Yes, I'd be worried.
McVay/Sneed are smart guys.
This hire was done to protect them.

It's not a home run defensive hire. If it was, then all eyes would be on McVay/Sneed for the Stafford Trade.
Many eyes are on them already.. but they have an "out". His name is Raheem Morris.

All they need is for Stafford to do better than Goff.. which isn't much after last year.
Then, any shortcomings of not getting to the super bowl would be on the Defense.

It is hard to improve on a #1 defense. You will see stuff that shows that the offense has improved in rankings from last year. But the Defense can only hold or drop.

If they hired a proven, well regarded DC, then things would be different. But Morris is setup to be a scapegoat for McVay/Sneed.