Against the top teams that could and most likely will be the difference. You can't give up a quarter+ to Arizona... or GB, or Dallas, or TB... or for that matter the next tier of teams down.
McVay is absolutely right being concerned about it. Even with D progress if they can't get the O into all 4 quarters things could get very tough.