How much can possibly go wrong for a kicker?
He's had to readjust his timing. That throws his kicking rhythm off.
Delaying the approach just a nanosecond leads to blocked kicks. The rushers get there just that same nanosecond sooner.
Fizzled blocking up-front doesn't help either.
The kicker thinks: "Lessee... I gotta delay my approach, and the rush is coming and will get closer tyhjan it should, and I gotta make this kick regardless how the hold comes down, and I dare not do anything except trust my holder and work with him, and alter my game to fit what is... "
That's a heavy burden to carry into every kick, knowing that you'll be blamed for every miss.