How to stop the push rush play.

I would send flying missiles.
Put some heavy duty shoulder pads on some backup players and give them a running start and fly at the quarterback.
I wouldn't care if they got a penalty.
Make teams think twice about playing rugby in a football game.