While I am in the minority and don't think the Rams would have done as well if Vermeil stayed, (i.e. Chiefs situation where they declined after he build it up and never made it back to the playoffs after that 1 game)
I still think he does belong in the Hall Of Fame and his career is similar to Kurt Warner's.
2 Superbowl appearances with 2 different teams and rebuilt both of them. Had big donut hole in the middle of his career (while in Vermeil's case, it is because he left coaching and with Kurt, it was mostly about his health)
But being the CEO of 2 bad teams and making them winners is a great thing. It is unfortunate as you say that he bailed both times because it would have been interesting.
I think his greatest strength is leadership and organization skills versus the actual coaching of football. He may have been a top executive or GM as well.
But the bottom line is, I think he deserves to be in the Hall of Fame, because what he accomplished, like Kurt Warner, really hasn't happened very much if at all. 2 great rebuilding jobs.