When he played on torn cartilage basically ending his career, that secured his doom.
These guys that don't have the longer careers of success needs to be at the very top of their game in the lesser time period they were good.
Kurt Warner for example, doesn't have career stats but was at the very top from 1999-2001, somewhat the same as Marshall Faulk who for the most part had a career like Warrick Dunn and Tiki Barber, but for those 3 years he was one of the best ever.
Holt never really had either. A dominant few years or a long career. He was just very good for a very good team for a shorter period of time.