They lose Reynolds and Brudzinski after 1980. Inexplicably allow Ferragamo to leave in 1981. Try to remedy that with Pastorini in 1981 (sigh...) and in 1982 by acquiring Bert Jones who, well, didn't work out. As you said guys like Brooks, Perry, Thomas, Fanning, etc. rapidly showed their age.
And while they drafted some good players in 1980 (Johnson, Pankey, Irvin, Guman) and 1981 (Owens, Collins, Meisner) that would become key pieces of their 1980s playoff teams it was not enough in the short term to make up for all those losses/declines.