You're not looking at things in context. A 100 rating back then was more rare than it is now. And Brees was #3 in the league with his 104.7 in just his third year starting. So, he wasn't just average or inconsistent when with the Chargers and it didn't take him until year 5 or later to blossom. Even in '05, he dropped in efficiency but he was still top 10 in the league. Only 2 guys had a 100+ rating that year.
Again, I don't understand what point you're trying to make with your examples. Peyton was a pro-bowler and top 5 rated QB in his 5th year. How is that the same as what Goff is doing in his 5th year?
You have to compare apples to apples and Goff is 22nd among his peers while Manning and Brees were top 5 -10. Their respective ratings can't be looked at in a vacuum, you have to consider context. In Peyton's 5th year, there was only 1 guy that had a 100 rating. Playing QB was different 18 years ago.