I think it had more to do with his Son living in New York, Brady could stay on the East Coast and be much closer, plus his wife Gisele (2nd wife) was not a huge fan of Los Angeles because of all the attention. In the end, IMO, taxes don't make that much of a difference because in California, you are more likely to receive endorsements that pay you more then the tax implication.
Look to the average fan or even an above average fan, lower taxes is always a plush, however, in this case I don't think it was that big of a difference. The East Coast, where weather was not a trade off and Bruce Arians as Head Coach sealed the deal for the Buccaneers IMHO.