I am the same way...I have always been challenged by remembering numbers: phone numbers, dates, combinations, jersey numbers, license tags, game scores, etc. The funny thing about that is I am actually pretty good at math but for some reason, recalling numbers is just not my thing. At least you can ID a player by body language, I just read the name on the back of the jersey and then link it to the number for the rest of the game.
An hour after the game I doubt I could tell you who wore which number. I attribute it to age now but honestly I have been that way since I was young it has simply gotten worse.