You did the right thing by keeping your mask on.
The anti-masker at my wife's job (who'd been counseled twice before for not following mask protocol in the office) called in sick over a week ago - Covid. All employees working in his unit (including my wife) were exposed. She received a single test and, thankfully, received a negative result.
Now we just hope she didn't receive a false-negative and has been asymptomatic.
It's this type of attitude/behavior that will likely kill more than a half-million Americans by the time vaccines get fully distributed.
But then again...it's just the flu, right?