I have been to three regular season games in Foxboro, MA (2001, 2008, 2016) and one preseason game (2010) and never had an issue. It's a pretty subdued crowd. I'm sure it's different when the Jets or Steelers come to town. But as a Rams fan at a Patriots game you're more looked at as a curiosity than anything...
Personally my worst experience was 1991 at the LA Coliseum vs. the Raiders. I was actually sort of glad the Raiders won on a last minute FG because I think the hostility on the way out would have been worse.
A few jerks at a bar the day of a 1996 game at Arizona but nothing major.