I have spent a lot of time in Baltimore over the years and can tell you that in and around the stadium is very nice. There are hotels within walking distance of the stadium and people are quite nice. It sits right next to Orioles Park at Camden Yards which is a beautiful facility.
Like any and every large city in America, there are sections of town to avoid. But in and around the stadiums it is quite safe. I would also encourage anyone going there to also enjoy the Inner Harbor where the National Aquarium is located. It is only a few blocks from the stadium.