You named two.... Fort Sumter and Andersonville.... although I'm not too keen on visiting Andersonville.
Was on my way to Fort Sumter on my motorcycle several years back when they turned me around due to a possible hurricane hitting Charleston.
I have been to many battlefields and various other non-war related Civil War Era locations.... the battlefield that impressed me the most was Shiloh, in Tennessee. The reason is: As far as major battlefields, maybe any battlefield, it remains nearly identical today to the way it was during the battle. As you know... battlefields are now only partially persevered.... giving way to malls, housing projects, etc.
I've been wanted to return to Gettysburg because when I was there a project was underway to return many of the famous sites on the Gettysburg Battlefield to it's original condition during the battle... meaning clearing trees, brush.... planting orchards, grains... lots of work! I have heard nothing of their progress in this endeavor and I'm afraid to research it fearing that like most other historical renovation they most likely ran out of funds.