Some flags and symbols have sadly been used throughout history as symbols of racism or other political ideas.
Remember that the swastika has a very long history and was only recently defiled through Nazi ideals.
I think that the Confederate Flag was invented specifically for the Confederates.
It's more like a symbol for rebellion than it is a symbol for racism.
However, popular culture and some groups feature it prominently as racist symbols. It's sad but unavoidable that people could take offense when seeing them. Just like all the fascist symbolism in WWII. (skull and bones, the two sig runes of the SS, the swastika, ...)
I'm afraid that most of those symbols will be forever tainted. Certainly in countries where they were abused.
Now there are multiple ways to deal with this, you either try to forget or you make the symbolism illegal. E.g. Almost all Nazi symbolism is forbidden in Germany today.
I honestly don't know which is best.