Sure, the confederate flag is a symbol of racism for a lot of people. It should be taken down for that reason. But few people have asked a more basic question: didn't the confederacy lose?
Since when does a country allow its opposition to fly their flag on their property? A flag is not just a symbol of ideologies and purpose, but of entitlement and ownership. When the confederate rebels that tried to leave the United States lost the war, they lost their entitlement and ownership over the property of the United States of America.
The confederate flag is a just as much a symbol of a racist ideology as it is a display of entitlement.
Guess what. You lost. A long time ago. Take the flag and fly it on your own property. And be lucky that the United States of America lets you fly it there.