It's racist currently because the red-neck idiots who wave it are racists using it as their symbol. Bit like the swastika is not inherently fascist, but the current connotations out-weigh its previous intent. Nothing is static, so citing 150 year old history is pretty moot. Not to mention the horror of the southern states during the first half of the 20th century.
-2 cents