Oh I agree. Where I come from, it was almost always a racist insult (or a classist insult, when it came from white people). But the problem for me was that the people who were on the receiving end of the insult use it of themselves.
The n-word is still different, though. I know, it's a double standard, but it applies. Redneck, no matter how viciously hurled, will never have the stinging power of the n-word. It's just reality.