Or they say it even if they don't know what it means.

You don't have to be offended by something for it to be racist, it's still racism if someone says something racist at someone who is a different race from you.