Maybe I do take this issue a little personally. I grew up in a majority black neighborhood. At school, I was picked on (spit on, beaten up, ect.) I was always treated as the instigator by the staff, even though I wasn't. (I was a bit of a dork, most of the bullying was not racially motivated, although some of it was) . I was accused of being racist from a young age because I was being picked on and attacked. I would get beat up, and then I would get detention, but not my attacker. So yes, I get defensive and "grumpy" when someone claims I get "special" treatment just because I'm white, because that's not how it felt when I was sitting in detentention for the crime of being spit on.
Once me and my siblings were attacked by a group of black kids (this incident was more severe than the school bullying). They even shouted "cracker" and "honkey". During the investigation, the faculty refused to entertain the idea that it might be racially motivated.
So yeah, when you're told, from a young age that you're horrible for existing, and that you deserve to be treated like crap because you're white, you get a little testy when someone tells you your skin color is a privilege.