If you work in customer service, you can get attacked with anonymity all the time. Besides, is it really anonymous? I was a victim of downvote abuse once (2-3 days having to wait 2 hours between comments was the result). I know exactly who did it. They were even proud of it. I made and submitted a couple of pity party memes, received an enormous amount of moral support from the community, it passed, and I moved on.
The downvote button is a tool, and like any tool, it can be abused and misused. It's not always fair, but if not the downvote button, then the trolls will flag your meme so that it gets taken down while the mods investigate.
I would get behind a campaign to reduce the effect that downvotes have, but I don't think I can get behind the campaign for their complete removal.