I made a meme recently that - without naming the culprit - made fun of a troll for making a BS claim on a meme of mine. It got featured quicker than any meme I've ever made, which surprised me to no end. The next morning when I logged in to imgflip it was back to 'submitted' despite many upvotes and comments on it. Someone discovered it and thus: troll swarm.
I suspect that moving up the hot list is based on an algorithm. The number of downvotes may affect that. I have no proof of this though. SirMemeAlot is right that memes can get downvoted out of submission. Downvotes on a comment can move it to the "low rated" status where you have to click to show.
It's happened to me several times recently when I chose to disagree with a couple notorious users. I had to beg for upvotes from friends, flag threatening evidence, and e-mail admins with copied evidence in order to get out of probation hell. Right now I and a couple others are trying to get downvotes on a timer so that trolls can't abuse it to wreak vengeance just because you don't think like them.