I think the meme just falls a tiny bit in the rankings if sorted by "Hot." It has no effect when you sort by "New." There is no way to know how much a meme was down-voted, as far as I know.
So, down-voting it's mostly harmless and therefore useless exercise. But: if something grinds my gears then I'll hit that down-vote button, and if something really, really grinds my gears, then I will post a comment or reaction meme saying that I down-voted it.
Although it's kind of silly, I do agree with this system. I think if the number of downvotes were made public, or if the screen-names of the ImgFlippers who down-voted were public, then things could get ugly here in a hurry.