I think if anything the downvote button should only effect the actual meme itself, but have no effect on comments. This would keep trolls from raging against people who have a differing opinion from them and would like to express it. Any more people cannot or will not say something negative in a comment because they know they will just get attacked.
So to take a look at how things are right now, if one was to take into account the number of views their meme gets with how many upvotes that meme gets during a certain period of time that it has been featured, that can tell them 1 of 2 things. If a few hours has gone by and there is a low view count that tells them that it got placed in a spot in which visibility of the meme was low and therefore can account for the low upvotes on the meme, whereas a high view count with low up votes would show that not very many people enjoyed it.