I'm not a downvote fairy, if they make a good comment or meme I up-vote; but I tend to judge them more harshly i.e: a comment that I'd normally leave alone because it's not good enough to up-vote nor bad enough to down vote I still down vote.
Perhaps it's because I expect more from them? I don't really know.