That's not right. I can definitely understand your frustrations about downvotes (who doesn't get frustrated with downvotes?), but it's not right to accuse people like that.
I go to the Latest page regularly. If I like a meme, I upvote it. If I don't like a meme, I'll either scroll past it or downvote it.