That's exactly the point of downvotes. We post what people like.
We d o get downvoted on Imgflip, and we do lose points for it but never get to know how many.
It'd be a lot better if the images could subtract the downvotes from the upvotes, and we'd have a fairer representation of how well the image did.