It's a valid point. I'm pretty sure you're not the only user that has brought that up. The best idea I've seen is to make downvotes cost points to the user who is giving them out.
I get the impression, the users who are most vocal against changing the downvote give them out like candy. Making it cost points would be somewhat amusing, watching downvote fairies go to zero, then they have to work for more points because they can't give out anymore, OR THEY LEAVE.
Whether there is a change to the system or not, we are all on the same field. I'm cool with whatever the owner of imgflip does, Im happy about being here and enjoy the fun I have here.