I agree with Beckett437, while adding that I have not seen the downvote button abused to any degree for almost a year. If you can prove me wrong, then 1) I will be surprised, and 2) instead of trying to get rid of a feature, you should try to find the source of the problem. There usually is a way to track down the 'trolls' behind the downvote.
The downvote button is designed to remove actively bad charts, memes, or gifs by popular consensus. Removing it would remove the popular opinion's ability to remove bad content from the site.