Although I agree the downvote is abused, getting rid of it means that there won't be any way for users to downvote a comment that is uncalled for and trollish in nature. And the flagging feature is also abused on this site. Anyone who "hates" another user can flag their memes and ruin the other user's reputation on here even if they didn't do anything wrong. This happens a lot on political and religious memes that make crybabies diapers get twisted. If anything, the flagging feature should end, but in my opinion the downvote feature is fine as it is.
Back in 2015 they removed the downvote bar that literally showed every downvote the meme received. They did this to curb some of the trolling and at first seemed to work well, but as trolls go, they adapt and find other ways to troll.