Well the same thing can be said about the secret attacks. You are on a public forum that doesn't use people's real identities to begin with. So we really don't know who each other is, so what does it matter if they attack in secret or attack to your face? You will never know who they really are, so you really can't do anything to them. Being a public site, people can use the site any way they wish. In all reality, the ability to vote on a meme or comment is a form of speech, so if you are really all about free speech, limiting how often they can use negative votes does in fact limit their freedom of speech. Whether it is alt accounts or people in a group like CTS that is casting the downvotes in secret, you are still limiting their speech by putting limits on it. The other issue I find with putting limits on votes is next you can start expecting spam comments from the people who want to do attacking. Say you limit to 5 downvotes an hour, so the person who attacks send 10 comments one hour, 8 comments the next, 12 the next hour, and so on. These can even be short comments or even single word comments. So since they can't attack with downvotes, now they attack with spam. What would you want to do at that point?
By being able to block them, you stop the attacks, both with what they say and with the downvotes, at least towards you directly. No matter what they will still find ways to talk shit about you to other people. And so what if people decide to crawl into their own little hole. Just means they have reduced the amount of points they will be getting and they removed nearly every reason to be on this site anyways. I would bet that if people lost their ability to attack others, both with words and with votes, they would give up and just leave.
Let me ask you this though. If you were to limit the amount of downvotes to 5 per hour, is that 5 per hour total or 5 per hour towards each account? So for instance, if you are going through the latest memes and find 10 memes that you disagree with and feel they should be downvoted, do you have to choose 5 of those to downvote and the rest are left alone, or if they are all created by different people are you still able to downvote all 10 of them? Then I guess the next question would be is that 5 downvotes total or 5 for comments and 5 for memes each hour? Really I am curious how far you have thought this out to how detailed you think it should be.