If that were the case, you would receive fewer upvotes for controversial memes. Users who would normally upvote might get scared away because they might receive ridicule for their vote.
The only thing I think it might be useful for would be to determine who is using alt accounts to upvote their own memes.