I don't think there's anything wrong with downvoting someone, and it certainly doesn't do them much harm (I mean, maybe a few of their recent submissions could do very, very slightly less well than they otherwise would, if the downvote caught them at the right time in Latest, and they didn't have many upvotes, but that's no big deal).
But it does seem like a strange way to spend time.
And it seems especially strange to tell them you're going to do it.