They take away an upvote behind the scenes. For instance I had one yesterday at 39 ups, a few hours later it was at 36. I guess if you pay attention, it lets you know you have haters :)
That's possible as well, I know I also had the conversation with Raydog more than a year ago. We were pretty much in agreement the downvotes were getting applied in some sort of delay. we were both finding memes and comments with deteriorating upvotes. Also Acuity12 admitted memes could get downvoted back to submission.
Yes, downvotes kill a member's upward momentum and can effect how many submissions you get. If a meme gets enough downvotes it is knocked back into submission which happens fairly often, especially to users who for whatever reason get on the bad side of trolls with numerous alt accounts. Downvotes on comments can also cause the comments to become "low rated" and not be visible. At times I've been targeted by downvote trolls and had numerous featured memes knocked back to submission and even friendly comments to my friends on my own memes become low rated and hidden. It's really ridiculous.
In the top few pages of Hot, downvotes are helpful to get the trash down more quickly and make more room for good memes to come up.
Downvotes are great when they're used responsibly - to help the best quality memes on the site become more visible. I wish more people used the downvote button regularly in the top few pages of Hot.