If I upvote your comment or meme and then decide I want to take the upvote back, you lose the point I gave you. Or if a bunch of people upvote your comment or meme and then they all decide to take them back, then you lose those points. That scenario isn't likely, though.
As for losing points, a moderator can take them away if you're caught cheating. It has happened to a few users in the past, but the chances of that happening is also rare. Usually they'll just delete your profile.