Unfortunately, I think you do. I remember several times where I checked my points, came back later, and actually had less points than before. Others have told me it's probably downvote trolls; they try hating on a person to bring them down as much as possible