I used to have a hard time with that. But only because I was genuinely afraid of someone breaking in and killing me.
I think the best thing to do is occupy yourself. Because if you sit around staring at the clock and counting the minutes until they return it will be torture. This site is a good distraction because you can interact with others, or if there is a good movie you have wanted to watch maybe take the opportunity to watch it. That is what helps me. Having something else to focus on.