"Still took him six books." Well, he could only have done it after the fourth, in which he was resurrected. However, he doesn't want Hogwarts, he wants Harry Potter. So he thinks to himself, "What if I lure Harry out of Hogwarts?" This is because Dumbledore is at Hogwarts, and that's why he said Hagrid said it's the "safest place ye' can find." Voldemort lures Harry to the Department of Mysteries in the fifth, but Harry, of course, escapes. In the sixth, his main objective was to kill Dumbledore. Why? Because Dumbledore is the only one he was afraid of. So, Dumbledore was the only one standing between Harry and him. Snape secretly makes the unbreakable vow and kills Dumbledore. Not because of the vow, but because Dumbledore says so. With Dumbledore out of the way at the end of the sixth book, he could only attack Hogwarts in the seventh.