That's actually explained. When the spell bounced off of Harry, Voldemort was turned into a mostly dead being forced to live off of others. He couldn't have.
IF VOLDEMORT REALLY WANTED TO KILL HARRY POTTER ON THE NIGHT THAT THE SPELL DIDN’T WORK; HE COULD’VE JUST PICKED HIM UP AND THROWN HIM OUT A WINDOW GIVEN THAT HARRY WAS A LITTLE BABY