He complained about Hermione to his dad before, in which his dad made clear that he should be ashamed that a "mudblood" had better marks in all subjects than he did. How do you think he'd respond if he allowed the same "mudblood" to punch him in his face? Not to mention that punching is a Muggle method of fighting. I don't think he'd like to to be embarrassed like that again, only with his father replying that he should be ashamed he allowed a "mudblood" to overpower him with a Muggle method of fighting like that.
Caption this Meme
