I get what you're saying, and I agree.
Though, if you think about it, you wouldn't get offended if you just brush off the insult because you know that insults are meant to bait you into anger or sadness, and by taking the insult to heart you are taking that insult.
But that's only for tough people, not everyone has thick skin.
That's why we have to be careful with our words, as they're a powerful weapon.