Yeah, it irritates me too (that's why I made the meme).
I think I get where they're coming from (slightly). That principle is okay if it's a matter only having to do with someone's personal preference (e.g. someone's favorite food, music, art, etc). However, if it's about a statement that contains a matter of fact, it's not okay to say that.
And the people who say that kind of show they themselves don't really follow that principle either, when they argue the point. If it were true, they couldn't argue against your view. :)