I know what it means. But I'm not going to Agree to Disagree with something that's factually false. The target of the joke, says it's not rude. That's the fact here.
You want to think it's rude - which is an opinion.
If you want - I'll agree that it's not rude, but that you're entitled to your opinion to think it is rude - regardless of the overwhelming evidence that has been presented otherwise.