No, it's fine.
What gets on my nerves is when people call others transphobic for criticizing trans people who did something wrong.
I remember I saw this online argument about how one guy was a gunman who killed 24 children, and someone defended him saying he was the victim because he was trans. Other people disagreed, because how is being trans linked to committing mass murder?
Worst part is, as long as they're in a minority, some people will defend them and accuse the other person for a hate crime they didn't commit