It's not a hate crime to not date a trans person. If you say, "No, f**k you, I wouldn't date a tr*nny", that is hateful. But saying, "No thank you, I'm single" is fine.
The problem is people are specifically hateful towards LGBTQ+ people because of their gender/sexuality and not because of other reasons.