Correct, not having sexual attraction towards someone is fine.
A homophobe is someone who treats others differently because the person feels sexual attraction towards people of a gender other than the opposite gender
A transphobe is someone who treats others differently because their sexuality doesn’t match their assigned gender
Not feeling sexual attraction towards someone who is Transsexual is perfectly fine, as long as you don’t discriminate against them