You've never met anyone who fears them?
So why are they acting like they do fear them
... or themselves?
Studies have been done utilizing gauges that detect increase in moisture (sweat, 'pre-chromosomal discharge'), electrical activity and blood flow in the gonadial region which indicated men who have no problem with those of such orientation showed no reaction when viewing 'gay' porn, while those who are homophobic all showed some degree of arousal. All.
Interestingly, the belongings of a surprisingly large number of ISIL members killed in battle have included homoerotic paraphernalia or had accessed homoerotic sites online.
Many Alt Right types, who are very like ISIL types in persona and ideals, also dabble in such material, despite, like ISILs, being vehemently 'against' homosexuality and whatever attached to it.
Many young men insecure or confused about their orientation or in denial of it lash out at those of homosexual orientation who are more open about their inclinations than these men are about their own.
So you can say that you are correct. The phobia is not of 'those' people, rather, it is a phobia of their own desires and what they are having difficulty dealing with about themselves.
It's like anybody - actual real intelligent people don't go around trying to make other people look stupid, stupid people who don't want to look stupid try to make others feel stupid.
Genuinely attractive people don't call others ugly, people who have to pile on the makeup and carefully coif their hair and workout excessively in order to feel like they look good call others, ugly.
There are two types of things people fear, that which is a potential danger to them (a grizzly bear charging), and that which reminds them about what they'd rather not know about themselves. That's what hate all to often is as well, just self loathing.