That doesn’t mean they have to have exact meanings transfer to the words.
The prefix in, which means “in, on, or not,” appears in numerous English vocabulary words, for example: inject, influx, and insane.
In some cases the prefix in can make words the opposite, like with insane, insane being the opposite of sane. Or change the meaning, like how flux and influx are different. Or do nothing at all, inflamable and flammable
Your logic for homophobic, and others words like it, being invalid is not sound. Prefixes and suffixes can imply a meaning, but do not require it. If you wanted to argue that path you would have to argue that inflammable means not flammable
