Problem is this is a conflict of absolutes. There are women/girls who only want to use the bathroom with other women/girls and there are men who want to use the bathroom with only women/girls (and other men who think they're women, presumably). This is a situation where both sides cannot get what they want, one side must necessarily trump the other.
So how do you resolve it? My solution: use which ever bathroom facility makes the most number of people comfortable. If you own the bathroom, then you should enforce whatever rules makes the most number of people comfortable. If that happens to make a tiny subfraction of the population have a sad, so be it.