The way you're brain works doesn't change whether you can go in or take in. If a man cuts off his pp, I wouldn't care if he's in the restroom with girls because he can't do much harm to them. In most cases, though, potential harm to others outweighs the benefit of one dude feeling more comfortable sitting on one toilet rather than another.
Given what I believe about that though, I do see the potential for a transgender person who's crossdressed getting picked on and bullied or highly embarrassed going into the bathroom of the person's real sex. I would
As for what public policy on publicly owned restrooms could be to minimize rape risk and bullying/embarrassment, I'd be fine if schools had a 3rd bathroom that can be used by either sex to keep the girls bathroom safe from people with p**ises while letting those willing to take that risk and boys who wear dresses use the bathroom without getting bullied. Another way could be to have a bunch of single person bathrooms that aren't set apart for one sex rather than large bathrooms, but that may not be feasible for a large school with hundreds or thousands of kids.
I do realize that this is more progressive of a view than much of the right holds, but I do many right-wing people's position of, "The government must tell people what is true and false, and that must be defined on what I personally believe" way too authoritarian and dangerous. I want as little input from the government as possible while still keeping young girls safe from being raped in a public bathroom.