It’s contextual. Example: A white kid attending a majority-black school is in the weaker position as far as being a racial minority. It’d be wrong for the black kids to bully him for that.
However, in a society-wide context, and certainly adding in America’s history of Jim Crow and slavery, it’s clear that blacks are in the weaker position.
If whites ever become 49%, they’ll be a plurality. Not a “minority“ necessarily. Other minority groups will still be smaller and weaker and more historically disenfranchised.
I don’t think speaking out against racism is the equivalent of assuming minorities are stupid and fragile. In fact you have already contradicted this idea yourself when you say “[racism] hurts everyone.”
We can and should speak out against racism even when it’s not directed at our own group.