No more racist than saying all whites are oppressors.
If it's okay for blacks to have their own "spaces" away from whites who hate blacks, then why is it wrong for a white guy to say the same thing? Why not just everyone go back to segregation policies and this time be okay with it?
Two wrongs don't make a right, but what's right for the goose shouldn't be wrong for the gander, either.