It really isn't. Whoever told you that is just showing their fear of people of color. If you really believe that we are all equal, then what you are expressing is a desire for antiracism: congratulations.
We are all morally and spiritually equal, but right now, our opportunities are not equal. Do you know that students of color get suspended or disciplined than white students, out of proportion to their behavior, which then damages their chances of getting a good education (how well can you learn from a teacher you can't trust to treat you fairly?) and their future prospects? See this:
https://usafacts.org/articles/black-students-more-likely-to-be-punished-than-white-students/
That's just a small piece of what's going on -- On the other hand, in schools where the students of color are treated fairly, the white students do better too. Antiracism is truly a good thing, for everybody. This broad study
( https://journals.sagepub.com/eprint/BGWXVTTADUHTE3TJAVTF/full )
concludes: "Drawing on survey responses from roughly 26,000 students, we find that racially diverse schools are associated with more positive social and emotional outcomes for all students. Strikingly, we find that these outcomes are most uniformly positive among white students, whose families have long represented the strongest opposition to systematic racial desegregation."
I am white myself, and I am very grateful for the chances I have had to learn from Black teachers, and to work with and learn from other people of color, and only sorry that there is so far to go before they and their communities get the full experience of equal treatment that is our country's aspiration.