Honestly?

True, every one is a person of color.

:)

Okay, still doesn't explain why not. I guess we need all sorts of euphemisms and replacement phrases so we can avoid acknowledging that black people are any different from whites (I mean in terms of skin color, nothing that really matters).

IIRC the justification is 'Person comes First" in 'Person of Color'... implying that their 'person-hood' is the focus, and 'color' is simply a descriptor

whereas in 'Colored Person" you're stressing that they are Colored first...

I usually mock it, by going one step further and go with 'Person of ______ Extraction" ie, African, Latino, East Asian, Island, Arabic depending the subject...