Don't you mean WHITE PEOPLE used to call black people colored people? Since slavery was abolished, white supremacy had no political affiliation for decades in this country. Black leaders started using person of color in the 60's in an effort to get away from being called the N word.
"If you're not OK with being called a "colored person", you should also not be OK with being called a "person of color"? - Says who?