Maybe 250 years ago.
That's like saying everyone was a slave at one point so we should use illegal immigrants as slaves.
Why do you want "brown people" in America when there is so much crime, gun violence, sexism, authoritarianism, racism, discrimination, etc., etc., etc.....do you hate them?