You said they were kidnapped. I'm pointing while yes, they removed from their kingdoms against their will, it was done in fact with the blessing of the black African leadership.
Ancient slavery, under the Roman empire for example, would not discriminate: slaves were both white and black (so were Emperors and Popes). In the middle ages, all European countries outlawed slavery (of course, Western powers retained countless "civilized" ways to enslave their citizens, but that's another story), whereas the African kingdoms happily continued in their trade. Therefore, only colored people could be slaves, and that is how the stereotype for African-American slavery was born.