Ok then. I think it's because people don't recognize that we are all humans made in the image of God. Even if they aren't prejudiced per se, they may not be exactly "comfortable" with people who are "different from" them.
Another thing is history. Us humans have a LONG track record of not responding well to those who are "different" and "invading our territory". And so, because of that, wounds have been given and received, and cultural gaps become wider and harder to bridge.
As I write this, I realize that you might've not been referring to racial diversity specifically. But I won't cover other types of diversity in this comment. Have a great rest of the week!