Well, no, that's not it. If you think about a country like Rhodesia, such as it was very briefly, the priviliged class was a minority in that instance - descended from the colonists on authority from foreign powers to establish themselves at the center of the legal and cultural infrastructure. And that's what it's all about.
Japan is super complicated because their power structures were built during a period of their history that can be generously described as downright confusing. However, a lot of relics of that still survive - people descended from prestigious clans of the singoku jidai face a lot of perks and the Barakumin still take the brunt of the discrimination - and then on top of THAT, you've got racial, ethnic, and religious discriminations to talk about.
But, my point is, it's not just about majorities - it's about dominance, as granted by cultural structures that were built historically.