Mr. Sowell's (totally unoriginal, by the way) thoughts on the detriments of affirmative action are a feel-good "just so" story but are pretty well contradicted by the evidence.
Mr. Sowell's thoughts on this, from https://www.newsmax.com/Newsfront/sowell-gop-blacks-unpromising/2013/03/22/id/495990/
--"Sowell says his new book developed from a previous work, “Intellectuals and Society,” and that affirmative action is 'one of the few policies that can be said to harm virtually every group in a different way.
--“'Obviously, whites and Asians lose out when you have preferential admission for black students or Hispanic students — but blacks and Hispanics lose out because what typically happens is the students who have all the credentials to succeed in college are admitted to colleges where the standards are so much higher that they fail.'"
Well? How should we measure that claim? Let's take a quick look at the data from Yale, which appears to be unusually transparent about this subject: https://datausa.io/profile/university/yale-university
Scroll down the page and you'll find a chart with graduation rates broken down. The available data goes from 2012-2017. The chart is interactive and you can play around with it.
Regardless of the settings you choose, you'll find that graduation rates by race group are strikingly consistent, and high. And in the case of 2017, actually a perfect 100% graduation rate among American Indians/Alaska Native.
How can this be so? Well -- hardly *anyone* drops out of Yale. Graduation rates are extremely high. There are all sorts of reasons for that, that I will skip over for now because this comment is already getting tl;dr. But basically, if you have the credentials to get into Yale, and lucky enough to be admitted, you are extremely likely to graduate.
The final thing I'd add is there are all sorts of non-racial "affirmative action" plans that greatly influence elite college admissions -- Alumni preferences, donor preferences, athlete preferences, etc. -- which end up claiming a lot more spots than the racial preferences.
Bottom-line: At least as applied to elite universities, the entire concept that race-based affirmative action is a detriment to the admitted students is just totally contradicted by the evidence.