This is about white privilege of the people on the right today not wanting to acknowledge slavery existed.
But 2Spirit is correct too. Most people didn't own slaves. Rich land owners like Jefferson did. He wrote about the immorality of slavery, but still wouldn't stop himself.
Further context: poor white people in the south went to war for the rich to keep owning slaves that they themselves couldn't afford. Much like conservatives today voting to give tax breaks to the already extremely rich so they have to pay more themselves. Don't make sense to me, but I also don't watch right-wing media.