27 million whites were counted in the 1860 US census.
8 million lived in the slave owning states of the South and the North. Maryland, Missouri, Delaware, and Kentucky were called border States of the North that allowed slavery.
Of these, 385,000 people owned about 3 million slaves who were counted in the census. Statistically, 4.8% of all Southern whites owned slaves. The average slave owner owned 8 slaves, however the average plantation had 20 slaves. One Union Soldier died in the Civil War to free 12 slaves