The latter is the case, which is why Lincoln was so amazing; abolitionism wasn't that popular in the North outside of liberal cities like Baltimore. Lincoln actually had to pitch himself as a racist sometimes in order to secure their political support.
But the Northern states all agreed that preservation of the Union was more important than maintaining slavery, and their economies didn't rely on it like many southern states. Therefore they went along with Lincoln's masterstroke (the 13th Amendment) and John Wilkes Booth made Lincoln a martyr for equality and the North. Johnson then f****d everything up and basically enabled a century of Jim Crow despite the amazing work his successor, U.S Grant (my favorite President) did to secure equality legislation (14th/15th Amendments, the First Civil Rights Act) and crushing Southern resistance to black equality with his newly-founded Justice Department and even the military. (If you haven't already, read Ron Chernow's "Grant". It's AMAZING and changed my entire understanding of Grant as a man and leader).
Fun fact: Two Confederate leaders did genuinely seem to regret fighting for the South and worked closely with Grant to help combat the KKK and other racist elements in the South. James Longstreet of all people put in good work as (I think) a governor and cavalry leader John Singleton Mosby, "The Gray Ghost" served Grant as a U.S ambassador.
And they say history is dull...