We'd still have the problem of the whims of voters in a small handful of swing states effectively deciding our national election. That's the biggest problem. Electoral College electioneering actually effects policy in significant ways, no more evident than now, I think.
Remember these curious Trump tweets, in apparent support of the nation's first significant anti-lockdown protests from 2 or 3 weekends ago?
These protests took place in several states and 47/50 states were in some sort of lockdown: All three of these singled out by Trump are large, purple states with Democratic governors.
Coincidence? I think not.