The full quote is; "If you have a problem figuring out whether you’re for me or Trump, then you ain’t black.” Biden was implying that black people have already made up their mind and probably going to vote for him because he thinks he's in their good graces. Yes, it is a gaffe, and one that might cost him votes and not just from black voters.
However, given Trump's current ability, or inability, it is fair to say that Biden is still a viable candidate and that a single gaffe (out of the dozens both men seem to spew on a daily basis) is not much of a reason to leave the Democrat party.
The mindset that I've seen from Republicans, especially Trump Supporters, is that if you acknowledge skin color then you're racist. Of course, this is an exaggeration of what the Left is doing, so it's perfectly understandable to take this position but not from a place of sincerity.
At worst, Biden is a poor attempt by the Democratic Party to prove they've not become full on socialists.