By punching Neo-Nazis you only deter them from talking alone, so instead they will just be tempted to talk in groups and this will make it so the next time some Antifa members try to attack them they will just be a big group getting into a fight with another big group causing a riot, so it's only going to do more damage in the long run.