The damage the Nazis did was not done by those who stood by and did nothing, it was done by the Nazis. And if those who stood by and did nothing did fight the Nazis, more damage would've been done than if there had been no Nazis in the first place.
That said, I wouldn't characterize those who stood and did nothing as complacent good men.