Gregory knows how to hide pretty well, which could throw any of the Funtimes off his location.
He doesn't directly attack to destroy the animatronics, he sets traps to catch them in a position where he can catch them off guard and have the advantage.
He may not be able to stop them completely, but he can definitely cause them enough inconvenience that they think he's not worth it.