That is infuriating but it is supposed to do that. It takes a certain amount of damage to break and holds the damage. The strat is to be the second one to hit it or hit it multiple times.
Me: *sees Smash Orb*
Also me: *hits Smash Orb with 100% of my power*
Smash Orb: *doesn't break*
Some other guy: *delivers a very weak punch to the smash orb*
Smash Orb: *actually breaks*
Me:; Bruh