Here's how it can break again.
Basically, villagers can do pretty much whatever they want when it comes to work stations if you haven't traded with them yet. They can even steal work stations that villagers who HAVE been traded with are presently using without even touching them
That's part of the reason it broke before, jobless villagers were taking work stations they weren't supposed to.
My solution was to give temporary work stations with a slab on top so I know not to trade with those villagers because I have later plans for them, hopefully this'll keep them from trying to steal from actual important villagers.
I haven't removed the mound of dirt under the farm yet so there is still potential for the golems to spawn where they aren't supposed to, which could also potentially be a problem.