you're probably almost 100% spot on about Shrines and Towers, Guardians probably went back into the five giant pillars that surrounded Hyrule castle, which is where they came from, not the observatory. side point, the Towers purpose was apparently to detect the arrival of Calamity Ganon, and a lot of good they did.