I have solved it! The house isn’t made of flesh. He isn’t made of house. The GingerBread men simply live in a universe where atoms are significantly larger and there is only ONE type of matter. Therefore that universe has no understanding of it. They just ARE there is no material only matter
"all things need shelter, a thing that is known,
So a surplus of dough served to fashion their home.
But one stops to consider a prospect quite grim,
Is he made of house or is house made of him?"