Scientific answer? It's too long to explain here, but I read some interesting theories about it. Most boil down to sex (they all boil down to sex)
But my point is it's because they are hidden. But unlike our and your hidden genital, the boobs are hidden, but not. Covered with clothing, but still out front and center. I use to have a girlfriend that was bartender at a strip club. I noticed then that everyone wanted to see the dancers breasts, until they were out....nice to look at, but not really what holds our interest.
I work with a woman that one day I asked her how she felt having the smallest breasts in the building, she laughed because she was the only woman on that shift and we have a lot of big boys on staff. She even went over to stand next to one of them and yelled back "you're right!" My point? men have breasts too, but for some reason they aren't considered sexy.
Caption this Meme 
