I thought about the lyrics a bit and I maybe see what you're getting at. "Little girl" didn't always specifically mean a child.
Little can refer to someone's height or weight. Think of it this way, there's a phrase that goes "Ain't you a pretty little thing?!". That phrase, while maybe a little outdated, is talking about a girl's or woman's appearance and weight/height being pleasing to the person who said it. In this context, "little" has nothing to do with her age.
Also, back then if someone is "your girl" it meant that you were exclusively seeing each other like you are a couple. Even today you have the term girlfriend which also uses girl in the word and doesn't refer to someone's age. Adults don't call their partners their manfriends and womanfriends...
For more context, the song it's self is from one of Elvis' movies and both love intrests are not minors.