No, it's kinda just the default that American society uses when nothing inherently girly pops up. For example, a stick figure isn't really female until you add something feminine to it. I guess it has to do with females usually being more complicated. Never really bothers me when people think I'm a dude, and I've done the same thing a lot. So no, you're not the only one, and no, it doesn't make you sexist.