I found that I was looking down at their midsection rather than their feet. That made some think I was "checking them out." If I looked at their feet they wondered why I was staring at their shoes. If I looked them in the eye they took it as a sign of aggression.... Or worse, they thought they could talk to me while I was having a partial seizure and couldn't respond to them. Its like staring blankly at them. Yeah, listening to music is the best way to get out of stupid conversation. Just be careful not to look midway down their body, though. I mean, there's just no way of making neurotypicals feel better about THEMSELVES no matter where we look.
For me, it's either partial seizures or exhaustion. I had to practice looking down before it got to the stage of not being able to look away. People would start asking if I was ok but I couldn't answer them for several seconds.