Well, the impairment itself does play a role, but frankly, having an impairment myself, I experienced that when I accepted my impairment I could deal with it easily, but you know the trouble? Accepting there are things you simply can't do, and hiding behind it are two different things, and people around me were hardly able to see the difference. You can't teach a blind what "red" means, and like that there are things you cannot teach me, and which you shouldn't even bother trying, and yet everybody demanded otherwise, against all impossibility. When I came in contact with others, this issue came clear is one of the must common issues, people dealt with. Along with "good (red: bad) advice" and so on. And of course, everybody is impaired in their own way... We all have our weak sides.... Only some impairments have a bigger effect than others, and lead to bigger limitations. And when people don't know enough about them, they easily judge, based on their ignorance and then you come to the point to getting either hurt, locked out of society or maybe worse, or a combination of all that.
Yeah, a half paralyzed face can be a reason for people to act odd. When I was a boy I remember a girl one class lower than me had a port-wine stain all over her face (and when I coincidentally saw her changing at a swimming pool, I even saw she had such stains all over her entire body), and some people did "tease" (read: bully) her about that. Now frankly all I knew about her was her name (which I won't reveal for obvious reasons) and what she looked like, but now that I look back on this as a man, I wonder who the girl behind those port-wine stains was, and if she thinks she's ugly because of those stains. If she suffers, it's the bullies who made her suffer, not the stains themselves. I never saw your face for real, so it's hard for me to say how I would react if I saw you "goofed face" as you call it, but you know what it also is? Some people are aware they may hurt you and look away afraid you might think they're staring at you, and people staring at you is al well known miserable experience too. Maybe playing a devil's advocate now, but I know that people always act in the extremes in situations like these.... Perhaps that's the most important lesson humanity has to learn... How to act to impairments, illnesses, or whatever... :-/