Yeah, that's literally where the word "sinister" comes from (means "left" or refers to the left hand in Latin), because the Romans (and many other cultures) associated being left-handed with being bad. If you're left-handed, you're weird/abnormal and thereby arouse suspicion so you must be a bit sinister.
"Dexter" means "right [hand]". Like Dexter's Lab. If you're ambidextrous, you have two right hands so you're extra skilful. You can't be ambisinistrous (well, you could be, but you'd be really clumsy). If you have "two left feet", you're a crap dancer.
We also have "gauche" to refer to a socially awkward person. Just means "left" in French. And we say stuff like "that's right" to mean something is correct and "I know my rights", etc.
(And of course we say "straight" to mean heterosexual, ie the "correct" way – straight or direct being other meanings of the word we got "right" from. Stick to the straight and narrow path. Rectum is also related, because it's your "straight intestine". If you have a "leftum" it's probably very twisted so you should see a doctor.)
This idea has been around for a long time. So lucky Conservatives, I guess. They're always right. 💪