I understand what you are saying but with "Freedom of Speech", as with any "Freedom", (like freedom to drive) comes Responsibility.
A Person shouldn't just come up to You and Accuse You of Murder, rape, molesting, arson, white supremacy, racism, hatred, etc. without hard evidence. Implying You are indeed guilty of what they accuse you of.
Sure, it's EASY to throw around allegations, charges, assumptions, or unjustifiable opinions without liability whatsoever of one's indictments but that doesn't make it RIGHT, Factual or Correct.