Does it really matter whether it's the government or other people doing the deed? Do you think a black person in American west had free speech before the onset of civil rights? Technically he could say whatever he wished but even tho the government would generally not do anything about it (at least not officially) he may well end up a target of a lynch mob, so it's effectively the same. Free speech means we allow anyone to speak their mind and we won't move to harm their person or livelihood on account of the opinions they express, at least that's the idea. Many people in US mix up the idea of free speech with First amendment, First amendment just enshrines in law that the government will not make laws that infringe on free speech, they are not one and the same.