That is one of the platforms of the Libertarian Party. Not all libertarians agree with that and some don't even know about it. It is not that popular of an issue. The big one that the Libertarians were fighting was was the decriminalization of marijuana. However, there are a lot who believe that all narcotics should be decriminalized because they believe the war on drugs failed. It did fail because it was never fought.
I just think the damage to society huge. It is not an issue of freedom because narcotics rob people of their freedom. They are highly addictive and change a persons ability to reason. It causes them to do things they, otherwise, would never think of doing. That is not freedom, that is servitude to an addiction.