well is it your idea that most people in society right now are irrational?
most people wouldn't even come to the conclusion of hurting someone else just for littering, and if they saw someone trying to hurt another person, most would intervene. In a true ancap society, most people would be well armed as to uphold the NAP at any given time if need be, so they would intervene if they saw this.
for you see, I should have specified earlier, only violent NAP violations count for forfeiter of rights, littering is not violent and is actually most likely not meant to be an offense (as in it was just carelessness). so the logical conclusion is simply to tell the litterer not to do it again and hope they don't.