It's really not that clear, actually. The laws of your state may be clear (lemme guess - yours is an open-carry state, with stipulations that no one can deny you your right to carry if you have the necessary permit?), which is all well and good. If you have a permit that allows you to openly carry a firearm in public, and that permit is valid in the specific place where you are (have you tried bringing it onto your local university campus, or into a city that has local ordinances that further regulate where you can carry? I wouldn't suggest it), you're entitled to protect that right. And hey, if you get a thrill out of making people uncomfortable and challenging whether they feel safe in your presence, two-fer!
I personally think you're being a d!ck to your neighbors. Just because it's legal doesn't make it the right choice. But you're totally allowed to make that choice, and defend it if need be.