Due process also applies to me. If we allow any rights to be taken away from some mythical "other" group, we are losing those rights ourselves. Want to be arrested and locked up indefinitely because someone "suspects" you're a terrorist. After all it would keep "everyone else" safer. So what if you're innocent.
That is the ultimate endgame when you eliminate due process. And don't assume you're safe because you don't associate with "those people". The political winds may change and the "those people" that we "need to fear" could change to Jews, maybe Christians, a political group that you belong to perhaps? Remember, anything that can be done to "others" can eventually be done to you.