It is. I don't want to assume the worst, I don't want to be so traditionalist or closed minded that I pre judge them unfairly, but I'm not so politically correct or brainwashed that I forgo the possibility.
I was raised religious. We didn't like hearing about Catholic Sex Scandals, and we didn't like seeing people who opposed religion use that as ammunition to kick all religious people in the butt with...but we also didn't deny that there had to be at least some truth to it. Even if we met a preacher we liked, and he seemed like a swell guy, there was always a chance of "one in every crowd."
So telling us to not assume they're all out to molest children? Sure. But there's one in every crowd.