No, it doesn't work like that. That would be used as an excuse to not engage in introspection and compassion, so that you could justify being inhuman and shutting out people that desperately need help.
The stories I hear on a daily basis are absolutely horrific. Mindbogglingly so.
Every morning at work we sit down to process the new referrals to the clinic, to see which professional will take which case.
It goes like this:
Blah-blah is from blah. Their family/loved ones were killed in front of them. The killers raped their family member/loved one before killing them. The killers then raped them too. They then jailed and tortured them. They escaped and found some smugglers. The smugglers sold them into slavery. Their captors beat and raped them. They escaped from their captors and were captured by police who jailed, tortured and beat them. They were released after some time and found other smugglers. They were captured by police while trying to cross into Greece on an overloaded dinghy whose engine cut out in the middle of the sea. They were returned to the country where they were previously enslaved, raped, beaten and jailed. They found new smugglers and managed to cross into Greece. Now they are living in a prison camp waiting to be processed. They face the possibility of being returned to the abovementioned countries they managed to escape from.
This story applies both for the men and women that I treat.
I listen to this EVERY MORNING.
Now, granted, we generally see the worst cases. But there are a lot of them. And there are a lot out there that fall through the net and do not receive treatment. There are others that are lucky enough to have friends that help them deal with these issues, so they don't come to us for help.
No, we cannot refuse to help these people.
If we are on the Bodhisattva path it would be a violation of our vows to not receive and help these people, no matter how hard it may be for us to do so.
The indignant, angry, exasperated and frustrated (white) types that prefer building walls and fretting over losing their privileges can just suck it up and blow it out their hole for all I care. Their fears are unfounded, pathetic and irrelevant.
Now if somebody else wants to go off and console their butthurt white ego, please feel free to do so. I personally have more important things to attend to.