I've been having some troubles of my own about this issue and my husband says he has a hard time talking to people about immigration because their response is always: "well of course you would think something like that, your wife is American". Like our views don't matter anyway because we are "too close to the issue".
I have had a lot of acquaintances say the "we don't mean you" speech, which makes me sad, because I have a little more respect for people who believe in no immigration than people who voice racist opinions.
I found this video helps when in these situations- How to Tell People They Sound Racist by J Smooth.
How To Tell People They Sound Racist"I can understand that your opinions come from your background and experience and I respect that, as I hope you can understand from my background as an immigrant, what you said can been either seen as a hurtful comment to/about me & my husband, or as voicing a racist idea. Both of these things are personally offensive to me, no matter who you meant to describe: I am an immigrant and every comment you make about immigration and every law that is passed affecting immigrants personally affects me. And when you choose to talk about how you "can't stand immigrants taking over your country- but I don't mean you kind of immigrant", well, what you say sounds racist and that is not okay to say." and then you can maybe inspire a little education on immigration.
It may not change their opinions, but I find people think about what they say in front of me more often, and in the end, my feelings get hurt less!