Back to the OP, I definitely have the same feeling of just not wanting to talk or just get by with a mumbled "hiya" or "sorry," both of which I can manage to sound like a native while saying. It's less out of negative reactions and more out of just wanting to be left alone to live my life and not answer a load of questions about where I'm from/why I'm here. People are usually just polite and inquisitive but obviously that stuff gets old really quickly.
This. Exactly this.
I've been here for three years now and I feel the biggest change in my way of speaking is that I enunciate more. Or so I thought...
I went home recently for an extended trip and pretty much all of my friends and family have decided that I now have a British accent. Even people I didn't even know that well, or at all, felt the same (waiters/waitresses, my hairdresser, etc). There were a couple of times I got a, "So, where are you from?" from cashiers in checkout lines. In my hometown!
Then I came back here and, because I work in a predominantly customer-facing environment, I am bombarded with very much the same types of statements/questions, only that I sound Canadian. Sometimes people ask which part of Canada I'm from, and I jokingly reply with a huge grin, "The Texan part," to which the asker turns red and apologizes profusely.
There is no escaping it now. Outside of work, I just try to get through the day without speaking to anyone. If I can manage with a smile and a nod, I'm perfectly happy. (I could only be so lucky...