When I lived in a non-English speaking country, I thought the problems understanding English speakers after living there a while had to do with learning a second language. I now think it has to do with rhythms more than anything.
It's funny though, now when I hear American accents, they often sound fake. My sister who lives in my hometown sounds like she's putting on a New York accent. My sister from Maine sounds like she's pretending to be an extra on a Stephen King film with all the downeast in it. And General American sometimes sounds like nails down a chalkboard (I never hear non-tourist Americans using it here, so none of you probably have it that strongly). The recorded voices over the tannoy in JFK make me blush. It's like a cross between General American and a southern belle.