I've felt this as well. We've had a bit of an argument because of it as well. There are things that culturally, I as an American, do differently. I shouldn't be faulted for not knowing unless he tells me.
One example....
We had a bit of an argument over the holidays because I was sitting at dinner next to one of his friends and I patted him, his friend, on the back. I was told that I was flirting. I disrespected my bf and embarrassed him in front of his friends!?
BF saw me do it and it wasn't until someone mentioned it to him later that he got furious. Of course I WASN'T flirting!!!! I didn't even remember doing it until he mentioned it. That's how insignificant it was. The entire table was telling a story to me to embarrass the friend. I patted him on the back as an "It's okay" kind of thing...two quick pats...that was it.
It was made clear to me that people DO NOT touch each other in casual conversation and I should have NEVER touched him. I am an "attached" woman and it was inappropriate behavior. The exact phrase that was used, "I know you all do this in America, but here it's not appropriate!" I find this confusing since we kiss each other on each cheek as a greeting??? IMO that's a bit more personal than a pat on the back.