Spelling things the American way isn't going to make you look like an idiot. After all, these spellings were taught to us in school growing up, expected to be spelled minus the oddly placed U's in everything. Just because the British have a different way of spelling certain words, doesn't make it the CORRECT way.
I say don't worry about it, most Brits are aware the two countries spell things differently.