I think it's one of those things where if you've thought about it, you probably shouldn't.
If it's something you always do unconsciously (like I'm a big poppy wearer, I always buy and wear at least one as soon as the British legion start selling them every year) you probably wouldn't have thought of it either way, but the fact that you're thinking you might look out of place or offend by wearing one means you probably shouldn't wear one.
I seriously don't think there's an issue with non-brits wearing poppies, my DB is wearing one. I don't always support war or the reasons for war, but I am very supportive of those who are injured or otherwise affected in the line of duty (and their families of course), if you feel like you want to make a contribution to support those service people then you should wear one with pride.