I think it depends on where you grew up. We called them chili peppers to differentiate from sweet peppers.
I think that they are just different varieties of peppers (chile peppers, sweet peppers etc.) so some people call them chiles and some people call them peppers, but both words are technically correct (think Red Hot Chilli Peppers

)... as phatbeetle mentioned, what you prefer to call them will likely depend on where you grew up.
Growing up in the UK, the only 'peppers' I came across were either as part of 'salt and pepper' or as red or green sweet peppers. I actually had no idea that chiles/jalapenos were actually referred to as peppers until I moved to the US (at first I thought they were separate things).