While I haven't used XE, I have used both bank and 3rd party FX exchange companies. I would say 3 - 7 business days is standard. I normally used Interchange FX and they take about 3-4 days. The cost isn't transfering money to the exchange company's account in the UK, it's the cost of them sending it from their account in UK to the other country. Transfers in country are generally free.
FX companies make their money one of two ways. Charging a fee and giving you a rate close to the interbank exchange rate or not charging a fee but giving you a slightly worse rate. Or if it's a high street bank, both! Sometimes sending more, say over a few thousand pounds will give you a better rate and waive any fees. Certainly, if you transferred every month or more frequently you might end up paying more than if you did it every quarter or half a year or whatever.