I had a refusal for ILR last summer (don't ask, I'm still sore) that was entirely the Home Office's fault. They even wrote me a letter of apology and refunded my application fee. However the first time that I re-entered the country from a short weekend trip abroad after the refusal, I was detained as well.
The IO investigated and my story checked out. But I was still served an IS51 (?) or something while they did so and was held back for about 30 minutes. When he came back, it was all fine, but he told me that I would continue to be detained for the next three times that I re-entered the country UNLESS I could get the Home Office to remove the flag on my record. Apparently any refused in-country application results in a flag, or at least that's what he told me.
Anyway, I immediately sent off a letter to my MP who followed this up with Jacqui Smith. I then got another apology letter in which I was told that the flag had now been removed. And ever since then, I've had no problem on the subsequent five times I've re-entered the UK.
Moral of the story: Write a polite but direct letter to your MP asking the Home Office to remove the flag. Mine was cleared in less than a month. Hopefully that will be the case for most people. But if not, be sure to carry all your relevant docs when you re-enter the country just in case. I did for the next few times, even though it turned out I didn't need them.