How thoroughly unpleasant!
I don't have the ID card - am one step ahead of the HO, thankfully (= I'm on ILR) - but the last time I came into the UK with my UK partner (I'm on an unmarried partner visa) I had no problems going through the UK/EU line with him. And I'm sure that other people on this board have done the same sort of thing. What I'd suggest, not that it will necessarily help if you get a dud IO, is to ask the person who hangs out at the front of the immigration area whether you can go through with your husband - at the very least, if you get hassled, you can say 'but that person said I could' and maybe the two of THEM can squabble about it.
Might want to travel with not only the letter but a raft of printouts, helpfully highlighted. Not that you should have to, of course! Sheesh. And yet ID cards are supposed to help keep Britain safer.... *wanders off muttering*