I find that whatever line I stand in, they always moan. Just ignore it, it's just to see if you are compliant.
The last time we came into the UK, the border agent was completely mellow. We talked about my visa (you guys are right about the ILR), the BRP (he didn't seem to care I don't have one) and the Brexit results for many minutes. I would have thought he'd be bored to tears talking about Brexit but he .just wanted to shoot the breeze.