I've been there too - sometimes you just have to put up with it.
I have a valid visitor visa (due to a DUI in 2007) and the first time I travelled to the US on a visitor visa, I was sent to the 'back room', where I sat there for about 20 minutes and then got 'interviewed' about my trip, my previous travel history and previous student visa. I've entered the US on my visitor visa several times since then with varying results - sometimes I'm just waved through, other times the IO will tell me I don't actually need a visa, and then I occasionally get a few questions.
However, even without visitor visas in advance and just travelling on the ESTA, it can sometimes take 2-3 hours to get through US immigration. I've missed several connecting flights in the past due to the long wait.