Shouldn’t be an issue at all.
Your spousal visa became invalid when you moved back to the US, as you did not maintain your residency in the UK, and now that it has expired, it’s irrelevant anyway.
So, now, you have to enter the UK as a visitor as you no longer have a valid spousal visa.
If they ask about the previous spousal visa, you can just explain that you decided to move back to the US (I once had a US student visa for grad school which I abandoned after a few months… occasionally when I enter the US on my visitor visa they ask about the abandoned student visa so I just explain why I decided to leave the US and it’s all been fine).