The general principle is that the income is earned where the work is performed, not the country from which it is paid for.
If HMRC argues that for whatever reason (and non-declaration of departure is not generally a "reason") that you remain taxable, you should seek credit for your federal and state tax on that income. You could ask for "competent authority" -- i.e. IRS and HMRC to decide who gets to tax, or tax first, but that seems a useless exercise and tends to antagonze the tax people.
Do a little research (I'm sure it's in the archives) on how you can continue to get NIC credits cheaply, preferable (I think it is) Class 2, voluntary Class 3 if you have to.
By and large the UK State Pension is a bargain. If your life expectancy is short for any reason, fuhgeddaboudit.
Sorry for the slang. As they say, I'm not a regular, only here for the beer, and only this week.