We recently did this and it looks like you're on the right track!
After our wedding we had planned to go over to the states for a month, which gave us plenty of time to get things sorted. We began by changing his social security card because, as Margo said, it could then be used as evidence elsewhere. Once we got that back we went to the DMV (who didn't much like our UK marriage certificate) and after that we sorted the bank.
We didn't change the passport until we were back in the UK, but this was an easy process. Be sure to remember that once you have changed your passport you're required to update your BRP within 3 months. As KFdancer said, it's expensive and takes a while to process, so bear in mind you may not be able to travel for a while once you've set the ball rolling.