Welcome to the forum

.
After this is done I plan to apply for a Spouse visa via VFS (possibly priority)
Just to clarify:
You apply for the visa through the UK government (UKVI), by way of an online application submitted on the 'visa4uk' website and a paper appendix form downloaded from the UKVI website and completed by hand.
Your biometrics are taken by US immigration on behalf of UKVI, while VFS are nothing more than a commercial partner website which allows you to purchase services offered by UKVI (i.e. priority processing) and UPS (i.e. shipping labels).
We want to spend Christmas together in the UK. Do I have to wait in the US whilst my application is processed or could I visit the UK for say 2 weeks at Christmas to be with him and his family whilst my Visa is being processed?
As has been said, you have to mail your passport to UKVI along with your application forms and supporting documents. So, you cannot leave the US while the visa is processing.
If you apply for the visa using priority processing in November right after you get married, you might get it in time to move to the UK for Christmas, but if not, you'll have to stay in the US until you have the visa.
In regards to booking flights for your actual move to the UK, DO NOT book anything until after you have received your visa and have checked it for errors. You will only have a 30-day window to enter the UK once the visa has been granted, and you don't want to book flights and then find either the visa isn't issued in time to catch them or your flight ends up being outside 30-day window.