I agree that it would be a good idea to get a visa. I think it will ring suspicion bells if you say to the immigration officer, "I'm visiting my husband who lives in the UK, but I'm not planning on settling in the UK." IO's do more than just act on the letter of the law, they are allowed to use discretion, and refuse anyone who they suspect might try to settle without the proper visa.
edit: You are going to need a visa anyway when you eventually do move their permanently. Is your husband implying that you never need a visa? That you can go back and forth from the US to the UK indefinitelty throughout your married life with just your passport, no visa? (Sorry if completely misunderstood your post.) If you keep doing this, the odds are that eventually you will be stopped by immigration.