What I was meaning is the ones who wouldn't go because they don't believe that way. For example, hubby doesn't believe anything, doesn't even claim atheist because he feels in a way it is its own form of religion. He would never go into a church because he doesn't want to be involved in what the building stands for and what is practiced inside. He won't even go into one for weddings, for that reason.
I wouldn't go as far as your hubby, as I said, i was brought up CofE, was an altar boy and church sacristan, sort of a family tradition! My parents always said that when we were 16 we could then choose whether to go or not...I continued til I was 18 but that was because of a girl who went!

Anyway, now I guess I'd consider myself athiest if pressed as I don't believe but probably take your hubby's viewpoint that, that suggests something as well - the best term I ever saw was Apathiest - basically you don't believe but you don't really care either!
I don't go as far as not going though, I'll go to a wedding or funeral because you are doing it for the other person, at my dad's funeral I even took communion, I can't explain why I did it, I just felt at the time it was the right thing to do for my dad. At my father-in-law's funeral I didn't, and nobody commented on it - in fact I think there might have been more comment had I done so as my brother-in-law is a minister and they know I am a non-believer, they were just pleased I'd gone. I asked him and my siste-in-law about certain aspects of the service, and gave them comparisons to ones I'd been to in England.
And I love going in old churches and cathedrals to look around.