I don't think there is anything wrong with it, but I, personally, wouldn't want to.
It wouldn't be as enjoyable for me to go somewhere if I couldn't share the experience with my husband If there was someplace he really wanted to go to, but I wasn't particularly interested at first, I would still want to go to find out what it is that he likes about the place, and learn more about it and about him. The same for a place that I wanted to go to but he wasn't really interested in - I think he would want to go just to learn about what I like and learn more about me.
I don't think there is any place one of us would want to visit that the other one would absolutely not want to go to, and have to be dragged there against our will. If our interests were that dissimilar, I don't think our marriage would be as successful.
Like Andee, my hubby and I found each other later in life.