Yeah, I think persevering with the carrier would be a good idea. You are really not going to want him in the vets waiting room with all those dogs there, just in a harness.
Our older cat hates it too but I just bundle her in when she has to go and she forgives me once she's home! With the younger ones, we encouraged them to use the carriers as a safe place when they were brand new to us and so it's a lot easier.