Welcome! Do you mind if I ask where in Switzerland you are and why you hate it? It's a place my husband and I are considering as a future place to live and this is the first negative statement I've seen from someone living there as an expat.
Oh my, where do I even begin? First off, I'll start with personal reasons and then go from there.
1. I am currently living in the Schaffhausen area, near the German border. While it is a beautiful country to visit, it isn't someplace that I have enjoyed living.
2. Not being able to speak the language (Swiss German) nor having much of a desire to learn High German, hasn't helped with liking things here either to be honest.
3. Coming from the US, the south in particular, the weather has been atrocious. I HATE the cold and since we moved here at the end of August, the warmth quickly left. There are very few if any sunny days in the winter. Mostly grey skies and fog. I understand that the weather in the UK isn't really any better but if it were ONLY the weather, meh, I could deal with it.
4. Life proceeds at a very slow pace. I'm used to things being open 24/7 and went into a bit of a shock coming here. Most stores typically close around 8pm M-Thur, around 9pm on Fridays and close at 6pm on Saturday.

5. The Swiss aren't the most friendly people around either. Again, not knowing the local language doesn't help.
6. It is EXTREMELY expensive here!!! To give you an idea, when we visited London for the first time after moving here, I found myself making comments as to how cheap things were.

I know, crazy right?!
7. Since I can't speak the language, the joy of shopping has been sucked right out of me. I used to love just wandering around and window shopping but that's not so much fun when you are reduced to looking at pictures.
8. The stores are small and lack variety.
So, those are just some of the personal reasons that I don't like it here. Now to get to the restrictions if you will.
They only give out a limited amount of B Visa's to non EU residents and since we didn't arrive until the end of August, we received an L Visa (Limited Visa). This means I was not allowed to get a cell phone (had to have my husbands company get one for me), I'm not allowed to purchase a car (luckily we brought mine over from the States). Nor are we allowed to get our VAT back when shopping in Germany.

We have to jump through hoops every year to renew our Visas and pay outrageous fees. Again, this is just a sample of the crap we have had to deal with. I'm sure there are others who have come over and would disagree but I can only tell you about my experiences here.
In a way, I'm glad that we came here first though. Things like the weather and different food won't bother me as much as I'm sure it would've had we gone straight to England from the US. Which btw, is where I wanted to go in the first place. NEVER wanted to move to Switzerland, which again, doesn't help with me not liking it here.