I think you may be underestimating how much Brits want an immigration cap, like other western countries.
The 'right to family life' can take place in either country.
Of course, I understand the want of immigration caps and that a family could choose to live elsewhere--I suppose more than thinking that it is impossible for them to further restrict settlement visas, as I know fully that they can and probably will at some point do so, I'm thinking it won't be the first thing, and it won't be some massive, overnight change, just given how prior changes to settlement visas have occured.
That is strictly my assumption, and it probably isn't without a healthy side of wishful thinking.
I definitely see the writing on the wall, immigration of any kind is going to be limited in the UK and soon--I just presume that spousal, dependents, et cetera will be one of the later groups to be culled.