Actually one time when I moved it did matter. The new GP would not accept me as a patient without it, regardless of the fact that I am British. It took her about 3 months to confirm with the local health authority (or whatever it's called now) that I had an NHS number (because they couldn't find me in the "system") and I was without a GP that whole time.
That's ridiculous! I hope you complained, I know I would've.
What I have found surprising is that my records weren't retrieved from other health authorities. My last surgery, where I was previously registered 13 years ago, could find no medical record for me from the years between 1993, (when I first got married and moved to the other side of the country) and 2006 when I moved back to the area.
Granted, I had lived abroad for five years in total out of those years, (though not all at the same time or the same place) but, even so, I had attended ante-natal classes and seen midwives there in '95 though it wasn't my permanent place of residence and I had given birth in the next county at the only major hospital in the area while under their care!
If my doctors from 1993 - 2000 had been army docs then I possibly could've understood it but that wasn't the case.
I am most surprised, what with all the moving around I've done, that I've encountered absolutely no complications with any of this stuff at all.