A final update which I forgot to mention but will do so now! After almost 6 months of waiting, in the middle of July, I finally got my residence card! Because of this whole debacle I didn't bother to go with my family on holiday earlier in the summer as I didn't want to deal with any more ignorant border agents on the way back. Now however, I have no qualms about travelling!