Uhhh no. One potato with the skin has 49ish% of your daily vitamin C. It depends on the size of the potato and how it was stored though. But according to the Potato Council, potatoes account for 15% of the UKs daily Vitamin C intake.
I just had a check, and google says that apparently if you just ate potatoes and milk you'd eventually die of molybdenum deficiency "signs of molybdenum deficiency . . . are headache, rapid breathing and heart rate, nausea and vomiting, acute asthma attacks, visual problems, disorientation, and, finally, coma."
So yes, you wouldn't die of scurvy, but you'd still eventually die

If you were just eating two things you could do much worse than potatoes and milk, but you can't survive on it long term. Sad face.
I totally didn't realise that potatoes had Vit C in them, but imo, they're still bland and gross (and lacking in the essential nutrient molybdenum apparently!)