The reason it matters is because something is being described as originating in a culture and then being characterised as rude. Simple as that. There are reasons for all sorts of etiquette, and if the reasons are lost in time, it probably is time to no longer follow that piece of etiquette. We're not talking about the reasons behind each step of the Japanese tea ceremony here, or why it's tradition to curtsey to the Queen, but day to day etiquette which has more to do with how we interact with others than echoes of culture and "that's just how we do things".
If you were to accept that it isn't polite or impolite, it would still be seen as bad grammar to many, which I can't help but see as the crux of the issue to begin with.
I'm sorry I just can't give you any other answer because, from me personally, there isn't one. Maybe it stems from grammar, maybe at some point people thought it was inappropriate/impolite to use bad grammar and somehow my grandparents and great grandparents and so on, caught onto the idea but left out the details for those of us further down the line. I don't know, but I'm sure someone out there does. But why is grammar a more suitable and convincing answer than manners? I just don't see why the reason is important, and I don't see why, if there isn't a reason for it anymore, people should just get over it. It's language, not obsolete technology. If it becomes irrelevant, then it will go away on it's own like thou, thy, methinks, etc. For the time being, it's still relevant to some people, so it stays.
When it comes down to it, it's all a bunch of hot garbage anyway, really, including grammar. No one's truly right, no one's truly wrong. But we're humans, so we feel like we have to categorise everything and we also have to disagree on everything.
So I'll go on using the manners I was taught and not really think about it. Like I said, it's not a big deal to me, and I really don't care if people don't think it's rude. I don't judge them as some heathen or savage for saying "Can I get...", it's just not how I would do it.