Yeah, I'm the same. Granted, I'm an early bird and prefer to get into the office around 8 or earlier (so nice and quiet and you can get so much more done!) so I'm rarely "late" even if there is a massive delay, just late by my schedule, I guess.
I think some of it may depend on the corporate culture, though. My old company I had bosses who did similar things (showing up early, strolling around) and the times when people arrived and left were definitely noticed. If you left before 6, it was a strike against you. At my new company, the same level of performance is expected - but not the same hours. I don't feel like we have this need for "face time". If you do your job and you are done by 5 and there is no need for you to stay late, by all means, go home. I like this a lot more. I hated the feeling (at my old company) that I was leaving early when I already put in a 10 or 11 hour day, simply because my boss hadn't left yet.
It was definitely an adjustment when I moved. There was some sort of issue with the tube early on, and it took me about 2 hours or so to get to work that morning, and I rushed in, all worried about how it would reflect on me, and nobody cared, as they heard about the tube issue.