Well, you might be right that it's a red flag, but it may also be a positive...I once saw a lovely doctor (he's since passed away), and he took every patient's story to heart and really listened. At the time, I had a very serious medical mystery going on, and he was the only one who was so empathetic and curious and interested that he picked away at "my" mystery until he solved it. I firmly believe that had I seen any other doctor, my problem wouldn't have gotten figured out as soon as it did, and I could have developed brain damage, or even died. (The problem turned out to be cryptococcal pneumonia.)
Anyway, my point (yes, there is one!) is that this doctor was always running very late, but none of his patients cared, because they knew that once it was their turn, they would have the full attention of the doctor for however long it took.
On a side note, his nurses and office staff used to get so worked up, because he took so long at each appointment, and so they were always running behind. The hospital this small-town practice was affiliated with dictated that patients be scheduled four-per-hour, but they couldn't make him see them at that pace! Because of the rural location, this clinic wasn't a popular place for doctors to locate, and so the hospital wouldn't get rid of him, as they couldn't easily find a replacement!
As it was, he was over seventy, and he no longer cared about assembly-line medicine...he practiced his way, and it was refreshing and delightful!
I don't have much hope that this is the case with your doctor, as mine was a rare gem indeed, but maybe, just maybe...
Please post after you actually get to see this doctor--I'm interested in hearing if s/he was worth the wait! BTW, I, too am very neurotic about time, and I think you did the right thing...their time is no more valuable than yours!