IMHO I think you were feeling very uncomfortable with this man in your face. He was inappropriate and I suspect you wanted to tell him to bug off but were not sure how to deal with it.
Your anxiety started to kick in and your head was spinning with trying to deal with how the whole situation was going.
Next time, just politely tell someone that behaves that way, "I am not comfortable with this conversation, I find it inappropirate." Walk away, ask him to take a seat in the waiting room or whatever.
I am assuming he was not a co-worker?
Gentle Hugs
Kitt