tell him you said no to lunch...and you are saying no to everything else including any further contact...if possible do this writing or with a witness and then if he contacts you again report him to HR and file a harrassment charge...it sounds like he has aproblem with the word NO or at the very least he is not hearing it correctly from you.
When or if you do "run" into him simply nod or give a flat hello only if he acknowledges you first, but do not engage him in converstaion and if he tries to engage you in any conversation or stands in your way impeding your travels...Say very sternly and in a strong but level voice "I have asked you to leave me alone and not to speak with me, if you persist I will have to file a complaint." leave it at that and walk away.
You may be very surprised by how empowering it will be to assert yourself in the manner describe above...YOU take control of the situation and your stress level will lower.
good luck