This seems very simple to me.
1. Agree to meet him with the kids. If you file a separation or divorce, he will get visitation. So sorry to say, suck it up. (((hugs))) I say this kindly from a been-there-done-that.
2. Do not let him come to your house. Meet him someplace neutral, with structure. A restaurant. A playground. Mini-putt, batting cages. Even just grocery shopping. Something with an obvious "time to go" so no awkwardness at goodbye time.
3. Prep your kids. If they don't know about bipolar, now is the time. Talk to each one individually and keep it at their level of understanding. My barely-4-yr-old son knows that I am"sick", that I have bad days, that I take a lot of medication, that my dog works to help me and is a service dog, that not all dogs are service dogs, and that I love him ALL THE TIME no matter how I feel.
Also prep them about the visit. "Seeing him leaving again" is YOUR perception. How about "Daddy loves you very much and misses you, so we are going to go see him for a few hours. We are going to [fill in the activity]. When we are done, Daddy will go home to his house and we will go home to our house. I think it sounds like fun!"
4. Don't let him play with your emotions. You are letting him do this. He is not doing this to you. He is giving you different responses only because of his mood changes. He's not trying to manipulate you. Once he gets stable on medication, he will be than man you love. Watch carefully for signs of his mood when you are trying to decode whether or not he truly means what he says.
Something else unrelated - try communicating in letters. Don't forget that he is hurting too! Send him a note. Like "I love you and I miss the healthy you that I have always loved. I feel hurt that you've left, but I hope that you are taking this time to get help and get started on medications. When you are stable, I want you to come home and we can start the rest of our lives" If that is true. :)
Calm down, good luck.