Sorry to hear you had to deal with the pain of a perianal abscess. At the same time, I'm glad to hear you don't have (or at least haven't been diagnosed with) Crohn's. That helps increase you're chances that this will be your only one. I can only really give you a Crohnies' view of abscesses - ours typically take longer to heal so the information may be a bit skewed.
Yellow discharge after 3 weeks is normal for me. It is just pus from the abscess, which as you said is healing from the inside out. Healing from the inside out just means that the outside of the wound is still open, and so it is able to allow the pus to escape. Indications of a fistula are usually brown discharge - that is, you'll often find some stool in the abscess. With a fistula you might also notice you pass gas through your abscess wound.
You also want to make sure that the infection doesn't return. So watch out for fevers, redness, swelling, other painful spots (that may indicate another abscess). If you get any of these symptoms then I would contact your doctor/surgeon since it may indicate that your abscess is healing improperly and forming again.
Anyway, it sounds to me like everything is normal. Given you don't have Crohn's I imagine it should be healing up fairly soon. I would mention the yellow drainage at your next doctor's appointment for peace of mind and to make sure you're following a healing pattern that your doctor/surgeon expects. However I wouldn't worry yourself about it, it sounds normal to me.