No, it doesn't mean you are sensitive to them, it means they are working!
What your doctor meant is that there are people that are allergic to the herbs.
Activated charcoal is one thing you could do, but there are time requirements with the use of any binder. It must be taken 2 - 3 hours away (before and after) from anything else you wish your body to absorb - otherwise the binder will bind to the products that you have taken instead of the toxins.
Many of us can sneak in one dose during the day, but if that's not the case, if you get up regularly during the night to use the bathroom, you can have the binder set out along with at least 8 ounces of water and take it then. Binders must be taken with at least 8 ounces of water to avoid constipation. When we are constipated, we are not able to move the toxins expelled by that route out of our body quickly enough to prevent them from reabsorbing back into the body.
You can use dry skin brushing oil pulling and other forms of detoxing though. Here is a great list to go over that is similar to the one that CD posted:
www.tiredoflyme.com/detox-methods.html