I've had similar experiences. Thankfully, for a good part of it, I've had a GP who is very cautious about
medications like Remicade and has pushed me to find a GI that works for me.
about three years ago, I had had it with standard-practice GIs and found an IBD center. I've since seen docs at two IBD centers, and it's like night and day compared to the other.
They still see drugs like Remicade as the end of the line for standard treatment, but they've been open to alternative treatments and generous with their time (even when we disagree). The one I'm at now also includes meeting with dieticians (and if one wants to, psychologists) as a standard part of their care.
If you're near a large teaching hospital or other large medical center, definitely see if they have an IBD center, clinic, or other IBD-focused care. I kick myself for not exploring this option much earlier.