If almonds bother you don't eat them, that theory is how I run my SCDiet.
I had trouble with almonds and almond flour in the beginning. Later I found a thread that said that soaking almonds then baking them helped digestion and I found that to be very true. Earlier I also de-skinned almonds with boiling hot water, less fiber to deal with.
However nuts have a lot of fiber which can be difficult on a wounded gut so if you find difficulty there just avoid them. Different strokes, etc. Have you tried pecans? Soaking and baking them is FABulous and pecan flour is in my grocery store which I mix with almond flour for baking.
5 times a day seems like a lot to me even if you don't have blood or D. I think you are right to try and head off any worse symptoms by going gluten/sugar free. Your colon is not holding and processing nutrients well so you know something's not right.
Even though SCD has give me my life back it is not a cure, rather a control of symptoms. But I know the diet has lowered inflammation and for me eliminated D and blood and urgency.
P.S. Look around your diet for other causes of gut stimulation. You might remove caffeine and other causes.