i am not heavily invested in this earther way - so just carrying on the discussion for the sake of conversation
ref - maybe only a kind of v slight infection vs something like an obvious mass
maybe - but even then, when fungal species grow they form hyphal cells - these are long threads of hair like chains of cells that will grow into a layer of fibrous fungal tissue on a surface. this the hallmark of growth - as apposed to just a few spores or cells settling on a surface and sitting there inactive. therefore colonisation requires growing hyphal threads ( there are yeasts - which are a subset of fungi that can grow by budding as well as by hyphae - but this does not apply to molds like aspergillus)
hyphal threads are actually pretty easy to detect - eg just by swabbing the area - transferring onto a slide and staining with basic microscopy techniques - fungal hyphal cells can easily be distinguished from bacterial cells and other artefacts
you could probably even do it at home - but you could certainly do it with an endoscope to guide the sampling.
also - if there were colonisation - even the lightest growth of fungus on the surface- you would also expect some immune activation locally - so inflammation redness etc - exactly as is seen with candida in the mouth or throat or anywhere else in the body ( it also grows hyphal threads when it colonises )
so again - i just don't see it being that hard to demonstrate - if it indeed did exist in the majority of mold patients
for reference this is what hyphal fungal cells look like in molds like aspergillus
https://ibb.co/2q0mrhw