Vertigo has been a friend of mine for even longer than anxiety. I get those short burst dizzy moments like you describe from time to time. I can even have them for a month or more, then they will vanish.
I've gone back and forth in my mind as to what it might be, perhaps it's a problem with my inner ear or perhaps sinus congestion that places too much pressure in my inner ear.
More recently, I've read for some it can be caused by the vagus nerve complex that runs throughout your abdomen, especially in those with stomach problems or GERD / acid reflux. Basically when this nerve gets stimulated a certain way it can cause you to feel dizzy and faint and cause your blood pressure to fall. I do know that when I've had the dizziness you describe that I can take a stomach med like Nexium (acid reducer) and the vertigo pretty much vanishes.
Of course I can't figure out if my stomach problems cause anxiety or anxiety causes stomach problems...that's another story for another post...hehe.
Hang in there!