Breathing is directly linked to heart rate and vice versa. Obviously both increase when we are anxious or panicy (as well as when we exercise), so my guess is you've subconsciously made a connection in your mind relating anxiety to heart rate/breathing and thus have developed a fear response to heavier breathing and/or increased heart rate.
This happens to me a lot, throughout the day and especially when I'm in bed. If I'm not thinking a lot then I'm probably noticing my breathing, which I have a fear response to... the fear increases my breathing... and it goes circular like that, haha.
This is my opinion anyway. But exercise is good because endorphines can numb anxiety and remove pent up energy, so you should do it as much as possible no matter what.