trouble breathing every time i get a cold or sore throat?

alright, so for years i had really bad allergies. i also have asthma but it usually just comes and goes. however over the years i noticed things are getting worse. usually, as long as im away from allergens and im not sick i usually breath normally. but in the recent years, i noticed every time i get something like a small cold or a sore throat i have trouble breathing. its just been getting worse. know im so scared to get a cold or sore throat because i just had it over the weekend, and my breathing got so bad i could not sleep at night. ive tried taking my inhaler but it wasent helping much. that night i only got 2 hours of sleep. i finally got fed up and went to the doctor to get some antibiotics and know 3 days later my breathing is near normal again. its so frustrating know because im so scared to get sick again. im worried that the years of allergies may have somehow damaged my lungs? my health problems have all lead me to feeling so depressed know. what could be wrong with me?