BeRock,I think perhaps we are married to brothers, lol, my husband has this very same issue.
It got to the point where I went to many social gatherings alone as it beat making him go when he was so miserable while at the event. People would jokingly ask me if I really had a husband which hurt me emotionally but I would smile and pretend their words were funny.
Having company to the house is tough as my hubby has Crohn's Disease and is on a schedule that works for him re, eating, nap times and how he likes to spend his evenings ensuring that he is stress free.
He was upset with me a couple of weeks ago as 3 late afternoons last week someone dropped unexpected. He likes to take a nap at 4:00PM, then shower, eat dinner and watch his favorite outdoor sport TV shows.
Having all the company staying at our home years ago caused me anxiety as my hubby was very uptight and I spent double time trying to prevent things that would cause him to feel angry at the loss of his privacy. I would take out of town visitors to Mall of American for the day, take them out to dinner, to movies and even pay for hotel rooms so "they could have a place to go and rest, sleep, use the bathrooms etc. " Our home is very small........it is a mini-me house
I never did solve this problem as it goes on, but not as often as it use to as the requests to come and visit us has declined a thousand times over.
My best advice for you, enjoy your family and know that your hubby will make it through. Plan some outings and leave your hubby home to enjoy his quet time.
I know my hubby felt like a stranger in his own home when people would come to stay with us.
Blessings,
Kitt