Good Morning!I suspect you are irritable as your anxiety increases when you are asked to perform a task for someone else that makes you feel uncomfortable. Like many of us you have not learned how to say "No" without feeling guilty as you are possibly a people pleaser?
Setting boundaries is healthy. You need to learn to respect and take care of yourself.
When you are not expecting someone to ask a favor of you - give yourself time to think if the favor is something you don't mind doing or do you recognize immediately this is a stress-inducing "favor" you have been asked to do.
Either way, a polite no, I am not able to help you out this time is all you need to say but keep your feelings of anger tamped down. Never apologize for saying no as you have not done anything to be sorry for.
No feelings of guilt. 
Kindly,
Kitt