I had 25 fractions of IMRT and two HDR brachytherapy procedures. The only work I missed was on the HDR days because the procedure required me to be at the clinic from 11:30 in the morning until 4:30 in the afternoon. At the same time my wife was also undergoing IMRT for breast cancer and had 35 fractions. Never missed a day of work.
Just arrange your schedule properly and it shouldn't be an issue. My wife had the first appointment of the day on her way to work. I had the last of the day on my way home from work. The radiation itself generally presents very mild side effects, nothing that disrupted my schedule.
As for recovery, many people have some mild fatigue, and some urinary and bowel issues. Basically increased urgency and frequency that most people manage well. This generally starts about two to three weeks after the radiation has started and subsides two to three weeks after the radiation is done. Of course some men have more significant issues, but they seem to be in the minority.