me myself personally don't have a problem waking up at 4 am - i'm usually awake before then actually - it's the perfect time of day = no kids, no wife, no interruptions/distractions, etc. oh, and it's QUIET !
if you are not falling asleep at your desk during the day, then i wouldn't stress over it - just make good use of that alone time