Better to take a week off by choice and be able to hit it hard again when you return than to push until you injure yourself and be forced to take a lot of time off and then have to ease your way back in. Plus sometimes I feel like I just need a break. A week to relax, just live life, not worry about if I missed a meal, or counting my calories, or anything else. Not only is a physical break good, but for me I need the break mentally as well.
And really, what's the big deal? I'm never going to compete, it's just a hobby. Is a couple days or a week off really going to make any difference in the long run? I doubt it. IMHO it is supposed to be fun. If it stops being fun, I know it's time for a break.