I think it depends on the compound, more specifically it's strength. If you are using "sucicide inhibitor style" anti-e then yes, it it is a good idea to take some time off. However, when dealing with most herbal testosterone boosters, I doubt that they can elevate your testosterone levels to supra-physiological levels (above the normal range). Seeing that women can take many of these compounds and not experience irreversable androgenic side effects seems to be a good testament to this fact.
I think that the biggest issue is sensitization. When you take most supplements for long enough, your body becomes sensitized to the compound and it no longer 'works like it used to.' Anyone who has played around with stims/fat-burners can attest to this. By cycling different test-booster supplements, like in the way Guejsn mentioned, should allow you to enjoy prolonged usage. A lot of the boosters have common ingredients so you can use what the label says as a guide.