You can over-stimulate them..
I recently read claims of taking GABA to increase testosterone when leydig cells were damaged. Not sure if it is true or not, but may be worth looking into?
Also opioid antagonists are claimed to fix GhRH issues.. Who knows, could be BS.