I wouldn't take a drastic approach to things like this based off of guesswork and intuition alone. Lethargy and low testosterone are very common and don't necessarily link in with the adrenals as the root cause.
I took time off training completely for the exact same reason and felt no better at the end of it despite extending it out to around three weeks.
Get an adrenal stress index (ASI) test. They are one of the cheaper tests you can do at home. You deposit saliva four times per day at specific times and they run DHEA and cortisol levels. You will get a much more accurate account of where you are at the moment and you will be able to work out where to go from there.
For example, you may have a higher than normal cortisol reading at the beginning of the day and lower cortisol levels at the end. You may have inverse cortisol patterns where they raise at night and are low in the morning (could be characteristic of shift workers etc) or lower than normal levels throughout the day.
You would treat each one differently hence the need for testing.
If you did have a mild form of adrenal fatigue then I would lower the volume depending on the severity.