I think urologists have a better track record, plus they deal with fertility issues as well. You're more likely to get various treatment options with a urologist whereas I think most endo's are quick to just pass you androgel.
In the end, I think the doctor who is the most willing to work with you and is very open with you and listen to you, is going to be your best match.