***DISCLAIMER: I am not an expert. That said...
Spermatogenesis takes time. Recent studies show about 74-90 days. So it's not really infertility ON cycle that you should be concerned with. For example, if you are 3 weeks into a cycle of halo or epi, you are probably still pretty fertile. What you are doing is inhibiting the body's current future-sperm production. So infertility can rear its ugly head well after you've stopped.
I would refrain from using broad statements such as "this only happens to people who abuse ph/aas." It is more likely to happen to people who are stupid about using ph/aas and don't plan a proper PCT. Like what has been said before, by using ph/aas, you are going to suppress your body's sperm production. However, if you are smart about your PCT, you should rebound just fine. Clomid is definitely the way to go in terms of a SERM as it will help your fertility. I am currently on Clomid 50mg EOD for trt because of fertility concerns.
Word of caution though - everyone's body reacts differently. I just don't want you to think that "this could never happen to me." Not trying to scare you haha, just saying that you need to be aware of the risks before you start. I did once cycle of halo, with proper PCT, before I discovered that I had low T and my sperm sucked (volume is ok but mobility and morphology are horrible). Now, I never got tested BEFORE my cycle and even my urologist said the halo was probably was not the main factor of my issues, so I never blamed the ph for my problems and I'm sure I had those problems well before my cycle. I am just sharing my story with you. Now that I'm on trt for life I've done other cycles, allowing my body time to recover regarding fertility. If you want to put your mind at ease, get bloods done and maybe even a sperm analysis. Then you'll know what your baseline is before starting. That way, if you get tests after and they are way off, your doctor will at least have the full story. Good luck brother!