The libido problems are more attributed to the change in hormones IMO. I've used it on cycle for gyno prevention and never had any libido issues during a cycle, just when I came off during the first 2-3 weeks of post cycle therapy.
There is also Toremifen for those who want to avoid Tamox. I'd be more concerned with potential liver and ocular damage from Tamox.