I've read (yes, again, but this I believe was from New Scientist) that too much estrogen can make people moody and irritable. I believe there is a reason why it was always fermented in some way, originally. Think about how powerful testosterone is in the female body, now think about how powerful estrogens are in the male body. Certainly it's menfolk who are going to suffer the worst of the unfermented soy products. Please remember that certain forms of breast cancer actually aren't negatively effected by soy phytoestrogens, it's not a simple issue at all.
I've worked in a health food store or two. Some men with very strange body fat deposits* who ate a lot of soy products, and never meat. And they were very pissy (of course, maybe it was because of their strange fat deposits, cause-and-effect and all that...). I can't say which happened first, the weird looking body or the emphasis on soy products and abstention (at least from what I saw) from animal products.
*a gut that bulged from a really low place, almost like pregnant, and of course some extra sexy boobness.
In the end I'd say if you're feeling anhedonia, less confidence, moodiness and especially irritability and you're on a lot of soy it's time to cut it out. And soybean oil, refined, unrefined, organic or the delightful Monsanto Roundup variety... no way. Limited quantities only.