Do you eat meat or fish? If so, you should be fine—just bump up the amounts of those, and then use the soy on an as-necessary basis when eating protein-rich whole foods is difficult or impossible (e.g., weekend motorcycle trip).
If you are vegetarian, whether on purpose or just de facto, then you're going to have to bite either the soy bullet or the dairy bullet. Pick your poison.
Regarding your question about estrogens: Remember, "There are no poisons—only poisonous doses."
Most of the soy/feminization stuff has been reported at extreme levels of consumption (like 10+ times the norm, for months or years straight). At more moderate levels of consumption (unless you are an infant or small child) the data are inconclusive, and whatever effects there might be are quite small.
So, as long as you aren't planning to drink 2 quarts of soy milk (or throw down 3-4 of those shakes) every day, you should be fine.
In this respect, soy is a lot like, say, red wine. Both of them have upsides—soy likely protects against prostate cancer and might even extend life, and wine may protect your heart—but excess consumption is bad. (Alcohol also has pro-estrogenic effects.)