I have heard similar things from guys at my gym too, most of which avoid dairy as much as possible.
As far as the reasons why, there is more sugar than protein.
Me personally, I drink atleast 32 oz of whole milk a day, preferbally when I wake up and with my Post workout drink. But, I am going for size and strength.
If your trying to keep it lean, keep the milk minimal