I used to drink 5 gallons of milk a week, and aside from 1 serving of cheese every 2-3 days I've cut out dairy completely.
While I was able to squeeze it into my daily macros while dieting, I have had much better and faster results after dropping milk.
Anecdotally, I believe (and have been told) that milk slows or stops complete nutrient absorption due to its coating effect on the stomach. That's apropos of nothing though, because I cut it due to the sugar content.