Highly individual. Do you need to avoid it to lose weight? No, you don't. Fat loss is about being in a caloric deficit. You can still eat fat and lose fat if you're in a deficit. Dairy is one of those things that will vary with each individual. Some people have lactose issues, myself included. I've cut it out just because my stomach effing hates me when I consume it. However, it wouldn't hinder my fat loss efforts should I decide to eat/drink it. If you're not lactose intolerant, do whatever you want. However, many don't know they're lactose intolerant until they eliminate it for about a month and then reintroduce and see what happens. Highly, highly individualized here.