I spent my entire adult life buying into the whole grain propaganda and scoffing at the idea of "a calorie is a calorie"; I also spent my entire adult life (until now) obese and not able to lose much weight even though I read so much and could talk like a nutrition know-it-all.
Now, don't get me wrong--I still think, as a general rule, the more natural and less processed the food, the healthier it is. Some foods are more satiating for the calories they provide, making calorie restrictions easier. Nutrition and satiation are definitely important considerations. But in terms of weight gain/loss, cals in/out is the #1 factor, macros a distant #2, and everything else is a drop in the bucket in comparison.
My own experience: During my weight loss, I read Wheat Belly and stopped eating wheat/gluten products but kept my calories the same. I continued losing weight at the same rate I was already losing at. I think the author's claims of people losing weight when they stopped eating wheat is due primarily to eating more satiating high protein/fat food, experiencing less insulin-spike cravings, and thus eating fewer calories. For the record, I've continued eating wheat/gluten free anyway because I realized I don't miss it (and I can occasionally indulge in gluten free versions of the few things (mmmm brownies) I otherwise would miss) and the author still convinced me of the health implications.
Also, more recently, I switched from buying brown rice to white rice. I forced myself to buy brown rice for years with no benefit, because it turns out I ****ing hate brown rice no matter how I try to cook it, and hating your own cooking is a good way to sabotage a diet. It also turns out that I LOOOOVE white basmati, so that's what I buy now. Now I also love my homemade curries over rice and meaty fried rice dishes. The frequent white rice intake has done nothing to slow down my weight loss.
One last note: If you're stressing over fiber content because you're constipated, you're not eating enough fruits and veggies.