Most if not all people have a genetic limit to the size they can gain. If you are already at 180, I don't think it will be too difficult to gain an extra 20 lbs if you dedicate yourself to your diet an training regimen. Start counting your calories; this might make a real difference.
Some people have it easier than others. Personally, I never had too much trouble gaining weight but when it comes to getting cut, it is a pain in the ass. So those same guys who are maintaining their size with a half-ass routine, could be pissed that they don't have cuts.
You shouldn't use other people as a source of reference, it's your progress, not theirs' that matters. As far as other people naysaying, just ignore that sh*t. People seem to accept mediocrity as a part of life and anyone who is trying to go above and beyond that is generally chastized. I guess it's good to have haters though; it means you're doing well.