I saw a documentary on the science of sex appeal. As it turns out... your hormone levels during puberty WILL be a determinate factor on how many of your facial features are defined. Your jaw line, chin definition, facial hair, and... another feature (which the ladies don't consider as much as guys do) penis size.
So if a young man in pre-pubescent years has a hormone deficiency and carries this through puberty... he may have some less manly features. Rounded chin, and face... and will probably have more feminine features. The basal structure is still there, but many of the definite masculine features will be lacking.
There will be other bodily features that will probably take shape, such as visceral fat on the hips, thighs, and lower abdomen. Not to mention possible gynecomastia. A significant decrease in male sex hormones may play a part in increased estrogen... and I'm sure we all know what that does to a guy.
Now... what ambulldog has said, may actually have some truth to it. Though I'm sure he meant it to be cheeky. Women have a sense of when a male is not sexually healthy. They don't know that they're doing it, but most sexually healthy women can sniff it out, so to speak. Through millions of years of evolution we may not be as in-tune with our more carnal instincts, but they're still there. Scientists haven't placed the exact reason as to how, but it is most certainly a fact of nature that we can pick out a "more attractive" individual due to their ability to provide our more carnal needs. A woman will find a more masculine man with the ability to not only protect, but to also PROVIDE. That means provide food, provide children, provide a house. They seek safety and reproduction. They can somehow tell if you can't provide these things.