True, calories are a measure of energy (mainly used with foods). But you are forgetting about the nutritional aspect of the foods we eat. Example: the good calories vs. bad calories thing. A peanut butter and jelly sandwich on whole wheat has about 460 calories, while 4 tablespoons of cooking oil has about 480 calories. These both provide the body with about the same amount of chemical energy, but i'm sure you can see which ones are bad calories and which ones are good. Calories from fat and calories from sugar are what people mostly cut down on when on a diet (bad calories).
just my input