First thing - how cut are you (really)?
My chest is the first place I get fat and the last place I lose it - so I look a bit "booby" even at about 9 or 10 % bf.
Second - Do you have hard lumps behind your nipples? If it is jelly like - chances are it is stubborn fatty deposits and not breast tissue growth.
This is good and bad - i.e. good cos it means you probably don't have true gyno and so you don't need to spend an ass load of $$$ on surgery, bad cos it means you need to get really lean before you lose this fat.
Third and last point - are you being harsh on yourself - I only ask this because I am harsh on myself - I don't like my chest and worry about man boobs - even thought I know (because ppl tell me) that I dont really ahve a problem with this.
Maybe put up a couple of pics of yourself ?