Melatonin:
I personally don't find melatonin to be very sedating per se. I feel that it works best if you cooperate with it, meaning you actually get in bed and try to go to sleep. If you take it but keep doing 'awake' stuff, expecting it to sedate you or knock you out at some point, you may be disappointed. Also if there is something keeping you awake, like stress or noise, it may not be strong enough to overcome that.
On the other hand some people do find melatonin highly sedating, but in my experience that's the minority.
Dosage is highly variable per user, lots more so than many other things. If you feel tired or groggy the next day, you took too much or the form you took was too slowly absorbed. (Lots of people like sublingual melatonin for this reason.)
Melatonin has a lot of interesting hormonal and life extension possibilities... one thing that does give me pause, though, is the fact that even a 1mg dose is something like 50x what your body normally makes in a day. I think that I will eventually be taking melatonin every day, but at this time I don't feel comfortable with picking an appropriate dose. I would especially advise caution for those under 40 with respect to taking melatonin long-term. On the other hand there are people who use large doses daily for various reasons, so I may be worrying about nothing here.
5HTP:
You might or might not find this sedating. It's not uncommon for human trials to use 300mg, 3x a day before meals, but this is considered to be a substantial dose, and gut upset is not rare at this level. I think a lot of people will find 5HTP to be more relaxing than sedative, but it's certainly worth a try.
Magnesium:
Some people swear by it, but I have not personally known anyone who felt it was an effective sedative. The doses needed seem to be large (400-1000mg), so be aware that you might need to run to the bathroom when you wake up.
Taurine:
Again some people swear by it, though I personally don't know anyone who really found it sedating. (That'd be kind of ironic, considering a number of energy drinks have some, on the theory that it's energizing.) I've used daily doses of 3-5g and single doses up to 10g with no noticable effects.
Valerian:
Basically a mild benzodiazepine. Unless you have built up a tolerance, if you take enough it will almost certainly be effective. You might need to take quite a few capsules to get "enough", especially if they are just root powder and not a standardized extract. You can expect this to be relaxing and sedating. For my money it's probably the most useful at getting to sleep in the face of some difficulty such as stress or anxiety or noise. On the other hand this is something I prefer not to use long-term. You will tend to develop at least some tolerance, sleep architecture seems to be impaired, and morning hangovers are not uncommon.
Kava:
I personally don't find kava very useful. For me it does almost nothing, until I reach some threshold dose, above which it has a kind of deleriant effect. I'm actually at something of a loss to describe it. It makes me feel kinda goofy and incoherent. It is somewhat sedating, in that it also makes me feel like lying down and closing my eyes would be a good idea, but it doesn't really knock me out. On the other hand I know lots of people like it, so probably I'm just a non-responder or weird-responder. This is another thing I would not want to take every day long-term.
GABA:
I get mild flushing (like niacin only MUCH milder) at doses well below 1g. I haven't noticed any sedative activity at those doses, and have not tried larger doses. I do continue to take some, sort of on faith, when using valerian or theanine. That is, I don't notice any additional effects but figure it's cheap insurance when using other GABA-related things.
Theanine:
This is probably my favorite of the sleep aids I have ready access to now. I used to use GHB, actually GBL, back when it was legal. For me it was nearly perfect: It rapidly induced sleep even in difficult cases, promoted strong, well defined and very normal sleep architecture, and I woke feeling very refreshed and energized. Theanine is reasonably good at inducing sleep, and definitely promotes strong and normal-seeming sleep architecture, though it does not do either of these things as well/strongly as GBL did. At least for me it does not have any sort of "wake up energized" effect.
One thing you may find is that you wake up briefly every 90 minutes. This is basically when your sleep cycle reaches the least-deep point. If you do notice this you will almost certainly find that you drop right back to sleep, very easily, in a few minutes. (With GBL you could damn near pass out again in mid-step, if you got out of bed at that time.) I know many people strongly dislike waking up mid-night, but if this does happen, don't fret it, just relax and stay in bed. Odds are you'll be asleep again shortly and none the worse for it. For me this effect was much more pronounced with GHB and GBL, but is still pretty noticable with theanine. Basically this is pretty normal, it's just your natural sleep cycle, in which you spend a few minutes almost awake during each 90 minute cycle.
For me theanine is about half as strong as valerian in terms of getting to sleep when things like stress or noise are causing problems, and thus much stronger than most other things.
Getting these effects requires unusually large doses, for me in the area of 750-1000mg. There seems to be some synergy between theanine and valerian (as you'd expect). I also try to take GABA whenever I take theanine, not because I notice any additional effect, but rather because theanine supposedly increases GABA release and I figure it can't hurt to try and replace some of that GABA. (It may not work but it doesn't seem to hurt either.)
Hops, passionflower:
Haven't tried either, but I suppose I should.
Overall summary:
I try to get enough magnesium and taurine daily, since they are both important nutrients, but I don't see any sleep-related effects from either one.
I would suggest you try either theanine and/or melatonin first. If you find theanine is helpful but sometimes needs a boost, I'd add valerian or kava. If you end up relying on theanine long-term, I'd add some GABA just in case. If none of the above are working I'd try 5HTP. If that failed too, then I'd think about OTC sleep remedies. (I find them powerful and effective, but very unpleasant in terms of sleep quality.)
I hope this is somewhat helpful. Unfortunately sleep aids seem to be pretty variable from person to person, so I'm afraid you may end up having to try a few things to see what works for you.