Some people have a natural confidence when cycling downhill, or at least a way of ignoring their natural fears, while others are much more cautious. This nervousness about going downhill is an important weakness to tackle since:
1) if you lose a significant amount of time cycling the descents it is extremely difficult to make it up when going up hills
2) if you arrive at the bottom of the hill at, say, 30 mph instead of 20 mph you have a great advantage for the next stage of the course, whether it is a flat or a hill, so the faster cyclists going down the hill will often be the fastest cyclists going up the hill straight after as well
3) if cycling downhill makes you stressed you will enjoy riding much less
Very often cyclists attribute their hesitation in descending to factors such as 'natural fear' of going downhill fast. I know all about this since I did the same for several years. I've always hated children's swings and slides as well, and would have sworn that I just couldn't cycle any faster downhill, because panic would set in, my arms would go rigid, the bike start shaking - and I'd need to brake rapidly to prevent an accident.
As an approximate guideline, I would say a 'normal' cyclist will go down a 5% descent quite happily at 25 - 35 mph, while many will happily reach 40 or even 50 mph without concern. I am still not within this second group!
Of course, everyone has a 'fear' point - but if yours is lower than other people you cycle with then perhaps something else is wrong. Some of the most common problems and solutions that will enable you to cycle downhill faster::
1) your bike is setup wrong or is the wrong size. If you are being forced to put your weight on the handlebars by a bike that is the wrong size or setup badly then it is extremely difficult to descend 'properly'. On normal descents your hands should only be holding the handlebars very gently (about the same as when you are writing with a pen, for example). If this setup could be your problem first try moving your saddle back - even a few centimeters might be necessary overall, but only make changes a small amount at a time. Your bike shop will be able to tell you if your bike is simply the wrong size for you.
2) you are sitting too heavily on the saddle. During descents cyclists should put almost all their weight on the pedals, or just the outside pedal if going around a corner. There should be only a small weight on your saddle (and almost none on your handlebars, as mentioned above). This process lowers the center of gravity of the bike and makes it far more stable - try it, it makes a very large difference.
3) you are not looking far enough ahead. It is clear that when we look at the road directly in front of us it appears to be moving much faster than the road 200 yards ahead of us. Try and look less at the immediate section of road and focus on a point quite a long way ahead. You will feel a dramatic improvement in your confidence, as you suddenly feel as if you are cycling slowly instead of hurtling along.
4) focus on breathing. If none of these help try focusing on your breathing - breathe regularly and calmly, with your focus on breathing out evenly and calmly. Forgetting to breathe enough is easier than you think and not at all helpful!
5) keep pedaling. OK this sounds like a bad idea but it works wonders, even if you are only pedaling very slowly and with little effort (in a big gear, of course). A bike being pedaled along is much more stable than one that is simply rolling along. Your goal is to improve your stability, not reach 100 mph, so you only need pedal calmly and slowly - pedaling fast might be even better but that can come later!
