I stopped fearing that when I became a freelancer. It's done wonders for my confidence, and I'm much more productive as a result.

It also helps to be married; two jobs give a lot more security than one. Even so, having savings that can last you through a downturn or possibly even fund an early retirement, is definitely a good idea.