If you find yourself saying no a lot you need new clients, or a new line of work. Being a designer is hard, and so is finding good clients.

As an aside -- nightmare clients tend to crop up more on projects with thin margins. If you are taking on a new client, make sure the margins are not thin.