I love building servers, it is amazingly geeky fun. And no doubt colocation with your own servers will give you the best bang for your buck, with renting a dedicated server coming in second. But you do have to be careful about redundancy. If you can't do without a day or two down-time, you'll need 2 or more servers, because when it's a dedicated server it's all up to you (or to someone you hire/pay for) when something fails.

Sadly I live a 5 hour drive from the nearest co-location facility, so I'm forced to rent a dedicated server.