OK, you're pretty ambiguous on what you want, but based on "easy to move to a collocation center" and "GUI isn't required" it sounds like you're mostly asking about servers.
For initial development, you probably don't even need a server. There's pretty much always an easy way to have a local server run on your computer for development.
When you are ready for a server (probably not until you're about ready to go live), I'd suggest Slicehost. They're well-documented and cheap ($20/mo. for their smallest server!), and you can scale the server all the way to 15.5GB of RAM in just a couple of minutes.
You could look at Linode too though. It seems like you get a little more RAM for your buck, especially on the smaller end, but you can only go to ~2.8 GB of RAM on one server.
As for your development computer, I love Ubuntu, but it really depends what you want. If you want to do everything through an IDE, it depends on your language, but the best IDE may be for Windows. If you want to do much of anything through the command line or think you might in the future, develop on either Mac or Linux. Their shells are just way better. There is always Cygwin, which sorta fixes the problem, but I think true Unix works better.
When you are ready for a server (probably not until you're about ready to go live)
I'd suggest getting it the day you start your project.
First: twenty-five bucks a month, seriously, what are you even doing thinking about this ($5 extra for backups, which is another "why are you even thinking about this" item).
Second: source control from the very first line of code. The production webstack, from the first line of code. The production deployment procedure, from the first line of code. etc, etc
"I'd suggest getting it the day you start your project."
I agree with this. Having the server sitting there ready and waiting, and costing you money, has a certain motivating effect. Every time that monthly accounts email comes in, and you haven't gone live yet, you're gonna feel like an idiot, and that feeling is good for you.
It's like a gym membership, just sitting there sucking your money, mocking you and shaming you into action. It appeals to the exact right part of your psyche - you're paying for this, where's the results?
Take the step. If you're serious, sign up for a server first thing, I say. Buy a domain and wire it up. Load your blank page and stare at it and feel like a loser. For best effect tell all your friends about it and make all sorts of grand promises, and then stare at your blank page some more and feel like even more of a loser until finally you cross the threshold of shame, get started and then - you're on the track my friend!
Ah, blackmailing yourself into action. Tried and true. Shouldn't be necessary but hey, we're only human, and the only thing that matters is results.
I agree with getting your VPS and experimenting with it when you start (or early in the process). I use Debian on Linode, and I have learned so much. The experience is invaluable. Everything from configuration to automation to security.
This prepares you for the day that you go live, as going live won't be a completely new experience, it will be a new experience, but if you iterate on hosting a blog and sandbox apps, then you are much better prepared than if you go VPS cold with a new product.
Went through everything from old office computer used as web/subversion server while still on windows xp to slicehost. It all depends on the details... but if you're not sure what they are keep it simple.
Depending on what you feel more confortable with you could either slap ubuntu on an old computer, or take a small slice at slicehost. The second has less bandwith (then a local computer) but overall it's much more flexible.
For initial development, you probably don't even need a server. There's pretty much always an easy way to have a local server run on your computer for development.
When you are ready for a server (probably not until you're about ready to go live), I'd suggest Slicehost. They're well-documented and cheap ($20/mo. for their smallest server!), and you can scale the server all the way to 15.5GB of RAM in just a couple of minutes.
You could look at Linode too though. It seems like you get a little more RAM for your buck, especially on the smaller end, but you can only go to ~2.8 GB of RAM on one server.
As for your development computer, I love Ubuntu, but it really depends what you want. If you want to do everything through an IDE, it depends on your language, but the best IDE may be for Windows. If you want to do much of anything through the command line or think you might in the future, develop on either Mac or Linux. Their shells are just way better. There is always Cygwin, which sorta fixes the problem, but I think true Unix works better.