My last company I worked at, they had like 30% of the staging environment running production stuff. They also had only one deploy target, which is drum roll production, which meant all of staging AND some of Dev/QA was live and could interact with the production environment. Also fun to see the production load at 300% of expected, burning, only to find out some event, data roll up, query, whatever was being run in duplicate on 10 different machines because somebody forget to to manually edit the configs after they rolled out a new version to staging/QA. Although I think this would qualify as "chaos engineering", I don't think it fits with what netflix is going for.
Yeah, ok that had nothing to do with the OP, sorry, I just had to vent.
The kinds of things I work on now are entire "environments" we sell, so "production" testing is more along the lines of "go plug the box in and do stuff to it".
2) The projects I work at on $DAY_JOB that I control all have 3 hardware nodes for HA in both staging & production. Staging is under artificial load that plays back the same series of events on dummy data. [e.g. Job A runs -> Completes -> Puts a mock job in Staging's Job Queue -> runs on Staging with mocked data -> I randomly restart a node to see what happens]
Buying 3 1Us is $750 for a staging environment. If you have 1 DC availability, you can just put them on your Local LAN (Free).
Now, you suddenly have a rough approximation of a 3 Node / 1 DC environment.
But what if you have multiple DCs, 3 Nodes per DC, Hot/Cold Loadbalancers and have DNS Failover Between Them?!
Say, you run a LBx2+Nodex3/LBx2+Nodex3/LBx2+Nodex3 across 3 DCs.
Assuming you automate the build, you could just pay by hour. But if not...
Linode's Failover IPs let you do the Hot/Cold LB thing [$60 for 3 DCs worth].
3 Nodes per DC [$40 Nodes for 4096 MB is probably more than enough to test most issues]. $360
So for $420/month, you could replicate a pretty big real world setup.
Linode's Failover IPs let you do the Hot/Cold LB thing [$40 for 2 DCs worth].
13 Nodes per DC [$40 Nodes for 4096 MB is probably more than enough to test most issues]. $520
$560/month you can basically duplicate the 2 coast hot/cold setup with 26 servers.
But its expensive!
...unless this is a one-man operation, not really. A fully loaded junior engineer is a minimum of $120k. A robust staging environment is very useful and isn't going to cost more than 1 engineer month to setup.
Yes, you might not use the same hardware. However, it is close enough and honestly...the same hardware 1:1 with real servers [based on the ebay servers] and a rack at Fremont w/ Hurricane Electric is $600/month. 13 servers is $3,250 in hardware. Get someone else on the East Coast in like Buffalo, NY and you can get as close as your budget affords. I doubt you'd hit 1 Junior Engineer Year worth of $$.
Not really Free. Power and A/C are things you'll pay for but its not a notable cost [less than $100/month].
