"Hell, whole racks lose power at times. Doesn't happen all that often, but it happens often enough that if your provider says "We blew breaker X" well, more often than not, it's a honest problem, and not the FBI yanking power to image a drive."
Now you've got me wondering whether the apparent disparity between manufacturers claimed MTBF and what we see in failure rates in the real world, might plausibly be attributed to mysterious government agencies coercing data center owners into unexpected-but-plausible downtime. (four or five nines of power uptime might just mean the FBI/NSA need to batch server imaging and grab a whole bunch in a particular data center at once)
>Now you've got me wondering whether the apparent disparity between manufacturers claimed MTBF and what we see in failure rates in the real world, might plausibly be attributed to mysterious government agencies coercing data center owners into unexpected-but-plausible downtime. (four or five nines of power uptime might just mean the FBI/NSA need to batch server imaging and grab a whole bunch in a particular data center at once)
It's far more likely that people are idiots. How many hardware techs do you know who even own an ESD wrist strap? I get actively ridiculed when I pull mine out.
Next, the SLAs claimed by datacenters are usually bullshit on multiple levels.
First, the penalty is usually "we will refund you for the time you were down, if you ask." - which is fine, but a 5 minute power outage can be brutal to clean up after, while 5 minutes of your monthly bill is hardly worth asking for. I'd be happy to give people a 100% sla on those terms. I mean, obviously, the service isn't going to be up 100%, but the penalties are so low that who cares?
Then, well, even if the facility doesn't lose power, there are a hundred different ways a server or a rack can lose power.
Hell, even I let a guy into my co-lo who plugged in one of those ancient computers with a manual 110-240v switch. (everything made in the last decade auto-switches.) He plugged it into my 208v power, with the switch on 110, causing the fuses on my PDU to blow (and taking out the whole rack)
And power cords. Especially if you don't have dual power supplies, power cords get bumped. The mark of a honest sysadmin is that s/he admits it when they bump the cord[1]
So yeah, while it /could/ be the FBI, the vast majority of the time, well, someone fucked up.
(Adds lsc to the list of likely NSA collaborators…) ;-)
And yeah, you're right about hosting SLAs - I've got a hosting account which proudly advertises "100% uptime guarantee", which in the fineprint/t&cs offers "pro rata refunds for _twice_ your costs of any downtime!" – on a $48/year invoice - so if they go down for an entire _week_, they'll owe me not quite two whole dollars. Thanks...
Even the much more expensive/professional hosting I arrange for other clients always includes something like:
Limitation of Damages
Recovery of damages from $hostingCompany may not exceed
the amount of fees it has collected on the account.
Now you've got me wondering whether the apparent disparity between manufacturers claimed MTBF and what we see in failure rates in the real world, might plausibly be attributed to mysterious government agencies coercing data center owners into unexpected-but-plausible downtime. (four or five nines of power uptime might just mean the FBI/NSA need to batch server imaging and grab a whole bunch in a particular data center at once)