The expertise in question isn’t really used at “hyperscale” companies. Not everyone wants to work at large companies anyway. And the expertise isn’t that difficult to train or acquire (compared to getting another software engineer).
> Or, to put it another way, mid-sized company X can never pay as much as hyperscale company Y for Z expertise. Because utilization at mid will always be less than at hyper.
Being able to run a PostgreSQL cluster is a handy skill but won’t get you hired at Google.
Google can afford to pay an engineer to make cool stuff like a faster malloc implementation or analyze the best possible way to encode data to be stored on disk. That’s because making Google’s malloc 1% faster will pay someone’s salary many times over, or likewise for saving 1% CPU in your disk servers.
But running your own database cluster and container orchestration ain’t exactly rocket science these days, and running your own stack means you have an expert in-house when the shit hits the fan—and that’s when your expert earns their salary, many times over.
One root cause I've seen over during database outages is insufficient IO. It's caused by the other most common root cause: lack of actual expertise. Until you've been through the wringer, it's easy to lack respect for data: how important it is, how fucking large it is, and how long it takes to do basically anything with a few TB of it. If you didn't hire a real expert, and your DB-for-a-day guy miscalculated the spec for your bare metal clusters, you'll find it extremely hard to magic up faster hardware during a 4am full-outage when Europe wakes up. If you're on cloud, at least you have the flexibility to increase block storage IOPS or quickly reboot a node on beefier hardware or faster network connections. But suddenly the "cheap" DIY database is a lot more expensive than you budgeted for, and the phrase "nobody needs a dedicated DB guy" starts sounding really short sighted.
I’m talking about managing your own software on IaaS as an alternative to running off SaaS, you seem to think I’m talking about ditching the cloud entirely!
> Or, to put it another way, mid-sized company X can never pay as much as hyperscale company Y for Z expertise. Because utilization at mid will always be less than at hyper.
Being able to run a PostgreSQL cluster is a handy skill but won’t get you hired at Google.
Google can afford to pay an engineer to make cool stuff like a faster malloc implementation or analyze the best possible way to encode data to be stored on disk. That’s because making Google’s malloc 1% faster will pay someone’s salary many times over, or likewise for saving 1% CPU in your disk servers.
But running your own database cluster and container orchestration ain’t exactly rocket science these days, and running your own stack means you have an expert in-house when the shit hits the fan—and that’s when your expert earns their salary, many times over.