Forget the study, let's just do a simple thought experiment. Your developer gets paid $140k/yr (let's round up to ~$70/hr). Let's say a given bug found in testing takes 1 hour to fix; that's $70 (not counting the costs of ci/cd etc). If they miss it in test, and it hits production, would it cost $7,000 to fix? Depends what you mean by "bug", what it affects, and what you mean by "fix in production".
- Did you screw up the font size on some text you just published? Ok, you can fix that in about 5 seconds, and it affects pretty much nothing. Doesn't cost 100x.
- Did your sql migration just delete all records in the production database? Ok, that's going to take longer than 5 seconds to fix. People's data is gone, apps stop working, the lack of or bad data fed to other systems causes larger downstream issues, there's the reputational harm, the money you'll have to pay back to advertisers for their ads / your content being down, and all of that multiplied by however long it takes you to restore the database from backup (um... you do test restoring your backups... right?). That's closer to 100x more expensive to fix in production.
- Did you release a car, airplane, satellite, etc with a bug? We're looking at potentially millions in losses. Way more than 1000x.
And those are just the easy ones. What about a bug you release, that then is adopted (and depended on) by downstream api consumers, and that you then spend decades to patch over and engineer around? How about when production bugs cause your product team to lose confidence in deployments, so they spend weeks and weeks to "get ready" for a single deploy, afraid of it failing and not being able to respond quickly? That fear will dramatically slow down the pace of development/shipping.
The "long tail" of fixing bugs in production involves a lot more complexity than in non-production; that's where the extra cost comes from. These costs could end up costing 10,000x over the long term, when all is said and done. Security bugs, reliability bugs, performance bugs, user interface bugs, etc. There's a universe of bugs which are much harder/costlier to fix in production.
But you know what is certain? It always costs more to fix in production. 1.2x, 10x, 1000x, that's not the point; the point is, fix your bugs before it goes to production. ("Shift Left" is how we refer to this in the DevOps space, but it applies to everything in the world that has to do with quality. Improve quality before it gets shipped to customers, and you save money in the long run.)
>> Did you screw up the font size on some text you just published? Ok, you can fix that in about 5 seconds, and it affects pretty much nothing. Doesn't cost 100x.
Actually, i find these to be worse, because i've been in scrum meetings where 6 people spend 2 minutes talking about this bug, then another 2 minutes talking about the QA of it the next day. Tiny issues are very expensive to fix if you have formulaic team members who arent taking the reigns.
And you're lucky if it's two. If they're not familiar with the rendering engine/docs system/API platform/middleware etc (or just think they aren't), or have low confidence in a debt ridden platform, they'll spend five minutes a day for a week or three theorizing on how it could have gone wrong, debating if, actually, it's correct the way it is, refreshing their knowledge of deep internals, debating if a fix could break something, and so on. Way safer to do that than risk making things worse in an uncertain system.
I will willingly risk karma hazard by suggesting this isn't about "bugs leaking into later phases" but about "corporate culture allowing endless nitpicking".
(Which I reckon is what my parent comment is saying too!.)
Yep, bike-shedding in action - the most mundane issues (and features too) will generate way too much discussion. At least with some complicated internal issue, no clueless manager is able to offer their 2c on how to fix it.
Bugs are more like viruses in practice. The cost is useful to measure in negative lifespan, not cost to fix, per se. This is why many bugs are never fixed. Those cost nothing to fix, because they don't have to be.
> Did your sql migration just delete all records in the production database? That's closer to 100x more expensive to fix in production.
Companies that do this often, don't stay in business. It's not 100x more expensive if you're not in business. Survivorship ensures that classes of bugs don't have a consistent negative return, because they are often fatal.
Is the opportunity cost of one hour of developer time only $70? Is every hour spent testing guaranteed to fix one bug?
I feel like the logic here gets a little twisted because it's comparing the value of a known outcome with an earlier probability. You can save millions of dollars by buying a winning lottery ticket before they announce the number.
- Did you screw up the font size on some text you just published? Ok, you can fix that in about 5 seconds, and it affects pretty much nothing. Doesn't cost 100x.
- Did your sql migration just delete all records in the production database? Ok, that's going to take longer than 5 seconds to fix. People's data is gone, apps stop working, the lack of or bad data fed to other systems causes larger downstream issues, there's the reputational harm, the money you'll have to pay back to advertisers for their ads / your content being down, and all of that multiplied by however long it takes you to restore the database from backup (um... you do test restoring your backups... right?). That's closer to 100x more expensive to fix in production.
- Did you release a car, airplane, satellite, etc with a bug? We're looking at potentially millions in losses. Way more than 1000x.
And those are just the easy ones. What about a bug you release, that then is adopted (and depended on) by downstream api consumers, and that you then spend decades to patch over and engineer around? How about when production bugs cause your product team to lose confidence in deployments, so they spend weeks and weeks to "get ready" for a single deploy, afraid of it failing and not being able to respond quickly? That fear will dramatically slow down the pace of development/shipping.
The "long tail" of fixing bugs in production involves a lot more complexity than in non-production; that's where the extra cost comes from. These costs could end up costing 10,000x over the long term, when all is said and done. Security bugs, reliability bugs, performance bugs, user interface bugs, etc. There's a universe of bugs which are much harder/costlier to fix in production.
But you know what is certain? It always costs more to fix in production. 1.2x, 10x, 1000x, that's not the point; the point is, fix your bugs before it goes to production. ("Shift Left" is how we refer to this in the DevOps space, but it applies to everything in the world that has to do with quality. Improve quality before it gets shipped to customers, and you save money in the long run.)