>At this point, we know that it's not enough to hire brilliant C programmers and have them exercise care.
I mostly agree with your points. In my experience (never worked at FAANG), the brilliant C programmer hypothesis is almost untested, since most companies hire mostly average C programmers. It's only logical that non-brilliant programmers will be working somewhere.
Even if a company has a few brilliant C programmers on staff, they won't be able to find all the issues the mediocre programmers produce.
This is why I think it's about time that any safety critical industry should think of retiring C in favor of safer alternatives.
Rust would personally be my favorite since it solves most of the really terrible issues around memory management in C and C++ (mediocre programmers not being able to compile due to the borrow checker is a feature).
Of course Rust doesn't have the necessary certifications yet, but industry would be wise to contribute. Such efforts are underway (Sealed Rust by Ferrous Systems), but I don't see big aerospace, automotive and industry sponsorship here, i.e. those who would benefit the most.
There is a safer language that is certified that is often ignored, which is ADA. It's actually quite modern in terms of features since the 2012 release. It has some other concepts than Rust for safety, but that doesn't mean it's worse. The heavy lifting in terms of design and certification here has been paid for by the DoD and aerospace companies, so there's no reason other industries couldn't use it.
> the brilliant C programmer hypothesis is almost untested, since most companies hire mostly average C programmers. It's only logical that non-brilliant programmers will be working somewhere.
Chromium has suffered from buffer-overflow issues, and that's a high-profile security-sensitive C++ codebase run by Google.
> Rust doesn't have the necessary certifications yet, but industry would be wise to contribute
I'd rather avionics code be written in verified SPARK than in Rust, but if they can build a rock-solid Rust compiler it would be good to have it compete in that space. I imagine it could be a nice alternative to MISRA C++, for instance.
As you've probably gathered from my mention of SPARK, I agree with you that Ada is sadly overlooked.
Turns out that the basic idea of an imperative language that's much like C/C++ but is much safer and better suited to automatic runtime checks, isn't a new one. Ada does this, it's existed since around 1980, it's mature, there are various compilers for it approved for life-critical work, and its performance matches that of C if runtime checks are disabled.
It's comparable to Zig and Rust in that regard, but those languages aren't the first to aim for C performance with far superior safety properties.
> there's no reason other industries couldn't use it.
Right, especially considering the fine work done by AdaCore, who make their tooling available as Free and Open Source software.
Perhaps Ada's reputation for being boring is its greatest vice, as well as its greatest virtue?
>Chromium has suffered from buffer-overflow issues, and that's a high-profile security-sensitive C++ codebase run by Google.
Like I said, I haven't worked at Google. But are you sure every programmer on Chromium is a genius? Google has a hard Interview process, but does it filter out every bad engineer?
And if Google's "probably better than other company's" engineers can't do it? How can we expect other companies to deal with C?
>I'd rather avionics code be written in verified SPARK than in Rust, but if they can build a rock-solid Rust compiler it would be good to have it compete in that space. I imagine it could be a nice alternative to MISRA C++, for instance.
Right now SPARK would definitely be favorable. But I think even current Rust would be much safer than MISRA C++.
> are you sure every programmer on Chromium is a genius? Google has a hard Interview process, but does it filter out every bad engineer?
I suspect that yes, it filters out all candidates who are outright bad. I don't think this point really matters though. If even Google aren't able to produce a large C++ codebase free of buffer-overflows, even when it really matters, it suggests that no-one can. edit This doesn't need to be an argument drawn from a single sample, either. Buffer overflows happen to just about all major C/C++ codebases. OpenSSL, the Linux kernel, Windows, the Apache Web Server, nginx, etc.
We could quibble about the way they don't use the sort of methodology used in avionics software development, but I don't think there's much value in exploring that. If Google consider that to be too slow/costly for Chromium, it highlights how rarely those methodologies can practically be used.
I don't think there are any individual C++ programmers who are too smart to ever introduce undefined behaviour, but even if there were it wouldn't matter. As you indicate, real large-scale software development happens with teams, not lone geniuses.
> I think even current Rust would be much safer than MISRA C++.
I agree that the language might be a reasonable choice, but I don't think the Sealed Rust project is anywhere near delivering a Rust compiler that can be trusted with your life. Even mature C compilers can have serious bugs, [0] and compiling Rust is far more challenging than compiling C.
I've run into (multiple) compiler bugs in certified C and C++ compilers, I know how difficult this is. But still a much larger percentage of bugs I've seen was introduced by the developers. Which is why I said even now Rust would probably beat MISRA C++ in terms of total amount of bugs. I do think Sealed Rust has a bit of a way to go, but it would be a worthy investment.
Seen from the point of total risk, the compiler errors are much harder to find, but the developer errors can still kill you.
In any case SPARK is indeed even farther ahead in terms of having a reliable toolchain paired with a language that makes safe code easier to produce.
"Even if a company has a few brilliant C programmers on staff, they won't be able to find all the issues the mediocre programmers produce."
Also, I am very sceptical, that even brilliant C programmers won't have a bad day, lack of sleep or distracted by personal problems - to avoid misstakes.
And about ADA, have you experience with it? What could be the reason, it is not used on a broader base? I assume also lack of libaries, etc.?
I mean, there are millions of free and open C and C++ libaries around for allmost any usecase, I assume this is not the case with ADA?
Sorry to double reply but I see I missed some of your points:
> Basically with ADA, you develope much slower (and performance is usually lower)
Ada deliberately emphasises readability over writeability. It might take a bit longer to do initial development work in Ada than in C, although it's likely that you'll encounter fewer bugs, so Ada might win out even here. Subsequent maintenance is likely to be cheaper/easier. I believe comparison studies have borne this out, although I'm not certain how trustworthy they are.
As for performance: with runtime checks disabled, Ada should perform about the same as C/C++, as its abstractions don't tend to have runtime costs, it's just pretty plain imperative code. Ada isn't typically used with garbage collection, for instance. Like C++, you have the option of using the language's OOP features.
With runtime checks enabled, you'll pay some runtime performance cost (let's say very roughly 15%, to simply make up a number out of nowhere). C and C++ don't give you the option of enabling as many runtime checks as Ada does, due to the free-form way arrays work, for instance. gcc and clang both give you the option of enabling runtime checks for certain kinds of errors, such as dereferencing NULL, but plenty of other kinds of error won't be caught.
> But if you want reliability, it does sound good.
Ada has some traction in the critical-systems space. The Boeing 777 flies on Ada, for instance. Of course, C and C++ are also both used for safety-critical software.
I think that's a part of it. Few GUI toolkits can easily be used from Ada, for instance. Ada has excellent support for interfacing with C code, but making decent bindings is never trivial.
I think lack of hype might be part of it too. Ada is seen as a stodgy and boring language good for writing autopilot software, whereas Rust and Zig are seen as new and exciting general-purpose programming languages. I'm not saying Rust and Zig bring nothing new to the table, of course, as they both do, but in terms of safety and programming with an emphasis on correctness, Ada has a lot to offer.
I have only hobbyist experience with ADA. It didn't seem nearly as hard as a lot of people said it was, and I was intrigued about in which ways it's safer than C.
My best guess is that people are unfamiliar with ADA, unwilling to learn, and most of all believe that hiring will turn into a problem if they switch to ADA. Also they bring up the brilliant programmer theory.
The hiring problem seems superficially compelling.
There are indeed fewer ADA programmers out there, which of course is a chicken and egg problem if companies are not trying to adopt it.
Now the fourth argument often follows the third argument. My problem with it is that most of the programmers I've worked with that I'd describe as brilliant are the first ones to tell you that we should use a safer language, because they know even they make mistakes. And a lot of the brilliant programmers wouldn't have trouble picking up ADA or learning something new.
So my answer would be that if you truly have brilliant programmers, you don't need to worry about hiring. If you don't have those brilliant programmers you really can't afford staying with C because you will have huge costs from debugging and being sued for the bugs you didn't find.
A process that can not survive a single programmer having a bad day is not a high reliability process. That problem is trivially easy to "solve" in any number of ways.
1. You can have multiple people whose job is to fully understand that code and review it.
2. You can have independent 3rd party reviewers whose job is fully understand the code and review it.
3. You can have adversarial reviewers from your competitors review the code with veto power over your code if they discover and can prove there is a defect.
4. You can have a requirements doc that traces to the code and a corresponding test to exercise that property.
5. You can verify that everybody understands the source by reading the compiled output to verify that it correctly traces to the expected source constructs.
6. You can have multiple people develop independent systems with the same functionality to run in parallel to crosscheck results with each other.
7. You can have those systems developed in isolation from each other so that they share no code or ideas giving even greater independence.
8. You can run these systems on the same and different hardware in parallel to provide even more independence and crosschecking.
Every single one of these raises the bar from a single programmer having a bad day to requiring multiple people independently failing on the same code which is multiplicatively less likely. Every single one of these is something that is actually done for high criticality avionics software. Any "high-reliability" process that seriously considers a single programmer having a bad day as a material unsolved risk is made by complete amateurs. This does not mean high criticality avionics software is perfect, it just means that the concerns are more about actually hard problems such as how to prevent errors where a dozen brilliant programmers cross-checking each other will be unable to prevent the problem without the process.
To put the difference in reliability between these systems into perspective, a "high-reliability" commercial system like AWS only promises 99.99% uptime in their standard SLA. In contrast, the 737 MAX, which most people consider an absolute deathtrap and evidence of the terrible quality of avionics code, had one failure per ~200,000 flights or 99.999% per-flight reliability (10x the reliability if we compare number-of-flights to seconds-of-uptime). If we were to consider all high-criticality avionics software, avionics software has not been implicated in any crash for the last 10 years except for the 737-MAX, so at ~10,000,000 flights per year for a total of ~100,000,000 that means avionics software has 99.999998% per-flight reliability or 5,000x the reliability if we compare number-of-flights to seconds-of-uptime.
The 737 Max crashes weren't the result of failures in software development. As far as the crashes go, the software engineers built the system they were asked to build, and the problems were at a higher level than the software (i.e. aeronautical engineering and systems engineering).
Software issues have since been discovered, but aren't thought to have been related to the crashes, as I understand it.
I actually happen to agree with you, but I deliberately decided to take the worst possible interpretation of the lowest reliability system (that is worse by orders of magnitude than the average) to highlight the differences in outcome. If the absolute dead-last system, a system that most would view as shameful and many an unconscionable deathtrap, is an order of magnitude better than best-practices in commercial software, it is probably not a good idea to learn from commercial software vendors or use any knowledge based on their inexperience as they can not even consistently make systems as unreliable as an unconscionable deathtrap, so there is no reason to believe they actually know how to make systems of higher reliability than that.
I mostly agree with your points. In my experience (never worked at FAANG), the brilliant C programmer hypothesis is almost untested, since most companies hire mostly average C programmers. It's only logical that non-brilliant programmers will be working somewhere.
Even if a company has a few brilliant C programmers on staff, they won't be able to find all the issues the mediocre programmers produce.
This is why I think it's about time that any safety critical industry should think of retiring C in favor of safer alternatives.
Rust would personally be my favorite since it solves most of the really terrible issues around memory management in C and C++ (mediocre programmers not being able to compile due to the borrow checker is a feature).
Of course Rust doesn't have the necessary certifications yet, but industry would be wise to contribute. Such efforts are underway (Sealed Rust by Ferrous Systems), but I don't see big aerospace, automotive and industry sponsorship here, i.e. those who would benefit the most.
There is a safer language that is certified that is often ignored, which is ADA. It's actually quite modern in terms of features since the 2012 release. It has some other concepts than Rust for safety, but that doesn't mean it's worse. The heavy lifting in terms of design and certification here has been paid for by the DoD and aerospace companies, so there's no reason other industries couldn't use it.