For sure. Maybe less accidental complexity. Embedded has been doing asynchronous IO since before it was cool. You also have distributed computing if you’re doing something like sensor networks.
I was studying for my MSc in CS some 25 years ago. Our curriculum included both automata/formal languages (multiple courses over multiple semesters) and parallel programming.
Since then, I've repeatedly felt that I've seriously benefited from my formal languages courses, while the same couldn't be said about my parallel programming studies. PVM is dead technology (I think it must have counted as "nearly dead" right when we were using it). And the only aspect I recall about the formal parallel stuff is that it resembles nothing that I've read or seen about distributed and/or concurrent programming ever since.
A funny old memory regarding PVM. (This was a time when we used landlines with 56 kbit/s modems and pppd to dial in to university servers.) I bought a cheap second computer just so I could actually "distribute" PVM over a "cluster". For connecting both machines, I used linux's PLIP implementation. I didn't have money for two ethernet cards. IIRC, PLIP allowed for 40 kbyte/s transfers! <https://en.wikipedia.org/wiki/Parallel_Line_Internet_Protoco...>
Sure, I did the same, BS/MS with a focus on Compilers/Programming Languages. It's been personally gratifying to understand programming "end-to-end" and to solve some tricky problems, but 99% of folks aren't going to hit those problems. There are tons of people interacting with Cloud Services every day that aren't aware of the basic issues like:
- Consistency models (can I really count on data being there? What do I have to do to make sure that stale reads/write conflicts don't occur?)
- Transactions (this has really fallen off, especially in larger companies outside of BI/Analytics)
- Causality (how can I handle write conflicts at the App Layer? Are there Data Structures ie CDTs that can help in certain cases?)
Even basic things like "use system time/monotonic clocks to measure elapsed time instead of wall-clock time" aren't well known, I've personally corrected dozens of CRs for this. Yes this can be built in to libs, AI agents etc but it never seems to actually be, and I see the same issues repeated over-and-over. So something is missing at the education layer
For the very good reason that the underlying math is insanely complicated and tiresome for mere practitioners (which, although I have a background in math, I openly aim to be).
For example, even if you assume sequential consistency (which is an expensive assumption) in a C or C++ language multi-threaded program, reasoning about the program isn't easy. And once you consider barriers, atomics, load-acqire/store-release explicitly, the "SMP" (shared memory) proposition falls apart, and you can't avoid programming for a message passing system, with independent actors -- be those separate networked servers, or separate CPUs on a board. I claim that struggling with async messaging between independent peers as a baseline is not why most people get interested in programming.
Our systems (= normal motherboards on one and, and networked peer to peer systems on the other end) have become so concurrent that doing nearly anything efficiently nowadays requires us to think about messaging between peers, and that's very-very foreign to our traditional, sequential, imperative programming languages. (It's also foreign to how most of us think.)
Thus, I certainly don't want a simple (but leaky) software / programming abstraction that hides the underlying hardware complexity; instead, I want the hardware to be simple (as little internally-distributed as possible), so that the simplicity of the (sequential, imperative) programming language then reflect and match the hardware well. I think this can only be found in embedded nowadays (if at all), which is why I think many are drawn to embedded recently.
I think SaaS and multicore hardware are evolving together because a queue of unrelated, partially ordered tasks running in parallel is a hell of a lot easier to think about than trying to leverage 6-128 cores to keep from ending up with a single user process that’s wasting 84-99% of available resources. Most people are not equipped to contend with Amdahl’s Law. Carving 5% out of the sequential part of a calculation is quickly becoming more time efficient than taking 50% out of the parallel parts, and we’ve spent 40 years beating the urge to reach for 1-4% improvements out of people. When people find out I got a 30% improvement by doing 8+6+4+4+3+2+1.5+1.5 they quickly find someplace else to be. The person who did the compressed pointer work on v8 to make it as fast as 64 bit pointers is the only other person in over a decade I’ve seen document working this way. If you’re reading this we should do lunch.
So because we discovered a lucrative, embarrassingly parallel problem domain that’s what basically the entire industry has been doing for 15 years, since multicore became unavoidable. We have web services and compilers being multi-core and not a lot in between. How many video games still run like three threads and each of those for completely distinct tasks?
Personally I've been inspired by nnethercote's logs (https://nnethercote.github.io/) of incremental single-digit percentage performance improvements to rustc over the past several years. The serial portion of compilers is still quite significant and efforts to e.g. parallelize the entire rustc frontend are heroic slogs that have run into subtle semantic problems (deadlocks and races) that have made it very hard to land them. Not to disparage those working on that approach, but it is really difficult! Meanwhile, dozens of small speedups accumulate to really significant performance improvements over time.
I know c++ has a lack luster implementation, but do coroutines and channels solve some of these complaints? although not inherently multithreaded, many things shouldn't be multithreaded , just paused. and channels insteaded of shared memory can control order
Coroutines basically make the same observation as transmit windows in TCP/IP: you don’t send data as fast as you can if the other end can’t process it, but also if you send one at a time then you’re going to be twiddling your fingers an awful lot. So you send ten, or twenty, and you wait for signs of progress before you send more.
On coroutines it’s not the network but the L1 cache. You’re better off running a function a dozen times and then running another than running each in turn.
I've found both explicit future/promise management and coroutines difficult (even irritating) to reason about. Co-routines look simpler at the surface (than explicit future chaining), and so their the syntax is less atrocious, but there are nasty traps. For example:
I think trying to shoehorn everything into sequential, imperative code is a mistake. The burden of performance should be on the programmer's cognitive load, aided where possible by the computer. Hardware should indeed be simple, but not molded to current assumptions. It's indeed true that concurrency of various fashions and the attempts at standardizing it are taxing on programmers. However, I posit this is largely essential complexity and we should accept that big problems deserve focus and commitment. People malign frameworks and standards (obligatory https://xckd.com/927), but the answer is not shying away from them but rather leveraging them while being flexible.
Distributed systems require insanely hard math at the bottom (paxos, raft, gossip, vector clocks, ...) It's not how the human brain works natively -- we can learn abstract thinking, but it's very hard. Embedded systems sometimes require the parallelization of some hot spots, but those are more like the exception AIUI, and you have a lot more control over things; everything is more local and sequential. Even data race free multi-threaded programming in modern C and C++ is incredibly annoying; I dislike dealing with both an explicit mesh of peers, and with a leaky abstraction that lies that threads are "symmetric" (as in SMP) while in reality there's a complicated messaging network underneath. Embedded is simpler, and it seems to require less that practitioners become advanced mathematicians for day to day work.
Most embedded systems are distributed systems these days, there's simply a cultural barrier that prevents most practitioners from fully grappling with that fact. A lot of systems I've worked on have benefited from copying ideas invented by distributed systems folks working on networking stuff 20 years ago.
I worked in an IoT platform that consisted of 3 embedded CPUs and one linux board. The kicker was that the linux board could only talk directly to one of the chips, but had to be capable of updating the software running on all of them.
That platform was parallelizable of up to 6 of its kind in a master-slave configuration (so the platform in the physical position 1 would assume the "master role" for a total of 18 embedded chips and 6 linux boards) on top of having optionally one more box with one more CPU in it for managing some other stuff and integrating with each of our clients hardware. Each client had a different integration, but at least they mostly integrated with us, not the other way around.
Yeah it was MUCH more complex than your average cloud. Of course the original designers didn't even bother to make a common network protocol for the messages, so each point of communication not only used a different binary format, they also used different wire formats (CAN bus, Modbus and ethernet).
But at least you didn't need to know kubernetes, just a bunch of custom stuff that wasn't well documented. Oh yeah and don't forget the boot loaders for each embedded CPU, we had to update the bootloaders so many times...
The only saving grace is that a lot of the system could rely on the literal physical security because you need to have physical access (and a crane) to reach most of the system. Pretty much only the linux boards had to have high security standards and that was not that complicated to lock down (besides maintaining a custom yocto distribution that is).
Many automotive systems have >100 processors scattered around the vehicle, maybe a dozen of which are "important". I'm amazed they ever work given the quality of the code running on them.
Indeed. I've been building systems that orchestrate batteries and power sources. Turns out, it's a difficult problem to temporally align data points produced by separate components that don't share any sort of common clock source. Just take the latest power supply current reading and subtract the latest battery current reading to get load current? Oops, they don't line up, and now you get bizarre values (like negative load power) when there's a fast load transient.
Even more fun when multiple devices share a single communication bus, so you're basically guaranteed to not get temporally-aligned readings from all of the devices.
I run a small SaaS side hustle where the core value proposition of the product - at least what got us our first customers, even if they did not realize what was happening under the hood - is, essentially, an implementation of NTP running over HTTPS that can be run on some odd devices and sync those devices to mobile phones via a front end app and backend server. There’s some other CMS stuff that makes it easy for the various customers to serve their content to their customers’ devices, but at the end of the day our core trade secret is just using a roll-your-own NTP implementation… I love how NTP is just the tip of the iceberg when it comes to the wicked problem of aligning clocks. This is all just to say - I feel your pain, but also not really since it sounds like you are dealing with higher precision and greater challenges than I ever had to!
Here’s a great podcast on the topic which you will surely like!
The ultimate frustration is when you have no real ability to fix the core problem. NTP (and its 'roided-up cousin PTP) are great, but they require a degree of control and influence over the end devices that I just don't have. No amount of pleading will get a battery vendor to implement NTP in their BMS firmware, and I don't have nearly enough stacks of cash to wave around to commission a custom firmware. So I'm pretty much stuck with the "black box cat herding" technique of interoperation.
Yeah, that makes sense. We are lucky in that we get to deploy our code to the devices. It’s not really “embedded” in the sense most people use as these are essentially sandboxed Linux devices that only run applications written in a programming language specific to these devices which is similar to Lua/python but the scripts get turned into byte code at boot IIRC, but none the less very powerful/fast.
You work on BMS stuff? That’s cool- a little bit outside my domain (I do energy modeling research for buildings) but have been to some fun talks semi-recently about BMs/BAS/telemetry in buildings etc. The whole landscape seems like a real mess there.
FYI that podcast I linked has some interesting discussion about some issues with PTP over NTP- worth listening to for sure.
Yes even 'simple' devices these days will have devices (ADC/SPI etc) running in parallel often using DMA, multiple semi-independent clocks, possibly nested interrupts etc. Oh and the UART for some reason always, always has bugs, so hopefully you're using multiple levels of error checking.
Yeah, it was a "fun" surprise to discover the errata sheet for the microcontroller I was working with after beating my head against the wall trying to figure out why it doesn't do what the reference manual says it should do. It's especially "fun" when the errata is "The hardware flow control doesn't work. Like, at all. Just don't even try."
> Distributed systems require insanely hard math at the bottom (paxos, raft, gossip, vector clocks, ...) It's not how the human brain works natively -- we can learn abstract thinking, but it's very hard.
I think this take is misguided. Most of the systems nowadays, specially those involving any sort of network cals, are already distributed systems. Yet, the amount of systems go even close to touching fancy consensus algorithms is very very limited. If you are in a position to design a system and you hear "Paxos" coming out of your mouth, that's the moment you need to step back and think about what you are doing. Odds are you are creating your own problems, and then blaming the tools.
I remember when I prepared for system design interviews in FAANG, I was anxious I would get asked about Paxos (which I learned at school). Now that I'm working there, never heard about Paxos or fancy distributed algorithms. We rely on various high-level services for deployment, partitioning, monitoring, logging, service discovery, storage...
And Paxos doesn't require much maths. It's pretty tricky to consider all possible interleavings, but in term of maths, it's really basic discrete maths.
I'm starting to believe these talks of fancy high-complexity solutions come from people who desperately try to come up with convoluted problems they create for themselves only to be able to say they did a fancy high-complexity solution. Instead of going with obvious simple reliable solutions, they opt for convoluted high-complexity unreliable hacks. Then, when they are confronted by the mess they created for themselves, they hide behind the high-complexity of their solution, as if the problem was the solution itself and not making the misjudged call to adopt it.
It's so funny how all of a sudden every single company absolutely must implement Paxos. No exception. Your average senior engineer at a FANG working with global deployments doesn't come close to even hearing about it, but these guys somehow absolutely must have Paxos. Funny.
this is completely backwards. the tools may have some internal consistency guarantees, handle some classes of failures, etc. They are leaky abstractions that are partially correct. There were not collectively designed to handle all failures and consistent views no matter their composition.
From the other direction, Paxos, two generals, serializability, etc. are not hard concepts at all. Implementing custome solutions in this space _is_ hard and prone to error, but the foundations are simple and sound.
You seem to be claiming that you shouldn't need to understand the latter, that the former gives you everything you need. I would say that if you build systems using existing tools without even thinking about the latter, you're just signing up to handling preventable errors manually and treating this box that you own and black and inscrutable.
- You work on postgres: you have to deal with the transaction engine's internals.
- You work in enterprise application intergration (EAI): you have ten legacy systems that inevitably don't all interoperate with any one specific transaction manager product. Thus, you have to build adapters, message routing and propagation, gateways, at-least-once-but-idempotent delivery, and similar stuff, yourself. SQL business logic will be part of it, but it will not solve the hard problems, and you still have to dig through multiple log files on multiple servers, hoping that you can rely on unique request IDs end-to-end (and that the timestamps across those multiple servers won't be overly contradictory).
In other words: same challenges at either end of the spectrum.
Yeah this is kind of an abstraction failure of the infrastructure. Ideally the surface visible to the user should be simple across the entire spectrum of use cases. In some very, very rare cases one necessarily has to spelunk under the facade and know something about the internals, but for some reason it seems to happen much more often in the real world. I think people often don't put enough effort into making their system model fit with the native model of the infrastructure, and instead torture the infrastructure interface (often including the "break glass" parts) to fit into their a priori system model.
That's true, but you can do a lot of that once, and then get on with your life, if you build the right structures. I've gotten a huge amount of mileage from consensus to decide where to send reads/writes to, then everyone sends their reads/writes for the same piece of data to the same place; that place does the application logic where it's simple, and sends the result back. If you don't get the result back in time, bubble it up to the end-user application and it may retry or not, depending.
This is built upon a framework of the network is either working or the server team / ops team is paged and will be actively trying to figure it out. It doesn't work nearly as well if you work in an environment where the network is consistently slightly broken.
> Even data race free multi-threaded programming in modern C and C++ is incredibly annoying; I dislike dealing with both an explicit mesh of peers, and with a leaky abstraction that lies that threads are "symmetric" (as in SMP) while in reality there's a complicated messaging network underneath.
If you're using traditional (p)threads-derived APIs to get work done on a message passing system, I'd say you're using the wrong API.
More likely, I don't understand what you might mean here.
Sorry, I figure I ended up spewing a bit of gibberish.
- By "explicit mesh of peers", I referred to atomics, and the modern (C11 and later) memory model. The memory model, for example as written up in the C11 and later standards, is impenetrable. While the atomics interfaces do resemble a messaging passing system between threads, and therefore seem to match the underlying hardware closely, they are discomforting because their foundation, the memory model, is in fact laid out in the PhD dissertation of Mark John Batty, "The C11 and C++11 Concurrency Model" -- 400+ pages! <https://www.cl.cam.ac.uk/~pes20/papers/topic.c11.group_abstr...>
- By "leaky abstraction", I mean the stronger posix threads / standard C threads interfaces. They are more intuitive and safer, but are more distant from the hardware, so people sometimes frown at them for being expensive.
> finally they are telling me they are doing a task so I know we are going in the right direction
Yes, but. :)
Assume a high-profile open source project where your direct report is a maintainer. The higher-profile the project is, the more political it becomes, of course. Your report will not need your (= the manager's) input on technical decisions; instead, they will inform you (in the optimal case) of where things have been heading. However, if that maintainer also participates in community governance -- which is quite likely --, they won't be able to avoid decisions that are more political than technical in nature. And whatever they decide there may easily reflect on their employer or client, one way or another. That kind of stuff is something that they should consult you on, regardless of their technical seniority.
Yes, but NATO members are not bound by contracts that classify them into superiors and subordinates. Whereas at work, you'll have some form of contract in which you agree to taking -- and usually: soliciting -- direction from your superior or client. Silence procedure works between equals, where each party is required, from the get-go, to (a) announce their intent to the shared message bus, (b) diligently listen to the shared message bus, and take action whenever necessary. The communication pattern between manager and report is totally different; the parties are not equals there.
Nice; instead of burdening your boss with a decision, now you're burdening them with a ultimatum, with a deadline. I guess, as a manager, I wouldn't love anything more than a report saddling me with an arbitrary deadline, one I'd have to schedule along with the rest.
One of my managers used to tell me this, instead: ask for forgiveness, not permission. That one seems way saner to me.
Agreed. I actually think consensus amongst the team is more important than your boss. If a group of reports under some manager come to consensus on something they need to do, the opinion of the boss is kind of moot. I tend to operate using a bias for action, and the boss is clued in with information about our decision making, but asking for permission (or forgiveness) really just opens the door for micromanagement. This assumes basic trust has been established between the team and the manager, but in the end the technical decisions should be chosen by those on the team who are technical (read: not management).
Management (in my opinion at least) is primarily for resource allocation, career growth, and shielding the team from endless meetings and acting as a point of contact for upper management
> Management (in my opinion at least) is primarily for resource allocation, career growth, and shielding the team from endless meetings and acting as a point of contact for upper management
Agreed! (And we can also call "resource allocation" "setting priorities".)
> One of my managers used to tell me this, instead: ask for forgiveness, not permission. That one seems way saner to me.
So you are suggesting making the change without consultation and then waiting for it to be discovered?
I guess that makes sense for certain kinds of situations, but the ones I can imagine don't sound very pleasant. Maybe I'm missing something. What is an example of a situation where you'd take this approach?
It’s really about (a) the wording (b) the level of risk.
For something that isn’t risky, I trust my reports to make reasonable decisions and wouldn’t benefit from the “I’m going to do this tomorrow unless you say no” approach. Instead, they can just tell me they’re doing it (or let me know after the fact, or add/FYI me on the code review, or not even mention it, depending on what it is).
For a decision that is more important/higher risk, they should get affirmative agreement rather than just hoping that I see the ultimatum and silently approve.
That’s why I’m with the GP that the ultimatum-with-deadline doesn’t seem like the best choice in any situation.
Thanks for the feedback. I guess I'd put the 'ultimatum-with-deadline' in between the two risk profiles you outline--something just on the edge of risk.
> Just hoping that I see the ultimatum and silently approve.
I never would have thought of this as an ultimatum--to me it's more like a 'default decision' that makes my manager's life easier while still looping them in and giving them control should they choose to exercise it.
But now I can definitely see how it might be seen as one, depending on the type of decision and the relationship between employee and manager. I should probably update the post to say that you need a degree of trust and alignment for this technique to work; you shouldn't roll this out on the first day.
The advice refers to situations where you, the report, are both able and (well-informedly) willing to take full responsibility. "Ability" here means that, if your decision backfires, you can revert it, and/or contain the damage otherwise. Otherwise, you may be willing to take responsibility, but are unable to. In other words, this piece of advice applies to things that are ultimately under your control.
The background is that, even when something is (mostly) in your control, you may be tempted to ask for permission, in advance, just to distribute the responsibility to others (shift the blame, cover your ass). That delays things, and usually the manager will sense that they got burdened with the request-for-permission somewhat needlessly. In those cases, it's better to take initiative, and be accountable later on, because the latter is in your power, in the end.
(Of course, if you and your manager have dedicated time slots anyway, then bringing the topic up is prudent, as it will not require them to scramble for otherwise unallocated time.)
Conversely, if containing the potential damage is indeed not in your power, then you shouldn't go ahead without explicit permission. For things where you simply can't bear responsibility, a timeout from the approver is not a default "yes", but a default "no". If you can't bear responsibility, then you need explicit signoff from someone higher up that, should shit hit the fan, they will bear responsibility on your behalf -- and so a timeout is meaningless (it doesn't give you what you responsibly need).
Whenever you can afford to interpret a timeout whichever way you want, then you don't need to ask the question in the first place (--> don't ask for permission, just revert/contain the damage, if needed). Otherwise (--> the potential damage is beyond you), you need an explicit "yes" (which is the only case when you're off the hook).
The problem with your suggestion is that it assumes that you, as a report, are in a position to set deadlines for your superior(s); in other words, to allocate their time and priorities. Usually the exact opposite is true (by contract): it's your manager who sets your priorities. You can consider their (repeated?) failure to respond timely a true failure, but that doesn't give you the right to do whatever you want; it would be in bad faith / a form of vigilantism. Your option is to leave that manager.
Right, that's called communication. Communication also enables you to ask your manager if this approach is okay to begin with, or ask that they acknowledge when you give them such heads up notifications. There's a lot of ways to make this work well.
> smart humans learned how to hide behind systems to avoid accountability
And "systems" is more general in this sense than just technology. Consider stoning, and firing squads. Both were invented to remove individual responsibility.
