> It's ok to think that things have flaws or could be improved. But it's a trap to believe that solutions are trivial and everyone else is just too enfeebled of character to push the miracle button.
I feel this so much. I've worked on products which many people insist could be done just as well by spreadsheets. I've worked in areas which people believe are overcomplicated - which they are - and therefore could be replaced with something simple. It is never that easy.
I see it all the time in other fields and I just shake my head. If you ever find yourself saying "Why don't they just..?" you almost certainly don't understand the problem.
> I see it all the time in other fields and I just shake my head. If you ever find yourself saying "Why don't they just..?" you almost certainly don't understand the problem.
A political corollary to this that I enjoy came from an old professor of mine- "If your political opinions require large amounts of people to be stupid or evil, you don't understand the issues well enough." (A rough quote, I don't remember exact phrasing.)
I wish that was the banner at the top of every political discussion forum. Maybe with some auto-flagging on posts which can be condensed down to "everyone who disagrees with me is an idiot"
this seems to discount historically known political movements in which almost everyone involved, with the benefit of hindsight, was obviously stupid or evil.
Probably these are outliers, so also probably your professor's quote fits well enough most of the time, and only some of the time returns false when it should return true.
Usually those movements intend to remove "evil" people.
"We weren't evil, how can attacking evil people be evil?"
You can see people say things like this even today. The evil in these cases was done by the people convincing the masses that the other group really is evil and needs to be attacked, not by the masses. The masses will attack and punish whatever they think is evil, everyone knows that, doesn't mean that they are evil, rather it is always evil to label large groups of people as evil since it encourages the masses to attack those.
I agree with you, but I also kind of hate that "Why don't you just..?" has become so tainted, because it is a great question to ask if you want to learn more about the heart of the matter. What went wrong when the 'obvious' solution was tried is interesting!
Or, huh, just spitballing here, I suppose one could rephrase it as "have you considered..?" to ask the same question without the implicit assumption that the task is easy and the person being asked has overlooked something/is an idiot.
"Why don't they use containers instead of tarballs?" is a excellent question which can help you understand a system better.
"Why don't they just use containers instead of tarballs?" strongly suggests that containers are obviously superior and there must be some flaw in the system that's keeping the tarballs around.
I think "just" is supposed to imply that the proposed solution might be simpler or easier in some way, not that it's obviously superior overall.
[Of course in the case of containers vs. tarballs ... it might actually make the system more complex and more difficult to implement, use, and maintain - something which might be nice to state explicitly along with the reasons why. >;-)]
I agree with you that "questioning why" it generally a good thing for understanding. The issue here is one of communication, whereby the connotations of "just" may be taken differently by Asker and Responder.
A Responder could take the use of "just" as implying "obvious thing you personally missed" which derides your efforts/understanding/intellect/ego.
Whereas I see the use of "just" as saying something about the Askers mental model. The Asker knows their mental model says this is easy. The Asker knows you did not do [easy thing]. They want to know why there is a disparity.
Anecdotally, I have been hobbying away when my wife walks in - is given a brief explanation of what I am doing - says "Why don't you just [xyz]?"
She has limited information. She knows it. I know it. We both know her mental model is not like mine. But through her building it how she wants we can explore options. When she says "just" it is because her mental model says this should be easy.
And sometimes I eat humble pie because it is. Because my mental model was built from scratch from a different direction and my perspective means I didn't see [easy thing]. Not often. But often enough that I appreciate every time my wife asks.
Maybe the issue is who/how often you are asked the "just" question. Are you being bothered by too many people who don't-really-know/don't-really-care ? Then the Nth person coming up saying "why don't you just?" is another time/attention sapping interaction.
i like to phrase this sort of question as "i'm assuming you considered this simpler solution, why was that not sufficient?" sometimes the answer is that they didn't consider it and now feel dumb for not thinking of it, but i've never seen anyone get defensive or offended or upset when phrasing it that way.
As a new programmer, I was also seduced by the FP hype, and learned web development in ClojureScript without knowing JavaScript.
This was a really hard way to learn, and once I finally went mainstream I found that the messaging about how bad other languages were was much exaggerated.
In general I think the ecosystem trumps the language quality, and the senior developers who think otherwise generally underestimate the difficulty for a junior developer of working on the bleeding edge. Pragmatism over purity (while keeping enough purity-focus to keep the codebase clean).
I also think JS is absolutely fine as a language and its warts are not that hard to work around.
I'm really tempted to write a blog post about "fake functional".
The thing is, I really like functional. I really like Haskell. But most of the things that I like about Haskell aren't present when you rip things out of it willy-nilly and drop it into a language not designed for those idioms. The thing I like about functional programming is the fluid composition of all those ideas, and just dropping "map" or "option" doesn't mean you've got the fluid composition. Jamming functional paradigms into languages where they don't compose like they do in Haskell is just fake functional.
I'd much rather work with a language in the space where it does provide fluid composition of its concepts then jam a foreign idiom into it and enter into a constant low-level war, where I'm likely going to have to roll out the Inner Platform gun to try to wrangle the language into being a bad imitation of another language. And that's a gun I want to be firing as a last resort, not using as my primary weapon.
Functional programming can mean a lot of things to a lot of different people so declaring something “fake functional” is difficult.
If we get down to brass tacks, functional programming is just solving problems by composing functions, if we want to talk “purely functional” then we’re making more promises regarding side effects and immutability.
So I’d hesitate to call anything “fake functional” myself, I suspect what we call functional languages are often just “languages that do a good job letting us apply functional programming concepts”, and languages that don’t do that well will certainly not feel as good to use as ones that do (if you like the ideas in functional programming that is).
So in a way I agree. I personally can solve problems in Elixir using functional ideas more easily than Ruby. At the same time I could absolutely write a mutable, side effect ridden mess in Elixir or some pretty respectable functional looking code in Ruby.
I think it’s more about what the tool encourages and makes easy for you to do than a binary functional/not-functional designation.
In contrast, I really like the simplistic functional elements added to a language, like the Java 8 stream API, but I feel like banging my head against a wall using scala with effects systems.
I actually like some of the things added; being able to write a map function that takes an already-existing function and applies it to an array con be more convenient than the for loop. Depends on the language, obviously.
What I dislike are people who get on a high horse about it, as if they're programming "functionally" and it's better than what everyone else in the language is doing. They're not "programming functionally", because they can't be, the language doesn't support it. A single map call is convenient. Sometimes you can put a couple of them together. Huge piles of code in imperative languages written in functional paradigms is a mess, because most of the putative advantages of functional aren't present.
The one that I notice the most is that the "functional" paradigm embedded in imperative languages creates a barrier against refactoring. Sometimes it is simply that all the noise and chaff prevents programmers from seeing through to the underlying essense, and simply not realizing that if you have a recurring pattern of mapping the same three functions over an array in multiple places that you can in fact refactor that into a function. Sometimes, though, it's just not possible to refactor properly through inner platform you've introduced. Sometimes, refactoring is possible, but requires bringing in yet more firepower to get around the inner platform, like defunctionalization. By the time you're defunctionalizing something (by necessity, not choice) because of your inner platform to do something that would have been 4 lines of code in the original core language if you'd just used that, you're in deep trouble.
And what's worse, people will call this good programming. It's just the functional programming equivalent of frameworkitis that you see in languages like Java. "We need all this machinery to program properly!" No. You don't.
FP programmer here and I agree insofar as ecosystem matters a lot and you want to be able to onboard juniors easily and on. To that end a pure FP language is often not the best choice, especially in lowering the learning curve. However it can be great a choice on the right project and with the right team. Overall it makes sense that we have way more JS and Go programmers than Haskell or 'pure FP' Scala. JS is fine as a language but so is F#, they fit different niches and different contexts.
I love the diversity of our industry and I see having all these different paradigms and languages as a massive strength. I welcome them all.
Not something on a scale of a project, but "bowling score" exercises are a lot more elegant in functional languages than more procedural ones due to things like pattern matching.
If you compare JS (not TS, TS is fine) and PHP (I have extensive experience in both) you'll find that once you go beyond the syntax on the top you'll find the same flaws in some form or another.
Yes, PHP is uglier, but that is about it.
> but I start do doubt the same can be said about the ecosystem.
JS is a horrible language that works because of the tireless effort of its community that has created an ecosystem so rich that even JS can be programmed efficiently.
The ecosystem might be flawed but it is the only reason why JS is usable at all.
I really enjoy writing JS/TS - I find that eg arrow functions mixed with React allow for really concise and expressive solutions. You are correct about the ecosystem... maybe not per se, but that supply chain attacks are hopelessly intractable at the moment. How could you possibly go about writing a modern SPA that notionally cares about security when the third party software you must rely on is impossible to verify? I would love if browsers let you run JS in a sand boxed mode wherein you could just pass a bundle of vendored upstream software without the ability to touch the window, network, global state, etc.
I don’t disagree, but for someone who has been using JS for almost a decade, it is liberating and relaxing to use Clojure. Similar can be said about other languages that have a deliberate and thoughtful design.
but for someone who has been using JS for almost a decade, it is liberating and relaxing to use Clojure.
Can you elaborate what you find liberating and relaxing in Clojure vs modern JS?
Putting aside runtime features like multi-threading the only thing I miss a bit after having use Clojure for one year was macros. But the for the rest, is all the same, just functions and data, and some polymorphic abstractions here and there.
Tons of things actually. If I have to choose a few:
For one, JS feels clunkier with its expression/statement mish-mash. Once I started getting expression based syntax, everything made so much sense. It's such a simple feature that lets code compose granularily and is one of the key enablers for _seamless_ interactive development (and macros). One instance where this is pronounced is when you compare React JSX with Reagent and ClojureScript.
Secondly, you can do FP in JS but no matter how you do it, it feels tacked on, is typically less performant, doesn't compose as well and relies more on discipline because it isn't the default. I'm not saying FP all the things either. It's just that in interactive web GUI's you have tons of state and asynchronicity, so the type of bugs and problems that occur from mismanaging those are incredibly frustrating. There are JS libraries that mutate the stuff you pass into for no reason at all. FP by default is very, very relaxing and composes well.
Sure but those are all minor things (and greatly exaggerated) in the grand scheme of shipping stuff and producing value IMO, today, a language being mainstream and its ecosystem matters. You are falling for the exact trap the article warns about.
You'll say you can use the Java ecosystem but them you loose all the benefits you mentioned, you say you will keep interop/mutation/side effects isolated but then again you can do the same in JS if some lib is causing unwanted mutation.
I have seen JS developers run in circles around Clojure developers because the JS devs have specialized in their language and ecosystem while the others keep moving between ecosystems and niche languages hoping for some magical thing that will fix all their problems.
Rich Hickey is a great software engineer and software architect but what most don't notice is that he also is a good salesman.
There is no "trap" I fell into here. The point of the article you refer to was that there is more value in being more proficient in some language/tech choice rather than jumping around and that the ecosystem matters.
From the article
> If anything, my progress was often hampered by the lack of libraries, unreliable tools and not spending enough time in any one ecosystem to develop real fluency. These got in the way of working on hard problems, and working on hard problems was the main thing that actually led to improvement.
> By way of counter-example, check out this ICFP contest retrospective. Nikita is using clojure, a pretty niche language, but has built up incredible fluency with both the language and the ecosystem so that he can quickly throw out web scrapers and gui editors. Whereas I wouldn't be able to quickly solve those problems in any language after flitting around from ecosystem to ecosystem for 12 years.
And I largely agree with this.
> You'll mention you can use the Java ecosystem but them you loose all the benefits you mentioned, you say you will keep interop/mutation/side effects isolated but then again you can do the same in JS if some lib is causing unwanted mutation.
Been there, done that and I explained honestly above why I prefer having more explicit and built-in boundaries, rather than tacking it on via libraries and discipline. Interop, mutation and side-effects stand out and there are utilities that let you do those things in a convenient, productive way without you having to worry about copying stuff around or similar.
There are some really good libraries in the JS world too, that are a joy to work with and are powerful productivity wise. No doubt about that. I merely cited the reasons most important to me why I have a preference and why it feels liberating and relaxing to me to use Clojure over JS in general. There are more, and there are also some downsides. None of them de-validate your judgement or preference.
while FP is hyped, it is also fundamentally good technology. I was very skeptical of Haskell for years, and finally really tried it, and I dont think I have ever changed my mind so completely on anything.
Don't get me wrong. It has warts. And you're right, a jr who has problems with like configuring their environment variables correctly for example is going to really have a hard time. This is mostly why I think that while I'd still recommend Haskell to people, you can't just throw it on a team of newly grads and expect it to work out. Not that you can do that with JS either per se, but you CANNOT do it with Haskell.
Sure, JS has warts, and a lot of them are bad, but heck, Haskell has a lot of warts too. They all have warts. Any language that has been around for any length of time has generally had its share of ideas which didn't really work out over the long term.
For me the bigger issue with JS is the ecosystem quality. If you aren't on the golden path of package usage, you may find that you are in for a world of hurt.
Here’s a meta question to the author: Having been around the circuit once, do they at least feel a little more humble about their opinions this time around, or are they just as strong but only in a different direction? :-)
IMHO, while the author has undoubtedly grown wiser, I think there’s just as much truth to their earlier opinions as the later ones; the value lies in being able to disambiguate the context in which each is valuable — not so much in trading one set of opinions for another. That humility, to me, is the bigger takeaway.
It did take me a while to write this (~20 hours, I think) because I kept worrying about that exact problem, and I kept catching myself writing generic advice rather than concrete experiences.
I tried as much as possible to only write things where I could think of multiple concrete examples in my own experience.
By way of counter-example, check out this ICFP contest retrospective. Nikita is using clojure, a pretty niche language, but has built up incredible fluency with both the language and the ecosystem so that he can quickly throw out web scrapers and gui editors. Whereas I wouldn't be able to quickly solve those problems in any language after flitting around from ecosystem to ecosystem for 12 years.
The thing with niche languages like Clojure is that you can leverage existing ecosystems(reducing the risk of getting stuck) unlike other languages that start from scratch. Probably the same for languages like F#, Kotlin and even Zig (using the C ecosystem).
Of course, I found the article great with lots of good advice(ex: I still wouldn't use Clojure if Javascript can get the job done perfectly fine) so don't let this nitpick distract you or look the other way ;)
> It's so easy to think that simple solutions exist. But if you look at the history of ideas that actually worked, they tend to only be simple from a distance. The closer you get, the more you notice that the working idea is surrounding by a huge number of almost identical ideas that don't work.
This brings to mind, two of my favorite H. L. Mencken quotes:
"There's always an easy solution to every human problem; Neat, plausible and wrong."
"The fact that I have no remedy for all the sorrows of the world is no reason for my accepting yours. It simply supports the strong probability that yours is a fake."
Simple solutions do exist... it's just, they're simple, so people solve them very quickly and nobody else even has time to start processing them as a "problem".
If people are still arguing about something ten years later, it's because it's not simple. But there's lots of stuff we don't argue about, because it's solved, because it was simple.
Simple solutions can be really hard to find sometimes (you know P != NP and all that). The thing you're talking about is solutions that are simple to think of.
> Being fluent in the core language of mathematics (basic logic
I have been programming for a living for 15 years, I'm most definitely not as good a programmer as OP is, but the least I can do is double-vouch for that advice. I first heard it a little over 20 years ago, at the end of my first calculus class at uni.
After scaring us, his students, about all the things that we will have to learn during the next semester I remember the professor telling us that we should be really learning what the NOT, OR and AND logical operators really do and stick that piece of information firmly in our heads, because that is really important for our jobs. He also bitched a little about our older colleagues for not remembering that stuff anymore (and causing different computer bugs because of that) and I remember saying to myself: "how stupid can one be in order to un-learn this stuff? It is easy!".
Like I said, 20 years have passed since then and I found out that even though it's easy lots of people seem to forget how basic logic actually works when it comes to programming.
> Being fluent in the core language of mathematics (basic logic, sets, functions, proof techniques) makes it easy to pick up domain-specific tools when I need them eg statistical relational learning when working at relational.ai, bidirectional type inference for imp.
I feel the most important part of learning higher math for programming is that you learn what it means for a statement to be true. Most people, even most programmers, don't really grasp that which makes most harder problems intractable to them as you can't build a stable tower on "basically true" statements.
Also even higher math is all about creating new useful abstractions for solving new problems, that is basically the same kind of thinking as writing useful reusable programming libraries.
If his baseline was Haskell. It was his first programming language. Then of course coding something else won't make him a better programmer. He is already a genious.
> Life is short and you don't get to learn more than a tiny fraction of the knowledge and skills available, so if you want to make really cool stuff then you need to spend most of your time on the highest-leverage options and spend only a little time on the lottery tickets.
Solid gold advice, and this summarizes the whole article.
Great article, I was nodding my head with almost every point! Although I think you also have to consider that the author has a strong math background, which both improves and warps your view of programming. If you don't know any math, then this post is probably less relevant to you (e.g. learning a bit Haskell or ML is probably a good idea in that case).
------
As a nitpick about the theory of automata, I would draw a big red line between regular languages and context free grammars.
Regular languages have very useful engineering properties: they have predictable (linear) performance and give you "free" lookahead. (That is, nondeterminism is a math-y idea that many programmers are not comfortable with, but it's useful for engineering.)
They are better than Perl-style regexes for building reliable and understandable systems, which "blow up" on a regular basis.
In contrast, context-free grammars have almost no useful engineering properties by themselves. (There are subsets like LALR(1) that do, but they come with a bunch of tradeoffs.) As the article mentions, learning about recursive descent parsing first is probably more practical.
Also, you're MUCH more likely to encounter a regular expression in real code than a context free grammar. In 15 years of professional programming I probably dealt with regexes on a weekly or monthly basis, but had to write or modify a grammar exactly zero times. It does help to be more familiar with regexes and regular languages.
> If you don't know any math ... learning a bit Haskell or ML is probably a good idea
This is very much counter to the point I was making. I didn't find that learning haskell or ml was anything like learning math, especially the kind of math that I've seen people get use out of in day-to-day engineering, and I don't understand why this idea is so persistent.
If you don't know any math, it might be worth learning some of whatever kind of math is applicable in your field. It's probably not haskell.
> It does help to be more familiar with regexes and regular languages.
Regexes are certainly useful, but I don't think that learning a lot about automata theory is a particularly efficient way to get better at using regexes in practice, compared to eg doing a bunch of drills on https://www.executeprogram.com/courses/regexes
Hm reading this section again, I don't doubt that you had that experience, but I doubt it generalizes.
I think the "latin" claim is pretty close to true -- getting exposure to different languages helps your general programming ability (though I have no idea if it's true for Latin itself :) ) You get to see how other people solve problems, and play with different abstractions, and both of those are valuable things.
For my particular experience, I've written about 80% Python code, 15% C/C++, and 5% other languages for 10+ years. So thoroughly mainstream, and I agree that's the way to get work done. But I did do SICP as a freshman in college and I think that permanently changed my thinking about programming.
I also spent a few weeks with Real World OCaml in 2014, and hacked on some open source OCaml code. That experience directly helped me with implementing a language (Oil). That might not generalize either, but I'd still stand by the claim that people who don't know any math could benefit from learning a functional language.
It's a long argument that's been made many times elsewhere, but exhaustive case analysis, recursion, and various types of composition are second nature to some programmers but not others. I also think it helps with the "basic logic, sets, functions" areas you point out. There is A LOT of production code in the world that can use help with these things.
Like another commenter, I do think it's possible to swing too far in the other direction :) You probably internalized a lot of stuff a long time ago from either math or Haskell that helps in programming, but then you have "the curse of knowledge" -- it's hard to imagine what it's like not to know those things.
----
I also disagree about regexes -- the main problem is that the syntax varies so much in practice, and people hack or copy and paste until it works (the regex as lingua franca paper in my blog post above talks about this.) I think learning the simple mathematical concepts of alternation, repetition, etc. first is actually easier than doing drills for JavaScript regex syntax, and you can use that knowledge in practical contexts like the shell (where none of the common tools use JavaScript syntax).
If none of that convinced, I now remember that John Carmack (who you quoted!) has advocated functional programming as a way to improve the design of your programs (and he also used Racket for a VR project as far as I remember):
Probably everyone reading this has heard "functional programming" put forth as something that is supposed to bring benefits to software development, or even heard it touted as a silver bullet ...
It isn't immediately clear what that has to do with writing better software. My pragmatic summary: A large fraction of the flaws in software development are due to programmers not fully understanding all the possible states their code may execute in. In a multithreaded environment, the lack of understanding and the resulting problems are greatly amplified, almost to the point of panic if you are paying attention. Programming in a functional style makes the state presented to your code explicit, which makes it much easier to reason about, and, in a completely pure system, makes thread race conditions impossible.
> Also, you're MUCH more likely to encounter a regular expression in real code than a context free grammar. In 15 years of professional programming I probably dealt with regexes on a weekly or monthly basis, but had to write or modify a grammar exactly zero times. It does help to be more familiar with regexes and regular languages.
This massively depends on what you’re doing. I’ve been programming for 30+ years, 16 of them professionally and not gone near regular expressions beyond a general knowledge of their existence and uses. Mind you the same is also true of context free grammars which is something I dimly remember learning about at university.
The tension here is that we all understand the concept of being 'too close to the problem'. It's the opposite fence-post from Dunning-Kruger, and the truth likes somewhere between them, not between DK and some center line.
It's not universally the case that an unbiased observer is lacking critical details that explain why the problem hasn't been solved better. The observer has different frames of reference, one that might include wisdom from other problem domains that have already solved a similar problem. The observer is unbiased or differently biased. They have no sunk cost fallacy. They have not attached their sense of identity to the existing work.
Best of all, they have not attached their sense of self to the struggle that got us where we are today. People who identify with their suffering resist attempts to fix it. Aggressively. Because if they sit with the idea that it could have ended years ago if only we had done X, that means they needlessly suffered. It makes them feel foolish, and small. The suffering has to mean something, other than that we were just grinding our gears.
Peter Norvig in his essay "Teach Yourself Programming in Ten Years"[0] wrote this:
> Learn at least a half dozen programming languages. Include one language that emphasizes class abstractions (like Java or C++), one that emphasizes functional abstraction (like Lisp or ML or Haskell), one that supports syntactic abstraction (like Lisp), one that supports declarative specifications (like Prolog or C++ templates), and one that emphasizes parallelism (like Clojure or Go).
There is also similar advice from other well respected veterans. Being able to choose from a smallish deliberate set of languages that introduce some powerful concept each or have some unique reach.
Then there is "Beating the Averages"[1] by Paul Graham.
> A suspicious person might begin to wonder if there was some correlation here. A big chunk of our code was doing things that are very hard to do in other languages. The resulting software did things our competitors' software couldn't do. Maybe there was some kind of connection. I encourage you to follow that thread. There may be more to that old man hobbling along on his crutches than meets the eye.
I think this hint is important and relates to the above. There are things that you can do in X, that are "very hard" to do in Y, probably vice versa. It is powerful and helpful to know a heterogeneous set of languages and tools.
The distinction between say Java, C# or Typescript is almost arbitrary from a technical, real brain "juice" kind of perspective. The distinction between say SQL, C or Lisp however lets you see a problem in a whole different light as each introduce things you cannot do easily in the other.
Example: Why does a particular Blub webapp suck? Because it runs slow, changes to it take too much time, it has data consistency issues that it moves to the user or introduces as bugs? Hint: There are solutions to these problems and some technologies make them just fall out, almost for free, if one has taken the time to learn them.
I don't yell this down from an ivory tower. I say this as someone who has went through the pain repeatedly, because I and most people around me didn't get this. Pragmatism is not always "what everyone else does."
And coming back to the article; I don't say the ecosystem and the culture around a language or platform don't matter. It is just one factor of many. When looking at these things I found it is _useful_ to consider languages (and so on) that are slightly niche and opinionated enough so they make particular things easy that would otherwise be hard.
I feel this so much. I've worked on products which many people insist could be done just as well by spreadsheets. I've worked in areas which people believe are overcomplicated - which they are - and therefore could be replaced with something simple. It is never that easy.
I see it all the time in other fields and I just shake my head. If you ever find yourself saying "Why don't they just..?" you almost certainly don't understand the problem.