Thank you for the kind words! Are you saying that AI model innovation stopped at GPT-2 and everyone has performance and gpu utilization figured out?
Are you talking about NVIDIA Hopper or any of the rest of the accelerators people care about these days? :). We're talking about a lot more performance and TCO at stake than traditional CPU compilers.
I’m saying actual algorithmic (as in not data) model innovation has never been a significant part of the revenue generation in the field. You get your random forest, or ResNet, or BERT, or MaskRCNN, or GPT-2-with-One-Weird-Trick, and then you spend four hours trying to figure out how to preprocess your data.
On the flipside, far from figuring out GPU efficiency, most people with huge jobs are network bottlenecked. And that’s where the problem arises: solutions for collective comms optimization tend to explode in complexity because, among other reasons, you now have to package entire orchestrators in your library somehow, which may fight with the orchestrators that actually launch the job.
Doing my best to keep it concise, but Hopper is like a good case study. I want to use Megatron! Suddenly you need FP8, which means the CXX11 ABI, which means recompiling Torch along with all those nifty toys like flash attention, flashinfer, vllm, whatever. Ray, jsonschema, Kafka and a dozen other things also need to match the same glibc and glibc++ versions. So using that as an example, suddenly my company needs C++ CICD pipelines, dependency management etc when we didn’t before. And I just spent three commas on these GPUs. And most likely, I haven’t made a dime on my LLMs, or autonomous vehicles, or weird cyborg slavebots.
So what all that boils down to is just that there’s a ton of inertia against moving to something new and better. And in this field in particular, it’s a very ugly, half-assed, messy inertia. It’s one thing to replace well-designed, well-maintained Java infra with Golang or something, but it’s quite another to try to replace some pile of shit deep learning library that your customers had to build a pile of shit on top of just to make it work, and all the while fifty college kids are working 16 hours a day to add even more in the next dev release, which will of course be wholly backwards and forwards incompatible.
Lattner's comment aside (which I'm fanboying a little bit at), I do tend to agree with your pessimism/realism for what it's worth. It's gonna be a long long time before that whole mess you're describing is sorted out, but I'm confident that over the next decade we will do it. There's just too much money to be made by fixing it at this point.
I don't think it's gonna happen instantly, but it will happen, and Mojo/Modular are really the only language platform I see taking a coherent approach to it right now.
I tend to agree with you, but I hoped the field would start collectively figuring out how to be big boys with CICD and dependency management back in 2017–I thought Google’s awkward source release of BERT was going to be the low point, and we’d switch to Torch and be saved. Instead, it’s gotten so much worse. And the kind of work that the Python core team has been putting into package and dependency management is nothing short of heroic, and it still falls short because PyTorch extends the Python runtime itself, and now Torch compile intercepting Py_FrameEval and NVIDIA is releasing Python CUDA bindings.
It’s just such a massive, uphill, ugly moving target to try to run down. And I sit here thinking the same as many of these comments—on the one hand, I can’t imagine we’re still using Python 3 in 2035? 2050?? But on the other hand I can’t envision a path leading out of the mess making money, or at least continue pretending they’ll start to soon.
Mojo doesn't have C++-like exceptions, but does support throwing. The codegen approach is basically like go's (where you return a bool + error conceptually) but with the python style syntax to make it way more ergonomic than Go.
So what did you decide to give up on? Overloading functions with the same name, or bidirectional constraint solving? :)
These days though the type checker is not where compile time is mostly spent in Swift; usually it’s the various SIL and LLVM optimization passes. While the front end could take care to generate less redundant IR upfront, this seems like a generally unavoidable issue with “zero cost abstraction” languages, where the obvious implementation strategy is to spit out a ton of IR, inline everything, and then reduce it to nothing by transforming the IR.
Bidirectional constraint solving. It's bad for compile time but even worse for predictable diagnostics. Mojo does contextual resolution, but it works more similar to how C++ resolves initializer lists etc.
> Bidirectional constraint solving. It's bad for compile time but even worse for predictable diagnostics.
That’s really only true if you have overloading though! Without overloading there are no disjunction choices to attempt, and if you also have principal typing it makes the problem of figuring out diagnostics easier, because each expression has a unique most general type in isolation (so your old CSDiag design would actually work in such a language ;-) )
But perhaps a language where you have to rely on generics for everything instead of just overloading a function to take either an Int or a String is a bridge too far for mainstream programmers.
Mojo has overloading, generics and a much more advanced type system than Swift (dependent and linear types etc), and compile time in all phases is very important. The Mojo design seems to be working well - it gives expressive power, good error messages etc.
I wasn’t trying to start a fight over languages, that would be silly. I also wrote a language once and then moved on from it (to your former language ;-)), so I know the feeling! I wish you luck with your new language, and I wish for many more languages in the future to try different approaches and learn from each other.
My original reply was just to point out that constraint solving, in the abstract, can be a very effective and elegant approach to these problems. There’s always a tradeoff, and it all depends on the combination of other features that go along with it. For example, without bidirectional inference, certain patterns involving closures become more awkward to express. You can have that, without overloading, and it doesn’t lead to intractability.
Sure, I wasn't trying to start a fight either, I was just sharing my experience and opinion on having worked on both. Mojo (and C++) have closures, for example c++ does lambda type inference without a constraint solver.
In my opinion, constraint solving would be a bad design point for Mojo, and I regret Swift using it. I'm not trying to say that constraint solving is bad for all languages and use-cases.
Thank you, that's not entirely wrongbut not the full picture. Our initial explanation had two problems actually,
1) we were "ambitiously optimistic" (different way of saying "ego-driven naïveté" perhaps :) ) and
2) the internet misread our long-term ambitions as being short-term goals.
We've learned that the world really really wants a better Python and the general attention spans of clickbait are very short - we've intentionally dialed back to very conservative claims to avoid the perception of us overselling.
Thank you for all the great interest in the podcast and in Mojo. If you're interested in learning more, Mojo has a FAQ that covers many topics (including "why not make Julia better" :-) here:
https://docs.modular.com/mojo/faq/
I watched a lot of your talks about mojo, where you mention how mojo benefits from very advanced compiler technology. But i've never seen you give concret example of this advanced technology. Please, can you write a blog about that, going as deep and hardcore tech as you can ? As i'm not a compiler dev i'll probably understand 20% of it, but hopefully i'll start to get a sense of how advanced the whole thing is.
They have some fancy tech around compile-time interpreter, mojopkg, and their code generator. They also have a graph compiler (not mentioned in the talk) that can fuse kernels written in Mojo.
interesting talk. I thought mojo actually had innovations at the IR level as well, but it seems to rely on what chris built at google with MLIR.
Chris mentioned pushing updates to MLIR while working at Mojo, i'd be curious to have a summary what those changes are, because i believe this is where the hard core innovations are happening.
Thanks for Swift, one of the best languages I've used (even if the tooling around it leaves something to be desired), along with GDScript, for different reasons ^^
The FAQ answers: "Is the Mojo Playground still available?" by pointing me to the playground, but, the playground itself says "Playground will be removed with the next release of 25.6"
As a FAQ answer to "is this available?" pointing me to a thing which says it will be removed at some unspecified point sort of misses the point. I guess maybe the answer is "Not for long" ?
That's a great idea. Matt is a great guy but also there's so much stuff in Compiler Explorer unrelated to the particulars of a programming language and you get to leverage all of that by just being there next to everything else.
I don't think Mojo is a big part of my future, but I've been wrong before and being in Compiler Explorer can't hurt you.
Super impressive app and experience, it is incredible that you can get Swift to do this with such interactivity! Rebuilding Swift to be interpreted is a bold move,
Are you talking about NVIDIA Hopper or any of the rest of the accelerators people care about these days? :). We're talking about a lot more performance and TCO at stake than traditional CPU compilers.