Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
We want to make Nix better (determinate.systems)
223 points by biggestlou on Sept 2, 2022 | hide | past | favorite | 198 comments


Writing the packaging expressions should not be harder than writing the program, for the average developer. The average developer doesn't know FP, at this point. Nix needs:

1. Approachability for those not indoctrinated in lazy, declarative, functional programming (i.e. Haskell); nope, Nix Pills are not sufficient for the average dev

2. Editor tooling to help guide the writing of expressions (just as anyone learning C# or Rust can use LSP); a better type system for Nix would help here

3. Better documentation for practical things like "Using Nix to manage the dependencies and package a C++ program using Meson", rather than having people piece this together from a bunch of disparate docs

4. A much better CLI UX than `nix-env -qa` and the like (this is ongoing and experimental, but even that broke recently, causing lots of confusion; now it requires `nix --extra-experimental-features nix-command --extra-experimental-features flakes search nixpkgs`)

5. To seriously answer the question: is the Nix language required for the Nix packaging system to exist? Laziness is required, to some degree, but can the next iteration provide an on-ramp which doesn't involve learning a new lang and paradigm? Guix folks sure think so.

I feel like Nix folks have been focused so long on solving the tough problems of declarative, deterministic packaging that they haven't been able to focus on the UX. I also feel like folks for whom Haskell is comfortable may not realize just how absurd it feels to everyone else. Perhaps like the early days of Git.

I do really hope Nix succeeds in this, though; I've been using it, or it's been using me, for several years. [1] and [2] for more info.

[1]: https://blog.jeaye.com/2015/11/24/nixos/

[2]: https://blog.jeaye.com/2017/07/30/nixos-revisited/


Having used Nix a bunch, the system has been hard to use and learn in exactly the same ways that messy legacy codebases I've worked with have been hard—which makes sense since Nixpkgs is an absolutely massive codebase that's evolved over the years with very distributed development. People latch onto the Nix language and the paradigm because that's the most tangible difference but, everything else held equal, Nix + Nixpkgs would be no easier to work with if everything were in Python. (And, if my limited experience is any guide, would be worse with an ad-hoc DSL pretending to be Python à la Starlark!)

This is a pattern I've seen over and over: if a project tries to do something different, the novel aspect gets blamed for issues that would not be surprising in any other context. When faced with a massive awful codebase in Java, well, sometimes legacy code can be a pain, right? It is what it is. People rarely jump to "OOP is a mistake" or "Java is completely unusable". But that's exactly what happens all the time with, say, functional programming.

Pointing to an uncommon choice as the underlying cause of a problem makes for a nice, pat story; the pervasive, diffuse problems inherent to messy legacy code do not. After all, here's a single thing that, presumably we can actually change! Nix being functional is an objective fact while people can—reasonably or not—disagree on how well-designed and well-written Nixpkgs-the-codebase is. Besides, even if we could agree, what could we do about it? We all know how much momentum legacy code has and how hard it is to make anything beyond local improvements. Being a socially convenient story doesn't mean it's a particularly strong explanation, but it does mean that people will believe it even if it isn't.


Yeah IMHO nix just needs someone like Brian Kernighan to sit down and write the definitive book about it and its internals. Something that clearly explains the practical and theoretical uses of nix and nixpkgs.

Nix pills, the official manual, etc. aren't really there yet--they talk about all kinds of paradigms that might (?) be going away like nix-env and its ilk. It's just extremely painful to track down the current best practice and way to do things in nix--you're bouncing off all kinds of docs and finding multiple conflicting or out of date ways to do things. Someone needs to give folks a slim book about nix flakes and the flakes way of doing things, the nix language, and a tour of nixpkgs, and IMHO there will be a lot more people using it.


I dislike that the way nix is onboarded for new folks is nix-env, and then for the next documentation to say: nix-env is bad don't use it.

If it was bad why show it to me at all?


Eelco likes `nix-env` and its new equivalent, but not many other people in the community share this opinion... That's why its in the nix documentation but everyone else uses something else.


I think the “famous” ‘Simple Made Easy’ talk makes this difference you mean quite palpable: simple is an objective term meaning something is not “interconnected” (not complected) with other parts of the code, while easy is a relative term originating from something being close to us, it being familiar.

So to most programmers, imperative code is easy and FP is hard, but on an objective basis both can be simple. In a way FP may be simpler (according to the talk), as e.g. mutable variables complect/mix together values and time.


I agree, even with FP and (some) Haskell experience, customising Nix and debugging what's going on is made hard by Nixpkgs being a complex codebase more than anything. However, it is different than the usual large web app. The problem of packaging software is fundamentally easy to break down into tiny pieces, the package definitions themselves. Arch's PKGBUILDs were very easy to understand for me as a novice, much better than RPM SPECs or whatever Debian has, so I think it's not impossible for Nix-equivalent packages to be similarly simple.

The biggest barrier, as I see it, is not actually Nix or Nixpkgs code, but the way NixOS differs from every other Linux distro. Most of my delving into Nixpkgs has been motivated by getting something working that would be trivial in a traditional distro. I agree the Nix language is a red herring, what's really needed is to make work other than "adding a package to Nixpkgs" easier. Actually usable introspection tools that the GP mentioned would be great.


> Besides, even if we could agree, what could we do about it?

The same thing we do for other paradigms (system architecture, programming concepts, etc.) - find and document common patterns, then find ways to transition to the patterns away from things out of pattern, and lastly introduce automatic refactorings that do fix these things in bulk.

I definitely agree with you on the evolution thing. Homebrew is a similar DSL for installing packages on macOS. It has the problem where things get deprecated from the DSLs (Casks and Formulae) which leads to not being able to run various commands until previously installed software is fixed.


Fascinating take! I've never heard this view before, thank you for sharing it!


> People rarely jump

I'd just like to take a moment to reiterate that object-oriented programming is a mistake produced by cargo-culting half a dozen different styles used by competent programmers (including the devolpers of smalltalk and its relatives, from which OOP appropiated its name), and that Java is completely unusable by anyone with a sense of self-respect and a awareness that programming languages can be better than least-common-denominator noe-COBOLs. I'm not familiar with the specific codebase you're refering, but it would probably be at least less massive and less awful if it had been written in a more decent language, like Haskell or C.


That’s like, your opinion, and we all know the common analogy to that :D

Java is a tiny language with not too many concepts going around sitting on top one of the best runtimes with one of the biggest ecosystems. It is a very sane and good choice for almost every program out there, especially that empirically we have absolutely zero evidence for any significant difference between language productivities, other than managed languages being a boost (so your mention of C, which is arguably a language with so many faults one could fill a whole book about, is frankly, quite laughable).


If it would be unusuable it would not have been used to do so many useful things. By all means, there may be better options out there but it is far from as bad as you describe it.


> To seriously answer the question: is the Nix language required for the Nix packaging system to exist? Laziness is required, to some degree, but can the next iteration provide an on-ramp which doesn't involve learning a new lang and paradigm? Guix folks sure think so.

I'd love to hear from someone deeply familiar with Nix and Guix about laziness.

I'm deeply familiar with Nix and I've concluded that lazy semantics is absolutely critical for a configuration language. It lets me refer to other attributes of my configuration from anywhere. For example, I can refer to port number from my whatever service in my firewall. Nix's system of overlays depends on laziness too to provide efficient late-binding familiar from OOP.

I don't need to topologically sort the evaluation of the various inter-dependencies of my configuration. So long as there exists an evaluation order, laziness finds it.

Laziness is compelling enough that I managed to convince the author of Jsonnet <https://jsonnet.org/> of it when he was designing it, and in turn he helped me design what is now known as overlays in Nix.

I don't even understand how Guix manages to work without laziness, though clearly it does somehow. I'm curious as to how that is possible, though I fear I will only ever truly understand by diving into Guix.


As an 18 month Nix packager/user, I continue to be blown away by how powerful laziness is and how expressive the interactions between multiple overlays can be.


I'm not convinced that any of the things you mentioned as benefits of lazy evaluation are a necessary condition to getting those benefits.


It’s not. Guix has exactly the same properties while using an eager language.


> Nix's system of overlays depends on laziness too to provide efficient late-binding familiar from OOP.

As an OOPer, inheritance is one of the things that is generally trotted out as the worst aspect of OO languages. This leads to the mantra "favor composition over inheritance". Much has been written about why and how. But it seems to cause the same pain points that people talk about with respect to laziness in Nix. "Spooky action at a distance" is one of my favorite ways of describing some of these problems.

(I'm not super familiar with Nix, so I could be misreading the points about laziness. Please forgive me if so.)


I actually agree with you about inheritance and OOP. It's funny because when I finally understood how to override packages with this late-binding technique, I said to myself "Finally I've found a use for inheritance!".

I certainly think people should exercise caution as this could probably be abused quite badly. Maybe people will come up with better designs in the future. However I will say, that the method it has replaced was much worse. The old way was adhoc and almost always failed to achieve ones desires in all but the simplest cases.

Don't just take my word for it. Peter Simons also agrees: <https://github.com/NixOS/nixpkgs/commit/3c8b33eee442fd573d47...>


> I don't need to topologically sort the evaluation of the various inter-dependencies of my configuration. So long as there exists an evaluation order, laziness finds it.

That’s not a property of laziness but of how Nix evaluates configurations. I think people overvalue what laziness brings them because they attribute to it things which could be done equally well with an eager language.


There is nothing special about how Nix evaluate system configurations that is any different from how Nix evaluates any other expression.

Being able to write

     let usercfg = { username = "nix-user";
                     homedir = "/home/" + usercfg.username;
                   };
     in usercfg
is nearly a hallmark of non-strict semantics, which is achieved by lazy evaluation.

Strict languages can simulate these non-strict semantics by using functions of no parameters to delay evaluation, but you lose the efficiency that comes from the memoization of record fields that you get with lazy evaluation.

P.S. The above would perhaps be more likely be written as

    rec { username = "nix-user"; homedir = "/home/" + username; }
but when specifying larger configurations composed of multiple nested records with wide cross-cutting inter-dependencies, "rec" doesn't really cut it as "rec" works best with extremely local inter-dependencies.


Recursion and merging semantics are also necessary. I use Nix and Jsonnet a lot, but Nix is much more expressive for complex structures, but the tooling being tightly coupled with the package manage make it impossible to adopt for common use cases.


There's lots of criticism about the Nix expression language, but it's way overblown IMO. The core language is small. It's essentially just JSON with functions, very similar to Jsonnet. But with Nix, people make too big of a deal out of implementation details like it being lazy, functional, etc. I personally like Nix as a configuration language because it's suitable for complex configurations despite it being small and simple.

There's a case to be made about improving the state of the Nixpkgs APIs though. Stabilizing the APIs and documenting it more thoroughly would dramatically change how people perceive the complexity of Nix. Experienced Nix users grep the Nixpkgs repo to learn about its APIs, but that's far from ideal for newcomers who won't even know where they should look for information.

Still, despite its reputation, Nix lowers the bar for software packaging. Package definitions tend to be more concise, customizable, and reusable than the competition. Nixpkgs contains many useful utilities for packaging and there's an abundance of real world examples contained in a single repo. I also frequently work with RPM, but Nix is much easier in comparison.


Point 4 is what lost me when I tried using NixOS for a machine I explicitly wanted as reproducible as possible. I needed to get a bunch of stuff working, then freeze/bundle/lock the useland as a documented working state I could rebuild later if anything went wrong.

When documented examples began steering me towards the still experimental nix flakes and using the sorts of with ever more flags required commands, I bailed and built it using salt. I want to like nix but the current state of things was a massive put off.


> When documented examples began steering me towards the still experimental nix flakes

There's a push to make them not be experimental anymore since so much of the community including myself is using them.


flakes are anything but experimental at this point, they've been out for many years and I pretty much use them for everything. you don't have to type those flags if you add them to the configuration file


A newcomer is going to treat anything documented as experimental as that, regardless of what the community thinks; if it's no longer experimental, it needs to be presented as such at all the normal entry points. (None of this is unique to Nix; imagine you needed to start packaging for Windows for some reason.)


This is exactly what I mean. It’s a big put off to see several year old experimental features and other signs that no one is making the decision to move things forward.

If I’m learning something as big as nix with all the facets it has, I don’t also have time to learn it’s entire community history. It’s not a good thing that I need to do that in order to make a decision about how much to trust experimental features of a tool when I’ve not even been using it more than a few hours from the moment of downloading the iso and installing it. It’s a put off to someone new coming to use nix for the first time.


> I also feel like folks for whom Haskell is comfortable may not realize just how absurd it feels to everyone else.

Yup. I can't even imagine what could I accomplish if I'd have something like a "python library" for nixpkgs. I still use nixos but gave up fiddling with nix for a while, after trying to wrap some stuff I am using/developing.*

* (Yes, I know it's easy. I just need flakes or correct amount of "lib: prev: { self"s. Sadly not for me.)


> they haven't been able to focus on the UX. I also feel like folks for whom Haskell is comfortable may not realize just how absurd it feels to everyone else.

The majority of communication I've seen from Nix maintainers over the last few months has been about exactly this.

The submitted post is about exactly that while Nix powerful, they want more people to be able to use it without having to pay a steep penalty to learn it.


It also doesn't have anything to do with Haskell or it looking somewhat like Haskell. I'm very comfortable in Haskell, and Nix looks awful to me. It's not that it's functional or it's declarative, it's that it's badly designed and not intuitive, making it hard to remember and therefore awkward to code in.


I think the only hard part is not Nix, but Nixpkgs’s not well-documented/not-stable APIs. It would be a problem in any language.


> Writing the packaging expressions should not be harder than writing the program, for the average developer.

Nah. The logical conclusion of this is that Nix should be some Java-like, Python-like, or whatever else qualifies as something that the average developer knows.

Nix needs to be declarative which rules out the average PL paradigm.

> […] , but can the next iteration provide an on-ramp which doesn't involve learning a new lang and paradigm? Guix folks sure think so.

Does the average developer know Scheme? I think you just played yourself.


Having just done some prototyping with Nix, this list is spot on in enumerating the pain points I hit.

I think most of the docs problems could be solved with the meta-strategy of “deeply understand all the things the Rust ecosystem got right with docs, discovery, and general pedagogical structure, and replicate that”. Actually have a canonical set of docs. Link liberally between packafes and stdlib. Figure out typing so you can get those hints and links in your IDE. Fix the package docs so they actually tell you what functions the packages export, instead of relying on the wiki for instructions. Etc.

Basically, pay a couple excellent docs people for a few years.

I found the language a bit weird but not terrible to learn. But if it’s possible to build some porcelain for common usecases that skips there need to grok the language, that could be quite transformative. I do wonder if the “dev env/packaging” use-case might be simpler than the “run and configure a whole OS” usecase.


> I think most of the docs problems could be solved with the meta-strategy of “deeply understand all the things the Rust ecosystem got right with docs, discovery, and general pedagogical structure, and replicate that”.

I think every project would love having the money to hire someone full time to work on the documentation and evangelism.


True, but the Nix project is actually investing in this area and did just hire a docs team.


I think you are missing the heart of the issue.

Learning to use a lazy functional programming language remains relatively easy - we use to teach the basis in Ocaml in three hours when I was a student - and Git shows people can tolerate awful CLI tool UX.

I think your point 3 is where most of people issues with Nix lie except I wouldn’t limit myself at a lack of practical enough documentation. Nix documentation pretty much just doesn’t exist period.

If you want to use Nix, you will have to go through the packages or the implementation to understand what’s happening. That’s not very welcoming especially when you consider that’s the other Linux distributions which tried to do things differently in the past and could be a source of users for Nix OS, first of them Gentoo, have absolutely stellar documentation.


Nix (the language) is just JSON with variables and arrow functions.

It's the easiest programming language in the world.

(Which is a problem when people start building huge projects like nixpkgs with it. It's not a language fit for programming-in-the-large, just like Javascript isn't.)


What nix needs more than anything is a centralized repository containing recipes for running all the different kinds of programs, as nic files. ie a dockerhub alternative.


nixpkgs?


Something a little more compact would be welcome. There is:

    nix flake show templates
But the examples in there are a bit out of date and don't necessarily reflect real world usage and are lacking in comments (e.g. still using defaultPackage, not using flake-utils, etc.).


> Everything should be made as simple as possible, but not simpler


Also: it should be able to deal with NVidia cards.


NixOS handles nvidia cards better than any linux distro i've ever uses including Debian, Ubuntu, and Archlinux.

I've only had to add a few lines to my configuration over 10 PCs and laptops.


Ok, that is good to know. But how well does it run on NVidia's Jetson line of systems?

https://developer.nvidia.com/embedded-computing


The Jetson environment provided by Nvidia runs Ubuntu and are setup for you to use docker.

I have trouble understanding your question. Why do you want to use Nix if you use a system which comes pre-configured to do something else?


Same reason why I removed Windows from my desktop computer and installed Linux :)


With Eelco as a cofounder of DetSys, I feel really excited about where we're going here. I think the world is in many ways primed and ready for Nix, as long as we can help Nix "meet them in the middle."

We're working on making Nix more accessible and producing good and usable, production-ready workflows so teams can just pick it up and go. I'd love to hear what y'all think, to help make sure we're going in the right direction.


My main gripe is documentation.

Look at this:

https://nixos.wiki/wiki/PostgreSQL

Now, the real "workflow" with nix is look at other setups, or just look at the code:

https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/s...

And look at the code is the only reliable way to see what exactly is supported.

My second gripe is the fact is hard to see what nix do. Today I hit this trouble:

https://www.reddit.com/r/NixOS/comments/x46w98/why_new_user_...

The thing is: Nixos not tell what is doing. `verbose` is too much noise.

What I wish now is something like:

    nix change nothing (if my changes somehow don't do anything, like in my issue) 

    nix setup users...
    nix setup postgresql


I'm looking at your second link, and it's not really that bad if you are familiar with nix. It's a standard pattern for a package with options. Mostly a long enumerated list, not any different from a long enumerated list of the postgres options themselves. The great thing about it is that it's a single DSL, instead of 10,000 DSLs, one for each package you install.

I do think command line options to do what you ask would be nice, but in order to keep the declarative reprudicibility you get with nix, those commands would need to operate on nix config rather than the system itself. They could update the config then update the system based on the config, so it could look like one step even though it is two behind the scenes.


Yeah, I think "look at the code" is not that bad if is embraced, like the docs of Rust. But half-docs in wiki, then in another links with options, then in code not work, IMHO.


> What I wish now is something like:

> nix change nothing (if my changes somehow don't do anything, like in my issue)

I miss this feedback greatly having used Ansible since 2013 and only this year diving into a single NixOS installation as my intro.


Yeah, dealing with user directory is a pain, and it is a pain because you can't make it completely functional (what about passwords). And it is the only place I found where nix doesn't work as you would expect it to in my 3 years of nixos use.


You can use agenix for secrets management. You can store all secrets encrypted with your SSH key in public config. The only non functional part is installing your SSH key, which is probably not a bad thing.


Are you using home-manager for that?


Hi, Graham, I have a few questions.

Does Eelco's departure from Tweag have any impact on Nickel as a possible replacement for nix?

There are lots of fragmented attempts at making nix easier/more immediately valuable/lower barrier to entry. Some of these are personal projects, some are businesses. Some of these just make nix easier, and some attempt to put something in between nix and the user. I'm thinking about things like Cachix, Flox, and divnix (formerly devos). How do you see your work interacting with these?

One of my barriers to adoption at work has been ensuring maintenance continuity. That's always going to be the case for tools that aren't ubiquitous, but I worry with nix that teams without my assistance will revert to other tools they know better, even if I've invested a good bit of time in teaching them enough to keep things going. Do you seek to address that?


My thinking is that this is long overdue!

I am working on a very large project where multiple solutions have been invented to solve problem which are all handled by Nix:

- Bake in the repository the clone of the dependencies.

- Compile the tool chain to create reproducible builds.

- Create "artifact" builds, to avoid large recompilation times for many developers.

- Use various tools to cache and distribute builds.

- Use a bleeding-edge version of some dependencies.

- Use Node, Java (Android), C++, Rust, …

If only Nix were to run natively on Windows, this would check all the boxes.


There was some work being done to make Nix work on Windows [0] - but it might have fizzled out. I too would really love Nix to work on Windows. I already use it on both macOS and Linux and it’s great.

[0] - https://github.com/nix-windows/nix


I wish this blog post had more meat to it. It was pretty minimal on information.

This is also due to the fact that I tend to value Graham's posts highly :)


Consider this post more of a "hello world." Substance is on the way :)


I might suggest making a typed superset of Nix, basically adding record types and function signatures. That in itself would be a huge boon for tooling and interacting with nixpkgs. nixpkgs describes a huge and varied meta-API for derivations in various build ecosystems, which makes it hard to use because you ultimately need to dive into source code.


We've heard this feedback pretty frequently. We have a few ideas of ways to add some types to Nix in place. I think even simple naive approaches would add a lot of safety and improve the usability of Nix.

However, one of the bigger problems here is the general API design of Nixpkgs. Most of the function interfaces use some named parameters and then use `...` to accept any additional parameters. All of those then get passed down to some other function, and on and on. This makes a mess! It is hard to tell what functions use what parameters, and none of them can restrict their interface.

We could see some significant DX here by creating smaller, more specific interfaces that actually restrict their inputs.


I think the biggest win with some progressive typing would be better error diagnostics. Right now the errors are horrifyingly bad in many cases, puking out massive recursive stack traces that have nothing to do with the real problem. Being able to pinpoint exactly where the square peg is going into a round hole would be amazing.


Some community members have looked into this quite a bit, and personally I have come to the conclusion that types don't really give that much benefit to Nix (and I say this as a Rust & Haskell developer).

Nix expressions are almost always[2] short-lived snippets that evaluate to a data structure, not long-running programs where the distinction between "static analysis time" and runtime is extremely relevant.

In fact, deriving the full set of potentially relevant type constraints is only possible at runtime due to how the import system works and the shape of most expressions. If you pick a single file from nixpkgs without context, you can't get much information from it at all (other than a lot of unknown types being passed to things that accept an unknown type) - it's only in the context of an evaluation of a graph node that uses that file that you can infer anything meaningful about it (and even then that information is only relevant in that context).

I think what people actually want is a way to derive more sensible documentation information from the code so that the various builders etc. could have checked documentation pages, as well as tooling for auto-completing members of attribute sets, names of (formals) function arguments and so on.

fwiw, there's an alternative implementation of the Nix language[0] that we (TVL[1]) are slowly open-sourcing at the moment, and we're aiming to design it in such a way that things like an LSP can be implemented on top of it, and to be able to dump out static information from a context that can help with documentation etc.

[0]: https://cs.tvl.fyi/depot/-/tree/tvix/eval

[1]: https://tvl.fyi/

[2]: Yes, I'm aware that Nix is turing-complete, but surely nobody would write a web framework[3] and HTML templater[4] in it.

[3]: https://cs.tvl.fyi/depot/-/blob/web/bubblegum/README.md

[4]: https://cl.tvl.fyi/c/depot/+/3410


> If you pick a single file from nixpkgs without context, you can't get much information from it at all (other than a lot of unknown types being passed to things that accept an unknown type) - it's only in the context of an evaluation of a graph node that uses that file that you can infer anything meaningful about it (and even then that information is only relevant in that context).

This is exactly where types would be the most beneficial, and what makes nix so hard to read. If a nix function could specify the types of its arguments, then I might have a chance of better understanding it.

Just because most nix functions currently don’t have many constraints on their inputs doesn’t mean that they couldn’t in the future.

> it's only in the context of an evaluation of a graph node that uses that file that you can infer anything meaningful about it (and even then that information is only relevant in that context)

That’s how nixpkgs works now. But we can make functions that work only in more limited contexts.


> tooling for auto-completing members of attribute sets

That's a great start, but it would be nice to know what the expected type of the member is! Often it's another attribute set, a list, or even a function, and it's impossible to tell what type is expected without looking at where the member is used.


I am originally opposed to blaming the Nix language, but now that I read your comment and thought a bit more about it, Clojure’s spec might be a much better tool than types. It would solve the readable error messages problem quite well and is flexible enough to accommodate any usage like parameter passing.


Something like Clojure's spec would IMO be a better fit, yeah. I've written an experiment with a runtime type-spec and checker for Nix: https://code.tvl.fyi/about/nix/yants/README.md

However, this has massive runtime cost (especially in hot code paths), so some sort of spec-like annotation system would be cool. We'll see how it shakes out over time ...

As for error messages, we might already be able to do big improvements in Tvix. Our bytecode has pretty exact tracking of the source spans that things come from all the way throughout, so we could trace things like a type error being raised somewhere to the last time that value was passed through as a function argument from user code etc.

This error message business will be a whole area of development of its own however, for now we're just making sure that as much of the relevant information is available as cheaply as possible.



I installed nix on my Mac but quickly backed out due to the complexity. I assumed the nix store would just be an ordinary directory with a tool for managing it, similar to brew. I discovered it creates a new Unix group, adds a separate APFS volume, installs a daemon. This was too invasive for a tool I was unsure if I even wanted to use, so I uninstalled it.

What is the reason for all this machinery? I went with the recommended multi-user install, should I have just used the single user mode instead?


> What is the reason for all this machinery?

> new Unix group

This is used for the daemon, so it doesn't run as root and expose your system to Nix build code.

> adds a separate APFS volume

I think this is required because of macOS security restrictions preventing direct modification of the root directory. The Nix store has to be housed in /nix because all references to runtime dependencies in the store are absolute paths in /nix, and that can't really change per system because it would break caching and reproducibility. The separate volume is added to /etc/synthetic.conf so it can live in /nix.

> installs a daemon

For multi-user installs, this allows non-privileged users to add build outputs to the Nix store, which ultimately allows these users to share build outputs. More info here: https://nixos.org/manual/nix/stable/installation/multi-user....


The Nix store has to be housed in /nix because all references to runtime dependencies in the store are absolute paths in /nix,

This is true, but in some sense Nix on Apple Silicon was a missed opportunity. It started as a blank slate (fresh binary cache) and it would've been a good opportunity to move the store to a writable path like /opt/nix. This would have solved the whole dance needed with volumes and synthetic.conf. I know that there are infrastructure issues (like Hydra using a single Nix store), but it would've made the macOS Nix story so much better.

It's a shame that only Apple can make firmlinks, because that would've been another possible solution (/nix could be a firmlink to the actual store location).

The same problem occurs on e.g. Fedora SilverBlue, because you can also not make arbitrary root directories. But at least on Linux, you don't want to throw away more than one decade of a x86_64 binary cache.


> It started as a blank slate (fresh binary cache) and it would've been a good opportunity to move the store to a writable path like /opt/nix.

That breaks caching cross-compiled outputs and x86_64 outputs, which do still run on Apple Silicon.


Which most users don’t really care about. I think it’s optimizing for the wrong use cases.

This is the main reason why I don’t use Nix on the Mac anymore, I just don’t want the pile of hacks that is necessary on my system. And I am a former nixpkgs contributor. Many people will just shrug and install Homebrew.


Actually the cache isn't that important, most binaries get rebuilt quite often due to their dependencies changing.

Although it is my hope that rfc 17 eventually makes it through: https://github.com/wmertens/rfcs/blob/master/rfcs/0017-inten...


Actually the cache isn't that important, most binaries get rebuilt quite often due to their dependencies changing.

Sure. But it's still nice that e.g. on Linux x86_64 you can use a nixpkgs commit from 2015 and get all stuff from the binary cache.


Single-user Nix installs are lighter weight, but also have a major compromise: the store is writable by the user. This means it is pretty easy for software you're using to accidentally change the store underneath Nix, and create unpredictable and unreproducible behavior.

I think the multi-user install is a better Nix experience, even if the install process is spookier.

Like is mentioned elsewhere, Docker does similar contortions to install itself. I wonder if it would have been more palatable if the Nix installer was less forward about what it is doing?


> I wonder if it would have been more palatable if the Nix installer was less forward about what it is doing?

I bet most wouldn't consider it spooky if the installer didn't print it out.

If docker started printing it, I bet lots of people would complain it seems similarly complex.


I recall the sqlite changed their tmpfiles on Windows from "sqlite" to "etilqs".


> What is the reason for all this machinery?

Nix wants to use /nix, but Apple wants macOS to be locked down and secure.

The compromise was putting /nix on a separate volume.

> I discovered it creates a new Unix group, ... installs a daemon

This is at about the same level as Docker, fwiw.


> The compromise was putting /nix on a separate directory.

Why does a separate directory need a whole other volume? These are just files, right? Why do they need to be on a separate volume?


Many binaries when compiled gets hardcoded paths compiled into them which are determined during configuration time. This means you cant use binary from a cache if the file hierarchy isnt identical. On linux this isnt a problem due to namespaces (feature originating from plan9).


OSX requires mounting root folders like /nix with synthetic.conf IIRC.


Right but the comment said they compromised on using a different directory in the end, so it could be under /opt like Homebrew, and not a root directory.


One of Nix's benefits is it aims to isolate where programs are installed; so you can have multiple versions of the same program available without conflict. (e.g. programs built with different feature flags, or multiple minor versions).

Nix achieves this by storing packages in a path as some hash of its inputs. -- This then allows either compiling or downloading a package, with confidence that it will behave the same way regardless.

But, since the files were under /nix, if you put them under /opt, then you wouldn't be able to make use of the compilation caches for /nix.


But, since the files were under /nix, if you put them under /opt, then you wouldn't be able to make use of the compilation caches for /nix.

Until a very short time ago there was no binary cache for Apple Silicon Macs.


There are for anything you run as x86_64 (using Rosetta). It's also important if you're cross-compiling anything.


Sure. But I think 90% of the Nix on Apple Silicon users does not compare about cross-compiling or running x86_64 binaries with Nix. So, everyone is worse off for supporting minor use cases.


Yes but they found a solution which works for 100% of users.

I don’t really understand the complain in this thread. Using an APFS volume just works and is entirely transparent to the user.

People could just learn instead of complaining they don’t understand.


I guess I don't understand what you meant by 'The compromise was putting /nix on a separate directory' then?


Oops. Wrong word. I meant "separate volume".

/nix could still be used if it's on a separate volume. Which still seems less nice than just having /nix on the same volume.


> it could be under /opt like Homebrew

You can install Homebrew other places, too, for that matter.

I'd only tried NixOS (bounced off, couldn't get X-Window working even following tutorials to the letter) not Nix on macOS. "Must be installed in a specific, root directory" is a hard no for me, when it comes to add-on package managers. That's one hell of an odor.


Homebrew sometimes pretends you can install it somewhere else. Like most things with homebrew, if you try, you will quickly realise this is not regularly tested.

> That's one hell of an odor.

That’s very much fine. They do that to be able to share the cache between system. There is nothing magical about the root folder by the way. It’s just Apple being annoying.


It's because Nix wants to install into /nix. Once upon a time doing "sudo mkdir /nix" wasn't a problem, but recent macOS releases have made that very hard.

Nix could switch to an alternate location on macOS (e.g. /opt/nix) but that has a lot of downsides for interoperability with other systems.


Yikes, I had no idea. What a tough pill to swallow all because the directory nix runs out of is hardcoded.


On Linux, users can put their Nix store in their home directory or other places and at run-time Nix remaps the directory using user namespaces. Unfortunately this isn't workable on macOS: the kernel doesn't support the features we need.

Using /nix and a separate group and daemon means the store can be read-only and be protected from modification in several ways. This is pretty helpful, as a lot of tools try very hard to write "next to" where they are installed -- corrupting the Nix store.

I sort of wonder if it would be more palatable if the Nix installer was a bit less in your face about what's going on? This would be similar to how Docker's works.


It would be less palatable when I found out. The group is fine. Why the daemon when other package managers use sudo is unclear. Even Homebrew moved to /opt.


Other package managers are okay with requiring sudo because they install stuff globally. Nix doesn't have that restriction, you can use it for local stuff, temporary shell environments, etc. So you need non-admin users to be able to use it too, and even admin users need to be able to use it without using sudo. For example, when entering a nix shell, you don't want the shell to run as root. Or when using direnv. Or just when using it as part of your build system.

Nix is a package manager, yes, but it's more than that, it's a generalized build system.


Nix isn't the only package manager for home directories or source packages. Working in a home directory doesn't require sudo or a daemon. sudo doesn't mean run everything as root.


You missed the crucial point:

> So you need non-admin users to be able to use it too

The build daemon and the user are used for privilege separation. The separation goes both ways. Users can't write directly to /nix/store and Nix can't write outside of /nix/store during build.

If anything, it's there to make things less invasive. It's nothing like the Docker daemon, which is a proxy for root.

Additionally, the daemon doesn't do anything unless users request that a package be built.


Other package managers have privilege separation without daemons. I don't need non admin users to run it. And I could configure sudo to let them if I did.


This is inherent to having binary packages. Binaries compiled to look for libraries in one location cannot just be copied to a system with a different layout. Nix also can't just use the OS conventions, since part of the point of Nix is that it does not use the global system state and only its own isolated, reproducible world. So then they had to make a decision and decided to just use /nix everywhere which worked fine for some years. And now MacOS has changed, choosing something else would invalidate all historical binary packages.


It's not hardcoded, but you end can't take advantage of the binary caches if you change the directory. A company could certainly create their own binary cache and distribute that to it's users.


Not entirely true, there are many ways in which you can use a custom location and still take advantage of the binary cache. You can do it with chroot, file system namespaces, bind mounts and so on. There's also a nice user friendly tool that does exactly this [1].

[1]: https://github.com/DavHau/nix-portable


> What a tough pill to swallow all because the directory nix runs out of is hardcoded.

Yikes? Well then... how would you solve remote binary caching with something like Nix on a platform such as OSX without userns remapping support?


It’s not actually hardcoded, but you would have to compile everything from scratch as the hash would depend on that as well, throwing out the whole binary cache.


Actually, there's quite a few hard-coded references to /nix in nixpkgs I believe. It's possible but a little more work than just changing the prefix.


There may be, but at least one BigCorp deploys Nix to a different prefix.


The install process was simpler before recent OSX updates.

> I discovered it creates a new Unix group, adds a separate APFS volume, installs a daemon. This was too invasive for a tool I was unsure if I even wanted to use, so I uninstalled it.

Since you'd already installed it, wouldn't trying it in some capacity before uninstalling it have made sense?

> What is the reason for all this machinery

Enforcing reproducibility basically.

I don't know all but...

The daemon and APFS volume otherwise readonly /nix that only the daemon can write to can't be created.

The group is probably for the daemon to be able to write to /nix.

The path /nix is important because the remote binary cache paths will miss otherwise and you'll compile everything from source.

> I went with the recommended multi-user install, should I have just used the single user mode instead?

I'm guessing it would work in the way you want, but I always opt for the daemon.


> Since you'd already installed it, wouldn't trying it in some capacity before uninstalling it have made sense?

Fair observation. I installed nix as a prerequisite for DevBox, discussed here https://news.ycombinator.com/item?id=32600821 . I thought DevBox sounded really cool (and still do!), but the Quickstart took frustratingly long, and ended up not working. Faced with the prospect of debugging it, I opted to cut my losses and uninstall it instead.

That said I'm very much open to trying nix again in the future. Also I want to acknowledge how much effort went into getting /nix to work on the Mac; it appears that was a heavy lift indeed.


The meta answer to the question is this info belongs at the non-existent doc - https://nixos.org/manual/nix/stable/installation/macos.html, but no-one has put it there yet.


It's because builds are completely reproducible, and therefore they have to be built in the same place (in `/nix`), and you can't create directories off of `/` on modern Mac setups.


APFS volume is required to have a read-only nix-store at /nix. The daemon is only required in multi-user installation, and you can opt out of it (by using a single-user install). New groups are for letting people submit jobs to builders in multi-user mode.


Note that the multi-user installation is these days the only suggested approach by the Nix team for new MacOS installations - the old instructions for single user installation on MacOS have been removed from their site.


As someone who has been using NixOS for a couple of years now, I really want to say how appreciative I am of everybody for making noticeable improvements to the system on a somewhat regular basis. The nix command keeps on adding great new features like flake templates and bundling as well as just being more user friendly (error messages, actionable hints, etc.) Additionally, tools like nix-ld [1] make nix more usable than ever with software from external sources. Things just keep on getting better for NixOS users!

Despite the reputation, I feel that NixOS or some derivative of it has the power to become the best distribution for non-technical users in the long run. What NixOS has done is effectively built an interface to every component of a modern Linux system, all that needs to be built is a user application to take advantage of it. Of course, there still needs be some improvements in Nix itself for it to blossom into its final form, but I really see a path to greatness here.

I have often thought about creating a simple unified Win2K-esq or BeOS-like X11 WM/DE specifically for NixOS but unfortunately I lack the time/motivation.

[1] https://github.com/Mic92/nix-ld


Yes, I could imagine a very locked down opinionated distro built on top of Nix that completely abstracts away all of it and makes it work somewhat like Ubuntu with some prepared nix templates/metaprogramming to automatically generate config. Over time more of the Nix functionality could be exposed through UI or more friendly cli tools as best practices and nix itself evolve, allowing the distro to remain friendly to non-technical users while opening up it's true powers to them.

It wouldn't be able to handle arbitrary manual changes to config at the beginning, so either there would always be a protected generated config region of the whole config and the distro would remain relatively standardized, or an alternative evolution would be a low-code user friendly UI that could map to any possible nix config.


Make the documentation up to the modern standards. If I have to open a single random blog or Github repository to find out what to do and piece it together like a puzzle - it is impossible to adapt at organizational scale.

It is also complex enough to require a modern Language Server.


This has been my experience also. I finished the Nix Pills, and got some personal Python/Haskell projects built using nixpkgs by following the nixpkgs language-specific documentation, but anything off the beaten path is going to involve lots of blog posts and clicking through source code. For example in the Haskell world there are so many blog posts that go in different directions from the nixpkgs docs, using flakes, haskell.nix, etc.

We adopted it in an organization of ~100 engineers, and the only way it’s been possible is having a full-time Nix team writing custom Nix functions specific for our environment and projects. That team also does “Nix help desk” work for one-off questions.

Once it works, it does a great job of hermetic builds, easy Docker images, easy to add cross-repo dependencies mixing C++/Python etc. But there are too many rough edges I can’t recommend it in the general case over Dockerfiles, Bazel or language-specific tooling, Cmake, etc. Pick something simpler, ideally whatever is popular for your language.


The name Nix is used for the language, package system, and sometimes the whole ecosystem. There is a recent post that tells the difference https://www.haskellforall.com/2022/08/stop-calling-everythin...

Knowing functional programming helps but Nix still is a hard language to learn. I have a lot of Haskell experience but the Nix language was very confusing for me. At first I couldn't even tell a variable name from a keyword (it is a convention to call variables "self" and "super" in some contexts). The Nix language has been created for research in build systems and has evolved over a long time, so it has accumulated quite a lot of idiosyncrasies and cruft.

I hope that a simpler typed language like https://github.com/tweag/nickel replaces it.


I believe blaming the language is misguided. The problem is that nixpkgs’ constructs are not well documented, or not yet stable. It would be a problem in any language (you can’t expect someone to feel familiar in a new codebase, and nixpkgs is essentially that).


Yeah, the conventions of nixpkgs are a much bigger obstacle than the language syntax. The language lacks static typing. Also, it is a common convention to create attribute sets crafted for logic that has indirect hard-to-follow links to module where the set is defined. This makes it hard to develop tooling like linters or LSP implementations.


Since Eelco (often treated as Nix's BDFL) and Graham (also very prominent in the Nix community) are involved hopefully this group will actually be able to make traction on some of these issues. This is not even close to the first of these posts saying "we're going to help make Nix more user friendly!", but maybe this group has a chance of making changes to the Nix ecosystem since they can just approve it themselves.

I've had concerns about Nix's governance in the past but it sounds like they may be going in the right direction, so I'm excited to see what they're planning.


"But there’s a catch: to make that happen you need to write some Nix, use Nix tools, and probably consult several documentation sources."

You can use bob[1] if you want a build tool which uses Nix to install dependencies in an easy manner: just list the package names for a task and then they will be installed.

I'm looking forward for all the changes in Nix ecosystem and it's a good sign the fact that they also started working on an initiative to improve Nix documentation which was spread all over the places.

[1] https://bob.build/


Bob is very cool and I think that there's room for lots of tools like this that build _on_ Nix but abstract away some of the trickier parts.

Suggestion: make Bob installable with Nix!


Thanks for the suggestion! It makes sense to have it on Nixpkgs collection since anyway you need to install Nix alongside bob

Already created an issue :)

https://github.com/benchkram/bob/issues/193


Is Bob complementary to Devbox recently featured in a Show HN here and would you be willing to explain how you might use both together if so?


I think they only way in which they are similar is the way a user would add dependencies for a project. If with devbox a user populates devbox.json with package names, with bob you add dependencies in list inside bob.yaml. You can find out more in a short guide[0] I wrote.

But bob intends to be more than a dependency manager, it wants to be fully-fledged build system similar to Dagger, Bazel, Earthly etc with features like:

* build pipelines[1]. You can define and manage your build pipelines in a familiar YAML format. You also get caching for your targets and so on.

* isolated environment. We completely clear the environment for a task. I see with devbox that if I do a `printenv` or an `echo PATH` while I'm still in an active devbox shell, I see variables from my local system which can affect reproducibility.

* workspaces: manage multi-repo setups in an easy manner

[0] https://bob.build/docs/usage/package-management

[1] https://bob.build/docs/usage/build


How does Bob handle network requirements (downloading dependencies during the build)? I think that is one of the biggest pain points for nix packaing. Many projects have build instructions that depend on downloading stuff in a complex manner that cannot be easily abstracted/shaped to what Nix expects.


I wrote a short guide[1] how bob uses Nix, but essentially we still use `nix-build` under the hood to build a list of packages you give under dependencies

> Many projects have build instructions that depend on downloading stuff in a complex manner that cannot be easily abstracted/shaped to what Nix expects.

Are you referring here to packages which are not available in Nixpkgs?

[1] https://bob.build/docs/usage/package-management


The point is: a couple of the tough points in packaging something with Nix are "how to come up with the Nix expression", but also how to get the software you're packaging to build with the constraints Nix has (no fetching external dependencies, shared libraries explicitly declared in non-standard locations, etc.).

A bespoke YAML format is arguably not simpler than just "here's the equivalent Nix expression". But, sure, can be more accessible for some people.

But, providing a nice facade over the Nix language doesn't help with cases of where Nix can be difficult to work with.


> But, providing a nice facade over the Nix language doesn't help with cases of where Nix can be difficult to work with.

This is true. If you reach the point where you need to write your own Nix expression you currently can become stuck.

But I think these are problems that are waiting to be solved. There are folks are already working on a new language tutorial at nix.dev for example and other tools that can abstract some of the problems we have now can be built. I believe we are still in an early stage when it come to reproducibility and we just have to keep working at it.


> Many projects have build instructions that depend on downloading stuff in a complex manner that cannot be easily abstracted/shaped to what Nix expects.

The basic way to fix this with nix is:

1) download the thing with a fixed ouput derivation: a derivation who's source has the URL and checksum of the artifact

2) copy the artifact from the nix store where said build tool stores it


In nix expressions, sure; but bob uses YAML, and uses that to drive nix.


Yes, basically you can list your packages in the yaml file and have them installed.

But you can also add .nix expression files in the same bob.yaml that can be used to build your packages when a package is not found on Nixpkgs.


Hot take as a NixOS user that uses Nix for work: "all" we need is a much better, sound, statically typed language to build better abstractions with.

The only hard requirements I can think of are algebraic data types with exhaustive pattern matching to go with, row polymorphism, purity and good inline documentation support.

I don't know if a good enough hostable language exists or if it should be a new version of Nixlang, but almost every single annoying problem that makes me go "Nix is getting in my way" can be traced back to the lack of a good, powerful type system leading to a house of cards situation whether it comes from nixpkgs or entirely of my own making.


The problem is that we need not just a language like that, but a version of Nixpkgs-the-codebase designed for that language with corresponding sensibilities. And that's a much bigger obstacle than coming up with a language!


I think this is true for Nix in the deployment/ops space, where debugging a broken build can be very frustrating. Language improvements are going to be less useful for app developers, the Flake learning curve is not going to get better with a type system.

Perhaps something like heroku buildpaks (https://github.com/railwayapp/nixpacks ?) would help devs get on the Nix train.


Can Dhall be that language?


How does Dhall compare to Nickel?


Funny enough I introduced Nix to our company which was acquired by Google -- so Google has software leveraging Nix. (I got Google to in fact sponsor Nix which was nice too. A small amount but it was meaningful to me).


I hope this goes the way of RedHat/SUSE. i.e. The corporate side of a FOSS project so that contracts can be signed for appropriate purposes, etc.


I have more than 20 years of Unix experience and was getting into DevOps and CI/CD in the recent years with GitHub Actions, Azure DevOps, AWS etc. I literally discovered Nix yesterday via Andrew Kelley’s Zig demo on YouTube. I had the similar feeling as I first learned Unix tool chains two decades ago.

I believe live demo and CI/CD pipeline might be some of the best use cases to get people interested in Determinate Systems. Don’t get distracted by language features and syntactic sugars. If it works, people will learn.


I've worked with large scale monorepos in FAANG - as a rule, they all suck. Eventually you always end up in this place where you have to understand, and be prepared to debug the entire universe just to build hello world.


Totally agree. For complicated system, this essential complexity won't go away no matter how you make the UX "friendly". But good tooling will help users grow to understand the challenge of the problems as well as to appreciate the power of the solution. For people who are really interested in the solution, they will eventually dive deeper: https://edolstra.github.io/pubs/phd-thesis.pdf


For the first time in about 8 years an Arch update rendered my OS unbootable last week. I'd had graphics issues once or twice but this time it just hung in systemd somewhere. Still don't know what happened. Rolled all packages back about a month and that worked, but I'll need to update eventually.

I'm considering Nix but don't like the custom language. I wish Guix were. more popular and less extreme on the freedom side. Chances are I'll just do a fresh Arch install or maybe try Debian rolling.

These days I like to install almost all software and libraries by hand in my home directory. All I really want is a solid system that basically boots fresh then loads my home directory. Maybe I should do a net boot of some sort?

Any other suggestions?


Just a heads up, the issue was likely grub, grub had a breaking change for certain EFI devices. You can check arch news etc for more info. I had to chroot and fix my laptop the other day cause it was stuck boot looping after I upgraded.


you can try something like Kinoite or Silverblue from fedora.

you can also manage tools with asdf if it makes it easier.

I personally run Fedora + Flatpaks + asdf and it covers everything, and is stable, up to date, and has a new release every 6 months. it feels midway between a rolling release and. apoint release


I mean, I just add packages to the list and some configuration options. If you only need it as a user it's fairly simple.


I just started to experiment with EndeavourOS, a prepackaged version of Arch. Looks nice so far.

https://endeavouros.com/about-us/


Eh, I'd just use Silverblue. You get the benefits of immutability, without having to learn a new language and weird tools to manage your workstation. I ain't got time for any of that.


I agree with the preamble and the "our mission" stuff. I wish them success at bringing the benefits of nix to more users, but without it being limited to enthusiasts, and people who can afford to deal with rough edges.

The "external dependencies" thing shows how Nix can be a tough sell to those who don't click with it.

I'd say that the other solutions are:

- a README which has a bunch of "apt-get install" commands you can copy-paste from,

- a "setup.sh" script that installs things,

- a container or VM image with the toolchain setup,

- or a tool like asdf.

IMO, Nix is a much nicer solution than these other options, and has the benefits, avoids the downsides of these (but an entirely different set of downsides).


How about Bazel, doesn't it solve the same issues?


Both Bazel and Nix have some powerful features to them. (Though, oddly, they don't quite mix).

I think they both also have downsides where, if your software does things in ways they don't like, it's difficult to use them. -- And, your team is going to have a tough time using the tool without sufficient expertise.

As I understand it, Bazel's emphasis is more about compilation, where the big win is building projects where you've got a team big enough (& a project complex enough) that dealing with Bazel saves more time than it costs.

Whereas with Nix.. I think one good use case is "declare & lock the dependencies for this toy project", which then allows picking the project up without having to update your code following updates to the system package manager.


I love nix, I've been using it for the last 2 years, I have a very stable setup from these 2 years of effort [0], and I just can't recommend Nix for Linux beginners, why?

It's not because of the nix language, It's not because of the CLI, it's because everything is scattered, you have to consult many places to find out how to do things with Nix, here is an example:

Usually, when I need a new complex program, like Steam, I first check the system-wide configuration [1], the wiki [2] and the package list [3], if I just want it on my user, I need to check if Home Manager has an option [4], if it doesn't, I can try using the "home.packages" option. Now, if I need to override something on the package, I need to remember how to do it with [5] [6] (while checking the source code for the package in parallel to find the options).

And then sometimes, on very rare occasions, I need to fine tune something with the nix language, so I need to check the builtins/lib docs [7], but some builtins are not there, so I need to either use nix-doc [8] or find the docs inside the code-bases [9] [10] (they are split between both repos)

For me, this is one of the main pain points of using Nix / NixOS that needs to be solved.

[0] - https://github.com/shiryel/nixos-dotfiles

[1] - https://search.nixos.org/options

[2] - https://nixos.wiki/wiki/Steam

[3] - https://search.nixos.org/packages

[4] - https://mipmip.github.io/home-manager-option-search/

[5] - https://nixos.org/manual/nixos/stable/#sec-customising-packa...

[6] - https://nixos.org/guides/nix-pills/nixpkgs-overriding-packag...

[7] - https://teu5us.github.io/nix-lib.html

[8] - https://github.com/lf-/nix-doc

[9] - https://github.com/NixOS/nix

[10] - https://github.com/NixOS/nixpkgs


The external dependency problem is somewhat solved in npm (at least, as far as many users are concerned) by writing a module that downloads the appropriate binary.

For example, esbuild is written in Go, which is compiled to a different binary for each system. The NPM for esbuild has 21 optional dependencies, one for each binary that it makes available. A post-install script [1] chooses which dependency to install.

It seems like a lot of work for the maintainer? But most users don't need to care.

It probably helps that the Go SDK builds static binaries.

[1] https://github.com/evanw/esbuild/blob/master/lib/npm/node-in...


> The external dependency problem is somewhat solved in npm (at least, as far as many users are concerned) by writing a module that downloads the appropriate binary.

This complicates working on node projects with nix in my experience.

Is there a way to tell npm "no, just use my binary"?

Best i've found is letting npm download it and then overwriting the npm binary with the nix one in a postinstall script.

Why you ask? Npm doesn't always have an aarch64 or arm32 binary.

Nix being a source based package manager typically does support it or at least makes it possible by overriding build/configure options.


If you know how to build the binary from scratch, could you teach the npm package's continuous build to do it? Sending a pull request fixing the npm package would fix it for everyone, not just nix users.

Also, I wonder if nix would be a suitable tool for generating npm packages that contain binaries? Maybe this is a way for people to benefit from nix without actually using it.


> If you know how to build the binary from scratch, could you teach the npm package's continuous build to do it? Sending a pull request fixing the npm package would fix it for everyone, not just nix users.

The "I have a binary already, just use this one" is in response to the problems NixOS users face dealing with helpful software like this.

NixOS doesn't provide shared libraries in the standard locations (for good reasons). Binaries that tools like NPM download will link against shared libraries in the standard location.

It's not really a question of "providing the binaries would help", because it's about assumptions of where the shared libraries are.


Beside being invasive on MacOS, as said by @ridiculous_fish, it took me more than 3 hours (and it didn't yet finish, I just quit all) to use QMK. I just cloned the Github repository of QMK and did `nix-shell` as they provide shell.nix file.

1. Does every nix-shell require building the whole world from ground-up? Seems impractical to me. 2. What is the right approach?

This is not to bash Nixpkgs, because I installed NixOS and it took me 10min to install a whole OS with Sway, Neovim and some other tools. I guess I'm doing something wrong. But, on MacOS, nixpkgs was not a pleasant experience at all.


> 1. Does every nix-shell require building the whole world from ground-up? Seems impractical to me.

It's the same as homebrew:

If there's a cached binary already, it will download the cached binary. If not, it will fetch the source and build it.

The QMK nix shell needs different GCCs for different targets. For the QMK shell.nix specifically, you can disable the targets you don't need (e.g. if you don't use a teensy board).


That makes sense. Is there a `nix-shell` option that outputs which packages will be compiled from source and which will get a precompiled binary? Something like `nix-shell --dry-run`?

I have to learn more about the nix language, but looking at the shell.nix seems that I have comment/remove lines from 25 to 30 according to my board.


The easiest is to change the default:

  { avr ? true, arm ? true, teensy ? true }:
Change `true` to `false` as needed, only keep the ones you need for you keyboard(s).


Thank you, I will try again with your suggestions!


> Something like `nix-shell --dry-run`?

Exactly that? Lol :)


Really? It is not written in the official nix-shell manual

https://nixos.org/manual/nix/stable/command-ref/nix-shell.ht...

I still will try when I get to my macbook. Thank you!


Just tried it, it does work. Not in the `--help` though.


This subthread is exactly why NixOS is getting much traction outside of enthusiasts and people with a lot of patience.


Missing a 'not', I think?

I think the idea that Nix is difficult to get started with, and undocumented, isn't a surprise to the community. -- These were some of the most repeated points in the community survey. And the topic of the submitted post.

That said, I do think Nix has seen a lot of growth over the last year, and I see fewer "nix is so easy" comments.


I've been on the sidelines for a while, as I keep hearing that NixOS devs are focusing on the documentation for years now... but to be honest I'm not seeing it yet. Documentation was the thing that turned me off it 4 years ago, and I'm a bit sad to see it's still a big problem even if I keep hearing it's being actively worked on.


I misread that as `nix-build --dry-run`, sorry!

You can use nix-build on a shell.nix though i'm pretty sure.


> 1. Does every nix-shell require building the whole world from ground-up? Seems impractical to me.

Agreed! Most repos with a shell.nix and heavy dependencies use their own binary cache with the service cachix.org (free for public repos).

> But, on MacOS, nixpkgs was not a pleasant experience at all.

Were there reasons besides the QMK nix shell building the world?


> Agreed! Most repos with a shell.nix and heavy dependencies use their own binary cache with the service cachix.org (free for public repos).

Created an issue:

https://github.com/qmk/qmk_firmware/issues/18248


Nix is source-first, but with binary cache. Many things have to be installed again on non-NixOS because nix is hermetic - it can't use library X from underlying system, it has to come from nix store.

MacOS has issues with binary cache because of apple's requirement on where macOS can be run.


Just let me a pin a single package to a version number, for the love of God.


You can do that with something like this:

    pkgs.yourpackage.overrideAttrs { prev: {
       src = fetchurl {
          url = "...";
          sha256 = "...";
       };
    }}
Bit annoying that you have to specific the full URL instead of just the 'version' number, but it'll work.

With flakes you can just add a 'ref' or 'rev' to the URL:

    nix run "github:juser/software?ref=v0.4.1"
If you want to override a package dependency across 'override'.


What does that version mean? What compile flags was it compiled with, which compiler, etc?


Is it Nix or NixOS?


Both exist. Nix is a functional language for describing how software is built. Nix is also a package manager of sorts that installs and manages software built with its language. NixOS is an operating system that is described entirely by a nix language configuration file that builds a bootable linux-based OS using nix language packages.


Nix is a package manager & programming language that you can install on any Linux or MacOS to manage software.

NixOS is a bootable OS build on top of Nix.

Nixpkgs is the default package collection used by both Nix and NixOS.


Disclaimer: I'm working on a build system that will eventually do what Nix does but make it much easier to use.

If they do manage to create a system where Nix is hidden, and end users never have to directly touch it, I think this could work and make my work never see the light of day.

But I have my doubts that they will be able to do that, and it boils down to one simple reason: declarative is not powerful enough.

Don't get me wrong, for 90%, possibly more, of use cases, it's enough. And it is preferable to keep things declarative as long as possible, so much so that my build system will have a way of restricting itself to purely declarative code when possible (and will error otherwise).

However, when more power is necessary, it is required; it is not possible, by the definition of Turing-completeness and declarative, to do the same thing with a declarative language that can be done with a Turing-complete language. So when that power is missing, the only option is to work around it.

This is why Nix is really just a front for bash. For all of its faults, at least bash is Turing-complete.

But this fact means that there will be things that Nix will fail to do on its own. I suspect this means that the abstraction around Nix will always be leaky. Maybe it won't be unacceptably leaky, but I'm not very hopeful.

I think Nix would be better served by doing the following:

1. Rewriting the language. This would require an auto-transformer to the new language in order to not throw away the entirety of nixpkgs, but transforming a declarative language to a Turing-complete one is easy.

2. Spend gobs of time on user experience. Make the usual commands short, easy to remember, and easy to use with few options. Make usual things easy and powerful things safe. For example, to make a powerful thing safe, make sure that screwing it up will not screw up their system.

3. But not only should they be safe, they should feel safe, giving users every opportunity to back out without consequences. This is where git goes wrong: it does not feel safe, so users are scared of it. [0] I believe Nix is the same way, even though it is safer than git.

4. Spend gobs of time on documentation. Use Fred Rogers' list of rules for talking with children [1] in each piece of documentation. This will make it easier to avoid the trap of not explaining something the user needs because the writer forgot that they didn't know it. In essence, the documentation should treat new users as Fred Rogers treated children: ignorant, but capable.

5. The documentation also needs to have a different focus. Instead of focusing on how great Nix is, it should focus on helping users get stuff done. Nix enthusiasts should be able to say, "Oh, you want to set up a development environment for your project? Great, go to such-and-such tutorial. It will tell you exactly how to do that even if you don't have Nix installed." This should be done for as many use cases as people have, including less common ones. Some examples: using Nix as a build system, using Nix to install multiple versions of glibc and how to switch between them, using Nix to set up systemd, using Nix to replace a Docker container in production, using Nix to distribute builds, etc.

[0]: https://xkcd.com/1597/

[1]: https://www.mentalfloss.com/article/547536/mr-rogers-rules-f...


> However, when more power is necessary, it is required; it is not possible, by the definition of Turing-completeness and declarative, to do the same thing with a declarative language that can be done with a Turing-complete language. So when that power is missing, the only option is to work around it.

I fall heavily into the use less powerful languages and "liberties constrain, constraints liberate" camps.

I'd be very interested in a real world example of this, though I know that class of examples is hard to come by.


> I fall heavily into the use less powerful languages "liberties constrain, constraints liberate" camp.

For what it's worth, so do I. That's why my build system will allow users to set those constraints. It's a great idea.

> I'd be very interested in a real world example of this, though I know that class of examples is hard to come by.

There are many, but I'll say one: anything that requires iteration until a fixed point is reached. A concrete example of this is building documentation in LaTeX.

Or it may even be simpler than that. Maybe your build requires some sort of special code should two arguments have specific values. In other words, edge cases often require Turing-compete power to resolve.


Ok then, would you consider markup languages declarative, XML... XSLT?


Yes. They are declarative.


Would you consider lisp to be (possibly) declarative?


Not all lisps are declarative, but a lisp certainly can be declarative.


No, it is Turing-complete.

I was going off of the article and Nix documentation that says Nix is declarative. But if it actually is not, then there shouldn't be a problem with power.

However, Nix proponents shouldn't claim it is declarative either.


Laziness, and purely functional go a long way toward declarative.


Declarative has a few definitions with respect to computer languages, and the Nix community seems to be using one and ignoring others.

When I think of declarative, I think of a language that has less power than a Turing-complete language. Think Make without bash or any shell. Or HTML without CSS.

In other words, the Nix community seems to use "declarative" as meaning "without side effects" where some programmers, like myself, hear "declarative" and think of something like HTML without CSS or JSON.


> This is why Nix is really just a front for bash. For all of its faults, at least bash is Turing-complete.

Nix is Turing-complete. This was even sometimes considered a problem because it makes the language too powerful:

https://nixos.org/~eelco/talks/guix-feb-2018.pdf


This is news to me, I admit. But if this is true, then people shouldn't say Nix is declarative.


This shouldn't really come as a surprise, any reasonably powerful language will end up Turing complete sooner or later, it's pretty difficult to avoid. Nix specifically doesn't even hide it, it has plain old function and recursion as primitives.

What makes it declarative is the lack of side effects and laziness.


What I'm trying to explain is that using the term "declarative" is confusing to a lot of developers who who not think of functional programming as declarative, but rather think of something like JSON.


I don't see it as confusing, it's a pretty accurate description. The Nix language is for most part a JSON-look-alike with a couple of additional features. You don't really program in it, you declare/return a data structure. The only programming you do in Nix is to make it a little easier to build that data structure. It's kind of like the old days when people would parse JSON with eval(), which would also grand you features that plain JSON wouldn't have.

Even NixOS itself doesn't do all that much actual programming in Nix, most of the code that does stuff is written in C++. The Nix language itself doesn't let you escape out of its sandbox.


Although it's debatable, it's very common for functional languages to be categorized as declarative.


Perhaps, but that categorization is going to confuse a lot of developers. It sure confused me until I said something wrong, and all of you Nix proponents descended on me.

This confusion is not good for Nix.


Nix is turing complete (it can implement the Lambda Calculus), even conveniently so.


Then people shouldn't say it is, such as in the article under discussion.


What aspect of lambda calculus is not declarative?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: