fasterthanlime

Learning Nix from the bottom up

Thanks to my sponsors: C J Silverio, Sam Leonard, Integer 32, LLC, SeniorMars, Zoran Zaric, Christoph Grabo, ofrighil, traxys, Matt Heise, Mikko Leppänen, Josh Triplett, Michael, Lennart Oldenburg, Xavier Groleau, Richard Pringle, James Leitch, Henrik Tudborg, Matěj Volf, Mikkel Rasmussen, Eugene Bulkin and 266 more C J Silverio, Sam Leonard, Integer 32, LLC, SeniorMars, Zoran Zaric, Christoph Grabo, ofrighil, traxys, Matt Heise, Mikko Leppänen, Josh Triplett, Michael, Lennart Oldenburg, Xavier Groleau, Richard Pringle, James Leitch, Henrik Tudborg, Matěj Volf, Mikkel Rasmussen, Eugene Bulkin, Tyler Schmidtke, Christian Bourjau, Lena Schönburg, notryanb, Andrew Henshaw, Geoffroy Couprie, Marc-Andre Giroux, Berkus Decker, Joseph Montanaro, Michael Alyn Miller, Yann Schwartz, Corey Alexander, Joshua Roesslein, Sean Bryant, Taneli Kaivola, Michael Mrozek, qrpth, Kyle Lacy, clement, Guy Waldman, Christopher Valerio, Ross Williams, Kai Kaufman, David Barsky, Andronik, ZacJW, Herman J. Radtke III, Ronen Cohen, Beth Rennie, Elnath, Borys Minaiev, Samit Basu, Daniel Papp, Alex Krantz, Timothée Gerber, Dave Minter, Justin Smith, Rufus Cable, Tyler Bloom, Tobias Bahls, Ahmad Alhashemi, Dominik Wagner, Duane Sibilly, Jack Duvall, Philipp Gniewosz, René Ribaud, Mattia Valzelli, Sylvie Nightshade, Tabitha, Ryan, James Brown, playest, Ronen Ulanovsky, Raphaël Thériault, Marcus Brito, Kamran Khan, Blake Johnson, Isak Sunde Singh, Ivo Murrell, Senyo Simpson, Michał Zalewski, Dom, AdrianEddy, Aleksandre Khokhiashvili, Aljaz Erzen, Dimitri Merejkowsky, Jack Maguire, Ives van Hoorne, Marcus Griep, Adam Lassek, Geoffrey Thomas, Elendol, Hadrien G., ShikChen, Chirag Jain, John VanEnk, Ian McLinden, Beat Scherrer, Raine Godmaire, L0r3m1p5um, Braidon Whatley, Yufan Lou, Xirvik Servers, Jim, Marcus Griep, Horváth-Lázár Péter, Tanner Muro, Jon Gjengset, Romain Kelifa, Olivia Crain, Shane Lillie, johbii, Max Bruckner, Chris Sims, Luke Yue, Tiziano Santoro, Gorazd Brumen, Torben Clasen, Noel, Walther, Johnathan Pagnutti, Marie Janssen, Michał Bartoszkiewicz, teor, Max von Forell, Cass, jatescher, Manuel Hutter, Andrew Neth, Sarah Berrettini, Jake Demarest-Mays, Cole Tobin, Mateusz Wykurz, xales, pinkhatbeard, Brooke Tilley, Richard Stephens, Hamilton Chapman, Matt Campbell, Pete Bevin, e9zaktw1, Laine Taffin Altman, hgranthorner, Jan De Landtsheer, old.woman.josiah, Nyefan, Egor Ternovoi, Chris Emery, compwhizii, milan, Paul Horn, Benjamin Röjder Delnavaz, Jelle Besseling, Andy F, prairiewolf, Sindre Johansen, Marco Carmosino, ACRL, Niels Abildgaard, Dylan Anthony, Garret Kelly, Diego Roig, Thor Kamphefner, Evan Relf, Geoff Cant, Yuriy Taraday, Vladimir, Alan O'Donnell, David White, Chris Thackrey, Mark Tomlin, Luiz Ferraz, Marcin Kołodziej, Paige Ruten, Julian Schmid, WeblWabl, Marky Mark, Damir Vandic, Michal Hošna, Victor Song, Wyatt Herkamp, Max Heaton, anichno, Nicolas Riebesel, Paul Marques Mota, Chris, Lyssieth, Thehbadger, Adrián Garnier Artiñano, Lucille Blumire, Jan-Stefan Janetzky, Dirkjan Ochtman, Mark, Guillaume E, Seth, Jean-David Gadina, Aalekh Patel, Mike English, Astrid, Guilherme Neubaner, Antoine Rouaze, Daniel Wagner-Hall, Pete LeVasseur, Wojciech Smołka, avborhanian, Philipp Hatt, genny, G, Steven Pham, callym, David E Disch, Ula, Justy, jer, Mark Old, Daniel Strittmatter, Arjen Laarhoven, Valentin Mariette, Toon Willems, Antoine PESTEL-ROPARS, David Cornu, Brandon Piña, John Horowitz, Scott Steele, Chris Biscardi, Josiah Bull, Òscar Pérez, Zaki, Radu Matei, Neil Blakey-Milner, Scott Sanderson, Nicholas Orta, Simon Menke, Antoine Boegli, villem, Jörn Huxhorn, Stephan Buys, Aiden Scandella, Mario Fleischhacker, Malik Bougacha, Nicholas, Jonathan Adams, Boris Dolgov, Luke Konopka, Cole Kurkowski, Jesse Luehrs, Yves, Mason Ginter, Urs Metz, James Rhodes, Zeeger Lubsen, Chris Walker, Adam Gutglick, Makoto Nakashima, belzael, Menno Finlay-Smits, Ben Mitchell, Daniel Silverstone, Matt Jadczak, budrick, Matt Jackson, Justin Ossevoort, Colin VanDervoort, Sawyer Knoblich, Gioele Pannetto, Ben Wishovich, bbutkovic, Enrico Zschemisch, Andy Gocke, Alex Rudy, Zachary Thomas, medzernik, Elijah Voigt, Guillaume Demonet, Matthew T, std__mpa, Mathias Brossard, you got maiL, Santiago Lema, Romet Tagobert, Luuk, Olly Swanson, Peter Shih, Marty Penner, Bob Ippolito, DaVince

Remember the snapshot we made allll the way back in Part 1? Now’s the time to use it.

Well, make sure you’ve committed and pushed all your changes, but when you’re ready, let’s go back in time to before we installed anything catscii-specific in our VM.

This should emulate the experience of a colleague onboarding onto the project well enough!

(I didn’t actually use VirtualBox’s snapshot feature for this, I actually set up a Ubuntu 22.10 VM on another computer entirely, but the effect should be much the same).

So, from that initial VM state, let’s clone our catscii project once again:

Oh, right, we have nothing. Well, let’s install git at least:

Then let’s go in there and try running cargo check:

Ah, right, we don’t have rustup/rustc/cargo either.

We could keep apt-installing things all day, but that’s kinda what we’re trying to avoid here. We’d install rustup, add its env setup to our ~/.bashrc, then notice we’re missing another thing, then another thing (docker, sqlite, flyctl, etc.), and that’s something we can solve, for everyone hacking on catscii, with nix.

Before we move on, we’ll have to install nix.

The nix download page has instructions, let’s follow the recommended “multi-user installation”.

It needs curl, so, sigh, and:

The install wizard is very nice and asks if you want to see a detailed list of what it’ll do, if it can use sudo (I let it), etc.

After it’s done, it lets us know nix won’t work in existing shells - I’ve been using VSCode SSH Remote so I’m able to just close the current shell with Ctrl-D (which sends EOF), open a new one with Ctrl+Shift+Backquote, and now I have:

This is not directly related to what we’re trying to achieve, but here’s something really neat you can do with nix: if you want to try out a package, you just can!

For example, cowsay isn’t installed in my VM right now:

But it’s in nixpkgs, so we can take it for a spin with nix-shell:

How does that work? From that shell, let’s find out where cowsay is in $PATH:

And what dynamic libraries it links to:

Ah. Well, let’s find out what it refers to when it runs, then:

It refers almost exclusively to things under /nix/store! In fact, the only things it refers to that aren’t under /nix/store are:

Let’s look at another executable in $PATH:

And that’s how nix works at runtime. There’s no containerization going on (no user namespaces), no virtual machines, no nothing: it’s just plain old executables, that only refer to things inside the /nix/store.

/nix/store is immutable, by the way:

Also, what’s that long hash about it? It’s derived from the inputs, which means, paths either never clash (because they have different inputs), or they deduplicate.

But also, you can have several versions of something and they won’t conflict. In my little nix-shell, I already have two versions of glibc, for example:

We can find out what needs either of them:

Using a slightly different query, we can see the full dependency tree of cowsay:

You can find these and more in the Nix to Debian phrasebook. I’ve found nix documentation hard to navigate, especially around newer stuff like flakes, but this gives us a nice starting point.

This is all temporary though: if we exit out of our nix-shell with Ctrl-D, we can’t execute cowsay anymore:

…or can we? It’s still there, after all…

Wait, does nix just leave stuff around? I thought nix-shell was temporary?

It does, until you garbage-collect! And then it only keeps stuff that’s referenced by a garbage collector root.

Ah, like docker system prune?

Kinda yeah!

So, how do we get cowsay to stick around?

Well, https://search.nixos.org/ tells us how to use nix-env, with nix-env -iA nixpkgs.cowsay, but that’s the “old” nix stuff, and we want to use the “new” nix stuff, so we’ll use nix profile instead:

Told you it was new! We don’t want to pass extra flags every time we run a nix command, so let’s create ~/.config/nix/nix.conf and add this to it:

That second line simply tells nix to take advantage of all cores when building stuff, which we haven’t really done yet.

Okay, now we can add cowsay to our profile:

And this time, it’ll survive a nix-collect-garbage:

Uh oh that still deleted stuff, are you sure it’s alright?

I’m sure it’s finnneee:

Good, but.. where is it? How does it work now?

Oh, it’s not a big mystery:

It’s “simply” a symbolic link to its location in /nix/store!

It’s also a Perl program, so it starts with a hash-pling:

…which points to the nix-installed perl, and not /usr/bin/perl, which also exists.

If we were using NixOS as our Linux distribution, there would be no /usr/bin/perl, only the nix-installed one, but I’m going for the pragmatic middle-of-the-road solution where you’re cozy and/or forced into using a more mainstream Linux distribution, like Ubuntu or Fedora, and you want to sprinkle a bit of nix on top.

nix profiles are neat, and we can list what’s installed in them with:

…but it’s not what we want for developing catscii. We want anyone to be able to jump in and develop the project, so if the README said “run nix profile install (a bunch of things)”, it wouldn’t be much better than saying “run apt install (another bunch of things)”.

Ideally, we want the list of things catscii needs to live directly in the repository, as a file.

And we can do that! In several ways, but again, let’s focus on the new, experimental, slightly-broken-at-times, opt-in stuff: flakes.

And my experience with nix has been a lot of “just copy paste this”, and having to deconstruct it later to understand what the heck is going on.

So, let’s start small.

Nix is a language. We can get a REPL!

We can have code in .nix files:

Cool Bear's hot tip

Before going any further: if you’re going to be running those locally as we go (and you should), here’s the recommended set-up with VS Code.

You want to install a formatter and an LSP server:

$ nix profile install nixpkgs#nixpkgs-fmt nixpkgs#rnix-lsp

Then grab the Nix IDE VS Code extension.

If you use VS Code with remote ssh, it’s possible Nix IDE complains about either of nixpkgs-fmt / rnix-lsp not being in $PATH (it’ll fail with ENOENT).

If killing the remote vscode server isn’t enough, you can always completely shut down VS Code (it caches too much stuff for its own good) and reboot the VM: that’s why we have VMs!

With that, you should have syntax highlighting, and you can enable “formatting on save” with this in your user settings:

{ "[nix]": { "editor.formatOnSave": true, "editor.defaultFormatter": "jnoortheen.nix-ide" }, }

All good? Good.

nix has sets, which you may be used to calling “dictionaries”, “objects” or “maps”, depending how your brain is wired. Anyway they associate keys and values:

And it has functions, which are called lambdas in some contexts:

Unfortunately, it’s hard to call it with nix eval due to an outstanding bug - you didn’t get warned because we opted into experimental stuff earlier, but nix eval is part of that experimental stuff and is a little half-baked.

We can have it print the lambda itself:

But to actually call it, we have to get creative. Instead of using --file, we can use --expr to evaluate an expression, and use import to import our .nix file:

Well, we can do that with --impure:

And then, well, we can call it:

Parentheses? What for?

Not to call the function, that’s for sure! This isn’t a lisp. Here they’re just used for grouping/priority: we want to access the .add_two property of the results of import ./sample.nix.

Note that ./sample.nix isn’t a string, it’s a path: that’s a distinct type in nix. We can’t import a string:

Okay, okay, this feels weird, but I’m assimilating.

Oh we’re not done! What if we want to bind the result of importing ./sample.nix to something before using it by name?

Since nix isn’t imperative, we don’t really have a concept of statements that are evaluated in order.

But there’s ways to work around that.

For simplicity, let’s make another nix file that’ll import sample.nix, so we don’t have to deal with --impure --expr.

Okay, so, I was saying: binding it to something. Well we can’t have “successive statements”, but we can have… sets!

Unfortunately, that doesn’t work:

Because by default, sets are non-recursive. We can fix that with rec:

This is a little untidy though: we don’t really need “sample” to be part of our output.

So instead, we can use a let:

Okay, we got pretty far already! So, let’s summarize:

• () are just like in C/Rust/whatever: they mean “evaluate that first”
• {} defines a set/object/dictionary/key-value map. it has key = value; inside (each item ends with a semicolon)
• [] defines a list/array: it has 1 2 3 inside (space-separated)
• foo.bar accesses property bar of set foo
• let lets you bind some names and use them in the in block later, which is preferred to using a recursive set.

You haven’t really explained why recursive sets are bad though.

Ah. Easy!

Oh. I see. Given that, I suppose order doesn’t matter in sets?

You are correct:

nix language is all well and good (…depending), but by itself it doesn’t do much.

The whole packagement system (sic.) built around it is the juicy bit.

And if you look at any nix tutorial, they’ll show you how to use mkDerivation. So of course, we won’t do that - we’ll go one level deeper.

So let’s remove all our nix files so far:

And create a default.nix file with a simple derivation:

There’s a lot going on here, so before we try to do something with this file, let’s make sure we understand what’s going.

We’re calling a function named derivation, passing a set to it. The value for name is a string, the value for args is a list, the value for src is a path.

What is system set to? Let’s find out:

Woops, that didn’t work. Does nix eval not give you builtins?

Oh, it does. It just doesn’t have currentSystem.

The reason nix eval didn’t recognize builtins.currentSystem is because it operates in a pure evaluation mode by default. In this mode, certain builtins, like currentSystem, aren’t available. But there’s another way to access these builtins.

How about the REPL?

That one has it, neat. The nix repl operates in an impure mode by default, which is why builtins.currentSystem is accessible there. If you want to use nix eval and see the value of currentSystem, you can add the --impure flag:

This flag switches the evaluation to impure mode, making currentSystem available.

The nastiest part of our default.nix file is definitely this, though:

Luckily, we know just enough nix to decipher what that means. Well, we didn’t know "${foo}" did string interpolation, but we do now.

We also didn’t know what <nixpkgs> is a magic path, but, yeah:

And so, if we follow logically, this must mean we build a string out of the “bash” property of whatever the result of calling import <nixpkgs> with an empty set gives us.

Because, yeah, import <nixpkgs> evaluates to a lambda:

So we must call it. But I strongly recommend not typing import <nixpkgs> {} in your REPL, because it’ll evaluate everything in there. At the time of this writing, there’s over 80K (eighty thousand) packages in there, and luckily, they warned fools like me:

Accessing a single field, though, is fine:

So what we’ve learned is that nix has lazy evaluation, which, if you’ve made it this far, you’ve probably already heard about, because it’s the one thing nobody will shut up about.

(But in their defense, that’s pretty important).

Now to do the same string interpolation as we have in default.nix, but in the REPL, to see what it gives us:

See, that’s interesting.

Because it just so happens that this path does exist right now:

But if we try with another package:

It doesn’t exist.

And that’s another thing you may have heard The Nix People go on about: the hash part is great, actually: because all inputs are statically defined, it can tell where a derivation is going to go (what its hash is going to be) without actually realizing it.

If we actually want to build dive, we first have to instantiate it:

We get a warning about the garbage collector – fair enough! We’re just playing around.

In return, we got the path of a .drv file, which I keep reading as “driver” but I suspect means “derivation”. This .drv file is human-readable (go look) but not really human-friendly.

Let’s use nix show /path/to/file.drv instead, which gives us JSON:

This specifies everything nix needs to realize that derivation.

But at this point, the output path still doesn’t exist:

If we want it, we have to realize it:

And now we have it!

Of course, it’ll be garbage-collected, just like they warned us:

Now onto our own derivation. It’s not in <nixpkgs>, it’s in a default.nix file.

We can evaluate it, just like any other .nix file:

That’s not terribly friendly. Normally we could ask for JSON with --json, but since this is a value of type “derivation”, it just shows the output path instead:

So, instead, I’ll just manually format what nix eval --file gave us:

So, essentially, a derivation seems like a spicy set. Okay!

Just like dive, we can instantiate our derivation to generate the .drv file:

Note that whereas to instantiate a nixpkgs package, we had to use --expr, here we can just pass a path: the current directory, ., which contains a default.nix file (that naming is not by accident).

Now that we have the .drv path, we can realize it with nix-store:

And now, it’s in our store!

I didn’t really explain what our derivation actually did. Well, it ran the bash builder with arguments -c "echo foo > $out". $out is an environment variable that contains the path where the derivation should generate their output.

Now for a bit of good news: we went much deeper than needed!

First off, we can clean up our default.nix - we know how to use let, so we can do this:

Alternatively, we can use the with syntax, which.. to me at least, “adds a set to the local scope”. If we do with something, we can refer to any property of something without doing something.property - we can just do property.

So in this version, we no longer need to refer to pkgs.bash, we can just refer to bash:

Cool Bear's hot tip

Bringing all of nixpkgs in scope is an anti-pattern. Using let and selecting what to inherit is preferred.

Secondly, we can use some bash powers to do both instantiation and realization in one command:

But let’s garbage-collect for a minute..

You know what does both instantiation and realization in one step?

That’s right! It’s nix-build!

What do you mean “that’s right”? That’s the first time you mention it?

Welllll maybe folks have used nix before on the surface, I don’t know.

Anyway:

Okay, cool, we know everything about nix!

Well.. no? What about mkDerivation?

Oh, right. So derivation is low-level and one hardly ever uses it.

Instead, mkDerivation is often used.

And this time, I think we’re ready to look at a bit of real-world nix. It’s something I wrote myself, for my video platform.

In this incarnation of my video platform, I use shaka-packager to package videos as MPEG-DASH (fragmented MP4 + an .mpd manifest).

It’s distributed as static binaries already, so the nix bit is actually fairly simple!

We can build it:

And then run it:

mkDerivation provides a bunch of nice things described in the nix derivation docs. It creates a temporary directory for the build (automatically removed after), clears the environment, sets a bunch of relevant environment variables, removes any existing output path, writes stdout and stderr to /nix/var/log/nix, sets timestamps to 00:00:01 1/1/1970 UTC (for reproducible builds), and also:

If the build was successful, Nix scans each output path for references to input paths by looking for the hash parts of the input paths. Since these are potential runtime dependencies, Nix registers them as dependencies of the output paths.

Isn’t that interesting!

You can also see in the build log (scroll up a little) that it tries to “patch script interpreter paths”. That’s right, executables are interpreted also, sort of: dynamically-loaded ones need a.. dynamic loader! The kind we built in making our own executable packer.

Before we move on to the more modern stuff, let’s note that derivations aren’t the only thing a .nix file can evaluate to.

Well… of course it’s not: you can have .nix files that evaluate to a single lambda for example, so that you can split out utility functions in other files.

But as far as “top-level” things go, it’s mostly derivations and.. dev shells!

Let’s remove our default.nix file for now – it doesn’t actually build catscii anyway, just shaka-packager, which is completely unrelated.

And let’s now create a shell.nix file with this:

Remember flyctl? That’s what we use to deploy our app to fly.io. Surely we’ll need that.

Let’s make sure we don’t have it installed in our Ubuntu VM (we shouldn’t, it certainly doesn’t ship with a default Ubuntu install and we reset our VM earlier):

Okay! And now the magic:

Hurray! We have a dev shell with flyctl in there.

Comment on /r/fasterthanlime

Here's another article just for you:

My ideal Rust workflow

Oct 26, 2021

57 min #rust · #release-engineering

Writing Rust is pretty neat. But you know what’s even neater? Continuously testing Rust, releasing Rust, and eventually, shipping Rust to production. And for that, we want more than plug-in for a code editor.

We want… a workflow.

Why I specifically care about this

Cool Bear's hot tip

This gets pretty long, so if all you want is the advice, feel free to jump to it directly.

Read more