fasterthanlime

Learning Nix from the bottom up

Thanks to my sponsors: Xavier Groleau, DaVince, Niels Abildgaard, Andronik, Mark Tomlin, Philipp Hatt, Michal Hošna, David E Disch, clement, Marcin Kołodziej, jatescher, Walther, Makoto Nakashima, Andy F, Kai Kaufman, Nyefan, Mike English, Timothée Gerber, Scott Sanderson, Brandon Piña and 267 more Xavier Groleau, DaVince, Niels Abildgaard, Andronik, Mark Tomlin, Philipp Hatt, Michal Hošna, David E Disch, clement, Marcin Kołodziej, jatescher, Walther, Makoto Nakashima, Andy F, Kai Kaufman, Nyefan, Mike English, Timothée Gerber, Scott Sanderson, Brandon Piña, Christian Bourjau, Òscar Pérez, James Rhodes, Adam Gutglick, Eugene Bulkin, Zoran Zaric, Michał Zalewski, Jean-David Gadina, Geoffroy Couprie, Raine Godmaire, Dirkjan Ochtman, Marc-Andre Giroux, prairiewolf, Mikkel Rasmussen, Braidon Whatley, Philipp Gniewosz, Borys Minaiev, Ahmad Alhashemi, Chris Walker, budrick, Jake Demarest-Mays, Mattia Valzelli, Tyler Schmidtke, Horváth-Lázár Péter, WeblWabl, Astrid, Jan De Landtsheer, Mathias Brossard, G, Hamilton Chapman, Sawyer Knoblich, Diego Roig, Berkus Decker, Matt Heise, Sarah Berrettini, Antoine Boegli, Romain Kelifa, Daniel Papp, David Barsky, Jon Gjengset, Marcus Griep, Pete Bevin, Marcus Griep, Cole Tobin, xales, Josh Triplett, Thehbadger, Alex Rudy, Evan Relf, Enrico Zschemisch, Dominik Wagner, Jesse Luehrs, Beat Scherrer, Alan O'Donnell, Alex Krantz, Tanner Muro, Mateusz Wykurz, Isak Sunde Singh, Hadrien G., Adrián Garnier Artiñano, Luke Yue, Ross Williams, Benjamin Röjder Delnavaz, Guy Waldman, Beth Rennie, Guillaume Demonet, Herman J. Radtke III, Daniel Silverstone, genny, Damir Vandic, Luuk, Menno Finlay-Smits, hgranthorner, Mark Old, pinkhatbeard, Kristoffer Winther Balling, Zaki, Scott Steele, Paul Marques Mota, René Ribaud, Olly Swanson, Matthew T, Jan-Stefan Janetzky, Senyo Simpson, James Leitch, Michael Alyn Miller, Bader, Andy Gocke, Lucille Blumire, Raphaël Thériault, Aleksandre Khokhiashvili, Luiz Ferraz, Ives van Hoorne, Taneli Kaivola, Noel, Aalekh Patel, Lennart Oldenburg, Marty Penner, Manuel Hutter, std__mpa, Joseph Montanaro, Lena Schönburg, Dylan Anthony, Jack Maguire, Ula, Urs Metz, Yuriy Taraday, Guillaume E, Simon Menke, you got maiL, SeniorMars, Peter Shih, David Cornu, Justin Ossevoort, playest, Antoine PESTEL-ROPARS, Elijah Voigt, Mikko Leppänen, callym, L0r3m1p5um, Dom, Tyler Bloom, Toon Willems, Mark, Vladimir, John VanEnk, Chris, Nicholas Orta, avborhanian, Chris Biscardi, Bob Ippolito, Radu Matei, Kamran Khan, Xirvik Servers, Max Heaton, Max von Forell, David White, Kyle Lacy, Yufan Lou, James Brown, Chirag Jain, Marco Carmosino, Justy, Duane Sibilly, Sindre Johansen, Cole Kurkowski, Sam Leonard, Jelle Besseling, Matěj Volf, Elnath, Olivia Crain, Elendol, Daniel Wagner-Hall, Zeeger Lubsen, Paige Ruten, Integer 32, LLC, Julian Schmid, Sylvie Nightshade, Christoph Grabo, Daniel Strittmatter, Joshua Roesslein, Luke Konopka, Morgan Rosenkranz, Seth, Marcus Brito, Das Gürteltier, Pete LeVasseur, AdrianEddy, compwhizii, Guilherme Neubaner, Ronen Ulanovsky, Adam Lassek, bbutkovic, Max Bruckner, Justin Smith, Sean Bryant, Christopher Valerio, Santiago Lema, Geoffrey Thomas, Garret Kelly, Boris Dolgov, belzael, Wojciech Smołka, Geoff Cant, Dimitri Merejkowsky, Aljaz Erzen, Marky Mark, ZacJW, Michael, Ben Mitchell, Richard Pringle, Steven Pham, Matt Jadczak, Ryan, Ian McLinden, Mario Fleischhacker, Romet Tagobert, Jonathan Adams, Ronen Cohen, Lyssieth, Samit Basu, Jim, Nicholas, Stephan Buys, johbii, Jack Duvall, Nicolas Riebesel, anichno, Richard Stephens, Gorazd Brumen, Zachary Thomas, Tabitha, Chris Sims, Antoine Rouaze, ofrighil, Valentin Mariette, Neil Blakey-Milner, Cass, medzernik, John Horowitz, Torben Clasen, Aiden Scandella, Yann Schwartz, Chris Emery, Mason Ginter, Henrik Tudborg, Michał Bartoszkiewicz, Matt Jackson, Victor Song, traxys, Michael Mrozek, Tobias Bahls, notryanb, e9zaktw1, Brooke Tilley, Johnathan Pagnutti, jer, Corey Alexander, Blake Johnson, Ivo Murrell, Ben Wishovich, C J Silverio, Egor Ternovoi, Josiah Bull, Shane Lillie, Rufus Cable, Gioele Pannetto, ACRL, teor, Jörn Huxhorn, Paul Horn, Matt Campbell, Yves, old.woman.josiah, ShikChen, Tiziano Santoro, villem, Andrew Neth, Dave Minter, Colin VanDervoort, Malik Bougacha, Marie Janssen, Chris Thackrey, Laine Taffin Altman, Wyatt Herkamp, qrpth, Thor Kamphefner

NOW RECRUITING

bearcove is recruiting a video editor for fasterthanlime.

Applicants need:

• a mac
• good english
• great internet
• willingness to learn a specific DaVinci Resolve-based workflow

Is this you? Or do you know someone who is? check out the complete job listing here

Remember the snapshot we made allll the way back in Part 1? Now’s the time to use it.

Well, make sure you’ve committed and pushed all your changes, but when you’re ready, let’s go back in time to before we installed anything catscii-specific in our VM.

This should emulate the experience of a colleague onboarding onto the project well enough!

(I didn’t actually use VirtualBox’s snapshot feature for this, I actually set up a Ubuntu 22.10 VM on another computer entirely, but the effect should be much the same).

So, from that initial VM state, let’s clone our catscii project once again:

Oh, right, we have nothing. Well, let’s install git at least:

Then let’s go in there and try running cargo check:

Ah, right, we don’t have rustup/rustc/cargo either.

We could keep apt-installing things all day, but that’s kinda what we’re trying to avoid here. We’d install rustup, add its env setup to our ~/.bashrc, then notice we’re missing another thing, then another thing (docker, sqlite, flyctl, etc.), and that’s something we can solve, for everyone hacking on catscii, with nix.

Before we move on, we’ll have to install nix.

The nix download page has instructions, let’s follow the recommended “multi-user installation”.

It needs curl, so, sigh, and:

The install wizard is very nice and asks if you want to see a detailed list of what it’ll do, if it can use sudo (I let it), etc.

After it’s done, it lets us know nix won’t work in existing shells - I’ve been using VSCode SSH Remote so I’m able to just close the current shell with Ctrl-D (which sends EOF), open a new one with Ctrl+Shift+Backquote, and now I have:

This is not directly related to what we’re trying to achieve, but here’s something really neat you can do with nix: if you want to try out a package, you just can!

For example, cowsay isn’t installed in my VM right now:

But it’s in nixpkgs, so we can take it for a spin with nix-shell:

How does that work? From that shell, let’s find out where cowsay is in $PATH:

And what dynamic libraries it links to:

Ah. Well, let’s find out what it refers to when it runs, then:

It refers almost exclusively to things under /nix/store! In fact, the only things it refers to that aren’t under /nix/store are:

Let’s look at another executable in $PATH:

And that’s how nix works at runtime. There’s no containerization going on (no user namespaces), no virtual machines, no nothing: it’s just plain old executables, that only refer to things inside the /nix/store.

/nix/store is immutable, by the way:

Also, what’s that long hash about it? It’s derived from the inputs, which means, paths either never clash (because they have different inputs), or they deduplicate.

But also, you can have several versions of something and they won’t conflict. In my little nix-shell, I already have two versions of glibc, for example:

We can find out what needs either of them:

Using a slightly different query, we can see the full dependency tree of cowsay:

You can find these and more in the Nix to Debian phrasebook. I’ve found nix documentation hard to navigate, especially around newer stuff like flakes, but this gives us a nice starting point.

This is all temporary though: if we exit out of our nix-shell with Ctrl-D, we can’t execute cowsay anymore:

…or can we? It’s still there, after all…

Wait, does nix just leave stuff around? I thought nix-shell was temporary?

It does, until you garbage-collect! And then it only keeps stuff that’s referenced by a garbage collector root.

Ah, like docker system prune?

Kinda yeah!

So, how do we get cowsay to stick around?

Well, https://search.nixos.org/ tells us how to use nix-env, with nix-env -iA nixpkgs.cowsay, but that’s the “old” nix stuff, and we want to use the “new” nix stuff, so we’ll use nix profile instead:

Told you it was new! We don’t want to pass extra flags every time we run a nix command, so let’s create ~/.config/nix/nix.conf and add this to it:

That second line simply tells nix to take advantage of all cores when building stuff, which we haven’t really done yet.

Okay, now we can add cowsay to our profile:

And this time, it’ll survive a nix-collect-garbage:

Uh oh that still deleted stuff, are you sure it’s alright?

I’m sure it’s finnneee:

Good, but.. where is it? How does it work now?

Oh, it’s not a big mystery:

It’s “simply” a symbolic link to its location in /nix/store!

It’s also a Perl program, so it starts with a hash-pling:

…which points to the nix-installed perl, and not /usr/bin/perl, which also exists.

If we were using NixOS as our Linux distribution, there would be no /usr/bin/perl, only the nix-installed one, but I’m going for the pragmatic middle-of-the-road solution where you’re cozy and/or forced into using a more mainstream Linux distribution, like Ubuntu or Fedora, and you want to sprinkle a bit of nix on top.

nix profiles are neat, and we can list what’s installed in them with:

…but it’s not what we want for developing catscii. We want anyone to be able to jump in and develop the project, so if the README said “run nix profile install (a bunch of things)”, it wouldn’t be much better than saying “run apt install (another bunch of things)”.

Ideally, we want the list of things catscii needs to live directly in the repository, as a file.

And we can do that! In several ways, but again, let’s focus on the new, experimental, slightly-broken-at-times, opt-in stuff: flakes.

And my experience with nix has been a lot of “just copy paste this”, and having to deconstruct it later to understand what the heck is going on.

So, let’s start small.

Nix is a language. We can get a REPL!

We can have code in .nix files:

Cool Bear's hot tip

Before going any further: if you’re going to be running those locally as we go (and you should), here’s the recommended set-up with VS Code.

You want to install a formatter and an LSP server:

$ nix profile install nixpkgs#nixpkgs-fmt nixpkgs#rnix-lsp

Then grab the Nix IDE VS Code extension.

If you use VS Code with remote ssh, it’s possible Nix IDE complains about either of nixpkgs-fmt / rnix-lsp not being in $PATH (it’ll fail with ENOENT).

If killing the remote vscode server isn’t enough, you can always completely shut down VS Code (it caches too much stuff for its own good) and reboot the VM: that’s why we have VMs!

With that, you should have syntax highlighting, and you can enable “formatting on save” with this in your user settings:

{ "[nix]": { "editor.formatOnSave": true, "editor.defaultFormatter": "jnoortheen.nix-ide" }, }

All good? Good.

nix has sets, which you may be used to calling “dictionaries”, “objects” or “maps”, depending how your brain is wired. Anyway they associate keys and values:

And it has functions, which are called lambdas in some contexts:

Unfortunately, it’s hard to call it with nix eval due to an outstanding bug - you didn’t get warned because we opted into experimental stuff earlier, but nix eval is part of that experimental stuff and is a little half-baked.

We can have it print the lambda itself:

But to actually call it, we have to get creative. Instead of using --file, we can use --expr to evaluate an expression, and use import to import our .nix file:

Well, we can do that with --impure:

And then, well, we can call it:

Parentheses? What for?

Not to call the function, that’s for sure! This isn’t a lisp. Here they’re just used for grouping/priority: we want to access the .add_two property of the results of import ./sample.nix.

Note that ./sample.nix isn’t a string, it’s a path: that’s a distinct type in nix. We can’t import a string:

Okay, okay, this feels weird, but I’m assimilating.

Oh we’re not done! What if we want to bind the result of importing ./sample.nix to something before using it by name?

Since nix isn’t imperative, we don’t really have a concept of statements that are evaluated in order.

But there’s ways to work around that.

For simplicity, let’s make another nix file that’ll import sample.nix, so we don’t have to deal with --impure --expr.

Okay, so, I was saying: binding it to something. Well we can’t have “successive statements”, but we can have… sets!

Unfortunately, that doesn’t work:

Because by default, sets are non-recursive. We can fix that with rec:

This is a little untidy though: we don’t really need “sample” to be part of our output.

So instead, we can use a let:

Okay, we got pretty far already! So, let’s summarize:

• () are just like in C/Rust/whatever: they mean “evaluate that first”
• {} defines a set/object/dictionary/key-value map. it has key = value; inside (each item ends with a semicolon)
• [] defines a list/array: it has 1 2 3 inside (space-separated)
• foo.bar accesses property bar of set foo
• let lets you bind some names and use them in the in block later, which is preferred to using a recursive set.

You haven’t really explained why recursive sets are bad though.

Ah. Easy!

Oh. I see. Given that, I suppose order doesn’t matter in sets?

You are correct:

nix language is all well and good (…depending), but by itself it doesn’t do much.

The whole packagement system (sic.) built around it is the juicy bit.

And if you look at any nix tutorial, they’ll show you how to use mkDerivation. So of course, we won’t do that - we’ll go one level deeper.

So let’s remove all our nix files so far:

And create a default.nix file with a simple derivation:

There’s a lot going on here, so before we try to do something with this file, let’s make sure we understand what’s going.

We’re calling a function named derivation, passing a set to it. The value for name is a string, the value for args is a list, the value for src is a path.

What is system set to? Let’s find out:

Woops, that didn’t work. Does nix eval not give you builtins?

Oh, it does. It just doesn’t have currentSystem.

The reason nix eval didn’t recognize builtins.currentSystem is because it operates in a pure evaluation mode by default. In this mode, certain builtins, like currentSystem, aren’t available. But there’s another way to access these builtins.

How about the REPL?

That one has it, neat. The nix repl operates in an impure mode by default, which is why builtins.currentSystem is accessible there. If you want to use nix eval and see the value of currentSystem, you can add the --impure flag:

This flag switches the evaluation to impure mode, making currentSystem available.

The nastiest part of our default.nix file is definitely this, though:

Luckily, we know just enough nix to decipher what that means. Well, we didn’t know "${foo}" did string interpolation, but we do now.

We also didn’t know what <nixpkgs> is a magic path, but, yeah:

And so, if we follow logically, this must mean we build a string out of the “bash” property of whatever the result of calling import <nixpkgs> with an empty set gives us.

Because, yeah, import <nixpkgs> evaluates to a lambda:

So we must call it. But I strongly recommend not typing import <nixpkgs> {} in your REPL, because it’ll evaluate everything in there. At the time of this writing, there’s over 80K (eighty thousand) packages in there, and luckily, they warned fools like me:

Accessing a single field, though, is fine:

So what we’ve learned is that nix has lazy evaluation, which, if you’ve made it this far, you’ve probably already heard about, because it’s the one thing nobody will shut up about.

(But in their defense, that’s pretty important).

Now to do the same string interpolation as we have in default.nix, but in the REPL, to see what it gives us:

See, that’s interesting.

Because it just so happens that this path does exist right now:

But if we try with another package:

It doesn’t exist.

And that’s another thing you may have heard The Nix People go on about: the hash part is great, actually: because all inputs are statically defined, it can tell where a derivation is going to go (what its hash is going to be) without actually realizing it.

If we actually want to build dive, we first have to instantiate it:

We get a warning about the garbage collector – fair enough! We’re just playing around.

In return, we got the path of a .drv file, which I keep reading as “driver” but I suspect means “derivation”. This .drv file is human-readable (go look) but not really human-friendly.

Let’s use nix show /path/to/file.drv instead, which gives us JSON:

This specifies everything nix needs to realize that derivation.

But at this point, the output path still doesn’t exist:

If we want it, we have to realize it:

And now we have it!

Of course, it’ll be garbage-collected, just like they warned us:

Now onto our own derivation. It’s not in <nixpkgs>, it’s in a default.nix file.

We can evaluate it, just like any other .nix file:

That’s not terribly friendly. Normally we could ask for JSON with --json, but since this is a value of type “derivation”, it just shows the output path instead:

So, instead, I’ll just manually format what nix eval --file gave us:

So, essentially, a derivation seems like a spicy set. Okay!

Just like dive, we can instantiate our derivation to generate the .drv file:

Note that whereas to instantiate a nixpkgs package, we had to use --expr, here we can just pass a path: the current directory, ., which contains a default.nix file (that naming is not by accident).

Now that we have the .drv path, we can realize it with nix-store:

And now, it’s in our store!

I didn’t really explain what our derivation actually did. Well, it ran the bash builder with arguments -c "echo foo > $out". $out is an environment variable that contains the path where the derivation should generate their output.

Now for a bit of good news: we went much deeper than needed!

First off, we can clean up our default.nix - we know how to use let, so we can do this:

Alternatively, we can use the with syntax, which.. to me at least, “adds a set to the local scope”. If we do with something, we can refer to any property of something without doing something.property - we can just do property.

So in this version, we no longer need to refer to pkgs.bash, we can just refer to bash:

Cool Bear's hot tip

Bringing all of nixpkgs in scope is an anti-pattern. Using let and selecting what to inherit is preferred.

Secondly, we can use some bash powers to do both instantiation and realization in one command:

But let’s garbage-collect for a minute..

You know what does both instantiation and realization in one step?

That’s right! It’s nix-build!

What do you mean “that’s right”? That’s the first time you mention it?

Welllll maybe folks have used nix before on the surface, I don’t know.

Anyway:

Okay, cool, we know everything about nix!

Well.. no? What about mkDerivation?

Oh, right. So derivation is low-level and one hardly ever uses it.

Instead, mkDerivation is often used.

And this time, I think we’re ready to look at a bit of real-world nix. It’s something I wrote myself, for my video platform.

In this incarnation of my video platform, I use shaka-packager to package videos as MPEG-DASH (fragmented MP4 + an .mpd manifest).

It’s distributed as static binaries already, so the nix bit is actually fairly simple!

We can build it:

And then run it:

mkDerivation provides a bunch of nice things described in the nix derivation docs. It creates a temporary directory for the build (automatically removed after), clears the environment, sets a bunch of relevant environment variables, removes any existing output path, writes stdout and stderr to /nix/var/log/nix, sets timestamps to 00:00:01 1/1/1970 UTC (for reproducible builds), and also:

If the build was successful, Nix scans each output path for references to input paths by looking for the hash parts of the input paths. Since these are potential runtime dependencies, Nix registers them as dependencies of the output paths.

Isn’t that interesting!

You can also see in the build log (scroll up a little) that it tries to “patch script interpreter paths”. That’s right, executables are interpreted also, sort of: dynamically-loaded ones need a.. dynamic loader! The kind we built in making our own executable packer.

Before we move on to the more modern stuff, let’s note that derivations aren’t the only thing a .nix file can evaluate to.

Well… of course it’s not: you can have .nix files that evaluate to a single lambda for example, so that you can split out utility functions in other files.

But as far as “top-level” things go, it’s mostly derivations and.. dev shells!

Let’s remove our default.nix file for now – it doesn’t actually build catscii anyway, just shaka-packager, which is completely unrelated.

And let’s now create a shell.nix file with this:

Remember flyctl? That’s what we use to deploy our app to fly.io. Surely we’ll need that.

Let’s make sure we don’t have it installed in our Ubuntu VM (we shouldn’t, it certainly doesn’t ship with a default Ubuntu install and we reset our VM earlier):

Okay! And now the magic:

Hurray! We have a dev shell with flyctl in there.

Comment on /r/fasterthanlime

NOW RECRUITING

bearcove is recruiting a video editor for fasterthanlime.

Applicants need:

• a mac
• good english
• great internet
• willingness to learn a specific DaVinci Resolve-based workflow

Is this you? Or do you know someone who is? check out the complete job listing here