Thanks to my sponsors: Joshua Roesslein, Peter Shih, hgranthorner, Ross Williams, Santiago Lema, Aleksandre Khokhiashvili, Vincent, Braidon Whatley, Luke Konopka, Miguel Piedrafita, Richard Stephens, Jack Duvall, Josh Triplett, Marty Penner, Colin VanDervoort, Lyssieth, James Rhodes, Ian McLinden, Guillaume Demonet, Timothée Gerber and 232 more

Mar 05, 2023

51 min #nix

Learning Nix from the bottom up

From the series Building a Rust service with Nix

1

2

3

4

5

6

7

8

9

10

11

12

Remember the snapshot we made allll the way back in Part 1? Now's the time to use it.

Well, make sure you've committed and pushed all your changes, but when you're ready, let's go back in time to before we installed anything catscii-specific in our VM.

This should emulate the experience of a colleague onboarding onto the project well enough!

(I didn't actually use VirtualBox's snapshot feature for this, I actually set up a Ubuntu 22.10 VM on another computer entirely, but the effect should be much the same).

So, from that initial VM state, let's clone our catscii project once again:

$ git clone git@github.com:fasterthanlime/catscii.git
Command 'git' not found, but can be installed with:
sudo apt install git

Oh, right, we have nothing. Well, let's install git at least:

$ sudo apt update
(cut)

$ sudo apt install git
(cut)

$ git clone git@github.com:fasterthanlime/catscii.git
Cloning into 'catscii'...
The authenticity of host 'github.com (140.82.121.3)' can't be established.
ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
remote: Enumerating objects: 137, done.
remote: Counting objects: 100% (137/137), done.
remote: Compressing objects: 100% (73/73), done.
remote: Total 137 (delta 63), reused 115 (delta 41), pack-reused 0
Receiving objects: 100% (137/137), 48.26 KiB | 484.00 KiB/s, done.
Resolving deltas: 100% (63/63), done.

Then let's go in there and try running cargo check:

$ cargo check
Command 'cargo' not found, but can be installed with:
sudo snap install rustup  # version 1.24.3, or
sudo apt  install cargo   # version 0.60.0ubuntu1-0ubuntu3
See 'snap info rustup' for additional versions.

Ah, right, we don't have rustup/rustc/cargo either.

We could keep apt-installing things all day, but that's kinda what we're trying to avoid here. We'd install rustup, add its env setup to our ~/.bashrc, then notice we're missing another thing, then another thing (docker, sqlite, flyctl, etc.), and that's something we can solve, for everyone hacking on catscii, with nix.

Installing nix

Before we move on, we'll have to install nix.

The nix download page has instructions, let's follow the recommended "multi-user installation".

It needs curl, so, sigh, and:

$ sudo apt install curl
(cut)

$ sh <(curl --location https://nixos.org/nix/install) --daemon

The install wizard is very nice and asks if you want to see a detailed list of what it'll do, if it can use sudo (I let it), etc.

After it's done, it lets us know nix won't work in existing shells - I've been using VSCode SSH Remote so I'm able to just close the current shell with Ctrl-D (which sends EOF), open a new one with Ctrl+Shift+Backquote, and now I have:

$ nix --version
nix (Nix) 2.13.2

Trying out nix-shell

This is not directly related to what we're trying to achieve, but here's something really neat you can do with nix: if you want to try out a package, you just can!

For example, cowsay isn't installed in my VM right now:

$ cowsay "Well hi there"
Command 'cowsay' not found, but can be installed with:
sudo apt install cowsay

But it's in nixpkgs, so we can take it for a spin with nix-shell:

$ nix-shell --packages cowsay
(cut: lots of downloading going on)

[nix-shell:~]$ cowsay "Well well well"
 ________________
< Well well well >
 ----------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

How does that work? From that shell, let's find out where cowsay is in $PATH:

[nix-shell:~]$ which cowsay
/nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/bin/cowsay

And what dynamic libraries it links to:

$ ldd $(which cowsay)
$       not a dynamic executable

Ah. Well, let's find out what it refers to when it runs, then:

[nix-shell:~]$ strace cowsay "Well well well" 2>&1 | grep --fixed-strings '"/'
execve("/nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/bin/cowsay", ["cowsay", "Well well well"], 0x7fff84a29b98 /* 103 vars */) = 0
access("/etc/ld-nix.so.preload", R_OK)  = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/bja5fzkjjn26zz742pgkvpm5dq3girv9-perl-5.36.0/lib/perl5/5.36.0/x86_64-linux-thread-multi/CORE/glibc-hwcaps/x86-64-v2/libperl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/bja5fzkjjn26zz742pgkvpm5dq3girv9-perl-5.36.0/lib/perl5/5.36.0/x86_64-linux-thread-multi/CORE/glibc-hwcaps/x86-64-v2", 0x7ffd39942530, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/bja5fzkjjn26zz742pgkvpm5dq3girv9-perl-5.36.0/lib/perl5/5.36.0/x86_64-linux-thread-multi/CORE/tls/x86_64/x86_64/libperl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
(cut)
newfstatat(AT_FDCWD, "/nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/share/cowsay/site-cows/default.cow.pm", 0x161e4a8, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/share/cowsay/cows/default.cow", {st_mode=S_IFREG|0444, st_size=175, ...}, 0) = 0
newfstatat(AT_FDCWD, "/nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/share/cowsay/cows/default.cow", {st_mode=S_IFREG|0444, st_size=175, ...}, 0) = 0
openat(AT_FDCWD, "/nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/share/cowsay/cows/default.cow", O_RDONLY|O_CLOEXEC) = 3

It refers almost exclusively to things under /nix/store! In fact, the only things it refers to that aren't under /nix/store are:

access("/etc/ld-nix.so.preload", R_OK)  = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/dev/urandom", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/dev/urandom", O_RDONLY|O_CLOEXEC) = 3

Let's look at another executable in $PATH:

[nix-shell:~]$ ldd $(which bash)
        linux-vdso.so.1 (0x00007ffdde95f000)
        libreadline.so.8 => /nix/store/pwhg9383i9j7vxhl2sb5mxsbdy70mrzj-readline-8.2p1/lib/libreadline.so.8 (0x00007f6e70d20000)
        libhistory.so.8 => /nix/store/pwhg9383i9j7vxhl2sb5mxsbdy70mrzj-readline-8.2p1/lib/libhistory.so.8 (0x00007f6e70d13000)
        libncursesw.so.6 => /nix/store/35s126gfkfrwbiv49kv9kxqdpd9zvcvm-ncurses-6.4/lib/libncursesw.so.6 (0x00007f6e70c9f000)
        libdl.so.2 => /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/libdl.so.2 (0x00007f6e70c9a000)
        libc.so.6 => /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/libc.so.6 (0x00007f6e70a00000)
        /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/ld-linux-x86-64.so.2 => /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib64/ld-linux-x86-64.so.2 (0x00007f6e70d7b000)

And that's how nix works at runtime. There's no containerization going on (no user namespaces), no virtual machines, no nothing: it's just plain old executables, that only refer to things inside the /nix/store.

/nix/store is immutable, by the way:

[nix-shell:~]$ touch /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/libdl.so.2
touch: cannot touch '/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/libdl.so.2': Permission denied

Also, what's that long hash about it? It's derived from the inputs, which means, paths either never clash (because they have different inputs), or they deduplicate.

But also, you can have several versions of something and they won't conflict. In my little nix-shell, I already have two versions of glibc, for example:

[nix-shell:~]$ find /nix/store/*glibc*/ -name 'libdl.so'
/nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libdl.so
/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/libdl.so

We can find out what needs either of them:

[nix-shell:~]$ nix-store --query --referrers /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/ | head -5
/nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163
/nix/store/026hln0aq1hyshaxsdvhg0kmcm6yf45r-zlib-1.2.13
/nix/store/3j1h6psl4pzn6b3yck6rk33bpwrmihb1-aws-c-common-0.8.5
/nix/store/4mxnw95jcm5a27qk60z7yc0gvxp42b9a-openssl-3.0.7
/nix/store/4rkhsf7sig2lh303bygqr3ph5mfwz0ah-s2n-tls-1.3.28

[nix-shell:~]$ nix-store --query --referrers /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/ | head -5
/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224
/nix/store/502zrfyc2agh928111zhwy6959bqd8m0-xz-5.4.1
/nix/store/12wczkzi5db1ajbjp4hdyif3v9y5rbi5-xz-5.4.1-bin
/nix/store/2a6yagz3pa8kiawg5mk2js70f8kwqzqd-bzip2-1.0.8
/nix/store/17pkxcz3js3549kn9dc3hhvp4adbwvs1-bzip2-1.0.8-bin

Using a slightly different query, we can see the full dependency tree of cowsay:

# piping to cat to avoid the pager
[nix-shell:~]$ nix-store --query --tree $(which cowsay) | cat
/nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0
└───/nix/store/bja5fzkjjn26zz742pgkvpm5dq3girv9-perl-5.36.0
    ├───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224
    │   ├───/nix/store/na1irnycfp8z5mab0g5jvrnhnscsaqsb-libidn2-2.3.2
    │   │   ├───/nix/store/jdjpni8kq3i95dj1d49nlf9m10wl0kqq-libunistring-1.0
    │   │   │   └───/nix/store/jdjpni8kq3i95dj1d49nlf9m10wl0kqq-libunistring-1.0 [...]
    │   │   └───/nix/store/na1irnycfp8z5mab0g5jvrnhnscsaqsb-libidn2-2.3.2 [...]
    │   └───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
    ├───/nix/store/7w1s50i8yplw2gkpf9hpha6n9vci8g98-coreutils-9.1
    │   ├───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
    │   ├───/nix/store/83q5sbysda8285svx8mjlpc17bd3n630-attr-2.5.1
    │   │   ├───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
    │   │   └───/nix/store/83q5sbysda8285svx8mjlpc17bd3n630-attr-2.5.1 [...]
    │   ├───/nix/store/aqd9cz92qz1pfnvw09bsidi6vrcfhl9r-gmp-with-cxx-stage4-6.2.1
    │   │   ├───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
    │   │   ├───/nix/store/k88zxp7cvd5gpharprhg9ah0vhz2asq7-gcc-12.2.0-lib
    │   │   │   ├───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
    │   │   │   └───/nix/store/k88zxp7cvd5gpharprhg9ah0vhz2asq7-gcc-12.2.0-lib [...]
    │   │   └───/nix/store/aqd9cz92qz1pfnvw09bsidi6vrcfhl9r-gmp-with-cxx-stage4-6.2.1 [...]
    │   ├───/nix/store/jwlr603mdcq2lkaqpxmg9kkp2acszb1n-acl-2.3.1
    │   │   ├───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
    │   │   ├───/nix/store/83q5sbysda8285svx8mjlpc17bd3n630-attr-2.5.1 [...]
    │   │   └───/nix/store/jwlr603mdcq2lkaqpxmg9kkp2acszb1n-acl-2.3.1 [...]
    │   └───/nix/store/7w1s50i8yplw2gkpf9hpha6n9vci8g98-coreutils-9.1 [...]
    ├───/nix/store/9dz5lmff9ywas225g6cpn34s0wbldnxa-zlib-1.2.13
    │   └───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
    ├───/nix/store/kaiinm7cjirmv4zd3pdqar23rsypfqlh-libxcrypt-4.4.33
    │   ├───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
    │   └───/nix/store/kaiinm7cjirmv4zd3pdqar23rsypfqlh-libxcrypt-4.4.33 [...]
    └───/nix/store/bja5fzkjjn26zz742pgkvpm5dq3girv9-perl-5.36.0 [...]

You can find these and more in the Nix to Debian phrasebook. I've found nix documentation hard to navigate, especially around newer stuff like flakes, but this gives us a nice starting point.

This is all temporary though: if we exit out of our nix-shell with Ctrl-D, we can't execute cowsay anymore:

$ cowsay
Command 'cowsay' not found, but can be installed with:
sudo apt install cowsay

...or can we? It's still there, after all...

$ /nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/bin/cowsay "Look who's still here"
 _______________________
< Look who's still here >
 -----------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Wait, does nix just leave stuff around? I thought nix-shell was temporary?

It does, until you garbage-collect! And then it only keeps stuff that's referenced by a garbage collector root.

$ nix-collect-garbage
finding garbage collector roots...
deleting garbage...
deleting '/nix/store/4gj19wzyqhkdji9xg7hipy2m2899s0gg-shell.drv'
deleting '/nix/store/43gc1j2wpjzs5avyk0fa4j77sxwdih2l-cowsay-3.7.0.drv'
deleting '/nix/store/fyh9bz0ba9f6ihlazs78mcjnpbxj0bz4-source.drv'
(cut)
deleting '/nix/store/py1pdcfhj845cyam6ndq0b2mh02zk4my-openssl-3.0.8.tar.gz.drv'
deleting '/nix/store/0v7w62jca8nw6w31zs4y2805jj670f6p-no-arch_only-8.2.patch'
deleting unused links...
note: currently hard linking saves -0.00 MiB
417 store paths deleted, 386.64 MiB freed

Ah, like docker system prune?

Kinda yeah!

nix profile

So, how do we get cowsay to stick around?

Well, https://search.nixos.org/ tells us how to use nix-env, with nix-env -iA nixpkgs.cowsay, but that's the "old" nix stuff, and we want to use the "new" nix stuff, so we'll use nix profile instead:

$ nix profile install 'nixpkgs#cowsay'
error: experimental Nix feature 'nix-command' is disabled; use '--extra-experimental-features nix-command' to override

Told you it was new! We don't want to pass extra flags every time we run a nix command, so let's create ~/.config/nix/nix.conf and add this to it:

experimental-features = nix-command flakes
max-jobs = auto

That second line simply tells nix to take advantage of all cores when building stuff, which we haven't really done yet.

Okay, now we can add cowsay to our profile:

$ nix profile install 'nixpkgs#cowsay'
$ cowsay "Guess who's back"
 __________________
< Guess who's back >
 ------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

And this time, it'll survive a nix-collect-garbage:

$ nix-collect-garbage
finding garbage collector roots...
deleting garbage...
deleting '/nix/store/43gc1j2wpjzs5avyk0fa4j77sxwdih2l-cowsay-3.7.0.drv'
deleting '/nix/store/fyh9bz0ba9f6ihlazs78mcjnpbxj0bz4-source.drv'
deleting '/nix/store/rrr6h7wib0c1mg3d79xas4ac6b0yblwg-unzip-6.0.drv'
deleting '/nix/store/cciw7lgkldvx25d77cxpjhh1iw4xghd9-setup-hook.sh'
deleting '/nix/store/kwprl4pibdiiz1pzvdqcxv1h3y962ncy-source'
deleting '/nix/store/q9bnj7gmw8mdrssm1mzn9j0dsb32vada-glibc-locales-2.35-224.drv'
deleting '/nix/store/d46j5s34jfs779lg392bz2ll81ngm26d-python3-minimal-3.10.9.drv'
deleting '/nix/store/xfqq6rb3i2r2n65l64c4s0lm2y5bcbz4-pkg-config-wrapper-0.29.2.drv'
deleting '/nix/store/wjs1ma47yzsxxiiyrl923iddfcf7a71d-expat-2.5.0.drv'
deleting '/nix/store/scq9sx9lfj49523a0fk1mmfzypzvbkgd-CVE-2019-13232-2.patch.drv'
deleting '/nix/store/pqz31j5dcw1pc5cszdrkrz3scpkvrav3-Python-3.10.9.tar.xz.drv'
deleting '/nix/store/l4aa6fimwdzlzbyihb845dk44zrsw2ml-pkg-config-0.29.2.drv'
deleting '/nix/store/kmvyag3kxqgc7km7phjq2v1adhf6rzzv-9e129fa0933cf1837672c97f5ae5ad4a1a10ec11.patch.drv'
deleting '/nix/store/k43ngx9q4xybxbfz5d7vriixhqrd30sl-unzip60.tar.gz.drv'
deleting '/nix/store/isyf5h4pwzzm4y6qdhdni5q304nbjvr7-libffi-3.4.4.drv'
deleting '/nix/store/d1iivmdhrfdi9n0ilpfp94gjl98w7hnb-nuke-references.drv'
deleting '/nix/store/byrqr5zh6lq097wy30ix9azdjgiqvs57-pkg-config-0.29.2.tar.gz.drv'
deleting '/nix/store/a15i7dsiv2l1s9ssyvc84v3wf125yw6i-CVE-2019-13232-3.patch.drv'
deleting '/nix/store/9y996ql1kszsr960p8bqnnyi7nfyczgh-glibc-2.35.tar.xz.drv'
deleting '/nix/store/92gmlz1g08lm7d92p43wz1b4sfcn9b6q-patchutils-0.3.3.drv'
deleting '/nix/store/ckrkr5sna6k8j41dsz0jlwk7v4mkaljl-patchutils-0.3.3.tar.xz.drv'
deleting '/nix/store/8y4bvjq5n9krg2s8zgkhbv6xk1hjidr3-bison-3.8.2.drv'
deleting '/nix/store/sndbhmzm6c7rh8mbpr373x4nvwfnxj6x-bison-3.8.2.tar.gz.drv'
deleting '/nix/store/8q43vcy567y9b749fyqfbkqw5mx0jbnm-linux-headers-6.1.drv'
deleting '/nix/store/lwhj44wjw92hlrg00z802nwbnwwc46qi-linux-6.1.tar.xz.drv'
deleting '/nix/store/86jb692kfkm05ksi9rsqa2w45ihg98gm-autoreconf-hook.drv'
deleting '/nix/store/f5rgi9bv4ghsz23z3rg0f7lvja0ba7q9-libtool-2.4.7.drv'
deleting '/nix/store/qgcpw59w37sa48yr6566c92kxnhr727s-file-5.44.drv'
deleting '/nix/store/bcvr111pfvziv1d62zzkcdkhd55mxn86-file-5.44.tar.gz.drv'
deleting '/nix/store/b9a3hmzq303dk33n7b7abzh22k4kkcg9-automake-1.16.5.drv'
deleting '/nix/store/q1cfch1x2ai0ww0jbcxh1mj0lnz37bx4-autoconf-2.71.drv'
deleting '/nix/store/q2a79fndpzqj7nnpcps69dc9x8wq7qwh-autoconf-2.71.tar.xz.drv'
deleting '/nix/store/fpdqsslr23vygwd8hb4rhq0n8h41svca-texinfo-6.8.drv'
deleting '/nix/store/ynl8y88v27ih4w6xnc5rx1y0l1wsxwcc-perl-5.36.0.drv'
deleting '/nix/store/icpmjjaxq9c902r2cxvzgkijhc0dw7lw-perl-5.36.0.tar.gz.drv'
deleting '/nix/store/f4scqssxpx249w3bknfpkcg10y3rxgli-texinfo-6.8.tar.xz.drv'
deleting '/nix/store/cszf8j3mlcv9n127ywiknfzg244bvk0w-gnum4-1.4.19.drv'
deleting '/nix/store/nyibn3sc1k866pg7zjx9vz9h86kdj3ip-m4-1.4.19.tar.bz2.drv'
deleting '/nix/store/7w75i7xdfg1h45z2fp3yz5i1864ygj6j-autoconf-archive-2022.09.03.drv'
deleting '/nix/store/vp90pdrz9gnjwwdkq8bqwbmln6qdfiff-autoconf-archive-2022.09.03.tar.xz.drv'
deleting '/nix/store/7vycjl4qsz42ba9fsx076ps645462fpv-gettext-0.21.drv'
deleting '/nix/store/7j1109rjq45knvrq81y5bd3x05ivmz1n-expat-2.5.0.tar.xz.drv'
deleting '/nix/store/5bnj2hxrjm69i7wr90jb27p7cchiyqkm-gettext-0.21.tar.gz.drv'
deleting '/nix/store/3qhamx7gvfz4c1jhxr2amcsshxj2lr6h-automake-1.16.5.tar.xz.drv'
deleting '/nix/store/3fqy66lzrcdrbrhchq34fj393n5szb8a-libffi-3.4.4.tar.gz.drv'
deleting '/nix/store/27r0bvzg617g1r4zmnhwcyvf5cf4h9v5-28-cve-2022-0529-and-cve-2022-0530.patch.drv'
deleting '/nix/store/1w8fxnpg6mjk7pqcvp8wkijkvvj9s8lf-CVE-2019-13232-1.patch.drv'
deleting '/nix/store/0nasry8aip2d0zc115ia4jwhk10b8mh8-06-initialize-the-symlink-flag.patch.drv'
deleting '/nix/store/0fbnd1w0qxkdl8ap0ddx5gx971km2kgm-libtool-2.4.7.tar.gz.drv'
deleting '/nix/store/0hjmcwg5rpq8kk87ngdvbkrm6zh8k3ll-curl-7.87.0.drv'
deleting '/nix/store/5pwp4fnx421hp06lyf8xgypzm0gpk2h5-libkrb5-1.20.1.drv'
deleting '/nix/store/3sk2f10sbr79sv90d0m7b45fghzdm3n6-libssh2-1.10.0.drv'
deleting '/nix/store/1wzbvkhygdyzlhng28f4gcq8dy1ydy9d-openssl-3.0.8.drv'
deleting '/nix/store/lzmcfv2m4ripknpvbsv8wcg1ik1kif4h-use-etc-ssl-certs.patch'
deleting '/nix/store/h22pscjl75ph7q0zcsn8gqc7qlizv6z5-mirrors-list.drv'
deleting '/nix/store/ng54snz2v2y7jjz21d22jz9wpmvkz5lp-nghttp2-1.51.0.drv'
deleting '/nix/store/l6wwzr6szj2xfhq4cvyvmpl9vqr9c9m7-pkg-config-wrapper-0.29.2.drv'
deleting '/nix/store/lq08qqqyrxyp2cmrkhpzwn5sjdq2z476-pkg-config-0.29.2.drv'
deleting '/nix/store/1x122mg8wkrla3sbzy00366aabzqxhhx-locales-setup-hook.sh.drv'
deleting '/nix/store/2cq4hsc1v8ylccspw8351r72s56w1fia-CVE-2015-7697.diff'
deleting '/nix/store/p8q62lpf4m88kp2s16apbibdagcrw93l-nghttp2-1.51.0.tar.bz2.drv'
deleting '/nix/store/i3ywgfm3bgqvy7n0iflbf5z11v5zwy5j-keyutils-1.6.3.drv'
deleting '/nix/store/d9b2qrrq32jzdsdx4y33inzrra5n5z5n-CVE-2014-8140.diff'
deleting '/nix/store/p46prhgmv7ibjh9igfkbc6zfxbbi6sk5-dont-hardcode-cc.patch'
deleting '/nix/store/0k8d71xza3275szx7n5r7vhj9m3ip0hh-glibc-iconv-2.35.drv'
deleting '/nix/store/6np2acjv1nxcg0xzsv9a76wyrpxznkna-CVE-2014-8141.diff'
deleting '/nix/store/sdj5d4lx170pffyvwwj2iac36c0avwhz-source'
deleting '/nix/store/p55a764pi2f4kkx3adb43bxb2dnb4z6r-CVE-2018-18384.patch'
deleting '/nix/store/1k1wn8807yizgz3ghnbd4k6zsc0dzfkr-CVE-2014-9913.patch'
deleting '/nix/store/vizgh9p9qc3v9f4c17f10smqb7zg90sg-libssh2-1.10.0.tar.gz.drv'
deleting '/nix/store/hs1p8pfraz8vhz0kj19wqd3jix6wcfh0-make-shell-wrapper-hook.drv'
deleting '/nix/store/sswjgxdkphxj9f3wn0q30nzz8cdkgbsm-die-hook.drv'
deleting '/nix/store/17yjw33i747j1v1md1fgi4s2i0hy8klq-locales-builder.sh'
deleting '/nix/store/ycwm35msmsdi2qgjax1slmjffsmwy8am-write-mirror-list.sh'
deleting '/nix/store/k51cqpa0nzhs09bnyv69liik1d86l9ww-coreutils-9.1.drv'
deleting '/nix/store/68bf2pfyxwfq1m0fzx4fqjw41vdkaard-perl-5.36.0.drv'
deleting '/nix/store/cklrwbwi889pp2fdsswdjvn12sdy5i5j-openssl-disable-kernel-detection.patch'
deleting '/nix/store/0ri4w1lxx2s1mazff8y7h5q2pkixqz7s-python-setup-hook.sh.drv'
deleting '/nix/store/rdkdki1f24q8mqgnbsyk7gmh28c027ks-CVE-2014-8139.diff'
deleting '/nix/store/6zqn6w9rwkgfa6z1hpagnh5xhz2dag6m-CVE-2015-7696.diff'
deleting '/nix/store/pdcj2chp5c2gvm2jc3shbajfc62kbx1i-CVE-2014-9636.diff'
deleting '/nix/store/97d26l91h0db8h0qkmhxwi5d8shrilv6-CVE-2016-9844.patch'
deleting '/nix/store/151w9968gazbxf52ijf2y7fh4f4sphi0-stdenv-linux.drv'
deleting '/nix/store/88a62ypvi5xpa3m8znhrl6l7jc307i8r-conf-symlink.patch'
deleting '/nix/store/w1c7ihmg8ykp1ddpjldq7i5407cqsbz4-7.79.1-darwin-no-systemconfiguration.patch'
deleting '/nix/store/qggqyqdmdsqdv4k19sa6rjxcq887930p-krb5-1.20.1.tar.gz.drv'
deleting '/nix/store/lkkzb8ybhp5jjgvb04b13n68i51lxwd9-keyutils-1.6.3.tar.gz.drv'
deleting '/nix/store/8djp1rizc1dblv8svnb0mpa0c3lwvc17-drop-comments.patch'
deleting '/nix/store/z0sjg5c7g1iqb4x5vyiqcm5n68mb5x5p-0001-Remove-unused-function-after_eq.patch'
deleting '/nix/store/57620l1168piiia2bmmsxxhh7sjb2n40-builder.sh'
deleting '/nix/store/bbq4vd80lj8m6qx3y0hl55aspir0diik-curl-7.87.0.tar.bz2.drv'
deleting '/nix/store/sq4h6bqjx12v9whvm65pjss25hg1538q-nix-ssl-cert-file.patch'
deleting '/nix/store/zrh6il3gp9xa58ldg40d57kwgsvljyb1-openssl_add_support_for_libressl_3_5.patch'
deleting '/nix/store/p0wk6m9z4r48b7dph66mkhc7y7kxhr5i-source'
deleting '/nix/store/py1pdcfhj845cyam6ndq0b2mh02zk4my-openssl-3.0.8.tar.gz.drv'
deleting unused links...
note: currently hard linking saves -0.00 MiB
95 store paths deleted, 151.50 MiB freed

Uh oh that still deleted stuff, are you sure it's alright?

I'm sure it's finnneee:

$ cowsay "Still kickin'"
 _______________
< Still kickin' >
 ---------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Good, but.. where is it? How does it work now?

Oh, it's not a big mystery:

$ which cowsay
/home/amos/.nix-profile/bin/cowsay

$ ls -lhA $(which cowsay)
-r-xr-xr-x 1 root root 8.9K Jan  1  1970 /home/amos/.nix-profile/bin/cowsay

$ readlink -f $(which cowsay)
/nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/bin/cowsay

It's "simply" a symbolic link to its location in /nix/store!

It's also a Perl program, so it starts with a hash-pling:

$ head $(which cowsay)
#!/nix/store/bja5fzkjjn26zz742pgkvpm5dq3girv9-perl-5.36.0/bin/perl

##
## Cowsay
##

use Cwd 'abs_path';
use File::Basename;
use File::Find;
use Getopt::Std;

...which points to the nix-installed perl, and not /usr/bin/perl, which also exists.

If we were using NixOS as our Linux distribution, there would be no /usr/bin/perl, only the nix-installed one, but I'm going for the pragmatic middle-of-the-road solution where you're cozy and/or forced into using a more mainstream Linux distribution, like Ubuntu or Fedora, and you want to sprinkle a bit of nix on top.

nix the language

nix profiles are neat, and we can list what's installed in them with:

$ nix profile list
0 flake:nixpkgs#legacyPackages.x86_64-linux.cowsay github:NixOS/nixpkgs/63b5955814db30d2e2ff7157aaa5665b502ed2f4#legacyPackages.x86_64-linux.cowsay /nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0 /nix/store/qgsl9bd6vvnpjaxhdnzg0xmycg3652c0-cowsay-3.7.0-man

...but it's not what we want for developing catscii. We want anyone to be able to jump in and develop the project, so if the README said "run nix profile install (a bunch of things)", it wouldn't be much better than saying "run apt install (another bunch of things)".

Ideally, we want the list of things catscii needs to live directly in the repository, as a file.

And we can do that! In several ways, but again, let's focus on the new, experimental, slightly-broken-at-times, opt-in stuff: flakes.

And my experience with nix has been a lot of "just copy paste this", and having to deconstruct it later to understand what the heck is going on.

So, let's start small.

Nix is a language. We can get a REPL!

$ nix repl
Welcome to Nix 2.13.2. Type :? for help.

nix-repl> 1 + 2
3

We can have code in .nix files:

󱄅# in sample.nix
[ 1 2 ] ++ [ 3 4 ]

$ nix eval --file ./sample.nix
[ 1 2 3 4 ]

Cool bear's hot tip

Before going any further: if you're going to be running those locally as we go (and you should), here's the recommended set-up with VS Code.

You want to install a formatter and an LSP server:

$ nix profile install nixpkgs#nixpkgs-fmt nixpkgs#rnix-lsp

Then grab the Nix IDE VS Code extension.

If you use VS Code with remote ssh, it's possible Nix IDE complains about either of nixpkgs-fmt / rnix-lsp not being in $PATH (it'll fail with ENOENT).

If killing the remote vscode server isn't enough, you can always completely shut down VS Code (it caches too much stuff for its own good) and reboot the VM: that's why we have VMs!

With that, you should have syntax highlighting, and you can enable "formatting on save" with this in your user settings:

{
  "[nix]": {
    "editor.formatOnSave": true,
    "editor.defaultFormatter": "jnoortheen.nix-ide"
  },
}

All good? Good.

nix has sets, which you may be used to calling "dictionaries", "objects" or "maps", depending how your brain is wired. Anyway they associate keys and values:

󱄅{
  list =
    [ 1 2 ] ++ [ 3 4 ];
  addition = 1 + 2;
}

$ nix eval --file ./sample.nix
{ addition = 3; list = [ 1 2 3 4 ]; }

$ nix eval --file ./sample.nix addition
3

$ nix eval --file ./sample.nix list
[ 1 2 3 4 ]

And it has functions, which are called lambdas in some contexts:

󱄅{
  add_two = x: x + 2;
}

Unfortunately, it's hard to call it with nix eval due to an outstanding bug - you didn't get warned because we opted into experimental stuff earlier, but nix eval is part of that experimental stuff and is a little half-baked.

We can have it print the lambda itself:

$ nix eval --file ./sample.nix add_two
<LAMBDA>

But to actually call it, we have to get creative. Instead of using --file, we can use --expr to evaluate an expression, and use import to import our .nix file:

$ nix eval --expr 'import ./sample.nix'
error: access to absolute path '/home/amos/catscii/sample.nix' is forbidden in pure eval mode (use '--impure' to override)
(use '--show-trace' to show detailed location information)

Well, we can do that with --impure:

$ nix eval --impure --expr 'import ./sample.nix'
{ add_two = <LAMBDA>; }

And then, well, we can call it:

$ nix eval --impure --expr '(import ./sample.nix).add_two 3'
5

Parentheses? What for?

Not to call the function, that's for sure! This isn't a lisp. Here they're just used for grouping/priority: we want to access the .add_two property of the results of import ./sample.nix.

Note that ./sample.nix isn't a string, it's a path: that's a distinct type in nix. We can't import a string:

$ nix eval --impure --expr '(import "./sample.nix").add_two 3'
error: string './sample.nix' doesn't represent an absolute path

       at «string»:1:2:

            1| (import "./sample.nix").add_two 3
             |  ^
(use '--show-trace' to show detailed location information)

Okay, okay, this feels weird, but I'm assimilating.

Oh we're not done! What if we want to bind the result of importing ./sample.nix to something before using it by name?

Since nix isn't imperative, we don't really have a concept of statements that are evaluated in order.

But there's ways to work around that.

For simplicity, let's make another nix file that'll import sample.nix, so we don't have to deal with --impure --expr.

󱄅# in caller.nix
(import ./sample.nix).add_two 5

$ nix eval --file caller.nix
7

Okay, so, I was saying: binding it to something. Well we can't have "successive statements", but we can have... sets!

󱄅# in caller.nix
{
  sample = import ./sample.nix;
  result = sample.add_two 5;
}

Unfortunately, that doesn't work:

$ nix eval -f caller.nix
error: undefined variable 'sample'

       at /home/amos/catscii/caller.nix:4:12:

            3|   sample = (import ./sample.nix).add_two 5;
            4|   result = sample.add_two 5;
             |            ^
            5| }

Because by default, sets are non-recursive. We can fix that with rec:

󱄅# in caller.nix
rec {
  sample = import ./sample.nix;
  result = sample.add_two 5;
}

$ nix eval --file caller.nix
{ result = 7; sample = { add_two = <LAMBDA>; }; }

This is a little untidy though: we don't really need "sample" to be part of our output.

So instead, we can use a let:

󱄅# in caller.nix
let
  sample = import ./sample.nix;
in
sample.add_two 5

$ nix eval -f caller.nix
7

Okay, we got pretty far already! So, let's summarize:

() are just like in C/Rust/whatever: they mean "evaluate that first"
{} defines a set/object/dictionary/key-value map. it has key = value; inside (each item ends with a semicolon)
[] defines a list/array: it has 1 2 3 inside (space-separated)
foo.bar accesses property bar of set foo
let lets you bind some names and use them in the in block later, which is preferred to using a recursive set.

You haven't really explained why recursive sets are bad though.

Ah. Easy!

󱄅# in caller.nix
rec {
  a = b;
  b = c;
  c = a;
}

$ nix eval --file caller.nix
error: infinite recursion encountered

       at /home/amos/catscii/caller.nix:4:7:

            3|   a = b;
            4|   b = c;
             |       ^
            5|   c = a;
(use '--show-trace' to show detailed location information)

Oh. I see. Given that, I suppose order doesn't matter in sets?

You are correct:

󱄅# in caller.nix
rec {
  result = sample.add_two 5;
  # this comes after we've used `sample`, but that doesn't matter.
  sample = import ./sample.nix;
}

$ nix eval --file caller.nix
{ result = 7; sample = { add_two = <LAMBDA>; }; }

nix derivations

nix language is all well and good (...depending), but by itself it doesn't do much.

The whole packagement system (sic.) built around it is the juicy bit.

And if you look at any nix tutorial, they'll show you how to use mkDerivation. So of course, we won't do that - we'll go one level deeper.

So let's remove all our nix files so far:

$ rm *.nix
(cut)

And create a default.nix file with a simple derivation:

󱄅# in default.nix
derivation {
  name = "simple";
  builder = "${(import <nixpkgs> {}).bash}/bin/bash";
  args = [ "-c" "echo foo > $out" ];
  src = ./.;
  system = builtins.currentSystem;
}

There's a lot going on here, so before we try to do something with this file, let's make sure we understand what's going.

We're calling a function named derivation, passing a set to it. The value for name is a string, the value for args is a list, the value for src is a path.

What is system set to? Let's find out:

$ nix eval --expr 'builtins.currentSystem'
error: attribute 'currentSystem' missing

       at «string»:1:1:

            1| builtins.currentSystem
             | ^

Woops, that didn't work. Does nix eval not give you builtins?

$  nix eval --raw --expr 'builtins.toXML { a = "b"; }'
<?xml version='1.0' encoding='utf-8'?>
<expr>
  <attrs>
    <attr name="a">
      <string value="b" />
    </attr>
  </attrs>
</expr>

Oh, it does. It just doesn't have currentSystem.

The reason nix eval didn't recognize builtins.currentSystem is because it operates in a pure evaluation mode by default. In this mode, certain builtins, like currentSystem, aren't available. But there's another way to access these builtins.

How about the REPL?

$ nix repl
Welcome to Nix 2.13.2. Type :? for help.

nix-repl> builtins.currentSystem
"x86_64-linux"

nix-repl>

That one has it, neat. The nix repl operates in an impure mode by default, which is why builtins.currentSystem is accessible there. If you want to use nix eval and see the value of currentSystem, you can add the --impure flag:

󱄅$ nix eval --impure --expr 'builtins.currentSystem'
"aarch64-darwin"

This flag switches the evaluation to impure mode, making currentSystem available.

The nastiest part of our default.nix file is definitely this, though:

󱄅# in default.nix
derivation {
  name = "simple";
  #              👇👇👇👇👇👇👇
  builder = "${(import <nixpkgs> {}).bash}/bin/bash";
  args = [ "-c" "echo foo > $out" ];
  src = ./.;
  system = builtins.currentSystem;
}

Luckily, we know just enough nix to decipher what that means. Well, we didn't know "${foo}" did string interpolation, but we do now.

We also didn't know what <nixpkgs> is a magic path, but, yeah:

$ nix repl
Welcome to Nix 2.13.2. Type :? for help.

nix-repl> <nixpkgs>
/nix/var/nix/profiles/per-user/root/channels/nixpkgs

And so, if we follow logically, this must mean we build a string out of the "bash" property of whatever the result of calling import <nixpkgs> with an empty set gives us.

Because, yeah, import <nixpkgs> evaluates to a lambda:

nix-repl> import <nixpkgs>
«lambda @ /nix/store/44vlp0hzcir0j7hmg48g8fmj321igpw1-nixpkgs/nixpkgs/pkgs/top-level/impure.nix:14:1»

So we must call it. But I strongly recommend not typing import <nixpkgs> {} in your REPL, because it'll evaluate everything in there. At the time of this writing, there's over 80K (eighty thousand) packages in there, and luckily, they warned fools like me:

nix-repl> import <nixpkgs> {}
{ AAAAAASomeThingsFailToEvaluate = «error: error: Please be informed that this pseudo-package is not the only part of
       Nixpkgs that fails to evaluate. You should not evaluate entire Nixpkgs
       without some special measures to handle failing packages, like those taken
       by Hydra.»; AMB-plugins = «derivation /nix/store/4fpqhkcsm53zlr5lgwb6hm7zjif6h5ci-AMB-plugins-0.8.1.drv»; ArchiSteamFarm = «derivation /nix/store/zg5wnx2d7lcfdir0dlb64vw1s2jkydaf-archisteamfarm-5.4.1.11.drv»; AusweisApp2 = «derivation /nix/store/c4cyfm3q2f91dq26nv5vspfnlvr7bpnq-AusweisApp2-1.26.2.drv»; BeatSaberModManager = «derivation /nix/store/b3yv9v0cyckmhgpl8b3bplk5ww9l8asi-BeatSaberModManager-0.0.4.drv»; CHOWTapeModel = «derivation /nix/store/cyap5ndhw5h5x21jllhr6vprxfdj7f06-CHOWTapeModel-2.10.0.drv»; ChowCentaur = «derivation /nix/store/mj1i3vs60asnrzl637b6iznivp3sqvqg-ChowCentaur-1.4.0.drv»; ChowKick = ^Cerror: interrupted by the user
«derivation

Accessing a single field, though, is fine:

nix-repl> (import <nixpkgs> {}).bash
«derivation /nix/store/0hnjp6s8k71xm62157v37zg3qzwvl8lx-bash-5.2-p15.drv»

So what we've learned is that nix has lazy evaluation, which, if you've made it this far, you've probably already heard about, because it's the one thing nobody will shut up about.

(But in their defense, that's pretty important).

Now to do the same string interpolation as we have in default.nix, but in the REPL, to see what it gives us:

nix-repl> "${(import <nixpkgs> {}).bash}/bin/bash"
"/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash"

See, that's interesting.

Because it just so happens that this path does exist right now:

$ file /nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash
/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/ld-linux-x86-64.so.2, BuildID[sha1]=ca0fcdc9d631f4ccf8bb36cdd843b7d2b7b9390f, for GNU/Linux 3.10.0, not stripped

But if we try with another package:

$ nix repl
Welcome to Nix 2.13.2. Type :? for help.

nix-repl> "${(import <nixpkgs> {}).dive}"
"/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0"

(pressing Ctrl-D to get back to bash)

$ file /nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0
/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0: cannot open `/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0' (No such file or directory)

It doesn't exist.

And that's another thing you may have heard The Nix People go on about: the hash part is great, actually: because all inputs are statically defined, it can tell where a derivation is going to go (what its hash is going to be) without actually realizing it.

If we actually want to build dive, we first have to instantiate it:

$ nix-instantiate --expr '(import <nixpkgs> {}).dive'
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/30r459s85cx70nprlcxyar9sa13qg9yr-dive-0.10.0.drv

We get a warning about the garbage collector – fair enough! We're just playing around.

In return, we got the path of a .drv file, which I keep reading as "driver" but I suspect means "derivation". This .drv file is human-readable (go look) but not really human-friendly.

Let's use nix show /path/to/file.drv instead, which gives us JSON:

{
  "/nix/store/30r459s85cx70nprlcxyar9sa13qg9yr-dive-0.10.0.drv": {
    "args": [
      "-e",
      "/nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh"
    ],
    "builder": "/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash",
    "env": {
      "CGO_ENABLED": "1",
      "GO111MODULE": "on",
      "GOARCH": "amd64",
      "GOFLAGS": "-mod=vendor -trimpath",
      "GOOS": "linux",
      "__structuredAttrs": "",
      "buildInputs": "/nix/store/d7g939y4lqixk8rfm385vwa7gc12rmck-btrfs-progs-6.1.2 /nix/store/h4fdrla07dzi0cp9fah3a9vdhfrih618-gpgme-1.18.0-dev /nix/store/9cw3b5nixp1xv7k3prj28lnxyfkj7k2f-lvm2-2.03.18-dev",
      "buildPhase": "runHook preBuild\n\nexclude='\\(/_\\|examples\\|Godeps\\|testdata'\nif [[ -n \"$excludedPackages\" ]]; then\n  IFS=' ' read -r -a excludedArr <<<$excludedPackages\n  printf -v excludedAlternates '%s\\\\|' \"${excludedArr[@]}\"\n  excludedAlternates=${excludedAlternates%\\\\|} # drop final \\| added by printf\n  exclude+='\\|'\"$excludedAlternates\"\nfi\nexclude+='\\)'\n\nbuildGoDir() {\n  local cmd=\"$1\" dir=\"$2\"\n\n  . $TMPDIR/buildFlagsArray\n\n  declare -a flags\n  flags+=($buildFlags \"${buildFlagsArray[@]}\")\n  flags+=(${tags:+-tags=})\n  flags+=(${ldflags:+-ldflags=\"$ldflags\"})\n  flags+=(\"-p\" \"$NIX_BUILD_CORES\")\n\n  if [ \"$cmd\" = \"test\" ]; then\n    flags+=(-vet=off)\n    flags+=($checkFlags)\n  fi\n\n  local OUT\n  if ! OUT=\"$(go $cmd \"${flags[@]}\" $dir 2>&1)\"; then\n    if ! echo \"$OUT\" | grep -qE '(no( buildable| non-test)?|build constraints exclude all) Go (source )?files'; then\n      echo \"$OUT\" >&2\n      return 1\n    fi\n  fi\n  if [ -n \"$OUT\" ]; then\n    echo \"$OUT\" >&2\n  fi\n  return 0\n}\n\ngetGoDirs() {\n  local type;\n  type=\"$1\"\n  if [ -n \"$subPackages\" ]; then\n    echo \"$subPackages\" | sed \"s,\\(^\\| \\),\\1./,g\"\n  else\n    find . -type f -name \\*$type.go -exec dirname {} \\; | grep -v \"/vendor/\" | sort --unique | grep -v \"$exclude\"\n  fi\n}\n\nif (( \"${NIX_DEBUG:-0}\" >= 1 )); then\n  buildFlagsArray+=(-x)\nfi\n\nif [ ${#buildFlagsArray[@]} -ne 0 ]; then\n  declare -p buildFlagsArray > $TMPDIR/buildFlagsArray\nelse\n  touch $TMPDIR/buildFlagsArray\nfi\nif [ -z \"$enableParallelBuilding\" ]; then\n    export NIX_BUILD_CORES=1\nfi\nfor pkg in $(getGoDirs \"\"); do\n  echo \"Building subPackage $pkg\"\n  buildGoDir install \"$pkg\"\ndone\nrunHook postBuild\n",
      "builder": "/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash",
      "checkPhase": "runHook preCheck\n# We do not set trimpath for tests, in case they reference test assets\nexport GOFLAGS=${GOFLAGS//-trimpath/}\n\nfor pkg in $(getGoDirs test); do\n  buildGoDir test \"$pkg\"\ndone\n\nrunHook postCheck\n",
      "cmakeFlags": "",
      "configureFlags": "",
      "configurePhase": "runHook preConfigure\n\nexport GOCACHE=$TMPDIR/go-cache\nexport GOPATH=\"$TMPDIR/go\"\nexport GOPROXY=off\nexport GOSUMDB=off\ncd \"$modRoot\"\nrm -rf vendor\ncp -r --reflink=auto /nix/store/1j8i1f7abvbd5p4z9ias3l914d6vbbv7-dive-0.10.0-go-modules vendor\n\n\nrunHook postConfigure\n",
      "depsBuildBuild": "",
      "depsBuildBuildPropagated": "",
      "depsBuildTarget": "",
      "depsBuildTargetPropagated": "",
      "depsHostHost": "",
      "depsHostHostPropagated": "",
      "depsTargetTarget": "",
      "depsTargetTargetPropagated": "",
      "disallowedReferences": "/nix/store/4z6hhnxkmd34kp0i2p0a7nz7idvkc753-go-1.19.5",
      "doCheck": "1",
      "doInstallCheck": "",
      "enableParallelBuilding": "1",
      "enableParallelChecking": "1",
      "installPhase": "runHook preInstall\n\nmkdir -p $out\ndir=\"$GOPATH/bin\"\n[ -e \"$dir\" ] && cp -r $dir $out\n\nrunHook postInstall\n",
      "ldflags": "-s -w -X main.version=0.10.0",
      "mesonFlags": "",
      "name": "dive-0.10.0",
      "nativeBuildInputs": "/nix/store/4z6hhnxkmd34kp0i2p0a7nz7idvkc753-go-1.19.5 /nix/store/lrb01sby9jg4hfqhr32s28igg77bhcwr-pkg-config-wrapper-0.29.2",
      "out": "/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0",
      "outputs": "out",
      "patches": "/nix/store/l8vidjbfpd8r5llai1zgby1p1lnsdpjl-fe9411c414418d839a8638bb9a12ccfc892b5845.patch",
      "pname": "dive",
      "propagatedBuildInputs": "",
      "propagatedNativeBuildInputs": "",
      "src": "/nix/store/p4zd2ls3yy4w6j430f5lcykkkc14f0yc-source",
      "stdenv": "/nix/store/b09v23lirgvci3wzszh22mbkdfj0h0yq-stdenv-linux",
      "strictDeps": "1",
      "system": "x86_64-linux",
      "version": "0.10.0"
    },
    "inputDrvs": {
      "/nix/store/0hnjp6s8k71xm62157v37zg3qzwvl8lx-bash-5.2-p15.drv": [
        "out"
      ],
      "/nix/store/1ps4mh2yih85fx76imq77q36whi0g1pn-go-1.19.5.drv": [
        "out"
      ],
      "/nix/store/c31f6dgzjjyn84d16ybaap9zr1j2l6ql-source.drv": [
        "out"
      ],
      "/nix/store/kv8gamr3m8s7pbpzqda5fm3b6z3i6cqf-dive-0.10.0-go-modules.drv": [
        "out"
      ],
      "/nix/store/l98jn7564nrzb7a5ik6iqqfx0vrmfwaz-gpgme-1.18.0.drv": [
        "dev"
      ],
      "/nix/store/likh4j2kkp2fparm884aywk4ma0yxwvl-btrfs-progs-6.1.2.drv": [
        "out"
      ],
      "/nix/store/nsb0bznfxnfyxnlacd4qzs4s1sz9jdj0-lvm2-2.03.18.drv": [
        "dev"
      ],
      "/nix/store/qf8nz4ixc1b5kvqd2h50mbxgj23n0kij-fe9411c414418d839a8638bb9a12ccfc892b5845.patch.drv": [
        "out"
      ],
      "/nix/store/r2h029bx2fbyxxj84s5hf1abp2vfkah2-stdenv-linux.drv": [
        "out"
      ],
      "/nix/store/xfqq6rb3i2r2n65l64c4s0lm2y5bcbz4-pkg-config-wrapper-0.29.2.drv": [
        "out"
      ]
    },
    "inputSrcs": [
      "/nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh"
    ],
    "outputs": {
      "out": {
        "path": "/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0"
      }
    },
    "system": "x86_64-linux"
  }
}

This specifies everything nix needs to realize that derivation.

But at this point, the output path still doesn't exist:

$ file /nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.
/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.: cannot open `/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.' (No such file or directory)

If we want it, we have to realize it:

$ nix-store --realize /nix/store/30r459s85cx70nprlcxyar9sa13qg9yr-dive-0.10.0.drv
these 4 paths will be fetched (3.06 MiB download, 11.82 MiB unpacked):
  /nix/store/66fr4mlfnwym5dw4bx61xs7nb81q58cb-tzdata-2022g
  /nix/store/ih35ndii8g86q4hlr2f1ma0z14ga4d0r-mailcap-2.1.53
  /nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0
  /nix/store/nl44nfqyyc7d5bw58v8g5ac2wcvmfp98-iana-etc-20221107
copying path '/nix/store/nl44nfqyyc7d5bw58v8g5ac2wcvmfp98-iana-etc-20221107' from 'https://cache.nixos.org'...
copying path '/nix/store/ih35ndii8g86q4hlr2f1ma0z14ga4d0r-mailcap-2.1.53' from 'https://cache.nixos.org'...
copying path '/nix/store/66fr4mlfnwym5dw4bx61xs7nb81q58cb-tzdata-2022g' from 'https://cache.nixos.org'...
copying path '/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0' from 'https://cache.nixos.org'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0

And now we have it!

$ /nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0/bin/dive --version
dive 0.10.0

Of course, it'll be garbage-collected, just like they warned us:

$ nix-collect-garbage
(cut)

$ /nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0/bin/dive --version
bash: /nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0/bin/dive: No such file or directory

Now onto our own derivation. It's not in <nixpkgs>, it's in a default.nix file.

We can evaluate it, just like any other .nix file:

$ nix eval --file default.nix
{ all = [ «repeated» ]; args = [ "-c" "echo foo > $out" ]; builder = "/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash"; drvAttrs = { args = «repeated»; builder = "/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash"; name = "simple"; src = /home/amos/catscii; system = "x86_64-linux"; }; drvPath = "/nix/store/zvknsw43ljqyw6ry9w5snzgscckvcp2m-simple.drv"; name = "simple"; out = «repeated»; outPath = "/nix/store/fv9v381if8y7kl5iajvskl7cawz38yri-simple"; outputName = "out"; src = /home/amos/catscii; system = "x86_64-linux"; type = "derivation"; }

That's not terribly friendly. Normally we could ask for JSON with --json, but since this is a value of type "derivation", it just shows the output path instead:

$ nix eval --json --file default.nix
"/nix/store/fv9v381if8y7kl5iajvskl7cawz38yri-simple"

So, instead, I'll just manually format what nix eval --file gave us:

󱄅{
  all = [ «repeated» ];
  args = [ "-c" "echo foo > $out" ];
  builder = "/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash";
  drvAttrs = {
    args = «repeated»;
    builder = "/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash";
    name = "simple";
    src = /home/amos/catscii;
    system = "x86_64-linux";
  };
  drvPath = "/nix/store/zvknsw43ljqyw6ry9w5snzgscckvcp2m-simple.drv";
  name = "simple";
  out = «repeated»;
  outPath = "/nix/store/fv9v381if8y7kl5iajvskl7cawz38yri-simple";
  outputName = "out";
  src = /home/amos/catscii;
  system = "x86_64-linux";
  type = "derivation";
}

So, essentially, a derivation seems like a spicy set. Okay!

Just like dive, we can instantiate our derivation to generate the .drv file:

$ nix-instantiate .
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/zvknsw43ljqyw6ry9w5snzgscckvcp2m-simple.drv

Note that whereas to instantiate a nixpkgs package, we had to use --expr, here we can just pass a path: the current directory, ., which contains a default.nix file (that naming is not by accident).

Now that we have the .drv path, we can realize it with nix-store:

$ nix-store -r /nix/store/zvknsw43ljqyw6ry9w5snzgscckvcp2m-simple.drv
this derivation will be built:
  /nix/store/zvknsw43ljqyw6ry9w5snzgscckvcp2m-simple.drv
this path will be fetched (0.43 MiB download, 1.60 MiB unpacked):
  /nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15
copying path '/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15' from 'https://cache.nixos.org'...
building '/nix/store/zvknsw43ljqyw6ry9w5snzgscckvcp2m-simple.drv'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/fv9v381if8y7kl5iajvskl7cawz38yri-simple

And now, it's in our store!

$ cat /nix/store/fv9v381if8y7kl5iajvskl7cawz38yri-simple
foo

I didn't really explain what our derivation actually did. Well, it ran the bash builder with arguments -c "echo foo > $out". $out is an environment variable that contains the path where the derivation should generate their output.

Now for a bit of good news: we went much deeper than needed!

First off, we can clean up our default.nix - we know how to use let, so we can do this:

󱄅# in default.nix
let
  pkgs = (import <nixpkgs> { });
in
derivation {
  name = "simple";
  builder = "${pkgs.bash}/bin/bash";
  args = [ "-c" "echo foo > $out" ];
  src = ./.;
  system = builtins.currentSystem;
}

Alternatively, we can use the with syntax, which.. to me at least, "adds a set to the local scope". If we do with something, we can refer to any property of something without doing something.property - we can just do property.

So in this version, we no longer need to refer to pkgs.bash, we can just refer to bash:

󱄅# in default.nix
with
(import <nixpkgs> { });
derivation
{
  name = "simple";
  builder = "${bash}/bin/bash";
  args = [ "-c" "echo foo > $out" ];
  src = ./.;
  system = builtins.currentSystem;
}

Cool bear's hot tip

Bringing all of nixpkgs in scope is an anti-pattern. Using let and selecting what to inherit is preferred.

Secondly, we can use some bash powers to do both instantiation and realization in one command:

$ nix-store -r $(nix-instantiate .)
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
this derivation will be built:
  /nix/store/5cpc0sjkzgn4xrc75a4brzpl6kxdnaxd-simple.drv
building '/nix/store/5cpc0sjkzgn4xrc75a4brzpl6kxdnaxd-simple.drv'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/mz5y3xxmas0y4595lk8w5kvpy5k9xgd3-simple

$ cat /nix/store/mz5y3xxmas0y4595lk8w5kvpy5k9xgd3-simple
foo

But let's garbage-collect for a minute..

$ nix-collect-garbage
finding garbage collector roots...
(cut)

You know what does both instantiation and realization in one step?

That's right! It's nix-build!

What do you mean "that's right"? That's the first time you mention it?

Welllll maybe folks have used nix before on the surface, I don't know.

Anyway:

$ nix-build
this derivation will be built:
  /nix/store/5cpc0sjkzgn4xrc75a4brzpl6kxdnaxd-simple.drv
this path will be fetched (0.43 MiB download, 1.60 MiB unpacked):
  /nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15
copying path '/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15' from 'https://cache.nixos.org'...
building '/nix/store/5cpc0sjkzgn4xrc75a4brzpl6kxdnaxd-simple.drv'...
/nix/store/mz5y3xxmas0y4595lk8w5kvpy5k9xgd3-simple

$ cat /nix/store/mz5y3xxmas0y4595lk8w5kvpy5k9xgd3-simple
foo

Okay, cool, we know everything about nix!

Well.. no? What about mkDerivation?

Oh, right. So derivation is low-level and one hardly ever uses it.

A real-world mkDerivation

Instead, mkDerivation is often used.

And this time, I think we're ready to look at a bit of real-world nix. It's something I wrote myself, for my video platform.

In this incarnation of my video platform, I use shaka-packager to package videos as MPEG-DASH (fragmented MP4 + an .mpd manifest).

It's distributed as static binaries already, so the nix bit is actually fairly simple!

󱄅# in default.nix
with
(import <nixpkgs> { });
let
  version = "2.6.1";
in
stdenv.mkDerivation
{
  name = "shaka-packager-${version}";

  # https://nixos.wiki/wiki/Packaging/Binaries
  src = pkgs.fetchurl {
    url =
      "https://github.com/shaka-project/shaka-packager/releases/download/v${version}/packager-linux-x64";
    sha256 = "sha256-MoMX6PEtvPmloXJwRpnC2lHlT+tozsV4dmbCqweyyI0=";
  };

  dontUnpack = true;
  sourceRoot = ".";

  installPhase = ''
    install -m755 -D $src $out/bin/shaka-packager
  '';

  meta = with lib; {
    homepage = "https://shaka-project.github.io/shaka-packager/html/";
    description =
      "Media packaging framework for VOD and Live DASH and HLS applications";
    platforms = platforms.x86_64-linux;
  };
}

We can build it:

$ nix-build
these 2 derivations will be built:
  /nix/store/4gllgh3bdxfpwj4hi2bcqa5ly390jz35-packager-linux-x64.drv
  /nix/store/668fy2xd3lc8anmqd6jmhk2a1vrzv79x-shaka-packager-2.6.1.drv
these 46 paths will be fetched (64.56 MiB download, 294.29 MiB unpacked):
  /nix/store/09vh6qzbvbavdlcfm4i9kcxj7sv1y2xh-curl-7.87.0-man
  /nix/store/12wczkzi5db1ajbjp4hdyif3v9y5rbi5-xz-5.4.1-bin
  (cut)
copying path '/nix/store/2a6yagz3pa8kiawg5mk2js70f8kwqzqd-bzip2-1.0.8' from 'https://cache.nixos.org'...
copying path '/nix/store/v316awk24xxz2dj84b5qw2vflfpg7if4-diffutils-3.8' from 'https://cache.nixos.org'...
copying path '/nix/store/p699p3vdqq4bn958dxzppbhn7n5k0c0n-ed-1.19' from 'https://cache.nixos.org'...
(cut)
building '/nix/store/4gllgh3bdxfpwj4hi2bcqa5ly390jz35-packager-linux-x64.drv'...

trying https://github.com/shaka-project/shaka-packager/releases/download/v2.6.1/packager-linux-x64
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 6745k  100 6745k    0     0  4487k      0  0:00:01  0:00:01 --:--:-- 16.9M
copying path '/nix/store/l0fvy72hpfdpjjs3dk4112f57x7r3dlm-gcc-wrapper-12.2.0' from 'https://cache.nixos.org'...
copying path '/nix/store/b09v23lirgvci3wzszh22mbkdfj0h0yq-stdenv-linux' from 'https://cache.nixos.org'...
building '/nix/store/668fy2xd3lc8anmqd6jmhk2a1vrzv79x-shaka-packager-2.6.1.drv'...
patching sources
configuring
no configure script, doing nothing
building
no Makefile or custom buildPhase, doing nothing
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/7i41i7mbhabnffykx8ns6wpk0p7y993r-shaka-packager-2.6.1
shrinking /nix/store/7i41i7mbhabnffykx8ns6wpk0p7y993r-shaka-packager-2.6.1/bin/shaka-packager
patchelf: cannot find section '.dynamic'. The input file is most likely statically linked
checking for references to /build/ in /nix/store/7i41i7mbhabnffykx8ns6wpk0p7y993r-shaka-packager-2.6.1...
patchelf: cannot find section '.dynamic'. The input file is most likely statically linked
patching script interpreter paths in /nix/store/7i41i7mbhabnffykx8ns6wpk0p7y993r-shaka-packager-2.6.1
stripping (with command strip and flags -S) in  /nix/store/7i41i7mbhabnffykx8ns6wpk0p7y993r-shaka-packager-2.6.1/bin
/nix/store/7i41i7mbhabnffykx8ns6wpk0p7y993r-shaka-packager-2.6.1

And then run it:

$ /nix/store/7i41i7mbhabnffykx8ns6wpk0p7y993r-shaka-packager-2.6.1/bin/shaka-packager --version
shaka-packager version v2.6.1-634af65-release

mkDerivation provides a bunch of nice things described in the nix derivation docs. It creates a temporary directory for the build (automatically removed after), clears the environment, sets a bunch of relevant environment variables, removes any existing output path, writes stdout and stderr to /nix/var/log/nix, sets timestamps to 00:00:01 1/1/1970 UTC (for reproducible builds), and also:

If the build was successful, Nix scans each output path for references to input paths by looking for the hash parts of the input paths. Since these are potential runtime dependencies, Nix registers them as dependencies of the output paths.

Isn't that interesting!

You can also see in the build log (scroll up a little) that it tries to "patch script interpreter paths". That's right, executables are interpreted also, sort of: dynamically-loaded ones need a.. dynamic loader! The kind we built in making our own executable packer.

A simple dev shell

Before we move on to the more modern stuff, let's note that derivations aren't the only thing a .nix file can evaluate to.

Well... of course it's not: you can have .nix files that evaluate to a single lambda for example, so that you can split out utility functions in other files.

But as far as "top-level" things go, it's mostly derivations and.. dev shells!

Let's remove our default.nix file for now – it doesn't actually build catscii anyway, just shaka-packager, which is completely unrelated.

And let's now create a shell.nix file with this:

󱄅# in shell.nix
{ pkgs ? import <nixpkgs> { } }:
with pkgs;
mkShell
{
  packages = [ flyctl ];
}

Remember flyctl? That's what we use to deploy our app to fly.io. Surely we'll need that.

Let's make sure we don't have it installed in our Ubuntu VM (we shouldn't, it certainly doesn't ship with a default Ubuntu install and we reset our VM earlier):

$ which fly
$ which flyctl

Okay! And now the magic:

$ nix-shell

[nix-shell:~/catscii]$ fly version
flyctl v0.0.456 linux/amd64 Commit: v0.0.456 BuildDate: 1970-01-01T00:00:00Z

[nix-shell:~/catscii]$
(pressing Ctrl-D)
exit

Hurray! We have a dev shell with flyctl in there.

You're reading the Building a Rust service with Nix series.

Here's another article just for you:

Understanding Rust futures by going way too deep

Jul 25, 2021

1 hour 42 min #rust · #async

So! Rust futures! Easy peasy lemon squeezy. Until it's not. So let's do the easy thing, and then instead of waiting for the hard thing to sneak up on us, we'll go for it intentionally.

Cool bear's hot tip

That's all-around solid life advice.

Choo choo here comes the easy part 🚂💨

We make a new project: