51 min
Learning Nix from the bottom up
👋 This page was last updated ~2 years ago. Just so you know.
Remember the snapshot we made allll the way back in Part 1? Now's the time to use it.
Well, make sure you've committed and pushed all your changes, but when you're ready, let's go back in time to before we installed anything catscii-specific in our VM.
This should emulate the experience of a colleague onboarding onto the project well enough!
(I didn't actually use VirtualBox's snapshot feature for this, I actually set up a Ubuntu 22.10 VM on another computer entirely, but the effect should be much the same).
So, from that initial VM state, let's clone our catscii project once again:
$ git clone git@github.com:fasterthanlime/catscii.git Command 'git' not found, but can be installed with: sudo apt install git
Oh, right, we have nothing. Well, let's install git at least:
$ sudo apt update (cut) $ sudo apt install git (cut) $ git clone git@github.com:fasterthanlime/catscii.git Cloning into 'catscii'... The authenticity of host 'github.com (140.82.121.3)' can't be established. ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU. This key is not known by any other names Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added 'github.com' (ED25519) to the list of known hosts. remote: Enumerating objects: 137, done. remote: Counting objects: 100% (137/137), done. remote: Compressing objects: 100% (73/73), done. remote: Total 137 (delta 63), reused 115 (delta 41), pack-reused 0 Receiving objects: 100% (137/137), 48.26 KiB | 484.00 KiB/s, done. Resolving deltas: 100% (63/63), done.
Then let's go in there and try running cargo check:
$ cargo check Command 'cargo' not found, but can be installed with: sudo snap install rustup # version 1.24.3, or sudo apt install cargo # version 0.60.0ubuntu1-0ubuntu3 See 'snap info rustup' for additional versions.
Ah, right, we don't have rustup/rustc/cargo either.
We could keep apt-installing things all day, but that's kinda what we're
trying to avoid here. We'd install rustup, add its env setup to our ~/.bashrc,
then notice we're missing another thing, then another thing (docker, sqlite,
flyctl, etc.), and that's something we can solve, for everyone hacking on
catscii, with nix.
Installing nix
Before we move on, we'll have to install nix.
The nix download page has instructions, let's follow the recommended "multi-user installation".
It needs curl, so, sigh, and:
$ sudo apt install curl (cut)
$ sh <(curl --location https://nixos.org/nix/install) --daemon
The install wizard is very nice and asks if you want to see a detailed list of what it'll do, if it can use sudo (I let it), etc.
After it's done, it lets us know nix won't work in existing shells - I've been
using VSCode SSH Remote so I'm able to just close the current shell with Ctrl-D
(which sends EOF), open a new one with Ctrl+Shift+Backquote, and now I have:
$ nix --version nix (Nix) 2.13.2
Trying out nix-shell
This is not directly related to what we're trying to achieve, but here's something really neat you can do with nix: if you want to try out a package, you just can!
For example, cowsay isn't installed in my VM right now:
$ cowsay "Well hi there" Command 'cowsay' not found, but can be installed with: sudo apt install cowsay
But it's in nixpkgs, so we can take it for a spin with nix-shell:
$ nix-shell --packages cowsay
(cut: lots of downloading going on)
[nix-shell:~]$ cowsay "Well well well"
________________
< Well well well >
----------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
How does that work? From that shell, let's find out where cowsay is in $PATH:
[nix-shell:~]$ which cowsay /nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/bin/cowsay
And what dynamic libraries it links to:
$ ldd $(which cowsay) $ not a dynamic executable
Ah. Well, let's find out what it refers to when it runs, then:
[nix-shell:~]$ strace cowsay "Well well well" 2>&1 | grep --fixed-strings '"/'
execve("/nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/bin/cowsay", ["cowsay", "Well well well"], 0x7fff84a29b98 /* 103 vars */) = 0
access("/etc/ld-nix.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/bja5fzkjjn26zz742pgkvpm5dq3girv9-perl-5.36.0/lib/perl5/5.36.0/x86_64-linux-thread-multi/CORE/glibc-hwcaps/x86-64-v2/libperl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/bja5fzkjjn26zz742pgkvpm5dq3girv9-perl-5.36.0/lib/perl5/5.36.0/x86_64-linux-thread-multi/CORE/glibc-hwcaps/x86-64-v2", 0x7ffd39942530, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/bja5fzkjjn26zz742pgkvpm5dq3girv9-perl-5.36.0/lib/perl5/5.36.0/x86_64-linux-thread-multi/CORE/tls/x86_64/x86_64/libperl.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
(cut)
newfstatat(AT_FDCWD, "/nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/share/cowsay/site-cows/default.cow.pm", 0x161e4a8, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/share/cowsay/cows/default.cow", {st_mode=S_IFREG|0444, st_size=175, ...}, 0) = 0
newfstatat(AT_FDCWD, "/nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/share/cowsay/cows/default.cow", {st_mode=S_IFREG|0444, st_size=175, ...}, 0) = 0
openat(AT_FDCWD, "/nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/share/cowsay/cows/default.cow", O_RDONLY|O_CLOEXEC) = 3
It refers almost exclusively to things under /nix/store! In fact, the only things
it refers to that aren't under /nix/store are:
access("/etc/ld-nix.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/dev/urandom", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/dev/urandom", O_RDONLY|O_CLOEXEC) = 3
Let's look at another executable in $PATH:
[nix-shell:~]$ ldd $(which bash)
linux-vdso.so.1 (0x00007ffdde95f000)
libreadline.so.8 => /nix/store/pwhg9383i9j7vxhl2sb5mxsbdy70mrzj-readline-8.2p1/lib/libreadline.so.8 (0x00007f6e70d20000)
libhistory.so.8 => /nix/store/pwhg9383i9j7vxhl2sb5mxsbdy70mrzj-readline-8.2p1/lib/libhistory.so.8 (0x00007f6e70d13000)
libncursesw.so.6 => /nix/store/35s126gfkfrwbiv49kv9kxqdpd9zvcvm-ncurses-6.4/lib/libncursesw.so.6 (0x00007f6e70c9f000)
libdl.so.2 => /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/libdl.so.2 (0x00007f6e70c9a000)
libc.so.6 => /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/libc.so.6 (0x00007f6e70a00000)
/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/ld-linux-x86-64.so.2 => /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib64/ld-linux-x86-64.so.2 (0x00007f6e70d7b000)
And that's how nix works at runtime. There's no containerization going on (no
user namespaces), no virtual machines, no nothing: it's just plain old executables,
that only refer to things inside the /nix/store.
/nix/store is immutable, by the way:
[nix-shell:~]$ touch /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/libdl.so.2 touch: cannot touch '/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/libdl.so.2': Permission denied
Also, what's that long hash about it? It's derived from the inputs, which means, paths either never clash (because they have different inputs), or they deduplicate.
But also, you can have several versions of something and they won't conflict. In my little nix-shell, I already have two versions of glibc, for example:
[nix-shell:~]$ find /nix/store/*glibc*/ -name 'libdl.so' /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libdl.so /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/libdl.so
We can find out what needs either of them:
[nix-shell:~]$ nix-store --query --referrers /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/ | head -5 /nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163 /nix/store/026hln0aq1hyshaxsdvhg0kmcm6yf45r-zlib-1.2.13 /nix/store/3j1h6psl4pzn6b3yck6rk33bpwrmihb1-aws-c-common-0.8.5 /nix/store/4mxnw95jcm5a27qk60z7yc0gvxp42b9a-openssl-3.0.7 /nix/store/4rkhsf7sig2lh303bygqr3ph5mfwz0ah-s2n-tls-1.3.28 [nix-shell:~]$ nix-store --query --referrers /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/ | head -5 /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 /nix/store/502zrfyc2agh928111zhwy6959bqd8m0-xz-5.4.1 /nix/store/12wczkzi5db1ajbjp4hdyif3v9y5rbi5-xz-5.4.1-bin /nix/store/2a6yagz3pa8kiawg5mk2js70f8kwqzqd-bzip2-1.0.8 /nix/store/17pkxcz3js3549kn9dc3hhvp4adbwvs1-bzip2-1.0.8-bin
Using a slightly different query, we can see the full dependency tree of cowsay:
# piping to cat to avoid the pager
[nix-shell:~]$ nix-store --query --tree $(which cowsay) | cat
/nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0
└───/nix/store/bja5fzkjjn26zz742pgkvpm5dq3girv9-perl-5.36.0
├───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224
│ ├───/nix/store/na1irnycfp8z5mab0g5jvrnhnscsaqsb-libidn2-2.3.2
│ │ ├───/nix/store/jdjpni8kq3i95dj1d49nlf9m10wl0kqq-libunistring-1.0
│ │ │ └───/nix/store/jdjpni8kq3i95dj1d49nlf9m10wl0kqq-libunistring-1.0 [...]
│ │ └───/nix/store/na1irnycfp8z5mab0g5jvrnhnscsaqsb-libidn2-2.3.2 [...]
│ └───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
├───/nix/store/7w1s50i8yplw2gkpf9hpha6n9vci8g98-coreutils-9.1
│ ├───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
│ ├───/nix/store/83q5sbysda8285svx8mjlpc17bd3n630-attr-2.5.1
│ │ ├───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
│ │ └───/nix/store/83q5sbysda8285svx8mjlpc17bd3n630-attr-2.5.1 [...]
│ ├───/nix/store/aqd9cz92qz1pfnvw09bsidi6vrcfhl9r-gmp-with-cxx-stage4-6.2.1
│ │ ├───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
│ │ ├───/nix/store/k88zxp7cvd5gpharprhg9ah0vhz2asq7-gcc-12.2.0-lib
│ │ │ ├───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
│ │ │ └───/nix/store/k88zxp7cvd5gpharprhg9ah0vhz2asq7-gcc-12.2.0-lib [...]
│ │ └───/nix/store/aqd9cz92qz1pfnvw09bsidi6vrcfhl9r-gmp-with-cxx-stage4-6.2.1 [...]
│ ├───/nix/store/jwlr603mdcq2lkaqpxmg9kkp2acszb1n-acl-2.3.1
│ │ ├───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
│ │ ├───/nix/store/83q5sbysda8285svx8mjlpc17bd3n630-attr-2.5.1 [...]
│ │ └───/nix/store/jwlr603mdcq2lkaqpxmg9kkp2acszb1n-acl-2.3.1 [...]
│ └───/nix/store/7w1s50i8yplw2gkpf9hpha6n9vci8g98-coreutils-9.1 [...]
├───/nix/store/9dz5lmff9ywas225g6cpn34s0wbldnxa-zlib-1.2.13
│ └───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
├───/nix/store/kaiinm7cjirmv4zd3pdqar23rsypfqlh-libxcrypt-4.4.33
│ ├───/nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224 [...]
│ └───/nix/store/kaiinm7cjirmv4zd3pdqar23rsypfqlh-libxcrypt-4.4.33 [...]
└───/nix/store/bja5fzkjjn26zz742pgkvpm5dq3girv9-perl-5.36.0 [...]
You can find these and more in the Nix to Debian phrasebook. I've found nix documentation hard to navigate, especially around newer stuff like flakes, but this gives us a nice starting point.
This is all temporary though: if we exit out of our nix-shell with Ctrl-D,
we can't execute cowsay anymore:
$ cowsay Command 'cowsay' not found, but can be installed with: sudo apt install cowsay
...or can we? It's still there, after all...
$ /nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/bin/cowsay "Look who's still here"
_______________________
< Look who's still here >
-----------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Wait, does nix just leave stuff around? I thought nix-shell was temporary?
It does, until you garbage-collect! And then it only keeps stuff that's referenced by a garbage collector root.
$ nix-collect-garbage finding garbage collector roots... deleting garbage... deleting '/nix/store/4gj19wzyqhkdji9xg7hipy2m2899s0gg-shell.drv' deleting '/nix/store/43gc1j2wpjzs5avyk0fa4j77sxwdih2l-cowsay-3.7.0.drv' deleting '/nix/store/fyh9bz0ba9f6ihlazs78mcjnpbxj0bz4-source.drv' (cut) deleting '/nix/store/py1pdcfhj845cyam6ndq0b2mh02zk4my-openssl-3.0.8.tar.gz.drv' deleting '/nix/store/0v7w62jca8nw6w31zs4y2805jj670f6p-no-arch_only-8.2.patch' deleting unused links... note: currently hard linking saves -0.00 MiB 417 store paths deleted, 386.64 MiB freed
Ah, like docker system prune?
Kinda yeah!
nix profile
So, how do we get cowsay to stick around?
Well, https://search.nixos.org/ tells us how to use nix-env, with nix-env -iA nixpkgs.cowsay, but that's the "old" nix stuff, and we want to use the
"new" nix stuff, so we'll use nix profile instead:
$ nix profile install 'nixpkgs#cowsay' error: experimental Nix feature 'nix-command' is disabled; use '--extra-experimental-features nix-command' to override
Told you it was new! We don't want to pass extra flags every time we run a nix
command, so let's create ~/.config/nix/nix.conf and add this to it:
experimental-features = nix-command flakes max-jobs = auto
That second line simply tells nix to take advantage of all cores when building stuff, which we haven't really done yet.
Okay, now we can add cowsay to our profile:
$ nix profile install 'nixpkgs#cowsay'
$ cowsay "Guess who's back"
__________________
< Guess who's back >
------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
And this time, it'll survive a nix-collect-garbage:
$ nix-collect-garbage finding garbage collector roots... deleting garbage... deleting '/nix/store/43gc1j2wpjzs5avyk0fa4j77sxwdih2l-cowsay-3.7.0.drv' deleting '/nix/store/fyh9bz0ba9f6ihlazs78mcjnpbxj0bz4-source.drv' deleting '/nix/store/rrr6h7wib0c1mg3d79xas4ac6b0yblwg-unzip-6.0.drv' deleting '/nix/store/cciw7lgkldvx25d77cxpjhh1iw4xghd9-setup-hook.sh' deleting '/nix/store/kwprl4pibdiiz1pzvdqcxv1h3y962ncy-source' deleting '/nix/store/q9bnj7gmw8mdrssm1mzn9j0dsb32vada-glibc-locales-2.35-224.drv' deleting '/nix/store/d46j5s34jfs779lg392bz2ll81ngm26d-python3-minimal-3.10.9.drv' deleting '/nix/store/xfqq6rb3i2r2n65l64c4s0lm2y5bcbz4-pkg-config-wrapper-0.29.2.drv' deleting '/nix/store/wjs1ma47yzsxxiiyrl923iddfcf7a71d-expat-2.5.0.drv' deleting '/nix/store/scq9sx9lfj49523a0fk1mmfzypzvbkgd-CVE-2019-13232-2.patch.drv' deleting '/nix/store/pqz31j5dcw1pc5cszdrkrz3scpkvrav3-Python-3.10.9.tar.xz.drv' deleting '/nix/store/l4aa6fimwdzlzbyihb845dk44zrsw2ml-pkg-config-0.29.2.drv' deleting '/nix/store/kmvyag3kxqgc7km7phjq2v1adhf6rzzv-9e129fa0933cf1837672c97f5ae5ad4a1a10ec11.patch.drv' deleting '/nix/store/k43ngx9q4xybxbfz5d7vriixhqrd30sl-unzip60.tar.gz.drv' deleting '/nix/store/isyf5h4pwzzm4y6qdhdni5q304nbjvr7-libffi-3.4.4.drv' deleting '/nix/store/d1iivmdhrfdi9n0ilpfp94gjl98w7hnb-nuke-references.drv' deleting '/nix/store/byrqr5zh6lq097wy30ix9azdjgiqvs57-pkg-config-0.29.2.tar.gz.drv' deleting '/nix/store/a15i7dsiv2l1s9ssyvc84v3wf125yw6i-CVE-2019-13232-3.patch.drv' deleting '/nix/store/9y996ql1kszsr960p8bqnnyi7nfyczgh-glibc-2.35.tar.xz.drv' deleting '/nix/store/92gmlz1g08lm7d92p43wz1b4sfcn9b6q-patchutils-0.3.3.drv' deleting '/nix/store/ckrkr5sna6k8j41dsz0jlwk7v4mkaljl-patchutils-0.3.3.tar.xz.drv' deleting '/nix/store/8y4bvjq5n9krg2s8zgkhbv6xk1hjidr3-bison-3.8.2.drv' deleting '/nix/store/sndbhmzm6c7rh8mbpr373x4nvwfnxj6x-bison-3.8.2.tar.gz.drv' deleting '/nix/store/8q43vcy567y9b749fyqfbkqw5mx0jbnm-linux-headers-6.1.drv' deleting '/nix/store/lwhj44wjw92hlrg00z802nwbnwwc46qi-linux-6.1.tar.xz.drv' deleting '/nix/store/86jb692kfkm05ksi9rsqa2w45ihg98gm-autoreconf-hook.drv' deleting '/nix/store/f5rgi9bv4ghsz23z3rg0f7lvja0ba7q9-libtool-2.4.7.drv' deleting '/nix/store/qgcpw59w37sa48yr6566c92kxnhr727s-file-5.44.drv' deleting '/nix/store/bcvr111pfvziv1d62zzkcdkhd55mxn86-file-5.44.tar.gz.drv' deleting '/nix/store/b9a3hmzq303dk33n7b7abzh22k4kkcg9-automake-1.16.5.drv' deleting '/nix/store/q1cfch1x2ai0ww0jbcxh1mj0lnz37bx4-autoconf-2.71.drv' deleting '/nix/store/q2a79fndpzqj7nnpcps69dc9x8wq7qwh-autoconf-2.71.tar.xz.drv' deleting '/nix/store/fpdqsslr23vygwd8hb4rhq0n8h41svca-texinfo-6.8.drv' deleting '/nix/store/ynl8y88v27ih4w6xnc5rx1y0l1wsxwcc-perl-5.36.0.drv' deleting '/nix/store/icpmjjaxq9c902r2cxvzgkijhc0dw7lw-perl-5.36.0.tar.gz.drv' deleting '/nix/store/f4scqssxpx249w3bknfpkcg10y3rxgli-texinfo-6.8.tar.xz.drv' deleting '/nix/store/cszf8j3mlcv9n127ywiknfzg244bvk0w-gnum4-1.4.19.drv' deleting '/nix/store/nyibn3sc1k866pg7zjx9vz9h86kdj3ip-m4-1.4.19.tar.bz2.drv' deleting '/nix/store/7w75i7xdfg1h45z2fp3yz5i1864ygj6j-autoconf-archive-2022.09.03.drv' deleting '/nix/store/vp90pdrz9gnjwwdkq8bqwbmln6qdfiff-autoconf-archive-2022.09.03.tar.xz.drv' deleting '/nix/store/7vycjl4qsz42ba9fsx076ps645462fpv-gettext-0.21.drv' deleting '/nix/store/7j1109rjq45knvrq81y5bd3x05ivmz1n-expat-2.5.0.tar.xz.drv' deleting '/nix/store/5bnj2hxrjm69i7wr90jb27p7cchiyqkm-gettext-0.21.tar.gz.drv' deleting '/nix/store/3qhamx7gvfz4c1jhxr2amcsshxj2lr6h-automake-1.16.5.tar.xz.drv' deleting '/nix/store/3fqy66lzrcdrbrhchq34fj393n5szb8a-libffi-3.4.4.tar.gz.drv' deleting '/nix/store/27r0bvzg617g1r4zmnhwcyvf5cf4h9v5-28-cve-2022-0529-and-cve-2022-0530.patch.drv' deleting '/nix/store/1w8fxnpg6mjk7pqcvp8wkijkvvj9s8lf-CVE-2019-13232-1.patch.drv' deleting '/nix/store/0nasry8aip2d0zc115ia4jwhk10b8mh8-06-initialize-the-symlink-flag.patch.drv' deleting '/nix/store/0fbnd1w0qxkdl8ap0ddx5gx971km2kgm-libtool-2.4.7.tar.gz.drv' deleting '/nix/store/0hjmcwg5rpq8kk87ngdvbkrm6zh8k3ll-curl-7.87.0.drv' deleting '/nix/store/5pwp4fnx421hp06lyf8xgypzm0gpk2h5-libkrb5-1.20.1.drv' deleting '/nix/store/3sk2f10sbr79sv90d0m7b45fghzdm3n6-libssh2-1.10.0.drv' deleting '/nix/store/1wzbvkhygdyzlhng28f4gcq8dy1ydy9d-openssl-3.0.8.drv' deleting '/nix/store/lzmcfv2m4ripknpvbsv8wcg1ik1kif4h-use-etc-ssl-certs.patch' deleting '/nix/store/h22pscjl75ph7q0zcsn8gqc7qlizv6z5-mirrors-list.drv' deleting '/nix/store/ng54snz2v2y7jjz21d22jz9wpmvkz5lp-nghttp2-1.51.0.drv' deleting '/nix/store/l6wwzr6szj2xfhq4cvyvmpl9vqr9c9m7-pkg-config-wrapper-0.29.2.drv' deleting '/nix/store/lq08qqqyrxyp2cmrkhpzwn5sjdq2z476-pkg-config-0.29.2.drv' deleting '/nix/store/1x122mg8wkrla3sbzy00366aabzqxhhx-locales-setup-hook.sh.drv' deleting '/nix/store/2cq4hsc1v8ylccspw8351r72s56w1fia-CVE-2015-7697.diff' deleting '/nix/store/p8q62lpf4m88kp2s16apbibdagcrw93l-nghttp2-1.51.0.tar.bz2.drv' deleting '/nix/store/i3ywgfm3bgqvy7n0iflbf5z11v5zwy5j-keyutils-1.6.3.drv' deleting '/nix/store/d9b2qrrq32jzdsdx4y33inzrra5n5z5n-CVE-2014-8140.diff' deleting '/nix/store/p46prhgmv7ibjh9igfkbc6zfxbbi6sk5-dont-hardcode-cc.patch' deleting '/nix/store/0k8d71xza3275szx7n5r7vhj9m3ip0hh-glibc-iconv-2.35.drv' deleting '/nix/store/6np2acjv1nxcg0xzsv9a76wyrpxznkna-CVE-2014-8141.diff' deleting '/nix/store/sdj5d4lx170pffyvwwj2iac36c0avwhz-source' deleting '/nix/store/p55a764pi2f4kkx3adb43bxb2dnb4z6r-CVE-2018-18384.patch' deleting '/nix/store/1k1wn8807yizgz3ghnbd4k6zsc0dzfkr-CVE-2014-9913.patch' deleting '/nix/store/vizgh9p9qc3v9f4c17f10smqb7zg90sg-libssh2-1.10.0.tar.gz.drv' deleting '/nix/store/hs1p8pfraz8vhz0kj19wqd3jix6wcfh0-make-shell-wrapper-hook.drv' deleting '/nix/store/sswjgxdkphxj9f3wn0q30nzz8cdkgbsm-die-hook.drv' deleting '/nix/store/17yjw33i747j1v1md1fgi4s2i0hy8klq-locales-builder.sh' deleting '/nix/store/ycwm35msmsdi2qgjax1slmjffsmwy8am-write-mirror-list.sh' deleting '/nix/store/k51cqpa0nzhs09bnyv69liik1d86l9ww-coreutils-9.1.drv' deleting '/nix/store/68bf2pfyxwfq1m0fzx4fqjw41vdkaard-perl-5.36.0.drv' deleting '/nix/store/cklrwbwi889pp2fdsswdjvn12sdy5i5j-openssl-disable-kernel-detection.patch' deleting '/nix/store/0ri4w1lxx2s1mazff8y7h5q2pkixqz7s-python-setup-hook.sh.drv' deleting '/nix/store/rdkdki1f24q8mqgnbsyk7gmh28c027ks-CVE-2014-8139.diff' deleting '/nix/store/6zqn6w9rwkgfa6z1hpagnh5xhz2dag6m-CVE-2015-7696.diff' deleting '/nix/store/pdcj2chp5c2gvm2jc3shbajfc62kbx1i-CVE-2014-9636.diff' deleting '/nix/store/97d26l91h0db8h0qkmhxwi5d8shrilv6-CVE-2016-9844.patch' deleting '/nix/store/151w9968gazbxf52ijf2y7fh4f4sphi0-stdenv-linux.drv' deleting '/nix/store/88a62ypvi5xpa3m8znhrl6l7jc307i8r-conf-symlink.patch' deleting '/nix/store/w1c7ihmg8ykp1ddpjldq7i5407cqsbz4-7.79.1-darwin-no-systemconfiguration.patch' deleting '/nix/store/qggqyqdmdsqdv4k19sa6rjxcq887930p-krb5-1.20.1.tar.gz.drv' deleting '/nix/store/lkkzb8ybhp5jjgvb04b13n68i51lxwd9-keyutils-1.6.3.tar.gz.drv' deleting '/nix/store/8djp1rizc1dblv8svnb0mpa0c3lwvc17-drop-comments.patch' deleting '/nix/store/z0sjg5c7g1iqb4x5vyiqcm5n68mb5x5p-0001-Remove-unused-function-after_eq.patch' deleting '/nix/store/57620l1168piiia2bmmsxxhh7sjb2n40-builder.sh' deleting '/nix/store/bbq4vd80lj8m6qx3y0hl55aspir0diik-curl-7.87.0.tar.bz2.drv' deleting '/nix/store/sq4h6bqjx12v9whvm65pjss25hg1538q-nix-ssl-cert-file.patch' deleting '/nix/store/zrh6il3gp9xa58ldg40d57kwgsvljyb1-openssl_add_support_for_libressl_3_5.patch' deleting '/nix/store/p0wk6m9z4r48b7dph66mkhc7y7kxhr5i-source' deleting '/nix/store/py1pdcfhj845cyam6ndq0b2mh02zk4my-openssl-3.0.8.tar.gz.drv' deleting unused links... note: currently hard linking saves -0.00 MiB 95 store paths deleted, 151.50 MiB freed
Uh oh that still deleted stuff, are you sure it's alright?
I'm sure it's finnneee:
$ cowsay "Still kickin'"
_______________
< Still kickin' >
---------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Good, but.. where is it? How does it work now?
Oh, it's not a big mystery:
$ which cowsay /home/amos/.nix-profile/bin/cowsay $ ls -lhA $(which cowsay) -r-xr-xr-x 1 root root 8.9K Jan 1 1970 /home/amos/.nix-profile/bin/cowsay $ readlink -f $(which cowsay) /nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0/bin/cowsay
It's "simply" a symbolic link to its location in /nix/store!
It's also a Perl program, so it starts with a hash-pling:
$ head $(which cowsay) #!/nix/store/bja5fzkjjn26zz742pgkvpm5dq3girv9-perl-5.36.0/bin/perl ## ## Cowsay ## use Cwd 'abs_path'; use File::Basename; use File::Find; use Getopt::Std;
...which points to the nix-installed perl, and not /usr/bin/perl, which
also exists.
If we were using NixOS as our Linux distribution, there would be no
/usr/bin/perl, only the nix-installed one, but I'm going for the pragmatic
middle-of-the-road solution where you're cozy and/or forced into using a more
mainstream Linux distribution, like Ubuntu or Fedora, and you want to sprinkle
a bit of nix on top.
nix the language
nix profiles are neat, and we can list what's installed in them with:
$ nix profile list 0 flake:nixpkgs#legacyPackages.x86_64-linux.cowsay github:NixOS/nixpkgs/63b5955814db30d2e2ff7157aaa5665b502ed2f4#legacyPackages.x86_64-linux.cowsay /nix/store/azn0g0m6yg6m9vmdp3wq6wjbsd1znv44-cowsay-3.7.0 /nix/store/qgsl9bd6vvnpjaxhdnzg0xmycg3652c0-cowsay-3.7.0-man
...but it's not what we want for developing catscii. We want anyone to be able
to jump in and develop the project, so if the README said "run nix profile
install (a bunch of things)", it wouldn't be much better than saying "run apt
install (another bunch of things)".
Ideally, we want the list of things catscii needs to live directly in the
repository, as a file.
And we can do that! In several ways, but again, let's focus on the new, experimental, slightly-broken-at-times, opt-in stuff: flakes.
And my experience with nix has been a lot of "just copy paste this", and having to deconstruct it later to understand what the heck is going on.
So, let's start small.
Nix is a language. We can get a REPL!
$ nix repl Welcome to Nix 2.13.2. Type :? for help. nix-repl> 1 + 2 3
We can have code in .nix files:
# in sample.nix [ 1 2 ] ++ [ 3 4 ]
$ nix eval --file ./sample.nix [ 1 2 3 4 ]
Before going any further: if you're going to be running those locally as we go (and you should), here's the recommended set-up with VS Code.
You want to install a formatter and an LSP server:
$ nix profile install nixpkgs#nixpkgs-fmt nixpkgs#rnix-lsp
Then grab the Nix IDE VS Code extension.
If you use VS Code with remote ssh, it's possible Nix IDE complains about either
of nixpkgs-fmt / rnix-lsp not being in $PATH (it'll fail with ENOENT).
If killing the remote vscode server isn't enough, you can always completely shut down VS Code (it caches too much stuff for its own good) and reboot the VM: that's why we have VMs!
With that, you should have syntax highlighting, and you can enable "formatting on save" with this in your user settings:
{
"[nix]": {
"editor.formatOnSave": true,
"editor.defaultFormatter": "jnoortheen.nix-ide"
},
}
All good? Good.
nix has sets, which you may be used to calling "dictionaries", "objects" or "maps", depending how your brain is wired. Anyway they associate keys and values:
{
list =
[ 1 2 ] ++ [ 3 4 ];
addition = 1 + 2;
}
$ nix eval --file ./sample.nix
{ addition = 3; list = [ 1 2 3 4 ]; }
$ nix eval --file ./sample.nix addition
3
$ nix eval --file ./sample.nix list
[ 1 2 3 4 ]
And it has functions, which are called lambdas in some contexts:
{
add_two = x: x + 2;
}
Unfortunately, it's hard to call it with nix eval due to an outstanding
bug - you didn't get warned because
we opted into experimental stuff earlier, but nix eval is part of that
experimental stuff and is a little half-baked.
We can have it print the lambda itself:
$ nix eval --file ./sample.nix add_two <LAMBDA>
But to actually call it, we have to get creative. Instead of using --file, we can
use --expr to evaluate an expression, and use import to import our .nix
file:
$ nix eval --expr 'import ./sample.nix' error: access to absolute path '/home/amos/catscii/sample.nix' is forbidden in pure eval mode (use '--impure' to override) (use '--show-trace' to show detailed location information)
Well, we can do that with --impure:
$ nix eval --impure --expr 'import ./sample.nix'
{ add_two = <LAMBDA>; }
And then, well, we can call it:
$ nix eval --impure --expr '(import ./sample.nix).add_two 3' 5
Parentheses? What for?
Not to call the function, that's for sure! This isn't a lisp. Here they're just
used for grouping/priority: we want to access the .add_two property of the
results of import ./sample.nix.
Note that ./sample.nix isn't a string, it's a path: that's a distinct type in
nix. We can't import a string:
$ nix eval --impure --expr '(import "./sample.nix").add_two 3'
error: string './sample.nix' doesn't represent an absolute path
at «string»:1:2:
1| (import "./sample.nix").add_two 3
| ^
(use '--show-trace' to show detailed location information)
Okay, okay, this feels weird, but I'm assimilating.
Oh we're not done! What if we want to bind the result of importing
./sample.nix to something before using it by name?
Since nix isn't imperative, we don't really have a concept of statements that are evaluated in order.
But there's ways to work around that.
For simplicity, let's make another nix file that'll import sample.nix, so
we don't have to deal with --impure --expr.
# in caller.nix (import ./sample.nix).add_two 5
$ nix eval --file caller.nix 7
Okay, so, I was saying: binding it to something. Well we can't have "successive statements", but we can have... sets!
# in caller.nix
{
sample = import ./sample.nix;
result = sample.add_two 5;
}
Unfortunately, that doesn't work:
$ nix eval -f caller.nix
error: undefined variable 'sample'
at /home/amos/catscii/caller.nix:4:12:
3| sample = (import ./sample.nix).add_two 5;
4| result = sample.add_two 5;
| ^
5| }
Because by default, sets are non-recursive. We can fix that with rec:
# in caller.nix
rec {
sample = import ./sample.nix;
result = sample.add_two 5;
}
$ nix eval --file caller.nix
{ result = 7; sample = { add_two = <LAMBDA>; }; }
This is a little untidy though: we don't really need "sample" to be part of our output.
So instead, we can use a let:
# in caller.nix let sample = import ./sample.nix; in sample.add_two 5
$ nix eval -f caller.nix 7
Okay, we got pretty far already! So, let's summarize:
()are just like in C/Rust/whatever: they mean "evaluate that first"{}defines a set/object/dictionary/key-value map. it haskey = value;inside (each item ends with a semicolon)[]defines a list/array: it has1 2 3inside (space-separated)foo.baraccesses propertybarof setfooletlets you bind some names and use them in theinblock later, which is preferred to using a recursive set.
You haven't really explained why recursive sets are bad though.
Ah. Easy!
# in caller.nix
rec {
a = b;
b = c;
c = a;
}
$ nix eval --file caller.nix
error: infinite recursion encountered
at /home/amos/catscii/caller.nix:4:7:
3| a = b;
4| b = c;
| ^
5| c = a;
(use '--show-trace' to show detailed location information)
Oh. I see. Given that, I suppose order doesn't matter in sets?
You are correct:
# in caller.nix
rec {
result = sample.add_two 5;
# this comes after we've used `sample`, but that doesn't matter.
sample = import ./sample.nix;
}
$ nix eval --file caller.nix
{ result = 7; sample = { add_two = <LAMBDA>; }; }
nix derivations
nix language is all well and good (...depending), but by itself it doesn't do much.
The whole packagement system (sic.) built around it is the juicy bit.
And if you look at any nix tutorial, they'll show you how to use
mkDerivation. So of course, we won't do that - we'll go one level deeper.
So let's remove all our nix files so far:
$ rm *.nix (cut)
And create a default.nix file with a simple derivation:
# in default.nix
derivation {
name = "simple";
builder = "${(import <nixpkgs> {}).bash}/bin/bash";
args = [ "-c" "echo foo > $out" ];
src = ./.;
system = builtins.currentSystem;
}
There's a lot going on here, so before we try to do something with this file, let's make sure we understand what's going.
We're calling a function named derivation, passing a set to it. The value for
name is a string, the value for args is a list, the value for src is a
path.
What is system set to? Let's find out:
$ nix eval --expr 'builtins.currentSystem'
error: attribute 'currentSystem' missing
at «string»:1:1:
1| builtins.currentSystem
| ^
Woops, that didn't work. Does nix eval not give you builtins?
$ nix eval --raw --expr 'builtins.toXML { a = "b"; }'
<?xml version='1.0' encoding='utf-8'?>
<expr>
<attrs>
<attr name="a">
<string value="b" />
</attr>
</attrs>
</expr>
Oh, it does. It just doesn't have currentSystem.
The reason nix eval didn't recognize builtins.currentSystem is because it operates in a pure evaluation mode by default. In this mode, certain builtins, like currentSystem, aren't available. But there's another way to access these builtins.
How about the REPL?
$ nix repl Welcome to Nix 2.13.2. Type :? for help. nix-repl> builtins.currentSystem "x86_64-linux" nix-repl>
That one has it, neat. The nix repl operates in an impure mode by default, which is why builtins.currentSystem is accessible there. If you want to use nix eval and see the value of currentSystem, you can add the --impure flag:
$ nix eval --impure --expr 'builtins.currentSystem' "aarch64-darwin"
This flag switches the evaluation to impure mode, making currentSystem available.
The nastiest part of our default.nix file is definitely this, though:
# in default.nix
derivation {
name = "simple";
# 👇👇👇👇👇👇👇
builder = "${(import <nixpkgs> {}).bash}/bin/bash";
args = [ "-c" "echo foo > $out" ];
src = ./.;
system = builtins.currentSystem;
}
Luckily, we know just enough nix to decipher what that means. Well, we didn't
know "${foo}" did string interpolation, but we do now.
We also didn't know what <nixpkgs> is a magic path, but, yeah:
$ nix repl Welcome to Nix 2.13.2. Type :? for help. nix-repl> <nixpkgs> /nix/var/nix/profiles/per-user/root/channels/nixpkgs
And so, if we follow logically, this must mean we build a string out of the
"bash" property of whatever the result of calling import <nixpkgs> with an
empty set gives us.
Because, yeah, import <nixpkgs> evaluates to a lambda:
nix-repl> import <nixpkgs> «lambda @ /nix/store/44vlp0hzcir0j7hmg48g8fmj321igpw1-nixpkgs/nixpkgs/pkgs/top-level/impure.nix:14:1»
So we must call it. But I strongly recommend not typing import <nixpkgs> {}
in your REPL, because it'll evaluate everything in there. At the time of this
writing, there's over 80K (eighty thousand) packages in there, and luckily, they
warned fools like me:
nix-repl> import <nixpkgs> {}
{ AAAAAASomeThingsFailToEvaluate = «error: error: Please be informed that this pseudo-package is not the only part of
Nixpkgs that fails to evaluate. You should not evaluate entire Nixpkgs
without some special measures to handle failing packages, like those taken
by Hydra.»; AMB-plugins = «derivation /nix/store/4fpqhkcsm53zlr5lgwb6hm7zjif6h5ci-AMB-plugins-0.8.1.drv»; ArchiSteamFarm = «derivation /nix/store/zg5wnx2d7lcfdir0dlb64vw1s2jkydaf-archisteamfarm-5.4.1.11.drv»; AusweisApp2 = «derivation /nix/store/c4cyfm3q2f91dq26nv5vspfnlvr7bpnq-AusweisApp2-1.26.2.drv»; BeatSaberModManager = «derivation /nix/store/b3yv9v0cyckmhgpl8b3bplk5ww9l8asi-BeatSaberModManager-0.0.4.drv»; CHOWTapeModel = «derivation /nix/store/cyap5ndhw5h5x21jllhr6vprxfdj7f06-CHOWTapeModel-2.10.0.drv»; ChowCentaur = «derivation /nix/store/mj1i3vs60asnrzl637b6iznivp3sqvqg-ChowCentaur-1.4.0.drv»; ChowKick = ^Cerror: interrupted by the user
«derivation
Accessing a single field, though, is fine:
nix-repl> (import <nixpkgs> {}).bash
«derivation /nix/store/0hnjp6s8k71xm62157v37zg3qzwvl8lx-bash-5.2-p15.drv»
So what we've learned is that nix has lazy evaluation, which, if you've made it this far, you've probably already heard about, because it's the one thing nobody will shut up about.
(But in their defense, that's pretty important).
Now to do the same string interpolation as we have in default.nix, but in the
REPL, to see what it gives us:
nix-repl> "${(import <nixpkgs> {}).bash}/bin/bash"
"/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash"
See, that's interesting.
Because it just so happens that this path does exist right now:
$ file /nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash /nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/ld-linux-x86-64.so.2, BuildID[sha1]=ca0fcdc9d631f4ccf8bb36cdd843b7d2b7b9390f, for GNU/Linux 3.10.0, not stripped
But if we try with another package:
$ nix repl
Welcome to Nix 2.13.2. Type :? for help.
nix-repl> "${(import <nixpkgs> {}).dive}"
"/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0"
(pressing Ctrl-D to get back to bash)
$ file /nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0
/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0: cannot open `/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0' (No such file or directory)
It doesn't exist.
And that's another thing you may have heard The Nix People go on about: the hash part is great, actually: because all inputs are statically defined, it can tell where a derivation is going to go (what its hash is going to be) without actually realizing it.
If we actually want to build dive, we first have to instantiate it:
$ nix-instantiate --expr '(import <nixpkgs> {}).dive'
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/30r459s85cx70nprlcxyar9sa13qg9yr-dive-0.10.0.drv
We get a warning about the garbage collector – fair enough! We're just playing around.
In return, we got the path of a .drv file, which I keep reading as "driver"
but I suspect means "derivation". This .drv file is human-readable (go look)
but not really human-friendly.
Let's use nix show /path/to/file.drv instead, which gives us JSON:
{
"/nix/store/30r459s85cx70nprlcxyar9sa13qg9yr-dive-0.10.0.drv": {
"args": [
"-e",
"/nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh"
],
"builder": "/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash",
"env": {
"CGO_ENABLED": "1",
"GO111MODULE": "on",
"GOARCH": "amd64",
"GOFLAGS": "-mod=vendor -trimpath",
"GOOS": "linux",
"__structuredAttrs": "",
"buildInputs": "/nix/store/d7g939y4lqixk8rfm385vwa7gc12rmck-btrfs-progs-6.1.2 /nix/store/h4fdrla07dzi0cp9fah3a9vdhfrih618-gpgme-1.18.0-dev /nix/store/9cw3b5nixp1xv7k3prj28lnxyfkj7k2f-lvm2-2.03.18-dev",
"buildPhase": "runHook preBuild\n\nexclude='\\(/_\\|examples\\|Godeps\\|testdata'\nif [[ -n \"$excludedPackages\" ]]; then\n IFS=' ' read -r -a excludedArr <<<$excludedPackages\n printf -v excludedAlternates '%s\\\\|' \"${excludedArr[@]}\"\n excludedAlternates=${excludedAlternates%\\\\|} # drop final \\| added by printf\n exclude+='\\|'\"$excludedAlternates\"\nfi\nexclude+='\\)'\n\nbuildGoDir() {\n local cmd=\"$1\" dir=\"$2\"\n\n . $TMPDIR/buildFlagsArray\n\n declare -a flags\n flags+=($buildFlags \"${buildFlagsArray[@]}\")\n flags+=(${tags:+-tags=})\n flags+=(${ldflags:+-ldflags=\"$ldflags\"})\n flags+=(\"-p\" \"$NIX_BUILD_CORES\")\n\n if [ \"$cmd\" = \"test\" ]; then\n flags+=(-vet=off)\n flags+=($checkFlags)\n fi\n\n local OUT\n if ! OUT=\"$(go $cmd \"${flags[@]}\" $dir 2>&1)\"; then\n if ! echo \"$OUT\" | grep -qE '(no( buildable| non-test)?|build constraints exclude all) Go (source )?files'; then\n echo \"$OUT\" >&2\n return 1\n fi\n fi\n if [ -n \"$OUT\" ]; then\n echo \"$OUT\" >&2\n fi\n return 0\n}\n\ngetGoDirs() {\n local type;\n type=\"$1\"\n if [ -n \"$subPackages\" ]; then\n echo \"$subPackages\" | sed \"s,\\(^\\| \\),\\1./,g\"\n else\n find . -type f -name \\*$type.go -exec dirname {} \\; | grep -v \"/vendor/\" | sort --unique | grep -v \"$exclude\"\n fi\n}\n\nif (( \"${NIX_DEBUG:-0}\" >= 1 )); then\n buildFlagsArray+=(-x)\nfi\n\nif [ ${#buildFlagsArray[@]} -ne 0 ]; then\n declare -p buildFlagsArray > $TMPDIR/buildFlagsArray\nelse\n touch $TMPDIR/buildFlagsArray\nfi\nif [ -z \"$enableParallelBuilding\" ]; then\n export NIX_BUILD_CORES=1\nfi\nfor pkg in $(getGoDirs \"\"); do\n echo \"Building subPackage $pkg\"\n buildGoDir install \"$pkg\"\ndone\nrunHook postBuild\n",
"builder": "/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash",
"checkPhase": "runHook preCheck\n# We do not set trimpath for tests, in case they reference test assets\nexport GOFLAGS=${GOFLAGS//-trimpath/}\n\nfor pkg in $(getGoDirs test); do\n buildGoDir test \"$pkg\"\ndone\n\nrunHook postCheck\n",
"cmakeFlags": "",
"configureFlags": "",
"configurePhase": "runHook preConfigure\n\nexport GOCACHE=$TMPDIR/go-cache\nexport GOPATH=\"$TMPDIR/go\"\nexport GOPROXY=off\nexport GOSUMDB=off\ncd \"$modRoot\"\nrm -rf vendor\ncp -r --reflink=auto /nix/store/1j8i1f7abvbd5p4z9ias3l914d6vbbv7-dive-0.10.0-go-modules vendor\n\n\nrunHook postConfigure\n",
"depsBuildBuild": "",
"depsBuildBuildPropagated": "",
"depsBuildTarget": "",
"depsBuildTargetPropagated": "",
"depsHostHost": "",
"depsHostHostPropagated": "",
"depsTargetTarget": "",
"depsTargetTargetPropagated": "",
"disallowedReferences": "/nix/store/4z6hhnxkmd34kp0i2p0a7nz7idvkc753-go-1.19.5",
"doCheck": "1",
"doInstallCheck": "",
"enableParallelBuilding": "1",
"enableParallelChecking": "1",
"installPhase": "runHook preInstall\n\nmkdir -p $out\ndir=\"$GOPATH/bin\"\n[ -e \"$dir\" ] && cp -r $dir $out\n\nrunHook postInstall\n",
"ldflags": "-s -w -X main.version=0.10.0",
"mesonFlags": "",
"name": "dive-0.10.0",
"nativeBuildInputs": "/nix/store/4z6hhnxkmd34kp0i2p0a7nz7idvkc753-go-1.19.5 /nix/store/lrb01sby9jg4hfqhr32s28igg77bhcwr-pkg-config-wrapper-0.29.2",
"out": "/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0",
"outputs": "out",
"patches": "/nix/store/l8vidjbfpd8r5llai1zgby1p1lnsdpjl-fe9411c414418d839a8638bb9a12ccfc892b5845.patch",
"pname": "dive",
"propagatedBuildInputs": "",
"propagatedNativeBuildInputs": "",
"src": "/nix/store/p4zd2ls3yy4w6j430f5lcykkkc14f0yc-source",
"stdenv": "/nix/store/b09v23lirgvci3wzszh22mbkdfj0h0yq-stdenv-linux",
"strictDeps": "1",
"system": "x86_64-linux",
"version": "0.10.0"
},
"inputDrvs": {
"/nix/store/0hnjp6s8k71xm62157v37zg3qzwvl8lx-bash-5.2-p15.drv": [
"out"
],
"/nix/store/1ps4mh2yih85fx76imq77q36whi0g1pn-go-1.19.5.drv": [
"out"
],
"/nix/store/c31f6dgzjjyn84d16ybaap9zr1j2l6ql-source.drv": [
"out"
],
"/nix/store/kv8gamr3m8s7pbpzqda5fm3b6z3i6cqf-dive-0.10.0-go-modules.drv": [
"out"
],
"/nix/store/l98jn7564nrzb7a5ik6iqqfx0vrmfwaz-gpgme-1.18.0.drv": [
"dev"
],
"/nix/store/likh4j2kkp2fparm884aywk4ma0yxwvl-btrfs-progs-6.1.2.drv": [
"out"
],
"/nix/store/nsb0bznfxnfyxnlacd4qzs4s1sz9jdj0-lvm2-2.03.18.drv": [
"dev"
],
"/nix/store/qf8nz4ixc1b5kvqd2h50mbxgj23n0kij-fe9411c414418d839a8638bb9a12ccfc892b5845.patch.drv": [
"out"
],
"/nix/store/r2h029bx2fbyxxj84s5hf1abp2vfkah2-stdenv-linux.drv": [
"out"
],
"/nix/store/xfqq6rb3i2r2n65l64c4s0lm2y5bcbz4-pkg-config-wrapper-0.29.2.drv": [
"out"
]
},
"inputSrcs": [
"/nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh"
],
"outputs": {
"out": {
"path": "/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0"
}
},
"system": "x86_64-linux"
}
}
This specifies everything nix needs to realize that derivation.
But at this point, the output path still doesn't exist:
$ file /nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10. /nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.: cannot open `/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.' (No such file or directory)
If we want it, we have to realize it:
$ nix-store --realize /nix/store/30r459s85cx70nprlcxyar9sa13qg9yr-dive-0.10.0.drv these 4 paths will be fetched (3.06 MiB download, 11.82 MiB unpacked): /nix/store/66fr4mlfnwym5dw4bx61xs7nb81q58cb-tzdata-2022g /nix/store/ih35ndii8g86q4hlr2f1ma0z14ga4d0r-mailcap-2.1.53 /nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0 /nix/store/nl44nfqyyc7d5bw58v8g5ac2wcvmfp98-iana-etc-20221107 copying path '/nix/store/nl44nfqyyc7d5bw58v8g5ac2wcvmfp98-iana-etc-20221107' from 'https://cache.nixos.org'... copying path '/nix/store/ih35ndii8g86q4hlr2f1ma0z14ga4d0r-mailcap-2.1.53' from 'https://cache.nixos.org'... copying path '/nix/store/66fr4mlfnwym5dw4bx61xs7nb81q58cb-tzdata-2022g' from 'https://cache.nixos.org'... copying path '/nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0' from 'https://cache.nixos.org'... warning: you did not specify '--add-root'; the result might be removed by the garbage collector /nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0
And now we have it!
$ /nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0/bin/dive --version dive 0.10.0
Of course, it'll be garbage-collected, just like they warned us:
$ nix-collect-garbage (cut) $ /nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0/bin/dive --version bash: /nix/store/mbf2jg732w1sg944ijick07p4kiqy9ga-dive-0.10.0/bin/dive: No such file or directory
Now onto our own derivation. It's not in <nixpkgs>, it's in a default.nix
file.
We can evaluate it, just like any other .nix file:
$ nix eval --file default.nix
{ all = [ «repeated» ]; args = [ "-c" "echo foo > $out" ]; builder = "/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash"; drvAttrs = { args = «repeated»; builder = "/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash"; name = "simple"; src = /home/amos/catscii; system = "x86_64-linux"; }; drvPath = "/nix/store/zvknsw43ljqyw6ry9w5snzgscckvcp2m-simple.drv"; name = "simple"; out = «repeated»; outPath = "/nix/store/fv9v381if8y7kl5iajvskl7cawz38yri-simple"; outputName = "out"; src = /home/amos/catscii; system = "x86_64-linux"; type = "derivation"; }
That's not terribly friendly. Normally we could ask for JSON with --json, but
since this is a value of type "derivation", it just shows the output path instead:
$ nix eval --json --file default.nix "/nix/store/fv9v381if8y7kl5iajvskl7cawz38yri-simple"
So, instead, I'll just manually format what nix eval --file gave us:
{
all = [ «repeated» ];
args = [ "-c" "echo foo > $out" ];
builder = "/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash";
drvAttrs = {
args = «repeated»;
builder = "/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15/bin/bash";
name = "simple";
src = /home/amos/catscii;
system = "x86_64-linux";
};
drvPath = "/nix/store/zvknsw43ljqyw6ry9w5snzgscckvcp2m-simple.drv";
name = "simple";
out = «repeated»;
outPath = "/nix/store/fv9v381if8y7kl5iajvskl7cawz38yri-simple";
outputName = "out";
src = /home/amos/catscii;
system = "x86_64-linux";
type = "derivation";
}
So, essentially, a derivation seems like a spicy set. Okay!
Just like dive, we can instantiate our derivation to generate the .drv file:
$ nix-instantiate . warning: you did not specify '--add-root'; the result might be removed by the garbage collector /nix/store/zvknsw43ljqyw6ry9w5snzgscckvcp2m-simple.drv
Note that whereas to instantiate a nixpkgs package, we had to use --expr,
here we can just pass a path: the current directory, ., which contains a
default.nix file (that naming is not by accident).
Now that we have the .drv path, we can realize it with nix-store:
$ nix-store -r /nix/store/zvknsw43ljqyw6ry9w5snzgscckvcp2m-simple.drv this derivation will be built: /nix/store/zvknsw43ljqyw6ry9w5snzgscckvcp2m-simple.drv this path will be fetched (0.43 MiB download, 1.60 MiB unpacked): /nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15 copying path '/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15' from 'https://cache.nixos.org'... building '/nix/store/zvknsw43ljqyw6ry9w5snzgscckvcp2m-simple.drv'... warning: you did not specify '--add-root'; the result might be removed by the garbage collector /nix/store/fv9v381if8y7kl5iajvskl7cawz38yri-simple
And now, it's in our store!
$ cat /nix/store/fv9v381if8y7kl5iajvskl7cawz38yri-simple foo
I didn't really explain what our derivation actually did. Well, it ran the
bash builder with arguments -c "echo foo > $out". $out is an environment
variable that contains the path where the derivation should generate their output.
Now for a bit of good news: we went much deeper than needed!
First off, we can clean up our default.nix - we know how to use let, so we
can do this:
# in default.nix
let
pkgs = (import <nixpkgs> { });
in
derivation {
name = "simple";
builder = "${pkgs.bash}/bin/bash";
args = [ "-c" "echo foo > $out" ];
src = ./.;
system = builtins.currentSystem;
}
Alternatively, we can use the with syntax, which.. to me at least, "adds a set
to the local scope". If we do with something, we can refer to any property of
something without doing something.property - we can just do property.
So in this version, we no longer need to refer to pkgs.bash, we can just refer
to bash:
# in default.nix
with
(import <nixpkgs> { });
derivation
{
name = "simple";
builder = "${bash}/bin/bash";
args = [ "-c" "echo foo > $out" ];
src = ./.;
system = builtins.currentSystem;
}
Bringing all of nixpkgs in scope is an
anti-pattern.
Using let and selecting what to inherit is preferred.
Secondly, we can use some bash powers to do both instantiation and realization in one command:
$ nix-store -r $(nix-instantiate .) warning: you did not specify '--add-root'; the result might be removed by the garbage collector this derivation will be built: /nix/store/5cpc0sjkzgn4xrc75a4brzpl6kxdnaxd-simple.drv building '/nix/store/5cpc0sjkzgn4xrc75a4brzpl6kxdnaxd-simple.drv'... warning: you did not specify '--add-root'; the result might be removed by the garbage collector /nix/store/mz5y3xxmas0y4595lk8w5kvpy5k9xgd3-simple $ cat /nix/store/mz5y3xxmas0y4595lk8w5kvpy5k9xgd3-simple foo
But let's garbage-collect for a minute..
$ nix-collect-garbage finding garbage collector roots... (cut)
You know what does both instantiation and realization in one step?
That's right! It's nix-build!
What do you mean "that's right"? That's the first time you mention it?
Welllll maybe folks have used nix before on the surface, I don't know.
Anyway:
$ nix-build this derivation will be built: /nix/store/5cpc0sjkzgn4xrc75a4brzpl6kxdnaxd-simple.drv this path will be fetched (0.43 MiB download, 1.60 MiB unpacked): /nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15 copying path '/nix/store/561wgc73s0x1250hrgp7jm22hhv7yfln-bash-5.2-p15' from 'https://cache.nixos.org'... building '/nix/store/5cpc0sjkzgn4xrc75a4brzpl6kxdnaxd-simple.drv'... /nix/store/mz5y3xxmas0y4595lk8w5kvpy5k9xgd3-simple $ cat /nix/store/mz5y3xxmas0y4595lk8w5kvpy5k9xgd3-simple foo
Okay, cool, we know everything about nix!
Well.. no? What about mkDerivation?
Oh, right. So derivation is low-level and one hardly ever uses it.
A real-world mkDerivation
Instead, mkDerivation is often used.
And this time, I think we're ready to look at a bit of real-world nix. It's something I wrote myself, for my video platform.
In this incarnation of my video platform, I use
shaka-packager to package
videos as MPEG-DASH (fragmented MP4 + an .mpd manifest).
It's distributed as static binaries already, so the nix bit is actually fairly simple!
# in default.nix
with
(import <nixpkgs> { });
let
version = "2.6.1";
in
stdenv.mkDerivation
{
name = "shaka-packager-${version}";
# https://nixos.wiki/wiki/Packaging/Binaries
src = pkgs.fetchurl {
url =
"https://github.com/shaka-project/shaka-packager/releases/download/v${version}/packager-linux-x64";
sha256 = "sha256-MoMX6PEtvPmloXJwRpnC2lHlT+tozsV4dmbCqweyyI0=";
};
dontUnpack = true;
sourceRoot = ".";
installPhase = ''
install -m755 -D $src $out/bin/shaka-packager
'';
meta = with lib; {
homepage = "https://shaka-project.github.io/shaka-packager/html/";
description =
"Media packaging framework for VOD and Live DASH and HLS applications";
platforms = platforms.x86_64-linux;
};
}
We can build it:
$ nix-build
these 2 derivations will be built:
/nix/store/4gllgh3bdxfpwj4hi2bcqa5ly390jz35-packager-linux-x64.drv
/nix/store/668fy2xd3lc8anmqd6jmhk2a1vrzv79x-shaka-packager-2.6.1.drv
these 46 paths will be fetched (64.56 MiB download, 294.29 MiB unpacked):
/nix/store/09vh6qzbvbavdlcfm4i9kcxj7sv1y2xh-curl-7.87.0-man
/nix/store/12wczkzi5db1ajbjp4hdyif3v9y5rbi5-xz-5.4.1-bin
(cut)
copying path '/nix/store/2a6yagz3pa8kiawg5mk2js70f8kwqzqd-bzip2-1.0.8' from 'https://cache.nixos.org'...
copying path '/nix/store/v316awk24xxz2dj84b5qw2vflfpg7if4-diffutils-3.8' from 'https://cache.nixos.org'...
copying path '/nix/store/p699p3vdqq4bn958dxzppbhn7n5k0c0n-ed-1.19' from 'https://cache.nixos.org'...
(cut)
building '/nix/store/4gllgh3bdxfpwj4hi2bcqa5ly390jz35-packager-linux-x64.drv'...
trying https://github.com/shaka-project/shaka-packager/releases/download/v2.6.1/packager-linux-x64
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 6745k 100 6745k 0 0 4487k 0 0:00:01 0:00:01 --:--:-- 16.9M
copying path '/nix/store/l0fvy72hpfdpjjs3dk4112f57x7r3dlm-gcc-wrapper-12.2.0' from 'https://cache.nixos.org'...
copying path '/nix/store/b09v23lirgvci3wzszh22mbkdfj0h0yq-stdenv-linux' from 'https://cache.nixos.org'...
building '/nix/store/668fy2xd3lc8anmqd6jmhk2a1vrzv79x-shaka-packager-2.6.1.drv'...
patching sources
configuring
no configure script, doing nothing
building
no Makefile or custom buildPhase, doing nothing
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/7i41i7mbhabnffykx8ns6wpk0p7y993r-shaka-packager-2.6.1
shrinking /nix/store/7i41i7mbhabnffykx8ns6wpk0p7y993r-shaka-packager-2.6.1/bin/shaka-packager
patchelf: cannot find section '.dynamic'. The input file is most likely statically linked
checking for references to /build/ in /nix/store/7i41i7mbhabnffykx8ns6wpk0p7y993r-shaka-packager-2.6.1...
patchelf: cannot find section '.dynamic'. The input file is most likely statically linked
patching script interpreter paths in /nix/store/7i41i7mbhabnffykx8ns6wpk0p7y993r-shaka-packager-2.6.1
stripping (with command strip and flags -S) in /nix/store/7i41i7mbhabnffykx8ns6wpk0p7y993r-shaka-packager-2.6.1/bin
/nix/store/7i41i7mbhabnffykx8ns6wpk0p7y993r-shaka-packager-2.6.1
And then run it:
$ /nix/store/7i41i7mbhabnffykx8ns6wpk0p7y993r-shaka-packager-2.6.1/bin/shaka-packager --version shaka-packager version v2.6.1-634af65-release
mkDerivation provides a bunch of nice things described in the nix derivation
docs. It creates
a temporary directory for the build (automatically removed after), clears the
environment, sets a bunch of relevant environment variables, removes any
existing output path, writes stdout and stderr to /nix/var/log/nix, sets
timestamps to 00:00:01 1/1/1970 UTC (for reproducible builds), and also:
If the build was successful, Nix scans each output path for references to input paths by looking for the hash parts of the input paths. Since these are potential runtime dependencies, Nix registers them as dependencies of the output paths.
Isn't that interesting!
You can also see in the build log (scroll up a little) that it tries to "patch script interpreter paths". That's right, executables are interpreted also, sort of: dynamically-loaded ones need a.. dynamic loader! The kind we built in making our own executable packer.
A simple dev shell
Before we move on to the more modern stuff, let's note that derivations aren't
the only thing a .nix file can evaluate to.
Well... of course it's not: you can have .nix files that evaluate to a single
lambda for example, so that you can split out utility functions in other files.
But as far as "top-level" things go, it's mostly derivations and.. dev shells!
Let's remove our default.nix file for now – it doesn't actually build
catscii anyway, just shaka-packager, which is completely unrelated.
And let's now create a shell.nix file with this:
# in shell.nix
{ pkgs ? import <nixpkgs> { } }:
with pkgs;
mkShell
{
packages = [ flyctl ];
}
Remember flyctl? That's what we use to deploy our app to fly.io. Surely
we'll need that.
Let's make sure we don't have it installed in our Ubuntu VM (we shouldn't, it certainly doesn't ship with a default Ubuntu install and we reset our VM earlier):
$ which fly $ which flyctl
Okay! And now the magic:
$ nix-shell [nix-shell:~/catscii]$ fly version flyctl v0.0.456 linux/amd64 Commit: v0.0.456 BuildDate: 1970-01-01T00:00:00Z [nix-shell:~/catscii]$ (pressing Ctrl-D) exit
Hurray! We have a dev shell with flyctl in there.
Thanks to my sponsors: Chris Biscardi, Eugene Bulkin, Marcin Kołodziej, Corey Alexander, Jarek Samic, Sam, Jonathan Adams, Raine Tingley, Marc-Andre Giroux, Mike Cripps, Rufus Cable, C J Silverio, Johnathan Pagnutti, David Souther, Integer 32, LLC, Steven Pham, Johan Saf, Hadrien G., Guillaume E, Jean-David Gadina and 227 more
If you liked what you saw, please support my work!
Donate on GitHub
Donate on Patreon
Here's another article just for you:
5 min
What's a ktls
I started work on ktls and ktls-sys, a pair of crates exposing Kernel TLS offload to Rust, about two years ago.
kTLS lets the kernel (and, in turn, any network interface that supports it) take care of encryption, framing, etc., for the entire duration of a TLS connection... as soon as you have a TLS connection.
For the handshake itself (hellos, change cipher, encrypted extensions, certificate verification, etc.), you still have to use a userland TLS implementation.