fasterthanlime

Generating a docker image with nix

Thanks to my sponsors: belzael, Damir Vandic, Astrid, Nicholas, Nicolas Riebesel, Zeeger Lubsen, Jack Duvall, Santiago Lema, Andrew Neth, Menno Finlay-Smits, Aiden Scandella, Pete LeVasseur, C J Silverio, Manuel Hutter, DaVince, Olivia Crain, Niels Abildgaard, Geoffroy Couprie, Mattia Valzelli, Jack Maguire and 266 more belzael, Damir Vandic, Astrid, Nicholas, Nicolas Riebesel, Zeeger Lubsen, Jack Duvall, Santiago Lema, Andrew Neth, Menno Finlay-Smits, Aiden Scandella, Pete LeVasseur, C J Silverio, Manuel Hutter, DaVince, Olivia Crain, Niels Abildgaard, Geoffroy Couprie, Mattia Valzelli, Jack Maguire, budrick, Matt Heise, Arjen Laarhoven, Sam Leonard, Hadrien G., Berkus Decker, Mathias Brossard, Marie Janssen, Tabitha, Ronen Cohen, David White, Cole Kurkowski, Henrik Tudborg, Boris Dolgov, Jesse Luehrs, Evan Relf, Yves, Mario Fleischhacker, milan, Horváth-Lázár Péter, Aleksandre Khokhiashvili, std__mpa, Simon Menke, Xavier Groleau, qrpth, Michał Zalewski, Ross Williams, Mason Ginter, Neil Blakey-Milner, Adam Gutglick, Cass, Herman J. Radtke III, Nicholas Orta, Daniel Silverstone, Jean-David Gadina, Joshua Roesslein, Yuriy Taraday, Jelle Besseling, Dylan Anthony, Borys Minaiev, Antoine Boegli, Christoph Grabo, Ryan, Enrico Zschemisch, Toon Willems, Senyo Simpson, Vladimir, Tiziano Santoro, ZacJW, Egor Ternovoi, genny, Max Heaton, Marcus Brito, hgranthorner, Dominik Wagner, old.woman.josiah, Daniel Strittmatter, Antoine Rouaze, Xirvik Servers, Mateusz Wykurz, Jonathan Adams, Shane Lillie, notryanb, Michał Bartoszkiewicz, Andy F, Scott Sanderson, Romet Tagobert, Òscar Pérez, Kamran Khan, callym, Philipp Gniewosz, Ben Wishovich, Luiz Ferraz, Radu Matei, clement, James Brown, Sean Bryant, johbii, Dave Minter, Corey Alexander, René Ribaud, Adrián Garnier Artiñano, WeblWabl, jer, Guilherme Neubaner, Zaki, Lucille Blumire, Mikkel Rasmussen, Philipp Hatt, Antoine PESTEL-ROPARS, Tyler Schmidtke, traxys, Jan De Landtsheer, Aljaz Erzen, xales, Taneli Kaivola, Marcus Griep, Luke Konopka, Matěj Volf, Andrew Henshaw, Max Bruckner, Sawyer Knoblich, ofrighil, Gorazd Brumen, Chirag Jain, Michael, Beat Scherrer, Wyatt Herkamp, Matt Jadczak, Justin Smith, Kyle Lacy, Aalekh Patel, Andy Gocke, Marcus Griep, Marco Carmosino, Braidon Whatley, Adam Lassek, ACRL, anichno, Dirkjan Ochtman, Julian Schmid, Ivo Murrell, playest, Peter Shih, Seth, Justy, bbutkovic, Josh Triplett, John Horowitz, Luke Yue, Laine Taffin Altman, Matt Jackson, Ben Mitchell, Duane Sibilly, Michael Mrozek, Sylvie Nightshade, Lena Schönburg, Geoffrey Thomas, Cole Tobin, Mark Old, avborhanian, Isak Sunde Singh, Scott Steele, Thor Kamphefner, Jake Demarest-Mays, Lennart Oldenburg, prairiewolf, Valentin Mariette, Ahmad Alhashemi, Matt Campbell, Mark, Chris Walker, you got maiL, compwhizii, David Barsky, Paige Ruten, Bob Ippolito, Victor Song, Ula, ShikChen, Samit Basu, Chris Emery, Marc-Andre Giroux, Guillaume E, Yann Schwartz, Max von Forell, Mike English, Guillaume Demonet, Ives van Hoorne, Benjamin Röjder Delnavaz, Marcin Kołodziej, Wojciech Smołka, Richard Stephens, Pete Bevin, Marky Mark, AdrianEddy, Guy Waldman, Dom, Raine Godmaire, Elijah Voigt, Mark Tomlin, Kai Kaufman, Joseph Montanaro, villem, Rufus Cable, Blake Johnson, Jon Gjengset, medzernik, Colin VanDervoort, Daniel Wagner-Hall, jatescher, Eugene Bulkin, Richard Pringle, Justin Ossevoort, Raphaël Thériault, Ian McLinden, David E Disch, G, Noel, Makoto Nakashima, Michal Hošna, Michael Alyn Miller, Urs Metz, Christian Bourjau, Thehbadger, SeniorMars, Daniel Papp, Yufan Lou, Zachary Thomas, Matthew T, Mikko Leppänen, Sindre Johansen, Chris, Brandon Piña, Chris Thackrey, John VanEnk, Paul Marques Mota, Lyssieth, Tobias Bahls, Malik Bougacha, L0r3m1p5um, Jörn Huxhorn, James Leitch, Torben Clasen, Nyefan, Alan O'Donnell, Andronik, Jim, Dimitri Merejkowsky, Jan-Stefan Janetzky, Paul Horn, Zoran Zaric, Elnath, teor, Olly Swanson, David Cornu, Marty Penner, Alex Rudy, Integer 32, LLC, e9zaktw1, Johnathan Pagnutti, Tyler Bloom, Hamilton Chapman, Sarah Berrettini, Walther, Steven Pham, Diego Roig, Tanner Muro, Gioele Pannetto, Ronen Ulanovsky, Josiah Bull, Geoff Cant, Brooke Tilley, James Rhodes, Garret Kelly, Elendol, Alex Krantz, Christopher Valerio, Beth Rennie, Luuk, Stephan Buys, Timothée Gerber, Chris Biscardi, Chris Sims, pinkhatbeard, Romain Kelifa

There it is. The final installment.

Over the course of this series, we’ve built a very useful Rust web service that shows us colored ASCII art cats, and we’ve packaged it with docker, and deployed it to https://fly.io.

We did all that without using nix at all, and then in the last few chapters, we’ve learned to use nix, and now it’s time to tell docker build goodbye, along with this whole-ass Dockerfile:

So, let’s begin with:

And let out a sigh of relief.

Before we need to be able to build a Docker image, we need to be able to build an executable.

Remember when we learned nix, how we were able to do nix build? Well it’s time to do that for catscii.

But luckily, we’ve set up everything we need, and so it’ll be painless.

We’ll use the crane nix library to build it.

The plan is to:

• add it as an input
• call one of its functions
• assign that to something under packages in our flake output

Good? Good.

Let’s start by adding it:

Before we go any further, vibe check:

Okay! crane depends on nixpkgs, rust-overlay, and flake-utils, which we already have, so, let’s deduplicate those:

Alright! That’s better.

Let’s keep going:

And that’s it! We’re good to go!

Oh, um. Almost. We do have a private registry in there, which I did on purpose, specifically to show how we could solve that problem with Docker and with crane.

Because we already have a secrets/shipyard-token file (protected by git-crypt), it’s actually fairly easy to get out of. Crane even has a whole docs page about it.

Replace the craneLib = line with these:

And this time, we’re actually good to go!

Cool Bear's hot tip

If this seems like a terrible no-good workaround until some standard is decided and implemented everywhere, that’s because it is!

You can check the tracking issue to see where the discussion is at by the time you actually read this.

Anyway, with that, nix build actually does work:

And in result, we have our binary!

Which only depends on things provided by nix:

And runs fine!

Here it is! The last thing we have to do.

nix comes with facilities to generate Docker images, which we should really call OCI images, without using Docker at all: those images are, after all, “just” tarballs with a well-known structure and some JSON manifests.

Let’s first try to make a docker image with just our binary: we know it won’t be enough, since we need to have the geo-ip database in there, but it’s a start.

In our outputs’ let block, next to bin, we can define a new key:

Let’s try building it and see what we got in there!

Huh, that was quick!

Yeah! The bin target was already built, and since everything is content-addressed, and none of the inputs changed, it really did just generate the Docker image from that.

Let’s see what the result file is now:

A symbolic link to a tarball. We can load this into our local docker registry with docker load:

…ah, we haven’t installed docker yet! If you needed proof that nix truly can build docker image without docker, I guess this is it.

In a real-world scenario, I’d probably want to have docker installed locally to be able to run the image, but for now let’s look at it another way: with the dive utility.

Let’s add it to our dev shell:

For some reason, pressing Enter didn’t do the trick for me this time, I had to close the shell and open a new one.

Now, dive generally takes a name:tag reference, which goes into the local Docker registry. But we don’t have Docker! It also supports the docker-archive:// protocol, but that requires .tar files, whereas we have a .tar.gz file.

So, let’s gunzip it first:

And we’re presented with a TUI (text-user interface), which we’ve actually already used in Part 6:

Wait… it’s only 58MB?

Yes it is! Remember how we struggled to get it below 100MB when we wrote the Dockerfile by hand? Well, here’s our first try, and it’s almost half.

There’s still a couple things missing from there, but nothing too big.

Note that we could’ve tried harder with the Dockerfile: the base Ubuntu system takes some space, we could’ve used a distroless image as a base - they even have an example for Rust!

But we would have to figure out how to ship sqlite, for example. I’m sure there’s a way, I just… I’m just not sure why I’d bother, since nix lets us solve both the “build a binary that can run anywhere” and “generate a small docker image while we’re at it”.

Before we deploy it to production, let’s make sure it runs locally, which is one of the big selling points of Docker.

Installing the latest Docker on Ubuntu was a bit of a hassle last time.

Can we just add docker to our dev shell and see if it works?

Out of curiosity, I tried that. First off, it’s not a small download (half a gig). Secondly, when you install Docker via Ubuntu packages, it does a couple things, like set up a systemd service to start the docker daemon, and set up user groups.

Without that, we have to start the daemon ourselves, and it’s more subtle than you’d think.

The daemon does need to run, so docker info doesn’t work out of the box:

Starting the docker daemon as an unprivileged user is possible in theory, it’s called “rootless mode”, but it’s not the default, so just running “dockerd” doesn’t work:

There’s a dockerd-rootless executable in $PATH, but it ends up failing to find newuidmap.

That executable is in nixpkgs#shadow, but adding it doesn’t work any better:

I suppose this all works swimmingly on NixOS, but it’s a bit more delicate intruding into another distribution like that.

Anyway, we can run dockerd as root:

Well, not like that - sudo sanitizes the environment, at least on Ubuntu. You can configure that, but let’s not:

Now, the docker daemon is running, but a non-root user cannot talk to it:

As root though, we can!

Violently chown-ing /var/run/docker.sock is an option, although not a great one:

Really, you’re probably better off installing Docker from Ubuntu packages. But if you really wanted to have docker in your dev shell, this is how you could do it.

So, as promised, we can load our built image into the local docker registry with docker load:

And then we can try to run it!

Ah yes! We must set a few environment variables.

If we had to do this a lot, a neat way to do it would be to:

• Split those environment variables from .envrc into .env
• Use the dotenv instruction to load the .env from the .envrc
• Pass --env-file to docker run, or set up a docker-compose file

But for now, let’s juse use -env (-e) to pass a few environment variables by hand:

Ah, containerization in action: it can’t read the geoip country database because… it’s stuck in its little container, and db/GeoLite2-Country.mmdb is outside of it.

We need a tiny bit more nix to fix this.

So, we already have the geoip database locally, so we can just copy it from the nix flake.

Mhh before we do that though - surely someone has already packaged that database into nixpkgs, right?

Ehhh I don’t know if the license terms permit th- oh would you look at that, there it is, as clash-geoip.

Well, I suppose we can try!

If the casing of Cmd and Env seem weirdly out-of-place, it’s because they map to the JSON format expected by the Docker Engine API, and Docker is written in Go, where public fields are UpperCamelCase.

Nice bit of trivia there but, wait, we can refer to any package? Just like that?

I’m doing what instinctively feels right!

Let’s see if it actually works:

Ah! Right. It has an unfree license.

Exporting the environment variable and using --impure seems… icky. Alternative “b)” seems to only apply to NixOS, not nixpkgs, and I don’t love the idea of changing the global config either… maybe we can fix it in the flake itself?

Hey, it built!

Hey, it loaded!

Hey, it ran!

Hey, something went wrong!

The app is running in sort of a weird environment here: it’s a release build, so sentry thinks it’s “production”, but it also has Sentry & Honeycomb keys for “development”.

The codepath we’re hitting is in root_get_inner, here:

We’re not printing anything useful to stdout or stderr, but we are attaching the error to the active opentelemetry span.

So if we open Honeycomb, we should be able to see it.

And yet, I see nothing.

Running the service again with a little more logging enabled reveals the source of the problem:

(This happened while running curl http://localhost:8080 from another shell).

In case your brain doesn’t have a built-in JSON.parse function yet, here’s a human-friendlier version of this:

Oh! It can’t find the CA certificates to verify cataas.com’s certificate!

Why yes, exactly!

When establishing a TLS connection, a lot of things happen (as wonderfully explained in The Illustrated TLS 1.3 Connection), but an important one is: the server presents a chain of certificates, starting with a certificate for the domain in question (here, cataas.com), and ending with something you trust.

At the time of this writing, the certificate chain for cataas.com goes:

• ISRG Root X1
  • R3
    • cataas.com

And how do we know we trust “ISRG Root X1”, a CA (“Certificate Authority”) from Let’s Encrypt?

Well, we have a file on disk that says so!

That’s why, if you do:

…it works!

But when our program inside the Docker image tries to do so, it does not work. In fact, even outside the Docker image it doesn’t work, because it links against a build of openssl that looks for CA certificates not in /etc/ssl/certs (like an Ubuntu package would), but in /nix/store/dsf1m9azqqz6c3nqj9yk0nnardqmaia0-openssl-3.0.8/etc/ssl/certs (like a nix package would).

So, long story short, we need to pull that into the Docker image:

Wait wait, but why didn’t we see it in Honeycomb?

Well, how do you think events are sent to Honeycomb?

Over HTTPS? Oh…. Ohhhhhhhhhhh!

Yeah.

Let’s build this once again:

Say, this seems a little wasteful, no? Writing the whole image to disk, only to import it back into the docker registry right after?

You’re right! We can live with that, of course, but say we didn’t, say we wanted to really obsess over that, we could switch to streamLayeredImage.

Let’s do that now!

Loading the image into the docker registry is now a bit different!

result is no longer a gzipped tarball (.tar.gz), but it is a script that streams out an uncompressed tarball (.tar).

So, we can call it and pipe the output to docker load.

You’ll notice this image has layers (we can control how many layers it has with maxLayers), so pushing updates should be faster, since dependencies like openssl, glibc, zlib etc. seldom change!

Note that our image is still delightfully small: dive catscii:latest shows a total image size of 55MB. Adding ca-certs only cost us 486 kB:

Let’s run it just like we did before:

And finally, we have our sweet ascii art cats again!

Deploying works much the same as it did before - we still have to opt out of fly.io’s “remote builders”, since we really do want to push a local image there.

Our Justfile could look like this:

But of course, we’ll want to to add just and flyctl to our dev shell first:

Then,fly auth login:

And… let’s just check our fly.toml one last time.. oh, right! We have that in there:

We should remove that GEOLITE2_COUNTRY_DB line from there - that environment variable is baked into the Docker image now.

And then we’re good to go!

Comment on /r/fasterthanlime