fasterthanlime

Making a dev shell with nix flakes

Thanks to my sponsors: Menno Finlay-Smits, Xirvik Servers, Mathias Brossard, Neil Blakey-Milner, notryanb, Ben Mitchell, Ivo Murrell, bbutkovic, Raphaël Thériault, Jon Gjengset, Ross Williams, Duane Sibilly, Chris Emery, Hadrien G., Adam Gutglick, Geoffrey Thomas, Dimitri Merejkowsky, Romet Tagobert, Marcus Brito, Blake Johnson and 266 more Menno Finlay-Smits, Xirvik Servers, Mathias Brossard, Neil Blakey-Milner, notryanb, Ben Mitchell, Ivo Murrell, bbutkovic, Raphaël Thériault, Jon Gjengset, Ross Williams, Duane Sibilly, Chris Emery, Hadrien G., Adam Gutglick, Geoffrey Thomas, Dimitri Merejkowsky, Romet Tagobert, Marcus Brito, Blake Johnson, Raine Godmaire, Elnath, Mason Ginter, Brooke Tilley, ShikChen, Lucille Blumire, Tyler Bloom, David White, DaVince, traxys, James Brown, C J Silverio, Michael, Benjamin Röjder Delnavaz, Christopher Valerio, Eugene Bulkin, Senyo Simpson, John Horowitz, std__mpa, Olly Swanson, Dylan Anthony, Ronen Cohen, Michael Mrozek, Jörn Huxhorn, ofrighil, Adam Lassek, Yves, Paul Marques Mota, Lena Schönburg, Marie Janssen, Isak Sunde Singh, Dave Minter, Shane Lillie, Valentin Mariette, Josiah Bull, Sylvie Nightshade, Zoran Zaric, Egor Ternovoi, Chris Sims, Jake Demarest-Mays, Yuriy Taraday, Alex Krantz, Alan O'Donnell, Mark, Alex Rudy, Marty Penner, Yufan Lou, Mikko Leppänen, Max Bruckner, Joshua Roesslein, Chris Walker, Christian Bourjau, Yann Schwartz, Mark Tomlin, Mario Fleischhacker, Damir Vandic, Marcus Griep, Manuel Hutter, Seth, Philipp Hatt, Daniel Silverstone, Zachary Thomas, Radu Matei, Henrik Tudborg, Niels Abildgaard, Guillaume Demonet, Nyefan, Joseph Montanaro, Andrew Neth, Herman J. Radtke III, Marcin Kołodziej, genny, Paul Horn, Guillaume E, Lennart Oldenburg, belzael, Aalekh Patel, Daniel Papp, Brandon Piña, Luke Konopka, Matt Jadczak, AdrianEddy, clement, Wyatt Herkamp, Garret Kelly, Matt Heise, Chris Thackrey, Laine Taffin Altman, Jack Duvall, John VanEnk, Guy Waldman, ACRL, Beth Rennie, Antoine Rouaze, Horváth-Lázár Péter, Sean Bryant, Andronik, Aiden Scandella, Tanner Muro, Stephan Buys, teor, Daniel Wagner-Hall, budrick, Noel, Ahmad Alhashemi, Beat Scherrer, ZacJW, Mike English, Timothée Gerber, villem, jatescher, Michał Zalewski, Andy F, playest, Antoine Boegli, old.woman.josiah, Jim, jer, you got maiL, Guilherme Neubaner, L0r3m1p5um, Ula, qrpth, WeblWabl, Pete LeVasseur, Tabitha, Ben Wishovich, Nicholas Orta, Kamran Khan, Braidon Whatley, compwhizii, johbii, Johnathan Pagnutti, Josh Triplett, Thehbadger, Lyssieth, James Rhodes, Malik Bougacha, Matěj Volf, Julian Schmid, James Leitch, Philipp Gniewosz, callym, Cass, Òscar Pérez, Jean-David Gadina, Mattia Valzelli, Ian McLinden, Marky Mark, Cole Kurkowski, Chris, Sindre Johansen, Justin Smith, Justin Ossevoort, Elijah Voigt, Luuk, Richard Pringle, SeniorMars, Richard Stephens, Jan-Stefan Janetzky, Kai Kaufman, Luke Yue, Gioele Pannetto, Kyle Lacy, Ryan, Jan De Landtsheer, pinkhatbeard, Steven Pham, Jonathan Adams, Pete Bevin, Corey Alexander, Scott Sanderson, Adrián Garnier Artiñano, Jack Maguire, e9zaktw1, Chirag Jain, Zeeger Lubsen, Max Heaton, Michael Alyn Miller, Antoine PESTEL-ROPARS, Thor Kamphefner, Gorazd Brumen, Jesse Luehrs, Sam Leonard, Wojciech Smołka, Victor Song, David Barsky, Berkus Decker, Borys Minaiev, Daniel Strittmatter, Michał Bartoszkiewicz, Geoff Cant, David E Disch, Cole Tobin, Vladimir, Boris Dolgov, Diego Roig, xales, Elendol, Zaki, Sarah Berrettini, Andrew Henshaw, Torben Clasen, Olivia Crain, Chris Biscardi, hgranthorner, Dirkjan Ochtman, Arjen Laarhoven, Mikkel Rasmussen, Bob Ippolito, Aljaz Erzen, Mark Old, medzernik, G, Geoffroy Couprie, milan, Max von Forell, Peter Shih, Marco Carmosino, Nicolas Riebesel, Aleksandre Khokhiashvili, Christoph Grabo, Nicholas, Andy Gocke, Justy, Matt Jackson, Simon Menke, Tyler Schmidtke, Luiz Ferraz, Toon Willems, anichno, Mateusz Wykurz, Jelle Besseling, Matthew T, Marcus Griep, Romain Kelifa, Sawyer Knoblich, Dom, René Ribaud, David Cornu, Taneli Kaivola, Urs Metz, Samit Basu, Ives van Hoorne, Evan Relf, Xavier Groleau, Santiago Lema, Makoto Nakashima, Rufus Cable, Dominik Wagner, Hamilton Chapman, Paige Ruten, avborhanian, Tobias Bahls, Colin VanDervoort, Ronen Ulanovsky, Integer 32, LLC, Matt Campbell, Scott Steele, Michal Hošna, prairiewolf, Walther, Tiziano Santoro, Marc-Andre Giroux, Enrico Zschemisch, Astrid

In the previous chapter, we’ve made a nix “dev shell” that contained the fly.io command-line utility, “flyctl”.

That said, that’s not how I want us to define a dev shell.

Our current solution has issues. I don’t like that it has import <nixpkgs>. Which version of nixpkgs is that? The one you’re on? Who knows what that is.

Also, we haven’t really seen a mechanism to use .nix files from elsewhere.

As you may be beginning to suspect, building Rust code (or a Node.js app, etc.) with nix is actually non-trivial, because we need to hash the whole input, download everything in advance (did I mention builders don’t have access to the internet? to make double-plus-sure everything is reproducible?), use the right toolchain, etc.

So we won’t be using mkDerivation at all - we’ll be using a third-party Nix library. And we need a way to refer to it cleanly.

Those problems are solved with nix flakes - part of the new, experimental, don’t-cry-if-it-breaks stuff.

A nix flake is just a regular nix file! It needs to evaluate to a set with inputs and outputs, and some others if we feel fancy.

So, let’s remove shell.nix, and create a mostly-empty flake.nix:

This isn’t enough, and any flakes-enabled commands (the new stuff, the nix subcommands, not the nix-dash-something commands) should let us know, like, nix flake check:

Wait, no, that’s… that’s not the error I expected. Due to how flakes work, the flake.nix file needs to be staged into our Git repository, so let’s do that:

So, fine, let’s add outputs:

Ah, outputs should be a function. A function of its inputs, no doubt:

And just like that, we’ve made our first, completely useless flake.

Let’s make it useful! One thing a flake can have as an output is… an executable! We can use derivation just like we did before!

The only difference is, now we specify where nixpkgs comes from:

And if we do nix build, it builds just like before.

Cool Bear's hot tip

Note that this is nix build with a space, as in “the build subcommand of nix”, not “the nix-build command”.

And in flake.lock, our inputs are locked:

…which we can update with nix flake update.

We can also show the outputs our flake has with nix flake show:

However, this isn’t a very good flake, yet.

The code is redundant all over.

Instead of referring to inputs.stuff, we could destructure the arguments of outputs:

And then, we can use let to great effect:

There. Isn’t that much tidier?

But see that system = system there? That’s not very DRY.

nix has a thing for this: inherit. In fact, we can inherit a bunch of things:

This version of the flake does exactly the same, but it’s clearer! (We didn’t really need to inherit name and src, but I wanted to show it off).

Finally, our flake doesn’t only work on x86_64-linux. We could define it manually for a bunch of other targets, or we could use a nix library to do it for us!

Any flake-enabled command updates flake.lock (since we have a new input), nix flake show now shows that we can build for a bunch of common targets:

Note that nix build still defaults to building.. the default package, for our current platform.

We can tell, if we ask it for extra debugging output with --debug:

We can also build it more explicitly, by passing the path of the thing to build:

Or we can be more specific:

Or even more specific:

Wait, does that mean we can cross-compile?

It’s, uhhh, complicated. I haven’t cracked it at the time of this writing.

If you try the obvious thing, it fails somewhat gracefully, at least:

Just for completeness’ sake, let’s see how our shaka-packager package would look like as flake.

Instead of removing our default package, we’ll simply add it as another output:

There! Isn’t that neat?

And we can build it with:

Wait.. isn’t it downloading an x86_64-linux binary for shaka-packager?

Oh, you’re right! This one should not be for all targets.

Well, what a good excuse to learn about the //! It returns the union of two sets, preferring the right-hand-side if there’s conflicts.

Note that this doesn’t do a deep merge, so we need to expand things a little:

And now we have the expected output:

Note that of course, shaka-packager has builds for all these platforms, so we could have our flake translate system to the proper download URL, with the proper sha256.

But this is not at all what we’re trying to do here, so let’s not.

Let’s start our flake.nix over, since we got distracted with learning nix and stuff.

There.

The most pressing thing we need in our dev shell is cargo / rustc - remember that? Way at the beginning of the article?

That’s still what we’re trying to do.

But at least, now we know enough of nix to do it easily, right?

Well….. almost. We haven’t talked about overlays yet. Remember how we said the result of an import <nixpkgs> is a function that can be called with a set?

Yes?

Well, that’s how we do overlays. Which provide packages that aren’t in nixpkgs by default.

For example, github:oxalica/rust-overlay provides various versions of the Rust toolchain.

We can do, for example:

And then:

Ah, finally!

Wait no no no, hold on a minute.

What. What? It does the thing!

It certainly does, but did you notice how it added inputs:

Well?

We already have those! And now we have some duplicates!

Ah, well. Isn’t that kinda inherent to the flake model?

Mhhhhhsorta but we can fix it:

Let’s check again:

Ah, much better!

Making sure you have the same inputs, down to the commit is important to avoid “rebuild the world” situations, as Pyrox explains on GitHub.

So, nix-shell is the old stuff, nix develop is the new, flake-enabled stuff. Notably, it’s much faster than nix-shell, so it’s not just novelty for novelty’s sake.

(Also, for me at least, it doesn’t change the shell prompt, which can be a bit confusing - it’s hard to tell when you’re in the dev shell or not).

The next thing I’m unhappy with, is that it gave us the latest stable:

…when we have a rust-toolchain.toml file.

We can fix that too!

And now, we have the proper version:

Yay! Finally!

We’re not quite done yet, though. There’s still a couple things missing, but before we move forward, let’s set up a thing that “enters the dev environment” anytime we cd into the directory.

We’ll use the direnv tool for that, just like we did in part 4.

We could install it with apt, or… we could use nix profile! Let’s try the latter:

Just like last time, we’ll set up the bash hook for it (it works similarly for zsh and others, if you couldn’t resist switching to them already):

We already have an .envrc file in there, and it’s already encrypted by git-crypt, but… we haven’t decrypted it yet.

Our .envrc is a little hard to read right now:

So, remember when I told you to save your git-crypt-key file? Time to copy it somewhere into your VM once again, and then install git-crypt.

Cool Bear's hot tip

If you haven’t saved your git-crypt-key file, bummer, but you can find or regenerate honeycomb tokens etc., and re-write the file from scratch, generate a new git-crypt-key, and lock it again with git-crypt.

Wait, wouldn’t it make sense to have it as part of the dev shell?

Mhhh not really: the instructions to activate the dev shell when cd-ing into the directory are in the .envrc, so it needs to be already decrypted by that time.

So “having git-crypt already installed before you start hacking on catscii” is going to be a hard requirement for us.

Let’s proceed:

That looks better:

Before we approve it, let’s just add one thing to the beginning of our .envrc file:

This sets up nix-direnv, which may have a newer version by the time you read this.

Cool Bear's hot tip

You also probably want to add /.direnv to your .gitignore, because the next steps will create this directory and put some symlinks in there.

Now we can run direnv allow and let it do its thing:

How interesting! We can see everything it exports here: we recognize some of the things from our .envrc file, like GEOLITE2_COUNTRY and HONEYCOMB_API_KEY, but also some definitely-nix-related things like NIX_BINTOOLS, AR, NM, STRIP, which are all part of the toolchain pulled in by Rust:

That’s not quite enough though - we do want to be able to develop catscii in VS Code, with rust-analyzer, which invokes cargo check, so it must have access to a Rust toolchain, which we’re getting from nix.

So how do we get VS Code extensions to find our nix-installed Rust toolchain? Well, there’s a direnv extension! That works with the SSH Remote extension we’re already using. And that we’ve also talked about in Part 4.

After installing it, it’ll tell us the environment was updated, and ask if we want to restart extensions, which we do (so we can click “Restart”).

If you now try to run cargo check, you may run into a cargo error:

And here, you have two options: either you have backed up your SSH key way back in part 7, or you haven’t, and then you can make a new SSH key and add it to Shipyard.

After that, cargo check gets a lot further, because it manages to clone/update the private repository.

But it wants libssl-dev! Very well then, let’s add it to our flake.nix:

Cool Bear's hot tip

Remember, you can use the NixOS package search site to find what you’re looking for. It’s pretty good.

Say amos, what’s the difference between buildInputs and nativeBuildInputs?

Well, for dev shells, there’s none. But buildInputs are things you’ll also need at run-time, whereas nativeBuildInputs are things you only need at compile-time.

It’s easier to remember if you think of cross-compiling: if you’re building from x86_64 for aarch64, you might be using a “native” GCC for x86_64, that generates aarch64 binaries, hence: native build inputs.

After saving flake.nix, simply hitting Enter will re-evaluate our flake.nix:

…and we should have pkg-config in $PATH, which should be able to find openssl:

And now, it’s a game of cat and mouse! Let’s keep going:

Huh. I could’ve sworn…

Yeah, where’s sqlite?

Mhhh. Mhhh!

Ahh! I thought so.

I would’ve expected it to fail earlier with a nicer message, because, after all, pkg-config cannot find it:

But it seems the libsqlite3-sys maintainers chose to spray and pray in that scenario. Ah well.

Anyway, a quick search reveals that in nixpkgs, sqlite 3 is just named “sqlite” (I have yet to encounter a project that uses sqlite 1 or 2, after all), so we can add it to our buildInputs in flake.nix:

Press Enter in our shell to re-evaluate the flake, and then cargo build succeeds!

It is, however, missing the MaxMindDB database:

…so we need to copy it over again. I used scp:

And with that, it runs!

And we’re almost done with this adventure. All we have to do now is rip out that Dockerfile. It’s going to be glorious. I can smell it from here.

Comment on /r/fasterthanlime

Here's another article just for you:

My ideal Rust workflow

Oct 26, 2021

57 min #rust · #release-engineering

Writing Rust is pretty neat. But you know what’s even neater? Continuously testing Rust, releasing Rust, and eventually, shipping Rust to production. And for that, we want more than plug-in for a code editor.

We want… a workflow.

Why I specifically care about this

Cool Bear's hot tip

This gets pretty long, so if all you want is the advice, feel free to jump to it directly.

Read more