fasterthanlime

Making a dev shell with nix flakes

Thanks to my sponsors: callym, Christian Bourjau, Mark Tomlin, David White, Bader, Jon Gjengset, Stephan Buys, Justin Smith, Olivia Crain, Alex Rudy, Laine Taffin Altman, johbii, Colin VanDervoort, Mark, Geoff Cant, Isak Sunde Singh, Tanner Muro, Rufus Cable, Brandon Piña, Matt Jackson and 266 more callym, Christian Bourjau, Mark Tomlin, David White, Bader, Jon Gjengset, Stephan Buys, Justin Smith, Olivia Crain, Alex Rudy, Laine Taffin Altman, johbii, Colin VanDervoort, Mark, Geoff Cant, Isak Sunde Singh, Tanner Muro, Rufus Cable, Brandon Piña, Matt Jackson, Zachary Thomas, Andy Gocke, Evan Relf, Paige Ruten, Scott Sanderson, Samit Basu, Romet Tagobert, playest, Jake Demarest-Mays, Yann Schwartz, Mario Fleischhacker, Michael, Mark Old, compwhizii, Nicholas, Òscar Pérez, Senyo Simpson, Marcus Brito, Taneli Kaivola, Antoine Boegli, xales, Aleksandre Khokhiashvili, belzael, Ula, Mikkel Rasmussen, Seth, budrick, Steven Pham, Justin Ossevoort, traxys, jatescher, Christoph Grabo, James Leitch, Andronik, Braidon Whatley, Nicolas Riebesel, Wojciech Smołka, Pete Bevin, Yufan Lou, Corey Alexander, Luiz Ferraz, C J Silverio, David Barsky, Romain Kelifa, Luke Yue, Horváth-Lázár Péter, SeniorMars, Marco Carmosino, Dominik Wagner, clement, anichno, Thehbadger, Elnath, Diego Roig, medzernik, Sean Bryant, Garret Kelly, Marcus Griep, Victor Song, Nyefan, Guy Waldman, Eugene Bulkin, Matt Heise, Jim, Manuel Hutter, G, Adrián Garnier Artiñano, Mathias Brossard, Boris Dolgov, Lucille Blumire, Kai Kaufman, Brooke Tilley, Integer 32, LLC, Enrico Zschemisch, Kyle Lacy, Marcin Kołodziej, Sawyer Knoblich, Xavier Groleau, Walther, Guillaume Demonet, Marky Mark, Joseph Montanaro, Sam Leonard, old.woman.josiah, Guillaume E, Olly Swanson, std__mpa, Xirvik Servers, Simon Menke, Josh Triplett, Astrid, pinkhatbeard, Yuriy Taraday, Hadrien G., Urs Metz, René Ribaud, Tabitha, Das Gürteltier, Jan De Landtsheer, David E Disch, Michael Mrozek, Henrik Tudborg, ShikChen, Aljaz Erzen, Tiziano Santoro, Jelle Besseling, Matt Campbell, Daniel Silverstone, Herman J. Radtke III, Zaki, Josiah Bull, Marcus Griep, Elendol, Dimitri Merejkowsky, Richard Pringle, Alex Krantz, Richard Stephens, Lena Schönburg, Ahmad Alhashemi, Duane Sibilly, Adam Lassek, Cole Tobin, Ryan, Toon Willems, Damir Vandic, Cole Kurkowski, Marie Janssen, Chris Walker, Benjamin Röjder Delnavaz, Matt Jadczak, Ronen Ulanovsky, Daniel Papp, Yves, Michael Alyn Miller, villem, Adam Gutglick, Bob Ippolito, Geoffrey Thomas, genny, ACRL, Dave Minter, you got maiL, Tobias Bahls, Jack Maguire, ZacJW, Aalekh Patel, James Brown, Torben Clasen, Noel, Neil Blakey-Milner, Valentin Mariette, Sylvie Nightshade, Peter Shih, AdrianEddy, Philipp Hatt, Chris Thackrey, Andy F, Max Heaton, Jonathan Adams, Max Bruckner, Chirag Jain, Lennart Oldenburg, Antoine Rouaze, Antoine PESTEL-ROPARS, Michal Hošna, Alan O'Donnell, Dylan Anthony, Gioele Pannetto, John VanEnk, Joshua Roesslein, Michał Bartoszkiewicz, Christopher Valerio, Dirkjan Ochtman, WeblWabl, Mateusz Wykurz, Chris, John Horowitz, Johnathan Pagnutti, Ben Wishovich, Ives van Hoorne, Beat Scherrer, Tyler Bloom, Jörn Huxhorn, Marty Penner, e9zaktw1, Sindre Johansen, Zoran Zaric, Elijah Voigt, Andrew Neth, Mattia Valzelli, Justy, bbutkovic, Ben Mitchell, David Cornu, Michał Zalewski, Luuk, Menno Finlay-Smits, Ross Williams, Paul Horn, Raphaël Thériault, Paul Marques Mota, Thor Kamphefner, Aiden Scandella, Tyler Schmidtke, Santiago Lema, Gorazd Brumen, Matěj Volf, DaVince, Hamilton Chapman, Matthew T, Mike English, Mikko Leppänen, Beth Rennie, Radu Matei, Vladimir, Guilherme Neubaner, Julian Schmid, Jean-David Gadina, Pete LeVasseur, Daniel Strittmatter, Marc-Andre Giroux, Berkus Decker, teor, jer, Chris Emery, Makoto Nakashima, Ronen Cohen, Shane Lillie, Jesse Luehrs, Max von Forell, prairiewolf, Ivo Murrell, Zeeger Lubsen, Jack Duvall, Jan-Stefan Janetzky, Dom, Philipp Gniewosz, Raine Godmaire, avborhanian, Kristoffer Winther Balling, Wyatt Herkamp, Chris Biscardi, Daniel Wagner-Hall, Scott Steele, Egor Ternovoi, qrpth, Geoffroy Couprie, Nicholas Orta, Lyssieth, Sarah Berrettini, Cass, Blake Johnson, James Rhodes, ofrighil, hgranthorner, Ian McLinden, L0r3m1p5um, notryanb, Mason Ginter, Niels Abildgaard, Luke Konopka, Borys Minaiev, Chris Sims, Kamran Khan, Timothée Gerber, Malik Bougacha

In the previous chapter, we’ve made a nix “dev shell” that contained the fly.io command-line utility, “flyctl”.

That said, that’s not how I want us to define a dev shell.

Our current solution has issues. I don’t like that it has import <nixpkgs>. Which version of nixpkgs is that? The one you’re on? Who knows what that is.

Also, we haven’t really seen a mechanism to use .nix files from elsewhere.

As you may be beginning to suspect, building Rust code (or a Node.js app, etc.) with nix is actually non-trivial, because we need to hash the whole input, download everything in advance (did I mention builders don’t have access to the internet? to make double-plus-sure everything is reproducible?), use the right toolchain, etc.

So we won’t be using mkDerivation at all - we’ll be using a third-party Nix library. And we need a way to refer to it cleanly.

Those problems are solved with nix flakes - part of the new, experimental, don’t-cry-if-it-breaks stuff.

A nix flake is just a regular nix file! It needs to evaluate to a set with inputs and outputs, and some others if we feel fancy.

So, let’s remove shell.nix, and create a mostly-empty flake.nix:

This isn’t enough, and any flakes-enabled commands (the new stuff, the nix subcommands, not the nix-dash-something commands) should let us know, like, nix flake check:

Wait, no, that’s… that’s not the error I expected. Due to how flakes work, the flake.nix file needs to be staged into our Git repository, so let’s do that:

So, fine, let’s add outputs:

Ah, outputs should be a function. A function of its inputs, no doubt:

And just like that, we’ve made our first, completely useless flake.

Let’s make it useful! One thing a flake can have as an output is… an executable! We can use derivation just like we did before!

The only difference is, now we specify where nixpkgs comes from:

And if we do nix build, it builds just like before.

Cool Bear's hot tip

Note that this is nix build with a space, as in “the build subcommand of nix”, not “the nix-build command”.

And in flake.lock, our inputs are locked:

…which we can update with nix flake update.

We can also show the outputs our flake has with nix flake show:

However, this isn’t a very good flake, yet.

The code is redundant all over.

Instead of referring to inputs.stuff, we could destructure the arguments of outputs:

And then, we can use let to great effect:

There. Isn’t that much tidier?

But see that system = system there? That’s not very DRY.

nix has a thing for this: inherit. In fact, we can inherit a bunch of things:

This version of the flake does exactly the same, but it’s clearer! (We didn’t really need to inherit name and src, but I wanted to show it off).

Finally, our flake doesn’t only work on x86_64-linux. We could define it manually for a bunch of other targets, or we could use a nix library to do it for us!

Any flake-enabled command updates flake.lock (since we have a new input), nix flake show now shows that we can build for a bunch of common targets:

Note that nix build still defaults to building.. the default package, for our current platform.

We can tell, if we ask it for extra debugging output with --debug:

We can also build it more explicitly, by passing the path of the thing to build:

Or we can be more specific:

Or even more specific:

Wait, does that mean we can cross-compile?

It’s, uhhh, complicated. I haven’t cracked it at the time of this writing.

If you try the obvious thing, it fails somewhat gracefully, at least:

Just for completeness’ sake, let’s see how our shaka-packager package would look like as flake.

Instead of removing our default package, we’ll simply add it as another output:

There! Isn’t that neat?

And we can build it with:

Wait.. isn’t it downloading an x86_64-linux binary for shaka-packager?

Oh, you’re right! This one should not be for all targets.

Well, what a good excuse to learn about the //! It returns the union of two sets, preferring the right-hand-side if there’s conflicts.

Note that this doesn’t do a deep merge, so we need to expand things a little:

And now we have the expected output:

Note that of course, shaka-packager has builds for all these platforms, so we could have our flake translate system to the proper download URL, with the proper sha256.

But this is not at all what we’re trying to do here, so let’s not.

Let’s start our flake.nix over, since we got distracted with learning nix and stuff.

There.

The most pressing thing we need in our dev shell is cargo / rustc - remember that? Way at the beginning of the article?

That’s still what we’re trying to do.

But at least, now we know enough of nix to do it easily, right?

Well….. almost. We haven’t talked about overlays yet. Remember how we said the result of an import <nixpkgs> is a function that can be called with a set?

Yes?

Well, that’s how we do overlays. Which provide packages that aren’t in nixpkgs by default.

For example, github:oxalica/rust-overlay provides various versions of the Rust toolchain.

We can do, for example:

And then:

Ah, finally!

Wait no no no, hold on a minute.

What. What? It does the thing!

It certainly does, but did you notice how it added inputs:

Well?

We already have those! And now we have some duplicates!

Ah, well. Isn’t that kinda inherent to the flake model?

Mhhhhhsorta but we can fix it:

Let’s check again:

Ah, much better!

Making sure you have the same inputs, down to the commit is important to avoid “rebuild the world” situations, as Pyrox explains on GitHub.

So, nix-shell is the old stuff, nix develop is the new, flake-enabled stuff. Notably, it’s much faster than nix-shell, so it’s not just novelty for novelty’s sake.

(Also, for me at least, it doesn’t change the shell prompt, which can be a bit confusing - it’s hard to tell when you’re in the dev shell or not).

The next thing I’m unhappy with, is that it gave us the latest stable:

…when we have a rust-toolchain.toml file.

We can fix that too!

And now, we have the proper version:

Yay! Finally!

We’re not quite done yet, though. There’s still a couple things missing, but before we move forward, let’s set up a thing that “enters the dev environment” anytime we cd into the directory.

We’ll use the direnv tool for that, just like we did in part 4.

We could install it with apt, or… we could use nix profile! Let’s try the latter:

Just like last time, we’ll set up the bash hook for it (it works similarly for zsh and others, if you couldn’t resist switching to them already):

We already have an .envrc file in there, and it’s already encrypted by git-crypt, but… we haven’t decrypted it yet.

Our .envrc is a little hard to read right now:

So, remember when I told you to save your git-crypt-key file? Time to copy it somewhere into your VM once again, and then install git-crypt.

Cool Bear's hot tip

If you haven’t saved your git-crypt-key file, bummer, but you can find or regenerate honeycomb tokens etc., and re-write the file from scratch, generate a new git-crypt-key, and lock it again with git-crypt.

Wait, wouldn’t it make sense to have it as part of the dev shell?

Mhhh not really: the instructions to activate the dev shell when cd-ing into the directory are in the .envrc, so it needs to be already decrypted by that time.

So “having git-crypt already installed before you start hacking on catscii” is going to be a hard requirement for us.

Let’s proceed:

That looks better:

Before we approve it, let’s just add one thing to the beginning of our .envrc file:

This sets up nix-direnv, which may have a newer version by the time you read this.

Cool Bear's hot tip

You also probably want to add /.direnv to your .gitignore, because the next steps will create this directory and put some symlinks in there.

Now we can run direnv allow and let it do its thing:

How interesting! We can see everything it exports here: we recognize some of the things from our .envrc file, like GEOLITE2_COUNTRY and HONEYCOMB_API_KEY, but also some definitely-nix-related things like NIX_BINTOOLS, AR, NM, STRIP, which are all part of the toolchain pulled in by Rust:

That’s not quite enough though - we do want to be able to develop catscii in VS Code, with rust-analyzer, which invokes cargo check, so it must have access to a Rust toolchain, which we’re getting from nix.

So how do we get VS Code extensions to find our nix-installed Rust toolchain? Well, there’s a direnv extension! That works with the SSH Remote extension we’re already using. And that we’ve also talked about in Part 4.

After installing it, it’ll tell us the environment was updated, and ask if we want to restart extensions, which we do (so we can click “Restart”).

If you now try to run cargo check, you may run into a cargo error:

And here, you have two options: either you have backed up your SSH key way back in part 7, or you haven’t, and then you can make a new SSH key and add it to Shipyard.

After that, cargo check gets a lot further, because it manages to clone/update the private repository.

But it wants libssl-dev! Very well then, let’s add it to our flake.nix:

Cool Bear's hot tip

Remember, you can use the NixOS package search site to find what you’re looking for. It’s pretty good.

Say amos, what’s the difference between buildInputs and nativeBuildInputs?

Well, for dev shells, there’s none. But buildInputs are things you’ll also need at run-time, whereas nativeBuildInputs are things you only need at compile-time.

It’s easier to remember if you think of cross-compiling: if you’re building from x86_64 for aarch64, you might be using a “native” GCC for x86_64, that generates aarch64 binaries, hence: native build inputs.

After saving flake.nix, simply hitting Enter will re-evaluate our flake.nix:

…and we should have pkg-config in $PATH, which should be able to find openssl:

And now, it’s a game of cat and mouse! Let’s keep going:

Huh. I could’ve sworn…

Yeah, where’s sqlite?

Mhhh. Mhhh!

Ahh! I thought so.

I would’ve expected it to fail earlier with a nicer message, because, after all, pkg-config cannot find it:

But it seems the libsqlite3-sys maintainers chose to spray and pray in that scenario. Ah well.

Anyway, a quick search reveals that in nixpkgs, sqlite 3 is just named “sqlite” (I have yet to encounter a project that uses sqlite 1 or 2, after all), so we can add it to our buildInputs in flake.nix:

Press Enter in our shell to re-evaluate the flake, and then cargo build succeeds!

It is, however, missing the MaxMindDB database:

…so we need to copy it over again. I used scp:

And with that, it runs!

And we’re almost done with this adventure. All we have to do now is rip out that Dockerfile. It’s going to be glorious. I can smell it from here.

Comment on /r/fasterthanlime

Here's another article just for you:

Cracking Electron apps open

Jul 03, 2023

15 min #electron · #rust · #asar

I use the draw.io desktop app to make diagrams for my website. I run it on an actual desktop, like Windows or macOS, but the asset pipeline that converts .drawio files, to .pdf, to .svg, and then to .svg again (but smaller) runs on Linux.

So I have a Rust program somewhere that opens headless chromium, and loads just the HTML/JS/CSS part of draw.io I need to render my diagrams, and then use Chromium’s “print to PDF” functionality to save a PDF.

Read more