fasterthanlime

Using the Shipyard private crate registry with Docker

Thanks to my sponsors: Andrew Neth, Max von Forell, Lena Schönburg, Herman J. Radtke III, Menno Finlay-Smits, Michael, Toon Willems, Jim, prairiewolf, Olivia Crain, Yann Schwartz, Matt Jackson, Mark, Duane Sibilly, Romet Tagobert, Matt Heise, Zaki, Nicolas Riebesel, Walther, Marcus Griep and 266 more Andrew Neth, Max von Forell, Lena Schönburg, Herman J. Radtke III, Menno Finlay-Smits, Michael, Toon Willems, Jim, prairiewolf, Olivia Crain, Yann Schwartz, Matt Jackson, Mark, Duane Sibilly, Romet Tagobert, Matt Heise, Zaki, Nicolas Riebesel, Walther, Marcus Griep, Braidon Whatley, Mark Tomlin, Malik Bougacha, DaVince, James Brown, Marty Penner, Daniel Silverstone, std__mpa, Simon Menke, Elnath, xales, James Rhodes, Adam Gutglick, Horváth-Lázár Péter, Geoffrey Thomas, Olly Swanson, Marc-Andre Giroux, Josh Triplett, David Barsky, Berkus Decker, compwhizii, Philipp Gniewosz, avborhanian, Taneli Kaivola, Niels Abildgaard, Joseph Montanaro, teor, jatescher, Andrew Henshaw, Elijah Voigt, Isak Sunde Singh, Stephan Buys, Dominik Wagner, Sarah Berrettini, Santiago Lema, Matěj Volf, Luke Konopka, Justin Smith, Senyo Simpson, traxys, anichno, ShikChen, Dimitri Merejkowsky, Radu Matei, James Leitch, Makoto Nakashima, Ben Wishovich, John VanEnk, Chirag Jain, Scott Steele, Mikko Leppänen, Mason Ginter, Tabitha, Sylvie Nightshade, Paul Horn, Pete Bevin, Guilherme Neubaner, Chris Walker, Raphaël Thériault, Alan O'Donnell, Marky Mark, e9zaktw1, Dave Minter, Astrid, Eugene Bulkin, genny, Lucille Blumire, Steven Pham, bbutkovic, Chris Emery, Arjen Laarhoven, Marcus Griep, old.woman.josiah, Mathias Brossard, Kyle Lacy, Raine Godmaire, Justin Ossevoort, Chris Biscardi, you got maiL, Aleksandre Khokhiashvili, Sawyer Knoblich, AdrianEddy, Guy Waldman, David E Disch, notryanb, L0r3m1p5um, Nicholas Orta, Sindre Johansen, Nicholas, Philipp Hatt, Christopher Valerio, Samit Basu, Gioele Pannetto, Laine Taffin Altman, Beth Rennie, jer, hgranthorner, Michal Hošna, Max Heaton, Michael Alyn Miller, Mike English, Zoran Zaric, Kai Kaufman, Neil Blakey-Milner, Sam Leonard, Dom, Daniel Strittmatter, Lyssieth, budrick, Alex Rudy, Marco Carmosino, Matt Jadczak, Andy F, Yves, Josiah Bull, SeniorMars, ACRL, Wyatt Herkamp, Beat Scherrer, Alex Krantz, Jack Maguire, Ivo Murrell, Chris Sims, Aalekh Patel, qrpth, Christoph Grabo, Brooke Tilley, Tiziano Santoro, Geoffroy Couprie, Adam Lassek, ZacJW, Mark Old, villem, Diego Roig, johbii, Ronen Ulanovsky, Richard Pringle, Egor Ternovoi, Damir Vandic, Elendol, Evan Relf, Marcin Kołodziej, Tyler Schmidtke, Jean-David Gadina, Hadrien G., Cole Kurkowski, Marie Janssen, Daniel Papp, Wojciech Smołka, Henrik Tudborg, Jan De Landtsheer, Michał Zalewski, Jan-Stefan Janetzky, Adrián Garnier Artiñano, Jörn Huxhorn, Ryan, Romain Kelifa, callym, Tanner Muro, Manuel Hutter, milan, Ben Mitchell, Dylan Anthony, Rufus Cable, Jesse Luehrs, clement, Mario Fleischhacker, John Horowitz, Matthew T, Cass, Corey Alexander, Christian Bourjau, Matt Campbell, Thor Kamphefner, Ian McLinden, Jonathan Adams, Johnathan Pagnutti, Richard Stephens, Jon Gjengset, Urs Metz, Daniel Wagner-Hall, Aljaz Erzen, Jelle Besseling, Zeeger Lubsen, Geoff Cant, Benjamin Röjder Delnavaz, Blake Johnson, Kamran Khan, Chris Thackrey, Pete LeVasseur, Tyler Bloom, Yuriy Taraday, Tobias Bahls, Luuk, Cole Tobin, Lennart Oldenburg, Noel, Julian Schmid, Enrico Zschemisch, Xavier Groleau, Thehbadger, Zachary Thomas, Paige Ruten, Brandon Piña, Joshua Roesslein, Michael Mrozek, playest, Dirkjan Ochtman, Marcus Brito, Xirvik Servers, G, Justy, Victor Song, Borys Minaiev, Hamilton Chapman, medzernik, Sean Bryant, Ross Williams, Bob Ippolito, Antoine Rouaze, David White, C J Silverio, Yufan Lou, Luke Yue, Boris Dolgov, Òscar Pérez, Andy Gocke, Chris, Luiz Ferraz, Nyefan, Ahmad Alhashemi, Ronen Cohen, Max Bruckner, Mateusz Wykurz, Antoine PESTEL-ROPARS, WeblWabl, Paul Marques Mota, Shane Lillie, Garret Kelly, Peter Shih, Timothée Gerber, Valentin Mariette, Mikkel Rasmussen, Ives van Hoorne, Integer 32, LLC, Aiden Scandella, René Ribaud, Scott Sanderson, Seth, Mattia Valzelli, belzael, David Cornu, Gorazd Brumen, Guillaume E, Ula, Antoine Boegli, Jack Duvall, Vladimir, Colin VanDervoort, Torben Clasen, pinkhatbeard, ofrighil, Guillaume Demonet, Jake Demarest-Mays, Andronik, Michał Bartoszkiewicz

Wait wait wait, so we’re not talking about nix yet?

Well, no! The service we have is pretty simple, and I want to complicate things a bit, to show how things would work in both the Dockerfile and the nix scenario.

And because I don’t like contrived examples, we’re going to do something somewhat real-world: we’re going to geo-locate visitors, and track how many visits we get from each country.

But we’re not logging IP addresses, right?

No, that would be problematic re: a bunch of privacy protection laws. I feel like the country is broad enough that it’s okay to store somewhere.

Very well, if we must.

In that particular context, it would make sense to have a cargo workspace, with both our main (web server) crate, and our “geo-locate and count visitors by country” crate in the same Git repository.

But for demonstration purposes, we’re going to make an entirely separate repository for it.

Let’s call it… locat. For location. And cat.

Let’s stub out our API:

Now’s a good time to create a repository on GitHub (or wherever), do an initial commit, and push locat to it.

Normally we’d be able to add a dependency on the crate simply by doing something like:

Or even:

But that’s not what I want to show off here.

I also don’t want to publish it to crates.io, since it’s probably too specific and I don’t want to assume the maintenance burden of it.

Instead, let’s publish it to Shipyard, a private crate registry that popped up recently.

They have a free tier, and we’re only going to publish a free crate there, so, let’s try it!

Signing up for Shipyard should give you a crate registry of your own, and the next step is to add it to the Cargo config for locat:

(My registry is named catscii, yours will probably be named something else).

Next we amend the Cargo manifest to indicate that the crate should only be published to our private registry:

Now that we have all that set up, let’s try publishing the crate with cargo publish:

That didn’t work! We need to authenticate in some way.

At the time of this writing, the proper authentication model isn’t done cooking yet, so I’m hesitant to show it here. Instead, we’ll use the hack that works on Rust stable.

First we need to generate a token over at https://shipyard.rs/tokens, and then, just so it’s shared across all our repositories, we can put it in the directory that contains both catscii and locat. In my VM, that’s just the home directory, /home/amos, or ~ for short:

This is a pretty bad workaround, since that’ll get sent with every request cargo makes, let’s make a note to try it the Rust nightly way after that.

Also: this is only half the authentication story. This takes care of making API requests to the crate registry, which is required when publishing a crate, but it doesn’t help with cloning the index, which is just a git repository.

Shipyard, like most private crate registries, provides SSH-based authentication, but this can be annoying to set up in CI. It also provides HTTPS authentication, which is annoying for other reasons.

Because we set our registry URL as https://git.shipyard.rs/REGISTRY/crate-index.git, it’s expecting HTTPs authentication, so let’s try that first.

In ~/.gitconfig, we’ll add:

And in ~/.git-credentials, we’ll add our Gitea username (find it here) and password, URL-encoded. Unless you changed it, this is the same password you signed up for Shipyard with, see their docs for details.

Cool Bear's hot tip

To get URI-encoding right, you can use encodeURIComponent("blah") in some browser dev tools.

For me, this managed to clone the repository, but it failed with:

Cool Bear's hot tip

You may also encounter:

error: unknown SSH host key The SSH host key for `ssh.shipyard.rs` is not known and cannot be validated.

You can run ssh git@ssh.shipyard.rs and accept the host key to fix it.

Running cargo login talks about crates.io, so that’s certainly wrong. Running cargo login --registry catscii does a little better but… you can tell third-party crate registries wasn’t on top of mind…

…because the URL it generates is crates.io-specific. Anyway, we have a token we generated earlier (it’s in ~/.cargo/config.toml), we can just paste it here.

Finally, publishing works:

The crate publishes successfully, and we can see it from the Shipyard UI:

However, this didn’t spark joy. Let’s try SSH authentication, too.

To reset the state a little, first let’s bump the version of our locat crate:

Then we’ll straight up:

• remove ~/.git-credentials
• remove the [credential] section from ~/.gitconfig
• delete ~/.cargo/config.toml (the ~ one, not the locat one)

And also, we’ll need to wipe the cloned crate index, to make sure it can do it again.

In locat’s cargo config, we now need to use an ssh URL:

Then we need to generate an SSH key from within the VM:

Then we need to add it to Shipyard, through the web UI.

Cool Bear's hot tip

Make sure to back up that SSH key for later, when we set up everything again from scratch!

And now, let’s try everything again!

This ended up failing for me because VS Code forwarded my host SSH agent (which is 1Password), so it only tried using the SSH keys I had on the host, as opposed to the key I just generated on the VM.

That’s fine, I can just add one of my host’s SSH keys to Shipyard instead.

I ended up having to mess with this a little more, and add back:

…otherwise nothing would work. Things are complicated by the fact that I have other SSH keys associated to another shipyard account, so I think I’m hitting an edge case you hopefully will not hit yourselves.

After that, publishing catscii@0.2.0 worked:

But keep in mind that it’s re-using the cargo login step from the “HTTP authentication” section: our token is still stored in here:

Now we can add a dependency on locat! First we need to set the registry URL:

And then, add the dependency! I wonder if cargo add will work there.

Yes it will! Here’s the resulting Cargo.toml bit:

Wonderful. Let’s do a quick cargo check just to see…

Ah. So there’s actually three halves to cargo authentication:

• Cloning the registry index (over Git: HTTPS or SSH)
• The publish API (token)
• The download API (token)

I said we’d try the nightly thing, so let’s try the nightly thing.

We only need to do that in the catscii directory. Let’s pin to a recent-ish nightly:

And set up our catscii registry:

Our next cargo check invocation downloads that nightly version:

And complains that we haven’t set the right experimental flags.

Let’s fix that:

And then… it just works!

Normally we’d have to make some code changes to get to the IP address, but since we’re deploying to fly.io, we’re behind a proxy already, and they set a header for that, so we can just use it.

We’ll add a locat field in our ServerState struct:

Initialize it:

And call it from root_get:

Even though we’re building the production version of our app using Docker, it should still run with cargo run, so let’s try that:

Trying to visit localhost:8080 from my host’s browser prints… nothing. Because the fly-client-ip header isn’t set.

We can try it with curl (in another terminal) and set an arbitrary IP there:

And the server shows:

Which makes perfect sense, since we haven’t yet connected the GeoIP database and everything. This is neat, because it lets us experiment with deploying the new code without breaking anything.

Even though our new code doesn’t do anything useful yet, let’s try deploying it to production.

We should probably make a Justfile, so we don’t forget which commands to run.

You can install just with cargo install just for now, and then:

Now tell me cool bear: do you think this’ll go smoothly?

You’re asking, so… probably not?

You’re absolutely correct. It’s time for.. more Docker golfing!

Contest number one! It says it can’t find a registry named catscii. And yet, the .cargo/config.toml is right there on disk.

Yes, but… looks at heading it’s not in the build context? We need to copy it?

Correct!

You know what else is a compile-time dependency now?

That’s right! It’s the square hole Git!

Does that mean it needs to re-download and re-install all packages from scratch on the next docker build run? You betcha!

It’s been two minutes already. I guess that was an unlucky run. How are you doing today? Is it a lovely day for you? Not necessarily the weather, but, you know, your internal weather? Have you been tending to your inner garden? It’s important, you know. Sometimes self-care might feel like you’re being selfish, but really you’re just getting yourself in a position where you can actually help others again, whereas if y- oh would you look at that, it finished.

Oh for crying out loud.

Well I guess we have time for more chit chat. What songs are you listening to right now? I’ve been trying to find Winter-themed music that isn’t offensive to my sensibilities. That mostly means Frank Sinatra, or some other jazz pianists. Teddy Wilson’s “As Times Goes By” is surprisingly g- oh, break’s over!

Oh. Uh. A little web search pointed to this, let’s try it:

No, wait, nevermind:

Okay, sure:

I’m fairly sure that defeats the purpose, but oh well.

Ok. I’ll save you the trouble. BuildKit also lets us “mount our SSH agent” (to Docker, every problem is mount-shaped), so we can do this:

The error message is exactly the same, except it includes --mount=type=ssh.

This is because, you need to tell docker build to share your SSH credentials. Otherwise any Dockerfile could just steal them. I still think the security model is kinda wonky there, but, we’ve written our own Dockerfile from start to finish and I trust it somewhat.

Let’s adjust our docker build invocation in the Justfile:

Because, again, I have multiple Shipyard accounts, and my SSH agent offers keys that are associated with the wrong account (an edge case you shouldn’t be hitting), I set up another agent quickly:

Here’s our next error:

Wait wait wait wait. I know this one. Can I try something?

Sure?

AhAh! It’s not using the right version of Rust inside Docker, because…

…because rust-toolchain.toml is not part of the build context.

Whoa, these are easy to miss, you weren’t kidding!

Yeah! But you know what? Some developers actually love technologies that make it easy to make mistakes, because then there’s always something to do! Every single one of these commits counts towards some metric, and it makes them look very productive: “saving the day” is usually remembering one of a hundred footguns and applying the same fix, over and over.

Huh. Funny how this works out.

Yes yep. Anyway:

(Also don’t forget to remove exit 251 from the Dockerfile)

Ah. So that one only worked locally because we had done cargo login. I see.

Well… this is a docker. Every problem is a mount away. Even secrets!

Because we’re going to commit that file, let’s make sure it’s encrypted with git-crypt first.

Then we can store just the value of our token in secrets/shipyard-token:

(Make sure that file has no trailing newline!)

Let’s make sure git-crypt picks it up:

Now we can mount it as a secret:

And pass it to docker build:

And finally, just deploy works!

Visiting the site, I can see this in my logs:

Nice! Now we “just” need to make it do the thing!

Comment on /r/fasterthanlime

Here's another article just for you:

Open sourcing the home CMS

Apr 22, 2025

7 min #oss · #rust · #ftl-infra

I’ve been bragging about my website software for years! For… whew, it’s been 5 years!

I’ve been teasing folks about the cool things I did from the beginning — here’s all the articles and series I’ve written that mention it:

• 2020: A new website
• 2021: Don’t shell out!
• 2022: Updating fasterthanli.me

Read more