Mar 05, 2023 10 min #rust · #docker · #flyio

Deploying catscii to fly.io

From the series Building a Rust service with Nix

12

👋 This page was last updated ~2 years ago. Just so you know.

Thanks to my sponsors: Manuel Hutter, Michal Hošna, Nyefan, Braidon Whatley, Bader, Niels Abildgaard, David White, Noel, WeblWabl, Urs Metz, Luuk, Valentin Mariette, traxys, Marky Mark, Nicholas Orta, Alex Rudy, Josiah Bull, Michał Bartoszkiewicz, James Leitch, Herman J. Radtke III and 267 more

Disclaimer:

Because I used to work for fly.io, I still benefit from an employee discount at the time of this writing: I don’t have to pay for anything deployed there for now.

fly.io is still sponsoring me for developing hring, but this isn’t a sponsored post. It’s just a good fit for what we’re doing here, with a generous free tier.

In the previous chapter, we’ve written a Dockerfile to build the catscii service inside Docker. The result is a container image that can be pushed to production!

Deploying to fly.io

You’ll need a fly.io account (you can log in with GitHub).

The fly.io docs have instructions on how to “Install Flyctl”, which you’ll need to follow. You’ll need to log in, too.

If everything goes fine, fly auth whoami should work:

$ fly auth whoami
(redacted e-mail address)

And now we’re ready to create an app!

$ fly apps create
? Choose an app name (leave blank to generate one):
? Select Organization: Amos Wenger (personal)
New app created: old-frost-6294

Note, some users have found it beneficial to launch the app before saving it’s configuration locally. Launching your app will also generate a fly.toml file for you. This step can help avoid potential “No machines configured for this app” errors. If you’re unfamiliar with launching, you can use the flyctl launch command or check out the official launch documentation.

I’ve intentionally let the name blank to generate a random name, so we’re not all fighting for the name catscii. Whenever you see old-frost-6294 in commands, simply replace it with your own app’s name.

Now that we have an app, we can deploy to it with fly deploy — we just need a fly.toml file.

First we can import the default config for our app:

$ fly config save --app old-frost-6294
Wrote config file fly.toml

Here’s what this gives me:

# fly.toml file generated for old-frost-6294 on 2022-12-23T17:21:37Z

app = "old-frost-6294"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

services[].internal_port is already set to 8080, what a coincidence. It has an HTTP handler on port 80, and a TLS handler on port 443, this is all we want.

Note that because that fly.toml has app = "old-frost-6294", as long as our shell is in that directory, we don’t need to pass -a / --app to flyctl.

Handy!

The only missing bit is “what should be deployed”, and we can specify that by adding a new section:

[build]
image = "catscii"

Cool Bear's hot tip

fly.io also provides “remote builders” that can build your Dockerfile directly. You can also force it to build using your local Docker daemon by passing --local-only to fly deploy.

However, because using Dockerfile is just a phase, and we’ll eventually use nix to generate the container image, we’re specifying an image directly here.

With that, we’re ready to deploy!

$ fly deploy --local-only
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'catscii' locally...
image found: sha256:c2a27c54dbe6940d7fd82a2d962c832c46b3596d1a328751daa43412e7fc5c3e
==> Pushing image to fly
The push refers to repository [registry.fly.io/old-frost-6294]
3c47a1ca65c2: Pushed
8a6f14bd6616: Pushed
42957de36bb8: Pushed
d49aaa665130: Pushed
5f70bf18a086: Pushed
531741825ed0: Pushing [======>                                            ]  145.8MB/1.157GB
57c57da67e69: Pushing [===========================>                       ]  142.2MB/255.6MB
0002c93bdb37: Pushing [======================>                            ]   32.6MB/72.79MB
^CWARN failed to finish build in graphql: Post "https://api.fly.io/graphql": context canceled

Wait no, hold up, Ctrl-C that. What’s that in the middle: a 1.157GB layer?

fly.io can handle that, but let’s… not. Let’s take a few minutes to make our image slimmer.

Multi-stage docker builds to make our image slimmer

The newer Dockerfile syntax, in conjunction with BuildKit, lets us have multi-stage builds, and so we can have a “builder” target (that has all the compile-time dependencies), and an “app” target (that has only the runtime dependencies).

Let’s try it out:

# syntax = docker/dockerfile:1.4

################################################################################
FROM ubuntu:20.04 AS base

################################################################################
FROM base AS builder

# Install compile-time dependencies
RUN set -eux; \
		apt update; \
		apt install -y --no-install-recommends \
			curl ca-certificates gcc libc6-dev pkg-config libssl-dev \
			;

# Install rustup
RUN set -eux; \
		curl --location --fail \
			"https://static.rust-lang.org/rustup/dist/x86_64-unknown-linux-gnu/rustup-init" \
			--output rustup-init; \
		chmod +x rustup-init; \
		./rustup-init -y --no-modify-path --default-toolchain stable; \
		rm rustup-init;

# Add rustup to path, check that it works
ENV PATH=${PATH}:/root/.cargo/bin
RUN set -eux; \
		rustup --version;

# Copy sources and build them
WORKDIR /app
COPY src src
COPY Cargo.toml Cargo.lock ./
RUN --mount=type=cache,target=/root/.rustup \
    --mount=type=cache,target=/root/.cargo/registry \
    --mount=type=cache,target=/root/.cargo/git \
		--mount=type=cache,target=/app/target \
		set -eux; \
		cargo build --release; \
		cp target/release/catscii .

################################################################################
FROM base AS app

# Install run-time dependencies, remove extra APT files afterwards.
# This must be done in the same `RUN` command, otherwise it doesn't help
# to reduce the image size.
RUN set -eux; \
		apt update; \
		apt install -y --no-install-recommends \
			ca-certificates \
			; \
		apt clean autoclean; \
		apt autoremove --yes; \
		rm -rf /var/lib/{apt,dpkg,cache,log}/

# Copy app from builder
WORKDIR /app
COPY --from=builder /app/catscii .

CMD ["/app/catscii"]

Let’s build it:

$ DOCKER_BUILDKIT=1 docker build --target app --tag catscii .
(cut)
 => => naming to docker.io/library/catscii

And run it:

$ docker run -it --env SENTRY_DSN --env HONEYCOMB_API_KEY --publish 8080:8080/tcp --rm catscii
{"timestamp":"2022-12-23T17:36:03.032745Z","level":"INFO","fields":{"message":"Creating honey client","log.target":"libhoney::client","log.module_path":"libhoney::client","log.file":"/root/.cargo/git/checkouts/libhoney-rust-3b5a30a6076b74c0/9871051/src/client.rs","log.line":78},"target":"libhoney::client"}
{"timestamp":"2022-12-23T17:36:03.032820Z","level":"INFO","fields":{"message":"transmission starting","log.target":"libhoney::transmission","log.module_path":"libhoney::transmission","log.file":"/root/.cargo/git/checkouts/libhoney-rust-3b5a30a6076b74c0/9871051/src/transmission.rs","log.line":124},"target":"libhoney::transmission"}
{"timestamp":"2022-12-23T17:36:03.036680Z","level":"INFO","fields":{"message":"Listening on 0.0.0.0:8080"},"target":"catscii"}
^C{"timestamp":"2022-12-23T17:36:03.958953Z","level":"WARN","fields":{"message":"Initiating graceful shutdown"},"target":"catscii"}

Still runs!

What’s the image size?

$ docker image ls catscii
REPOSITORY   TAG       IMAGE ID       CREATED              SIZE
catscii      latest    413b40968a62   About a minute ago   139MB

Okay, that’s much better.

Shrinking down the image size further

We can probably do better still, though… I’ve used dive to look into the image size:

$ dive catscii
(see screenshot below)

Terminal screenshot, shows the dive text-based UI. On the left there’s a panel that shows various layers. The focused layer is the one that installs runtime dependencies. On the right is a panel with the current layer contents. The var folder is 42MB, which is suspicious

Looking at this, I don’t know about you, but I… I think this line from our Dockerfile isn’t working as intended:

		rm -rf /var/lib/{apt,dpkg,cache,log}/

In fact, let’s look around:

$ docker run -it --rm catscii /bin/sh
# ls /var/lib/
apt  dpkg  misc  pam  systemd
# rm -rf /var/lib/{apt,dpkg,cache,log}/
# ls /var/lib/
apt  dpkg  misc  pam  systemd

Oh, okay, cool.

So it turns out brace expansion is “a bash-ism”. It is supported by the bash shell, but not by a strictly POSIX-compliant shell, which is used to execute our Dockerfile RUN directives.

So, it’s not actually running:

$ rm -rf /var/lib/apt/ /var/lib/dpkg/ /var/lib/cache/ /var/lib/log/

It’s actually running:

$ rm -rf '/var/lib/{apt,dpkg,cache,log}/'

And -f makes rm not complain if the target doesn’t exist, so… nothing happens.

If we were writing regular shell scripts, a tool like shellcheck may have saved us:

$ shellcheck ./rm.sh

In ./rm.sh line 2:
rm -rf /var/lib/{apt,dpkg,cache,log}/
                ^------------------^ SC3009 (warning): In POSIX sh, brace expansion is undefined.

For more information:
  https://www.shellcheck.net/wiki/SC3009 -- In POSIX sh, brace expansion is u...

…but I don’t think shellcheck is able to parse/interpret Dockerfiles, so.

There’s two potential fixes for that: we can either do the expansion ourselves, or we can switch the shell used for RUN commands:

################################################################################
FROM base AS app

# 👇 new!
SHELL ["/bin/bash", "-c"]

# Install run-time dependencies, remove extra APT files afterwards.
# This must be done in the same `RUN` command, otherwise it doesn't help
# to reduce the image size.
RUN set -eux; \
		apt update; \
		apt install -y --no-install-recommends \
			ca-certificates \
			; \
		apt clean autoclean; \
		apt autoremove --yes; \
		# Note: 👇 this only works because of the `SHELL` instruction above.
		rm -rf /var/lib/{apt,dpkg,cache,log}/

And now our image is…

$ DOCKER_BUILDKIT=1 docker build --target app --tag catscii .
(cut)

$ docker image ls catscii
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
catscii      latest    d4efaf33cd3e   16 minutes ago   99.1MB

Under 100MB! That’s better, but we can still improve on it.

The layer where we copy our app’s binary in-place is 20MB right now. If we change cp to an objcopy invocation that compresses debug info, we can bring that down a little.

# in "FROM base AS builder"
# omitted: installing compile-time dependencies, etc.

# Copy sources and build them
WORKDIR /app
COPY src src
COPY Cargo.toml Cargo.lock ./
RUN --mount=type=cache,target=/root/.rustup \
    --mount=type=cache,target=/root/.cargo/registry \
    --mount=type=cache,target=/root/.cargo/git \
		--mount=type=cache,target=/app/target \
		set -eux; \
		cargo build --release; \
		# 👇 new!
		objcopy --compress-debug-sections ./target/release/catscii ./catscii

I’m not even actually sure why this does anything, since debug defaults to false for the release profile, see the docs.

Without going too far down the rabbit hole, we can pick a few tricks from the excellent min-sized-rust page, and end up with something I’d personally be happy to deploy in production:

# in Cargo.toml

# at the bottom of the file

[profile.release]
debug = 1 # Include enough debug info for sentry to be useful
opt-level = "z"  # Optimize for size.
lto = true # Enable link-time optimization

For me, this comes out at 16MB too, but this time we actually have debug info. Not that I expect our service to panic in production, but, you know, this way we have it.

Actually deploying it to fly.io

Anyway, enough bikeshedding - let’s push that <100MB image to fly:

$ DOCKER_BUILDKIT=1 docker build --target app --tag catscii .
(cut)

$ fly deploy --local-only
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'catscii' locally...
image found: sha256:13cd689191106e5a503fa73af43ca51bd2d9c82c91ba5b4a6a52f7574873222b
==> Pushing image to fly
The push refers to repository [registry.fly.io/old-frost-6294]
2df41807956e: Pushed
81569901f7fe: Pushed
50161719fe01: Pushed
0002c93bdb37: Pushed
deployment-01GN030CXYTQ784CKH0TDMNT5C: digest: sha256:5d00cfb31fa901e1de011aa67250b5a4e982d0990dd0f2ace3c98958ea847f27 size: 1158
--> Pushing image done
==> Creating release
--> release v2 created

--> You can detach the terminal anytime without stopping the deployment
==> Monitoring deployment
Logs: https://fly.io/apps/old-frost-6294/monitoring

v0 is being deployed

Spoiler alert: this deployment will fail. Can you guess why?

I, uhh… no?

Luckily, fly deploy tells us why. After a couple tries, here’s the rest of the output:

# (continued)

==> Monitoring deployment
Logs: https://fly.io/apps/old-frost-6294/monitoring

 1 desired, 1 placed, 0 healthy, 1 unhealthy [restarts: 2] [health checks: 1 total]
Failed Instances

Failure #1

Instance
ID              PROCESS VERSION REGION  DESIRED STATUS  HEALTH CHECKS   RESTARTS        CREATED
523218db        app     0       cdg     run     running 1 total         2               22s ago

Recent Events
TIMESTAMP               TYPE            MESSAGE
2022-12-23T18:13:16Z    Received        Task received by client
2022-12-23T18:13:16Z    Task Setup      Building Task Directory
2022-12-23T18:13:24Z    Started         Task started by client
2022-12-23T18:13:26Z    Terminated      Exit Code: 101
2022-12-23T18:13:26Z    Restarting      Task restarting in 1.20387713s
2022-12-23T18:13:37Z    Started         Task started by client
2022-12-23T18:13:39Z    Terminated      Exit Code: 101
2022-12-23T18:13:39Z    Restarting      Task restarting in 1.214914905s
2022-12-23T18:13:47Z    Started         Task started by client

2022-12-23T18:13:24Z   [info]2022/12/23 18:13:24 listening on [fdaa:0:3614:a7b:ae02:5232:18db:2]:22 (DNS: [fdaa::3]:53)
2022-12-23T18:13:24Z   [info]thread 'main' panicked at '$SENTRY_DSN must be set: NotPresent', src/main.rs:37:37
2022-12-23T18:13:24Z   [info]note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
(cut: the same thing two more times)
--> v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1

--> Troubleshooting guide at https://fly.io/docs/getting-started/troubleshooting/
Error abort

Oh right! The environment variables.

Yes indeed!

Setting secrets

Easy enough, we can use fly secrets set and pass them as arguments, or fly secrets import.

For Honeycomb, we should make a “production API key”, whereas for Sentry, we can re-use the same API key, and it should auto-detect the environment.

Cool Bear's hot tip

The way the sentry crate “detects production environment” is simply by checking whether the debug_assertions cfg-attribute is set or not. It’s usually set for debug builds, and unset for release builds, so it all works out… unless you run release builds in development.

To make a new Honeycomb API key, click the environment picker on the top-left (below the logo), then click “Manage environments”. If needed, create the “production” environment, and from there, there should be an “API keys” tab where you can see existing API keys / create new ones.

Note that the security around those monitoring/telemetry API keys is significantly relaxed compared to others: the worst thing someone can do if they get their hands on it is send data somewhere they’ll never see. You still shouldn’t commit them in plaintext, but it’s not as radioactive as most credentials (and very easy to rotate).

Anyway, let’s import our secrets:

# note: we're running this from our `catscii/` directory, that has a `fly.toml`
# file, so we don't need to specify -a / --app
$ fly secrets import
SENTRY_DSN=https://redacted@redacted.ingest.sentry.io/redacted
HONEYCOMB_API_KEY=redacted

# press Ctrl-D when done

Release v1 created
==> Monitoring deployment
Logs: https://fly.io/apps/old-frost-6294/monitoring

v1 is being deployed

Cool Bear's hot tip

Note that double-quoting (ie. KEY="VALUE") is not supported at the time of this writing, unlike in Docker .env files, see that discussion.

Our deploy should succeed.

Watching our application in production

We can now look at logs in the CLI:

$ fly logs
(cut)
2022-12-23T18:31:39Z app[89d7ad81] cdg [info]Starting init (commit: f447594)...
2022-12-23T18:31:39Z app[89d7ad81] cdg [info]Preparing to run: `/app/catscii` as root
2022-12-23T18:31:39Z app[89d7ad81] cdg [info]2022/12/23 18:31:39 listening on [fdaa:0:3614:a7b:abd:89d7:ad81:2]:22 (DNS: [fdaa::3]:53)
2022-12-23T18:31:39Z app[89d7ad81] cdg [info]{"timestamp":"2022-12-23T18:31:39.396353Z","level":"INFO","fields":{"message":"Creating honey client","log.target":"libhoney::client","log.module_path":"libhoney::client","log.file":"/root/.cargo/git/checkouts/libhoney-rust-3b5a30a6076b74c0/9871051/src/client.rs","log.line":78},"target":"libhoney::client"}
2022-12-23T18:31:39Z app[89d7ad81] cdg [info]{"timestamp":"2022-12-23T18:31:39.396519Z","level":"INFO","fields":{"message":"transmission starting","log.target":"libhoney::transmission","log.module_path":"libhoney::transmission","log.file":"/root/.cargo/git/checkouts/libhoney-rust-3b5a30a6076b74c0/9871051/src/transmission.rs","log.line":124},"target":"libhoney::transmission"}
2022-12-23T18:31:39Z app[89d7ad81] cdg [info]{"timestamp":"2022-12-23T18:31:39.406458Z","level":"INFO","fields":{"message":"Listening on 0.0.0.0:8080"},"target":"catscii"}

We can look up the app’s status:

$ fly status
App
  Name     = old-frost-6294
  Owner    = personal
  Status   = running
  Version  = 2
  Platform = nomad
  Hostname = old-frost-6294.fly.dev

Services
PROTOCOL        PORTS
TCP             80 => 8080 [HTTP]
                443 => 8080 [TLS, HTTP]

IP Addresses
TYPE    ADDRESS                 REGION  CREATED AT
v6      2a09:8280:1::3:bb39     global  24m40s ago

We get our own IPv6 address, and a shared IPv4 address:

$ fly ips list
VERSION IP                      TYPE            REGION  CREATED AT
v6      2a09:8280:1::3:bb39     public          global  25m5s ago
v4      66.241.125.118          public (shared)

Visiting https://old-frost-6294.fly.dev/ in a browser works for me!

Over at https://fly.io/dashboard, there’s a ton of cool things to see:

The fly.io app dashboard, showing that our app uses 40/232MB of RAM, and has an average data in of 5 kB/s over the past hour. It shows the VM size, public IPv6 address, and release activity

Like logs:

The Monitoring tab, showing our logs nicely colored.

And metrics:

Graphs showing HTTP 200 requests

We even get Grafana:

fly-hosted Grafana screenshot, showing a world map and list of regions+traffic for Data Out, a Network I/O graph with separate input/output metrics, memory utilization and CPU utilization

Pretty neat!

You're reading the Building a Rust service with Nix series.

1

2

3

4

5

6

7

8

9

10

11

12

Comment on /r/fasterthanlime

(JavaScript is required to see this. Or maybe my stuff broke)

Here's another article just for you:

Abstracting away correctness

Jun 27, 2020

45 min #golang · #rust

I’ve been banging the same drum for years: APIs must be carefully designed.

This statement doesn’t resonate the same way with everyone. In order to really understand what I mean by “careful API design”, one has to have experienced both ends of the spectrum.

But there is a silver lining - once you have experienced “good design”, it’s really hard to go back to the other kind. Even after acknowledging that “good design” inevitably comes at a cost, whether it’s cognitive load, compile times, making hiring more challenging, etc.