Deploying catscii to fly.io
fasterthanlime
This article is part of the Building a Rust service with Nix series.
- Deploying to fly.io
- Multi-stage docker builds to make our image slimmer
- Shrinking down the image size further
- Actually deploying it to fly.io
- Setting secrets
- Watching our application in production
Contents
Disclosure: Because I used to work for fly.io, I still benefit from an employee discount at the time of this writing: I don't have to pay for anything deployed there for now.
fly.io is still sponsoring me for developing hring, but this isn't a sponsored post. It's just a good fit for what we're doing here, with a generous free tier.
In the previous chapter, we've written a Dockerfile to build the
catscii service inside Docker. The result is a container image that can be
pushed to production!
Deploying to fly.io
You'll need a fly.io account (you can log in with GitHub).
The fly.io docs have instructions on how to "Install Flyctl", which you'll need to follow. You'll need to log in, too.
If everything goes fine, fly auth whoami should work:
$ fly auth whoami (redacted e-mail address)
And now we're ready to create an app!
$ fly apps create ? Choose an app name (leave blank to generate one): ? Select Organization: Amos Wenger (personal) New app created: old-frost-6294
Note, some users have found it beneficial to launch the app before saving it's configuration locally. Launching your app will also generate a fly.toml file for you. This step can help avoid potential "No machines configured for this app" errors. If you're unfamiliar with launching, you can use the flyctl launch command or check out the official launch documentation.
I've intentionally let the name blank to generate a random name, so we're not
all fighting for the name catscii. Whenever you see old-frost-6294 in
commands, simply replace it with your own app's name.
Now that we have an app, we can deploy to it with fly deploy — we just need
a fly.toml file.
First we can import the default config for our app:
$ fly config save --app old-frost-6294 Wrote config file fly.toml
Here's what this gives me:
# fly.toml file generated for old-frost-6294 on 2022-12-23T17:21:37Z
app = "old-frost-6294"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []
[env]
[experimental]
allowed_public_ports = []
auto_rollback = true
[[services]]
http_checks = []
internal_port = 8080
processes = ["app"]
protocol = "tcp"
script_checks = []
[services.concurrency]
hard_limit = 25
soft_limit = 20
type = "connections"
[[services.ports]]
force_https = true
handlers = ["http"]
port = 80
[[services.ports]]
handlers = ["tls", "http"]
port = 443
[[services.tcp_checks]]
grace_period = "1s"
interval = "15s"
restart_limit = 0
timeout = "2s"
services[].internal_port is already set to 8080, what a coincidence. It has an
HTTP handler on port 80, and a TLS handler on port 443, this is all we want.
Note that because that fly.toml has app = "old-frost-6294", as long as our
shell is in that directory, we don't need to pass -a / --app to flyctl.
Handy!
The only missing bit is "what should be deployed", and we can specify that by adding a new section:
[build] image = "catscii"
fly.io also provides "remote builders" that can build your Dockerfile
directly. You can also
force it to build using your local Docker daemon by passing --local-only
to fly deploy.
However, because using Dockerfile is just a phase, and we'll eventually
use nix to generate the container image, we're specifying an image directly
here.
With that, we're ready to deploy!
$ fly deploy --local-only ==> Verifying app config --> Verified app config ==> Building image Searching for image 'catscii' locally... image found: sha256:c2a27c54dbe6940d7fd82a2d962c832c46b3596d1a328751daa43412e7fc5c3e ==> Pushing image to fly The push refers to repository [registry.fly.io/old-frost-6294] 3c47a1ca65c2: Pushed 8a6f14bd6616: Pushed 42957de36bb8: Pushed d49aaa665130: Pushed 5f70bf18a086: Pushed 531741825ed0: Pushing [======> ] 145.8MB/1.157GB 57c57da67e69: Pushing [===========================> ] 142.2MB/255.6MB 0002c93bdb37: Pushing [======================> ] 32.6MB/72.79MB ^CWARN failed to finish build in graphql: Post "https://api.fly.io/graphql": context canceled
Wait no, hold up, Ctrl-C that. What's that in the middle: a 1.157GB layer?
fly.io can handle that, but let's... not. Let's take a few minutes to make our image slimmer.
Multi-stage docker builds to make our image slimmer
The newer Dockerfile syntax, in conjunction with BuildKit, lets us have multi-stage builds, and so we can have a "builder" target (that has all the compile-time dependencies), and an "app" target (that has only the runtime dependencies).
Let's try it out:
# syntax = docker/dockerfile:1.4
################################################################################
FROM ubuntu:20.04 AS base
################################################################################
FROM base AS builder
# Install compile-time dependencies
RUN set -eux; \
apt update; \
apt install -y --no-install-recommends \
curl ca-certificates gcc libc6-dev pkg-config libssl-dev \
;
# Install rustup
RUN set -eux; \
curl --location --fail \
"https://static.rust-lang.org/rustup/dist/x86_64-unknown-linux-gnu/rustup-init" \
--output rustup-init; \
chmod +x rustup-init; \
./rustup-init -y --no-modify-path --default-toolchain stable; \
rm rustup-init;
# Add rustup to path, check that it works
ENV PATH=${PATH}:/root/.cargo/bin
RUN set -eux; \
rustup --version;
# Copy sources and build them
WORKDIR /app
COPY src src
COPY Cargo.toml Cargo.lock ./
RUN --mount=type=cache,target=/root/.rustup \
--mount=type=cache,target=/root/.cargo/registry \
--mount=type=cache,target=/root/.cargo/git \
--mount=type=cache,target=/app/target \
set -eux; \
cargo build --release; \
cp target/release/catscii .
################################################################################
FROM base AS app
# Install run-time dependencies, remove extra APT files afterwards.
# This must be done in the same `RUN` command, otherwise it doesn't help
# to reduce the image size.
RUN set -eux; \
apt update; \
apt install -y --no-install-recommends \
ca-certificates \
; \
apt clean autoclean; \
apt autoremove --yes; \
rm -rf /var/lib/{apt,dpkg,cache,log}/
# Copy app from builder
WORKDIR /app
COPY --from=builder /app/catscii .
CMD ["/app/catscii"]
Let's build it:
$ DOCKER_BUILDKIT=1 docker build --target app --tag catscii . (cut) => => naming to docker.io/library/catscii
And run it:
$ docker run -it --env SENTRY_DSN --env HONEYCOMB_API_KEY --publish 8080:8080/tcp --rm catscii
{"timestamp":"2022-12-23T17:36:03.032745Z","level":"INFO","fields":{"message":"Creating honey client","log.target":"libhoney::client","log.module_path":"libhoney::client","log.file":"/root/.cargo/git/checkouts/libhoney-rust-3b5a30a6076b74c0/9871051/src/client.rs","log.line":78},"target":"libhoney::client"}
{"timestamp":"2022-12-23T17:36:03.032820Z","level":"INFO","fields":{"message":"transmission starting","log.target":"libhoney::transmission","log.module_path":"libhoney::transmission","log.file":"/root/.cargo/git/checkouts/libhoney-rust-3b5a30a6076b74c0/9871051/src/transmission.rs","log.line":124},"target":"libhoney::transmission"}
{"timestamp":"2022-12-23T17:36:03.036680Z","level":"INFO","fields":{"message":"Listening on 0.0.0.0:8080"},"target":"catscii"}
^C{"timestamp":"2022-12-23T17:36:03.958953Z","level":"WARN","fields":{"message":"Initiating graceful shutdown"},"target":"catscii"}
Still runs!
What's the image size?
$ docker image ls catscii REPOSITORY TAG IMAGE ID CREATED SIZE catscii latest 413b40968a62 About a minute ago 139MB
Okay, that's much better.
Shrinking down the image size further
We can probably do better still, though... I've used dive to look into the image size:
$ dive catscii (see screenshot below)
Looking at this, I don't know about you, but I... I think this line from our
Dockerfile isn't working as intended:
rm -rf /var/lib/{apt,dpkg,cache,log}/
In fact, let's look around:
$ docker run -it --rm catscii /bin/sh
# ls /var/lib/
apt dpkg misc pam systemd
# rm -rf /var/lib/{apt,dpkg,cache,log}/
# ls /var/lib/
apt dpkg misc pam systemd
Oh, okay, cool.
So it turns out brace
expansion
is "a bash-ism". It is supported by the bash shell, but not by a strictly
POSIX-compliant shell, which is used to execute our Dockerfile RUN directives.
So, it's not actually running:
$ rm -rf /var/lib/apt/ /var/lib/dpkg/ /var/lib/cache/ /var/lib/log/
It's actually running:
$ rm -rf '/var/lib/{apt,dpkg,cache,log}/'
And -f makes rm not complain if the target doesn't exist, so... nothing
happens.
If we were writing regular shell scripts, a tool like shellcheck may have saved us:
$ shellcheck ./rm.sh
In ./rm.sh line 2:
rm -rf /var/lib/{apt,dpkg,cache,log}/
^------------------^ SC3009 (warning): In POSIX sh, brace expansion is undefined.
For more information:
https://www.shellcheck.net/wiki/SC3009 -- In POSIX sh, brace expansion is u...
...but I don't think shellcheck is able to parse/interpret Dockerfiles, so.
There's two potential fixes for that: we can either do the expansion ourselves,
or we can switch the shell used for RUN commands:
################################################################################
FROM base AS app
# 👇 new!
SHELL ["/bin/bash", "-c"]
# Install run-time dependencies, remove extra APT files afterwards.
# This must be done in the same `RUN` command, otherwise it doesn't help
# to reduce the image size.
RUN set -eux; \
apt update; \
apt install -y --no-install-recommends \
ca-certificates \
; \
apt clean autoclean; \
apt autoremove --yes; \
# Note: 👇 this only works because of the `SHELL` instruction above.
rm -rf /var/lib/{apt,dpkg,cache,log}/
And now our image is...
$ DOCKER_BUILDKIT=1 docker build --target app --tag catscii . (cut) $ docker image ls catscii REPOSITORY TAG IMAGE ID CREATED SIZE catscii latest d4efaf33cd3e 16 minutes ago 99.1MB
Under 100MB! That's better, but we can still improve on it.
The layer where we copy our app's binary in-place is 20MB right now. If we
change cp to an objcopy invocation that compresses debug info, we can bring
that down a little.
# in "FROM base AS builder"
# omitted: installing compile-time dependencies, etc.
# Copy sources and build them
WORKDIR /app
COPY src src
COPY Cargo.toml Cargo.lock ./
RUN --mount=type=cache,target=/root/.rustup \
--mount=type=cache,target=/root/.cargo/registry \
--mount=type=cache,target=/root/.cargo/git \
--mount=type=cache,target=/app/target \
set -eux; \
cargo build --release; \
# 👇 new!
objcopy --compress-debug-sections ./target/release/catscii ./catscii
I'm not even actually sure why this does anything, since debug defaults to
false for the release profile, see the docs.
Without going too far down the rabbit hole, we can pick a few tricks from the excellent min-sized-rust page, and end up with something I'd personally be happy to deploy in production:
# in Cargo.toml # at the bottom of the file [profile.release] debug = 1 # Include enough debug info for sentry to be useful opt-level = "z" # Optimize for size. lto = true # Enable link-time optimization
For me, this comes out at 16MB too, but this time we actually have debug info. Not that I expect our service to panic in production, but, you know, this way we have it.
Actually deploying it to fly.io
Anyway, enough bikeshedding - let's push that <100MB image to fly:
$ DOCKER_BUILDKIT=1 docker build --target app --tag catscii . (cut) $ fly deploy --local-only ==> Verifying app config --> Verified app config ==> Building image Searching for image 'catscii' locally... image found: sha256:13cd689191106e5a503fa73af43ca51bd2d9c82c91ba5b4a6a52f7574873222b ==> Pushing image to fly The push refers to repository [registry.fly.io/old-frost-6294] 2df41807956e: Pushed 81569901f7fe: Pushed 50161719fe01: Pushed 0002c93bdb37: Pushed deployment-01GN030CXYTQ784CKH0TDMNT5C: digest: sha256:5d00cfb31fa901e1de011aa67250b5a4e982d0990dd0f2ace3c98958ea847f27 size: 1158 --> Pushing image done ==> Creating release --> release v2 created --> You can detach the terminal anytime without stopping the deployment ==> Monitoring deployment Logs: https://fly.io/apps/old-frost-6294/monitoring v0 is being deployed
Spoiler alert: this deployment will fail. Can you guess why?
I, uhh... no?
Luckily, fly deploy tells us why. After a couple tries, here's the rest of
the output:
# (continued) ==> Monitoring deployment Logs: https://fly.io/apps/old-frost-6294/monitoring 1 desired, 1 placed, 0 healthy, 1 unhealthy [restarts: 2] [health checks: 1 total] Failed Instances Failure #1 Instance ID PROCESS VERSION REGION DESIRED STATUS HEALTH CHECKS RESTARTS CREATED 523218db app 0 cdg run running 1 total 2 22s ago Recent Events TIMESTAMP TYPE MESSAGE 2022-12-23T18:13:16Z Received Task received by client 2022-12-23T18:13:16Z Task Setup Building Task Directory 2022-12-23T18:13:24Z Started Task started by client 2022-12-23T18:13:26Z Terminated Exit Code: 101 2022-12-23T18:13:26Z Restarting Task restarting in 1.20387713s 2022-12-23T18:13:37Z Started Task started by client 2022-12-23T18:13:39Z Terminated Exit Code: 101 2022-12-23T18:13:39Z Restarting Task restarting in 1.214914905s 2022-12-23T18:13:47Z Started Task started by client 2022-12-23T18:13:24Z [info]2022/12/23 18:13:24 listening on [fdaa:0:3614:a7b:ae02:5232:18db:2]:22 (DNS: [fdaa::3]:53) 2022-12-23T18:13:24Z [info]thread 'main' panicked at '$SENTRY_DSN must be set: NotPresent', src/main.rs:37:37 2022-12-23T18:13:24Z [info]note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace (cut: the same thing two more times) --> v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1 --> Troubleshooting guide at https://fly.io/docs/getting-started/troubleshooting/ Error abort
Oh right! The environment variables.
Yes indeed!
Setting secrets
Easy enough, we can use fly secrets set and pass them as arguments, or fly secrets import.
For Honeycomb, we should make a "production API key", whereas for Sentry, we can re-use the same API key, and it should auto-detect the environment.
The way the sentry crate "detects production environment" is simply by
checking
whether the debug_assertions cfg-attribute is set or not. It's usually set for
debug builds, and unset for release builds, so it all works out... unless you
run release builds in development.
To make a new Honeycomb API key, click the environment picker on the top-left (below the logo), then click "Manage environments". If needed, create the "production" environment, and from there, there should be an "API keys" tab where you can see existing API keys / create new ones.
Note that the security around those monitoring/telemetry API keys is significantly relaxed compared to others: the worst thing someone can do if they get their hands on it is send data somewhere they'll never see. You still shouldn't commit them in plaintext, but it's not as radioactive as most credentials (and very easy to rotate).
Anyway, let's import our secrets:
# note: we're running this from our `catscii/` directory, that has a `fly.toml` # file, so we don't need to specify -a / --app $ fly secrets import SENTRY_DSN=https://redacted@redacted.ingest.sentry.io/redacted HONEYCOMB_API_KEY=redacted # press Ctrl-D when done Release v1 created ==> Monitoring deployment Logs: https://fly.io/apps/old-frost-6294/monitoring v1 is being deployed
Note that double-quoting (ie. KEY="VALUE") is not supported at the time of
this writing, unlike in Docker .env
files, see that discussion.
Our deploy should succeed.
Watching our application in production
We can now look at logs in the CLI:
$ fly logs
(cut)
2022-12-23T18:31:39Z app[89d7ad81] cdg [info]Starting init (commit: f447594)...
2022-12-23T18:31:39Z app[89d7ad81] cdg [info]Preparing to run: `/app/catscii` as root
2022-12-23T18:31:39Z app[89d7ad81] cdg [info]2022/12/23 18:31:39 listening on [fdaa:0:3614:a7b:abd:89d7:ad81:2]:22 (DNS: [fdaa::3]:53)
2022-12-23T18:31:39Z app[89d7ad81] cdg [info]{"timestamp":"2022-12-23T18:31:39.396353Z","level":"INFO","fields":{"message":"Creating honey client","log.target":"libhoney::client","log.module_path":"libhoney::client","log.file":"/root/.cargo/git/checkouts/libhoney-rust-3b5a30a6076b74c0/9871051/src/client.rs","log.line":78},"target":"libhoney::client"}
2022-12-23T18:31:39Z app[89d7ad81] cdg [info]{"timestamp":"2022-12-23T18:31:39.396519Z","level":"INFO","fields":{"message":"transmission starting","log.target":"libhoney::transmission","log.module_path":"libhoney::transmission","log.file":"/root/.cargo/git/checkouts/libhoney-rust-3b5a30a6076b74c0/9871051/src/transmission.rs","log.line":124},"target":"libhoney::transmission"}
2022-12-23T18:31:39Z app[89d7ad81] cdg [info]{"timestamp":"2022-12-23T18:31:39.406458Z","level":"INFO","fields":{"message":"Listening on 0.0.0.0:8080"},"target":"catscii"}
We can look up the app's status:
$ fly status
App
Name = old-frost-6294
Owner = personal
Status = running
Version = 2
Platform = nomad
Hostname = old-frost-6294.fly.dev
Services
PROTOCOL PORTS
TCP 80 => 8080 [HTTP]
443 => 8080 [TLS, HTTP]
IP Addresses
TYPE ADDRESS REGION CREATED AT
v6 2a09:8280:1::3:bb39 global 24m40s ago
We get our own IPv6 address, and a shared IPv4 address:
$ fly ips list VERSION IP TYPE REGION CREATED AT v6 2a09:8280:1::3:bb39 public global 25m5s ago v4 66.241.125.118 public (shared)
Visiting https://old-frost-6294.fly.dev/ in a browser works for me!
Over at https://fly.io/dashboard, there's a ton of cool things to see:
Like logs:
And metrics:
We even get Grafana:
Pretty neat!
This article is part 6 of the Building a Rust service with Nix series.
If you liked what you saw, please support my work!
