Avatar for fasterthanlime fasterthanlime
Nov 15, 2022 clock icon 10 min #rust · #cargo-edit · #ftl-infra

Cleaning up and upgrading third-party crates

From the series Updating fasterthanli.me for 2022

1

2

3

4

5

6

7

8

👋 This page was last updated ~2 years ago. Just so you know.

The bleeding edge of rustc and clippy

Typically, you'd want a production application to use a stable version of Rust. At the time of this writing, that's Rust 1.65.0, which stabilizes a bunch of long-awaited features (GATs, let-else, MIR inlining, split debug info, etc.).

Cool bear

Cool bear's hot tip

For every Rust release, Mara makes a wonderful recap thread on Twitter, on top of the official announcement.

All my repositories have a rust-toolchain.toml file that "pins" to a specific version of the rust compiler. Today, it looks like this:

[toolchain]
channel = "nightly-2022-11-07"
components = [ "rustfmt", "clippy" ]

Nightly versions are a great fit for me, since it lets me play with features before they're ready (good for me), and report breakage early on (good for you!), like I did with let-else and async fn in trait.

Upgrading to a new Rust version is always a bit of a treat. Rarely there will be breaking changes, for extremely good reasons, like "that was unsound", but otherwise, things just generally keep working. But each new rustc version also comes with a new clippy version, and that means I get to find new ways to improve my code!

One thing rustc has gotten very good over the past year is finding needless borrows.

Consider this code, that uses the katex crate, something I use for LaTeX-like rendering of mathematical notation on that site (..and that I rarely reach for):

	let kopts = katex::Opts::builder().display_mode(false).build().unwrap();
	let res = katex::render_with_opts(math, &kopts);

The signature for render_with_opts is this:

	pub fn render_with_opts(input: &str, opts: impl AsRef<Opts>) -> Result<String>

For opts, it takes any type that implements AsRef<Opts>. This is a neat way to accept either a reference or a value. Opts implements AsRef<Opts>, and so does &Opts, thanks to this blanket impl:

impl<T, U> AsRef<U> for &T
where
		T: AsRef<U> + ?Sized,
		U: ?Sized,

However, in this case, kopts is used once then never again, so there's no point in passing &kopts rather than kopts, and that's what clippy tells us:

$ cargo clippy
cargo clippy
    Blocking waiting for file lock on build directory
warning: the borrowed expression implements the required traits
  --> crates/futile-templating/src/markup.rs:48:57
   |
48 |                 let res = katex::render_with_opts(math, &kopts);
   |                                                         ^^^^^^ help: change this to: `kopts`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow
   = note: `#[warn(clippy::needless_borrow)]` on by default

warning: `futile-templating` (lib) generated 1 warning (run `cargo fix --lib -p futile-templating` to apply 1 suggestion)
    Finished dev [unoptimized + debuginfo] target(s) in 2.61s

This suggestion is machine-applicable, which means running "cargo fix" (or accepting the suggestion from your code editor) will fix it automatically. Not only does it make the code simpler (one less byte, yes!), it also means less work for the compiler to do, hence, faster compiles.

This is the tiniest, silliest example of all these, and of course fixing just one won't do much, but it adds up, and some of the clippy lints are getting increasingly clever, which is good! Most of the time!

Sometimes clippy lints are false positives, but you can always choose to ignore them.

Consider this code:

use liquid_core::{to_value, Display_filter, Filter, FilterReflection, ParseFilter};

#[derive(Clone, ParseFilter, FilterReflection)]
#[filter(
    name = "reading_time",
    description = "Estimate text reading time",
    parsed(ReadingTimeFilter)
)]
pub struct ReadingTime;

This uses the liquid family of crates, which I use for templating on this here website.

clippy's warning is as follows:

$ cargo clippy
warning: `Box::new(_)` of default value
  --> crates/futile-templating/src/filters/reading_time.rs:11:12
   |
11 |     parsed(ReadingTimeFilter)
   |            ^^^^^^^^^^^^^^^^^ help: try: `Box::default()`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#box_default
   = note: `#[warn(clippy::box_default)]` on by default

warning: `futile-templating` (lib) generated 1 warning (run `cargo fix --lib -p futile-templating` to apply 1 suggestion)
    Finished dev [unoptimized + debuginfo] target(s) in 0.22s

Clicking on the provided link (how helpful!) lets us know that it essentially wants us to turn Box::new(T::default()) to something like Box::<T>::default().

Here it's not just about making code simpler (although it is one less call, so, less work for the compiler), it's also about performance: the latter form lets the compiler create the value directly on the heap.

In this case though, the problem lies in the upstream derive macro, and so I can't do anything about it except ignore the lint for the time being:

// at the top of that file:

#![allow(clippy::box_default)]

This ignores the clippy::box_default lint for the whole module.

Normally you can be more specific about where you ignore a lint, for example, at the start of a function:

#[allow(clippy::box_default)]
fn i_know_what_im_doing() -> Box<u64> {
	// doing this on purpose, leave me alone clippy
	Box::new(Default::default())
}

Or even directly on some expression:

fn i_know_what_im_doing() -> Box<u64> {
	#[allow(clippy::box_default)] // doing this on purpose, leave me alone clippy
	Box::new(Default::default())
}

...but since here it's in code generated by a derive macro, I can't really do much better than ignoring it for the whole module. If I really did want to scope it down, I could add a nested module, and re-export the type:

mod inner {
    #![allow(clippy::box_default)]

    use super::ReadingTimeFilter;
    use liquid_core::{FilterReflection, ParseFilter};

    #[derive(Clone, ParseFilter, FilterReflection)]
    #[filter(
        name = "reading_time",
        description = "Estimate text reading time",
        parsed(ReadingTimeFilter)
    )]
    pub struct ReadingTime;
}

pub use inner::ReadingTime;

...but here it's already in a pretty small module to start with, so I won't bother.

So, almost all of the time, clippy lints are good: they catch bugs (remember my mutex horror story? there's a lint for that now), they reduce compile times, and they make code faster.

Because the gap between the last nightly I was using and today's nightly was of almost two months, there's some new features to play with, and I'll talk about them later in this article.

A recent cargo feature: workspace dependencies

In My ideal Rust workflow, I mentioned cargo-hakari. The basic idea is: you have a large cargo workspace, with a lot of crates.

Some of those crates have common dependencies, but maybe they have different features turned on.

If you build those individual targets (say you're building one of multiple CLI tools, or you're running tests) from the workspace, those dependencies are built several times: for each combination of feature flags.

cargo-hakari unifies feature flags, so that crates are built at most once: it implements what we call the "workspace hack". And what do with hacks? We retire them!

I was a happy hakari user, more or less — even with git hooks and CI pipelines, it was easy for the workspace-hack crate to get out of sync. Sometimes hakari wouldn't run, and it was hard to pinpoint why. It added a certain maintenance burden.

cargo recently (as of 1.64.0) got support for workspace dependencies.

This is what my workspace (top-level) Cargo.toml used to look like:

# in `Cargo.toml`

[workspace]
members = [
	"crates/base62",
	"crates/futile-asset-rewriter",
	"crates/futile-backtrace-printer",
	"crates/futile-config",
	"crates/futile-content",
	"crates/futile-db",
	"crates/futile-extract-text",
	"crates/futile-friendcodes",
	"crates/futile-frontmatter",
	"crates/futile-highlight",
	"crates/futile-parent-path",
	"crates/futile-patreon",
	"crates/futile-reddit",
	"crates/futile-templating",
	"crates/futile",
	"workspace-hack",
]

And each individual crates/xxx/Cargo.toml looked something like:

# in `crates/futile-reddit/Cargo.toml`

[package]
name = "futile-reddit"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
eyre = "0.6.8"
serde = { version = "1.0.145", features = ["derive"] }
tracing = "0.1.36"
url = "2.3.1"
reqwest = { version = "0.11.12", default-features = false, features = ["json", "gzip", "rustls-tls"] }
thiserror = "1.0.37"

Now, my top-level Cargo.toml looks like this: first off, and this is tangential, because every subdirectory of crates/ is.. a crate, I can just use a glob. But also, I can specify dependencies directly in the workspace:

[workspace]
members = [
	"crates/*",
]

[workspace.dependencies]
eyre = "0.6.8"
serde = { version = "1.0.145", features = ["derive"] }
tracing = "0.1.36"
url = "2.3.1"
reqwest = { version = "0.11.12", default-features = false, features = ["json", "gzip", "rustls-tls"] }
thiserror = "1.0.37"

And then, individual crate manifests look like this:

# in `crates/futile-reddit/Cargo.toml`

[package]
name = "futile-reddit"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
eyre.workspace = true
serde.workspace = true
tracing.workspace = true
url.workspace = true
reqwest.workspace = true
thiserror.workspace = true

This isn't a full replacement for cargo-hakari. As Rain points out, since not every crate of the workspace pulls in every dependency (what the workspace-hack crate does), transitive dependencies might enable features conditionally, and so, even with workspace dependencies, you'll have unnecessary rebuilds.

For me though, I'm happy to take the compromise and simplify maintenance a little: now, there's a single place for me to upgrade crate versions, and a single place for me to look when I want to reduce the amount of crates I depend on (without having to run cargo tree).

As of right now, I have 92 direct dependencies (and 1602 transitive ones). In my defense, my website does a lot. But I still think this could be reduced easily.

Finding unused dependencies with cargo-udeps

One way to reduce your carbon footprint number of dependencies is to find unused ones! cargo-udeps is a very neat tool for that.

Previously I had to disable and re-enable hakari to be able to run cargo-udeps, but now I don't need to anymore. Also, because I'm on Rust nightly, I don't need to have another toolchain just to run udeps, or export RUSTC_BOOTSTRAP=1 (which is frowned upon) just to be able to use the same toolchain.

$ just udeps
cargo udeps --all-targets --backend depinfo
    Checking futile-config v0.1.0 (/home/amos/bearcove/futile/crates/futile-config)
    (cut)
    Checking futile-content v0.1.0 (/home/amos/bearcove/futile/crates/futile-content)
    Checking futile v1.9.0 (/home/amos/bearcove/futile/crates/futile)
    Finished dev [unoptimized + debuginfo] target(s) in 2.50s
info: Loading depinfo from "/home/amos/bearcove/futile/target/debug/deps/futile_config-bb658f027edfa45f.d"
(cut)
info: Loading depinfo from "/home/amos/bearcove/futile/target/debug/deps/futile-8f4c9db8156e313c.d"
unused dependencies:
`futile-content v0.1.0 (/home/amos/bearcove/futile/crates/futile-content)`
└─── dependencies
     └─── "ulid"
Note: They might be false-positive.
      For example, `cargo-udeps` cannot detect usage of crates that are only used in doc-tests.
      To ignore some dependencies, write `package.metadata.cargo-udeps.ignore` in Cargo.toml.
error: Recipe `udeps` failed on line 34 with exit code 1
Cool bear

Cool bear's hot tip

The just command you see above is the just command runner, which is a lot simpler than GNU make, conceptually, and is a great replacement for most of the things folks use make for (ie. anything that doesn't involve dependency tracking, generated targets etc.)

Here, udeps is right on the money and I am indeed able to remove ulid from the dependencies of futile-content. It doesn't matter much since it is being used by other crates in the workspace, but, when it no longer is, udeps will catch that, too.

Upgrading dependencies with (a recent) cargo-edit

There's a few approaches to upgrading dependencies: cargo-outdated will show you everything you ought to update. Let's see what it tells us:

$ cargo outdated
error: no matching package named `futile-asset-rewriter` found
location searched: registry `crates-io`
required by package `futile v1.9.0 (/home/amos/bearcove/futile/crates/futile)`

Ah. It seems it's not fully compatible with workspace dependencies yet! The dependency is simply futile-asset-rewriter.workspace = true, which points to:

# in the top-level `Cargo.toml` file

[workspace.dependencies]
futile-asset-rewriter = { version = "0.1.0", path = "crates/futile-asset-rewriter" }

...however cargo-outdated doesn't see that, and seems to think it's just a crate off of https://crates.io, so it won't work for us for now. (It usually does work just fine!).

Next up, we can try cargo-edit, with its upgrade subcommand.

We'll do a "dry run" first, just to see what it would do.

$ cargo upgrade --dry-run
    Updating 'https://github.com/rust-lang/crates.io-index' index
    Checking virtual workspace's dependencies
    Updating 'https://github.com/bearcove/crates.git' index
name               old req compatible latest  new req note
====               ======= ========== ======  ======= ====
async-trait        0.1.57  0.1.58     0.1.58  0.1.58
aws-config         0.49.0  0.49.0     0.51.0  0.49.0  incompatible
aws-sdk-s3         0.19.0  0.19.0     0.21.0  0.19.0  incompatible
aws-types          0.49.0  0.49.0     0.51.0  0.49.0  incompatible
clap               4.0.4   4.0.19     4.0.19  4.0.19
fs-err             2.8.1   2.9.0      2.9.0   2.9.0
futures            0.3.24  0.3.25     0.3.25  0.3.25
hmac-sha256        1.1.4   1.1.5      1.1.5   1.1.5
html-escape        0.2.11  0.2.12     0.2.12  0.2.12
hyper              0.14.20 0.14.22    0.14.22 0.14.22
ipnet              2.5.0   2.5.1      2.5.1   2.5.1
moka               0.9.4   0.9.6      0.9.6   0.9.6
regex              1.6.0   1.7.0      1.7.0   1.7.0
sentry             0.27.0  0.27.0     0.28.0  0.27.0  incompatible
serde              1.0.145 1.0.147    1.0.147 1.0.147
serde_json         1.0.85  1.0.87     1.0.87  1.0.87
serde_yaml         0.9.13  0.9.14     0.9.14  0.9.14
time               0.3.14  0.3.17     0.3.17  0.3.17
tokio-stream       0.1.10  0.1.11     0.1.11  0.1.11
tracing            0.1.36  0.1.37     0.1.37  0.1.37
tracing-subscriber 0.3.15  0.3.16     0.3.16  0.3.16
    Checking base62's dependencies
    Checking futile's dependencies
    Checking futile-asset-rewriter's dependencies
    Checking futile-backtrace-printer's dependencies
    Checking futile-config's dependencies
    Checking futile-content's dependencies
    Checking futile-db's dependencies
    Checking futile-extract-text's dependencies
    Checking futile-friendcodes's dependencies
    Checking futile-frontmatter's dependencies
    Checking futile-highlight's dependencies
    Checking futile-parent-path's dependencies
    Checking futile-patreon's dependencies
    Checking futile-query's dependencies
    Checking futile-reddit's dependencies
    Checking futile-templating's dependencies
note: Re-run with `--incompatible` to upgrade incompatible version requirements
note: Re-run with `--verbose` to show all dependencies
  local: futile-query
  unchanged: arc-swap, async-stream, base-x, base62, broth, bytes, camino, chrono, color-backtrace, color-eyre, cookie, derive-try-from-primitive, eyre, feed-rs, flume, futile-asset-rewriter, futile-backtrace-printer, futile-config, futile-content, futile-db, futile-extract-text, futile-friendcodes, futile-frontmatter, futile-highlight, futile-parent-path, futile-patreon, futile-reddit, futile-templating, grass, http, ignore, jsonapi, katex, kstring, lazy_static, liquid, liquid-core, lol_html, memmap, notify, opentelemetry, opentelemetry-otlp, opentelemetry-semantic-conventions, parking_lot, pin-project-lite, pithy, pretty_assertions, pulldown-cmark, reqwest, rmp-serde, seahash, serde_urlencoded, slug, sqlx, tempfile, termcolor, test-log, thiserror, tokio, tokio-io-timeout, tokio-util, toml, tower, tower-request-id, tracing-opentelemetry, tree-sitter-collection, trust-dns-resolver, ulid, url, warp
warning: aborting upgrade due to dry run

That one seems to work with workspace dependencies, and lets us know that all upgrades except for aws-* and sentry are "compatible" updates, which means, if they follow semver, we should be able to upgrade all of these without breaking anything.

This is the default behavior of cargo upgrade, so let's do that:

$ cargo upgrade
    Updating 'https://github.com/rust-lang/crates.io-index' index
    Checking virtual workspace's dependencies
    Updating 'https://github.com/bearcove/crates.git' index
(cut, same output as `--dry-run`)
    Checking base62's dependencies
    Checking futile's dependencies
    Checking futile-asset-rewriter's dependencies
    (cut)
    Checking futile-templating's dependencies
   Upgrading recursive dependencies
    Blocking waiting for file lock on package cache
    Updating async-io v1.8.0 -> v1.10.0
    Updating async-lock v2.5.0 -> v2.6.0
    Updating bumpalo v3.8.0 -> v3.11.0
    Updating cc v1.0.73 -> v1.0.74
    Updating getrandom v0.2.7 -> v0.2.8
    Updating h2 v0.3.14 -> v0.3.15
    Updating itoa v1.0.3 -> v1.0.4
    Updating js-sys v0.3.59 -> v0.3.60
    Updating libc v0.2.134 -> v0.2.137
    Updating mio v0.8.4 -> v0.8.5
    Updating num_cpus v1.13.1 -> v1.14.0
    Updating once_cell v1.15.0 -> v1.16.0
    Updating os_str_bytes v6.3.0 -> v6.3.1
    Updating parking_lot_core v0.9.3 -> v0.9.4
    Updating polling v2.3.0 -> v2.4.0
    Updating proc-macro2 v1.0.46 -> v1.0.47
    Updating raw-cpuid v10.5.0 -> v10.6.0
    Updating regex-syntax v0.6.27 -> v0.6.28
    Updating smallvec v1.9.0 -> v1.10.0
    Updating syn v1.0.101 -> v1.0.103
    Updating triomphe v0.1.5 -> v0.1.8
    Updating ucd-util v0.1.8 -> v0.1.9
    Updating unicode-ident v1.0.4 -> v1.0.5
    Updating utf8-ranges v1.0.4 -> v1.0.5
    Updating utf8-width v0.1.5 -> v0.1.6
    Updating uuid v1.1.2 -> v1.2.1
    Updating wasm-bindgen v0.2.82 -> v0.2.83
    Updating wasm-bindgen-backend v0.2.82 -> v0.2.83
    Updating wasm-bindgen-macro v0.2.82 -> v0.2.83
    Updating wasm-bindgen-macro-support v0.2.82 -> v0.2.83
    Updating wasm-bindgen-shared v0.2.82 -> v0.2.83
    Updating web-sys v0.3.55 -> v0.3.60
      Adding windows-sys v0.42.0
      Adding windows_aarch64_gnullvm v0.42.0
      Adding windows_aarch64_msvc v0.42.0
      Adding windows_i686_gnu v0.42.0
      Adding windows_i686_msvc v0.42.0
      Adding windows_x86_64_gnu v0.42.0
      Adding windows_x86_64_gnullvm v0.42.0
      Adding windows_x86_64_msvc v0.42.0
note: Re-run with `--incompatible` to upgrade incompatible version requirements
note: Re-run with `--verbose` to show all dependencies
  local: futile-query
  unchanged: arc-swap, async-stream, base-x, base62, broth, bytes, camino, chrono, color-backtrace, color-eyre, cookie, derive-try-from-primitive, eyre, feed-rs, flume, futile-asset-rewriter, futile-backtrace-printer, futile-config, futile-content, futile-db, futile-extract-text, futile-friendcodes, futile-frontmatter, futile-highlight, futile-parent-path, futile-patreon, futile-reddit, futile-templating, grass, http, ignore, jsonapi, katex, kstring, lazy_static, liquid, liquid-core, lol_html, memmap, notify, opentelemetry, opentelemetry-otlp, opentelemetry-semantic-conventions, parking_lot, pin-project-lite, pithy, pretty_assertions, pulldown-cmark, reqwest, rmp-serde, seahash, serde_urlencoded, slug, sqlx, tempfile, termcolor, test-log, thiserror, tokio, tokio-io-timeout, tokio-util, toml, tower, tower-request-id, tracing-opentelemetry, tree-sitter-collection, trust-dns-resolver, ulid, url, warp

Everything still builds, my tests still pass, a quick QA round indicates that the site still works as intended.

As for the semver-breaking changes, we can target sentry:

$ cargo upgrade -p sentry
(cut)

A quick look at the changelog indicates that there's nothing to worry about. They bumped the MSRV (Minimum Supported Rust Version) to 1.60.0, whereas the nightly I'm using is the equivalent of 1.67.0 — we're good.

Then, we can upgrade the aws-* family of crates, the last few "incompatible" upgrades available:

$ cargo upgrade --incompatible
(cut)
    Updating 'https://github.com/rust-lang/crates.io-index' index
    Checking virtual workspace's dependencies
    Updating 'https://github.com/bearcove/crates.git' index
name       old req compatible latest new req
====       ======= ========== ====== =======
aws-config 0.49.0  0.49.0     0.51.0 0.51.0
aws-sdk-s3 0.19.0  0.19.0     0.21.0 0.21.0
aws-types  0.49.0  0.49.0     0.51.0 0.51.0
sentry     0.27.0  0.27.0     0.28.0 0.28.0
(cut)

Neither of these were actually "incompatible" in the sense that nothing broke, but we weren't covered by semver guarantees (regardless of how much you believe in them).

You're reading the Updating fasterthanli.me for 2022 series.

1

2

3

4

5

6

7

8

Comment on /r/fasterthanlime

(JavaScript is required to see this. Or maybe my stuff broke)

Here's another article just for you:

I am a Java, C#, C or C++ developer, time to do some Rust

Aug 12, 2020
clock icon 1 hour 13 min #rust · #traits · #lifetimes

As I've said before, I'm working on a book about lifetimes. Or maybe it's just a long series - I haven't decided the specifics yet. Like every one of my series/book things, it's long, and it starts you off way in the periphery of the subject, and takes a lot of detours to get there.

In other words - it's great if you want an adventure (which truly understanding Rust definitely is), but it's not the best if you are currently on the puzzled end of a conversation with your neighborhood lifetime enforcer, the Rust compiler.

Read more
About Legal Notice Privacy Policy Terms and Conditions