Cleaning up and upgrading third-party crates
👋 This page was last updated ~2 years ago. Just so you know.
The bleeding edge of rustc and clippy
Typically, you'd want a production application to use a stable version of Rust. At the time of this writing, that's Rust 1.65.0, which stabilizes a bunch of long-awaited features (GATs, let-else, MIR inlining, split debug info, etc.).
For every Rust release, Mara makes a wonderful recap thread on Twitter, on top of the official announcement.
All my repositories have a rust-toolchain.toml
file that "pins" to a specific
version of the rust compiler. Today, it looks like this:
[toolchain] channel = "nightly-2022-11-07" components = [ "rustfmt", "clippy" ]
Nightly versions are a great fit for me, since it lets me play with features before they're ready (good for me), and report breakage early on (good for you!), like I did with let-else and async fn in trait.
Upgrading to a new Rust version is always a bit of a treat. Rarely there will
be breaking changes, for extremely good reasons, like "that was unsound", but
otherwise, things just generally keep working. But each new rustc
version also
comes with a new clippy
version, and that means I get to find new ways to
improve my code!
One thing rustc
has gotten very good over the past year is finding needless
borrows.
Consider this code, that uses the katex crate, something I use for LaTeX-like rendering of mathematical notation on that site (..and that I rarely reach for):
let kopts = katex::Opts::builder().display_mode(false).build().unwrap(); let res = katex::render_with_opts(math, &kopts);
The signature for render_with_opts
is this:
pub fn render_with_opts(input: &str, opts: impl AsRef<Opts>) -> Result<String>
For opts
, it takes any type that implements AsRef<Opts>
. This is a neat way
to accept either a reference or a value. Opts
implements AsRef<Opts>
, and so
does &Opts
, thanks to this blanket impl:
impl<T, U> AsRef<U> for &T where T: AsRef<U> + ?Sized, U: ?Sized,
However, in this case, kopts
is used once then never again, so there's no
point in passing &kopts
rather than kopts
, and that's what clippy tells us:
$ cargo clippy cargo clippy Blocking waiting for file lock on build directory warning: the borrowed expression implements the required traits --> crates/futile-templating/src/markup.rs:48:57 | 48 | let res = katex::render_with_opts(math, &kopts); | ^^^^^^ help: change this to: `kopts` | = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow = note: `#[warn(clippy::needless_borrow)]` on by default warning: `futile-templating` (lib) generated 1 warning (run `cargo fix --lib -p futile-templating` to apply 1 suggestion) Finished dev [unoptimized + debuginfo] target(s) in 2.61s
This suggestion is machine-applicable, which means running "cargo fix" (or accepting the suggestion from your code editor) will fix it automatically. Not only does it make the code simpler (one less byte, yes!), it also means less work for the compiler to do, hence, faster compiles.
This is the tiniest, silliest example of all these, and of course fixing just one won't do much, but it adds up, and some of the clippy lints are getting increasingly clever, which is good! Most of the time!
Sometimes clippy lints are false positives, but you can always choose to ignore them.
Consider this code:
use liquid_core::{to_value, Display_filter, Filter, FilterReflection, ParseFilter}; #[derive(Clone, ParseFilter, FilterReflection)] #[filter( name = "reading_time", description = "Estimate text reading time", parsed(ReadingTimeFilter) )] pub struct ReadingTime;
This uses the liquid family of crates, which I use for templating on this here website.
clippy's warning is as follows:
$ cargo clippy warning: `Box::new(_)` of default value --> crates/futile-templating/src/filters/reading_time.rs:11:12 | 11 | parsed(ReadingTimeFilter) | ^^^^^^^^^^^^^^^^^ help: try: `Box::default()` | = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#box_default = note: `#[warn(clippy::box_default)]` on by default warning: `futile-templating` (lib) generated 1 warning (run `cargo fix --lib -p futile-templating` to apply 1 suggestion) Finished dev [unoptimized + debuginfo] target(s) in 0.22s
Clicking on the provided
link
(how helpful!) lets us know that it essentially wants us to turn
Box::new(T::default())
to something like Box::<T>::default()
.
Here it's not just about making code simpler (although it is one less call, so, less work for the compiler), it's also about performance: the latter form lets the compiler create the value directly on the heap.
In this case though, the problem lies in the upstream derive macro, and so I can't do anything about it except ignore the lint for the time being:
// at the top of that file: #![allow(clippy::box_default)]
This ignores the clippy::box_default
lint for the whole module.
Normally you can be more specific about where you ignore a lint, for example, at the start of a function:
#[allow(clippy::box_default)] fn i_know_what_im_doing() -> Box<u64> { // doing this on purpose, leave me alone clippy Box::new(Default::default()) }
Or even directly on some expression:
fn i_know_what_im_doing() -> Box<u64> { #[allow(clippy::box_default)] // doing this on purpose, leave me alone clippy Box::new(Default::default()) }
...but since here it's in code generated by a derive macro, I can't really do much better than ignoring it for the whole module. If I really did want to scope it down, I could add a nested module, and re-export the type:
mod inner { #![allow(clippy::box_default)] use super::ReadingTimeFilter; use liquid_core::{FilterReflection, ParseFilter}; #[derive(Clone, ParseFilter, FilterReflection)] #[filter( name = "reading_time", description = "Estimate text reading time", parsed(ReadingTimeFilter) )] pub struct ReadingTime; } pub use inner::ReadingTime;
...but here it's already in a pretty small module to start with, so I won't bother.
So, almost all of the time, clippy lints are good: they catch bugs (remember my mutex horror story? there's a lint for that now), they reduce compile times, and they make code faster.
Because the gap between the last nightly I was using and today's nightly was of almost two months, there's some new features to play with, and I'll talk about them later in this article.
A recent cargo feature: workspace dependencies
In My ideal Rust workflow, I mentioned cargo-hakari. The basic idea is: you have a large cargo workspace, with a lot of crates.
Some of those crates have common dependencies, but maybe they have different features turned on.
If you build those individual targets (say you're building one of multiple CLI tools, or you're running tests) from the workspace, those dependencies are built several times: for each combination of feature flags.
cargo-hakari
unifies feature flags, so that crates are built at most once: it
implements what we call the "workspace hack". And what do with hacks? We retire
them!
I was a happy hakari user, more or less — even with git hooks and CI pipelines,
it was easy for the workspace-hack
crate to get out of sync. Sometimes hakari
wouldn't run, and it was hard to pinpoint why. It added a certain maintenance burden.
cargo recently (as of 1.64.0) got support for workspace dependencies.
This is what my workspace (top-level) Cargo.toml
used to look like:
# in `Cargo.toml` [workspace] members = [ "crates/base62", "crates/futile-asset-rewriter", "crates/futile-backtrace-printer", "crates/futile-config", "crates/futile-content", "crates/futile-db", "crates/futile-extract-text", "crates/futile-friendcodes", "crates/futile-frontmatter", "crates/futile-highlight", "crates/futile-parent-path", "crates/futile-patreon", "crates/futile-reddit", "crates/futile-templating", "crates/futile", "workspace-hack", ]
And each individual crates/xxx/Cargo.toml
looked something like:
# in `crates/futile-reddit/Cargo.toml` [package] name = "futile-reddit" version = "0.1.0" edition = "2021" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] eyre = "0.6.8" serde = { version = "1.0.145", features = ["derive"] } tracing = "0.1.36" url = "2.3.1" reqwest = { version = "0.11.12", default-features = false, features = ["json", "gzip", "rustls-tls"] } thiserror = "1.0.37"
Now, my top-level Cargo.toml
looks like this: first off, and this is
tangential, because every subdirectory of crates/
is.. a crate, I can just
use a glob. But also, I can specify dependencies directly in the workspace:
[workspace] members = [ "crates/*", ] [workspace.dependencies] eyre = "0.6.8" serde = { version = "1.0.145", features = ["derive"] } tracing = "0.1.36" url = "2.3.1" reqwest = { version = "0.11.12", default-features = false, features = ["json", "gzip", "rustls-tls"] } thiserror = "1.0.37"
And then, individual crate manifests look like this:
# in `crates/futile-reddit/Cargo.toml` [package] name = "futile-reddit" version = "0.1.0" edition = "2021" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [dependencies] eyre.workspace = true serde.workspace = true tracing.workspace = true url.workspace = true reqwest.workspace = true thiserror.workspace = true
This isn't a full replacement for cargo-hakari
. As Rain points
out, since not every crate of the
workspace pulls in every dependency (what the workspace-hack crate does),
transitive dependencies might enable features conditionally, and so, even
with workspace dependencies, you'll have unnecessary rebuilds.
For me though, I'm happy to take the compromise and simplify maintenance a
little: now, there's a single place for me to upgrade crate versions, and a
single place for me to look when I want to reduce the amount of crates I depend
on (without having to run cargo tree
).
As of right now, I have 92 direct dependencies (and 1602 transitive ones). In my defense, my website does a lot. But I still think this could be reduced easily.
Finding unused dependencies with cargo-udeps
One way to reduce your carbon footprint number of dependencies is to find
unused ones! cargo-udeps is a very neat
tool for that.
Previously I had to disable and re-enable hakari
to be able to run
cargo-udeps
, but now I don't need to anymore. Also, because I'm on Rust
nightly, I don't need to have another toolchain just to run udeps, or export
RUSTC_BOOTSTRAP=1
(which is frowned upon) just to be able to use the same
toolchain.
$ just udeps cargo udeps --all-targets --backend depinfo Checking futile-config v0.1.0 (/home/amos/bearcove/futile/crates/futile-config) (cut) Checking futile-content v0.1.0 (/home/amos/bearcove/futile/crates/futile-content) Checking futile v1.9.0 (/home/amos/bearcove/futile/crates/futile) Finished dev [unoptimized + debuginfo] target(s) in 2.50s info: Loading depinfo from "/home/amos/bearcove/futile/target/debug/deps/futile_config-bb658f027edfa45f.d" (cut) info: Loading depinfo from "/home/amos/bearcove/futile/target/debug/deps/futile-8f4c9db8156e313c.d" unused dependencies: `futile-content v0.1.0 (/home/amos/bearcove/futile/crates/futile-content)` └─── dependencies └─── "ulid" Note: They might be false-positive. For example, `cargo-udeps` cannot detect usage of crates that are only used in doc-tests. To ignore some dependencies, write `package.metadata.cargo-udeps.ignore` in Cargo.toml. error: Recipe `udeps` failed on line 34 with exit code 1
The just
command you see above is the just command
runner, which is a lot simpler than GNU make,
conceptually, and is a great replacement for most of the things folks use make
for (ie. anything that doesn't involve dependency tracking, generated targets
etc.)
Here, udeps is right on the money and I am indeed able to remove ulid
from the
dependencies of futile-content
. It doesn't matter much since it is being
used by other crates in the workspace, but, when it no longer is, udeps will
catch that, too.
Upgrading dependencies with (a recent) cargo-edit
There's a few approaches to upgrading dependencies: cargo-outdated will show you everything you ought to update. Let's see what it tells us:
$ cargo outdated error: no matching package named `futile-asset-rewriter` found location searched: registry `crates-io` required by package `futile v1.9.0 (/home/amos/bearcove/futile/crates/futile)`
Ah. It seems it's not fully compatible with workspace dependencies yet! The
dependency is simply futile-asset-rewriter.workspace = true
, which points to:
# in the top-level `Cargo.toml` file [workspace.dependencies] futile-asset-rewriter = { version = "0.1.0", path = "crates/futile-asset-rewriter" }
...however cargo-outdated
doesn't see that, and seems to think it's just
a crate off of https://crates.io, so it won't work for us for now. (It usually
does work just fine!).
Next up, we can try cargo-edit, with its
upgrade
subcommand.
We'll do a "dry run" first, just to see what it would do.
$ cargo upgrade --dry-run Updating 'https://github.com/rust-lang/crates.io-index' index Checking virtual workspace's dependencies Updating 'https://github.com/bearcove/crates.git' index name old req compatible latest new req note ==== ======= ========== ====== ======= ==== async-trait 0.1.57 0.1.58 0.1.58 0.1.58 aws-config 0.49.0 0.49.0 0.51.0 0.49.0 incompatible aws-sdk-s3 0.19.0 0.19.0 0.21.0 0.19.0 incompatible aws-types 0.49.0 0.49.0 0.51.0 0.49.0 incompatible clap 4.0.4 4.0.19 4.0.19 4.0.19 fs-err 2.8.1 2.9.0 2.9.0 2.9.0 futures 0.3.24 0.3.25 0.3.25 0.3.25 hmac-sha256 1.1.4 1.1.5 1.1.5 1.1.5 html-escape 0.2.11 0.2.12 0.2.12 0.2.12 hyper 0.14.20 0.14.22 0.14.22 0.14.22 ipnet 2.5.0 2.5.1 2.5.1 2.5.1 moka 0.9.4 0.9.6 0.9.6 0.9.6 regex 1.6.0 1.7.0 1.7.0 1.7.0 sentry 0.27.0 0.27.0 0.28.0 0.27.0 incompatible serde 1.0.145 1.0.147 1.0.147 1.0.147 serde_json 1.0.85 1.0.87 1.0.87 1.0.87 serde_yaml 0.9.13 0.9.14 0.9.14 0.9.14 time 0.3.14 0.3.17 0.3.17 0.3.17 tokio-stream 0.1.10 0.1.11 0.1.11 0.1.11 tracing 0.1.36 0.1.37 0.1.37 0.1.37 tracing-subscriber 0.3.15 0.3.16 0.3.16 0.3.16 Checking base62's dependencies Checking futile's dependencies Checking futile-asset-rewriter's dependencies Checking futile-backtrace-printer's dependencies Checking futile-config's dependencies Checking futile-content's dependencies Checking futile-db's dependencies Checking futile-extract-text's dependencies Checking futile-friendcodes's dependencies Checking futile-frontmatter's dependencies Checking futile-highlight's dependencies Checking futile-parent-path's dependencies Checking futile-patreon's dependencies Checking futile-query's dependencies Checking futile-reddit's dependencies Checking futile-templating's dependencies note: Re-run with `--incompatible` to upgrade incompatible version requirements note: Re-run with `--verbose` to show all dependencies local: futile-query unchanged: arc-swap, async-stream, base-x, base62, broth, bytes, camino, chrono, color-backtrace, color-eyre, cookie, derive-try-from-primitive, eyre, feed-rs, flume, futile-asset-rewriter, futile-backtrace-printer, futile-config, futile-content, futile-db, futile-extract-text, futile-friendcodes, futile-frontmatter, futile-highlight, futile-parent-path, futile-patreon, futile-reddit, futile-templating, grass, http, ignore, jsonapi, katex, kstring, lazy_static, liquid, liquid-core, lol_html, memmap, notify, opentelemetry, opentelemetry-otlp, opentelemetry-semantic-conventions, parking_lot, pin-project-lite, pithy, pretty_assertions, pulldown-cmark, reqwest, rmp-serde, seahash, serde_urlencoded, slug, sqlx, tempfile, termcolor, test-log, thiserror, tokio, tokio-io-timeout, tokio-util, toml, tower, tower-request-id, tracing-opentelemetry, tree-sitter-collection, trust-dns-resolver, ulid, url, warp warning: aborting upgrade due to dry run
That one seems to work with workspace dependencies, and lets us know that all
upgrades except for aws-*
and sentry
are "compatible" updates, which means,
if they follow semver, we should be able to upgrade all of these without breaking
anything.
This is the default behavior of cargo upgrade
, so let's do that:
$ cargo upgrade Updating 'https://github.com/rust-lang/crates.io-index' index Checking virtual workspace's dependencies Updating 'https://github.com/bearcove/crates.git' index (cut, same output as `--dry-run`) Checking base62's dependencies Checking futile's dependencies Checking futile-asset-rewriter's dependencies (cut) Checking futile-templating's dependencies Upgrading recursive dependencies Blocking waiting for file lock on package cache Updating async-io v1.8.0 -> v1.10.0 Updating async-lock v2.5.0 -> v2.6.0 Updating bumpalo v3.8.0 -> v3.11.0 Updating cc v1.0.73 -> v1.0.74 Updating getrandom v0.2.7 -> v0.2.8 Updating h2 v0.3.14 -> v0.3.15 Updating itoa v1.0.3 -> v1.0.4 Updating js-sys v0.3.59 -> v0.3.60 Updating libc v0.2.134 -> v0.2.137 Updating mio v0.8.4 -> v0.8.5 Updating num_cpus v1.13.1 -> v1.14.0 Updating once_cell v1.15.0 -> v1.16.0 Updating os_str_bytes v6.3.0 -> v6.3.1 Updating parking_lot_core v0.9.3 -> v0.9.4 Updating polling v2.3.0 -> v2.4.0 Updating proc-macro2 v1.0.46 -> v1.0.47 Updating raw-cpuid v10.5.0 -> v10.6.0 Updating regex-syntax v0.6.27 -> v0.6.28 Updating smallvec v1.9.0 -> v1.10.0 Updating syn v1.0.101 -> v1.0.103 Updating triomphe v0.1.5 -> v0.1.8 Updating ucd-util v0.1.8 -> v0.1.9 Updating unicode-ident v1.0.4 -> v1.0.5 Updating utf8-ranges v1.0.4 -> v1.0.5 Updating utf8-width v0.1.5 -> v0.1.6 Updating uuid v1.1.2 -> v1.2.1 Updating wasm-bindgen v0.2.82 -> v0.2.83 Updating wasm-bindgen-backend v0.2.82 -> v0.2.83 Updating wasm-bindgen-macro v0.2.82 -> v0.2.83 Updating wasm-bindgen-macro-support v0.2.82 -> v0.2.83 Updating wasm-bindgen-shared v0.2.82 -> v0.2.83 Updating web-sys v0.3.55 -> v0.3.60 Adding windows-sys v0.42.0 Adding windows_aarch64_gnullvm v0.42.0 Adding windows_aarch64_msvc v0.42.0 Adding windows_i686_gnu v0.42.0 Adding windows_i686_msvc v0.42.0 Adding windows_x86_64_gnu v0.42.0 Adding windows_x86_64_gnullvm v0.42.0 Adding windows_x86_64_msvc v0.42.0 note: Re-run with `--incompatible` to upgrade incompatible version requirements note: Re-run with `--verbose` to show all dependencies local: futile-query unchanged: arc-swap, async-stream, base-x, base62, broth, bytes, camino, chrono, color-backtrace, color-eyre, cookie, derive-try-from-primitive, eyre, feed-rs, flume, futile-asset-rewriter, futile-backtrace-printer, futile-config, futile-content, futile-db, futile-extract-text, futile-friendcodes, futile-frontmatter, futile-highlight, futile-parent-path, futile-patreon, futile-reddit, futile-templating, grass, http, ignore, jsonapi, katex, kstring, lazy_static, liquid, liquid-core, lol_html, memmap, notify, opentelemetry, opentelemetry-otlp, opentelemetry-semantic-conventions, parking_lot, pin-project-lite, pithy, pretty_assertions, pulldown-cmark, reqwest, rmp-serde, seahash, serde_urlencoded, slug, sqlx, tempfile, termcolor, test-log, thiserror, tokio, tokio-io-timeout, tokio-util, toml, tower, tower-request-id, tracing-opentelemetry, tree-sitter-collection, trust-dns-resolver, ulid, url, warp
Everything still builds, my tests still pass, a quick QA round indicates that the site still works as intended.
As for the semver-breaking changes, we can target sentry
:
$ cargo upgrade -p sentry (cut)
A quick look at the changelog indicates that there's nothing to worry about. They bumped the MSRV (Minimum Supported Rust Version) to 1.60.0, whereas the nightly I'm using is the equivalent of 1.67.0 — we're good.
Then, we can upgrade the aws-*
family of crates, the last few "incompatible"
upgrades available:
$ cargo upgrade --incompatible (cut) Updating 'https://github.com/rust-lang/crates.io-index' index Checking virtual workspace's dependencies Updating 'https://github.com/bearcove/crates.git' index name old req compatible latest new req ==== ======= ========== ====== ======= aws-config 0.49.0 0.49.0 0.51.0 0.51.0 aws-sdk-s3 0.19.0 0.19.0 0.21.0 0.21.0 aws-types 0.49.0 0.49.0 0.51.0 0.51.0 sentry 0.27.0 0.27.0 0.28.0 0.28.0 (cut)
Neither of these were actually "incompatible" in the sense that nothing broke, but we weren't covered by semver guarantees (regardless of how much you believe in them).
Thanks to my sponsors:
If you liked what you saw, please support my work!
Here's another article just for you:
It happened when I least expected it.
Someone, somewhere (above me, presumably) made a decision. "From now on", they declared, "all our new stuff must be written in Rust".
I'm not sure where they got that idea from. Maybe they've been reading propaganda. Maybe they fell prey to some confident asshole, and convinced themselves that Rust was answer to their problems.