1 hour 38 min
complete
Don't shell out!
👋 This page was last updated ~3 years ago. Just so you know.
In this series, I change a critical component of this website's asset pipeline from "just calling a bunch of external tools" to statically linking with everything I need to process assets. It involves autoconf, CMake, Meson, CI, pkg-config, and some code crimes.
Series overview
1. Truly headless draw.io exports
16 min
I love diagrams. I love them so much!
In fact, I have fairly poor visualization skills, so making a diagram is extremely helpful to me: I'll have some vague idea of how different things are connected, and then I'll make a diagram, and suddenly there's a tangible thing I can look at and talk about.
Of course the diagram only represents a fraction of what I had in mind in the first place, but that's okay: the point is to be able to talk about of a concept, and so I have to make choices about what to include in the diagram. And maybe make several diagrams.
2. From Inkscape to poppler
10 min
What's next? Well... poppler is the library Inkscape uses to import PDFs.
Cool bear's hot tip
Yes, the name comes from Futurama.
Turns out, poppler comes with a bunch of CLI tools, including pdftocairo!
Halfway through this article, I realized the "regular weight" on my system was in fact Iosevka SS01 (Andale Mono Style) (see ), but the "bold weight" was the default Iosevka.
3. A static poppler build: the easy way
16 min
So! Now our asset processing pipeline is almost complete. But we've just traded dependencies against CLI tools, for dependencies against dynamic libraries:
$ ldd ./target/debug/pdftocairo
linux-vdso.so.1 (0x00007ffd615be000)
libpoppler-glib.so.8 => /lib64/libpoppler-glib.so.8 (0x00007f2ba1bb4000)
libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007f2ba1b59000)
libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f2ba1a1e000)
libcairo.so.2 => /lib64/libcairo.so.2 (0x00007f2ba1902000)
libcairo-gobject.so.2 => /lib64/libcairo-gobject.so.2 (0x00007f2ba18f6000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f2ba18dc000)
libm.so.6 => /lib64/libm.so.6 (0x00007f2ba17fe000)
libc.so.6 => /lib64/libc.so.6 (0x00007f2ba15f4000)
/lib64/ld-linux-x86-64.so.2 (0x00007f2ba216c000)
libpoppler.so.112 => /lib64/libpoppler.so.112 (0x00007f2ba1288000)
libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007f2ba11bd000)
libgio-2.0.so.0 => /lib64/libgio-2.0.so.0 (0x00007f2ba0fe4000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f2ba0dc5000)
libffi.so.6 => /lib64/libffi.so.6 (0x00007f2ba0db8000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f2ba0d40000)
libpixman-1.so.0 => /lib64/libpixman-1.so.0 (0x00007f2ba0c94000)
libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x00007f2ba0c45000)
libpng16.so.16 => /lib64/libpng16.so.16 (0x00007f2ba0c0c000)
libxcb-shm.so.0 => /lib64/libxcb-shm.so.0 (0x00007f2ba0c07000)
libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f2ba0bda000)
libxcb-render.so.0 => /lib64/libxcb-render.so.0 (0x00007f2ba0bca000)
libXrender.so.1 => /lib64/libXrender.so.1 (0x00007f2ba0bbd000)
libX11.so.6 => /lib64/libX11.so.6 (0x00007f2ba0a75000)
libXext.so.6 => /lib64/libXext.so.6 (0x00007f2ba0a60000)
libz.so.1 => /lib64/libz.so.1 (0x00007f2ba0a46000)
libjpeg.so.62 => /lib64/libjpeg.so.62 (0x00007f2ba09c2000)
libopenjp2.so.7 => /lib64/libopenjp2.so.7 (0x00007f2ba0968000)
liblcms2.so.2 => /lib64/liblcms2.so.2 (0x00007f2ba0903000)
libtiff.so.5 => /lib64/libtiff.so.5 (0x00007f2ba087c000)
libsmime3.so => /lib64/libsmime3.so (0x00007f2ba0850000)
libnss3.so => /lib64/libnss3.so (0x00007f2ba0712000)
libplc4.so => /lib64/libplc4.so (0x00007f2ba0709000)
libnspr4.so => /lib64/libnspr4.so (0x00007f2ba06c6000)
libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f2ba06b3000)
libharfbuzz.so.0 => /lib64/libharfbuzz.so.0 (0x00007f2ba05dd000)
libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f2ba05cf000)
libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007f2ba05c8000)
libmount.so.1 => /lib64/libmount.so.1 (0x00007f2ba0581000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f2ba0556000)
libxml2.so.2 => /lib64/libxml2.so.2 (0x00007f2ba03cd000)
libXau.so.6 => /lib64/libXau.so.6 (0x00007f2ba03c7000)
libwebp.so.7 => /lib64/libwebp.so.7 (0x00007f2ba0358000)
libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f2ba0260000)
libjbig.so.2.1 => /lib64/libjbig.so.2.1 (0x00007f2ba0252000)
libnssutil3.so => /lib64/libnssutil3.so (0x00007f2ba021f000)
libplds4.so => /lib64/libplds4.so (0x00007f2ba021a000)
libgraphite2.so.3 => /lib64/libgraphite2.so.3 (0x00007f2ba01f9000)
libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007f2ba01d4000)
libblkid.so.1 => /lib64/libblkid.so.1 (0x00007f2ba019c000)
libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f2ba0105000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f2ba00d9000)
4. Building poppler for Windows
14 min
I know what you're thinking: haven't we strayed from the whole "content pipeline" theme in this series?
Well... fair. But compiling and distributing software is part of software engineering, and unless you're in specific circles, I see that taught a lot less than the "just write code and stuff happens" part.
5. Porting poppler to meson
10 min
It took a hot minute.
Try several weeks.
Well, yeah. I got to contribute to a bunch of open-source projects in the meantime though, so I'm fairly pleased with it!
- libffi (for static linking)
- cairo (more static linking!)
- proxy-libintl (more static linking!)
- expat (static linking strikes again)
- poppler (for file descriptor stuff not properly gated on Windows, closed in favor of a similar MR)
6. Productionizing our poppler build
19 min
I was a bit anxious about running our poppler meson build in CI, because it's the real test, you know? "Works on my machine" only goes so far, things have a tendency to break once you try to make them reproducible.
And I was right to worry... but not for the reasons I thought. As I tried to get everything to build in CI, there was a Pypi maintenance that prevented me from installing meson, and then .
7. The rest of the fucking owl
7 min
NO! No no no.
What?
WE WERE DONE!
Well... yes! But also no. We still shell out to a bunch of tools:
$ rg 'Command::new'
src/commands/mod.rs
126: let variant = if let Ok(output) = run_command(Command::new("wslpath").arg("-m").arg("/")) {
src/commands/cavif.rs
29: Command::new("cavif")
src/commands/imagemagick.rs
25: Command::new(&self.bin)
src/commands/cwebp.rs
25: Command::new("cwebp")
src/commands/svgo.rs
25: Command::new("svgo")
8. One funny way to bundle assets
6 min
There's one thing that bothers me. In part 1, why are we using
hyper-staticfile? Couldn't we just use file:/// URLs?
Well, first off: showing off how easy it is to serve some static files, even in a "scary" language like Rust, is just not something I could pass up.
But also: think about distributing as a tool. Will we want to distribute all those HTML/CSS/JS/font files alongside it?
This series is complete.