fasterthanlime

Dec 31, 2021 14 min #build-systems · #pkg-config · #cmake · #ftl-infra

Building poppler for Windows

Thanks to my sponsors: Yuriy Taraday, medzernik, xales, Andy F, Marcin Kołodziej, Ben Mitchell, Alejandro Angulo, Urs Metz, Paige Ruten, Scott Sanderson, Colin VanDervoort, Chris Walker, Joshua Roesslein, C J Silverio, Mattia Valzelli, Dylan Anthony, G, Damir Vandic, Max von Forell, Christopher Valerio and 267 more Yuriy Taraday, medzernik, xales, Andy F, Marcin Kołodziej, Ben Mitchell, Alejandro Angulo, Urs Metz, Paige Ruten, Scott Sanderson, Colin VanDervoort, Chris Walker, Joshua Roesslein, C J Silverio, Mattia Valzelli, Dylan Anthony, G, Damir Vandic, Max von Forell, Christopher Valerio, Makoto Nakashima, Elendol, Romain Kelifa, Ives van Hoorne, Elnath, anichno, Mike English, Matthew T, Borys Minaiev, Walther, Andronik, Vincent, Yves, Brandon Piña, Herman J. Radtke III, Daniel Silverstone, Justin Ossevoort, Max Heaton, James Rhodes, Ryan, Chris, Aleksandre Khokhiashvili, villem, jer, Geoffroy Couprie, notryanb, Nicolas Riebesel, Malik Bougacha, Victor Song, Lena Schönburg, Jon Gjengset, jatescher, David Souther, Guillaume Demonet, Aljaz Erzen, Timothée Gerber, Vladimir, Enrico Zschemisch, David Barsky, Christian Bourjau, John Horowitz, Alex Krantz, Lyssieth, Jonathan Adams, Antoine Rouaze, Marky Mark, Simon Menke, Jim, Manuel Hutter, Sean Bryant, David White, old.woman.josiah, prairiewolf, compwhizii, Mateusz Wykurz, Matt Jadczak, Marcus Griep, Luiz Ferraz, Steven Pham, Thor Kamphefner, Zachary Thomas, Mario Fleischhacker, kuerbsikakteen, Tobias Bahls, Beat Scherrer, Corey Alexander, Mathias Brossard, Isak Sunde Singh, Jelle Besseling, qrpth, Cole Kurkowski, Marie Janssen, Tanner Muro, Santiago Lema, (18D)eezNuts, Hadrien G., Paul Schuberth, Duane Sibilly, Ahmad Alhashemi, Sam Leonard, Tabitha, Astrid, Dimitri Merejkowsky, Eugene Bulkin, René Ribaud, Aalekh Patel, Richard Pringle, Yufan Lou, Benjamin Röjder Delnavaz, Rufus Cable, Braidon Whatley, Toon Willems, Michał Bartoszkiewicz, avborhanian, Richard Stephens, Morgan Rosenkranz, Torben Clasen, Kai Kaufman, Dennis Henderson, Nicholas, Sawyer Knoblich, Tiziano Santoro, ZacJW, Sylvie Nightshade, Raine Godmaire, pinkhatbeard, Geoffrey Thomas, Evan Relf, Guillaume E, Dirkjan Ochtman, Aiden Scandella, Michal Hošna, Luke Konopka, Philipp Gniewosz, Xavier Groleau, hardfist, Gorazd Brumen, Adam Gutglick, Luke Yue, budrick, Beth Rennie, ShikChen, bbutkovic, AdrianEddy, hgranthorner, Blake Johnson, Berkus Decker, Johnathan Pagnutti, Jean-David Gadina, Paul Horn, Jan De Landtsheer, Matt Heise, Josiah Bull, Michael, Jack Duvall, Michael Mrozek, traxys, Tyler Schmidtke, Zeeger Lubsen, Lennart Oldenburg, Jake Demarest-Mays, Antoine Boegli, Integer 32, LLC, Guy Waldman, teor, Valentin Mariette, Jesse Luehrs, Alex Rudy, Ula, Chris Emery, Shane Lillie, Lev Khoroshansky, Joseph Montanaro, Mike Cripps, Daniel Wagner-Hall, Marcus Griep, Olly Swanson, Marco Carmosino, Jörn Huxhorn, Marty Penner, Nyefan, Noel, Raphaël Thériault, Das Gürteltier, Olivia Crain, Daniel Papp, Alan O'Donnell, Sonny Scroggin, Chris Sims, Pete Bevin, Romain Ruetschi, Radu Matei, Mason Ginter, Ronen Ulanovsky, Philipp Hatt, Ivo Murrell, Chris Thackrey, Laine Taffin Altman, Mark Tomlin, Stephan Buys, Paul Marques Mota, Niels Abildgaard, Sarah Berrettini, Daniel Strittmatter, James Leitch, Boris Dolgov, Jack Maguire, Matt Jackson, Bob Ippolito, clement, Samit Basu, Zoran Zaric, Chirag Jain, Scott Steele, ofrighil, Guilherme Neubaner, Julius Riegel, Horváth-Lázár Péter, ACRL, Sindre Johansen, Christoph Grabo, Elijah Voigt, Ronen Cohen, SeniorMars, John VanEnk, Zaki, Egor Ternovoi, Neil Blakey-Milner, Johan Saf, Ross Williams, Andy Gocke, Gioele Pannetto, Yann Schwartz, Josh Triplett, L0r3m1p5um, Ian McLinden, David E Disch, Leigh Oliver, Diego Roig, Cass, Ben Wishovich, Wojciech Smołka, genny, Mark Old, Michael Alyn Miller, Adam Lassek, Antoine PESTEL-ROPARS, Senyo Simpson, Matthew Planchard, Jacob Cheriathundam, James Brown, Romet Tagobert, Max Bruckner, Jan-Stefan Janetzky, Geoff Cant, playest, you got maiL, Matt Campbell, belzael, Marcus Brito, Marc-Andre Giroux, Peter Shih, Wyatt Herkamp, Chris Biscardi, Julian Schmid, Kyle Lacy, Kristoffer Winther Balling, David Cornu, Tyler Bloom, Thehbadger, Garret Kelly, Lucille Blumire, Cole Tobin, Xirvik Servers, std__mpa, Seth, WeblWabl, Mikkel Rasmussen, Kamran Khan, Matěj Volf, Andrew Neth, Helge Eichhorn, callym, Pete LeVasseur, Henrik Tudborg, Dom, Mark

I know what you’re thinking: haven’t we strayed from the whole “content pipeline” theme in this series?

Well… fair. But compiling and distributing software is part of software engineering, and unless you’re in specific circles, I see that taught a lot less than the “just write code and stuff happens” part.

Technically it’s release engineering, but who’s keeping track.

So we’re going to think about how we could build all this for Windows. Because every single of my internal tools for my website builds for Windows just fine, in case I, you know, need to write a series on Windows, from Windows.

So my little fork of salvage that uses headless_chrome to render a .drawio file to /tmp/export.pdf? That compiles and runs on Windows!

Well huh… ok maybe we can’t write to /tmp/export.pdf. Let’s quickly turn this:

Into this:

And then… it runs!

Isn’t it amazing that this was the only change we needed to make?

Heck yeah!

This goes to show how great of a job crate maintainers have been doing!

Our diagram looks just as fine on Windows:

Our pdftocairo executable however… it’s a different story. Well, to be fair, I have no idea what to expect, so let’s find out:

Well, system-deps wasn’t lying, it really does only support pkg-config!

pkg-config can be built for windows, and in fact, we can install it with scoop:

But of course, it can’t find anything:

Where does it even look?

Well bear, you know what RTFM means?

Read the.. fine manual?

Yes! But after a couple searches, I cannot for the life of me figure out where pkg-config looks for .pc files on Windows by default.

So what do we do then?

Well, even if I had found something, the docs always lie. So what we do is spy on the process instead.

But this is Windows, we don’t have strace. What we do have, is Process Monitor.

And now we have our answer! It looks in C:\Users\faste\scoop\apps\pkg-config\current\share\pkgconfig, which is a symbolic link to the currently installed version:

So, yay, there’s a global path! But nothing stable like /usr/lib64/pkgconfig on Fedora, or /usr/lib/x86_64-linux-gnu/pkgconfig on Ubuntu.

I’m sure if we install our .pc files somewhere though, pkg-config will be able to find them, right?

Well, our build scripts right now are bash, so… unless we install bash, we can’t run them as-is, but we can try to run similar instructions from a more Windows-y shell, like PowerShell?

Let’s try with pcre:

So far so good! Apparently Windows 11 ships the real curl, not just an alias?

Next we need to extract the archive. Apparently tar also ships with Windows 11!

Now, what build system is PCRE using again? Autotools? Right. Well…

…that’s not gonna work.

Luckily, PCRE also comes with a CMakeLists.txt, so we can “just” use cmake!

Visual Studio 2022 Build Tools ships with cmake, it’s just a bit hidden, but if we start the “x64 Native Tools Command Prompt for VS 2022”, it’s right there in the PATH:

So, let’s make a build folder and see what happens!

Ah, forgot to specify a prefix, let’s do that now:

And.. let’s build?

And install?

Oh, huh, it’s trying to install a release build but we made a debug build.

Fine, let’s fix our mistakes…

Oh look, it installed .pc files!

Now I guess we can get pkg-config to find those!

We can!

Mhh but those look like command-line flags for GCC/Clang. You just said the flags for MSVC looked different! (And then gave no example).

Well, time for an example then, because on Windows, pkg-config definitely has a setting for that:

Look, it’s different! Using /flag instead of -flag / --flag. It doesn’t seem to change anything for --cflags though, so uhhh I’m curious how everything will work out.

Let’s maybe try to make a quick project using libpcre from Rust with system-deps, to see how it’ll handle that.

If we try cargo run, it fails!

…as expected!

Let’s just set PKG_CONFIG_PATH and…

It works perfectly! I’m really happy about that. There was hardly a mention of windows in the system-deps docs, it was honestly a 50/50 between “just works” and “woops sorry we assumed Linux all along”.

For learning purposes, I did scoop uninstall pkg-config, and cargo clean && cargo run failed - it does need the real thing in path, they haven’t reimplemented pkg-config as a Rust library or anything.

Should they?

I don’t know, can you think of an argument for doing that?

Well, you wouldn’t need to install pkg-config on Windows. It’s a pretty nonstandard tool to have on there…

And can you think of an argument against doing that?

I suppose the Rust implementation would have to have the same hardcoded paths as the “real” pkg-config on Linux (or shell out to it to discover them?), and there’s always the risk the implementations could drift apart…

Exactly! So, should they?

Ehhhhhhhhhhhhhhhhh.

Exactly.

So, at this point, in this article about checks notes vector graphics, all we need to do is build all those libraries and we should be good, right?

But to be completely honest with y’all, I don’t feel like porting my build scripts to Windows, and maintaining them so that they work on both Windows and Linux.

This feels like a lot of work, /especially/ since I discovered that Meson can download and compile dependencies.

Alright, let’s start simple! We’ll make a new folder, poppler-meson, and add this file:

This should only pull in glib and its dependencies. Right now, our project fails to configure:

And that’s because we added a dependency on glib-2.0, but didn’t give meson any information on where to find it. So, it tried to find it with pkg-config and cmake, but it didn’t. On Fedora, with glib2-devel installed, this would probably work.

But we’re on Windows now! So meson needs additional info. Luckily, meson now comes with “wraptool”, which lets us pick from a list of libraries that have been “wrapped” so they can be built with meson instead of whatever their maintainers thought they should be built with (autotools, cmake, a home-grown organic set of Makefiles, etc.)

Hey there it is! Let’s add it:

Just like that, wraptool added the subprojects/glib.wrap file:

(With hashes and everything!)

Running meson setup again goes further! It actually creates subprojects/libpcre.wrap, since it’s required by glib:

But unfortunately, it eventually fails, because the default mirror for pcre sources hates me, apparently:

Luckily, there’s other mirrors we can pick from! By replacing the contents of subprojects/libpcre.wrap with a slightly tuned version of subprojects/glib-2.70.1/subprojects/libpcre.wrap…

…we can keep going.

Look at that, it pulled in zlib, libffi and proxy-libintl as well!

It did! How convenient.

A short meson compile -C build and many MSVC warnings later… it fails to build. Well, it turns out glib static builds on Windows are kind of a work-in-progress, so I had to learn about a few more meson features…

Some C defines were missing, but luckily, we can override them at the project level. We can even have the project default to a static build.

As for the glib dependency, we can also set things there, like ask for it to be statically linked with static: true, and set some default options, like: “don’t bother with tests”.

And because glib really insists on including DllMain in its multiple libraries (gobject, gio, etc.), even when they’re built to a static .a library on Windows, I had to bring in a little patch.

Which is a thing meson lets you do! Usually to “add meson build files”, but what’s a little crime among friends?

The patch is actually not a patch, more like an overlay, so I had to create subprojects/packagefiles/glib-2.70.1/glib/glib-init.c with my already-patched version of this source file. Handy!

And finally, it builds:

And installs:

Cool Bear's hot tip

Oddly enough, meson install wouldn’t find ninja the way meson compile did, and a scoop install ninja fixed it there.

Another option would’ve been meson devenv -C build, which enters a “dev environment”, with C/C++ compilers/linkers (like cl.exe, link.exe) and build tools (cmake.exe, ninja.exe) in PATH.

So let’s look at what meson/ninja has actually installed in our prefix.

That is great, we definitely needed a bunch of these.

Our static libraries are all installed in prefix\lib\pkgconfig as well:

I didn’t immediately notice something was wrong, because it had been a while since I had to think about MSVC idiosyncracies, but…

…that’s not how libraries should be named for MSVC. They shouldn’t be named libfoobar.a, they should be named foobar.lib.

Luckily, a single PowerShell command lets us solve that (quickly adds powershell to the tags of this article):

Tada! Wish zsh would let me do that this easily. Maybe nushell can?

Anyway, now it works:

Nope, nope, go back, meson defaults to a debug build, we want a release build for sure.

Ok surely now it’ll work:

Err almost. Just missing a couple libraries…

YES! Yes! Finally!

The source for this sample program, by the way, is:

And the glib-2.0 dependency is specified like so:

And I include it so you can find out that the system dep’s key cannot include dots, but that you can save it by specifying “name”.

Okay! So that’s glib. I mean, that’s glib.

What else do we need… well, cairo’s a big one. And good news, cairo, alongside autotools, actually ships its own meson.build file!

But that still leaves poppler itself. And it doesn’t ship with its own meson build files. It ships with CMake, and in theory, I could try to port everything to CMake, and if I did, I would use Izzy’s wonderful CMake articles, like Everything you never wanted to know about CMake, and How to Find Packages With CMake.

This time, I chose to go the opposite route: everything except poppler already had meson build files, so it seemed like a good opportunity for me to simply write meson.build files for poppler! Which is exactly what I did.

Comment on /r/fasterthanlime

Here's another article just for you:

Peeking inside a Rust enum

Aug 28, 2020

53 min #rust · #c · #memory

During a recent Rust Q&A Session on my twitch channel, someone asked a question that seemed simple: why are small string types, like SmartString or SmolStr, the same size as String, but small vec types, like SmallVec, are larger than Vec?

Now I know I just used the adjective simple, but the truth of the matter is: to understand the question, we’re going to need a little bit of background.

Read more