Building poppler for Windows

This article is part of the Don't shell out! series.

I know what you're thinking: haven't we strayed from the whole "content pipeline" theme in this series?

Well... fair. But compiling and distributing software is part of software engineering, and unless you're in specific circles, I see that taught a lot less than the "just write code and stuff happens" part.

Technically it's release engineering, but who's keeping track.

So we're going to think about how we could build all this for Windows. Because every single of my internal tools for my website builds for Windows just fine, in case I, you know, need to write a series on Windows, from Windows.

So my little fork of salvage that uses headless_chrome to render a .drawio file to /tmp/export.pdf? That compiles and runs on Windows!

PowerShell session
$ .\target\debug\salvage.exe
2021-11-26T13:56:29.450869Z  INFO headless_chrome::browser::fetcher: Getting project dir
2021-11-26T13:56:29.454840Z  INFO headless_chrome::browser::process: Launching Chrome binary at "C:\\Users\\faste\\AppData\\Roaming\\headless-chrome\\data\\win-634997\\chrome-win\\chrome.exe" 
2021-11-26T13:56:29.533551Z  INFO headless_chrome::browser::process: Started Chrome. PID: 18936
2021-11-26T13:56:29.712694Z  INFO salvage::chrome_stuff: Navigating...
2021-11-26T13:56:30.044934Z  INFO headless_chrome::browser::tab: Navigating a tab to http://localhost:5000/index.html
2021-11-26T13:56:31.049236Z  INFO salvage::chrome_stuff: Navigating... done!
2021-11-26T13:56:31.061422Z  INFO salvage::chrome_stuff: Got dimensions width=994 height=643
2021-11-26T13:56:31.192107Z  INFO salvage::chrome_stuff: Writing pdf... pdf_path=/tmp/export.pdf
2021-11-26T13:56:31.192335Z  INFO headless_chrome::browser: Dropping browser
2021-11-26T13:56:31.192442Z  INFO headless_chrome::browser::process: Killing Chrome. PID: 18936
2021-11-26T13:56:31.192552Z  INFO headless_chrome::browser::transport::web_socket_connection: Sending shutdown message to message handling loop
2021-11-26T13:56:31.192694Z  INFO headless_chrome::browser::transport: Received shutdown message
2021-11-26T13:56:31.192785Z  INFO headless_chrome::browser::transport: Shutting down message handling loop
2021-11-26T13:56:31.192886Z  INFO headless_chrome::browser::transport: cleared listeners, I think
2021-11-26T13:56:31.192904Z  INFO headless_chrome::browser::tab: finished tab's event handling loop
2021-11-26T13:56:31.192900Z  INFO headless_chrome::browser: Finished browser's event handling loop
2021-11-26T13:56:31.198980Z  INFO headless_chrome::browser::transport: dropping transport
2021-11-26T13:56:31.199044Z  INFO headless_chrome::browser::transport::web_socket_connection: dropping websocket connection
The application panicked (crashed).
Message:  called `Result::unwrap()` on an `Err` value: Os { code: 3, kind: NotFound, message: "The system cannot find the path specified." }
Location: src\chrome_stuff.rs:17

Backtrace omitted.
Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
Error:
   0: chrome error: JoinError::Panic(...)

Location:
   src\main.rs:47

Backtrace omitted.
Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

Well huh... ok maybe we can't write to /tmp/export.pdf. Let's quickly turn this:

Rust code
    let pdf_path = Utf8PathBuf::from("/tmp/export.pdf");

Into this:

Rust code
    let pdf_path = Utf8PathBuf::try_from(std::env::temp_dir())?.join("export.pdf");

And then... it runs!

PowerShell session
$ cargo run
   Compiling salvage v1.2.0 (C:\Users\faste\bearcove\salvage)
    Finished dev [unoptimized + debuginfo] target(s) in 2.75s
     Running `target\debug\salvage.exe`
2021-11-26T14:03:34.463367Z  INFO headless_chrome::browser::fetcher: Getting project dir
2021-11-26T14:03:34.468192Z  INFO headless_chrome::browser::process: Launching Chrome binary at "C:\\Users\\faste\\AppData\\Roaming\\headless-chrome\\data\\win-634997\\chrome-win\\chrome.exe" 
2021-11-26T14:03:34.550763Z  INFO headless_chrome::browser::process: Started Chrome. PID: 14984
2021-11-26T14:03:34.853213Z  INFO salvage::chrome_stuff: Navigating...
2021-11-26T14:03:35.188660Z  INFO headless_chrome::browser::tab: Navigating a tab to http://localhost:5000/index.html
2021-11-26T14:03:37.092808Z  INFO salvage::chrome_stuff: Navigating... done!
2021-11-26T14:03:37.104829Z  INFO salvage::chrome_stuff: Got dimensions width=994 height=643
2021-11-26T14:03:37.241291Z  INFO salvage::chrome_stuff: Writing pdf... pdf_path=C:\Users\faste\AppData\Local\Temp\export.pdf
2021-11-26T14:03:37.241867Z  INFO salvage::chrome_stuff: Writing pdf... done! pdf_path=C:\Users\faste\AppData\Local\Temp\export.pdf
2021-11-26T14:03:37.241957Z  INFO headless_chrome::browser: Dropping browser
2021-11-26T14:03:37.242089Z  INFO headless_chrome::browser::process: Killing Chrome. PID: 14984
2021-11-26T14:03:37.242193Z  INFO headless_chrome::browser::transport::web_socket_connection: Sending shutdown message to message handling loop
2021-11-26T14:03:37.242320Z  INFO headless_chrome::browser::transport: Received shutdown message
2021-11-26T14:03:37.242398Z  INFO headless_chrome::browser::transport: Shutting down message handling loop
2021-11-26T14:03:37.242484Z  INFO headless_chrome::browser::transport: cleared listeners, I think
2021-11-26T14:03:37.242521Z  INFO headless_chrome::browser::tab: finished tab's event handling loop
2021-11-26T14:03:37.242516Z  INFO headless_chrome::browser: Finished browser's event handling loop

Isn't it amazing that this was the only change we needed to make?

Heck yeah!

This goes to show how great of a job crate maintainers have been doing!

Our diagram looks just as fine on Windows:

Shell session
$ start $env:TMP\export.pdf

Our pdftocairo executable however... it's a different story. Well, to be fair, I have no idea what to expect, so let's find out:

PowerShell session
$ cargo build
    Updating crates.io index
  Downloaded anyhow v1.0.47
  Downloaded cairo-rs v0.14.9
(cut)
  Downloaded glib-sys v0.14.0
  Downloaded proc-macro-error v1.0.4
  Downloaded 19 crates (615.9 KB) in 1.39s
   Compiling proc-macro2 v1.0.32
   Compiling unicode-xid v0.2.2
(cut)
   Compiling poppler-sys-rs v0.18.0
The following warnings were emitted during compilation:

warning: Could not run `"pkg-config" "--libs" "--cflags" "glib-2.0" "glib-2.0 >= 2.48"`

error: failed to run custom build command for `glib-sys v0.14.0`

Caused by:
  process didn't exit successfully: `C:\Users\faste\bearcove\pdftocairo\target\debug\build\glib-sys-0231afba2b42ecdf\build-script-build` (exit code: 1)
  --- stdout
  cargo:rerun-if-env-changed=GLIB_2.0_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-pc-windows-msvc
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_pc_windows_msvc
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-pc-windows-msvc
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_pc_windows_msvc
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-pc-windows-msvc
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_pc_windows_msvc
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-pc-windows-msvc
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_pc_windows_msvc
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  cargo:warning=Could not run `"pkg-config" "--libs" "--cflags" "glib-2.0" "glib-2.0 >= 2.48"`
  The pkg-config command could not be found.

  Most likely, you need to install a pkg-config package for your OS.

  If you've already installed it, ensure the pkg-config command is one of the
  directories in the PATH environment variable.

  If you did not expect this build to link to a pre-installed system library,
  then check documentation of the glib-sys crate for an option to
  build the library from source, or disable features or dependencies
  that require pkg-config.
warning: build failed, waiting for other jobs to finish...
error: build failed

Well, system-deps wasn't lying, it really does only support pkg-config!

pkg-config can be built for windows, and in fact, we can install it with scoop:

PowerShell session
$ scoop install pkg-config
Installing 'pkg-config' (0.26-1) [64bit]
Loading pkg-config_0.26-1_win32.zip from cache
Checking hash of pkg-config_0.26-1_win32.zip ... ok.
Loading glib_2.28.8-1_win32.zip from cache
Checking hash of glib_2.28.8-1_win32.zip ... ok.
Loading gettext-runtime_0.18.1.1-2_win32.zip from cache
Checking hash of gettext-runtime_0.18.1.1-2_win32.zip ... ok.
Extracting pkg-config_0.26-1_win32.zip ... done.
Extracting glib_2.28.8-1_win32.zip ... done.
Extracting gettext-runtime_0.18.1.1-2_win32.zip ... done.
Linking ~\scoop\apps\pkg-config\current => ~\scoop\apps\pkg-config\0.26-1
Creating shim for 'pkg-config'.
'pkg-config' (0.26-1) was installed successfully!

But of course, it can't find anything:

PowerShell session
$ pkg-config --cflags --libs cairo
Package cairo was not found in the pkg-config search path.
Perhaps you should add the directory containing `cairo.pc'
to the PKG_CONFIG_PATH environment variable
No package 'cairo' found

Where does it even look?

Well bear, you know what RTFM means?

Read the.. fine manual?

Yes! But after a couple searches, I cannot for the life of me figure out where pkg-config looks for .pc files on Windows by default.

So what do we do then?

Well, even if I had found something, the docs always lie. So what we do is spy on the process instead.

But this is Windows, we don't have strace. What we do have, is Process Monitor.

And now we have our answer! It looks in C:\Users\faste\scoop\apps\pkg-config\current\share\pkgconfig, which is a symbolic link to the currently installed version:

PowerShell session
$ Get-Item C:\Users\faste\scoop\apps\pkg-config\current\

    Directory: C:\Users\faste\scoop\apps\pkg-config

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
l-r--          26/11/2021    15:11                current -> C:\Users\faste\scoop\apps\pkg-config\0.26-1

So, yay, there's a global path! But nothing stable like /usr/lib64/pkgconfig on Fedora, or /usr/lib/x86_64-linux-gnu/pkgconfig on Ubuntu.

I'm sure if we install our .pc files somewhere though, pkg-config will be able to find them, right?

Well, our build scripts right now are bash, so... unless we install bash, we can't run them as-is, but we can try to run similar instructions from a more Windows-y shell, like PowerShell?

Let's try with pcre:

PowerShell session
$ curl -f -L -o pcre.tar.bz2 https://sourceforge.net/projects/pcre/files/pcre/8.45/pcre-8.45.tar.bz2/download
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   605  100   605    0     0    605      0  0:00:01 --:--:--  0:00:01   880
100   331  100   331    0     0    331      0  0:00:01  0:00:01 --:--:--   331
100 1541k  100 1541k    0     0  1541k      0  0:00:01  0:00:01 --:--:-- 9883k

So far so good! Apparently Windows 11 ships the real curl, not just an alias?

PowerShell session
$ Get-Command curl

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     curl.exe                                           7.55.1.0   C:\Windows\system32\curl.exe

Next we need to extract the archive. Apparently tar also ships with Windows 11!

PowerShell session
$ tar xjf .\pcre.tar.bz2
$ cd .\pcre-8.45\

Now, what build system is PCRE using again? Autotools? Right. Well...

...that's not gonna work.

Luckily, PCRE also comes with a CMakeLists.txt, so we can "just" use cmake!

Visual Studio 2022 Build Tools ships with cmake, it's just a bit hidden, but if we start the "x64 Native Tools Command Prompt for VS 2022", it's right there in the PATH:

PowerShell session
$ Get-Command cmake

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     cmake.exe                                          3.21.2108… C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe

So, let's make a build folder and see what happens!

PowerShell session
$ mkdir build
$ cmake -S . -B build
-- Building for: Visual Studio 17 2022
-- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.22000.
-- The C compiler identification is MSVC 19.30.30705.0
-- The CXX compiler identification is MSVC 19.30.30705.0
(cut)
-- Could NOT find BZip2 (missing: BZIP2_LIBRARIES BZIP2_INCLUDE_DIR)
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)
-- Could not find OPTIONAL package Readline
-- Could not find OPTIONAL package Editline
-- Looking for dirent.h
-- Looking for dirent.h - not found
(cut)
--
--
-- PCRE-8.45 configuration summary:
--
--   Install prefix .................. : C:/Program Files (x86)/PCRE
--   C compiler ...................... : C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.30.30705/bin/Hostx64/x64/cl.exe
--   C++ compiler .................... : C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.30.30705/bin/Hostx64/x64/cl.exe
--   C compiler flags ................ : /DWIN32 /D_WINDOWS /W3
--   C++ compiler flags .............. : /DWIN32 /D_WINDOWS /W3 /GR /EHsc
--
--   Build 8 bit PCRE library ........ : ON
(cut)
--   Install MSVC .pdb files ..........: OFF
--
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/faste/bearcove/poppler-build/pcre-8.45/build

Ah, forgot to specify a prefix, let's do that now:

PowerShell session
$ cmake -S . -B build -DCMAKE_INSTALL_PREFIX=C:/Users/faste/bearcove/poppler-build/prefix
(output omitted)

And.. let's build?

PowerShell session
$ cmake --build build --parallel (Get-CimInstance Win32_ComputerSystem).NumberOfLogicalProcessors
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

  Checking Build System
  Building Custom Rule C:/Users/faste/bearcove/poppler-build/pcre-8.45/CMakeLists.txt
  pcre_byte_order.c
  pcre_chartables.c
  pcre_compile.c
(cut)
  pcre_stringpiece_unittest.vcxproj -> C:\Users\faste\bearcove\poppler-build\pcre-8.45\build\Debug\pcre_stringpiece_unittest.exe
  pcrecpp_unittest.vcxproj -> C:\Users\faste\bearcove\poppler-build\pcre-8.45\build\Debug\pcrecpp_unittest.exe
  Building Custom Rule C:/Users/faste/bearcove/poppler-build/pcre-8.45/CMakeLists.txt

And install?

PowerShell session
$ cmake --install build
-- Install configuration: "Release"
CMake Error at build/cmake_install.cmake:39 (file):
  file INSTALL cannot find
  "C:/Users/faste/bearcove/poppler-build/pcre-8.45/build/Release/pcre.lib":
  No error.

Oh, huh, it's trying to install a release build but we made a debug build.

Fine, let's fix our mistakes...

PowerShell session
$ cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=C:/Users/faste/bearcove/poppler-build/prefix
(cut)
$ cmake --build build --config Release --parallel (Get-CimInstance Win32_ComputerSystem).NumberOfLogicalProcessors
(cut)
$ cmake --install .\build\ --config Release
-- Installing: C:/Users/faste/bearcove/poppler-build/prefix/lib/pcre.lib
-- Installing: C:/Users/faste/bearcove/poppler-build/prefix/lib/pcreposix.lib
-- Installing: C:/Users/faste/bearcove/poppler-build/prefix/lib/pcrecpp.lib
-- Installing: C:/Users/faste/bearcove/poppler-build/prefix/bin/pcregrep.exe
(cut)
-- Installing: C:/Users/faste/bearcove/poppler-build/prefix/lib/pkgconfig/libpcre.pc
-- Installing: C:/Users/faste/bearcove/poppler-build/prefix/lib/pkgconfig/libpcrecpp.pc
-- Installing: C:/Users/faste/bearcove/poppler-build/prefix/lib/pkgconfig/libpcreposix.pc
-- Installing: C:/Users/faste/bearcove/poppler-build/prefix/bin/pcre-config

Oh look, it installed .pc files!

Now I guess we can get pkg-config to find those!

PowerShell session
$ pkg-config --static --cflags libpcre
-DPCRE_STATIC -IC:/Users/faste/bearcove/poppler-build/prefix/include

$ pkg-config --static --libs libpcre
-LC:/Users/faste/bearcove/poppler-build/prefix/lib -lpcre

We can!

Mhh but those look like command-line flags for GCC/Clang. You just said the flags for MSVC looked different! (And then gave no example).

Well, time for an example then, because on Windows, pkg-config definitely has a setting for that:

PowerShell session
$ pkg-config --msvc-syntax --static --libs libpcre
/libpath:C:/Users/faste/bearcove/poppler-build/prefix/lib pcre.lib

Look, it's different! Using /flag instead of -flag / --flag. It doesn't seem to change anything for --cflags though, so uhhh I'm curious how everything will work out.

Let's maybe try to make a quick project using libpcre from Rust with system-deps, to see how it'll handle that.

PowerShell session
$ cargo new pcre-test
     Created binary (application) `pcre-test` package

$ cd pcre-test
$ cargo add -B system-deps
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding system-deps v6.0.0 to build-dependencies
Rust code
// in `pcre-test/build.rs`

fn main() {
    system_deps::Config::new().probe().unwrap();
}
TOML markup
// in `pcre-test/Cargo.toml`

[package.metadata.system-deps]
libpcre = { version = "8.45" }
Rust code
// in `pcre-test/src/main.rs`

use std::ffi::CStr;

extern "C" {
    fn pcre_version() -> *const i8;
}

fn main() {
    let version = unsafe { CStr::from_ptr(pcre_version()) };
    dbg!(&version);
}

If we try cargo run, it fails!

PowerShell session
$ cargo run
   Compiling serde v1.0.130
   Compiling smallvec v1.7.0
   Compiling unicode-segmentation v1.8.0
   Compiling pkg-config v0.3.22
   Compiling version-compare v0.1.0
   Compiling cfg-expr v0.9.0
   Compiling heck v0.3.3
   Compiling toml v0.5.8
   Compiling system-deps v6.0.0
   Compiling pcre-test v0.1.0 (C:\Users\faste\bearcove\pcre-test)
error: failed to run custom build command for `pcre-test v0.1.0 (C:\Users\faste\bearcove\pcre-test)`

Caused by:
  process didn't exit successfully: `C:\Users\faste\bearcove\pcre-test\target\debug\build\pcre-test-128e1f1f3d1e876c\build-script-build` (exit code: 101)
  --- stdout
  cargo:rerun-if-env-changed=LIBPCRE_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-pc-windows-msvc
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_pc_windows_msvc
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-pc-windows-msvc
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_pc_windows_msvc
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-pc-windows-msvc
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_pc_windows_msvc
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-pc-windows-msvc
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_pc_windows_msvc
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR

  --- stderr
  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: PkgConfig(`"pkg-config" "--libs" "--cflags" "libpcre" "libpcre >= 8.45"` did not exit successfully: exit code: 1
  --- stderr
  Package libpcre was not found in the pkg-config search path.
  Perhaps you should add the directory containing `libpcre.pc'
  to the PKG_CONFIG_PATH environment variable
  No package 'libpcre' found
  Package libpcre was not found in the pkg-config search path.
  Perhaps you should add the directory containing `libpcre.pc'
  to the PKG_CONFIG_PATH environment variable
  No package 'libpcre' found
  )', build.rs:2:40
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

...as expected!

Let's just set PKG_CONFIG_PATH and...

PowerShell session
$ $env:PKG_CONFIG_PATH = "C:/Users/faste/bearcove/poppler-build/prefix/lib/pkgconfig"
$ cargo run
   Compiling pcre-test v0.1.0 (C:\Users\faste\bearcove\pcre-test)
    Finished dev [unoptimized + debuginfo] target(s) in 0.39s
     Running `target\debug\pcre-test.exe`
[src\main.rs:9] &version = "8.45 2021-06-15"

It works perfectly! I'm really happy about that. There was hardly a mention of windows in the system-deps docs, it was honestly a 50/50 between "just works" and "woops sorry we assumed Linux all along".

For learning purposes, I did scoop uninstall pkg-config, and cargo clean && cargo run failed - it does need the real thing in path, they haven't reimplemented pkg-config as a Rust library or anything.

Should they?

I don't know, can you think of an argument for doing that?

Well, you wouldn't need to install pkg-config on Windows. It's a pretty nonstandard tool to have on there...

And can you think of an argument against doing that?

I suppose the Rust implementation would have to have the same hardcoded paths as the "real" pkg-config on Linux (or shell out to it to discover them?), and there's always the risk the implementations could drift apart...

Exactly! So, should they?

Ehhhhhhhhhhhhhhhhh.

Exactly.

So, at this point, in this article about checks notes vector graphics, all we need to do is build all those libraries and we should be good, right?

But to be completely honest with y'all, I don't feel like porting my build scripts to Windows, and maintaining them so that they work on both Windows and Linux.

This feels like a lot of work, /especially/ since I discovered that Meson can download and compile dependencies.

A crash course in meson wraps

Alright, let's start simple! We'll make a new folder, poppler-meson, and add this file:

meson.build file
# in `poppler-meson/meson.build`

project('poppler-meson', 'c',
          version: '1.0.0',
          meson_version: '>= 0.60.1'
)

glib_dep = dependency('glib-2.0')

This should only pull in glib and its dependencies. Right now, our project fails to configure:

PowerShell session
$ meson setup build --wipe
The Meson build system
Version: 0.60.1
Source dir: C:\Users\faste\bearcove\poppler-meson
Build dir: C:\Users\faste\bearcove\poppler-meson\build
Build type: native build
Project name: poppler-meson
Project version: 1.0.0
Activating VS 17.0.1
C compiler for the host machine: cl (msvc 19.30.30705 "Microsoft (R) C/C++ Optimizing Compiler Version 19.30.30705 for x64")
C linker for the host machine: link link 14.30.30705.0
Host machine cpu family: x86_64
Host machine cpu: x86_64
Found pkg-config: C:\Users\faste\scoop\shims\pkg-config.EXE (0.26)
Found CMake: C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.EXE (3.21.21080301)
Traceback (most recent call last):
  (cut)
Build type: native build
Project name: poppler-meson
Project version: 1.0.0
Activating VS 17.0.1
C compiler for the host machine: cl (msvc 19.30.30705 "Microsoft (R) C/C++ Optimizing Compiler Version 19.30.30705 for x64")
C linker for the host machine: link link 14.30.30705.0
Host machine cpu family: x86_64
Host machine cpu: x86_64
Found pkg-config: C:\Users\faste\scoop\shims\pkg-config.EXE (0.26)
Found CMake: C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.EXE (3.21.21080301)        
Run-time dependency glib-2.0 found: NO (tried pkgconfig and cmake)

meson.build:6:0: ERROR: Dependency "glib-2.0" not found, tried pkgconfig and cmake

A full log can be found at C:\Users\faste\bearcove\poppler-meson\build\meson-logs\meson-log.txt

And that's because we added a dependency on glib-2.0, but didn't give meson any information on where to find it. So, it tried to find it with pkg-config and cmake, but it didn't. On Fedora, with glib2-devel installed, this would probably work.

But we're on Windows now! So meson needs additional info. Luckily, meson now comes with "wraptool", which lets us pick from a list of libraries that have been "wrapped" so they can be built with meson instead of whatever their maintainers thought they should be built with (autotools, cmake, a home-grown organic set of Makefiles, etc.)

PowerShell session
$ meson wrap search glib-2.0
Dependency glib-2.0 found in wrap glib

Hey there it is! Let's add it:

PowerShell session
$ meson wrap install glib
Installed glib version 2.70.1 revision 1

Just like that, wraptool added the subprojects/glib.wrap file:

meson .wrap file
[wrap-file]
directory = glib-2.70.1
source_url = https://download.gnome.org/sources/glib/2.70/glib-2.70.1.tar.xz
source_filename = glib-2.70.1.tar.xz
source_hash = f9b7bce7f51753a1f43853bbcaca8bf09e15e994268e29cfd7a76f65636263c0

[provide]
dependency_names = gthread-2.0, gobject-2.0, gmodule-no-export-2.0, gmodule-export-2.0, gmodule-2.0, glib-2.0, gio-2.0, gio-win32-2.0, gio-unix-2.0
program_names = glib-genmarshal, glib-mkenums, glib-compile-schemas, glib-compile-resources, gio-querymodules, gdbus-codegen

(With hashes and everything!)

Running meson setup again goes further! It actually crates subprojects/libpcre.wrap, since it's required by glib:

meson .wrap file
[wrap-redirect]
filename = glib-2.70.1\subprojects\libpcre.wrap

But unfortunately, it eventually fails, because the default mirror for pcre sources hates me, apparently:

PowerShell session
$ meson setup --wipe build --prefix C:\Users\faste\bearcove\poppler-meson\prefix\ --default-library static
(cut)
glib| Run-time dependency libpcre found: NO (tried pkgconfig and cmake)
glib| Library pcred found: NO
glib| Run-time dependency libpcre found: NO (tried pkgconfig and cmake)
glib| Looking for a fallback subproject for the dependency libpcre
glib| Using subprojects\glib-2.70.1\subprojects\libpcre.wrap
glib| Downloading libpcre source from https://ftp.pcre.org/pub/pcre/pcre-8.37.tar.bz2
glib| <urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>glib| A fallback URL could be specified using source_fallback_url key in the wrap file

subprojects\glib-2.70.1\meson.build:2002:2: ERROR: could not get https://ftp.pcre.org/pub/pcre/pcre-8.37.tar.bz2 is the internet available?

Luckily, there's other mirrors we can pick from! By replacing the contents of subprojects/libpcre.wrap with a slightly tuned version of subprojects/glib-2.70.1/subprojects/libpcre.wrap...

meson .wrap file
[wrap-file]
directory = pcre-8.37
source_url = https://ftp.halifax.rwth-aachen.de/osdn/sfnet/p/pc/pcre/pcre/8.37/pcre-8.37.tar.bz2
source_filename = pcre-8.37.tar.bz2
source_hash = 51679ea8006ce31379fb0860e46dd86665d864b5020fc9cd19e71260eef4789d
patch_filename = pcre_8.37-2_patch.zip
patch_url = https://wrapdb.mesonbuild.com/v2/pcre_8.37-2/get_patch
patch_hash = 6b80f72385e1bf06721e26fbc83aced576e9c0d3182d86a55dd173a04050fe26

[provide]
libpcre = pcre_dep

...we can keep going.

PowerShell session
$ meson setup --wipe build --prefix C:\Users\faste\bearcove\poppler-meson\prefix\ --default-library static
(cut)

Build targets in project: 341

poppler-meson 1.0.0

  Subprojects
    glib           : YES 3 warnings
    libffi         : YES
    libpcre        : YES
    proxy-libintl  : YES
    zlib           : YES

  User defined options
    default_library: static
    prefix         : C:\Users\faste\bearcove\poppler-meson\prefix\

Found ninja-1.10.2 at C:\Users\faste\scoop\shims\ninja.EXE

Visual Studio environment is needed to run Ninja. It is recommended to use Meson wrapper:
C:\Users\faste\AppData\Local\Programs\Python\Python310\Scripts\meson compile -C build

Look at that, it pulled in zlib, libffi and proxy-libintl as well!

It did! How convenient.

A short meson compile -C build and many MSVC warnings later... it fails to build. Well, it turns out glib static builds on Windows are kind of a work-in-progress, so I had to learn about a few more meson features...

Some C defines were missing, but luckily, we can override them at the project level. We can even have the project default to a static build.

As for the glib dependency, we can also set things there, like ask for it to be statically linked with static: true, and set some default options, like: "don't bother with tests".

meson.build file
# in poppler-meson/meson.build

project('poppler-meson', 'c',
          version: '1.0.0',
          meson_version: '>= 0.60.1',
          default_options: ['c_args=-DG_INTL_STATIC_COMPILATION=1 -DFFI_STATIC_BUILD=1', 'default_library=static'],
)

glib_dep = dependency('glib-2.0', static: true, default_options: ['tests=false'])

And because glib really insists on including DllMain in its multiple libraries (gobject, gio, etc.), even when they're built to a static .a library on Windows, I had to bring in a little patch.

Which is a thing meson lets you do! Usually to "add meson build files", but what's a little crime among friends?

meson .wrap file
# in `subprojects/glib.wrap`

[wrap-file]
directory = glib-2.70.1
source_url = https://download.gnome.org/sources/glib/2.70/glib-2.70.1.tar.xz
source_filename = glib-2.70.1.tar.xz
source_hash = f9b7bce7f51753a1f43853bbcaca8bf09e15e994268e29cfd7a76f65636263c0
# 👇 new!
patch_directory = glib-2.70.1

[provide]
dependency_names = gthread-2.0, gobject-2.0, gmodule-no-export-2.0, gmodule-export-2.0, gmodule-2.0, glib-2.0, gio-2.0, gio-win32-2.0, gio-unix-2.0
program_names = glib-genmarshal, glib-mkenums, glib-compile-schemas, glib-compile-resources, gio-querymodules, gdbus-codegen

The patch is actually not a patch, more like an overlay, so I had to create subprojects/packagefiles/glib-2.70.1/glib/glib-init.c with my already-patched version of this source file. Handy!

And finally, it builds:

PowerShell session
$ meson compile -C build     
Activating VS 17.0.1
ninja: Entering directory `C:/Users/faste/bearcove/poppler-meson/build'
(cut)
[361/361] Linking target subprojects/glib-2.70.1/fuzzing/fuzz_inet_socket_address_new_from_string.exe

And installs:

PowerShell session
$ meson install -C build
ninja: Entering directory `C:\Users\faste\bearcove\poppler-meson\build'
ninja: no work to do.
Installing subprojects\libffi\src\libffi.a to C:\Users\faste\bearcove\poppler-meson\prefix\lib
(cut)
Installing C:\Users\faste\bearcove\poppler-meson\subprojects\glib-2.70.1\m4macros/gsettings.m4 to C:\Users\faste\bearcove\poppler-meson\prefix\share/aclocal
Cool bear's hot tip

Oddly enough, meson install wouldn't find ninja the way meson compile did, and a scoop install ninja fixed it there.

Another option would've been meson devenv -C build, which enters a "dev environment", with C/C++ compilers/linkers (like cl.exe, link.exe) and build tools (cmake.exe, ninja.exe) in PATH.

So let's look at what meson/ninja has actually installed in our prefix.

PowerShell session
$ dir prefix\lib\pkgconfig

    Directory: C:\Users\faste\bearcove\poppler-meson\prefix\lib\pkgconfig

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          01/12/2021    17:34            751 gio-2.0.pc
-a---          01/12/2021    17:34            311 gio-windows-2.0.pc
-a---          01/12/2021    17:34            428 glib-2.0.pc
-a---          01/12/2021    17:34            278 gmodule-2.0.pc
-a---          01/12/2021    17:34            278 gmodule-export-2.0.pc
-a---          01/12/2021    17:34            295 gmodule-no-export-2.0.pc
-a---          01/12/2021    17:34            294 gobject-2.0.pc
-a---          01/12/2021    17:34            262 gthread-2.0.pc
-a---          01/12/2021    17:34            248 libffi.pc
-a---          01/12/2021    17:34            223 zlib.pc

That is great, we definitely needed a bunch of these.

Our static libraries are all installed in prefix\lib\pkgconfig as well:

PowerShell session
$ dir prefix\lib

    Directory: C:\Users\faste\bearcove\poppler-meson\prefix\lib

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          01/12/2021    18:26         127230 libffi.a
-a---          01/12/2021    18:26       13584544 libgio-2.0.a
-a---          01/12/2021    18:26        6860652 libglib-2.0.a
-a---          01/12/2021    18:26          77578 libgmodule-2.0.a
-a---          01/12/2021    18:26        1561240 libgobject-2.0.a
-a---          01/12/2021    18:26          19220 libgthread-2.0.a
-a---          01/12/2021    18:26          28104 libintl.a
-a---          01/12/2021    18:26         309480 libz.a

I didn't immediately noticed something was wrong, because it had been a while since I had to think about MSVC idiosyncracies, but...

PowerShell session
$ cd glib-test
$ cargo run
   Compiling glib-test v0.1.0 (C:\Users\faste\bearcove\glib-test)
error: linking with `link.exe` failed: exit code: 1181
  |
  (cut)
  = note: LINK : fatal error LNK1181: cannot open input file 'glib-2.0.lib'

error: could not compile `glib-test` due to previous error

...that's not how libraries should be named for MSVC. They shouldn't be named libfoobar.a, they should be named foobar.lib.

Luckily, a single PowerShell command lets us solve that (quickly adds powershell to the tags of this article):

PowerShell session
$ Get-ChildItem .\prefix\lib\lib*.a | Rename-Item -NewName { $_.Name -replace "^lib", "" -replace ".a$", ".lib" }
$ Get-ChildItem .\prefix\lib\*.lib

    Directory: C:\Users\faste\bearcove\poppler-meson\prefix\lib

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          01/12/2021    18:26         127230 ffi.lib
-a---          01/12/2021    18:26       13584544 gio-2.0.lib
-a---          01/12/2021    18:26        6860652 glib-2.0.lib
-a---          01/12/2021    18:26          77578 gmodule-2.0.lib
-a---          01/12/2021    18:26        1561240 gobject-2.0.lib
-a---          01/12/2021    18:26          19220 gthread-2.0.lib
-a---          01/12/2021    18:26          28104 intl.lib
-a---          01/12/2021    18:26         309480 z.lib

Tada! Wish zsh would let me do that this easily. Maybe nushell can?

Anyway, now it works:

PowerShell session
$ cargo run
(cut)
LINK : warning LNK4098: defaultlib 'MSVCRTD' conflicts with use of other libs; use /NODEFAULTLIB:library
(cut)

Nope, nope, go back, meson defaults to a debug build, we want a release build for sure.

PowerShell session
$ meson setup build --prefix C:\Users\faste\bearcove\poppler-meson\prefix\ --buildtype release --wipe
$ meson compile -C build
$ meson install -C build
$ # don't forget to rename libfoobar.a to foobar.lib again...

Ok surely now it'll work:

PowerShell session
$ cargo run
(cut)
  = note: glib-2.0.lib(gutils.c.obj) : error LNK2019: unresolved external symbol __imp_CoTaskMemFree referenced in function g_build_home_dir
          glib-2.0.lib(gutils.c.obj) : error LNK2019: unresolved external symbol SHGetKnownFolderPath referenced in function g_build_home_dir
          glib-2.0.lib(gmessages.c.obj) : error LNK2019: unresolved external symbol __imp_MessageBoxW referenced in function g_log_writer_default
          glib-2.0.lib(gwin32.c.obj) : error LNK2019: unresolved external symbol __imp_CommandLineToArgvW referenced in function g_win32_get_command_line
          glib-2.0.lib(giowin32.c.obj) : error LNK2019: unresolved external symbol __imp_PeekMessageA referenced in function g_io_win32_check
          glib-2.0.lib(giowin32.c.obj) : error LNK2019: unresolved external symbol __imp_PostMessageA referenced in function g_io_win32_msg_write
          glib-2.0.lib(giowin32.c.obj) : error LNK2019: unresolved external symbol __imp_IsWindow referenced in function g_io_channel_win32_new_messages
          glib-2.0.lib(gpoll.c.obj) : error LNK2019: unresolved external symbol __imp_MsgWaitForMultipleObjectsEx referenced in function g_poll
          C:\Users\faste\bearcove\glib-test\target\debug\deps\glib_test.exe : fatal error LNK1120: 8 unresolved externals


error: could not compile `glib-test` due to previous error

Err almost. Just missing a couple libraries...

Rust code
// in `glib-test/build.rs`

fn main() {
    system_deps::Config::new().probe().unwrap();

    let target = std::env::var("TARGET").unwrap();
    if target.contains("msvc") {
        println!("cargo:rustc-link-lib=ole32");
        println!("cargo:rustc-link-lib=user32");
        println!("cargo:rustc-link-lib=shell32");
    }
}
PowerShell session
$ cargo run
   Compiling glib-test v0.1.0 (C:\Users\faste\bearcove\glib-test)
    Finished dev [unoptimized + debuginfo] target(s) in 0.78s
     Running `target\debug\glib-test.exe`
[src\main.rs:9] &home_dir = "C:\\Users\\faste"

YES! Yes! Finally!

The source for this sample program, by the way, is:

Rust code
// in `glib-test/src/main.rs`

use std::ffi::CStr;

extern "C" {
    fn g_get_home_dir() -> *const i8;
}

fn main() {
    let home_dir = unsafe { CStr::from_ptr(g_get_home_dir()) };
    dbg!(&home_dir);
}

And the glib-2.0 dependency is specified like so:

TOML markup
[package.metadata.system-deps]
"glib" = { name = "glib-2.0", version = "2.70.1" }

And I include it so you can find out that the system dep's key cannot include dots, but that you can save it by specifying "name".

Okay! So that's glib. I mean, that's glib.

What else do we need... well, cairo's a big one. And good news, cairo, alongside autotools, actually ships its own meson.build file!

But that still leaves poppler itself. And it doesn't ship with its own meson build files. It ships with CMake, and in theory, I could try to port everything to CMake, and if I did, I would use Izzy's wonderful CMake articles, like Everything you never wanted to know about CMake, and How to Find Packages With CMake.

This time, I chose to go the opposite route: everything except poppler already had meson build files, so it seemed like a good opportunity for me to simply write meson.build files for poppler! Which is exactly what I did.

This article is part 4 of the Don't shell out! series.

Read the next part

If you liked what you saw, please support my work!

Patreon logo Become a Patron

Latest video

video cover image
This is a video about video

A descent into madness.

You wouldn't remux a movie. Or would you?

Watch now

You can watch more videos over there

Looking for the homepage?
Another article: A half-hour to learn Rust