Thanks to my sponsors: Victor Song, Tiziano Santoro, Paul Marques Mota, Mark Tomlin, Morgan Rosenkranz, Tanner Muro, Herman J. Radtke III, Vincent, Zachary Thomas, Romain Kelifa, Tyler Bloom, Ernest French, Joonas Koivunen, Xavier Groleau, Ryan, Luke Konopka, Tyler Schmidtke, Boris Dolgov, jer, Olly Swanson and 230 more

Feb 21, 2021

34 min #os · #assembly · #linux · #rust · #elf

Between libcore and libstd

From the series Making our own executable packer

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

👋 This page was last updated ~4 years ago. Just so you know.

You're still here! Fantastic.

I have good news, and bad news. The good news is, we're actually going to make an executable packer now!

Hurray!

I know right? No lie, we're actually really going to start working on the final product from this point onwards.

What uhhh what about the previous fourteen parts?

Ah, yes, the previous fourteen parts. Well, we had fun, didn't we? And we learned a lot about ELF, how it's basically a database format that different tools look at in different ways, how it's mapped in memory (more or less), what we really need to set up before starting up another executable, all that good stuff.

Yeah, yes, okay, but I mean — surely our final product is building upon all the code we've written so far, right?

Well, that's where the bad news come in.

Surely we're not going to just... throw it away?

Yes, exactly!

...exactly what?

We're exactly going to throw it away. Everything we've written. Every single line of code. It's not that much code anyway.

NOT THAT MUCH CO- oh no, yeah, it's actually just under 3000 lines of code, I thought we'd have more than that by this point.

Right?? I was surprised too, but yeah, point is, let's forget about everything we've written up to this point. Well, all the code we've written. We want to remember the lessons.

And the friends we've made along the way!

Mh? Oh yes, the friends, too.

Anyway, we're starting from scratch.

But why?

...a fair question, which I'll try to address as well as I can.

elk is not an executable packer

No, that's not a recursive bacronym.

It's simply that, well, our goal up from Part 2 up until now was merely to "run an executable without exec".

And we did! We got pretty far.

We had to worry about position-independent code, relocations, shared libraries, load orders, weak symbols, indirect functions, thread-local storage, and other fun things.

In the last part, we even managed to launch nano, and vi, etc.

It was really really neat!

But we also discovered that, really, when a process is asking for /lib64/ld-linux-x86-64.so.2, well, it wants /lib64/ld-linux-x86-64.so.2. To get the last few executables to run, we had to pretend that _dl_addr always failed, for example. And we even patched exit so it didn't crash!

We also took a lot of shortcuts that mean that, for example, if an application that uses thread-local storage actually decided to spawn another thread, well, with our loader, they'd be out of luck.

In short: it's really hard to cosplay as glibc.

But that's not the worst part! The worst part is: we didn't even make an executable packer! We made an executable loader!

Yeah, right, sure, but I always figured the plan, at some point, was to just somehow join elk and an actual executable, so that-

Oh yeah? Oh yeah cool bear? Well how? How would even do that?

Uhh I don't know, like maybe gzip the target executable and use cat to concatenate the tw-

God, you really do think you're cool don't you.

Well the readers seem to lik-

You fool. You absolute madnimal. Do you have any ideas how many libraries we've used? Do you know how large our binary is?

$ ll ./target/release/elk
-rwxr-xr-x 2 amos amos 8.3M Feb 20 04:58 ./target/release/elk

Well we can always str-

EVEN STRIPPED it's a whopping 539KiB! Our elders landed on the freaking moon with less code than that! Do you mean to tell me that we really need all that cruft to launch a measly executable? Huh? Do you?

I don't know, Amos, you're always the one saying that obsessing over file sizes is misguided and that one should consider that the constraints of the present are not the constraints of the past, and that a few decades ago they didn't have to worry about Unic-

Unicode??? Where do we need Unicode? We're just mapping four segments and setting up the stack! Hell, we could be writing this in assembly!!

That sounds cool actually, why don't we do that?

Do th- in assembly? Whoa, whoa, bear, come on, this is just banter, I'm not about to write the whole th- I mean, nah, there's still a bunch of logic in there that using a higher-level language could be justified, because-

Right, right, because having an Addr type with a write method so you can just write anywhere in memory is very much in the spirit of Rust. Sure.

Well, no, not that part, but for example..

Oh yeah, sure, in debug builds Rust will check that when you add up two random numbers you've read from fuck-knows-where in a poor ELF file, they don't underflow an u64. Oh yeah. Well done. Bravo. Super safe.

But-

Admit it. You just don't have what it takes to write in assembly. You're a little chicken. Cluck cluck, little chicken, you're scaaaa-

Alright ENOUGH. I write your paychecks you know.

My what?

I may not give them to you. They may be stacked up in the top drawer, but I do write them.

So?

So we're going to make an executable packer, with as much Rust as we can, and as little assembly as we can, bec-

Because?

BECAUSE I SAID SO.

What we take for granted

...but this is not just an appeal to authority.

I have very good reasons for wanting to get rid of libstd.

See, libstd is extremely nice, it gives us so many good things. For example, it lets us access files, with std::fs::File! It lets us access command-line arguments and environment variables with std::env. And it bequeaths upon us a hodgepodge of data structures, like Vec, and HashMap, and more!

But how does it do all those things?

Well,

$ gdb --quiet --args ./target/release/elk run /bin/ls
Reading symbols from ./target/release/elk...
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/amos/ftl/elf-series/target/release/elk.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) break open
Function "open" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (open) pending.
(gdb) r
Starting program: /home/amos/ftl/elf-series/target/release/elk run /bin/ls
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".

Breakpoint 1, 0x00007ffff7ecae50 in open64 () from /usr/lib/libc.so.6
(gdb) bt
#0  0x00007ffff7ecae50 in open64 () from /usr/lib/libc.so.6
#1  0x00005555555a407a in std::sys::unix::fs::{{impl}}::open_c::{{closure}} ()
    at /rustc/5fa22fe6f821ac3801d05f624b123dda25fde32c//library/std/src/sys/unix/fs.rs:745
#2  std::sys::unix::cvt_r<i32,closure-0> () at /rustc/5fa22fe6f821ac3801d05f624b123dda25fde32c//library/std/src/sys/unix/mod.rs:218
#3  std::sys::unix::fs::File::open_c () at /rustc/5fa22fe6f821ac3801d05f624b123dda25fde32c//library/std/src/sys/unix/fs.rs:745
#4  0x000055555559b0ad in std::sys::unix::fs::File::open ()
    at /rustc/5fa22fe6f821ac3801d05f624b123dda25fde32c//library/std/src/sys/unix/fs.rs:733
#5  std::fs::OpenOptions::_open () at /rustc/5fa22fe6f821ac3801d05f624b123dda25fde32c//library/std/src/fs.rs:923
#6  0x000055555556643d in std::fs::OpenOptions::open<&std::path::Path> (self=0x0,
    path=<error reading variable: access outside bounds of object referenced via synthetic pointer>)
    at /home/amos/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/fs.rs:919
#7  std::fs::File::open<&std::path::PathBuf> (path=0x7fffffffcfb8)
    at /home/amos/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/fs.rs:336
#8  elk::process::Process<elk::process::Loading>::load_object<&alloc::string::String> (self=0x7fffffffd700, path=...)
    at /home/amos/ftl/elf-series/elk/src/process.rs:216
#9  elk::process::Process<elk::process::Loading>::load_object_and_dependencies<&alloc::string::String> (self=0x7fffffffd700,
    path=<optimized out>) at /home/amos/ftl/elf-series/elk/src/process.rs:384
#10 0x000055555557027e in elk::cmd_run (args=...) at /home/amos/ftl/elf-series/elk/src/main.rs:79
#11 elk::do_main () at /home/amos/ftl/elf-series/elk/src/main.rs:71
#12 elk::main () at /home/amos/ftl/elf-series/elk/src/main.rs:63
(gdb)

...it wraps libc.

Which is fine, I guess, except it means we now have dependencies on a bunch of libraries, like libc.so.6:

$ ldd ./target/release/elk
        linux-vdso.so.1 (0x00007ffd78bfb000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f2c2b6cd000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f2c2b500000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f2c2b774000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f2c2b4e6000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f2c2b4c5000)

But that's not all! To provide data structures like Vec, and HashMap, it also needs a memory allocator, which, since Rust doesn't default to jemalloc anymore, also happens to be provided by...

$ gdb --quiet --args ./target/release/elk run /bin/ls
Reading symbols from ./target/release/elk...
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/amos/ftl/elf-series/target/release/elk.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) break malloc
Function "malloc" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (malloc) pending.
(gdb) r
Starting program: /home/amos/ftl/elf-series/target/release/elk run /bin/ls
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".

Breakpoint 1, 0x00007ffff7e66620 in malloc () from /usr/lib/libc.so.6
(gdb) bt
#0  0x00007ffff7e66620 in malloc () from /usr/lib/libc.so.6
#1  0x00007ffff7e5021f in __fopen_internal () from /usr/lib/libc.so.6
#2  0x00007ffff7e60d7b in pthread_getattr_np@GLIBC_2.2.5 () from /usr/lib/libc.so.6
#3  0x00005555555a2f67 in std::sys::unix::thread::guard::get_stack_start ()
    at /rustc/5fa22fe6f821ac3801d05f624b123dda25fde32c//library/std/src/sys/unix/thread.rs:288
#4  std::sys::unix::thread::guard::get_stack_start_aligned ()
    at /rustc/5fa22fe6f821ac3801d05f624b123dda25fde32c//library/std/src/sys/unix/thread.rs:305
#5  std::sys::unix::thread::guard::init () at /rustc/5fa22fe6f821ac3801d05f624b123dda25fde32c//library/std/src/sys/unix/thread.rs:336
#6  std::rt::lang_start_internal () at /rustc/5fa22fe6f821ac3801d05f624b123dda25fde32c//library/std/src/rt.rs:37
#7  0x0000555555573e6b in main ()
#8  0x00007ffff7e02b25 in __libc_start_main () from /usr/lib/libc.so.6
#9  0x000055555555d07e in _start ()
(gdb)

...glibc.

And this is sort of a problem, because we don't want to use glibc in the loader part of our packed executable.

We don't want to mess with the brk, because the guest program might use it. But of course, since libstd uses glibc, it does mess with the brk:

$ strace -e trace=brk ./target/release/elk
brk(NULL)                               = 0x5568c20c2000
brk(NULL)                               = 0x5568c20c2000
brk(0x5568c20e3000)                     = 0x5568c20e3000
One of the following subcommands must be present:
    help
    autosym
    run
    dig

+++ exited with 1 +++

Heck, it messes with the %fs register too!

$ strace -e trace=arch_prctl ./target/release/elk
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffef70b10f0) = -1 EINVAL (Invalid argument)
arch_prctl(ARCH_SET_FS, 0x7ff9e10f4c00) = 0
One of the following subcommands must be present:
    help
    autosym
    run
    dig

+++ exited with 1 +++

We cannot have that. And you may be thinking: well, can't we just use another libc, that we can link statically against, like musl?

Well, yes, we could!

$ cargo build --quiet --release --target x86_64-unknown-linux-musl --manifest-path ./elk/Cargo.toml

And then it would have no dynamic dependencies:

$ ldd ./target/x86_64-unknown-linux-musl/release/elk
        statically linked
$ file ./target/x86_64-unknown-linux-musl/release/elk
./target/x86_64-unknown-linux-musl/release/elk: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=111dfd32dcbe213541e36b8c7ff12b4e98d19708, stripped

...but then our program would be even larger!

$ ll ./target/x86_64-unknown-linux-musl/release/elk
-rwxr-xr-x 2 amos amos 8.8M Feb 21 12:30 ./target/x86_64-unknown-linux-musl/release/elk

...even stripped:

$ ll ./target/x86_64-unknown-linux-musl/release/elk
-rwxr-xr-x 2 amos amos 626K Feb 21 12:31 ./target/x86_64-unknown-linux-musl/release/elk

...and it would still mess with brk (for the heap) and the %fs register (for thread-local storage):

$ strace -e trace=brk,arch_prctl ./target/x86_64-unknown-linux-musl/release/elk
arch_prctl(ARCH_SET_FS, 0x7f8bd91ad0c0) = 0
brk(NULL)                               = 0x555556452000
brk(0x555556453000)                     = 0x555556453000
One of the following subcommands must be present:
    help
    autosym
    run
    dig

+++ exited with 1 +++

And we still would lose track of the initial value of %rsp, the top of the stack, which we need because we're going to be setting up auxiliary vectors as well.

But, and that's a fair question, could we just provide our own entry point yet still use musl? And provide our own global allocator?

Well, maybe? I honestly haven't tried.

This is a decision software developers are often faced with: do I take something that does more than what I need, and try to cut it down? Or do I take something that does less than what I need, and try to add the parts I need?

In that case, we want to know exactly what our packer (and packed executable) does, so we're going to start small — with just libcore — and build up from there.

Why do we want to know exactly what our code does? Well, because we're going to do terrible, awful, not-good but technically-legal things to our code.

Planning the heist

Before we write down a single line of code, let's think about what we actually need.

What we want is to turn a big program into a smaller program.

Since a diagram is worth roughly ${2^{10}}$ words, I made one:

Of course the diagram is only a rough approximation of what we're really going for. In real life, there is no magical cloud that solves all our problems. Well... depends who you trust, but that's besides the point.

In real life, it's not a cloud, it's a program. We're going to call it minipak!

But let's think about what's inside all those programs. Since the job of minipak is to take guest and make it smaller, it probably has some way of compressing it.

It also has some ways of... reading files, and since libcore, the minimalist subset of Rust, does not provide that, we're going to come up with our own set of APIs, which we'll call encore.

Finally, it also understands how to read and write ELF files (and manipulate them), which, that's what delf was for, but this time we don't want to rely on libstd, only on libcore, so we're going to make a smaller one called pixie..

As for the guest program, well, it could contain anything! We don't know! It could be a C program, or a Go program, or a Rust program. It could link against libc, or be linked statically with musl, or use no C library whatsoever. It could depend on some shared libraries or not! It could even open some of them at runtime with dlopen! Anything goes.

Hey uh what's that gray box in the smaller guest?

Ah, well it's the fun part! See, the guest innards are compressed - that's how guest gets smaller in the first place. So whenever the packed guest starts executing, its innards first need to be decompressed in memory, and then it can be launched.

So, in "guest (but smaller)", we actually need to include some code of our own, to do the decompressing and the mapping and the setting up of the stack and registers and auxiliary vectors and the jumping to the entry point.

This is also probably going to require a bunch of things that are not in libcore, so we might use most of our libraries there too.

And to avoid using the word "innards" any more, we'll switch to simply calling them the "segments" of the guest. That's all we care about after all, right?

But! There is something very conspicuous about this diagram.

The beginning of guest.mpk is called stage1.

Why would we start numbering things of all a sudden?

Well, to justify that, we need to consider both the file layout and the memory layout of the guest.

For the non-compressed version of the guest, everything is nice! And easy! The different segments get mapped somewhere in memory, and after the last segment, the heap beings (at the initial value of the brk).

For the compressed version however...

...our stage1 code gets mapped, so it's ready to execute.

But, for example, the heap starts at the wrong place. This might not matter for some programs, but we're going to try and be good citizens here and have it at the right place.

Luckily, it's not so hard to fix that problem. ELF is just a database format! We can totally add a dummy segment that isn't mapped from the file, but only exists in memory, and just reserves space for the to-be-decompressed guest segments, so that the heap starts at the right place in memory, right?

And now everything lines up! There's just one problem left...

Let's zoom in:

So, when our compressed executable starts (on the right), stage1 gets mapped into memory, and so does our dummy segments so that the heaps line up. Wonderful!

What is stage1 supposed to do? Well, it's supposed to read the compressed payload, which contains the real segments of the guest, decompress it somewhere in memory, and eventually copy them in place.

But "in place" happens to be exactly where stage1 is.

Which means that if we only had one stage, it would end up overwriting itself with the guest segments, leading to memory corruption and, ultimately, a spectacular crash.

This wouldn't be an issue if we could put stage1 somewhere else, like here:

...but then the heaps would no longer line up.

However, if we had a second stage... we could have the first stage map the second stage wayyy out of the way, and transfer control to it:

And with that little trick, we can load anything.

But let's start small.

Starting small

Our actual plan has quite a few moving parts, and I definitely handwaved over a few important bits, but, let's get started with something.

So, let's make a new cargo workspace, in a folder called minipak, with the following top-level Cargo manifest:

# in `Cargo.toml`
[workspace]
members = [
    "crates/encore",
]

encore, according to plan, will be our utility library on top of libcore that provides a safe interface over pesky stuff like system calls.

$ cargo new --lib crates/encore
     Created library `crates/encore` package

Great. That way, we can open the minipak folder in VS Code and be able to edit all our crates. (Of course, right now, we only have encore).

But before we do that: we're going to need Rust nightly. Instead of switching our default rustup channel, let's simply add a rust-toolchain file at the top.

# in `rust-toolchain`
[toolchain]
channel = "nightly-2021-02-14"
components = ["rustfmt", "clippy"]
targets = ["x86_64-unknown-linux-gnu"]

I normally wouldn't pin to a specific version like nightly-2021-02-14, but since we're going to be using a bunch of unstable features, I feel like it's justified here.

Note that we're installing the x86_64-unknown-linux-gnu target, but in reality, we're going to be using very little of it, since we do not actually intend on using any C library — or any GNU stuff at all, for that matter.

In fact, let's opt out of a whole bunch of stuff immediately.

Since we're going to be needing that in a bunch of other crates (minipak, stage1, and stage2), let's put those items directly in encore — that way we have them everywhere!

// in `crates/encore/src/lib.rs`

// Don't use libstd
#![no_std]
// Allow using inline assembly
#![feature(asm)]
// Allow specifying our own eh_personality
#![feature(lang_items)]
// Allow using intrinsics
#![feature(core_intrinsics)]

// Bring in heap-allocated types like `Vec`, and `String`
extern crate alloc;

pub mod items;

And in that items.rs file, we're going to immediately get down to business:

// in `crates/encore/src/items.rs`

/// Panic handler
#[panic_handler]
fn panic(_info: &core::panic::PanicInfo) -> ! {
    core::intrinsics::abort();
}

/// Exception handling personality
#[lang = "eh_personality"]
fn eh_personality() {}

// Provides memset, memcpy
extern crate rlibc;

/// Used by parts of libcore that aren't panic=abort
#[no_mangle]
#[allow(non_snake_case)]
unsafe extern "C" fn _Unwind_Resume() {}

/// Provides bcmp (aliased below)
extern crate compiler_builtins;

#[no_mangle]
unsafe extern "C" fn bcmp(s1: *const u8, s2: *const u8, n: usize) -> i32 {
    compiler_builtins::mem::bcmp(s1, s2, n)
}

What does all of this do? Well, even though we're not using libstd, the distribution of libcore we're using is built in a certain way. For example, it assumes we're unwinding on panics, to show stack traces.

Something we are not going to do. Instead we are going to abort on panics, as show in our panic handler above. Thus, we don't need (and don't want) to compile in stack unwinding information in our various binaries.

For the parts we're using pre-compiled (mostly libcore), we provide a dummy implementation of _Unwind_Resume, which is actually never going to be used, but will make the linker very happy.

The rest are just functions that rustc sort of assumes are there. Well, it's not rustc so much as it is LLVM, its backend. If you write code that looks like memcpy, it'll actually use memcpy. Same goes for memset, etc.

Normally those would be provided by the C library, but since we don't have one, we'll use this teeny tiny crate called rlibc that provides just a few functions:

$ cargo add rlibc@1.0.0 --manifest-path ./crates/encore/Cargo.toml
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding rlibc v1.0.0 to dependencies

Cool bear's hot tip

The cargo add and cargo rm subcommands are provided by the third-party tool cargo-edit.

Similarly, bcmp is something LLVM assumes is there (probably because we're using a GNU target, now that I think about it), but never fear, we can fish out an implementation for it as well, in the aptly-named compiler_builtins crate:

$ cargo add compiler_builtins@0.1.39 --manifest-path ./crates/encore/Cargo.toml
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding compiler_builtins v0.1.39 to dependencies

And now, encore should typecheck, and even build:

$ cargo check
    Finished dev [unoptimized + debuginfo] target(s) in 0.00s
$ cargo build
   Compiling rlibc v1.0.0
   Compiling compiler_builtins v0.1.39
   Compiling encore v0.1.0 (/home/amos/ftl/minipak/crates/encore)
    Finished dev [unoptimized + debuginfo] target(s) in 0.53s

But if you're using VS Code like me, with the rust-analyzer (RA for short) extension enabled, you may notice a lot of red squigglies around panic and eh_personality, with messages like "found duplicate lang item".

Well, that's because by default, the rust-analyzer extension checks all targets, including test. And, well, the test harness used by cargo test is not no_std friendly:

$ cargo test --quiet
error[E0152]: found duplicate lang item `panic_impl`
 --> crates/encore/src/items.rs:3:1
  |
3 | fn panic(_info: &core::panic::PanicInfo) -> ! {
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: the lang item is first defined in crate `std` (which `test` depends on)
  = note: first definition in `std` loaded from /home/amos/.rustup/toolchains/nightly-2021-02-14-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-6f5658153d127ddd.so, /home/amos/.rustup/toolchains/nightly-2021-02-14-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-6f5658153d127ddd.rlib
  = note: second definition in the local crate (`encore`)

error[E0152]: found duplicate lang item `eh_personality`
 --> crates/encore/src/items.rs:9:1
  |
9 | fn eh_personality() {}
  | ^^^^^^^^^^^^^^^^^^^
  |
  = note: the lang item is first defined in crate `panic_unwind` (which `std` depends on)
  = note: first definition in `panic_unwind` loaded from /home/amos/.rustup/toolchains/nightly-2021-02-14-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-b69e89b2ef7e009e.rlib
  = note: second definition in the local crate (`encore`)

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0152`.
error: could not compile `encore`

To learn more, run the command again with --verbose.

That's actually a really good error message.

It is! So, for the purposes of minimizing the amount of red squigglies going forward, we're going to add a VS Code workspace config, in which we're going to tell the rust-analyzer extension to not check all targets, and we'll sprinkle another couple options that will make completion better with procedural macros.

To open workspace settings, we can simply hit F1 to bring up VS Code's menu, and start typing "workspace settings", eventually picking the "Preferences: Open Workspace Settings (JSON)" option, which will create and open .vscode/setting.json

{
    "rust-analyzer.checkOnSave.allTargets": false,
    "rust-analyzer.procMacro.enable": true,
    "rust-analyzer.cargo.loadOutDirsFromCheck": true
}

This file has autocompletion with docs-on-hover and everything, thanks to JSON schema.

Changing those settings require reloading the VS Code window, so, if you haven't already, do that (it should prompt for you to do it, but if not, F1 => Reload Window).

And just like that, the red squiggles are gone!

Now's a good time to make a first commit, if you're following along at home. Unfortunately, since we didn't create our workspace with cargo new, there's no .gitignore.

And since it is a cargo workspace, all the build artifacts are going to end up in a folder named target at the top of the workspace (for all the crates in the workspace).

So, let's ignore it:

# in `.gitignore`
/target

Also, since we created minipak/crates/encore with cargo new before initializing minipak as a git repository itself, it created a Git repository for us, which we can get rid of:

$ rm -rf crates/encore/.git

Now let's make an executable:

$ cargo new --bin crates/minipak
warning: compiling this new package may not work due to invalid workspace configuration

current package believes it's in a workspace when it's not:
current:   /home/amos/ftl/minipak/crates/minipak/Cargo.toml
workspace: /home/amos/ftl/minipak/Cargo.toml

this may be fixable by adding `crates/minipak` to the `workspace.members` array of the manifest located at: /home/amos/ftl/minipak/Cargo.toml
Alternatively, to keep it out of the workspace, add the package to the `workspace.exclude` array, or add an empty `[workspace]` table to the package's manifest.
     Created binary (application) `crates/minipak` package

The warning, as often, is right on the money. If we want crates/minipak to be part of the workspace, we have to edit our top-level Cargo.toml.

[workspace]
members = [
    "crates/encore",
    # 👇 new!
    "crates/minipak",
]

Now then! The generated code for our new minipak executable isn't quite what we want.

Sure, it works:

$ cargo run --bin minipak
    Finished dev [unoptimized + debuginfo] target(s) in 0.00s
     Running `target/debug/minipak`
Hello, world!

But it depends on libc:

$ ldd ./target/debug/minipak
        linux-vdso.so.1 (0x00007ffc4828d000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007ff27639e000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007ff27637d000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007ff276376000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007ff2761a9000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007ff276419000)

And libstd:

$ nm --demangle ./target/debug/minipak | grep std:: | head -5
00000000000180f0 T <std::error::<impl core::convert::From<alloc::string::String> for alloc::boxed::Box<dyn std::error::Error+core::marker::Send+core::marker::Sync>>::from::StringError as core::fmt::Debug>::fmt
00000000000180c0 T <std::error::<impl core::convert::From<alloc::string::String> for alloc::boxed::Box<dyn std::error::Error+core::marker::Send+core::marker::Sync>>::from::StringError as std::error::Error>::description
00000000000180d0 T <std::error::<impl core::convert::From<alloc::string::String> for alloc::boxed::Box<dyn std::error::Error+core::marker::Send+core::marker::Sync>>::from::StringError as core::fmt::Display>::fmt
00000000000480d8 b std::sys_common::at_exit_imp::LOCK
0000000000048100 b std::sys_common::at_exit_imp::QUEUE

So let's make it no_std!

// in `crates/minipak/src/lib.rs`

// 👇
// Opt out of libstd
#![no_std]

fn main() {
    println!("Hello, world!");
}

Of course, now it doesn't build anymore:

$ cargo run --bin minipak
   Compiling minipak v0.1.0 (/home/amos/ftl/minipak/crates/minipak)
error: cannot find macro `println` in this scope
 --> crates/minipak/src/main.rs:5:5
  |
5 |     println!("Hello, world!");
  |     ^^^^^^^

error: language item required, but not found: `eh_personality`

error: `#[panic_handler]` function required, but not found

error: aborting due to 3 previous errors

error: could not compile `minipak`

To learn more, run the command again with --verbose.

The println!, we can just remove:

// in `crates/minipak/src/lib.rs`

// Opt out of libstd
#![no_std]

fn main() {
    // muffin!
}

As for the language items, we have them in encore!

$ (cd crates/minipak && cargo add ../encore)
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding encore (unknown version) to dependencies

Cool bear's hot tip

Using the --manifest-path option that cargo-edit accepts is complicated here. If we cargo add crates/encore, the path in minipak's Cargo manifest will be wrong.

If we cargo add ../encore, cargo-edit will immediately fail as it cannot read encore's Cargo manifest. I'm fairly sure this is a bug in cargo-edit.

Anyway, cd does the trick here.

Does it run now?

$ cargo run --bin minipak
   Compiling minipak v0.1.0 (/home/amos/ftl/minipak/crates/minipak)
error: language item required, but not found: `eh_personality`

error: `#[panic_handler]` function required, but not found

error: aborting due to 2 previous errors

error: could not compile `minipak`

To learn more, run the command again with --verbose.

Mh, no. It's almost like it doesn't actually include encore in the build...

Can we force it to?

// in `crates/minipak/src/lib.rs`

// below global attributes like `#![no_std]`, which must be the first thing in
// the main source file.
extern crate encore;

$ cargo run --bin minipak
   Compiling minipak v0.1.0 (/home/amos/ftl/minipak/crates/minipak)
error: no global memory allocator found but one is required; link to std or add `#[global_allocator]` to a static item that implements the GlobalAlloc trait.

error: `#[alloc_error_handler]` function required, but not found.

note: Use `#![feature(default_alloc_error_handler)]` for a default error handler.

error: aborting due to 2 previous errors

error: could not compile `minipak`

To learn more, run the command again with --verbose.

Yes!

So that's why we had extern crate rlibc and extern crate alloc. We want them to be linked together with our code even if we don't explicitly use any of their symbols!

Correct! In fact, before the Rust 2018 Edition, one had to add extern crate foobar for all crates you wanted to use.

And now it's implicit? They're pulled in if you have a use directive involving them?

Yep!

Let's move on to the next errors: it's complaining that we have no global memory allocator. That's fair.

In encore, we added extern crate alloc, because we want to be able to use types like Vec and String, but we haven't set an allocator. So, it fails at compile time!

Let's pick one from another prolific Rust article writer, Philipp Oppermann: linked_list_allocator.

Since we'll want an allocator pretty much everywhere, we'll add it directly to encore:

$ (cd crates/encore && cargo add linked_list_allocator@0.8.11)
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding linked_list_allocator v0.8.11 to dependencies

Cool bear's hot tip

Whenever you do cargo add or cargo rm, Cargo.toml is edited outside of VS Code, so the rust-analyzer extension does not pick up the changes (at the time of this writing, at least).

However, if you bring up the VS Code menu with F1 and search for "Rust Analyzer: Reload Workspace", you can have it pick up the new crates without reloading the whole window.

What I like about this allocator is that it doesn't use heuristics to decide whether to use brk or mmap to allocate memory. In fact, it has no knowledge of Linux syscalls at all! It makes zero assumptions. All it wants is a bottom address, and a size.

// in `crates/encore/src/items.rs`

use linked_list_allocator::LockedHeap;

// Set a global allocator
#[global_allocator]
static ALLOCATOR: LockedHeap = LockedHeap::empty();

/// Heap size, in megabytes
const HEAP_SIZE_MB: u64 = 128;

/// Initialize a global allocator that only uses `mmap`, with a fixed heap size.
///
/// # Safety
/// Calling this too late (or not at all) and doing a heap allocation will
/// fail. The `mmap` syscall can also fail, which would be disastrous.
pub unsafe fn init_allocator() {
    let heap_size = HEAP_SIZE_MB * 1024 * 1024;
    let heap_bottom = MmapOptions::new(heap_size).map().unwrap();
    ALLOCATOR.lock().init(heap_bottom as _, heap_size as _);
}

All we need to do now is remember to call init_allocator from all the crates that use encore, and we're good to go!

// in `crates/minipak/src/main.rs`

// Opt out of libstd
#![no_std]

fn main() {
    unsafe {
        encore::items::init_allocator();
    }
}

Cool bear's hot tip

We no longer need extern crate encore, because we're using a symbol from it!

And just like that, we should be good to go:

$ cargo run --quiet --bin minipak
error[E0433]: failed to resolve: use of undeclared type `MmapOptions`
  --> crates/encore/src/items.rs:43:23
   |
43 |     let heap_bottom = MmapOptions::new(heap_size).map().unwrap();
   |                       ^^^^^^^^^^^ use of undeclared type `MmapOptions`

Oh, uh, no. Where did this MmapOptions come from?

Getting our timelines crossed?

Yeah, that's what I get for doing prep work ahead of time...

Okay so, obviously, our current init_allocator function uses some code that does not exist yet. Well, no worries. We'll just stub a bunch of things!

// in `crates/encore/src/lib.rs`

pub mod items;

pub mod error; // new!
pub mod items;
pub mod memmap; // also new!

// in `crates/encore/src/error.rs`

#[derive(Debug)]
pub enum EncoreError {}

// in `crates/encore/src/memmap.rs`

use crate::error::EncoreError;

pub struct MmapOptions {
    len: u64,
}

impl MmapOptions {
    /// Create a new set of mmap options
    pub fn new(len: u64) -> Self {
        Self { len }
    }

    /// Create a memory mapping,
    pub fn map(&mut self) -> Result<u64, EncoreError> {
        todo!()
    }
}

// in `crates/encore/src/items.rs`

use crate::memmap::MmapOptions;

Finally, as requested, we'll add an allocation error handler, but this time, we need to add it directly to minipak - it can't be set in encore.

// in `crates/minipak/src/main.rs`

// Opt out of libstd
#![no_std]
// Use the default allocation error handler
#![feature(default_alloc_error_handler)]

fn main() {
    unsafe {
        encore::items::init_allocator();
    }
}

Surely now, minipak will run:

$ cargo run --quiet --bin minipak

error: requires `start` lang_item

Oh, uh, that's interesting. It complains that init_allocator is not used anywhere... but we call it from main?

Do we even have a main in a no_std binary? Doesn't a main need pre-main initialization code that would normally be provided by a C library?

Ohhhh right! Our entry point is not main, it's more like _start, like we've done before for assembly programs, or echidna, our previous no_std binary.

Well, the compiler output mentions the start language item, but a quick search reveals that it's far from stable (and also its documentation is kinda lacking?).

Instead, let's simply opt out of Rust's main handling altogether and provide a symbol named _start, unmangled, that we know our linker (GNU ld) will pick up.

Since it's an entry point, it's not called, it's "jumped to". We also want to avoid any stack allocations, since the first thing want to remember is the value of %rsp, the stack pointer, so that later on we can read our arguments, environment variables, and auxiliary vectors.

So, we need to opt in and out of a bunch more stuff:

// in `crates/minipak/src/main.rs`

// Opt out of libstd
#![no_std]
// Let us worry about the entry point.
#![no_main]
// Use the default allocation error handler
#![feature(default_alloc_error_handler)]
// Let us make functions without any prologue - assembly only!
#![feature(naked_functions)]
// Let us use inline assembly!
#![feature(asm)]

/// Our entry point.
#[naked]
#[no_mangle]
unsafe extern "C" fn _start() {
    asm!("mov rdi, rsp", "call pre_main", options(noreturn))
}

#[no_mangle]
unsafe fn pre_main(_stack_top: *mut u8) {
    encore::items::init_allocator();
}

Good, good. I like the look of this. Very low-level. Very "hardcore", as they say. Much street cred.

Of course this doesn't actually link.

The actual linker output is quite lengthy, so allow me to highlight just the bit that matters:

$ cargo run --quiet --bin minipak

(cut)

  = note: /usr/sbin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../lib/Scrt1.o: in function `_start':
          (.text+0x16): undefined reference to `__libc_csu_fini'
          /usr/sbin/ld: (.text+0x1d): undefined reference to `__libc_csu_init'
          /usr/sbin/ld: (.text+0x24): undefined reference to `main'
          /usr/sbin/ld: (.text+0x2a): undefined reference to `__libc_start_main'
          collect2: error: ld returned 1 exit status

AhAH! We're still getting some code from the C library linked in, and it expects us to have a main symbol. But we don't have one!

Let us worry about the entry point. Take a day off, glibc. After a few decades, you definitely deserve it.

Let's just opt into one... more... unstable feature...

// in `crates/minipak/src/main.rs`

// (omitted: other features / attributes)
// Let us pass arguments to the linker directly
#![feature(link_args)]

/// Don't link any glibc stuff, also, make this executable static.
#[allow(unused_attributes)]
#[link_args = "-nostartfiles -nodefaultlibs -static"]
extern "C" {}

// (omitted: entry point, pre_main)

And... now it compiles!

$ cargo run --quiet --bin minipak
[1]    9205 illegal hardware instruction  cargo run --quiet --bin minipak

...it even almost runs!

Where does it fail?

$ gdb --quiet -ex "run" --args ./target/debug/minipak
Reading symbols from ./target/debug/minipak...
Starting program: /home/amos/ftl/minipak/target/debug/minipak

Program received signal SIGILL, Illegal instruction.
encore::items::panic (_info=0x7fffffffdb98) at /home/amos/ftl/minipak/crates/encore/src/items.rs:4
4           core::intrinsics::abort();
(gdb) bt
#0  encore::items::panic (_info=0x7fffffffdb98) at /home/amos/ftl/minipak/crates/encore/src/items.rs:4
#1  0x0000000000402301 in core::panicking::panic_fmt ()
    at /rustc/8e54a21139ae96a2aca3129100b057662e2799b9//library/core/src/panicking.rs:92
#2  0x00000000004022cd in core::panicking::panic ()
    at /rustc/8e54a21139ae96a2aca3129100b057662e2799b9//library/core/src/panicking.rs:50
#3  0x0000000000401164 in encore::memmap::MmapOptions::map (self=0x7fffffffdc48)
    at /home/amos/ftl/minipak/crates/encore/src/memmap.rs:15
#4  0x0000000000401078 in encore::items::init_allocator () at /home/amos/ftl/minipak/crates/encore/src/items.rs:45
#5  0x000000000040101e in minipak::pre_main (_stack_top=0x7fffffffdc90)
    at /home/amos/ftl/minipak/crates/minipak/src/main.rs:28
#6  0x0000000000401008 in minipak::_start () at /home/amos/ftl/minipak/crates/minipak/src/main.rs:23

When we panic! Of course! That's just what abort does. It attempts to execute an illegal instruction, which causes the program to brutally stop.

And, see, we don't need panic=unwind, because we can just run our program under GDB, and boom, we get a stack trace.

Although, it would be neat if our program printed something before it crashed.

So, okay, fine, let's actually wrap some syscalls.

Safe-ish syscall wrappers

We've done this before, again, with echidna, our previous no_std program, in Part 12.

But we'll try to do it a little better this time, mh?

Let's add a couple crates to encore that we'll need very soon:

$ (cd crates/encore && cargo add bitflags@1.2.1 && cargo add displaydoc@0.1.7 --no-default-features)
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding bitflags v1.2.1 to dependencies
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding displaydoc v0.1.7 to dependencies

Both these crates are no_std-friendly, provided you disable default feature for displaydoc, which we've just done in our cargo-edit invocation (look closer!).

Next up, we'll add a syscall module to encore:

// in `crates/encore/src/lib.rs`

pub mod syscall;

Let's start at the beginning. It would be neat to write messages to the standard output, affectionately nicknamed "stdout". For that we'll need the write syscall.

The first argument that the write syscall takes is a file descriptor, which on 64-bit Linux is just a 64-bit integer (or is it limited to 32-bit integers? doesn't matter, all registers are 64-bit wide anyway).

Let's make a newtype just for file descriptors:

// in `crates/encore/src/syscall.rs`

/// A file descriptor
#[derive(Default, Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct FileDescriptor(pub u64);

impl FileDescriptor {
    /// Standard input file descriptor
    pub const STDIN: Self = Self(0);
    /// Standard output file descriptor
    pub const STDOUT: Self = Self(1);
    /// Standard error file descriptor
    pub const STDERR: Self = Self(2);
}

/// # Safety
/// Calls into the kernel.
#[inline(always)]
pub unsafe fn write(fd: FileDescriptor, buf: *const u8, count: u64) -> u64 {
    let syscall_number: u64 = 1;
    let mut rax = syscall_number;

    asm!(
        "syscall",
        inout("rax") rax,
        in("rdi") fd.0,
        in("rsi") buf,
        in("rdx") count,
        lateout("rcx") _, lateout("r11") _,
        options(nostack),
    );
    rax
}

Now, there's a lot I like about this wrapper. It forces you to consider that the first argument is not just any integer, it's actually a file descriptor. We provide values for stdin, stdout, and stderr, that have the right type.

And we use lateout to specify that rcx and r11 are clobbered, which means even in release (optimized) builds, everything should work. That's one mistake we won't be making again!

We can use it right now, from minipak:

// in `crates/minipak/src/main.rs`

#[no_mangle]
unsafe fn pre_main(_stack_top: *mut u8) {
    let s = "Hello from minipak!\n";
    encore::syscall::write(
        encore::syscall::FileDescriptor::STDOUT,
        s.as_bytes().as_ptr(),
        s.as_bytes().len() as _,
    );

    encore::items::init_allocator();
}

$ cargo run --quiet --bin minipak
Hello from minipak!
[1]    13309 illegal hardware instruction  cargo run --quiet --bin minipak

It works fine, but it's not very ergonomic.

Well, we've been down the "write our custom formatting routines" road before. It was fun! But we have a lot on our plate, so let's just accept a gift from ~~the gods~~ libcore, shall we?

// in `crates/encore/src/lib.rs`

pub mod utils;

// in `crates/encore/src/utils.rs`

use core::fmt;

#[macro_export]
macro_rules! println {
    ($($arg:tt)*) => {
        {
            use ::core::fmt::Write;
            ::core::writeln!($crate::utils::Stdout, $($arg)*).ok();
        }
    }
}

pub struct Stdout;
impl fmt::Write for Stdout {
    fn write_str(&mut self, s: &str) -> fmt::Result {
        unsafe {
            crate::syscall::write(
                crate::syscall::FileDescriptor::STDOUT,
                s.as_ptr(),
                s.len() as _,
            );
        }
        Ok(())
    }
}

Of course we have to define println! ourselves because libcore does not assume we even have anywhere to print to.

Consider this:

Where would this print to? Huh? That's what I thought.

Our definition of println! is not entirely trivial though, let's go through it real quick:

// We want to be able to access this macro from other crates, so it must be
// exported. Macro namespacing is a bit of a hard problem, because it changes
// *what's parsed*, yet the visibility of macros is also part of, well, what's
// parsed.
// More info here: https://github.com/rust-lang/rust/issues/54727
#[macro_export]
// Our macro has the name `println`
macro_rules! println {
    // It accepts 0 or more "token trees" - any token at all, including
    // commas, parentheses, etc.
    ($($arg:tt)*) => {
        // defines a new scope...
        {
            // ...so that we can import this trait, which we need for the
            // next macro invocation.
            use ::core::fmt::Write;

            // invoke the `writeln!` macro which *is* provided by `libcore`
            // since it accepts a destination.
            ::core::writeln!($crate::utils::Stdout, $($arg)*).ok();

            // note that `$crate::` is absolutely necessary here - we *cannot*
            // just use `crate::`, because since a macro's contents are
            // copy-pasted wherever it's used, it would end up referring to
            // the crate that _invokes_ the macro, rather than the crate where
            // it's defined. we could use `encore::` instead, but then we would
            // have to change it if we ever decide to rename this crate, which
            // I feel too lazy to do just _thinking_ about it.
        }
    }
}

Now, normally when you use libstd, your program implicitly uses a prelude, with a bunch of stuff in it, like Vec, and... the println! macro.

We can do the same with encore:

// in `crates/encore/src/lib.rs`

pub mod prelude;

// in `crates/encore/src/prelude.rs`

pub use crate::{error::EncoreError, items::init_allocator, memmap::MmapOptions, println, syscall};
pub use alloc::{fmt::Write, format, string::String, vec::Vec};

And now, minipak's main.rs can look like this:

// in `crates/minipak/src/main.rs`

// (omitted: everything up until and including `_start`)

use encore::prelude::*;

#[no_mangle]
unsafe fn pre_main(_stack_top: *mut u8) {
    println!("Initializing allocator...");
    init_allocator();
    println!("Initializing allocator... done!");
}

And now we get:

$ cargo run --quiet --bin minipak
Initializing allocator...
[1]    15284 illegal hardware instruction  cargo run --quiet --bin minipak

Nice!

Of course we still can't initialize our allocator, and our panics still straight up abort, which is not very informative.

Just in case this isn't our last panic, let's zhuzh up our panic handler:

// in `crates/encore/src/items.rs`

/// Panic handler
#[panic_handler]
fn panic(info: &core::panic::PanicInfo) -> ! {
    crate::println!("{}", info);
    core::intrinsics::abort();
}

Cool bear's hot tip

Even though we defined the println! macro in the utils module of the encore crate, its full path is crate::println!, not crate::utils::println!.

Told you macro namespacing was a bit fuzzy!

Now we have this:

$ cargo run --quiet --bin minipak
Initializing allocator...
panicked at 'not yet implemented', crates/encore/src/memmap.rs:16:9
[1]    15709 illegal hardware instruction  cargo run --quiet --bin minipak

Which feels much nicer.

Setting the RUST_BACKTRACE environment variable to 1 does sweet nothing, because stack unwinding is something libstd does, and we've opted out of it.

Speaking of, we should probably opt into panic=abort as well, just so we don't generate unnecessary code.

Since it goes into a "profile", it has to be in the top-level Cargo manifest, that of the workspace, not any specific crate:

# in `Cargo.toml`

[workspace]
members = [
    "crates/encore",
    "crates/minipak",
]

[profile.dev]
panic = "abort"

[profile.release]
debug = true
lto = "thin"
panic = "abort"

While we're at it, I've also signed us up for debug symbols in release builds, and Thin link-time optimization.

Just by switching to panic = "abort", our binary size reduced from 1.1 MiB to 799 KiB. Not bad! It's one of the tricks recommended in min-sized rust.

Our release binary is currently 89 KiB unstripped, 17 KiB stripped. That's fairly close to being optimal, given that we have 4 sections:

One with the ELF headers
One with the code
One with constants
One with read-write globals

...so the minimum size is 4*4KiB = 16KiB. Taking a quick look at a stripped version of our release build reveals that our code takes up slightly more than 4 KiB:

$ readelf -Wl ./target/release/minipak

Elf file type is EXEC (Executable file)
Entry point 0x402560
There are 8 program headers, starting at offset 64

Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  LOAD           0x000000 0x0000000000400000 0x0000000000400000 0x000224 0x000224 R   0x1000
                                                                   👇
  LOAD           0x001000 0x0000000000401000 0x0000000000401000 0x0015d5 0x0015d5 R E 0x1000
  LOAD           0x003000 0x0000000000403000 0x0000000000403000 0x0004c8 0x0004c8 R   0x1000
  LOAD           0x003e90 0x0000000000404e90 0x0000000000404e90 0x000170 0x000170 RW  0x1000

Doesn't our program still segfault?

Ah, right! We never actually wrapped mmap.

Well, let's do so now.

Making mmap sorta-kinda safe

The mmap syscall is a tad more complex than open, at least in its interface.

It accepts two kinds of flags: the memory protection (readable, writable, executable), and some options (private, fixed, anonymous).

For flags like those, the bitflags crate will come in really handy:

// in `crates/encore/src/syscall.rs`

use bitflags::*;

bitflags! {
    /// Memory protection: readable, writable, executable
    #[derive(Default)]
    pub struct MmapProt: u64 {
        /// The mapping will be readable
        const READ = 0x1;
        /// The mapping will be writable
        const WRITE = 0x2;
        /// The mapping will be executable
        const EXEC = 0x4;
    }
}

bitflags! {
    /// Flags for the `mmap` syscall
    pub struct MmapFlags: u64 {
        /// Create a private copy-on-write mapping.
        const PRIVATE = 0x02;
        /// Don't interpret addr as a hint: place the mapping at exactly that address.
        const FIXED = 0x10;
        /// Mapping is not backed by any file
        const ANONYMOUS = 0x20;
    }
}

/// # Safety
/// Calls into the kernel. May unmap running code.
/// One of the most unsafe syscalls in existence.
#[inline(always)]
pub unsafe fn mmap(
    addr: u64,
    len: u64,
    prot: MmapProt,
    flags: MmapFlags,
    fd: FileDescriptor,
    off: u64,
) -> u64 {
    let syscall_number: u64 = 9;
    let mut rax = syscall_number;

    asm!(
        "syscall",
        inout("rax") rax,
        in("rdi") addr,
        in("rsi") len,
        in("rdx") prot.bits(),
        in("r10") flags.bits(),
        in("r8") fd.0,
        in("r9") off,
        lateout("rcx") _, lateout("r11") _,
        options(nostack),
    );
    rax
}

The bitflags! macro lets us define a custom type that behaves like bitflags: the BitAnd and BitOr traits are implemented, for example, so we can do MmapFlags::PRIVATE | MmapFlags::FIXED and things just work.

The bits method returns the underlying type, here u64, so that we can "pass it to assembly".

So, we have an mmap wrapper! Still, that's a lot of parameters. Calling it directly is not ergonomic at all. Not one bit.

Let's try to make a safer abstraction on top of it!

There's already a handful of Rust crates that try to abstract mmap safely, but they rely on libc, which we cannot have, and also, they have opinions that are slightly different from mine.

I'd like mine to be able to:

Allocate big blocks of memory, read-write
Occasionally map entire files, read-only
Check that offsets are page-aligned before doing the syscall

The builder pattern seems like a good fit here, so let's go for it:

// in `crates/encore/src/memmap.rs`

use syscall::FileDescriptor;

use crate::{
    error::EncoreError,
    syscall::{self, MmapFlags, MmapProt},
};

/// Options for creating a memory mapping
pub struct MmapOptions {
    prot: MmapProt,
    flags: MmapFlags,
    len: u64,
    file: Option<FileOpts>,
    at: Option<u64>,
}

/// Options for mapping a file
#[derive(Default, Clone)]
pub struct FileOpts {
    /// An open file descriptor
    pub fd: FileDescriptor,
    /// The offset at which to map the file
    pub offset: u64,
}

impl MmapOptions {
    pub fn new(len: u64) -> Self {
        Self {
            prot: MmapProt::READ | MmapProt::WRITE,
            flags: MmapFlags::ANONYMOUS | MmapFlags::PRIVATE,
            len,
            file: None,
            at: None,
        }
    }

    /// Specify a file that should be mapped
    pub fn file(&mut self, file: FileOpts) -> &mut Self {
        self.file = Some(file);
        self
    }

    /// Sets protections - defaults to READ+WRITE
    pub fn prot(&mut self, prot: MmapProt) -> &mut Self {
        self.prot = prot;
        self
    }

    /// Set flags. Note that `ANONYMOUS` and `PRIVATE` are the default,
    /// and this overwrites them. If `at` is set, `FIXED` is also used.
    pub fn flags(&mut self, flags: MmapFlags) -> &mut Self {
        self.flags = flags;
        self
    }

    /// Specify a fixed address for this mapping (sets the `FIXED` flag)
    pub fn at(&mut self, at: u64) -> &mut Self {
        self.at = Some(at);
        self
    }

    /// Actually create the mapping
    pub fn map(&mut self) -> Result<u64, EncoreError> {
        let mut flags = self.flags;

        if let Some(at) = &self.at {
            if !is_aligned(*at) {
                return Err(EncoreError::MmapMemUnaligned(*at));
            }
            flags.insert(MmapFlags::FIXED);
        }

        if let Some(file) = &self.file {
            if !is_aligned(file.offset) {
                return Err(EncoreError::MmapFileUnaligned(file.offset));
            }
            flags.remove(MmapFlags::ANONYMOUS);
        }

        let file = self.file.clone().unwrap_or_default();
        let addr = self.at.unwrap_or_default();

        let res = unsafe { syscall::mmap(addr, self.len, self.prot, flags, file.fd, file.offset) };
        if res as i64 == -1 {
            return Err(EncoreError::MmapFailed);
        }
        Ok(res)
    }
}

fn is_aligned(x: u64) -> bool {
    x & 0xFFF == 0
}

There!

Note that we don't have to worry about flags much. mmap can do a lot more, like for example sharing memory across multiple processes, but we don't need that here — all our mappings are PRIVATE.

They're also ANONYMOUS, unless the user passes a FileOpts, in which case they're able to specify a FileDescriptor and an offset in one fell swoop.

And finally, calling at automatically sets the FIXED flag. Oh, and any offsets specified are checked for alignment. We assume 4KiB pages because... it's convenient.

We've used EncoreError variants that don't exist yet, though: let's fix that.

Normally, I'd use a crate like thiserror, but it is unfortunately not no_std friendly, so instead, it's time to give Jane's wonderful displaydoc a try.

// in `crates/encore/src/error.rs`

#[derive(Debug)]
pub enum EncoreError {
    /// mmap fixed address provided was not aligned to 0x1000: {0}
    MmapMemUnaligned(u64),
    /// mmap file offset provided was not aligned to 0x1000: {0}
    MmapFileUnaligned(u64),
    /// mmap syscall failed
    MmapFailed,
}

Et voilà! Le tour est joué.

$ cargo run --quiet --bin minipak
Initializing allocator...
Initializing allocator... done!
[1]    19138 illegal hardware instruction  cargo run --quiet --bin minipak

Oh, forgot about exit.

// in `crates/encore/src/syscall.rs`

#[inline(always)]
pub fn exit(code: i32) -> ! {
    let syscall_number: u64 = 60;
    unsafe {
        asm!(
            "syscall",
            in("rax") syscall_number,
            in("rdi") code,
            options(noreturn, nostack),
        );
    }
}

// in `crates/minipak/src/main.rs`

#[no_mangle]
unsafe fn pre_main(_stack_top: *mut u8) {
    println!("Initializing allocator...");
    init_allocator();
    println!("Initializing allocator... done!");
    syscall::exit(0);
}

$ cargo run --quiet --bin minipak
Initializing allocator...
Initializing allocator... done!

Cool! It doesn't crash anymore! It's time to wish y'all a good week, as I go back to my ~~cave~~ day job until ne-

...but we didn't do anything?

Oh what, planning doesn't count? Plus we did a ton of preparation!

Right, but that's really boring. And we've done it all before, with echidna. There was nothing new here, nothing new at all!

Ahhhh well we've done it before, but not that well. You can tell that this time, we're preparing for war. We have proper string formatting, a println! macro, a nice panic handler, even a memory allocator.

Speaking of, we haven't even tried using that memory allocator! How do we know it works??

You know what bear, that's fair. Let's have a little bit of fun before we close out. Just to showcase what we can do from our no_std binary.

For example, we could print the contents of /etc/hosts - that's always fun!

So first we open a file, and then we... ah crap, I forgot about open.

See? Plenty to do still.

Fine, FINE, we'll make an abstraction for files, but then I'm out of here.

First, a couple syscalls:

// in `encore/src/syscall.rs`

bitflags! {
    /// Flags for the `open` syscall
    pub struct OpenFlags: u64 {
        /// Read-only (open flag)
        const RDONLY = 0o0;
        /// Read-write (open flag)
        const RDWR = 0o2;
        /// Create (open flag)
        const CREAT = 0o100;
        /// Truncate (open flag)
        const TRUNC = 0o1000;
    }
}

/// # Safety
/// Calls into the kernel.
#[inline(always)]
pub unsafe fn open(filename: *const u8, flags: OpenFlags, mode: u64) -> FileDescriptor {
    let syscall_number: u64 = 2;
    let mut rax = syscall_number;

    asm!(
        "syscall",
        inout("rax") rax,
        in("rdi") filename,
        in("rsi") flags.bits(),
        in("rdx") mode,
        lateout("rcx") _, lateout("r11") _,
        options(nostack),
    );
    FileDescriptor(rax)
}

#[repr(C)]
pub struct Stat {
    // As found using `offsetof` and `sizeof`
    _unused1: [u8; 48],
    pub size: u64,
    _unused2: [u8; 88],
}

/// # Safety
/// Calls into the kernel.
#[inline(always)]
pub unsafe fn fstat(fd: FileDescriptor, buf: *mut Stat) -> u64 {
    let syscall_number: u64 = 5;
    let mut rax = syscall_number;

    asm!(
        "syscall",
        inout("rax") rax,
        in("rdi") fd.0,
        in("rsi") buf,
        lateout("rcx") _, lateout("r11") _,
        options(nostack),
    );
    rax
}

/// # Safety
/// Calls into the kernel.
#[inline(always)]
pub unsafe fn close(fd: FileDescriptor) -> u64 {
    let syscall_number: u64 = 3;
    let mut rax = syscall_number;

    asm!(
        "syscall",
        inout("rax") rax,
        in("rdi") fd.0,
        lateout("rcx") _, lateout("r11") _,
        options(nostack),
    );
    rax
}

/// # Safety
/// Calls into the kernel.
#[inline(always)]
pub unsafe fn munmap<T>(addr: *const T, len: u64) -> u64 {
    let syscall_number: u64 = 11;
    let mut rax = syscall_number;

    asm!(
        "syscall",
        inout("rax") rax,
        in("rdi") addr,
        in("rsi") len,
        lateout("rcx") _, lateout("r11") _,
        options(nostack),
    );
    rax
}

And then, an fs module!

// in `crates/encore/src/lib.rs`

pub mod fs;

// in `crates/encore/src/fs.rs`

use crate::{
    error::EncoreError,
    memmap::{FileOpts, MmapOptions},
    syscall::{self, FileDescriptor, MmapProt, OpenFlags, Stat},
};
use alloc::{format, string::String};
use core::{
    mem::MaybeUninit,
    ops::{Index, Range},
};

/// A read-only file
pub struct File {
    path: String,
    fd: FileDescriptor,
}

#[allow(clippy::clippy::len_without_is_empty)]
impl File {
    /// Opens a file (read-only)
    pub fn open(path: &str) -> Result<Self, EncoreError> {
        Self::raw_open(path, OpenFlags::RDONLY, 0)
    }

    /// Creates a file (for writing)
    pub fn create(path: &str, mode: u64) -> Result<Self, EncoreError> {
        Self::raw_open(
            path,
            OpenFlags::RDWR | OpenFlags::CREAT | OpenFlags::TRUNC,
            mode,
        )
    }

    /// Internal: open a file with given flags and mode
    fn raw_open(path: &str, flags: OpenFlags, mode: u64) -> Result<Self, EncoreError> {
        let nul_path = format!("{}\0", path);
        let fd = unsafe { syscall::open(nul_path.as_ptr(), flags, mode) };
        if (fd.0 as i64) < 0 {
            return Err(EncoreError::Open(path.into()));
        }

        Ok(Self {
            path: path.into(),
            fd,
        })
    }

    /// Write a whole buffer to this file, or fail. This may involve
    /// multiple syscalls.
    pub fn write_all(&mut self, mut buf: &[u8]) -> Result<(), EncoreError> {
        while !buf.is_empty() {
            let written = unsafe { syscall::write(self.fd, buf.as_ptr(), buf.len() as u64) };
            if written as i64 == -1 {
                return Err(EncoreError::Write(self.path.clone()));
            }
            buf = &buf[written as usize..];
        }
        Ok(())
    }

    /// Returns the length of the file, in bytes
    pub fn len(&self) -> Result<u64, EncoreError> {
        let mut stat = MaybeUninit::<Stat>::uninit();
        let ret = unsafe { syscall::fstat(self.fd(), stat.as_mut_ptr()) };
        if ret != 0 {
            return Err(EncoreError::Stat(self.path.clone()));
        }
        let stat = unsafe { stat.assume_init() };
        Ok(stat.size)
    }

    /// Returns the file descriptor
    pub fn fd(&self) -> FileDescriptor {
        self.fd
    }

    /// Map this file in memory (read-only)
    pub fn map(&self) -> Result<Map<'_>, EncoreError> {
        let self_data = MmapOptions::new(self.len()?)
            .file(FileOpts {
                fd: self.fd,
                offset: 0,
            })
            .prot(MmapProt::READ)
            .map()? as *const u8;
        let data = unsafe { core::slice::from_raw_parts(self_data, self.len()? as _) };

        Ok(Map { file: self, data })
    }
}

impl Drop for File {
    fn drop(&mut self) {
        // Close on drop
        unsafe { syscall::close(self.fd) };
    }
}

pub struct Map<'a> {
    /// This file exists so the file isn't closed until the mapping is dropped.
    #[allow(unused)]
    file: &'a File,

    data: &'a [u8],
}

impl<'a> Drop for Map<'a> {
    fn drop(&mut self) {
        // Munmap on drop
        unsafe { syscall::munmap(self.data.as_ptr(), self.data.len() as _) };
    }
}

impl<'a> AsRef<[u8]> for Map<'a> {
    fn as_ref(&self) -> &[u8] {
        self.data
    }
}

impl<'a> Index<Range<usize>> for Map<'a> {
    type Output = [u8];

    fn index(&self, index: Range<usize>) -> &Self::Output {
        &self.data[index]
    }
}

Add the missing error variants:

// in `crates/encore/src/error.rs`

use alloc::string::String;

#[derive(Debug)]
pub enum EncoreError {
    /// Could not open file `0`
    Open(String),
    /// Could not write to file `0`
    Write(String),
    /// Could not statfile `0`
    Stat(String),

    /// mmap fixed address provided was not aligned to 0x1000: {0}
    MmapMemUnaligned(u64),
    /// mmap file offset provided was not aligned to 0x1000: {0}
    MmapFileUnaligned(u64),
    /// mmap syscall failed
    MmapFailed,
}

And for good measure, let's add some of these to our custom prelude:

// in `crates/encore/src/prelude.rs`

pub use crate::{
    error::EncoreError,
    fs::File,
    items::init_allocator,
    memmap::MmapOptions,
    println,
    syscall::{self, MmapFlags, MmapProt, OpenFlags},
};
pub use alloc::{
    fmt::Write,
    format,
    string::{String, ToString},
    vec::Vec,
};

And now we can print the contents of some file:

// in `crates/minipak/src/main.rs`

#[no_mangle]
unsafe fn pre_main(_stack_top: *mut u8) {
    init_allocator();
    main().unwrap();
    syscall::exit(0);
}

fn main() -> Result<(), EncoreError> {
    let file = File::open("/etc/lsb-release")?;
    let map = file.map()?;

    let s = core::str::from_utf8(&map[..]).unwrap();
    for l in s.lines() {
        println!("> {}", l);
    }

    Ok(())
}

$ cargo run --quiet --bin minipak
> DISTRIB_DESCRIPTION="Arch Linux"
> DISTRIB_RELEASE=rolling
> DISTRIB_ID=Arch
> LSB_VERSION=1.4

Hurray!

Cool, cool, but... we haven't done anything with executables in this article.

And yet it takes about an hour to read from end to end! Let this be your lesson for the day bear: things that don't feel like work, but are necessary for other work to happen, are still work.

And we did damn good work. It'll serve us well in the future.

Grhmph. I still would've liked to load an ELF object. Just one.

Fine, you want to load an executable? Here you go:

// in `crates/minipak/src/main.rs`

fn main() -> Result<(), EncoreError> {
    let file = File::open("/lib64/ld-linux-x86-64.so.2")?;
    let map = file.map()?;

    let there_you_go = core::str::from_utf8(&map[1..4]).unwrap();
    println!("{}", there_you_go);

    Ok(())
}

$ cargo run --quiet --bin minipak
ELF

There 😇

Grahhhhhhhhhhh that's not what I m-

Goodbye!

You're reading the Making our own executable packer series.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

Comment on /r/fasterthanlime

(JavaScript is required to see this. Or maybe my stuff broke)

Here's another article just for you:

Proc macro support in rust-analyzer for nightly rustc versions

Jul 28, 2022

39 min #rust · #rust-analyzer

I don't mean to complain. Doing software engineering for a living is a situation of extreme privilege. But there's something to be said about how alienating it can be at times.

Once, just once, I want to be able to answer someone's "what are you working on?" question with "see that house? it wasn't there last year. I built that".