Extra credit

We've achieved our goals already with this series: we have a web service written in Rust, built into a Docker image with nix, with a nice dev shell, that we can deploy to fly.io.

Generating a docker image with nix

There it is. The final instalment.

Over the course of this series, we've built a very useful Rust web service that shows us colored ASCII art cats, and we've packaged it with docker, and deployed it to https://fly.io.

Doing geo-location and keeping analytics

I sold you on some additional functionality for catscii last chapter, and we got caught up in private registry / docker shenanigans, so, now, let's resume web development as promised.

Using the Shipyard private crate registry with Docker

Wait wait wait, so we're not talking about nix yet?

Well, no! The service we have is pretty simple, and I want to complicate things a bit, to show how things would work in both the Dockerfile and the nix scenario.

Deploying catscii to fly.io

Disclosure: Because I used to work for fly.io, I still benefit from an employee discount at the time of this writing: I don't have to pay for anything deployed there for now.

Writing a Dockerfile for catscii

Now that our service is production-ready, it's time to deploy it somewhere.

There's a lot of ways to approach this: what we are going to do, though, is build a docker image. Or, I should say, an OCI image.

Serving ASCII cats over HTTP

Our catscii program does everything we want it to do, except that it's a command-line application rather than a web server. Let's fix that.

## Enter axum

Printing ASCII cats to the terminal

Now that our development environment is all set up, let's make something useful!

## Creating the catscii crate

The bottom emoji breaks rust-analyzer

Some bugs are merely fun. Others are simply delicious!

Today's pick is the latter.

## Reproducing the issue, part 1

C++ vs Rust: which is faster?

I ported some Advent of Code solutions from C/C++ to Rust, and used the opportunity to compare performance. When I couldn't explain why they performed differently, I had no choice but to disassemble both and look at what the codegen was like!

Day 18 (Advent of Code 2022)

This time around, we're porting a solution from C++ to Rust and seeing how it feels, how it performs, and what we can learn about both languages by doing that.

Day 17 (Advent of Code 2022)

Advent of Code gets harder and harder, and I'm not getting any smarter. Or any more free time. So, in order to close out this series anyway, I'm going to try and port other people's solutions from "language X" to Rust. That way, they already figured out the hard stuff, and we can just focus on the Rust bits!

Day 16 (Advent of Code 2022)

Let's tackle the day 16 puzzle!

## Parsing

Day 15 (Advent of Code 2022)

The day 15 puzzle falls into the "math puzzle" territory more than "let's learn something new about Rust", but since several folks asked if I was going to continue... let's continue.

Day 14 (Advent of Code 2022)

I like how the day 14 puzzle sounds, because I think it'll give me an opportunity to show off yet another way to have Rust embedded in a web page.

Day 13 (Advent of Code 2022)

The day 13 puzzle needs a speech therapist.

???

...because it has an awful lisp!! Ahhhahahahhhh

Are you ok? What is.. what is going on with you?

Day 12 (Advent of Code 2022)

Alright! The day 12 puzzle involves path finding, and it seems like a good time to lean more heavily on the WASM embeds I've set up for the previous parts.

Day 11 (Advent of Code 2022)

It's a new day, it's a new advent of code puzzle.

In that one, we have to apparently cosplay as an IBM mainframe and just.. crunch them numbers. This doesn't look fun, and I can't think of a clever twist to make it fun, so let's try to make it short and sweet.

## Parsing

Day 10 (Advent of Code 2022)

Onwards! To the day 10 puzzle.

I don't see a way to make part 1 especially fun — so let's just get to it.

## Parsing

Day 9 (Advent of Code 2022)

The Advent of Code is not a sprint: it's a marathon: sometimes you've got to stop and smell the roses.

I... what? That's not.. have you done a marathon before?

Day 8 (Advent of Code 2022)

In the day 8 problem, our input is a height map:

30373
25512
65332
33549
35390


This is a 5x5 grid, and every number denotes the height of a tree. For part 1, we must find out how many trees are visible from the outside of the grid.

Day 7 (Advent of Code 2022)

The day 7 challenge talks about trees! File trees that is.

The temptation to solve it before starting to write this article so I don't look silly is high, but I'm explicitly not doing so, so that we can bang our collective heads against any walls at the same time, and see how we can get out of it! Trees are serious business!

## Part 1

Day 6 (Advent of Code 2022)

Today I am joining you from the relative discomfort of my living room (since my better half has commandeered the home office due to Way Too Many Calls) to tackle the day 6 challenge, which I'm excited about: maybe despite, maybe because of, the low-grade fever I'm under.

## Part 1

Day 5 (Advent of Code 2022)

## Part 1

Day 4 (Advent of Code 2022)

## Part 1

Day 3 (Advent of Code 2022)

## Part 1

Day 2 (Advent of Code 2022)

## Part 1

Day 1 (Advent of Code 2022)

Two years ago, I did part of Advent of Code 2020 using the Rust language. It was a lot of fun, so let's try it again!

## The problem statement

Async fn in trait, for real this time

## async_trait's one weird type ascription trick

Because I started accepting donations via GitHub Sponsors, and because donating at the "Silver" tier or above gives you advance access to articles and your name in the credits, I need to interface with the GitHub API the same way I do the Patreon API.

Trying to use nix

Now that my website is deployed as a container image, I wanted to give nix a try. I'm still doing it the old-fashioned way right now: with a Dockerfile, running cargo in a "builder" image, copying stuff out of there into a slimmer image (that still has an Ubuntu base, even though distroless images are a thing now).

## But why?

Deploying at the edge

Disclosure: Although I no longer work for the company my website is hosted on, and this article is written in way that mentions neither my previous or current hosting provider: at the time of this writing, I don't pay for hosting.

Async fn in trait... not

## Async fn in trait... not

Migrating from warp to axum

## Falling out of love with warp

Cleaning up and upgrading third-party crates

## The bleeding edge of rustc and clippy

The HTTP crash course nobody asked for

HTTP does a pretty good job staying out of everyone's way.

How does the detour crate work?

We want to hook a function, so that our code gets called... but we also want the original code to execute. How the heck does that work?

Proc macro support in rust-analyzer for nightly rustc versions

I don't mean to complain. Doing software engineering for a living is a situation of extreme privilege. But there's something to be said about how alienating it can be at times.

When rustc explodes

One could say I have a bit of an obsession with build times.

I believe having a "tight feedback loop" is extremely valuable: when I work on a large codebase, I want to be able to make small incremental changes and check very often that things are going as expected.

Remote development with Rust on fly.io

Disclosure: At the time of this writing, I benefit from the fly.io "Employee Free Tier". I don't pay for side projects hosted there "within reasonable limits". The project discussed here qualifies for that.

## Why you might want a remote dev environment

The curse of strong typing

It happened when I least expected it.

Someone, somewhere (above me, presumably) made a decision. "From now on", they declared, "all our new stuff must be written in Rust".

Long story short: a couple of my articles got really popular on a bunch of sites, and someone, somewhere, went "well, let's see how much traffic that smart-ass can handle", and suddenly I was on the receiving end of a couple DDoS attacks.

Getting good at SNES games through DLL injection

Are you ever confronted with a problem and then think to yourself "wait a minute, I know how to code?" — that's exactly what happened there.

Hey Notepad! Nice process you got there. Would be a shame if someone were to... butt in.

In this video, we learn about applications and processes and threads, and use Win32 APIs to create a remote thread in another process, running into all kinds of complications on the way there.

Futures Nostalgia

Up until recently, hyper was my favorite Rust HTTP framework. It's low-level, but that gives you a lot of control over what happens.

Request coalescing in async Rust

As the popular saying goes, there are only two hard problems in computer science: caching, off-by-one errors, and getting a Rust job that isn't cryptocurrency-related.

A Rust match made in hell

I often write pieces that showcase how well Rust can work for you, and how it can let you build powerful abstractions, and prevents you from making a bunch of mistakes.

Some mistakes Rust doesn't catch

I still get excited about programming languages. But these days, it's not so much because of what they let me do, but rather what they don't let me do.

Computers as a social construct

Those who learn from history are doomed to repeat that fucking proverb.

Messing with the recipe

What if we learned just enough to be a little dangerous?

In the wake of Why is my Rust build so slow?, developers from the mold and lld linkers reached out, wondering why using their linker didn't make a big difference.

One funny way to bundle assets

There's one thing that bothers me. In part 1, why are we using hyper-staticfile? Couldn't we just use file:/// URLs?

The rest of the fucking owl

NO! No no no.

What?

WE WERE DONE!

Well... yes! But also no. We still shell out to a bunch of tools:

Shell session
\$ rg 'Command::new'
src/commands/mod.rs
126:        let variant = if let Ok(output) = run_command(Command::new("wslpath").arg("-m").arg("/")) {

src/commands/cavif.rs
29:            Command::new("cavif")

src/commands/imagemagick.rs
25:            Command::new(&self.bin)

src/commands/cwebp.rs
25:            Command::new("cwebp")

src/commands/svgo.rs
25:            Command::new("svgo")

Cool bear's hot tip
Productionizing our poppler build

I was a bit anxious about running our poppler meson build in CI, because it's the real test, you know? "Works on my machine" only goes so far, things have a tendency to break once you try to make them reproducible.

From Inkscape to poppler

What's next? Well... poppler is the library Inkscape uses to import PDFs.

Cool bear's hot tip

Yes, the name comes from Futurama.

Why is my Rust build so slow?

I've recently come back to an older project of mine (that powers this website), and as I did some maintenance work: upgrade to newer crates, upgrade to a newer rustc, I noticed that my build was taking too damn long!

My ideal Rust workflow

Writing Rust is pretty neat. But you know what's even neater? Continuously testing Rust, releasing Rust, and eventually, shipping Rust to production. And for that, we want more than plug-in for a code editor.

A terminal case of Linux

Has this ever happened to you?

You want to look at a JSON file in your terminal, so you pipe it into jq so you can look at it with colors and stuff.

Cool bear's hot tip
Understanding Rust futures by going way too deep

So! Rust futures! Easy peasy lemon squeezy. Until it's not. So let's do the easy thing, and then instead of waiting for the hard thing to sneak up on us, we'll go for it intentionally.

Cool bear's hot tip
Fine, we'll relocate our own binary!

Welcome back to the eighteenth and final part of "Making our own executable packer".

In the last article, we had a lot of fun. We already had a "packer" executable, minipak, which joined together stage1 (a launcher), and a compressed version of whichever executable we wanted to pack.

What's in the box?

Here's a sentence I find myself saying several times a week:

...or we could just box it.

Pin and suffering

I'd like to think that my understanding of "async Rust" has increased over the past year or so. I'm 100% onboard with the basic principle: I would like to handle thousands of concurrent tasks using a handful of threads. That sounds great!

Running a self-relocatable ELF from memory

Welcome back!

In the last article, we did foundational work on minipak, our ELF packer.

It is now able to receive command-line arguments, environment variables, and auxiliary vectors. It can parse those command-line arguments into a set of options. It can make an ELF file smaller using the LZ4 compression algorithm, and pack it together with stage1, our launcher.

Everything but ELF

And we're back!

In the last article, we thanked our old code and bade it adieu, for it did not spark joy. And then we made a new, solid foundation, on which we planned to actually make an executable packer.

Between libcore and libstd

You're still here! Fantastic.

I have good news, and bad news. The good news is, we're actually going to make an executable packer now!

In the bowels of glibc

Good morning, and welcome back to "how many executables can we run with our custom dynamic loader before things get really out of control".

Day 14 (Advent of Code 2020)

It's time for the Day 14 problem!

After the hassle that was Day 13, I hope this time we'll have a relatively chill time. And, at least for Part 1, that is true.

Day 13 (Advent of Code 2020)

In the Day 13 problem, we're trying to take the bus.

Our input looks like this:

939
7,13,x,x,59,x,31,19


The first line indicates the earliest minute we can leave from the bus terminal, and the second line indicates the "identifier" of the buses that are active.

Day 12 (Advent of Code 2020)

Time for the Day 12 problem!

In this problem, we have a ship. And we have navigation instructions:

• Action N means to move north by the given value.
• Action S means to move south by the given value.
• Action E means to move east by the given value.
• Action W means to move west by the given value.
• Action L means to turn left the given number of degrees.
• Action R means to turn right the given number of degrees.
• Action F means to move forward by the given value in the direction the ship is currently facing.
Day 11 (Advent of Code 2020)

Another day, another problem.

This time the problem looks suspiciously like Conway's Game of Life, or, I guess, any old Cellular automaton.

Day 10 (Advent of Code 2020)

Day, 10! Day, 10!

Okay, Day 10.

Again, the problem statement is very confusing - but what it all boils down to is this. We have a list of numbers:

16
10
15
5
1
11
7
19
6
12
4

Day 9 (Advent of Code 2020)

Day 9's problem statement is convoluted - the "ah maybe that's why I don't usually do Advent of Code" kind of convoluted, but let's give it a go anyway.

Day 8 (Advent of Code 2020)

Time for another Advent of Code 2020 problem!

That one sounds like it's going to be fun. Our input is pretty much assembly, like this:

nop +0
acc +1
jmp +4
acc +3
jmp -3
acc -99
acc +1
jmp -4
acc +6

Day 7 (Advent of Code 2020)

Another day, another Advent of Code 2020 problem.

That one seems fun! For some nerdy values of fun.

Our input is a set of rules:

light red bags contain 1 bright white bag, 2 muted yellow bags.
dark orange bags contain 3 bright white bags, 4 muted yellow bags.
bright white bags contain 1 shiny gold bag.
muted yellow bags contain 2 shiny gold bags, 9 faded blue bags.
shiny gold bags contain 1 dark olive bag, 2 vibrant plum bags.
dark olive bags contain 3 faded blue bags, 4 dotted black bags.
vibrant plum bags contain 5 faded blue bags, 6 dotted black bags.
faded blue bags contain no other bags.
dotted black bags contain no other bags.

Day 6 (Advent of Code 2020)

The end of Advent of Code 2020 is fast approaching, and we're nowhere near done. Time to do Day 6!

The problem statement here is a little contrived, as uh, as the days that came before it, but that won't stop us.

Day 5 (Advent of Code 2020)

Time for another day of Advent of Code 2020.

For Day 5, we're going to have to do...

Let me guess: more parsing?

Day 4 (Advent of Code 2020)

It's time for Day 4 of the Advent of Code 2020!

Now, I've already had a look at the problem statement, at least for part 1, and I'm not particularly excited.

Day 3 (Advent of Code 2020)

Hello all, and welcome back to Advent of Code 2020, featuring Cool Bear.

Hey y'all!

Let's get right to it.

The problem statement for Day 3 is as follows: we're given a map, that looks like this:

..##.......
#...#...#..
.#....#..#.
..#.#...#.#
.#...##..#.
..#.##.....
.#.#.#....#
.#........#
#.##...#...
#...##....#
.#..#...#.#

Day 2 (Advent of Code 2020)

Day 2, Day 2! Woo!

Basically, our input looks like this:

1-3 a: abcde
1-3 b: cdefg
2-9 c: ccccccccc

Day 1 (Advent of Code 2020)

I was not planning on doing anything specific this December, but a lot of folks around me (on Twitter, at work) have chosen this Advent of Code to pick up Rust, and I've got big FOMO energy, so, let's see where this goes.

Aiming for correctness with types

The Nature weekly journal of science was first published in 1869. And after one and a half century, it has finally completed one cycle of carcinization, by publishing an article about the Rust programming language.

So you want to live-reload Rust

Good morning! It is still 2020, and the world is literally on fire, so I guess we could all use a distraction.

Peeking inside a Rust enum

During a recent Rust Q&A Session on my twitch channel, someone asked a question that seemed simple: why are small string types, like SmartString or SmolStr, the same size as String, but small vec types, like SmallVec, are larger than Vec?

Frustrated? It's not you, it's Rust

Learning Rust is... an experience. An emotional journey. I've rarely been more frustrated than in my first few months of trying to learn Rust.

I am a Java, C#, C or C++ developer, time to do some Rust

As I've said before, I'm working on a book about lifetimes. Or maybe it's just a long series - I haven't decided the specifics yet. Like every one of my series/book things, it's long, and it starts you off way in the periphery of the subject, and takes a lot of detours to get there.

Surviving Rust async interfaces

I used to be afraid of async Rust. It's easy to get into trouble!

But thanks to the work done by the whole community, async Rust is getting easier to use every week. One project I think is doing particularly great work in this area is async-std.

I write a ton of articles about rust. And in those articles, the main focus is about writing Rust code that compiles. Once it compiles, well, we're basically in the clear! Especially if it compiles to a single executable, that's made up entirely of Rust code.

Getting in and out of trouble with Rust futures

I started experimenting with asynchronous Rust code back when futures 0.1 was all we had - before async/await. I was a Rust baby then (I'm at least a toddler now), so I quickly drowned in a sea of .and_then, .map_err and Either<A, B>.

Small strings in Rust

Hey everyone!

Image decay as a service

Since I write a lot of articles about Rust, I tend to get a lot of questions about specific crates: "Amos, what do you think of oauth2-simd? Is it better than openid-sse4? I think the latter has a lot of boilerplate."

Abstracting away correctness

I've been banging the same drum for years: APIs must be carefully designed.

This statement doesn't resonate the same way with everyone. In order to really understand what I mean by "careful API design", one has to have experienced both ends of the spectrum.

A new website for 2020

Hi everyone. Has it been two months since I last posted something? Yes it has!

That seems like a nice round duration, so let's break the silence with a few announcements.

## I have a new website

Welcome back and thanks for joining us for the reads notes... the thirteenth installment of our series on ELF files, what they are, what they can do, what does the dynamic linker do to them, and how can we do it ourselves.

A no_std Rust binary

In Part 11, we spent some time clarifying mechanisms we had previously glossed over: how variables and functions from other ELF objects were accessed at runtime.

More ELF relocations

In our last installment of "Making our own executable packer", we did some code cleanups. We got rid of a bunch of unsafe code, and found a way to represent memory-mapped data structures safely.

Safer memory-mapped structures

Welcome back to the "Making our own executable packer" series, where digressions are our bread and butter.

Last time, we implemented indirect functions in a no-libc C program. Of course, we got lost on the way and accidentally implemented a couple of useful elk-powered GDB functions - with only the minimal required amount of Python code.

GDB scripting and Indirect functions

In the last article, we cleaned up our dynamic linker a little. We even implemented the Dynamic relocation.

But it's still pretty far away from running real-world applications.

Working with strings in Rust

There's a question that always comes up when people pick up the Rust programming language: why are there two string types? Why is there String, and &str?

Dynamic symbol resolution

Let's pick up where we left off: we had just taught elk to load not only an executable, but also its dependencies, and then their dependencies as well.

A half-hour to learn Rust

In order to increase fluency in a programming language, one has to read a lot of it. But how can you read a lot of it if you don't know what it means?

Up until now, we've been loading a single ELF file, and there wasn't much structure to how we did it: everyhing just kinda happened in main, in no particular order.

The simplest shared library

In our last article, we managed to load and execute a PIE (position-independent executable) compiled from the following code:

x86 assembly
; in samples/hello-pie.asm

global _start

section .text

_start: mov rdi, 1      ; stdout fd
lea rsi, [rel msg]
mov rdx, 9      ; 8 chars + newline
mov rax, 1      ; write syscall
syscall

xor rdi, rdi    ; return code 0
mov rax, 60     ; exit syscall
syscall

section .data

msg:    db "hi there", 10

ELF relocations

The last article, Position-independent code, was a mess. But who could blame us? We looked at the world, and found it to be a chaotic and seemingly nonsensical place. So, in order to blend in, we had to let go of a little bit of sanity.

Position-independent code

In the last article, we found where code was hiding in our samples/hello executable, by disassembling the whole file and then looking for syscalls.

Running an executable without exec

In part 1, we've looked at three executables:

• sample, an assembly program that prints "hi there" using the write system call.
• entry_point, a C program that prints the address of main using printf
• The /bin/true executable, probably also a C program (because it's part of GNU coreutils), and which just exits with code 0.
What's in a Linux executable?

Executables have been fascinating to me ever since I discovered, as a kid, that they were just files. If you renamed a .exe to something else, you could open it in notepad! And if you renamed something else to a .exe, you'd get a neat error dialog.

Crafting ICMP-bearing IPv4 packets with the help of bitvec

So. Serializing IPv4 packets. Easy? Well, not exactly.

IPv4 was annoying to parse, because we had 3-bit integers, and 13-bit integers, and who knows what else. Serializing it is going to be exactly the same.

Crafting ARP packets to find a remote host's MAC address

Alright. ALRIGHT. I know, we're all excited, but let's think about what we're doing again.

So we've managed to look at real network traffic and parse it completely. We've also taken some ICMP packets, parsed them, and then serialized them right back and we got the exact same result.

Parsing and serializing ICMP packets with cookie-factory.

In the last part, we've finally parsed some IPv4 packets. We even found a way to filter only IPv4 packets that contain ICMP packets.

Parsing IPv4 packets, including numbers smaller than bytes

Hello and welcome to Part 11 of this series, wherein we finally use some of the code I prototyped way back when I was planning this series.

## Where are we standing?

Improving error handling - panics vs. proper errors

Before we move on to parsing more of our raw packets, I want to take some time to improve our error handling strategy.

Currently, the ersatz codebase contains a mix of Result<T, E>, and some methods that panic, like unwrap() and expect().

Rust 2020: Funding

Blog posts that praise Rust are many but funding is generally in short supply.

If even a small percentage of the money Rust saves companies was put back into the ecosystem it would help secure the future of the platform tremendously.

## Multiple sources of funding

Consuming Ethernet frames with the nom crate

Now that we've found the best way to find the "default network interface"... what can we do with that interface?

Binding C APIs with variable-length structs and UTF-16

Okay, I lied.

I'm deciding - right this instant - that using wmic is cheating too. Oh, it was fair game when we were learning about Windows, but we're past that now.

Finding the default network interface through WMI

Let's set aside our sup project for a while.

Don't get me wrong - it's a perfectly fine project, and, were we simply rewriting "ping" for Windows in Rust, we could (almost) stop there.

The builder pattern, and a macro that keeps FFI code DRY

Our ping API is simple, but it's also very limited:

Rust code
pub fn ping(dest: ipv4::Addr) -> Result<(), String>

// called as:

A simple ping library, parsing strings into IPv4 address

We've just spent a lot of time abstracting over LoadLibrary, but we still have all the gory details of the Win32 ICMP API straight in our main.rs file! That won't do.

Designing and implementing a safer API on top of LoadLibrary

### It's refactor time!

FFI-safe types in Rust, newtypes and MaybeUninit

It's time to make sup, our own take on ping, use the Win32 APIs to send an ICMP echo. Earlier we discovered that Windows's ping.exe used IcmpSendEcho2Ex. But for our purposes, the simpler IcmpSendEcho will do just fine.

Windows dynamic libraries, calling conventions, and transmute

So, how does ping.exe actually send a ping? It seems unrealistic that ping.exe itself implements all the protocols involved in sending a ping. So it must be calling some sort of library. Also, since it ends up talking to the outside world via a NIC (network interface controller), the kernel is probably involved at some point.

Declarative memory management

It feels like an eternity since I've started using Rust, and yet I remember vividly what it felt like to bang my head against the borrow checker for the first few times.

Reading files the hard way - Part 3 (ftrace, disk layouts, ext4)

So far, we've seen many ways to read a file from different programming languages, we've learned about syscalls, how to make those from assembly, then we've learned about memory mapping, virtual address spaces, and generally some of the mechanisms in which userland and the kernel interact.

Reading files the hard way - Part 1 (node.js, C, rust, strace)

Everybody knows how to use files. You just open up File Explorer, the Finder, or a File Manager, and bam - it's chock-full of files. There's folders and files as far as the eye can see. It's a genuine filapalooza. I have never once heard someone complain there were not enough files on their computer.

Rust modules vs files

A while back, I asked on Twitter what people found confusing in Rust, and one of the top topics was "how the module system maps to files".

Rust generics vs Java generics

In my previous article, I said I needed to stop thinking of Rust generics as Java generics, because in Rust, generic types are erased.

Recursive iterators in Rust

I've been looking for this blog post everywhere, but it doesn't exist, so I guess it's my turn to write about Some Fun with Rust.