This article is part 1 of the series itch v25 postmortem:
It’s a bit daunting to write a postmortem for the v25 release of the itch.io desktop app.
In part because, it being mostly a reliability/performance update, it’s not press-friendly. But also, because there’s so much to share!
Before I jump into specific subjects, I want to throw a bunch of numbers at you so you can get a better sense of the scope of the project.
I joined the itch.io team in early 2015, and was soon a full-time member. I worked on payments code, and features like co-op bundles.
Installing games manually always seemed to me like one of itch.io’s pain points. So when I discovered Leaf had been playing around with the idea of a desktop app, I jumped on board.
I kept working on the app, learning a ton as I went on. It was a wonderful time. Not everybody hated Electron yet. I had no users, so I was blissfully unaware of many bugs. So many bugs.
That year, Christmas came early: on December 14, I released v0.8.1 to the public. It was clearly still a beta, but it shipped with:
- A .dmg and .zip for macOS, with self-update (Squirrel.Mac)
- An installer for Windows, with self-update (Squirrel.Windows)
- A .deb package and an .rpm package
I had to buy my own code signing certificate to release it. We didn’t see any reason to not make the app open-source, so I got a cheap Certum PL:
The login screen was charming. I sort of miss it.
That’s all I can show you from the first public build, because it can’t log in anymore! 2
After that, the app grew organically for 3 years. I discovered many things along the way. First, I discovered in how many shapes and size game builds come: zip files are my ideal scenario, but then there’s the .tar family, the .rar syndicate, the .7z diehards, the .dmg priests, the church of InnoSetup, and many, many more.
I discovered that even the most basic assumptions don’t hold if you get enough users.
For example, a popular StackOverflow response to fixing problems with SourceTree
recommended changing a Windows registry value that somehow disabled stdout. The
message I was seeing was
EHANGUP. I don’t think anybody signed up for this in
the history of ever.
I discovered that some people are running Windows versions that lack basic utilities
%windir%. I discovered that some people apparently still own and operate PowerPC macs.
But most importantly, I discovered I knew nothing. I had built compilers, threw
together games for Ludum Dare, studied Computer Science for a bunch of years, led
R&D in a music start-up, but nothing could prepare me for the itch desktop app.
Despite all of that, features were added and bugs got fixed, painstakingly. We launched the itch.io refinery toolset. I launched a translation effort, and thanks to over a hundred contributors, itch is now available in 30+ languages.
I rewrote the entire app in TypeScript. I went through all possible React dropdown components. I adopted and abandoned webpack probably around five times throughout the duration of the project.
I added a lot of polish for macOS, and then a bunch of jerks kicked my backpack in a train station and my MacBookPro died. I continued testing for macOS by renting virtual machines, then physical MacBooks in the US.
I made friends at Google when we noticed that uploads had gotten really unreliable for European customers - resulting in a 91-comment whopper of a Github issue.
About seven thousand commits separate the first public build of itch, and itch v25.
Deep into the v23 release cycle, it was painful to think about any change at all. Despite constant cleanups, the codebase was a monster too terrible to tame. Things worked, except when they didn’t. And when they didn’t, finding out what exactly happened was far from guaranteed.
v23 derived its power from multiple external tools - file, unarchiver, icacls, etc. But they were all afterthoughts. Band-aids for really annoying bugs or scapegoats for features I really wanted to add. There was nothing holistic about the design, if we generously assume there was a design.
And yet, v23 has been downloaded over 2.5M times (counting updates, mostly for Windows). I was seeing a steady stream of about 5000 daily game installs using the app - keeping in mind that it was only modestly showcased on the website.
I had a really good experience developing butler though… After eating tons of Java and C for so long, it was almost refreshing. Sure, it had its flaws (and still does), but the mandatory simplicity forced me to concentrate on the core problems, and write something concurrent and correct.
butler was already used by v23 for downloads themselves. It brought better proxy support, better resumable download support, transparent integrity checking, just a more robust HTTP(S) stack in general. But extraction was still shelled out to unarchiver (or 7-zip, depending on the versions. or hdiutil, for DMGs).
So I started fantasizing about butler driving the entire install process. From downloading, to installing, to configuring, then launching, updating, and uninstalling a game. Things could be so clean. It would be designed from the ground up to be interruptible at any moment - whether the big bad OOM decided we’d be their next victim, or whether a thunderstorm hit the wrong pole and knocked out electricity for the whole neighborhood.
Of course, the rabbithole went much deeper than I expected. As it turns out, golang is very good at packing and unpacking zip file variants - better than 7-zip. So I started using it for that too. Then I decided it was silly to keep .zip files on disk when, I could just feed bytes read from the network directly into the zip decompressor. Then I realized if we did that, we’d lose the ability to pause & resume an install. So I went to work on saving the decompressor’s state, with Jesus’s help.
It all snowballed from there. Golang is pretty good at tar files too. Hey, we can make C bindings, doesn’t 7-zip come as a library too? Oh, it has this weird COM interface, but nothing one, two, three layers of abstractions can’t fix.
What about patches? They’re compressed with brotli - but they could sure use that same level of pause/resume support. Oh, someone wrote a pure Golang brotli decompressor ? I’m sure that’s hackable.
Now butler handles installation, updates, configuration, launch, and uninstallation. It’s graduated from “that command-line tool you call with arguments” to “a JSON-RPC daemon with TCP and HTTP transports”. It’s deployed using itself, straight to itch.io, and the app knows how to install it and keep it up-to-date, and restart it if it crashes, and, and…
…and why does it handle everything related to files, but not the local database? The one we store games, and collections, and uploads, and your profile info, so that search works offline and everything is fast by default? Why shouldn’t it talk directly to SQLite? And handle login, and all other API calls?
That was a few more months of work. And before I even knew it, I had separated all the business logic (butlerd) from the user interface (itch). As of v25, it’s possible to develop a completely different user interface with the same, robust and fast installation/configuration/launch engine. Multiple interfaces can even share the same library.
But there were still dirty corners. The popular Golang SQLite binding wasn’t so great. David Crawshaw’s take on it seemed much better. But no ORMs worked with it. But that’s okay, because popular ORMs try to support everything, and end up doing it poorly. We just need a subset of those features, but we need them to be strict. We need mass persistence, and a query builder. That’s what we need.
Oh, I forgot about the sandbox. Executing icacls.exe is dirty. It tends to fail. Can’t we use the Win32 APIs directly? Sure we can! Let’s port that over. And split it into its own project, with its own tests. In fact, let’s split as much functionality as we can into separate projects, with colorful names like boar, and savior, and hades, and smaug, and pelican and wizardry.
“It’s releasing soon”, I promise. How long has it been - 11 months? That much? Well, it still looks the same. Nobody’s going to even want to try it. Plus, v23’s interface looks amateur. And it’s hard to hack on, because I tried to follow Redux so closely. What if the renderer made requests to butlerd directly? What if we could see those requests in the Chrome devtools? What if we changed the entire structure of the app’s state so it would be easier to develop new components?
What if we added multi-window support last-minute? What if we got rid of tabs, because many users found them confusing? What if I picked up that crazy itch-setup project I started months ago - we’re already breaking everything, it sorta feels like now or never. And that way, everything is distributed via itch.io. The app, butler, its installer, all the prerequisites, everything. Everybody gets CDN speeds, bye GitHub’s Amazon S3 buckets.
What’s our migration path? Why am I tired all the time? Oh no, we have to stub out the old update server don’t we. The beta has been running much better than the “stable” v23 for months now. Fuck it. It’s time. Let’s release. It’s happening. I can’t believe it’s happening. This never happens. Big rewrites never end well.
It’s out. People are downloading it. Only a handful of issues trickle in. They’re easy fixes. The system works. I’m so tired. I’m waiting for a big bad issue to come in. It never comes. How am I going to write about all these changes. 2400 COMMITS? And that’s just for the interface.
“What’s next?”, Spencer asks for the blog. “A long fucking vacation, that’s what’s next.”
- Leaf wasn’t a huge react component.. I mean, proponent at the time. But when he saw that I was still bugging him about it one year in, he finally tried it and started liking it for real. I’m not that much of an excited puppy anymore, now that I have to maintain production stuff as well, but hey, I’ll take credit for that scout. [return]
- Our API is generally stable, but we’ve had to make a few breaking changes to the login endpoints, courtesy of evil internet actors everywhere. I’ll spare you the details. [return]
If you liked this article, please support my work on Patreon!