11 min
Things that can go wrong when downloading
👋 This page was last updated ~8 years ago. Just so you know.
When I get a little bit too emotional about my current baby, the itch.io app, there's always a timely support ticket reminding me that it is currently, still a glorified game downloader.
However true that is, that doesn't mean it's easy! In the past year, I've had to account for a bunch of failure conditions that can happen, some of which I didn't realize were even possible. Let's review them, for fun!
Your computer may refuse to run butler
butler is a little helper program for the itch.io app and developers. It serves as both an uploader, and a downloader (and a differ, a patcher, and various little utility commands).
It's coded in the Go language, which is great for compatibility, because golang executables will run pretty much anywhere - and it links (almost) everything statically, so there's no dependency hell, just "whoops your exe is 17MB" hell.
There's this GitHub issue that's puzzling both the reporter and me: a Linux install that's seemingly 64-bit, but refuses to run the 64-bit build of butler. It'll run the 32-bit build fine though!
As far as I can tell, it's an isolated case. butler runs on many flavors of Linux, both 32 and 64-bit, without any issues. Maybe we'll find the answer to this riddle in the next few months, or maybe we won't! Computers are fun.
Your API key may be expired/revoked
When the itch.io app runs butler, it passes your API key, and a download key, if applicable. But as a user, you're free to revoke it any time from your user settings.
If it's revoked, API calls will fail. This not a recoverable error - the only fix is to log in again from the itch.io app and try again. It's not a particularly bizarre error condition, but it's not necessarily the first thing you think about either!
You may be using a proxy
Thankfully, that's (almost) an easy one. The itch.io app detects your system's proxy settings and passes them along to butler, which uses them.
But proxies love to break HTTP in funny ways. Especially when you have long, streaming requests. Or any kind of caching.
And when someone reports a proxy problem, it's not usually one of these open-source, easy-to-configure proxies you can just spin up locally to see what's going on. They're usually these closed-source corporate/university proxies, and then presto, you-the-developer-of-the-tool-that-doesnt-work-for-some-user are starring in an episode of Sherlock - if the writers had hired a technical consultant or ten.
DNS resolution might misbehave
Before making any API requests or downloads, we need to find the address of, respectively, the API server, and the CDN node we'll be talking to. That's where DNS comes in. If we can't find their address, even we could reach them - we just don't know who to reach, so the whole thing falls flat.
In 2017, it's easy to get solid DNS. Just surrender your eternal soul to the Google overlord (set your DNS servers to 8.8.8.8 and 8.8.4.4) and, if you're not extremely unlucky, you'll have fast, accurate DNS resolution.
But if you're just a regular Joe itching to play some games, you might inadvertently have lent your soul to your ISP instead. Or maybe you're running some Linux distribution that has dnsmasq set up by default - and then butler will see some funny error conditions.
dnsmasq for example, relays DNS requests to another DNS server. If your internet access is temporarily cut, it'll accept the request, try to relay it, wait, eventually fail and time out. That's a different failure condition than "not being able to open a connection to 8.8.8.8 because the network is down". So we have to account for that.
Your computer might think you have IPv6, but you don't
Once upon a time, it looked like all the RFC we'll ever need were already written. IPv6 was the answer to that[0]. But it turns out deployment took a little while. It's not uncommon for part of your internet route to support IPv6, while other parts don't know what to do with it.
Thankfully, there's a solution to that, and it even has a cute name: Happy Eyeballs. The gist is to attempt both IPv6 and IPv4 requests, and see what works best for us.
That's not something I personally had to deal with, but it's quirky enough that it warranted a mention in this list.
The HTTP(S) server might reject our request
As of late 2016, all itch.io downloads that aren't external uploads are served by our CDN. Aside from the fact that CDNs are, uh, a bit of an oligopoly, it's great:
- Everyone gets fast downloads, no matter their region
- It saves costs, since bandwidth is cheaper from the CDN than from the storage server
- It's a well-behaved, reliable HTTP server that supports Range requests and a bunch of other cool stuff
But there's a few games still using external uploads, which means butler might be talking to any HTTP server at all.
For example, it might be talking to Github releases, which:
- Redirects any requests to a US-based Amazon S3 bucket
- ...which is slow for anyone outside the US
- ...refuses HEAD requests (only accepts GET)
- ...and requires us to forward headers there when we're doing Range requests
It gets worse - we might be talking to a self-hosted HTTP server that not only won't scale but will also be pretty poor at HTTP.
For example, it might not set the Content-Length header, so we won't know how large the file
we're downloading is. Or it might reply to Range requests with 200, sending us the whole file every time (which means
we can't resume downloads if they're paused/stopped/abruptly shut down).
And finally, it might not be a static file server but one of those terrible we'll-make-you-wait-in-a-queue-while-you-watch-ads sites that eventually let you download at 80KiB/s, but only from a browser. There's no hope for these. The jdownloader team spent a decade writing crawlers for all these, so, use that. I won't!
Note: if you're using external uploads, please don't. Make yourself and your users a favor and upload directly to itch.io, whether using web (better), or butler (best). You can even automate uploads if you're doing CI, it generates small patchs for itch.io app users and fixes a bunch of other problems. More on that later!
The HTTP(S) server might close the connection
Even the best of HTTP servers sometimes decide a connection has been going on for too darn long and decide to
bail out of here before you stick your toothbrush in their bathroom. If they support Range requests, we can
simply pick up where we left off. If not, we're out of luck.
Your computer might close the connection
Maybe it's the full moon, maybe you just installed VirtualBox's network drivers, maybe it was something else,
I'm not here to judge - but sometimes, your OS will just close a connection. Same dance - Range support makes
for a happy butler, lack of it means we have to start over from the beginning.
Data might just stop coming in
Closing a connection isn't something to be celebrated, but at least we know something's up. Sometimes, data will
just stop flowing. To fight against that, we have timeouts - if the connection is idle for too long, we give it
up and try again with a fresh one (given - you guessed it - that we have Range support).
A bit might get flipped
TCP/IP has a bunch of measures to avoid data corruption - but given all the agents involved, once in a very rare while, a bit might get flipped. This means the server thinks it's sending the right data, but it's sending something very slightly different - and then there's a few scenarios:
- Some check will catch it (for example, the corrupted data results in an invalid zip)
- Nothing catches it, it extracts fine, and you end up with a corrupted copy of the game
The latter is even rarer because, even if by some freakish lack of luck, the resulting file can be extracted (zip headers aren't corrupted, etc.), the zip format has built-in checksums, and it'll notice on extraction that the checksum and the content doesn't match (and then you get to try again, yay!).
But maybe it corrupted something else, like a file name - so you end up with Level0[1] instead of level0, and
the game won't start.
Writing to disk may fail
So, all the network operations are going fine, we're downloading at full speed, and suddenly, writing to disk fails!
There could be a bunch of reasons:
- A network disk was disconnected (because the host went down, for example)
- An external disk was disconnected (because someone tripped on the USB/SATA cable, or you're Jesus and your SATA controller hates you)
- You don't have permissions to write to a folder
- You're trying to write to a file that's locked[2].
What you're downloading may be funny
Sometimes, new developers think an .exe can run anywhere - so they add Linux and macOS tags, and call it a day. But, of course, it won't run![3]
Sometimes, developers using löve aren't quite sure how to make macOS or Linux builds, so they just upload the .love file[4] and hope it'll run. (But of course, it won't)
Or sometimes, it is a proper Windows/Linux/macOS build, but the uploaded .zip file was compressed using some software that decided standard zips are too good for them, and shove another compression format in there, like LZMA (or cthulhu forbid, Wavpack). butler's unzipper doesn't really handle that (yet!)
Maybe extraction goes perfectly fine, but then, on macOS/Linux, your executable is missing the right permissions (maybe because you compressed it from Windows!). Luckily, the itch.io app fixes that. But, your users downloading from web will definitely be staring at a broken game, unless they know what they're doing.
Note: butler fixes permissions at upload time, solving the problem both for web users and app users!
Or maybe it's a Windows build but it needs some dependency to run, like the MSVC runtime, or .NET, or... in which case, the itch.io app can install it for your users, if you tell it to.
Closing words
These are only some of the most entertaining failure conditions that can happen when downloading something.
Thanks for reading, I hope you enjoyed the article, and I wish I could say my little sister drew the illustrations but unfortunately it was all me.
If you liked this, maybe you'll enjoy trying out the itch.io app - that's what I spend most of my time working on, either on the visible or invisible parts of it.
Until next time, take care!
I kid, I kid. IPv6 was the answer to IPv4 address exhaustion. The big promise was that every device could get its own IP. Making IoT DDoS botnets more scalable! Ahhh sorry, I'm snarking again :)
Yes, flipping a given bit of an ASCII character can turn lowercase into uppercase and vice versa. That's on purpose! Aren't text encodings great? Don't try to do that on emojis though!
In an early version of the itch.io app, Adam sent me a fun bug report. "Amos, sometimes the game runs fine, but once you quit it won't launch again." Turns out the app was updating the game while it was running. Turns out it's not such a hot idea. Linux and macOS will happily let you do that, while Windows is going to complain about you writing to a running .exe pretty hard.
And that can be solved - we can detect what kind of files an archive contains, and we could tag it automatically. I'll get around to it any day now. Not now though, I'm in the middle of writing a rant.
Fun fact, .love files are just zip files. For recent versions of love, they even include a version number
somewhere in a .lua file inside the zip. So there is a way
Here's another article just for you:
Abstracting away correctness
44 min
I've been banging the same drum for years: APIs must be carefully designed.
This statement doesn't resonate the same way with everyone. In order to really understand what I mean by "careful API design", one has to have experienced both ends of the spectrum.
But there is a silver lining - once you have experienced "good design", it's really hard to go back to the other kind. Even after acknowledging that "good design" inevitably comes at a cost, whether it's cognitive load, compile times, making hiring more challenging, etc.