Image decay as a service

Jul 1, 2020 33 minute read rust · web · tide · warp

The tide is rising (at its own pace)
Something something warp speed.
Closing words

Since I write a lot of articles about Rust, I tend to get a lot of questions about specific crates: "Amos, what do you think of oauth2-simd? Is it better than openid-sse4? I think the latter has a lot of boilerplate."

And most of the time, I'm not sure what to responds. There's a lot of crates out there. I could probably review one crate a day until I retire!

Now, I personally think having so many crates available is a good th...

Cool bear's hot tip

Shhhhhh. Drop it and move on.

O..kay then.

Well, I recently relaunched my website as a completely custom-made web server on top of tide. And a week later, mostly out of curiosity (but not exclusively), I ported it over to warp.

So these I can review. And let's do so now.

Cool bear's hot tip

This article is from July of 2020! The landscape has changed a bit since.

If you insist on following it anyway, you can refer to the source code for this project.

The tide is rising (at its own pace)

I'll start with tide, as it's my personal favorite.

We'll build a small web app with it.

Cool bear's hot tip

Before proceeding - you're going to want a recent version of Rust.

If you're picking it up again after some time, make sure to run rustup update or equivalent, so that you have at least rustc 1.44.1.

Also, the samples in this article are run on Linux.

Shell session

$ cargo new more-jpeg
     Created binary (application) `more-jpeg` package
$ cd more-jpeg
$ cargo add tide
      Adding tide v0.11.0 to dependencies

Now, the thing with Rust http servers, is that you're not choosing a single crate. A single decision will determine a lot of the other crates you depend on.

You should know that there efforts to bridge that gap are underway, and there's often solutions you can use to pick your favorites from either ecosystem.

Your mileage may vary. I had no problem using tokio::sync::broadcast inside my original tide-powered app. However, I wasn't able to use reqwest. This is a known issue, and I expect it'll be fixed over time.

More than anything else, I'm interested in showing you both approaches, and talk about their respective strengths. Think of it as two very good restaurants. The chefs may have their own take on a lot of things, but either way, you're getting a delicious meal.

So!

On tide's side, we're going to go with async-std, just like the README recommends:

Shell session

$ cargo add async-std

Actually, we'll also want to opt into the attributes feature of async-std, so let's edit our Cargo.toml a bit:

TOML markup

[dependencies]
tide = "0.11.0"
async-std = { version = "1.6.2", features = ["attributes"] }

Thanks to that, we can declare our main function as async:

Rust code

#[async_std::main]
async fn main() {
    println!("Hello from async rust! (well, sort of)");
}

Shell session

$ cargo run --quiet
Hello from async rust! (well, sort of)

Of course we're not actually doing any asynchronous work yet.

But we could!

Rust code

use async_std::{fs::File, io::prelude::*};
use std::{collections::hash_map::DefaultHasher, error::Error, hash::Hasher};

#[async_std::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let path = "./target/debug/more-jpeg";
    let mut f = File::open(path).await?;
    let mut hasher = DefaultHasher::new();
    let mut buf = vec![0u8; 1024];

    loop {
        match f.read(&mut buf).await? {
            0 => break,
            n => hasher.write(&buf[..n]),
        }
    }
    println!("{}: {:08x}", path, hasher.finish());

    Ok(())
}

Shell session

$ cargo run --quiet
./target/debug/more-jpeg: b0d272206e97a665

Cool bear's hot tip

Two things not to do in this code sample:

Don't use DefaultHasher - the internal algorithm is not specified. It was used here to avoid adding a dependency just for a digression.
Don't use a 1KiB buffer. Also, in some cases, async_std::fs::read is a better idea.

Okay. Cool! That's not a web server though.

Let's serve up some text:

Rust code

use std::error::Error;

#[async_std::main]
async fn main() -> Result<(), Box<dyn Error>> {
    // Make a new tide app
    let mut app = tide::new();
    // Handle the `/` route.
    // Note that async closures are still unstable - this
    // is a regular closure with an async block in it.
    // The argument we're discarding (with `_`) is the request.
    app.at("/").get(|_| async { Ok("Hello from tide") });
    // The argument to `listen` is an `impl ToSocketAddrs`,
    // but it's async-std's `ToSocketAddrs`, so it accepts strings:
    app.listen("localhost:3000").await?;
    Ok(())
}

Shell session

$ cargo run --quiet &
[1] 464865
$ curl http://localhost:3000
Hello from tide% 
$ kill %1
[1]  + 464865 terminated  cargo run --quiet

Cool bear's hot tip

The final % is not a typo - it's just zsh's way of saying the command's output did not finish with a new line - but it inserted one anyway, otherwise our command prompt would be misaligned.

The prompt shown here is starship, by the way.

Text is cool and all, but how about some HTML?

Let's get fancy immediately and use liquid for templating:

HTML

<!-- in `templates/index.html.liquid` -->

<!DOCTYPE html>
<html lang="en">
    <head>
        <title>More JPEG!</title>
    </head>

    <body>
        <p>Hello from <em>Tide</em>.</p>
    </body>
</html>

Rust code

// in `src/main.rs`

use async_std::fs::read_to_string;
use liquid::Object;
use std::error::Error;
use tide::{Response, StatusCode};

#[async_std::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let mut app = tide::new();
    app.at("/").get(|_| async {
        let path = "./templates/index.html.liquid";
        let source = read_to_string(path).await.unwrap();
        let compiler = liquid::ParserBuilder::with_stdlib().build().unwrap();
        let template = compiler.parse(&source).unwrap();
        let globals: Object = Default::default();
        let markup = template.render(&globals).unwrap();
        let mut res = Response::new(StatusCode::Ok);
        res.set_body(markup);
        Ok(res)
    });
    app.listen("localhost:3000").await?;
    Ok(())
}

This code isn't good:

We're reading and compiling the template for every request
We unwrap() a lot - our app will panic if anything goes wrong

...but let's try it anyway.

We forgot something! In order to get a browser to render HTML, we have to set the content-type header:

Rust code

// new: `Mime` import
use tide::{http::Mime, Response, StatusCode};
// new: `FromStr` import
use std::{error::Error, str::FromStr};

#[async_std::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let mut app = tide::new();
    app.at("/").get(|_| async {
        // omitted: everything up until `let markup`
        let mut res = Response::new(StatusCode::Ok);
        res.set_content_type(Mime::from_str("text/html; charset=utf-8").unwrap());
        res.set_body(markup);
        Ok(res)
    });
    app.listen("localhost:3000").await?;
    Ok(())
}

That should be better

And it is!

Now let's look at how our server behaves:

Shell session

$ curl http://localhost:3000
<!DOCTYPE html>
<html lang="en">
    <head>
        <title>More JPEG!</title>
    </head>

    <body>
        <p>Hello from <em>Tide</em>.</p>
    </body>
</html>%

So far so good.

Shell session

$ curl -I http://localhost:3000
HTTP/1.1 200 OK
content-length: 162
date: Wed, 01 Jul 2020 11:45:25 GMT
content-type: text/html;charset=utf-8

Seems okay.

Shell session

$ curl -X HEAD http://localhost:3000
Warning: Setting custom HTTP method to HEAD with -X/--request may not work the 
Warning: way you want. Consider using -I/--head instead.
<!DOCTYPE html>
<html lang="en">
    <head>
        <title>More JPEG!</title>
    </head>

    <body>
        <p>Hello from <em>Tide</em>.</p>
    </body>
</html>%

Woops, that's wrong! For http HEAD requests, a server should set the content-length header, but it "must not" actually send the body.

This is a bug in async-h1, and there's already a fix in the works.

Shell session

$ curl -v -d 'a=b' http://localhost:3000
*   Trying ::1:3000...
* Connected to localhost (::1) port 3000 (#0)
> POST / HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.70.0
> Accept: */*
> Content-Length: 3
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 3 out of 3 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 405 Method Not Allowed
< content-length: 0
< date: Wed, 01 Jul 2020 11:47:28 GMT
< 
* Connection #0 to host localhost left intact

That is correct. We specified a GET handler, and it refuses to reply to POST.

Finally, let's request a route that doesn't exist:

Shell session

$ curl -v http://localhost:3000/nope
*   Trying ::1:3000...
* Connected to localhost (::1) port 3000 (#0)
> GET /nope HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.70.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< content-length: 0
< date: Wed, 01 Jul 2020 11:58:37 GMT
< 
* Connection #0 to host localhost left intact

Wonderful.

Let's look at this line:

Rust code

        let mut res = Response::new(StatusCode::Ok);

Here we used a variant from the StatusCode enum - but we don't have to, we could also just use 200, and it would work, because Response::new takes an <S: TryInto<StatusCode>>.

Now this line:

Rust code

        res.set_content_type(Mime::from_str("text/html; charset=utf-8").unwrap());

Response::set_content_type takes an impl Into<Mime>. Before setting the content-type header, it'll check that the value we're setting it to is a valid mime type.

This is actually one of the few places I'm going to allow an unwrap() - you really should never be sending invalid mime types.

Okay - what about error handling?

Cool bear's hot tip

What about it?

Well, we don't actually want our server to crash. We want it to gracefully reply with an HTTP 500.

Let's try refactoring our code to make our template serving code re-usable:

Rust code

use async_std::fs::read_to_string;
use liquid::Object;
use std::{error::Error, str::FromStr};
use tide::{http::Mime, Response, StatusCode};

#[async_std::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let mut app = tide::new();
    app.at("/").get(|_| async {
        let path = "./templates/index.html.liquid";
        serve_template(path).await
    });
    app.listen("localhost:3000").await?;
    Ok(())
}

async fn serve_template(path: &str) -> Result<Response, Box<dyn Error>> {
    let source = read_to_string(path).await?;
    let compiler = liquid::ParserBuilder::with_stdlib().build()?;
    let template = compiler.parse(&source)?;
    let globals: Object = Default::default();
    let markup = template.render(&globals)?;
    let mut res = Response::new(StatusCode::Ok);
    res.set_content_type(Mime::from_str("text/html; charset=utf-8").unwrap());
    res.set_body(markup);
    Ok(res)
}

Shell session

$ cargo check
    Checking more-jpeg v0.1.0 (/home/amos/ftl/more-jpeg)
error[E0271]: type mismatch resolving `<impl std::future::Future as std::future::Future>::Output == std::result::Result<_, http_types::error::Error>`
 --> src/main.rs:9:17
  |
9 |     app.at("/").get(|_| async {
  |                 ^^^ expected struct `std::boxed::Box`, found struct `http_types::error::Error`
  |
  = note: expected enum `std::result::Result<tide::response::Response, std::boxed::Box<dyn std::error::Error>>`
             found enum `std::result::Result<_, http_types::error::Error>`
  = note: required because of the requirements on the impl of `tide::endpoint::Endpoint<()>` for `[closure@src/main.rs:9:21: 12:6]`

Ah, that doesn't compile.

It looks like squints returning a Result is correct, but the Error type is wrong. Luckily, tide ship with its own Error type, so we can just map our error to it.

Let's even add a little bit of logging:

Shell session

$ cargo add log
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding log v0.4.8 to dependencies
$ cargo add pretty_env_logger
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding pretty_env_logger v0.4.0 to dependencies

That way, we can log the error on the server, without showing visitors sensitive information:

Rust code

#[async_std::main]
async fn main() -> Result<(), Box<dyn Error>> {
    if std::env::var_os("RUST_LOG").is_none() {
        std::env::set_var("RUST_LOG", "info");
    }
    pretty_env_logger::init();

    let mut app = tide::new();
    app.at("/").get(|_| async {
        log::info!("Serving /");
        let path = "./templates/index.html.liquid-notfound";
        serve_template(path).await.map_err(|e| {
            log::error!("While serving template: {}", e);
            tide::Error::from_str(
                StatusCode::InternalServerError,
                "Something went wrong, sorry!",
            )
        })
    });
    app.listen("localhost:3000").await?;
    Ok(())
}

I'm not super fond of the empty body there - Firefox just display a blank page, whereas Chromium shows its own 500 page. But we could always have our own error handling middleware! It's fixable.

Next up - what would we do if we wanted to parse templates at server startup? Instead of doing it on every request?

Shell session

$ cargo add thiserror
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding thiserror v1.0.20 to dependencies

First, let's make a function to compile a bunch of templates:

Rust code

// new: `Template` import
use liquid::{Object, Template};
// new: `HashMap` import
use std::{error::Error, str::FromStr, collections::HashMap};

pub type TemplateMap = HashMap<String, Template>;

#[derive(Debug, thiserror::Error)]
enum TemplateError {
    #[error("invalid template path: {0}")]
    InvalidTemplatePath(String),
}

async fn compile_templates(paths: &[&str]) -> Result<TemplateMap, Box<dyn Error>> {
    let compiler = liquid::ParserBuilder::with_stdlib().build()?;

    let mut map = TemplateMap::new();
    for path in paths {
        let name = path
            .split('/')
            .last()
            .map(|name| name.trim_end_matches(".liquid"))
            .ok_or_else(|| TemplateError::InvalidTemplatePath(path.to_string()))?;
        let source = read_to_string(path).await?;
        let template = compiler.parse(&source)?;
        map.insert(name.to_string(), template);
    }
    Ok(map)
}

Next up, we can call it from main:

Rust code

async fn main() -> Result<(), Box<dyn Error>> {
    // (cut)

    let templates = compile_templates(&["./templates/index.html.liquid"]).await?;
    log::info!("{} templates compiled", templates.len());

    // etc.
}

This works well enough:

     Running `target/debug/more-jpeg`
 INFO  more_jpeg > 1 templates compiled

But how do we use it from our handler? First we'll want to change our serve_template function:

Rust code

#[derive(Debug, thiserror::Error)]
enum TemplateError {
    #[error("invalid template path: {0}")]
    InvalidTemplatePath(String),
    // new
    #[error("template not found: {0}")]
    TemplateNotFound(String),
}

async fn serve_template(templates: &TemplateMap, name: &str) -> Result<Response, Box<dyn Error>> {
    let template = templates
        .get(name)
        .ok_or_else(|| TemplateError::TemplateNotFound(name.to_string()))?;
    let globals: Object = Default::default();
    let markup = template.render(&globals)?;
    let mut res = Response::new(StatusCode::Ok);
    res.set_content_type(Mime::from_str("text/html; charset=utf-8").unwrap());
    res.set_body(markup);
    Ok(res)
}

And adjust our route handler accordingly:

Rust code

    app.at("/").get(|_| async {
        log::info!("Serving /");
        let name = "index.html";
        serve_template(&templates, name).await.map_err(|e| {
            log::error!("While serving template: {}", e);
            tide::Error::from_str(
                StatusCode::InternalServerError,
                "Something went wrong, sorry!",
            )
        })
    });

Right?

Shell session

$ cargo check
    Checking more-jpeg v0.1.0 (/home/amos/ftl/more-jpeg)
error[E0373]: closure may outlive the current function, but it borrows `templates`, which is owned by the current function
  --> src/main.rs:17:21
   |
17 |     app.at("/").get(|_| async {
   |                     ^^^ may outlive borrowed value `templates`
...
20 |         serve_template(&templates, name).await.map_err(|e| {
   |                         --------- `templates` is borrowed here
   |
note: function requires argument type to outlive `'static`
  --> src/main.rs:17:5
   |
17 | /     app.at("/").get(|_| async {
18 | |         log::info!("Serving /");
19 | |         let name = "index.html";
20 | |         serve_template(&templates, name).await.map_err(|e| {
...  |
26 | |         })
27 | |     });
   | |______^
help: to force the closure to take ownership of `templates` (and any other referenced variables), use the `move` keyword
   |
17 |     app.at("/").get(move |_| async {
   |                     ^^^^^^^^

Okay, uh, let's try move, if you say so rustc:

Rust code

    app.at("/").get(move |_| async {
        // etc.
    });

Shell session

$ cargo check
    Checking more-jpeg v0.1.0 (/home/amos/ftl/more-jpeg)
error: lifetime may not live long enough
  --> src/main.rs:17:30
   |
17 |       app.at("/").get(move |_| async {
   |  _____________________--------_^
   | |                     |      |
   | |                     |      return type of closure is impl std::future::Future
   | |                     lifetime `'1` represents this closure's body
18 | |         log::info!("Serving /");
19 | |         let name = "index.html";
20 | |         serve_template(&templates, name).await.map_err(|e| {
...  |
26 | |         })
27 | |     });
   | |_____^ returning this value requires that `'1` must outlive `'2`
   |
   = note: closure implements `Fn`, so references to captured variables can't escape the closure

Mhhhhh not quite.

What if we try to move move elsewhere? To our async block?

Rust code

    app.at("/").get(|_| async move {
        // etc.
    });

Shell session

$ cargo check
    Checking more-jpeg v0.1.0 (/home/amos/ftl/more-jpeg)
error[E0525]: expected a closure that implements the `Fn` trait, but this closure only implements `FnOnce`
  --> src/main.rs:17:21
   |
17 |       app.at("/").get(|_| async move {
   |  _________________---_^^^^^^^^^^^^^^_-
   | |                 |   |
   | |                 |   this closure implements `FnOnce`, not `Fn`
   | |                 the requirement to implement `Fn` derives from here
18 | |         log::info!("Serving /");
19 | |         let name = "index.html";
20 | |         serve_template(&templates, name).await.map_err(|e| {
...  |
26 | |         })
27 | |     });
   | |_____- closure is `FnOnce` because it moves the variable `templates` out of its environment

Different text, same wall. This was definitely the biggest problem I encountered when first doing web development in Rust.

The problem is as follows:

Our handler captures a reference to templates
But our handler may live forever!
Whereas templates dies at the end of main

In practice, we await the Future returned by app.listen(), so this isn't a problem as far as I can tell. That's just one of those cases where the borrow checker knows less than we do. It happens!

tide has a solution for that, though - you can simply put some state in your application.

Rust code

// new: `Request` import
use tide::{http::Mime, Request, Response, StatusCode};

struct State {
    templates: TemplateMap,
}

#[async_std::main]
async fn main() -> Result<(), Box<dyn Error>> {
    // cut

    let mut app = tide::with_state(State { templates });
    app.listen("localhost:3000").await?;
        app.at("/").get(|req: Request<State>| async {
        log::info!("Serving /");
        let name = "index.html";
        serve_template(&req.state().templates, name)
            .await
            .map_err(|e| {
                // etc.
            })
    });
    Ok(())
}

That way, the application owns the State instance, and it hands out (counted) reference to it. Internally, it uses Arc, but that implementation detail is hidden.

The above almost compiles:

Shell session

$ cargo check
    Checking more-jpeg v0.1.0 (/home/amos/ftl/more-jpeg)
error[E0373]: async block may outlive the current function, but it borrows `req`, which is owned by the current function
  --> src/main.rs:21:49
   |
21 |       app.at("/").get(|req: Request<State>| async {
   |  _________________________________________________^
22 | |         log::info!("Serving /");
23 | |         let name = "index.html";
24 | |         serve_template(&req.state().templates, name)
   | |                         --- `req` is borrowed here
...  |
32 | |             })
33 | |     });
   | |_____^ may outlive borrowed value `req`
   |
note: async block is returned here
  --> src/main.rs:21:43
   |
21 |       app.at("/").get(|req: Request<State>| async {
   |  ___________________________________________^
22 | |         log::info!("Serving /");
23 | |         let name = "index.html";
24 | |         serve_template(&req.state().templates, name)
...  |
32 | |             })
33 | |     });
   | |_____^
help: to force the async block to take ownership of `req` (and any other referenced variables), use the `move` keyword
   |
21 |     app.at("/").get(|req: Request<State>| async move {
22 |         log::info!("Serving /");
23 |         let name = "index.html";
24 |         serve_template(&req.state().templates, name)
25 |             .await
26 |             .map_err(|e| {
 ...

And this time, rustc has the right idea. Since we have an async block within a closure, we want that async block to take ownership of the closure's arguments - so that it may live forever.

With that fix, everything compiles and run just as it did before.

Now let's try to make our app do something useful!

HTML

<!-- in `templates/index.html.liquid` -->

<!DOCTYPE html>
<html lang="en">
    <head>
        <title>More JPEG!</title>
        <link href="https://fonts.googleapis.com/css2?family=Indie+Flower&display=swap" rel="stylesheet">
        <link href="/style.css" rel="stylesheet">
        <script src="/main.js"></script>
    </head>

    <body>
        <p>You can always use more JPEG.</p>

        <div id="drop-zone">
            Drop an image on me!
        </div>
    </body>
</html>

css

// in `templates/style.css.liquid`

body {
    max-width: 960px;
    margin: 20px auto;
    font-size: 1.8rem;
    padding: 2rem;
}

* {
    font-family: 'Indie Flower', cursive;
}

#drop-zone {
    width: 100%;
    height: 400px;
    border: 4px dashed #ccc;
    border-radius: 1em;

    display: flex;
    justify-content: center;
    align-items: center;
}

#drop-zone.over {
    border-color: #93b8ff;
    background: #f0f4fb;
}

.result {
    width: 100%;
    height: auto;
}

JavaScript code

// in `templates/main.js.liquid`

// @ts-check
"use strict";

(function () {
    document.addEventListener("DOMContentLoaded", () => {
        /** @type {HTMLDivElement} */
        let dropZone = document.querySelector("#drop-zone");
        dropZone.addEventListener("dragover", (ev) => {
            ev.preventDefault();
            ev.dataTransfer.dropEffect = "move";
            dropZone.classList.add("over");
        });
        dropZone.addEventListener("dragleave", (ev) => {
            dropZone.classList.remove("over");
        });
        dropZone.addEventListener("drop", (ev) => {
            ev.preventDefault();
            dropZone.classList.remove("over");

            if (ev.dataTransfer.items && ev.dataTransfer.items.length > 0) {
                let item = ev.dataTransfer.items[0].getAsFile();
                console.log("dropped file ", item.name);

                fetch("/upload", {
                    method: "post",
                    body: item,
                }).then((res) => {
                    if (res.status !== 200) {
                        throw new Error(`HTTP ${res.status}`);
                    }

                    return res.json();
                }).then((payload) => {
                    /** @type {HTMLImageElement} */
                    var img = document.createElement("img");
                    img.src = payload.src;
                    img.classList.add("result");
                    dropZone.replaceWith(img);
                }).catch((e) => {
                    alert(`Something went wrong!\n\n${e}`);
                });
            }
        });
        console.log("drop zone", dropZone);
    });
})();

We'll need to add those templates to our TemplateMap:

Rust code

    let templates = compile_templates(&[
        "./templates/index.html.liquid",
        "./templates/style.css.liquid",
        "./templates/main.js.liquid",
    ])
    .await?;

And also adjust our serve_template helper to take a Mime!

Rust code

async fn serve_template(
    templates: &TemplateMap,
    name: &str,
    mime: Mime,
) -> Result<Response, Box<dyn Error>> {
    // (cut)
    res.set_content_type(mime);
    res.set_body(markup);
    Ok(res)
}

Next up, we'll make ourselves a nice little collection of mime types:

Rust code

// still in `src/main.rs`

mod mimes {
    use std::str::FromStr;
    use tide::http::Mime;

    pub(crate) fn html() -> Mime {
        Mime::from_str("text/html; charset=utf-8").unwrap()
    }

    pub(crate) fn css() -> Mime {
        Mime::from_str("text/css; charset=utf-8").unwrap()
    }

    pub(crate) fn js() -> Mime {
        Mime::from_str("text/javascript; charset=utf-8").unwrap()
    }
}

And then we can adjust our route accordingly:

Rust code

        serve_template(&req.state().templates, "index.html", mimes::html())
            .await
            .map_err(|e| {
                log::error!("While serving template: {}", e);
                tide::Error::from_str(
                    StatusCode::InternalServerError,
                    "Something went wrong, sorry!",
                )
            })

But we're going to have three more routes, and, you know the rule.

That error mapping logic is a bit lengthy, so let's simplify it a bit:

Rust code

trait ForTide {
    fn for_tide(self) -> Result<tide::Response, tide::Error>;
}

impl ForTide for Result<tide::Response, Box<dyn Error>> {
    fn for_tide(self) -> Result<Response, tide::Error> {
        self.map_err(|e| {
            log::error!("While serving template: {}", e);
            tide::Error::from_str(
                StatusCode::InternalServerError,
                "Something went wrong, sorry!",
            )
        })
    }
}

And now, our handlers can be nice and tidy:

Rust code

    let mut app = tide::with_state(State { templates });

    app.at("/").get(|req: Request<State>| async move {
        serve_template(&req.state().templates, "index.html", mimes::html())
            .await
            .for_tide()
    });

    app.at("/style.css").get(|req: Request<State>| async move {
        serve_template(&req.state().templates, "style.css", mimes::css())
            .await
            .for_tide()
    });

    app.at("/main.js").get(|req: Request<State>| async move {
        serve_template(&req.state().templates, "main.js", mimes::js())
            .await
            .for_tide()
    });

    app.listen("localhost:3000").await?;

Let's try it out!

Wonderful!

Dropping an image doesn't work - yet:

But we can make it do something pretty easily.

Let's start with reading the whole body the browser sends us, encoding it with base64, and sending it back as a Data URL.

Shell session

$ cargo add base64
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding base64 v0.12.3 to dependencies

We're also going to need to be able to format a JSON response, and I can think of two crates that will work just fine:

Shell session

$ cargo add serde
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding serde v1.0.114 to dependencies
$ cargo add serde_json
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding serde_json v1.0.56 to dependencies

Now. We'll need a struct to determine the shape of our JSON response:

Rust code

// in `src/main.rs`

use serde::Serialize;

#[derive(Serialize)]
struct UploadResponse<'a> {
    src: &'a str,
}

And then we're good to go:

Rust code

    app.at("/upload")
        .post(|mut req: Request<State>| async move {
            let body = req.body_bytes().await?;
            let payload = base64::encode(body);
            let src = format!("data:image/jpeg;base64,{}", payload);

            let mut res = Response::new(StatusCode::Ok);
            res.set_content_type(tide::http::mime::JSON);
            res.set_body(tide::Body::from_json(&UploadResponse { src: &src })?);
            Ok(res)
        });

It works!

It's a bit sluggish, but that's only because the resulting data URL is a whopping 8430091 bytes. We'll need to take care of that.

For now though - what we'd like to do is:

Accept an image of any (popular) format
Compress as a JPEG
Send it back

Let's try it:

Shell session

$ cargo add image
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding image v0.23.6 to dependencies

Rust code

    app.at("/upload")
        .post(|mut req: Request<State>| async move {
            let body = req.body_bytes().await?;
            let img = image::load_from_memory(&body[..])?;
            let mut output: Vec<u8> = Default::default();
            let mut encoder = image::jpeg::JPEGEncoder::new_with_quality(&mut output, 90);
            encoder.encode_image(&img)?;

            let payload = base64::encode(output);
            let src = format!("data:image/jpeg;base64,{}", payload);

            let mut res = Response::new(StatusCode::Ok);
            res.set_content_type(tide::http::mime::JSON);
            res.set_body(tide::Body::from_json(&UploadResponse { src: &src })?);
            Ok(res)
        });

Wonderful! Everything still works.

Now, returning the image as a base64 URL isn't great.

So let's do something slightly better:

Shell session

$ cargo add ulid
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding ulid v0.4.0 to dependencies

Rust code

// new import: `RwLock`
use async_std::{fs::read_to_string, sync::RwLock};
// new import
use ulid::Ulid;

struct Image {
    mime: Mime,
    contents: Vec<u8>,
}

struct State {
    templates: TemplateMap,
    // new:
    images: RwLock<HashMap<Ulid, Image>>,
}

#[async_std::main]
async fn main() -> Result<(), Box<dyn Error>> {
    // (cut)

    let state = State {
        templates,
        images: Default::default(),
    };
    let mut app = tide::with_state(state);

    // omitted: other routes

    // new!
    app.at("/upload")
        .post(|mut req: Request<State>| async move {
            let body = req.body_bytes().await?;

            let img = image::load_from_memory(&body[..])?;
            let mut output: Vec<u8> = Default::default();
            use image::jpeg::JPEGEncoder;
            let mut encoder = JPEGEncoder::new_with_quality(&mut output, 90);
            encoder.encode_image(&img)?;

            let id = Ulid::new();
            let src = format!("/images/{}.jpg", id);

            let img = Image {
                mime: tide::http::mime::JPEG,
                contents: output,
            };
            {
                let mut images = req.state().images.write().await;
                // TODO: expire those images at some point. Right now we have
                // an unbounded cache. Seeing as this service is bound to become
                // hugely popular, this seems ill-advised.
                images.insert(id, img);
            }

            let mut res = Response::new(StatusCode::Ok);
            res.set_content_type(tide::http::mime::JSON);
            res.set_body(tide::Body::from_json(&UploadResponse { src: &src })?);
            Ok(res)
        });

    // also new!
    app.at("/images/:name")
        .get(|req: Request<State>| async { serve_image(req).await.for_tide() });

    app.listen("localhost:3000").await?;
    Ok(())
}

async fn serve_image(req: Request<State>) -> Result<Response, Box<dyn Error>> {
    let id: Ulid = req.param("name").map_err(|_| ImageError::InvalidID)?;

    let images = req.state().images.read().await;
    if let Some(img) = images.get(&id) {
        let mut res = Response::new(200);
        res.set_content_type(img.mime.clone());
        res.set_body(&img.contents[..]);
        Ok(res)
    } else {
        Ok(Response::new(StatusCode::NotFound))
    }
}

Now, if you're following along at home (or at work!) you may notice that our application is a tad slow for now. We're going to do two things to fix it.

First, even in development, we can ask cargo to build our dependencies with optimizations.

TOML markup

# in `Cargo.toml`

[profile.dev.package."*"]
opt-level = 2

The next cargo build is going to take a little while, as all the dependencies are compiled with optimizations for the first time (for this project), so here's cool bear's thought of the day:

Cool bear's hot tip

Have you ever wondered about the trend of YouTube videos sponsored by VPN providers? Isn't it kinda strange?

There's no way the ad spend is worth it on face value, right? The conversion rates can't be that high, and they have to pay content creators enough for them to make a custom segment in which they pretty openly admit that, yeah, they could use that money.

So, are they just bleeding VC money into advertisement to show some growth? Or do they have another incentive?

Someone should look into that. THE BEARS WANT ANSWERS.

Wait, the bears? How many of you are there exactl...oh look it compiled.

Cool bear's hot tip

About 1.2 million, but the sun bears never want to fill out the census so I guess we'll never know for sure.

Now it only takes a second at most between the time I drop the file on the zone, and I see it again.

It's time for the finishing touches. Our app is called more JPEG, but we're using 90% quality. Also, we have the image stay the same exact orientation, so eventually the encoder will settle on a "midpoint" and stop degrading quality.

Let's fix that.

Shell session

$ cargo add rand
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding rand v0.7.3 to dependencie

Rust code

// new imports
use image::{imageops::FilterType, jpeg::JPEGEncoder, DynamicImage, GenericImageView};
use rand::Rng;

pub const JPEG_QUALITY: u8 = 25;

trait BitCrush: Sized {
    type Error;

    fn bitcrush(self) -> Result<Self, Self::Error>;
}

impl BitCrush for DynamicImage {
    type Error = image::ImageError;

    // Content warning: this method is gruesome
    fn bitcrush(self) -> Result<Self, Self::Error> {
        let mut current = self;
        let (orig_w, orig_h) = current.dimensions();

        // So, as it turns out, if you just decode and re-eencode
        // an image as JPEG repeatedly, nothing very interesting happens.
        // So, we *help* the artifacts surface *just a little bit*
        let mut rng = rand::thread_rng();
        let (temp_w, temp_h) = (
            rng.gen_range(orig_w / 2, orig_w * 2),
            rng.gen_range(orig_h / 2, orig_h * 2),
        );

        let mut out: Vec<u8> = Default::default();
        for _ in 0..2 {
            current = current
                // nearest neighbor because why not?
                .resize_exact(temp_w, temp_h, FilterType::Nearest)
                // that'll throw the JPEG encoder off the scent...
                // (also that's why we do it twice)
                .rotate180()
                // and so will that
                .huerotate(180);
            out.clear();
            {
                // changing the quality level helps a lot with surfacing fun artifacts
                let mut encoder =
                    JPEGEncoder::new_with_quality(&mut out, rng.gen_range(10, 30));
                encoder.encode_image(&current)?;
            }
            current = image::load_from_memory_with_format(&out[..], image::ImageFormat::Jpeg)?
                .resize_exact(orig_w, orig_h, FilterType::Nearest);
        }
        Ok(current)
    }
}

In our /upload handler, we just have to call BitCrush::bitcrush(), which does everything we want to:

Rust code

    app.at("/upload")
        .post(|mut req: Request<State>| async move {
            let body = req.body_bytes().await?;
            let img = image::load_from_memory(&body[..])?.bitcrush()?;
            let mut output: Vec<u8> = Default::default();
            let mut encoder = JPEGEncoder::new_with_quality(&mut output, JPEG_QUALITY);
            encoder.encode_image(&img)?;

            // etc.
        });

Finally, let's make some client-side changes, so we can make the whole process more user-friendly.

We'll change the HTML:

HTML

<!-- in `templates/index.html.liquid` -->

<!DOCTYPE html>
<html lang="en">
    <head>
        <title>More JPEG!</title>
        <link href="https://fonts.googleapis.com/css2?family=Indie+Flower&display=swap" rel="stylesheet">
        <link href="/style.css" rel="stylesheet">
        <script src="/main.js"></script>
    </head>

    <body>
        <p id="status">You can always use more JPEG.</p>
        <p>
            <label>
                Auto click:
                <input id="autoclick" type="checkbox"/>
            </label>
        </p>

        <div id="drop-zone">
            <span id="instructions">
                Drop an image on me!
            </span>
        </div>
    </body>
</html>

The CSS:

css

body {
    max-width: 960px;
    margin: 0 auto;
    font-size: 1.8rem;
    padding: 2rem;
}

* {
    font-family: 'Indie Flower', cursive;
}

#drop-zone {
    position: relative;

    width: 100%;
    min-height: 200px;
    border: 4px dashed #ccc;
    border-radius: 1em;

    display: flex;
    justify-content: center;
    align-items: center;
}

#drop-zone.over {
    border-color: #93b8ff;
    background: #f0f4fb;
}

#drop-zone .spinner {
    position: absolute;
    left: 1em;
    top: 1em;

    z-index: 2;

    width: 40px;
    height: 40px;
    border-radius: 100%;

    border: 4px dashed #333;
    border-radius: 50%;

    animation: spin 4s linear infinite;
}

@keyframes spin {
    0% {
        transform: rotateZ(0deg);
    }
    100% {
        transform: rotateZ(360deg);
    }
}

#drop-zone img {
    cursor: pointer;

    width: auto;
    height: auto;
    max-height: 80vh;
}

And sprinkle a tiny bit more JavaScript:

JavaScript code

// @ts-check
"use strict";

(function () {
  document.addEventListener("DOMContentLoaded", () => {
    /** @type {HTMLDivElement} */
    let dropZone = document.querySelector("#drop-zone");
    /** @type {HTMLParagraphElement} */
    let status = document.querySelector("#status");
    /** @type {HTMLInputElement} */
    let autoclick = document.querySelector("#autoclick");
    /** @type {HTMLSpanElement} */
    let instructions = document.querySelector("#instructions");
    let spinner = document.createElement("div");
    spinner.classList.add("spinner");

    /**
     * @param {Error} e
     */
    let showErrorDialog = (e) => {
      alert(`Something went wrong!\n\n${e}\n\n${e.stack}`);
    };

    autoclick.addEventListener("change", (ev) => {
      if (autoclick.checked) {
        let img = dropZone.querySelector("img");
        if (img) {
          img.click();
        }
      }
    })

    /** @param {BodyInit} body */
    let bitcrush = (body) => {
      dropZone.appendChild(spinner);

      fetch("/upload", {
        method: "post",
        body,
      })
        .then((res) => {
          if (res.status !== 200) {
            throw new Error(`HTTP ${res.status}`);
          }

          return res.json();
        })
        .then((payload) => {
          /** @type {HTMLImageElement} */
          var img = document.createElement("img");
          img.src = payload.src;
          img.addEventListener("load", () => {
            img.decode().then(() => {
              img.addEventListener("click", onImageClick);
              status.innerText = "Click image to add more JPEG";

              dropZone.innerHTML = "";
              dropZone.appendChild(img);

              if (autoclick.checked) {
                img.click();
              }
            });
          });
        })
        .catch(showErrorDialog);
    };

    /**
     * @param {MouseEvent} ev
     */
    let onImageClick = (ev) => {
      /** @type {HTMLImageElement} */
      // @ts-ignore
      let img = ev.currentTarget;
      if (img.tagName.toLowerCase() !== "img") {
        return;
      }

      console.log("src is", img.src);
      fetch(img.src)
        .then((body) => body.blob())
        .then(bitcrush)
        .catch(showErrorDialog);
    };

    dropZone.addEventListener("dragover", (ev) => {
      ev.preventDefault();
      ev.dataTransfer.dropEffect = "move";
      dropZone.classList.add("over");
    });

    dropZone.addEventListener("dragleave", () => {
      dropZone.classList.remove("over");
    });

    dropZone.addEventListener("drop", (ev) => {
      ev.preventDefault();
      dropZone.classList.remove("over");
      instructions.remove();

      if (ev.dataTransfer.items && ev.dataTransfer.items.length > 0) {
        let item = ev.dataTransfer.items[0].getAsFile();
        bitcrush(item);
      }
    });
  });
})();

I wasn't sure what picture to use for testing. I didn't want to use Lenna, because she's straight out of Playboy and also, overdone.

So instead - fruit!

Okay! That's enough fun for one day. If you want to watch it go for longer than one minute, or on non-fruit images, feel free to use your local copy - or go back in time and follow along.

Cool bear's hot tip

You're not going to run a public instance?

You're a guest here, cool bear. Things can change.

Cool bear's hot tip

Alright, sheesh.

Let's move on to our second framework.

Something something warp speed.

I said picking a web framework usually means picking a collection of crates.

Our first collection was:

Our next collection will be:

Here's what we're going to do: we're gonna move all the non-tide-specific functionality over to a new module, and then we'll have both a tide_server and a warp_server module.

Sounds good? Good. Let's go.

Let's start by adding tokio:

TOML markup

# in `Cargo.toml`

tokio = { version = "0.2.21", features = ["sync", "rt-core", "rt-util", "rt-threaded", "macros", "fs"] }

And change up our main function signature:

Rust code

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    // cut
}

At this point, our app still works! Wonderful.

We can switch to tokio's fs facilities:

Rust code

// removed import: `read_to_string`
use async_std::sync::RwLock;
// new import: `read_to_string`
use tokio::fs::read_to_string;

Everything still works the same. This is going to be easier than I thought!

The same goes for locks:

Rust code

// removed import: `async_std`
use tokio::{fs::read_to_string, sync::RwLock};

And the function signatures are also compatible.

We can now remove async-std:

Shell session

$ cargo rm async-std
    Removing async-std from dependencies

It's definitely still pulled in by tide, but we'll get to that later.

Most of our main is still good - up until the creation of the State instance.

But we now want to be creating a warp app instead.

Shell session

$ cargo add warp
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding warp v0.2.3 to dependencies

Rust code

// new imports
use std::net::SocketAddr;
use warp::Filter;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    // (cut)

    let state = State {
        templates,
        images: Default::default(),
    };

    let index = warp::path::end().map(|| "Hello from warp!");
    let addr: SocketAddr = "127.0.0.1:3000".parse()?;
    warp::serve(index).run(addr).await;
    Ok(())
}

Shell session

$ curl -v http://localhost:3000
*   Trying ::1:3000...
* connect to ::1 port 3000 failed: Connection refused
*   Trying 127.0.0.1:3000...
* Connected to localhost (127.0.0.1) port 3000 (#0)
> GET / HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.70.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: text/plain; charset=utf-8
< content-length: 16
< date: Wed, 01 Jul 2020 21:09:23 GMT
< 
* Connection #0 to host localhost left intact
Hello from warp!%

Okay, neat, we can serve stuff!

The code probably deserves some explanations though. Coming from tide, I was thoroughly confused by warp's approach for quite some time.

Whereas tide lets you build up an app - adding handlers one by one, that can all access the server state (read-only), and have full access to the request object, warp would really like it if you used filters.

For everything.

And I do mean everything.

Want to match a route? That's a filter. Only accept get requests? That's a filter. Matching paths? Filter. Want to read the request body? You want a filter. Need to get some headers maybe? Filter, again. Hell, even the handlers themselves are filters.

It certainly was a new way to think about web applications for me.

Did I like it? Well, enough to ship it in production, so, let's go.

So, right now our, uh, route, has two filters:

A path filter, that only matches /
A Map filter, that returns an impl Reply

So, for example, GET-ing /hello will 404:

Shell session

$ curl -f http://localhost:3000/hello
curl: (22) The requested URL returned error: 404 Not Found

But POST-ing to / will work just fine:

Shell session

$ curl -f -d 'cool=bear' http://localhost:3000/
Hello from warp!%

If we want to only accept GET requests, we need to add another filter. More to the point, we need to and it with our current filter. Preferably before the map, since it's a precondition.

Rust code

    let index = warp::path::end().and(warp::filters::method::get()).map(|| "Hello from warp!");

And now POST-ing returns 405 - as it should!

$ curl -f -d 'cool=bear' http://localhost:3000/
curl: (22) The requested URL returned error: 405 Method Not Allowed

Progress!

Returning a &'static str is not that exciting, to be honest. Let's try serving up some HTML.

Shell session

$ cargo add http
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding http v0.2.1 to dependencies

Rust code

    let index = warp::path::end()
        .and(warp::filters::method::get())
        .map(|| http::Response::builder().body("I do <em>not</em> miss XHTML."));

Whereas tide has a mutable Response type, http uses the builder pattern. All the methods take self and return Self, so you have to either chain them, or sprinkle a copious amount of let bindings.

I recommend just chaining them.

Woops, forgot the content type.

Unfortunately, http doesn't have a Mime equivalent, so we'll have to make do:

Rust code

    let index = warp::path::end().and(warp::filters::method::get()).map(|| {
        http::Response::builder()
            .header("content-type", "text/html; charset=utf-8")
            .body("I do <em>not</em> miss XHTML.")
    });

Perfect.

Cool bear's hot tip

Huhhhhhhh

No, you know what? Not perfect. I like the Mime type. Let's extend warp a bit.

Cool bear's hot tip

Yeah, also fix your markup maybe.

Shhh stand back bear, I'm doing traits.

Rust code

trait MimeAware {
    // calling this one `content_type` instead of `set_content_type`
    // to stick with warp conventions.
    fn content_type(self, mime: Mime) -> Self;
}

impl MimeAware for http::response::Builder {
    fn content_type(self, mime: Mime) -> Self {
        self.header("content-type", mime.to_string())
    }
}

Rust code

    let index = warp::path::end().and(warp::filters::method::get()).map(|| {
        http::Response::builder()
            .content_type(mimes::html())
            .body("<html><body><p>I do <em>not</em> miss XHTML.</p></body></html>")
    });

Cool bear's hot tip

Theeeere you go. Are you sure you're okay to write? Not getting tired?

Nonsense! This is one of my short articles.

Cool bear's hot tip

Ah, yes.

Let's keep moving. There's one important question we haven't answered yet: how do we access the state?

Well, let's try - what happens if we do this?

Rust code

    let index = warp::path::end().and(warp::filters::method::get()).map(|| {
        let template = state.templates.get("index.html").unwrap();
        let globals: Object = Default::default();
        let markup = template.render(&globals).unwrap();

        http::Response::builder()
            .content_type(mimes::html())
            .body(markup)
    });

Shell session

$ cargo check                                                                                                                      
    Checking more-jpeg v0.1.0 (/home/amos/ftl/more-jpeg)                                                                           
error[E0373]: closure may outlive the current function, but it borrows `state`, which is owned by the current function
   --> src/main.rs:97:73                                                                                                           
    |                    
97  |     let index = warp::path::end().and(warp::filters::method::get()).map(|| {                                                 
    |                                                                         ^^ may outlive borrowed value `state`
98  |         let template = state.templates.get("index.html").unwrap();
    |                        ----- `state` is borrowed here                                                                        
    |                                                            
note: function requires argument type to outlive `'static`                                                                         
   --> src/main.rs:97:17                                         
    |                                                                                                                              
97  |       let index = warp::path::end().and(warp::filters::method::get()).map(|| {
    |  _________________^                                                                                                          
98  | |         let template = state.templates.get("index.html").unwrap();                                                         
99  | |         let globals: Object = Default::default();
100 | |         let markup = template.render(&globals).unwrap();                                                                   
...   |
104 | |             .body(markup)                                                                                                  
105 | |     });                                                                                                                    
    | |______^
help: to force the closure to take ownership of `state` (and any other referenced variables), use the `move` keyword
    |
97  |     let index = warp::path::end().and(warp::filters::method::get()).map(move || {
    |                                                                         ^^^^^^^

Oh, rustc.

You sweet, sweet summer child.

Sure, okay, let's give it a go.

Rust code

    let index = warp::path::end()
        .and(warp::filters::method::get())
        .map(move || {
            // etc.
        });

Shell session

$ cargo check                                                    
    Checking more-jpeg v0.1.0 (/home/amos/ftl/more-jpeg)                                                                          
error[E0277]: the trait bound `State: std::clone::Clone` is not satisfied in `[closure@src/main.rs:99:14: 107:10 state:State]`    
   --> src/main.rs:99:10                                         
    |                           
99  |           .map(move || {                                                                                                    
    |  __________^^^_-                                           
    | |          |                                               
    | |          within `[closure@src/main.rs:99:14: 107:10 state:State]`, the trait `std::clone::Clone` is not implemented for `State`                           
100 | |             let template = state.templates.get("index.html").unwrap();                                                    
101 | |             let globals: Object = Default::default();                                                                     
102 | |             let markup = template.render(&globals).unwrap();                                                              ...   |                                                                                                                           106 | |                 .body(markup)                            
107 | |         });             
    | |_________- within this `[closure@src/main.rs:99:14: 107:10 state:State]`                                                   
    |                           
    = note: required because it appears within the type `[closure@src/main.rs:99:14: 107:10 state:State]`

Well that didn't work. But the error is interesting - very interesting.

Cool bear's hot tip

Yes, I too love walls.

No cool bear, you don't get it - it's not complaining that our function is FnOnce. That's the error we got with tide.

Cool bear's hot tip

Yeah, because state is now moved into our closure - so it can only be called once, hence FnOnce.

So what?

So, it's complaining we're not Clone. Which means warp's Map filters can be FnOnce, as long as they're clonable.

Cool bear's hot tip

...but it's not. It's not clonable.

Well, not right now, but we can definitely fix that - the same way tide does, internally - with an Arc.

Rust code

// new import
use std::sync::Arc;

Rust code

    let state = State {
        templates,
        images: Default::default(),
    };
    let state = Arc::new(state);

    let index = warp::path::end()
        .and(warp::filters::method::get())
        .map(move || {
            let template = state.templates.get("index.html").unwrap();
            let globals: Object = Default::default();
            let markup = template.render(&globals).unwrap();

            http::Response::builder()
                .content_type(mimes::html())
                .body(markup)
        });

Cool bear's hot tip

Ohhhhhh.

And cloning an Arc is cheap, right? Because it's only adding one to the reference counter, not cloning the actual data?

Yeah. Well, not as cheap as cloning an Rc. But in the lands of async, you are either Send or not at all.

Okay, now we're getting somewhere! We're missing some routes though, let's add /style.css, for example:

Rust code

    let style = warp::path!("style.css")
        .and(warp::filters::method::get())
        .map(move || {
            let template = state.templates.get("style.css").unwrap();
            let globals: Object = Default::default();
            let markup = template.render(&globals).unwrap();

            http::Response::builder()
                .content_type(mimes::css())
                .body(markup)
        });

$ cargo check
    Checking more-jpeg v0.1.0 (/home/amos/ftl/more-jpeg)
warning: unused variable: `style`
   --> src/main.rs:110:9
    |
110 |     let style = warp::path!("style.css")
    |         ^^^^^ help: if this is intentional, prefix it with an underscore: `_style`
    |
    = note: `#[warn(unused_variables)]` on by default

error[E0382]: use of moved value: `state`
   --> src/main.rs:112:14
    |
96  |     let state = Arc::new(state);
    |         ----- move occurs because `state` has type `std::sync::Arc<State>`, which does not implement the `Copy` trait
...
100 |         .map(move || {
    |              ------- value moved into closure here
101 |             let template = state.templates.get("index.html").unwrap();
    |                            ----- variable moved due to use in closure
...
112 |         .map(move || {
    |              ^^^^^^^ value used here after move
113 |             let template = state.templates.get("style.css").unwrap();
    |                            ----- use occurs due to use in closure

Cool bear's hot tip

Okay, okay okay okay.

I have several questions.

I'm all ears bear, go ahead.

Cool bear's hot tip

First off: how do you plan on serving both routes.

If I'm not mistaken, serve takes a single Filter, and it's taking index right now:

Rust code

warp::serve(index).run(addr).await;

Well, seeing how everything else works in warp, I'm assuming there's an easy way to combine them.

Like, let me pick a page at random in the docs and... there. There's an or() method.

That way, if one route fails, it tries the next one. And they probably get filtered out pretty quickly, too, since it first matches on stuff like the path, and the method, and whatnot.

Rust code

    warp::serve(index.or(style)).run(addr).await;

Cool bear's hot tip

Good.

Next question: how.. how are you going to get out of that "value used after move"?

Oh I can think of a couple way. This isn't my first lifetime rodeo.

If it moves into the closure. And we can clone it. Then let's just move clones into it.

Rust code

    let index = {
        let state = state.clone();
        warp::path::end()
            .and(warp::filters::method::get())
            .map(move || {
                // etc.
            })
    };

    let style = {
        let state = state.clone();
        warp::path!("style.css")
            .and(warp::filters::method::get())
            .map(move || {
                // etc.
            })
    };

Cool bear's hot tip

sigh

I can't believe that worked.

Well, I can! Because it builds. And if it builds, it's good to ship.

Cool bear's hot tip

I don't know, it doesn't seem very "idiomatic" for warp.

You're absolutely correct. You know what is warp-y? A filter.

Rust code

    let with_state = warp::filters::any::any().map(move || state.clone());

    let index = warp::filters::method::get()
        .and(warp::path::end())
        .and(with_state.clone())
        .map(|state: Arc<State>| {
            // omitted
        });

    let style = warp::filters::method::get()
        .and(warp::path!("style.css"))
        .and(with_state.clone())
        .map(|state: Arc<State>| {
            // omitted
        });

Cool bear's hot tip

And that works too?

Okay... still feels weird having to call those clone() by hand.

And that's why every warp filter is.. a function!

Rust code

    let with_state = {
        let filter = warp::filters::any::any().map(move || state.clone());
        move || filter.clone()
    };

    let index = warp::filters::method::get()
        .and(warp::path::end())
        .and(with_state())
        .map(|state: Arc<State>| {
            // omitted
        });

Cool bear's hot tip

Now we're talking!

That seems warp-y.

Very warpy. Much combinators. Just don't make any errors. Oh, nevermind, you do like walls. I don't.

So, moving on - we're about to write a third route that serves a template, so we need to think about having a serve_template function again.

Let's not worry too much about error handling for now:

Rust code

async fn serve_template(state: &State, name: &str, mime: Mime) -> impl warp::Reply {
    let template = state
        .templates
        .get(name)
        .ok_or_else(|| TemplateError::TemplateNotFound(name.to_string()))
        .unwrap();
    let globals: Object = Default::default();
    let markup = template.render(&globals).unwrap();

    http::Response::builder().content_type(mime).body(markup)
}

Rust code

    let index = warp::filters::method::get()
        .and(warp::path::end())
        .and(with_state())
        .map(|state: Arc<State>| async move {
            serve_template(&state, "index.html", mimes::html()).await
        });

Wall incoming!

$ cargo check
    Checking more-jpeg v0.1.0 (/home/amos/ftl/more-jpeg)
error[E0277]: the trait bound `impl core::future::future::Future: warp::reply::Reply` is not satisfied
   --> src/main.rs:119:17
    |
119 |     warp::serve(index.or(style)).run(addr).await;
    |                 ^^^^^^^^^^^^^^^ the trait `warp::reply::Reply` is not implemented for `impl core::future::future::Future`
    | 
   ::: /home/amos/.cargo/registry/src/github.com-1ecc6299db9ec823/warp-0.2.3/src/server.rs:25:17
    |
25  |     F::Extract: Reply,
    |                 ----- required by this bound in `warp::server::serve`
    |
    = note: required because of the requirements on the impl of `warp::reply::Reply` for `(impl core::future::future::Future,)`
    = note: required because of the requirements on the impl of `warp::reply::Reply` for `warp::generic::Either<(impl core::future::future::Future,), (impl core::future::future::Future,)>`
    = note: required because of the requirements on the impl of `warp::reply::Reply` for `(warp::generic::Either<(impl core::future::future::Future,), (impl core::future::future::Future,)>,)`

Oh right! We forgot about one detail haha.

Usually, handlers are async. Just like our serve_template method.

Cool bear's hot tip

Wait, why is serve_template async? It only does synchronous work...

Ever heard about this new concept called "the needs of the story"?

Anyway - .map isn't going to work here. You know what will?

.and_then will work. There's one catch - it wants a TryFuture, not a Future, so we have to return a Result.

Rust code

    let index = warp::filters::method::get()
        .and(warp::path::end())
        .and(with_state())
        .and_then(|state: Arc<State>| async move {
            Ok(serve_template(&state, "index.html", mimes::html()).await)
        });

I'm sure rustc will have no problem with that code..

Cool bear's hot tip

I'll take "famous last words" for 500.

Shell session

$ cargo check
    Checking more-jpeg v0.1.0 (/home/amos/ftl/more-jpeg)
error[E0698]: type inside `async` block must be known in this context
   --> src/main.rs:107:13
    |
107 |             Ok(serve_template(&state, "index.html", mimes::html()).await)
    |             ^^ cannot infer type
    |
note: the type is part of the `async` block because of this `await`
   --> src/main.rs:118:5
    |
118 |     warp::serve(index.or(style)).run(addr).await;
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Ah, right - we only ever specify the Ok variant of Result, so it doesn't know what the error type would be.

There's a type for that, it's called Infallible.

Rust code

// new import
use std::convert::Infallible;

Then we simply annotate our result:

Rust code

    let index = warp::filters::method::get()
        .and(warp::path::end())
        .and(with_state())
        .and_then(|state: Arc<State>| async move {
            let res: Result<_, Infallible> =
                Ok(serve_template(&state, "index.html", mimes::html()).await);
            res
        });

And then everything works again!

But that's a little bit silly. Okay, it's very silly.

Our serve_template method can fail!

With proper error handling, it looks like this:

Rust code

async fn serve_template(
    state: &State,
    name: &str,
    mime: Mime,
) -> Result<impl warp::Reply, Box<dyn Error>> {
    let template = state
        .templates
        .get(name)
        .ok_or_else(|| TemplateError::TemplateNotFound(name.to_string()))?;
    let globals: Object = Default::default();
    let markup = template.render(&globals)?;

    Ok(http::Response::builder().content_type(mime).body(markup))
}

Cool bear's hot tip

Calling this now: there is no way that warp accepts a Box<dyn std::error::Error> as an Error type for and_then.

Shh no spoilers.

Cool bear's hot tip

I said what I said.

Now we can simplify our handlers:

Rust code

    let index = warp::filters::method::get()
        .and(warp::path::end())
        .and(with_state())
        .and_then(|state: Arc<State>| async move {
            serve_template(&state, "index.html", mimes::html()).await
        });

Shell session

$ cargo check
    Checking more-jpeg v0.1.0 (/home/amos/ftl/more-jpeg)

error[E0277]: the trait bound `std::boxed::Box<dyn std::error::Error>: warp::reject::sealed::CombineRejection<warp::reject::Rejection>` is not satisfied
   --> src/main.rs:108:10
    |
108 |         .and_then(|state: Arc<State>| async move {
    |          ^^^^^^^^ the trait `warp::reject::sealed::CombineRejection<warp::reject::Rejection>` is not implemented for `std::boxed::Box<dyn std::error::Error>`

error[E0277]: the trait bound `std::boxed::Box<dyn std::error::Error>: warp::reject::sealed::CombineRejection<warp::reject::Rejection>` is not satisfied
   --> src/main.rs:115:10
    |
115 |         .and_then(|state: Arc<State>| async move {
    |          ^^^^^^^^ the trait `warp::reject::sealed::CombineRejection<warp::reject::Rejection>` is not implemented for `std::boxed::Box<dyn std::error::Error>`

error[E0599]: no method named `or` found for struct `warp::filter::and_then::AndThen<warp::filter::and::And<warp::filter::and::And<impl warp::filter::Filter+std::marker::Copy, impl warp::filter::Filter+std::marker::Copy>, warp::filter::map::Map<impl warp::filter::Filter+std::marker::Copy, [closure@src/main.rs:101:52: 101:73 state:_]>>, [closure@src/main.rs:108:19: 110:10]>` in the current scope
   --> src/main.rs:120:23
    |
120 |     warp::serve(index.or(style)).run(addr).await;
    |                       ^^ method not found in `warp::filter::and_then::AndThen<warp::filter::and::And<warp::filter::and::And<impl warp::filter::Filter+std::marker::Copy, impl warp::filter::Filter+std::marker::Copy>, warp::filter::map::Map<impl warp::filter::Filter+std::marker::Copy, [closure@src/main.rs:101:52: 101:73 state:_]>>, [closure@src/main.rs:108:19: 110:10]>`
    | 
   ::: /home/amos/.cargo/registry/src/github.com-1ecc6299db9ec823/warp-0.2.3/src/filter/and_then.rs:12:1
    |
12  | pub struct AndThen<T, F> {
    | ------------------------
    | |
    | doesn't satisfy `_: warp::filter::FilterBase`
    | doesn't satisfy `_: warp::filter::Filter`
    |
    = note: the method `or` exists but the following trait bounds were not satisfied:
            `warp::filter::and_then::AndThen<warp::filter::and::And<warp::filter::and::And<impl warp::filter::Filter+std::marker::Copy, impl warp::filter::Filter+std::marker::Copy>, warp::filter::map::Map<impl warp::filter::Filter+std::marker::Copy, [closure@src/main.rs:101:52: 101:73 state:_]>>, [closure@src/main.rs:108:19: 110:10]>: warp::filter::FilterBase`
            which is required by `warp::filter::and_then::AndThen<warp::filter::and::And<warp::filter::and::And<impl warp::filter::Filter+std::marker::Copy, impl warp::filter::Filter+std::marker::Copy>, warp::filter::map::Map<impl warp::filter::Filter+std::marker::Copy, [closure@src/main.rs:101:52: 101:73 state:_]>>, [closure@src/main.rs:108:19: 110:10]>: warp::filter::Filter`
            `&warp::filter::and_then::AndThen<warp::filter::and::And<warp::filter::and::And<impl warp::filter::Filter+std::marker::Copy, impl warp::filter::Filter+std::marker::Copy>, warp::filter::map::Map<impl warp::filter::Filter+std::marker::Copy, [closure@src/main.rs:101:52: 101:73 state:_]>>, [closure@src/main.rs:108:19: 110:10]>: warp::filter::FilterBase`
            which is required by `&warp::filter::and_then::AndThen<warp::filter::and::And<warp::filter::and::And<impl warp::filter::Filter+std::marker::Copy, impl warp::filter::Filter+std::marker::Copy>, warp::filter::map::Map<impl warp::filter::Filter+std::marker::Copy, [closure@src/main.rs:101:52: 101:73 state:_]>>, [closure@src/main.rs:108:19: 110:10]>: warp::filter::Filter`
            `&mut warp::filter::and_then::AndThen<warp::filter::and::And<warp::filter::and::And<impl warp::filter::Filter+std::marker::Copy, impl warp::filter::Filter+std::marker::Copy>, warp::filter::map::Map<impl warp::filter::Filter+std::marker::Copy, [closure@src/main.rs:101:52: 101:73 state:_]>>, [closure@src/main.rs:108:19: 110:10]>: warp::filter::FilterBase`
            which is required by `&mut warp::filter::and_then::AndThen<warp::filter::and::And<warp::filter::and::And<impl warp::filter::Filter+std::marker::Copy, impl warp::filter::Filter+std::marker::Copy>, warp::filter::map::Map<impl warp::filter::Filter+std::marker::Copy, [closure@src/main.rs:101:52: 101:73 state:_]>>, [closure@src/main.rs:108:19: 110:10]>: warp::filter::Filter`

error: aborting due to 3 previous errors

Some errors have detailed explanations: E0277, E0599.
For more information about an error, try `rustc --explain E0277`.
error: could not compile `more-jpeg`.

To learn more, run the command again with --verbose.

Cool bear's hot tip

Caaaaaaalled it.

Nice wall, by the way.

sigh where's HR when you need it.

Okay, so, it doesn't work. But we've been down that error handling road before.

We can just make an adapter!

Rust code

trait ForWarp {
    type Reply;

    fn for_warp(self) -> Result<Self::Reply, warp::Rejection>;
}

impl<T> ForWarp for Result<T, Box<dyn Error>>
where
    T: warp::Reply + 'static,
{
    type Reply = Box<dyn warp::Reply>;

    fn for_warp(self) -> Result<Self::Reply, warp::Rejection> {
        let b: Box<dyn warp::Reply> = match self {
            Ok(reply) => Box::new(reply),
            Err(e) => {
                log::error!("Error: {}", e);
                let res = http::Response::builder()
                    .status(500)
                    .body("Something went wrong, apologies.");
                Box::new(res)
            }
        };
        Ok(b)
    }
}

Cool bear's hot tip

Uhh Amos? I watched you port your website from tide to warp and you definitely didn't do it that way.

Yeah well, it was late, and, sometimes you figure stuff out as you go along.

Cool bear's hot tip

Also, doesn't warp::reject::custom exist? Why not use it?

Because, I don't know, when I used them I got a nice log message, but empty replies. Chrome showed its own 500 page, but Firefox just showed a blank one, and that didn't seem very friendly.

Anyway.

Now, we have some happy little traits:

Rust code

    let index = warp::filters::method::get()
        .and(warp::path::end())
        .and(with_state())
        .and_then(|state: Arc<State>| async move {
            serve_template(&state, "index.html", mimes::html())
                .await
                .for_warp()
        });

    let style = warp::filters::method::get()
        .and(warp::path!("style.css"))
        .and(with_state())
        .and_then(|state: Arc<State>| async move {
            serve_template(&state, "style.css", mimes::css())
                .await
                .for_warp()
        });

    let js = warp::filters::method::get()
        .and(warp::path!("main.js"))
        .and(with_state())
        .and_then(|state: Arc<State>| async move {
            serve_template(&state, "main.js", mimes::js())
                .await
                .for_warp()
        });

    let addr: SocketAddr = "127.0.0.1:3000".parse()?;
    warp::serve(index.or(style).or(js)).run(addr).await;
    Ok(())

And we're finally done with our po...

Cool bear's hot tip

The uploads. You forgot about the uploads.

Right! The uploads! Of course.

Well, same stuff different route, really.

Shell session

$ cargo add bytes
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding bytes v0.5.5 to dependencies

Rust code

async fn handle_upload(state: &State, bytes: Bytes) -> Result<impl warp::Reply, Box<dyn Error>> {
    let img = image::load_from_memory(&bytes[..])?.bitcrush()?;
    let mut output: Vec<u8> = Default::default();
    let mut encoder = JPEGEncoder::new_with_quality(&mut output, JPEG_QUALITY);
    encoder.encode_image(&img)?;

    let id = Ulid::new();
    let src = format!("/images/{}", id);

    let img = Image {
        mime: tide::http::mime::JPEG,
        contents: output,
    };
    {
        let mut images = state.images.write().await;
        images.insert(id, img);
    }

    let payload = serde_json::to_string(&UploadResponse { src: &src })?;
    let res = http::Response::builder()
        .content_type(tide::http::mime::JSON)
        .body(payload);
    Ok(res)
}

Rust code

    let upload = warp::filters::method::post()
        .and(warp::path!("upload"))
        .and(with_state())
        .and(warp::filters::body::bytes())
        .and_then(|state: Arc<State>, bytes: Bytes| async move {
            handle_upload(&state, bytes).await.for_warp()
        });

    let addr: SocketAddr = "127.0.0.1:3000".parse()?;
    warp::serve(index.or(style).or(js).or(upload))
        .run(addr)
        .await;

Cool thing alert: there's a content_length_limit filter we could use to fight against one of the numerous ways our server could be DoS'd.

Finally, we need to serve the images again:

Rust code

async fn serve_image(state: &State, name: &str) -> Result<impl warp::Reply, Box<dyn Error>> {
    let id: Ulid = name.parse().map_err(|_| ImageError::InvalidID)?;

    let images = state.images.read().await;
    let res = if let Some(img) = images.get(&id) {
        http::Response::builder()
            .content_type(img.mime.clone())
            .body(img.contents.clone())
    } else {
        http::Response::builder()
            .status(404)
            .body("Image not found")
    };
    Ok(res)
}

Mhh... that doesn't build though:

Shell session

$ cargo check
error[E0308]: `if` and `else` have incompatible types
   --> src/main.rs:255:9
    |
250 |         let res = if let Some(img) = images.get(&id) {
    |    _______________-
251 |   |         http::Response::builder()
    |  _|_________-
252 | | |             .content_type(img.mime.clone())
253 | | |             .body(img.contents.clone())
    | |_|_______________________________________- expected because of this
254 |   |     } else {
255 | / |         http::Response::builder()
256 | | |             .status(404)
257 | | |             .body("Image not found")
    | |_|____________________________________^ expected struct `std::vec::Vec`, found `&str`
258 |   |     };
    |   |_____- `if` and `else` have incompatible types
    |
    = note: expected type `std::result::Result<http::response::Response<std::vec::Vec<u8>>, _>`
               found enum `std::result::Result<http::response::Response<&str>, _>`

Mhhhhh okay. I'm new here, so, I'm going to get out of this the cowardly way

with a Box.

But if you know better, you know, reach out. I'm sure we can work something out.

Cool bear's hot tip

I mean, you could go for an Either type.

Yeah, no, I still have nightmares of my first few weeks with tokio, pre-async/await. I'm good.

You do it.

Rust code

async fn serve_image(state: &State, name: &str) -> Result<impl warp::Reply, Box<dyn Error>> {
    let id: Ulid = name.parse().map_err(|_| ImageError::InvalidID)?;

    let images = state.images.read().await;
    let res: Box<dyn warp::Reply> = if let Some(img) = images.get(&id) {
        Box::new(
            http::Response::builder()
                .content_type(img.mime.clone())
                .body(img.contents.clone()),
        )
    } else {
        Box::new(
            http::Response::builder()
                .status(404)
                .body("Image not found"),
        )
    };
    Ok(res)
}

Now, all that's left is to set up a route for it...

Rust code

    let images = warp::filters::method::get()
        .and(warp::path!("images" / String))
        .and(with_state())
        .and_then(|name: String, state: Arc<State>| async move {
            serve_image(&state, &name).await.for_warp()
        });

    let addr: SocketAddr = "127.0.0.1:3000".parse()?;
    warp::serve(index.or(style).or(js).or(upload).or(images))
        .run(addr)
        .await;
    Ok(())

...and remove a few dependencies:

Shell session

$ cargo rm tide
    Removing tide from dependencies
$ cargo add http-types
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding http-types v2.2.1 to dependencies

Rust code

// this was imported from `tide` before:
use http_types::Mime;

mod mimes {
    // same here
    use http_types::Mime;
    use std::str::FromStr;

    pub(crate) fn html() -> Mime {
        Mime::from_str("text/html; charset=utf-8").unwrap()
    }

    pub(crate) fn css() -> Mime {
        Mime::from_str("text/css; charset=utf-8").unwrap()
    }

    pub(crate) fn js() -> Mime {
        Mime::from_str("text/javascript; charset=utf-8").unwrap()
    }

    // new, use instead of `tide::http::mime::JSON`
    pub(crate) fn json() -> Mime {
        Mime::from_str("application/json").unwrap()
    }

    // new, use instead of `tide::http::mime::JPEG` 
    pub(crate) fn jpeg() -> Mime {
        Mime::from_str("image/jpeg").unwrap()
    }
}

// remove the `ForTide` trait.

And now our port is complete.

Phew. That was a bunch of work.

I think if you got this far, you deserve another video. You know, just to show it's still working.

This time, with a picture from https://thispersondoesnotexist.com/:

Closing words

So, what's the verdict?

Who's the winner of the Rust Web Framework 2020 Jamboree?

Well, neither, really.

I like tide's types better. I like a strongly-typed Mime, I like having access to a Request, and a mutable Response. I like http-types' Cookie (it's so good).

But tide can only do http/1.1. If you want http/2, you'll need a reverse proxy (there are, of course, good Rust options for that, like sozu. Or you could just go with nginx, the devil I know).

I encounter bugs in tide and its satellite crates more often than I'd like. And that's perfectly understandable - those used to have more of an "experimental" vibe. Just, just playing with new stuff. That has changed, lately, and the change is not yet complete.

I'm intrigued by warp's "everything is a Filter" concept, but that's just not the way I think about web apps - yet. And the screen-fulls of errors you get when you do something bad are absolutely not something I want to have to deal with.

But, for now, I do.

So who wins? No one! There's good in both, and I hope I've showed in this article that you can extend whatever you want (with happy little traits), and pick off some types from the tide ecosystem if you really like them.

I hope you enjoyed reading this. I sure enjoyed writing it.

I don't feel nearly as anxious about async Rust as I did before. The errors aren't as good as sync Rust, and there's a lot of trial and error before it clicks, but it's not insurmountable.

Until next time, take care!

Discuss this article on /r/fasterthanlime

If you liked what you saw, please support my work!

Donate on GitHub Donate on Patreon

Looking for the homepage?

Another article: Rust: The wrong people are resigning

Image decay as a service

Contents

The tide is rising (at its own pace)

Mhh.

Woo!

Something something warp speed.

Closing words