Avatar for fasterthanlime fasterthanlime
Nov 15, 2022 clock icon 11 min #rust · #patreon · #github · #ftl-infra

Implementing "Log in with GitHub"

From the series Updating fasterthanli.me for 2022

1

2

3

4

5

6

7

8

👋 This page was last updated ~2 years ago. Just so you know.

Because I started accepting donations via GitHub Sponsors, and because donating at the "Silver" tier or above gives you advance access to articles and your name in the credits, I need to interface with the GitHub API the same way I do the Patreon API.

Because I'd rather rely on third-party identity providers than provide my own sign up / log in / password forgotten / 2FA flow, user identifiers on my website are simply {provider}:{provider_specific_user_id}:

$ just psql-dev
docker compose exec db psql -U user futile
psql (14.3 (Debian 14.3-1.pgdg110+1))
Type "help" for help.

futile=# select * from user_preferences limit 1;
      id       |                 data
---------------+--------------------------------------
 patreon:47556 | {"theme":"device","ligatures":false}
(1 row)

That seems pretty extensible! Sort of!

Doing OAuth 2.0 by hand, because why not

Adding GitHub login was an experience in "finding out what they did different". GitHub, for example, has several kinds of apps: it has "GitHub Apps", which can be installed onto projects/organizations, and are definitely not what I wanted. And it has "OAuth Apps", your regular OAuth 2.0 thingamaji.

Cool bear

Cool bear's hot tip

Back in the bad old days, we had to implement OAuth 1.0, which came with its own signature scheme, essentially HMAC-SHA1 of the "signature base string", which included the request method, the authority, the path and query components, the protocol parameters, and parameters included in the request entity-body.

It was very thorny, and super easy to get wrong. OAuth 2.0 in comparison is an absolute breeze, as you'll see.

The OAuth part didn't really come with any surprises. Although there are existing OAuth libraries for Rust, I have my own code for Patreon, and adjusting it for GitHub was rather easy.

/// Redirects to GitHub login URL
pub fn make_github_login_url(config: &Config) -> Result<String, GithubError> {
    let github_secrets = config.github_secrets();

    let mut u = Url::parse("https://github.com/login/oauth/authorize").unwrap();
    let mut q = u.query_pairs_mut();
    q.append_pair("response_type", "code");
    q.append_pair("client_id", &github_secrets.oauth_client_id);
    q.append_pair("redirect_uri", &github_secrets.oauth_redirect_uri);
    q.append_pair("scope", "read:user,read:org");
    drop(q);

    Ok(u.to_string())
}

Boom, done.

Well, not really — that'll let us redirect a user to the GitHub login page, but then we need to handle the callback, wherein we exchange a temporary code against an access token.

And in there, due to how everything is set up, we need to make a request to the GitHub API to fetch profile information.

And here, there's big differences.

First off... GitHub access tokens for OAuth apps don't expire? Or if they do... there's no mechanism to refresh them, unlike... every other OAuth 2.0 implementation I've had the pleasure of consuming? "Github Apps" tokens can get refreshed though. Ah well.

Second, when exchanging the code for a token, Patreon defaults to a JSON response, so this works fine:

#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct PatreonCredentials {
    /// example: "ajba90sd098w0e98f0w9e8g90a8ed098wgfae_w"
    pub access_token: String,
    /// example: "ajba90sd098w0e98f0w9e8g90a8ed098wgfae_w"
    pub refresh_token: String,
    /// example: 2678400 (31 days)
    pub expires_in: i64,
    /// example: "identity identity.memberships"
    pub scope: String,
}

/// Request an access token
pub async fn handle_patreon_oauth_callback(
    config: &Config,
    query: &Query,
) -> Result<PatreonCredentials> {
    #[derive(Deserialize)]
    pub struct QueryParams {
        pub code: String,
    }
    let params: QueryParams = query
        .deserialize()
        .map_err(|e| PatreonError::Any(format!("while deserializing: {}", e)))?;

    #[derive(Serialize)]
    struct TokenRequestBody<'a> {
        code: &'a str,
        grant_type: &'a str,
        client_id: &'a str,
        client_secret: &'a str,
        redirect_uri: &'a str,
    }

    let patreon_secrets = config.patreon_secrets();
    let tok_params = TokenRequestBody {
        code: &params.code,
        grant_type: "authorization_code",
        client_id: &patreon_secrets.oauth_client_id,
        client_secret: &patreon_secrets.oauth_client_secret,
        redirect_uri: &patreon_secrets.oauth_redirect_uri,
    };

    let client = Client::new();
    let res = client
        .post("https://patreon.com/api/oauth2/token")
        .form(&tok_params)
        .send()
        .await
        .map_err(|e| PatreonError::Any(e.to_string()))?;

    let status = res.status();
    if !status.is_success() {
        let error = res
            .text()
            .await
            .unwrap_or_else(|_| "Could not get error text".into());
        return Err(
            PatreonError::Any(format!("got HTTP {}, server said: {}", status, error)).into(),
        );
    }

    let creds: PatreonCredentials = res
        .json()
        .await
        .map_err(|e| PatreonError::Any(e.to_string()))?;

    Ok(creds)
}

But on GitHub, it defaults to url-encoding! So you have to request JSON explicitly:

#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct GithubCredentials {
    /// example: "ajba90sd098w0e98f0w9e8g90a8ed098wgfae_w"
    pub access_token: String,
    /// example: "read:user"
    pub scope: String,
    /// example: "bearer"
    pub token_type: String,
}

/// Request an access token
pub async fn handle_github_oauth_callback(
    config: &Config,
    query: &Query,
) -> Result<GithubCredentials> {
    #[derive(Deserialize)]
    pub struct QueryParams {
        pub code: String,
    }
    let params: QueryParams = query
        .deserialize()
        .map_err(|e| GithubError::Any(format!("while deserializing: {}", e)))?;

    #[derive(Serialize)]
    struct TokenRequestBody<'a> {
        code: &'a str,
        client_id: &'a str,
        client_secret: &'a str,
        redirect_uri: &'a str,
    }

    let github_secrets = config.github_secrets();
    let tok_params = TokenRequestBody {
        code: &params.code,
        client_id: &github_secrets.oauth_client_id,
        client_secret: &github_secrets.oauth_client_secret,
        redirect_uri: &github_secrets.oauth_redirect_uri,
    };

    let client = Client::new();
    let res = client
        .post("https://github.com/login/oauth/access_token")
        .form(&tok_params)
        // need to opt explicitly into JSON here
        .header(header::ACCEPT, "application/json")
        .send()
        .await
        .map_err(|e| GithubError::Any(e.to_string()))?;

    let status = res.status();
    if !status.is_success() {
        let error = res
            .text()
            .await
            .unwrap_or_else(|_| "Could not get error text".into());
        return Err(
            GithubError::Any(format!("got HTTP {}, server said: {}", status, error)).into(),
        );
    }

    // reqwest provides `res.json()` instead, but in case of deserialization
    // failure, it doesn't show you the payload. doing it in two steps makes
    // for easier troubleshooting.
    let creds = res
        .text()
        .await
        .map_err(|e| GithubError::Any(e.to_string()))?;

    let creds: GithubCredentials = serde_json::from_str(&creds)
        .map_err(|e| GithubError::Any(format!("while deserializing {creds}: {}", e)))?;

    Ok(creds)
}

Once we've got an access token, we want to save it to the database immediately, not a big problem with sqlx

    {
        let github_id = fut_creds.user_info.profile.github_id()?;
        sqlx::query(
            "
                INSERT INTO github_credentials
                (github_id, data) VALUES
                ($1, $2)
                ON CONFLICT (github_id) DO UPDATE SET data = $2
            ",
        )
        .bind(github_id)
        .bind(serde_json::to_string(&git_creds)?)
        .execute(&tr.state.users_pool)
        .await?;
    }

I am not fond of JSON:API

And then we reach another difference. Patreon's API (which seems to be in "maintenance mode" and which could disappear any day now, definitely the kind of news you want to hear about your primary income source) uses JSON:API, which essentially provides a way to normalize data in API responses (avoiding duplication):

See the example payload they give:

{
  "links": {
    "self": "http://example.com/articles",
    "next": "http://example.com/articles?page[offset]=2",
    "last": "http://example.com/articles?page[offset]=10"
  },
  "data": [{
    "type": "articles",
    "id": "1",
    "attributes": {
      "title": "JSON:API paints my bikeshed!"
    },
    "relationships": {
      "author": {
        "links": {
          "self": "http://example.com/articles/1/relationships/author",
          "related": "http://example.com/articles/1/author"
        },
        "data": { "type": "people", "id": "9" }
      },
      "comments": {
        "links": {
          "self": "http://example.com/articles/1/relationships/comments",
          "related": "http://example.com/articles/1/comments"
        },
        "data": [
          { "type": "comments", "id": "5" },
          { "type": "comments", "id": "12" }
        ]
      }
    },
    "links": {
      "self": "http://example.com/articles/1"
    }
  }],
  "included": [{
    "type": "people",
    "id": "9",
    "attributes": {
      "firstName": "Dan",
      "lastName": "Gebhardt",
      "twitter": "dgeb"
    },
    "links": {
      "self": "http://example.com/people/9"
    }
  }, {
    "type": "comments",
    "id": "5",
    "attributes": {
      "body": "First!"
    },
    "relationships": {
      "author": {
        "data": { "type": "people", "id": "2" }
      }
    },
    "links": {
      "self": "http://example.com/comments/5"
    }
  }, {
    "type": "comments",
    "id": "12",
    "attributes": {
      "body": "I like XML better"
    },
    "relationships": {
      "author": {
        "data": { "type": "people", "id": "9" }
      }
    },
    "links": {
      "self": "http://example.com/comments/12"
    }
  }]
}

The data being queried here is an article, with id "1", some attributes, and relationships. .data[0].relationships.author doesn't contain any information about the author except their type ("people") and ID (9). The actual information can be found in .included[0], where we find attributes like firstName, lastName, twitter, etc.

This is sort of a neat, standard way to do this — it makes it easy to write clients that cache some records locally. They also make it possible (and Patreon's v2 API does implement that) to request only a subset of fields, like so:

GET /articles?include=author&fields[articles]=title,body&fields[people]=name HTTP/1.1
Accept: application/vnd.api+json

I'm not sure how hard it is to take advantage of this to make the backend implementation more efficient (propagating field selection all the way to database queries, so it's not just about saving network bandwidth).

Overall though, I'm not super fond of JSON:API. See the example above for example? It's "simplified":

Note: The above example URI shows unencoded [ and ] characters simply for readability. In practice, these characters should be percent-encoded. See "Square Brackets in Parameter Names".

Much like OAuth 1.0, there's a bunch of papercuts associated to JSON:API. I was able to use an existing crate, but still ended up writing my own extensions, for ergonomics:

// in `crates/futile-patreon/src/jsonapi_ext.rs`

use eyre::Result;
use jsonapi::model::{DocumentData, Resource};
use serde::de::DeserializeOwned;

use crate::PatreonError;

pub trait BetterDoc {
    fn get_resource(&self, id: &str) -> Result<&Resource>;
}

impl BetterDoc for DocumentData {
    fn get_resource(&self, id: &str) -> Result<&Resource> {
        if let Some(resources) = self.included.as_ref() {
            if let Some(resource) = resources.iter().find(|res| res.id == id) {
                return Ok(resource);
            }
        }

        Err(PatreonError::Any(format!("Could not find resource {} in jsonapi doc", id)).into())
    }
}

pub trait BetterResource {
    fn get_single_relationship<'doc>(
        &self,
        doc: &'doc DocumentData,
        name: &str,
    ) -> Result<&'doc Resource>;

    fn get_multi_relationship<'doc>(
        &self,
        doc: &'doc DocumentData,
        name: &str,
    ) -> Result<Vec<&'doc Resource>>;

    fn get_attributes<T>(&self) -> Result<T>
    where
        T: DeserializeOwned;
}

impl BetterResource for Resource {
    fn get_single_relationship<'doc>(
        &self,
        doc: &'doc DocumentData,
        name: &str,
    ) -> Result<&'doc Resource> {
        if let Some(relationship) = self.get_relationship(name) {
            if let Ok(Some(id)) = relationship.as_id() {
                return doc.get_resource(id);
            }
        }

        Err(PatreonError::Any(format!(
            "Could not get single relationship {} for {} in jsonapi doc",
            name, self._type
        ))
        .into())
    }
    fn get_multi_relationship<'doc>(
        &self,
        doc: &'doc DocumentData,
        name: &str,
    ) -> Result<Vec<&'doc Resource>> {
        if let Some(relationship) = self.get_relationship(name) {
            if let Ok(Some(ids)) = relationship.as_ids() {
                return ids.iter().map(|id| doc.get_resource(id)).collect();
            }
        }

        Err(PatreonError::Any(format!(
            "Could not get multi relationship {} for {} in jsonapi doc",
            name, self._type
        ))
        .into())
    }

    fn get_attributes<T>(&self) -> Result<T>
    where
        T: DeserializeOwned,
    {
        let val = serde_json::to_value(&self.attributes)?;
        Ok(serde_json::from_value(val)?)
    }
}
Cool bear

Wait, does this round-trip a HashMap<String, Value> through JSON just to deserialize it again as another type that implements Deserialize?

Not quite. It builds a serde_json::Value, which is then "mapped" to the struct using serde_json. No actual JSON was harmed generated in the making of this code, it's just JSON-shaped data structures.

Even with those helpers though, this isn't pleasant code. Building the query URL isn't pleasant:

        let mut identity_url = Url::parse("https://www.patreon.com/api/oauth2/v2/identity")?;
        {
            let mut q = identity_url.query_pairs_mut();
            let include = vec![
                "memberships",
                "memberships.currently_entitled_tiers",
                "memberships.campaign",
            ]
            .join(",");
            q.append_pair("include", &include);
            q.append_pair("fields[member]", "patron_status");
            q.append_pair("fields[user]", "full_name,thumb_url");
            q.append_pair("fields[tier]", "title");
        }

And extracting data from it isn't pleasant either:

        // there's two levels of error handling here: the `?` bubbles up "this
        // was malformatted JSON", whereas the match takes care of the "server
        // didn't like our request" case.
        let doc: DocumentData = match serde_json::from_str::<JsonApiDocument>(&payload)? {
            JsonApiDocument::Data(doc) => doc,
            JsonApiDocument::Error(errors) => {
                return Err(PatreonError::Any(format!("jsonapi errors: {:?}", errors)).into())
            }
        };

        // Honestly not sure why this is optional, maybe the spec has the answer
        let user = match &doc.data {
            Some(PrimaryData::Single(user)) => user,
            _ => return Err(PatreonError::Any("no top-level user resource".into()).into()),
        };

        let mut tier_title = None;

        #[derive(Debug, Deserialize)]
        struct UserAttributes {
            full_name: String,
            thumb_url: String,
        }
        // this is fallible because it maps the fields we requested to the
        // struct above, we do want to find out soon if they get out of sync
        let user_attrs: UserAttributes = user.get_attributes()?;

        let memberships = user.get_multi_relationship(&doc, "memberships")?;
        'each_membership: for &membership in &memberships {
            let campaign = membership.get_single_relationship(&doc, "campaign")?;
            if !is_campaign_blessed(&campaign.id) {
                continue;
            }

            // Why yes, this has an `has_many` relationship. I'm not sure
            // when that would ever happen, but that's poorly modelled data
            // for you.
            let tiers = membership.get_multi_relationship(&doc, "currently_entitled_tiers")?;
            if let Some(tier) = tiers.get(0) {
                #[derive(Debug, Deserialize)]
                struct TierAttributes {
                    title: String,
                }
                let tier_attrs: TierAttributes = tier.get_attributes()?;

                tier_title = Some(tier_attrs.title);
                break 'each_membership;
            }
        }

        // just mapping to a non-Patreon-specific data structure (that ends up
        // in the database), more on that later
        let profile = Profile {
            patreon_id: Some(user.id.clone()),
            github_id: None,
            full_name: user_attrs.full_name,
            thumb_url: user_attrs.thumb_url,
        };

        let res = if let Some(tier_title) = tier_title {
            UserInfo {
                profile,
                tier: Some(Tier { title: tier_title }),
            }
        } else if is_user_blessed(&user.id) {
            // I can't subscribe to my own Patreon campaign, so, there's some
            // allowlisting going on here.
            UserInfo {
                profile,
                tier: Some(Tier {
                    title: "Creator".into(),
                }),
            }
        } else {
            UserInfo {
                profile,
                tier: None,
            }
        };

        let fc = FutileCredentials {
            expires_at: Utc::now() + patreon_refresh_interval(config),
            user_info: res,
        };

Doing GraphQL by hand because screw it

And I'm spending a lot of effort (and words) showing you what the JSON:API thing looks like because our next stop is GitHub's take, which is GraphQL, which everyone loves to hate apparently?

Well I love to love it.

I have internalized the compromise and decided that it is good.

Amos

Except the part where everyone settled on JSON, preventing efficient binary payloads and forcing big numbers like identifiers to be stringified, but this too is a compromise and can be solved with more violence codegen.

Here's the equivalent GitHub code.

First, in github_sponsorship_for_viewer.graphql, I have:

query ($login: String!) {
  viewer {
    databaseId
    login
    name
    avatarUrl
  }
  user(login: $login) {
    sponsorshipForViewerAsSponsor {
      tier {
        isOneTime
        monthlyPriceInDollars
      }
    }
  }
}

And then:

impl GithubCredentials {
    pub async fn to_futile_credentials(&self) -> Result<(Self, FutileCredentials)> {
        #[derive(Serialize)]
        struct GraphqlQuery<'a> {
            query: &'a str,
            variables: serde_json::Value,
        }

        let query = include_str!("github_sponsorship_for_viewer.graphql");
        let variables = if is_development() {
            // just testing!
            serde_json::json!({ "login": "gennyble" })
        } else {
            serde_json::json!({ "login": "fasterthanlime" })
        };

        #[derive(Deserialize, Debug)]
        #[serde(rename_all = "camelCase")]
        struct GraphqlResponse {
            data: GraphqlResponseData,
        }

        #[derive(Deserialize, Debug)]
        #[serde(rename_all = "camelCase")]
        struct GraphqlResponseData {
            viewer: Viewer,
            user: User,
        }

        #[derive(Deserialize, Debug)]
        #[serde(rename_all = "camelCase")]
        struct Viewer {
            database_id: i32,
            login: String,
            name: Option<String>,
            avatar_url: String,
        }

        #[derive(Deserialize, Debug)]
        #[serde(rename_all = "camelCase")]
        struct User {
            sponsorship_for_viewer_as_sponsor: Option<Sponsorship>,
        }

        #[derive(Deserialize, Debug)]
        #[serde(rename_all = "camelCase")]
        struct Sponsorship {
            tier: Tier,
        }

        #[derive(Deserialize, Debug)]
        #[serde(rename_all = "camelCase")]
        struct Tier {
            is_one_time: bool,
            monthly_price_in_dollars: i32,
        }

        let client = Client::new();
        let res = client
            .post("https://api.github.com/graphql")
            .header(header::USER_AGENT, "github.com/bearcove/futile")
            .header(header::CONTENT_TYPE, "application/json")
            .json(&GraphqlQuery { query, variables })
            .bearer_auth(&self.access_token)
            .send()
            .await?;

        if !res.status().is_success() {
            let status = res.status();
            let error = res
                .text()
                .await
                .unwrap_or_else(|_| "Could not get error text".into());
            return Err(
                GithubError::Any(format!("got HTTP {}, server said: {}", status, error)).into(),
            );
        }

        let response = res
            .text()
            .await
            .map_err(|e| GithubError::Any(e.to_string()))?;

        let response: GraphqlResponse = serde_json::from_str(&response)
            .map_err(|e| GithubError::Any(format!("while deserializing {response}: {}", e)))?;

        info!("got response: {response:#?}");

        let viewer = &response.data.viewer;
        let full_name: String = viewer.name.clone().unwrap_or_else(|| viewer.login.clone());

        let fut_creds = FutileCredentials {
            expires_at: Utc::now() + Duration::days(365),
            user_info: UserInfo {
                profile: Profile {
                    full_name,
                    patreon_id: None,
                    github_id: Some(viewer.database_id.to_string()),
                    thumb_url: viewer.avatar_url.clone(),
                },
                tier: response
                    .data
                    .user
                    .sponsorship_for_viewer_as_sponsor
                    .and_then(|s| {
                        if s.tier.is_one_time {
                            return None;
                        }

                        match s.tier.monthly_price_in_dollars {
                            5 => Some(futile_credentials::Tier {
                                title: if is_development() { "Silver" } else { "Bronze" }.into(),
                            }),
                            10 => Some(futile_credentials::Tier {
                                title: "Silver".into(),
                            }),
                            50 => Some(futile_credentials::Tier {
                                title: "Gold".into(),
                            }),
                            _ => None,
                        }
                    }),
            },
        };

        Ok((self.clone(), fut_creds))
    }
}

Again, because I can't be my own sponsor on GitHub and I had to test this code some way, I started sponsoring gennyble instead, and, in development, that's what the code looks for.

For GraphQL too, there's Rust crates: on the server side, there's the excellent async-graphql, which I have used in other projects.

On the client side, there's a varity of crates: I've used cynic before, and graphql_client seems very popular according to lib.rs.

But here's the beauty of GraphQL: you don't need to care about it if you don't want to care about it (as a consumer).

After playing in the GitHub GraphQL Explorer for a bit, I had my two queries ready to go: the "as someone who just logged in, what level of access do I have?", and the other, to fetch the list of sponsors so that it can be shown at the bottom of every post.

Listing sponsors: Patreon vs GitHub

That code is actually a nice comparison point for JSON:API vs GraphQL, so let's take a look at the Patreon code:

async fn get_patreon_sponsors(
    client: &reqwest::Client,
    state: &ServerState,
) -> Result<HashSet<String>, Report> {
    let mut credited_patrons: HashSet<String> = Default::default();

    let credited_tiers: HashSet<String> = ["Silver", "Gold"]
        .into_iter()
        .map(|x| x.to_string())
        .collect();

    let patreon_credentials: PatreonCredentials = {
        #[derive(FromRow)]
        struct Row {
            data: String,
        }

        let row: Row = sqlx::query_as(
            "
            SELECT data
            FROM patreon_credentials
            WHERE patreon_id = $1
        ",
        )
        .bind(CREATOR_PATREON_USER_ID)
        .fetch_one(&state.users_pool)
        .await
        .map_err(|e| eyre!("sqlx error: creator needs to log in with Patreon first: {e}"))?;

        serde_json::from_str(&row.data)?
    };

    let (pat_creds, _fut_creds) = patreon_credentials
        .to_futile_credentials(&state.config)
        .await?;

    debug!("Using Patreon credentials for creator: {:#?}", pat_creds);

    let mut api_url = Url::parse(&format!(
        "https://www.patreon.com/api/oauth2/v2/campaigns/{CAMPAIGN_ID}/members"
    ))?;
    {
        let mut q = api_url.query_pairs_mut();
        q.append_pair("include", "currently_entitled_tiers");
        q.append_pair("fields[member]", "full_name");
        q.append_pair("fields[tier]", "title");
        q.append_pair("page[size]", "100");
    }

    let mut api_url = api_url.to_string();

    loop {
        info!(%api_url, "Performing request to Patreon API");

        let res = client
            .get(api_url.as_str())
            .bearer_auth(&pat_creds.access_token)
            .header("user-agent", "futile 1.0")
            .send()
            .await?;

        let status = res.status();
        if !status.is_success() {
            return Err(eyre!("got HTTP {}", status));
        }

        let patreon_payload = res.text().await?;
        trace!("Got patreon_payload: {}", patreon_payload);

        let patreon_response: PatreonResponse = serde_json::from_str(&patreon_payload)?;

        let mut tiers_per_id: HashMap<String, Tier> = Default::default();
        for tier in patreon_response.included {
            if let Item::Tier(tier) = tier {
                tiers_per_id.insert(tier.common.id.clone(), tier);
            }
        }

        for item in patreon_response.data {
            if let Item::Member(member) = item {
                if let Some(full_name) = member.attributes.full_name.as_deref() {
                    if let Some(entitled) = member.rel("currently_entitled_tiers") {
                        for item_ref in entitled.data.iter() {
                            let ItemRef::Tier(tier_id) = item_ref;
                            if let Some(tier) = tiers_per_id.get(&tier_id.id) {
                                if let Some(title) = tier.attributes.title.as_deref() {
                                    if credited_tiers.contains(title) {
                                        credited_patrons.insert(full_name.to_string());
                                    } else {
                                        trace!("Tier {title} not credited");
                                    }
                                }
                            } else {
                                trace!("Tier for id {} not found", tier_id.id);
                            }
                        }
                    } else {
                        trace!("No currently_entitled_tiers for member: {}", full_name);
                    }
                }
            }
        }

        match patreon_response.links.and_then(|l| l.next) {
            Some(next) => {
                api_url = next;
                continue;
            }
            None => break,
        }
    }

    Ok(credited_patrons)
}

JSON:API has pagination built-in: you simply have a links section with a next entry, so you only need to build the first page's URL, that's nice.

Everything else is kinda painful, to be honest: indirection through attributes (which I'm sure other libraries abstract over, making it less aggravating), having to denormalize the data yourself (by looking up related entities).

I like the idea of it, but not actually using it.

By way of comparison, here's the GitHub version of the code:

async fn get_github_sponsors(
    client: &reqwest::Client,
    state: &ServerState,
) -> Result<HashSet<String>, Report> {
    let mut credited_patrons: HashSet<String> = Default::default();

    // same idea so far
    let github_credentials: GithubCredentials = {
        #[derive(FromRow)]
        struct Row {
            data: String,
        }

        let row: Row = sqlx::query_as(
            "
            SELECT data
            FROM github_credentials
            WHERE github_id = $1
            ",
        )
        .bind(CREATOR_GITHUB_USER_ID)
        .fetch_one(&state.users_pool)
        .await
        .map_err(|e| eyre!("sqlx error: creator needs to log in with GitHub first: {e}"))?;

        serde_json::from_str(&row.data)?
    };

    #[derive(Serialize)]
    struct GraphqlQuery<'a> {
        query: &'a str,
    }

    let query_template = r#"
    {
        viewer {
            sponsors(${SPONSORS_PARAMS}) {
                pageInfo {
                    endCursor
                }
                nodes {
                    ... on User {
                        login
                        name
                        sponsorshipForViewerAsSponsorable {
                            privacyLevel
                            tier {
                                monthlyPriceInDollars
                                isOneTime
                            }
                        }
                    }
                    ... on Organization {
                        login
                        name
                        sponsorshipForViewerAsSponsorable {
                            privacyLevel
                            tier {
                                monthlyPriceInDollars
                                isOneTime
                            }
                        }
                    }
                }
            }
        }
    }
    "#;

    #[derive(Deserialize, Debug)]
    #[serde(rename_all = "camelCase")]
    struct GraphqlResponse {
        data: GraphqlResponseData,
    }

    #[derive(Deserialize, Debug)]
    #[serde(rename_all = "camelCase")]
    struct GraphqlResponseData {
        viewer: Viewer,
    }

    #[derive(Deserialize, Debug)]
    #[serde(rename_all = "camelCase")]
    struct Viewer {
        sponsors: Sponsors,
    }

    #[derive(Deserialize, Debug)]
    #[serde(rename_all = "camelCase")]
    struct Sponsors {
        page_info: PageInfo,
        nodes: Vec<Node>,
    }

    #[derive(Deserialize, Debug)]
    #[serde(rename_all = "camelCase")]
    struct PageInfo {
        end_cursor: Option<String>,
    }

    #[derive(Deserialize, Debug)]
    #[serde(rename_all = "camelCase")]
    struct Node {
        login: String,
        name: Option<String>,
        sponsorship_for_viewer_as_sponsorable: Option<SponsorshipForViewerAsSponsorable>,
    }

    #[derive(Deserialize, Debug)]
    #[serde(rename_all = "camelCase")]
    struct SponsorshipForViewerAsSponsorable {
        // e.g. "PUBLIC" or "PRIVATE"
        privacy_level: String,
        tier: GithubTier,
    }

    #[derive(Deserialize, Debug)]
    #[serde(rename_all = "camelCase")]
    struct GithubTier {
        monthly_price_in_dollars: Option<i32>,
        is_one_time: bool,
    }

    let mut sponsors_param = "first: 100".to_string();
    loop {
        info!("Fetching GitHub page with sponsors_param({sponsors_param})");

        let query = query_template.replace("${SPONSORS_PARAMS}", &sponsors_param);

        let res = client
            .post("https://api.github.com/graphql")
            .header(header::USER_AGENT, "github.com/bearcove/futile")
            .header(header::CONTENT_TYPE, "application/json")
            .json(&GraphqlQuery { query: &query })
            .bearer_auth(&github_credentials.access_token)
            .send()
            .await?;

        if !res.status().is_success() {
            let status = res.status();
            let error = res
                .text()
                .await
                .unwrap_or_else(|_| "Could not get error text".into());
            return Err(color_eyre::eyre::eyre!(format!(
                "got HTTP {}, server said: {}",
                status, error
            )));
        }

        let res_raw: String = res.text().await?;

        let res: GraphqlResponse = serde_json::from_str(&res_raw).map_err(|e| {
            color_eyre::eyre::eyre!(
                "could not deserialize GitHub API response: {res_raw}, error: {e}"
            )
        })?;

        let viewer = &res.data.viewer;

        for sponsor in &viewer.sponsors.nodes {
            if let Some(sponsorship) = sponsor.sponsorship_for_viewer_as_sponsorable.as_ref() {
                if sponsorship.privacy_level != "PUBLIC" {
                    continue;
                }

                if sponsorship.tier.is_one_time {
                    continue;
                }

                if let Some(monthly_price_in_dollars) = sponsorship.tier.monthly_price_in_dollars {
                    if monthly_price_in_dollars < 10 {
                        continue;
                    }
                }

                let name = sponsor.name.as_ref().unwrap_or(&sponsor.login);
                credited_patrons.insert(name.clone());
            }
        }

        match viewer.sponsors.page_info.end_cursor.as_deref() {
            Some(end_cursor) => {
                sponsors_param = format!("first: 100, after: \"{}\"", end_cursor);
            }
            None => {
                // all done!
                break;
            }
        }
    }

    Ok(credited_patrons)
}

Because the type of nodes items is a union (User | Organization), there's a bit of duplication on there. That could be alleviated with GraphQL fragments probably, this can all be done a lot better, but again the point I'm belaboring is that you don't have to care about all that. You can just treat it as a JSON endpoint tailored to exactly what you need, and deserialize it with serde_json.

Pagination here is done as parameters passed to sponsors, with opaque cursors. Page size is limited to 100, which is a magical number everyone has decided is the maximum their backend can handle before falling over, apparently.

The monthlyPriceInDollars made me chuckle, as it's very US-centric, but I appreciated the thought that went into privacyLevel: some folks want to support me privately, so they don't appear on my GitHub Sponsors profile, but they should still have access — so the first query ("what access do I have?") doesn't take it into account, but the second query ("who should be credited at the bottom of posts?") does.

A more flexible GraphQL API would let me filter directly by privacy level: maybe GitHub does allow that and I just looked in all the wrong places.

The rest of GitHub integration was just rewording a bunch of templates to be vocab-agnostic (using "sponsors" instead of "patrons"), adding a /donate page to explain why there's two options, what donating does, etc., creating a new database table for GitHub credentials:

use sqlx::{Postgres, Transaction};

pub struct Migration {}

#[async_trait::async_trait]
impl crate::migrations::MigrationPostgres for Migration {
    fn tag(&self) -> &'static str {
        "m0004_github_credentials"
    }

    async fn up(&self, db: &mut Transaction<Postgres>) -> eyre::Result<()> {
        sqlx::query(
            "
			CREATE TABLE github_credentials (
				github_id TEXT NOT NULL,
				data TEXT NOT NULL,
				PRIMARY KEY (github_id)
			)
			",
        )
        .execute(db)
        .await?;

        Ok(())
    }
}

(This is the Migration trait we saw earlier! It's not part of a fancy framework or anything, it's just a quick thing on top of sqlx).

Breaking the (dependency) cycle

One thing that was interesting was the crate structure: previously I had a futile-patreon crate, which kinda mixed concerns. It had Patreon-specific stuff like PatreonCredentials, but also a generic Profile struct, etc.

After integrating GitHub, I ended up with:

  • futile-patreon: All the Patreon-specific types/logic
  • futile-github: All the GitHub-specific types/logic
  • futile-credentials: All the generic code

That let me write some code that works for both Patreon & GitHub, for example when saving settings:

#[axum::debug_handler(state = crate::serve::ServerState)]
pub async fn save(tr: TemplateRenderer, Json(ud): Json<UserData>) -> HttpResult {
    let cookies = tr.state.private_cookies(tr.cookies.clone());

    let fut_creds =
        FutileCredentials::load_from_cookies(&tr.state.config, &cookies, &tr.state).await;

    let fut_creds = fut_creds.ok_or_else(|| {
        warn!("Tried to save settings but no session cookies");
        HttpError::Internal {
            err: "tried to save settings but no session cookies".into(),
        }
    })?;

    {
        sqlx::query(
            "
                INSERT INTO user_preferences
                (id, data) VALUES
                ($1, $2)
                ON CONFLICT (id) DO UPDATE SET data = $2
            ",
        )
        .bind(&fut_creds.user_info.profile.global_id()?)
        .bind(serde_json::to_string(&ud)?)
        .execute(&tr.state.users_pool)
        .await?;
    }

    (StatusCode::NO_CONTENT, ()).into_http()
}

There was, however, one catch — cyclic dependencies.

See, each of PatreonCredentials and GithubCredentials have a to_futile_credentials which returns a type from the futile-credentials crate... but FutileCredentials::load_from_cookies must also be able to refresh OAuth access tokens, which is provider-specific (and apparently impossible in the case of GitHub), which means we'd have a dependency cycle:

So, what do we do when we have a cyclic dependency? Make up a trait!

We could use async-trait but I've written about async so much I have brainrot and I figured I could just throw together something Pin<Box> based and move on to the next topic. And I did!

// in `crates/futile-credentials/src/lib.rs`

/// Something that can refresh credentials
pub trait CredentialsRefresher {
    fn refresh<'b>(
        &'b self,
        creds: &'b FutileCredentials,
    ) -> Pin<Box<dyn Future<Output = eyre::Result<FutileCredentials>> + Send + 'b>>;
}

What's interesting here is that the returned future is Send, and Unpin (since it's on the heap), a-

Cool bear

Whoa whoa whoa you're not going to explain any of that?

Amos

Already did.

...but its lifetime is 'b, which is shorter than 'static. In other words, the future can borrow from self and creds, and so, it does:

// in `crates/futile/src/serve/mod.rs`

impl CredentialsRefresher for ServerState {
    fn refresh<'b>(
        &'b self,
        creds: &'b FutileCredentials,
    ) -> Pin<Box<dyn Future<Output = eyre::Result<FutileCredentials>> + Send + 'b>> {
        Box::pin(async move {
            let patreon_id = creds
                .user_info
                .profile
                .patreon_id
                .as_deref()
                .ok_or_else(|| eyre::eyre!("can only renew patreon credentials"))?;

            let patreon_credentials: PatreonCredentials = {
                #[derive(FromRow)]
                struct Row {
                    data: String,
                }

                let row: Row = sqlx::query_as(
                    "
                        SELECT data
                        FROM patreon_credentials
                        WHERE patreon_id = $1
                    ",
                )
                .bind(patreon_id)
                .fetch_one(&self.users_pool)
                .await
                .map_err(|e| {
                    eyre::eyre!("sqlx error: could not find user patreon credentials: {e}")
                })?;

                serde_json::from_str(&row.data).map_err(|e| {
                    eyre::eyre!(
                        "serde_json error: could not deserialize user patreon credentials {}: {e}",
                        &row.data
                    )
                })?
            };

            info!(
                "Refreshing patreon credentials (expired at {:?})",
                creds.expires_at,
            );

            let mut refresh_creds = patreon_credentials.clone();
            if is_development() && test_patreon_renewal() {
                refresh_creds.access_token = "bad-token-for-testing".into()
            }

            let (pat_creds, fut_creds) = refresh_creds.to_futile_credentials(&self.config).await?;

            {
                let patreon_id = fut_creds.user_info.profile.patreon_id()?;
                sqlx::query(
                    "
                        INSERT INTO patreon_credentials
                        (patreon_id, data) VALUES
                        ($1, $2)
                        ON CONFLICT (patreon_id) DO UPDATE SET data = $2
                    ",
                )
                .bind(patreon_id)
                .bind(serde_json::to_string(&pat_creds)?)
                .execute(&self.users_pool)
                .await?;
            }

            Ok(fut_creds)
        })
    }
}

On the consumer side, we've got this:

    // in `crates/futile-credentials/src/lib.rs`

    pub async fn load_from_cookies(
        config: &Config,
        cookies: &PrivateCookies<'_>,
        refresher: &impl CredentialsRefresher,
    ) -> Option<Self> {
        let cookie = cookies.get(Self::COOKIE_NAME)?;

        let creds: Self = match serde_json::from_str(cookie.value()) {
            Ok(v) => v,
            Err(e) => {
                warn!(?e, "Got undeserializable cookie, removing");
                cookies.remove(cookie);
                return None;
            }
        };

        let now = Utc::now();
        if now < creds.expires_at {
            // credentials aren't expired yet
            return Some(creds);
        }

        let creds = match refresher.refresh(&creds).await {
            Err(e) => {
                warn!("Refreshing credentials failed, will log out: {:?}", e);
                cookies.remove(cookie);
                return None;
            }
            Ok(creds) => creds,
        };

        cookies.add(creds.as_cookie(config));
        Some(creds)
    }

And it.. works just fine.

Arguably there is a way to structure all this that is much less convoluted, I just don't have time for it yet, which is the point I want to make here: everyone encountering Rust suddenly gets these sky-high standards: "ooh my code must never allocate, it must have the perfect structure" when it turns out, nah, you can wing it, even under the watchful eye of a compiler with a special interest in discipline.

You're reading the Updating fasterthanli.me for 2022 series.

1

2

3

4

5

6

7

8

Comment on /r/fasterthanlime

(JavaScript is required to see this. Or maybe my stuff broke)

Here's another article just for you:

A dynamic linker murder mystery

Jul 08, 2020
clock icon 23 min #rust · #linkers · #linux · #postmortem

I write a ton of articles about rust. And in those articles, the main focus is about writing Rust code that compiles. Once it compiles, well, we're basically in the clear! Especially if it compiles to a single executable, that's made up entirely of Rust code.

That works great for short tutorials, or one-off explorations.

Unfortunately, "in the real world", our code often has to share the stage with other code. And Rust is great at that. Compiling Go code to a static library, for example, is relatively finnicky. It insists on being built with GCC (and no other compiler), and linked with GNU ld ().

not
Read more
About Legal Notice Privacy Policy Terms and Conditions