Thanks to my sponsors: Isak Sunde Singh, Morgan Rosenkranz, Marcus Griep, you got maiL, avborhanian, Henrik Tudborg, Dirkjan Ochtman, Noel, Santiago Lema, Seth, Paul Schuberth, Thor Kamphefner, Marcus Brito, Sarah Berrettini, Beth Rennie, Sylvie Nightshade, Yann Schwartz, Joshua Roesslein, Alex Doroshenko, Chris Sims and 232 more Isak Sunde Singh, Morgan Rosenkranz, Marcus Griep, you got maiL, avborhanian, Henrik Tudborg, Dirkjan Ochtman, Noel, Santiago Lema, Seth, Paul Schuberth, Thor Kamphefner, Marcus Brito, Sarah Berrettini, Beth Rennie, Sylvie Nightshade, Yann Schwartz, Joshua Roesslein, Alex Doroshenko, Chris Sims, James Leitch, Matt Jackson, Zeeger Lubsen, Josh Triplett, Andronik, Ernest French, joseph, Horváth-Lázár Péter, Richard Pringle, Duane Sibilly, John VanEnk, Joonas Koivunen, bbutkovic, L0r3m1p5um, Philipp Gniewosz, Yuriy Taraday, thbkrshw, Alex Krantz, budrick, xales, Cole Kurkowski, Daniel Silverstone, Jake Demarest-Mays, Justin Ossevoort, Brandon Piña, Michael Alyn Miller, Andy Gocke, Josiah Bull, Zaki, Mark Tomlin, Guillaume Demonet, Corey Alexander, Romet Tagobert, ofrighil, Jarek Samic, jatescher, Marie Janssen, Max Heaton, Tyler Schmidtke, Aleksandre Khokhiashvili, Miguel Piedrafita, Tabitha, Chris Walker, Nicolas Riebesel, Senyo Simpson, Max von Forell, Herman J. Radtke III, Mason Ginter, Tom Forbes, Raphaël Thériault, Romain Kelifa, Johan Saf, Michael, Eugene Bulkin, David Cornu, Makoto Nakashima, Tobias Bahls, Dom, Daniel Strittmatter, Lennart Oldenburg, Dylan Anthony, Marcin Kołodziej, Diego Roig, Zoran Zaric, Paige Ruten, Johnathan Pagnutti, Marc-Andre Giroux, Neil Blakey-Milner, Michał Bartoszkiewicz, Michal Hošna, notryanb, belzael, Shane Lillie, Paul Horn, Cass, compwhizii, Boris Dolgov, WeblWabl, Walther, Dennis Henderson, Ben Wishovich, Mario Fleischhacker, Jörn Huxhorn, Marky Mark, hgranthorner, Antoine Rouaze, Kristoffer Winther Balling, David White, Philipp Hatt, villem, Ian McLinden, Benjamin Röjder Delnavaz, Alejandro Angulo, AdrianEddy, Hadrien G., Max Bruckner, Torben Clasen, Daniel Wagner-Hall, playest, Paul Marques Mota, Mark Old, Andy F, Moritz Lammerich, Mike Cripps, Tanner Muro, Laine Taffin Altman, Matt Heise, Jelle Besseling, Braidon Whatley, Jack Duvall, Lyssieth, Colin VanDervoort, James Brown, Scott Sanderson, Gran PC, David Barsky, Ryan, Ives van Hoorne, Egor Ternovoy, Andrew Neth, prairiewolf, Pete Bevin, Foxie Solutions, Marty Penner, Ramen, Valentin Mariette, Yves, James Rhodes, Mikkel Rasmussen, Luke Konopka, old.woman.josiah, Jean-David Gadina, Blake Johnson, Johan Andersson, Daniel Papp, Geoff Cant, Scott Steele, Tiziano Santoro, Lucille Blumire, Olly Swanson, Guillaume E, Chris Emery, SeniorMars, Jacob Cheriathundam, Manuel Hutter, ZacJW, Raine Tingley, Sindre Johansen, David Souther, Ivo Murrell, Marcus Griep, Timothée Gerber, Steven Pham, Chris Biscardi, Jonathan Adams, Mattia Valzelli, Nikolai Vincent Vaags, Evan Relf, Jan De Landtsheer, Garret Kelly, Matt Jadczak, Luiz Irber, Ronen Ulanovsky, Chirag Jain, Adam Gutglick, Xirvik Servers, René Ribaud, David E Disch, Xavier Groleau, Michael Mrozek, Kamran Khan, Luke Yue, Julian Schmid, Matěj Volf, Alan O'Donnell, Ronen Cohen, Victor Song, Lena Schönburg, Rufus Cable, Guy Waldman, Dimitri Merejkowsky, Nicholas Bishop, Berkus Decker, Christian Bourjau, Chris, Yufan Lou, Lev Khoroshansky, genny, Sean Bryant, anichno, Aiden Scandella, Jean Manguy, Aalekh Patel, Katie Janzen, C J Silverio, clement, Matthew T, henrique-pinto, Enrico Zschemisch, std__mpa, Zachary Thomas, Niels Abildgaard, Mathias Brossard, Peter Shih, Radu Matei, Tyler Bloom, callym, Jon Gjengset, Sawyer Knoblich, Richard Stephens, Geoffroy Couprie, zaurask, qrpth, pinkhatbeard, Helge Eichhorn, Ben Mitchell, Borys Minaiev, Marco Carmosino, Romain Ruetschi, Steven McGuire, Integer 32, LLC, Ross Williams, Malik Bougacha, Chris Thackrey, Olivia Crain, Ahmad Alhashemi, Ula, Bob Ippolito, jer, Christoph Grabo, Astrid, Vincent

fasterthanlime

May 31, 2023 23 min #rust · #governance

The RustConf Keynote Fiasco, explained

Recently, RustConf offered ThePhD (aka JeanHeyd Meneide) the keynote speech, then on May 26, suggested it be downgraded to a "regular talk", despite the author's extreme clarity about the exploratory nature of the work being presented.

This is relevant, because some think the keynote should be "orthodox" (representative of the official direction of the Rust project), even though there's precedent for "heterodox" keynotes being given in the past at RustConf and other confs.

JeanHeyd then withdrew from RustConf altogether, then Sophia resigned over it and then speculation ran wild!

Speculation? On the internet?!

Right??

For example, some people thought JeanHeyd would've been the first black keynote speaker for RustConf.

And that, if the only justification for the take-back was "some people expressed concerns" (who? and what were they?), it's hard to blame observers for tentatively connecting the dots, given JeanHeyd called out the lack of diversity at RustConf 2020.

Because I was part of the same private chats as some Rust project members, I had insider information about the whole thing, and eventually chose to quit those private chats and petition the people who actually were on the relevant teams for an official response.

I didn't name anyone involved, knowing that pressure was already applied internally, and counting the hours until people stepped forward.

Predictably, I was met with frustration from folks who felt I was complicit in the whole thing.

And before we take a look at the full timeline and where we're at, let me take a moment to explain myself: I've been on both sides of internet mobs before.

I've been the canceller, the cancellee (boy did I fuck up, several times), and I've had my site DDoS'd for the lulz. I've been the public figure taking the flak for an org, and I've been the "man of the people" asking a company for accountability.

And even though I'm the kind of neuroatypical who feels, like, really strongly about justice, I have significantly improved my chill over the years.

The RustConf Keynote Fiasco unfolded over a three-day weekend for a lot of folks (including me), at a time where "Rust leadership" was already in the middle of being overhauled, the previous fiasco was still deemed unresolved by some, teams were still understaffed, many people burnt out for months at this point, etc.

I know the Rust project has come dangerously close to imploding before, and so I wanted to be mindful of how to proceed from there.

It's a delicate balance between "demanding justice" and "remembering we're dealing with human beings here". I've blown everything up and left enough times before, I was hoping for a better outcome here, despite the project's track record.

And the good news: I think we've got that outcome, or at least, part of it.

There's lots to talk about, so, forgive the length and the sprawling nature of this article: I've done my best with the time I had.

Before we look at personal and collective statements, we need to spend some time understanding which organizations are involved, and how they're related (to each other, and the topic at hand).

The Rust Foundation describes itself as "an independent non-profit organization dedicated to stewarding the Rust programming language, nurturing the Rust ecosystem, and supporting the set of maintainers governing and developing the project."

My understanding of it has always been that it's a legal requirement for funneling money from the big corporate users of Rust (like AWS, Google, Huawei, Meta, and Microsoft) to the people who actually maintain Rust.

So, there's employees from all these companies who work on Rust, and who are part of various teams (and people tend to freak about that, oh no, a corporate takeover!), but we also have "neutral", Rust-foundation-employed people now, who work on various things: you can check the complete list on the foundation's website.

And then there's everyone else, who can receive grants. In 2022 the foundation's grant budget was $625K, which is "a lot" and "not a lot" of money at the same time. Those grants are generally not enough to live on, by a long shot, but they qualify as "support" (perhaps even "recognition").

If you check the list of grant recipients for 2023 you'll notice that, among others, I'm on it: I have received two payments of $2500 for my Rust-focused articles and videos for that year. This isn't "buy a house" nor "buy my silence" money, but you know: it's nice to be seen.

The foundation also covers infrastructure costs related to hosting and operating rust-lang.org, crates.io and docs.rs. Before that, the on-call rotation for these was volunteers only, and many folks burned out as a result.

Some of the costs are not so much "paid for by the foundation" as "donated by member companies of the foundation"

All the details are in their annual reports, including their income (who donated and how much?) and expenditures (what was money spent on?).

But none of these are the reason the foundation got.. founded!

As the Rust project grew, its relationship with Mozilla became more and more strained. For example, the first two "Rust All Hands" events took place in Mozilla's Berlin office. But it eventually outgrew the venue, and in 2020, when the project tried to host it in Greece, things didn't go well.

Eventually, several folks from the Rust project worked tirelessly to put together the foundation as a Delaware non-profit corporation, and it was announced on February 8 2021.

What's important to note here is that The Rust Foundation was not involved at all in the RustConf Keynote Fiasco (RKNF).

It was involved in the Rust Trademark Fiasco (RTMF), in March of 2023. Although there's similarities in the dysfunction, different people and roles are involved.

Oh we're using acronyms now huh?

I'm nothing if not a nomenclature nerd.

I'm going to be honest with you here - even as a "partial insider", it's been extremely hard for me to figure out the structure of the Rust Project.

Essentially, there's teams, and they do stuff.

Looking at the list, you would be forgiven for assuming that some teams may naturally hold more power than others, that there's overlapping responsibilities, and that cross-team discussions occasionally need to be had!

And that these discussions, occasionally, break down.

And that's what the moderation team was here for!

Not only did they moderate discussions on GitHub (but NOT the /r/rust subreddit, as I've recently learned), but also on Zulip, between teams, and they were responsible for removing people from teams, if they didn't abide by the Code of Conduct.

Until every single one of them resigned in November of 2021, claiming that the core team was only accountable to itself.

Note: I'm not linking to the current "Moderation team" and "Core team" pages on purpose: their composition has varied over the years.

This Rust core team alumni blog post attempts to list various line-ups.

There's a lot left unsaid about that episode, especially since the core team hasn't published their side of the story as of yet.

The vagueness of the moderation team's mass resignation letter frustrated the wider community, was acknowledged officially three days later, and a "leadership chat" was formed (in private, for the time being).

A couple weeks later, an e-mail sent to the ~300 members of the Rust project was published on the Inside Rust Blog, wherein the following high-level goals are stated:

1. Figure out how to handle complex moderation issues
2. Fix Rust governance by changing its structure (but how?)
3. Fix the issue underlying the moderation team's resignation

In May of 2022, a governance update was posted informing everyone of the creation of said "leadership chat". The update notes that the chat was beneficial initially:

This has led to better communication around project management issues and allowed for healthy, productive, and good faith collaboration among all project leads.

...yet, one year later, there is consensus that it has existed for far too long (there's no such thing as a "temporary fix", etc. etc.), as this blog article from May 29 of 2023 notes:

This leadership chat was meant as a short-term solution and lacked clear rules and processes for decision making and communication. This left a lot of room for misunderstandings about when a decision had actually been made and when individuals were speaking for the project versus themselves.

All the while, an RFC (request for comments) for reforming Rust governance has been in the works: RFC 3392, establishing the Leadership Council.

That RFC has reached FCP (final comment period) a little over a month ago at this point, on April 19, 2023. FCP lasted 10 days, and the PR is now ready to be merged.

It hasn't been merged yet, because (as I'm writing this), representatives from all teams are still being picked. But things are being fast-tracked, because, well, of the RKNF (RustConf Keynote Fiasco, remember?).

All clear? No? Well of course not.

There's been a lot of issues over the years, which isn't exactly a surprise given the meteoric rise of Rust as a language. What worked to some extent in 2015 didn't work anymore in 2020, and over time, the wounds deepened, several individuals burned out and left, quietly or not.

It would be unwise to try and summarize those conflicts, but what I do want to close this section with is to note patterns of behavior: the Rust project has repeatedly encountered problems it didn't know how to solve, couldn't communicate publicly about, and which looked from the outside as if malicious actors were involved, especially in the absence of swift, clear, and coordinated communication.

Regularly, the worst was only narrowly avoided through hours of thankless work from folks who acted as makeshift mediators, temporary figureheads, and human lightning rods.

And that's why, in my previous post, I linked to Saoirse's recent post about governance. To quote it here:

[...] sometimes the problem is governance, but usually the problem is power.

What I meant with this remark was that organizational problems that appear as inadequacies in the formal system of governance are most commonly just the manifestations of an underlying problem that has arisen from the informal exercise of power by members of that organization, outside of (if not in direct contradiction to) the formal rules of governance of the organization.

Because, as it turns out, despite Rust's very much in-progress governance structure, that's very much what happened during the RKNF.

And I can now definitely say so, because some of the individuals in question have released their own statements, which I can point to.

But before we look at those, let's talk about the third "Rust".

RustConf bills itself "the largest annual gathering of the Rust-language community".

In my mind, it's "the official Rust conference", even though there's other large Rust conferences, like EuroRust, which I plan to attend in 2023.

RustConf the organization (led by Leah Silber) handles the logistics of RustConf the event (having a website, a way to sell tickets, a venue, a place to broadcast talks, communicating publicly about dates etc.).

The program of RustConf, however (who's invited to talk, and about what topics), is handled by the program committee, made up of a lead (this year, like a few of the previous years, Sage Griffin) and multiple volunteers.

In fact, I was on the program committee in 2022, which means I helped select speakers and talks for the previous edition of RustConf. There's a platform for reviewing proposals, the first part of the process is all anonymized, there's a mechanism for leaving feedback for submitters, etc.

Then the committee meets over several video calls to hammer out the specifics (at which point concerns like "should we give the audience a break between those two highly technical topics?" start to come up).

You don't need much more background there to follow the accounts of RKNF from different perspectives, so, let's proceed!

Leah made two statements first from her personal account on May 28 where she apologizes for the situation, then later that same day, again from her personal account, explaining the difference between the Rust Foundation, the Rust Project, and RustConf, and what happened from her perspective.

Sage released their statement the day after, on May 29 - since they're both sides of the same coin, I'll try to summarize what both statements have in common.

In essence, they explain that the keynote is a little different from the rest of the program, and that there's some sort of consensus-based system involving the Rust project to pick it.

Specifically, Leah initially ran RustConf as a favor to friends, and consulted them for input on keynote selection. Later, as those friends ended up leaving, the Rust project started assuming they had a say on the keynote spot.

This wasn't clearly codified or clearly communicated for a while. For 2023, the RustConf program committee had explicitly agreed to have project leadership select that year's keynote.

That's why, when a representative from the project said they wanted to move JeanHeyd's talk from the keynote to a regular slot, it was not strictly speaking overreach - however bad the idea might be.

Sage feared that simply saying "no" to the Rust project might lead to different drama, and, in hindsight, regrets considering giving in to that ask (and conveying the idea to JeanHeyd) especially since the consensus they thought was there.. wasn't.

They note that:

[...] there were plenty of structural [failures]. The project is working on fixing its structural issues that let this happen.

...while taking personal responsibility for failing to push back properly:

Ultimately this was the result of a shitty game of telephone leading to disastrous consequences. My over-reliance on backchannelling is what led to this game of telephone happening in the first place, which I will improve going forward.

Ultimately the buck should have stopped with me when it came to this decision. I'm sorry for my lack of judgement in this matter. It won't happen again.

Leah and Sage capped this with a joint statement from the @rustconf Twitter account.

So, when I said "all the good people who could've left haven't yet", well there's two of them. Nevertheless, I appreciate both of them taking responsibility for their part and apologizing publicly here.

On May 29 (3 days into RKNF), "leadership chat" released the first official response to the whole thing:

First off, I want to note that it's a real apology:

That decision was not right, and first off we want to publicly apologize for the harm we caused. We failed you JeanHeyd. The idea of downgrading a talk after the invitation was insulting, and nobody in leadership should have been willing to entertain it.

A fake apology would say "we're sorry if JeanHeyd feels like [...]", or things of that nature.

There's many good articles on the art of apologizing, here's one I picked at random. It's written with relationships in mind, but largely applicable here.

The statement notes that "[the] work isn't finished yet", but shares some steps that have been taken, notably that, until the new leadership council is ready, "leadership chat will enforce strict consensus rule for all decision making, so that ther is no longer ambiguity of whether something is an individual opinion or a group decision."

Additionally, the post explicitly says it's "leaving room for individuals to publicly acknowledge their own role".

And some of them did!

Josh Triplett released his own statement on May 30, in which he apologizes and takes personal responsibility for his part in all this, in addition to stepping down from:

• The upcoming leadership council
• The interim leadership chat
• The language team co-lead position

Some of these links point to rust-lang/team, which is effectively a multi-year record of who belonged to which team - except for the "leadership chat team", which doesn't have a 1-1 mapping with who was/currently is in the Zulip chat.

Nevertheless it would be really cool to generate some sort of graphical timeline based on that information, methinks.

Additionally, he will not be giving the RustConf talk The Rust Constitution and the Leadership Council, and will not be leading the RustConf PostConf UnConf as was previously planned.

Looking around the rust-lang/team repo some more, we can see Josh Triplett is still also on the libs team, the libs-api team, the cargo team, the crate-maintainers team, and the style team.

In his statement, Josh lays out a timeline, which I've completed based on several other accounts:

• May 5: Josh Triplett, alongside Josh Gould (who has released their own statement separately), relay a proposed pair of keynote speakers to Sage, which some people in leadership chat expressed support for (and no objections were raised, despite objections being explicitly solicited)
• May 18: Josh Triplett receives "several complaints" from "a few Rust project members" about "various aspects of [JeanHeyd's project] and the keynote selection".
• May 18: Josh Triplett starts a discussion about "people having concerns" in leadership chat. Initially, those concerns are echoed by a couple people.
• May 18: Both Josh Triplett and Josh Gould attempt to convey "some people have concerns" to Sage. However, this is done in such a way that Sage is led to believe that leadership chat is asking them to uninvite a speaker
• May 18: Josh Triplett brings up the idea of demoting the talk to Sage, who takes some time to think about it.
• May 18: In the meantime, in leadership chat, several people are pushing back on the concerns re JeanHeyd's talk, and pushing back on the idea of demoting their talk, which they feel would amount to uninviting them.
• May 22: Sage asks JeanHeyd to schedule some time to discuss their talk, they pick May 26
• May 26: JeanHeyd is asked how they'd feel about their keynote proposal being a regular talk instead, and, later that same day, JeanHeyd withdraws from RustConf altogether

Those statements and Josh Triplett's stepping away from leadership responsibilities were what I was calling for in my previous article.

With all the context I have today, I feel like this is an appropriate response from Josh.

I was wondering how meaningful "stepping down as language team co-lead" is in reality: some have argued that "team member" is a position of power in itself.

In addition, several discussions of Rust's proposed new form of governance specifically point out that changing the structure is meaningless if "implicit power" or "shadow power" is still being used.

I've quoted Saoirse already (emphasis mine):

But maybe the real problem that led to the collapse of the old system was not the explicit manifestations of power in the formal leadership body, but the informal and hidden power held by key stakeholders, who use that power through backchannels outside of the official system [...]

Saoirse, A governance system, if you can keep it

And I'll add Graydon:

I don't know if it does enough to imbue positions of power in the project -- including informal power -- with accountability for their actions, or to communicate that publicly to instil confidence.

Graydon Hoare, Batten Down Fix Later

When reached for comment, Josh Triplett mentioned that "team lead" did come with various capabilities, some well-defined, and some "soft", like:

• making posts on the Inside Rust Blog, since it was assumed team leads were "empowered to speak for their team", with "the expectation that they know enough about what's going on across the project that they can approve other people's messaging, etc."
• vetting new proposed members of the team (checking with moderation and sometimes leadership) and sending updates to the rust-lang/team repository.
• define policy for the team (for the lang team, this was, for example, the "initiative" process)
• being consulted by the Rust Foundation to vet project grant proposals

And that these are what he was stepping away from.

In our discussion, Josh Triplett also emphasized how different this was from the expected role of a team member (according to him, at least for the lang team):

As a team member, my role is to read RFCs and proposals and requests, collectively discuss them with the team in meetings that have public minutes, check boxes for RFCs, start FCPs, and the like.

I am aware that having literally any participation in the project means that my words will carry weight in other contexts. This is something I am acutely aware of at this point. What I'm trying to avoid is being in any leadership role or any poorly-defined poorly-scoped role.

And, frankly, I also plan to spend a lot of time saying "let's make sure we're talking about this in public", and watching like a hawk for any cases where the team ever starts something unaccountable rather than transparent.

To fully appreciate how insulting the whole thing was, one need look no further than JeanHeyd's May 26 withdrawal from the conference (emphasis mine):

But, to date:

• Shepherd’s Oasis and I have not received any e-mail communication regarding the work or any concerns from the Rust Project;
• Shepherd’s Oasis and I have not been given any warnings or asked to reconsider things through any kind of instant messaging communication or otherwise (Zulip, Discord, etc.) from the Rust Project; and,
• Shepherd’s Oasis and I were not the original ones seeking to give presentations about this subject material until we were explicitly asked to.

JeanHeyd, having worked with other standards organization before (C, C++), has, excuse my French, no patience for this bullshit:

This is a mark of both vindictive behavior and severe unprofessionalism that I expected from various organizations I am forced to interact with from day to day as a human being living in a flawed world, but not the Rust Project.

I have tried my best to rescue whatever goodwill was left after this episode, but just recently, Shepherd's Oasis, has pulled out from their Rust foundation grant and will no longer be working on their generic compile-time introspection pre-pre-RFC.

I am incredibly sad at this outcome, although it's completely understandable.

I still wish that the person or persons who expressed concerns about JeanHeyd's work in a private chat rather than directly to them, would release their own statement, explaining how they intend to deal with conflicts like these in the future.

I understand how difficult this might be given how things unfolded, but I believe the community deserves such candor to be able to truly move past this.

I'll be updating this article as further statements come out.

Manish just released his own statement detailing his role in the whole thing.

Manish was the person who initially nominated JeanHeyd for the keynote, based on their perspective from the C/C++ community.

JeanHeyd has worked on the C23 standard as project editor, for example.

When objections were brought up later, Manish pushed back, but also noted that deep technical dives seldom make for great keynotes (but that the reflection topic would be decent for an invited talk).

(Much earlier, Manish had tried to advise JeanHeyd to pick a topic other than reflection for the keynote, but was nevertheless excited about this project being talked about).

This was one of the sentiments that got picked up and relayed, eventually resulting in that whole situation. Manish explains that the remark was made off-the-cuff, in-between other real-life commitments, and that in retrospect he wish he'd have pushed back on the other concerns more strongly, and been more careful about how his remark might be interpreted.

Manish agrees with JT that the issue was handled in an extremely disrespectful way, that it should've never gotten to this point, and that the Rust project needs to be better than this.

Mara released her own statement detailing her part in this.

I haven't had time to summarize it yet, as I'm fighting worsening headache.

I'm still personally hopeful for more statements.

Comment on /r/fasterthanlime

Here's another article just for you:

Frustrated? It's not you, it's Rust

Aug 14, 2020

50 min #rust · #traits · #sized · #lifetimes

Learning Rust is... an experience. An emotional journey. I've rarely been more frustrated than in my first few months of trying to learn Rust.

What makes it worse is that it doesn't matter how much prior experience you have, in Java, C#, C or C++ or otherwise - it'll still be unnerving.

In fact, more experience probably makes it worse! The habits have settled in deeper, and there's a certain expectation that, by now, you should be able to get that done in a shorter amount of time.

will

Read more