ktls now under the rustls org

What's a ktls

I started work on ktls and ktls-sys, a pair of crates exposing Kernel TLS offload to Rust, about two years ago.

kTLS lets the kernel (and, in turn, any network interface that supports it) take care of encryption, framing, etc., for the entire duration of a TLS connection... as soon as you have a TLS connection.

For the handshake itself (hellos, change cipher, encrypted extensions, certificate verification, etc.), you still have to use a userland TLS implementation.

The Illustrated TLS 1.3 Connection shows the handshake in great detail.

In Rust, the natural choice is rustls, which got a very favorable audit in 2020, and so that's why the higher-level of my two crates, ktls, has a dependency on rustls.

This posed a challenge early on: for the kernel to take over encryption, continuing where rustls left off, we have to, well, not only deal with any data rustls may have already decrypted (TLS frames, TCP segments, and whatever read/recvmsg writes into your buffer don't necessarily align), but also to exfiltrate the session keys (and sequence numbers), something that rustls keeps well-hidden by design.

The exact data being exported varies depending on which cipher is used, and it was a bit of work to find an API design both the rustls maintainers and myself were happy with — that work took place two years ago and landed in rustls in a little under a month.

The ktls API today

Today, ktls's API takes a rustls ClientConnection or ServerConnection (kTLS works in either direction) and gives you back a TcpStream and a Vec<u8> of already-decrypted data.

Here's an excerpt of a code sample from loona, my HTTP/1+2 implementation:

Rust code

// use a self-signed certificate for testing
let certified_key = rcgen::generate_simple_self_signed(vec!["localhost".to_string()]).unwrap();
let crt = certified_key.cert.der();
let key = certified_key.key_pair.serialize_der();

let mut server_config = ServerConfig::builder()
    .with_no_client_auth()
    .with_single_cert(
        vec![crt.clone()],
        PrivatePkcs8KeyDer::from(key.clone()).into(),
    )
    .unwrap();

// not strictly needed for kTLS, but useful for debugging:
// this reads the SSLKEYLOGFILE environment variable, and writes
// the secrets to it
server_config.key_log = Arc::new(rustls::KeyLogFile::new());

// we'll need to extract secrets to give them to the kernel — the
// rustls API has us explicitly opt into this
server_config.enable_secret_extraction = true;

server_config.alpn_protocols = vec![b"h2".to_vec(), b"http/1.1".to_vec()];

let acceptor = tokio_rustls::TlsAcceptor::from(Arc::new(server_config));

let stream: TcpStream = todo!("TCP listen/accept logic goes here");

// this wrapper is able to stop reading at the boundary between
// two TLS messages — which is done in `config_ktls_{client,server}`
// further down, once the handshake is done.
let stream = CorkStream::new(stream);

// this is the usual way for a server to establish a TLS connection
// asynchronously with `tokio-rustl`
let stream = acceptor.accept(stream).await?;

// as usual, we get access to the rustls session, including
// the negotiated ALPN protocol
let sc = stream.get_ref().1;
let alpn_proto = sc.alpn_protocol().and_then(|p| std::str::from_utf8(p).ok().map(|s| s.to_string()));
debug!(?alpn_proto, "Performed TLS handshake");

// this extracts the secrets, configures kTLS as the "ULP"
// (upper-layer protocol), and returns a `TcpStream` that
// transparently does TLS.
let stream = ktls::config_ktls_server(stream).await?;

debug!("kTLS successfully set up");
let (drained, stream) = stream.into_raw();
let drained = drained.unwrap_or_default();
debug!("{} bytes already decoded by rustls", drained.len());

Coordinating & collaborating

But rustls occasionally evolves, and undergoes APIs changes (whenever better — safer, more correct, more flexible — interfaces are found).

And whenever a new version comes out, crates like tokio-rustls and ktls must be updated to be compatible with it.

In the past, I've been irritated at the several months lag between a rustls update and the corresponding tokio-rustls release — I recently complained about it online and the maintainers let me know that this is unlikely to happen again in the future because both packages are now under the same GitHub organization, https://github.com/rustls — they also offered to adopt ktls, so that it would also stay in-sync, and so we could easily orchestrate simultaneous releases of rustls, tokio-rustls, and ktls.

I'd like to thank Dirkjan for offering to adopt ktls — even though I'll be around too, to deal with whatever comes up :)

Thanks to my sponsors: Miguel Piedrafita, Mike Cripps, Alejandro Angulo, Ryan, Ronen Cohen, Andy F, Paul Horn, Makoto Nakashima, Lucille Blumire, Astrid, Sammy, Aleksandre Khokhiashvili, Jörn Huxhorn, Zaki, Jon Gjengset, Luke Yue, Josiah Bull, Aiden Scandella, Chris Emery, Yann Schwartz and 241 more