Skip to content

feat(velo-hub): TLS on the hub's listeners, with client certificates - #155

Draft
ryanolson wants to merge 5 commits into
mainfrom
ai/hub-tls
Draft

ryanolson wants to merge 5 commits into
mainfrom
ai/hub-tls

Conversation

@ryanolson

@ryanolson ryanolson commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

What

The hub's public and control listeners can serve TLS, with optional client certificate verification. The verified client identity is available to routes, extensions, and the registration path. Plain HTTP stays the default.

  • HubTls::new(chain, key), and HubBuilder::public_tls / HubBuilder::control_tls. Each listener has its own setting, so a common setup is server TLS on the public listener and client certificates on the control listener. The library reads no files. The program passes DER certificates and keys (velo_hub::pki_types, re-exported from rustls).
  • HubTls::client_certs(roots, ClientCerts::Required | ClientCerts::Optional). Required: a client without a certificate that verifies fails the handshake. Optional: an anonymous client connects, but a certificate that it presents must verify.
  • ClientIdentity (leaf, chain, URI SANs):
    • Extractor: answers unauthorized (401) without a verified certificate. Option<ClientIdentity> serves anonymous clients too.
    • Registration hooks: RegistrationView::client, Txn::client, Committed::client. An extension that admits only some identities refuses in admit.
  • Rotation: HubTls::rotate and HubTls::set_client_roots. A resolver and a verifier read the current values on each handshake, so new connections use them without a restart.
  • TLS 1.3 only, and no session resumption. A resumed session skips the client certificate check and keeps the first identity, so it would outlive a rotation. A test proves this.
  • TLS listener:
    • It owns its socket and accepts only inside axum's accept. So it takes no connection before the hub serves, and its port is free when shutdown returns.
    • Each accepted handshake runs in its own task under HANDSHAKE_TIMEOUT (10 s), so a client that connects and sends nothing does not hold up the next one.
    • At most MAX_PENDING_TLS (256) connections are in a handshake or wait for axum. Past that, the listener stops accepting, and the kernel backlog holds the rest, as for a plain listener.
    • ALPN offers HTTP/1.1 only, because the hub does not build axum's HTTP/2 support.
  • Both listeners now serve with connect info (into_make_service_with_connect_info), so routes read the identity the same way on plain and TLS listeners.

The client side needs no change. A program builds a reqwest::Client with TLS, and with a client certificate if needed, and passes it to HubClientBuilder::http_client with .scheme("https").

Dependencies

Under server: rustls >= 0.23.23 and tokio-rustls (ring provider, no process-wide default provider), and rustls-webpki >= 0.103.8 for URI SANs. All three are already in the lockfile. Dev: reqwest gets rustls-tls-manual-roots (ring), and rcgen makes the test certificates. With the no-provider reqwest feature, every default reqwest::Client in the other suites panicked, so the dev feature includes a provider.

velo-hub and velo-hub-ext were never published, so the semver gate skips them. cargo tree -p velo-hub-ext shows no velo and no prometheus. RegistrationView, Txn, and Committed get a new public field, which is a break for code that builds them. Only the hub builds them.

Tests

lib/velo-hub/tests/tls.rs:

  • The existing HubClient over TLS on both listeners through http_client: register, discover, unregister. A plain-HTTP request to a TLS listener gets no answer.
  • Client certificates required: a client without a certificate, and one with a certificate from another CA, cannot register. Control: a certificate from the configured CA registers.
  • An extension route reads ClientIdentity. Without a certificate, it answers 401, and the Option route answers "anonymous". admit refuses an identity that the extension does not authorize, and admit, prepare and commit each saw the identity of the admitted client.
  • The service watch (SSE) over TLS: initial set, then Added.
  • Rotation: one client config is kept through rotate (the new certificate is served) and set_client_roots (the old-CA client is then refused, and a new-CA client is accepted).
  • Eight silent TCP clients do not delay a TLS request behind them.
  • No TLS handshake completes while an extension's start holds the hub's startup.
  • With a limit of 2 (unit test), two silent clients hold both slots, and a real client waits until one leaves.
  • A client that offers h2 and http/1.1 gets http/1.1.
  • Dropping a TLS listener frees its port at once (unit test, current-thread runtime), and 20 hubs in a row free their ports at shutdown.
  • Key mismatch on new and rotate, and set_client_roots without client_certs, are refused.
  • Mutations:
    • Re-enable session resumption: the rotation test fails.
    • Drop the identity: the extension test fails.
    • Never offer client auth: three tests fail.
    • Run the handshake on the accept path: the silent-client test fails.
    • Raise the limit past the silent clients: the bound test fails.

All velo-hub suites pass with --all-features, except the etcd scale test a_prefix_at_target_scale_loads. On this machine, that test fails on origin/main too, in 3 of 3 runs. It is not from this PR.

Review ledger

An independent reviewer did four rounds (thermo-nuclear-code-quality-review and rust-code-review). Each valid code finding got a test that failed first.

Round Finding Ruling Change
1 High. TLS connection growth was unbounded. Accepting started at bind, before the hub served, and each finished handshake waited in a task of its own. Valid. no_tls_handshake_completes_before_the_hub_serves and pending_tls_connections_are_bounded. The listener accepts only inside axum's accept. A semaphore of 256 slots covers the connections in a handshake or waiting for axum.
1 Medium. ALPN offered h2, which a hub built with only server cannot serve. Valid. the_listener_negotiates_http_1_1. ALPN offers http/1.1 only.
1 Medium. shutdown could return while a detached accept task still held the TLS port, for up to 1 s after an accept error. Valid. dropping_a_tls_listener_closes_its_socket_at_once. No detached accept task: axum's serve future owns the socket, and the hub joins that future.
2 None Verdict: ready
3 Medium (docs). "Silent clients delay others for up to HANDSHAKE_TIMEOUT" is not true: the timer starts at accept, and the backlog has no deadline. Valid. The book says what the timeout covers, and makes no overall promise.
4 None Verdict: ready

From the V1 review: the rustls requirement went up to 0.23.23 (CertifiedKey::from_der).

The hub's public and control listeners served plain HTTP only. Each
can now serve TLS, configured by the program that builds the hub:
HubBuilder::public_tls and HubBuilder::control_tls take a HubTls. The
library reads no files; the program passes DER certificates and keys.
Plain HTTP stays the default.

HubTls::client_certs asks clients for a certificate and verifies it
against the given roots, Required or Optional. The verified
ClientIdentity (leaf, chain, URI SANs) is available to routes through
an extractor (401 without one, or Option<ClientIdentity>), and to
registration hooks as RegistrationView::client, Txn::client and
Committed::client, so an extension can admit only the identities that
it authorizes.

HubTls::rotate and HubTls::set_client_roots apply to new connections
without a restart. TLS 1.3 only, with no session resumption, because a
resumed session skips the client certificate check and keeps the first
identity. Each handshake runs in its own task under a 10 s timeout,
so a silent client does not hold up accept.

Both listeners serve with connect info, so routes read the identity
the same way on plain and TLS listeners.

Tests: the existing HubClient over TLS through http_client; a client
without a certificate, or with one from another CA, is refused when
verification is on; an extension route and every registration hook see
the identity, and admit refuses an identity it does not authorize; the
service watch (SSE) over TLS; rotation of the certificate and the
client roots with one client config kept throughout; a silent client
does not block other handshakes.

Signed-off-by: Ryan Olson <rolson@nvidia.com>
@coderabbitai

coderabbitai Bot commented Oct 22, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.10758% with 65 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
lib/velo-hub/src/hub/tls.rs 83.28% 56 Missing and 7 partials ⚠️
lib/velo-hub/src/hub/mod.rs 90.47% 0 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

Three faults in the TLS listener:

- It accepted from the moment it was built, before the hub served,
  and each finished handshake then waited in a task of its own with
  no limit. The listener now accepts only inside axum's accept, and
  at most MAX_PENDING_TLS (256) connections are in a handshake or
  waiting for axum. Past that it stops accepting, and the kernel
  backlog queues the rest, as for a plain listener.
- A detached task owned the listening socket, so shutdown could
  return while the port was still bound, up to a second after an
  accept error. The listener now owns the socket, and it closes when
  axum::serve drops the listener, before the server task that the
  hub joins ends.
- ALPN offered h2, which the hub does not build axum support for.
  It offers http/1.1 only.

Also require rustls 0.23.23, the first release with
CertifiedKey::from_der.

Signed-off-by: Ryan Olson <rolson@nvidia.com>
Signed-off-by: Ryan Olson <rolson@nvidia.com>
Signed-off-by: Ryan Olson <rolson@nvidia.com>
Signed-off-by: Ryan Olson <rolson@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant