Velo

Velo is a distributed messaging library for Rust. It gives active messages, typed streams, distributed events, large-payload transfer (with RDMA over UCX), and work queues over pluggable transports. It has peer discovery and Prometheus metrics built in.

NOTE: Velo is experimental. Its design, development, and tests are still in progress, and the APIs are not stable. Do not use Velo in production.

Documentation: the Velo book. The source is in docs/src/.

Typed streams support graceful stop, cancellation, and ordered finalization, including prebound tickets. See Stream lifecycle.

A producer can wait until velo has handed a stream's output on: StreamSender::retired() resolves to Sent or Discarded. TCP and UDS write the frames that they admitted before they close at graceful shutdown. See Know when the output has left, and Known limitations for what Sent does not cover yet.

Streams of bytes that the application already serialized can skip msgpack on both ends. StreamSender<Bytes>::send_bytes and StreamAnchor<Bytes>::next_bytes use the same frames as send and next, and the consumer gets each item as a slice of the received buffer, not a copy. See Byte items.

Velo 0.19 gives Velo and Messenger explicit drop lifetimes, removes value-level Messenger cloning, and removes duplicate dispatch and cancellation APIs. See Migrate ownership, dispatch, and cancellation.

Velo 0.21 makes Velo a handle. VeloBuilder::build() returns Velo, not Arc<Velo>. To share it, clone it. See Migrate to the Velo handle.

Use Velo::shutdown(policy) to close resources and join owned tasks while Tokio keeps running. The instance and its handles stay valid after shutdown, and shutdown does not release them from memory. Accepted requests remain counted until their replies are submitted, including error replies. See Shutdown and drain.

Call Velo::unregister_peer when a peer leaves. A restarted peer has a new instance id, so without the call a long-lived node keeps state and connections for every peer it ever served. See Forget a peer.

Batched streams give each stream a window of 32 records in flight by default. A larger window lets a saturated consumer node queue a new stream's first record behind every other stream's backlog. See Change the credit window.

Crates

Crate	For	Contents
velo	Application authors	The runtime: messaging, streaming, rendezvous, events, queues, discovery, all in-tree transports (TCP, UDS, NATS, gRPC, ZMQ, UCX, QUIC), metrics
velo-ext	Authors of out-of-tree plugins	The stable trait surface: Transport, FrameTransport, PeerDiscovery, ServiceDiscovery, TransportObservability, and the types they use
velo-hub-ext	Programs that speak the hub's wire protocol without the velo runtime	The hub's wire protocol on velo-ext alone: paths, request and response bodies, errors, and ids. velo-hub re-exports it as velo_hub::protocol. Not published yet.
velo-hub	Authors of a hub service, and programs that use a hub	A hub (server): peer and service discovery, registration, and liveness for a velo application, plus hosted extensions. Its etcd store (etcd): registrations that survive a hub restart. Its client (client): registration, peer and service discovery through the hub, and calls to extensions. Not published yet.

Application authors depend on velo only. velo-hub builds one service that is the peer discovery for an application and hosts extensions. It takes the workspace version and requires velo and velo-hub-ext with a caret on it. See The hub, Workspace crates, and Versioning.

Streams to one peer can share its connection through the messenger mux. What the mux costs a serving frontend, and how that was measured, is in Response-plane performance.

QUIC and TCP can keep several lanes, each its own connection, to one peer. See TCP lanes and QUIC lanes.

Streams through the mux spread over those lanes, and each stream stays on one lane. If the caller gives a key (attach_anchor_keyed, prebind_anchor_keyed), the receiver places the stream by that key, so streams with one key to one receiver share a lane. The key is a placement hint: the mux never orders records across streams. Without a key, the receiver uses the lane with the least load. See Lanes.

Build with default-features = false to omit distributed events, work queues, and built-in discovery services. The default services feature keeps those APIs available. Use .mux_only() to avoid an extra stream listener when all peers support the messenger mux. See Services and mux-only mode.

Quick start

cargo add velo
cargo add serde --features derive
cargo add tokio --features macros,rt-multi-thread
cargo add anyhow

use std::sync::Arc;
use serde::{Deserialize, Serialize};
use velo::transports::tcp::TcpTransportBuilder;
use velo::{Handler, TypedContext, Velo};

#[derive(Serialize, Deserialize)]
struct AddRequest { a: i64, b: i64 }

#[derive(Serialize, Deserialize)]
struct AddResponse { sum: i64 }

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let node_a = Velo::builder()
        .add_transport(Arc::new(TcpTransportBuilder::new().build()?))
        .build()
        .await?;
    let node_b = Velo::builder()
        .add_transport(Arc::new(TcpTransportBuilder::new().build()?))
        .build()
        .await?;

    node_b.register_handler(
        Handler::typed_unary_async("add", |ctx: TypedContext<AddRequest>| async move {
            Ok(AddResponse { sum: ctx.input.a + ctx.input.b })
        })
        .build(),
    )?;
    node_a.register_peer(node_b.peer_info())?;

    let resp: AddResponse = node_a
        .typed_unary::<AddResponse>("add")?
        .payload(&AddRequest { a: 3, b: 4 })?
        .instance(node_b.instance_id())
        .send()
        .await?;
    assert_eq!(resp.sum, 7);
    Ok(())
}

See Getting started for the feature flags.

Stream reservations default to 120 seconds before ordinary attachment and 180 seconds for a messenger-mux bind to receive its first batch. VeloBuilder::reservation_config configures these lifetimes. Applications can batch held reservations through AnchorManager::renew_reservations; see reservation renewal.

Build and test

cargo build --all-features
cargo test --all-features --all-targets   # NATS on localhost:4222 and etcd on :2379 for those tests
cargo clippy --all-features --no-deps --all-targets -- -D warnings
cargo fmt --check
cargo machete
bash scripts/check-semver.sh
bash scripts/build-book.sh                # needs mdbook, mdbook-mermaid, mdbook-linkcheck

The zmq feature needs cmake. The ucx feature needs the libibverbs-dev and librdmacm-dev headers. See Testing.

A change on a hot path also needs a test that counts its cost. See Review for performance.

On macOS the whole workspace builds with --all-features, but the ucx transport is Linux only. See Platforms.

Contributing

See CONTRIBUTING.md and Documentation.