webdataset-tar

webdataset-tar provides the synchronous, storage-independent mechanics for working with uncompressed WebDataset TAR shards:

• index regular-file payload byte ranges without reading payloads;
• group consecutive key.suffix members with canonical WebDataset naming;
• plan deterministic, prefix-stable draws across a collection of shard bytes;
• resynchronize a range read to aligned TAR headers;
• select a primary member and its consecutive sample parts; and
• repack selected members as an ordinary, smaller TAR archive.

The crate has no async runtime or storage-client dependency. Callers decide how to list objects and fetch ranges; this crate handles the archive and sample semantics.

Example

use webdataset_tar::build_tar_member_index;

let archive_bytes = std::fs::read("shard-000000.tar")?;
let index = build_tar_member_index(archive_bytes.as_slice())?;

for sample in index.samples()? {
    println!("{} has {} parts", sample.key, sample.parts.len());
    for part in sample.parts {
        println!("  {}: {:?}", part.name, part.byte_range());
    }
}

# Ok::<(), Box<dyn std::error::Error>>(())

WebDataset compatibility

Sample parsing matches WebDataset's reference base_plus_ext and group_by_keys behavior:

• the key ends at the first dot in the final path component, so scene/clip.rgb.jpg is key scene/clip with suffix rgb.jpg;
• only consecutive members with the same key form one sample;
• suffix matching is case-insensitive;
• duplicate suffixes within a sample are rejected; and
• structural __name__ metadata trees and names without key.suffix are not samples.

Full indexing and window resynchronization resolve local PAX metadata and GNU long names through the tar crate. Resynchronization can only preserve an extension record when that record is present in the supplied window. A returned member header may be inside the window even when its payload continues beyond the window; callers can fetch that payload with a second range read.

repack_sparse_tar uses “sparse” to mean a selected subset of members. Its output is a normal uncompressed TAR, not the GNU sparse-file format.

Deterministic draws

plan_byte_draws hashes (seed, draw_index) and maps the leading 128 digest bits into the concatenated shard byte space. Increasing the requested count extends the same sequence, which makes sample identities reproducible. The modulo mapping has negligible bias, and no bias when the total byte count divides 2^128; the exact mapping is intentionally stable across releases.

Minimum supported Rust version

Rust 1.85 or newer is required.

License

Apache-2.0.