Skip to content

About

Convert model tensors in real time across formats and architectures

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

8 Commits

Folders and files

Repository files navigation

TensorBridge

A reliable, dependency-free C++17 framework for converting model tensors in real time across formats and architectures — safetensors ⇄ GGUF, dtype changes (bf16/fp16/fp32), and the architecture transforms that actually block model interop: MoE expert fuse/unfuse and tensor-name remapping.

It exists because of a concrete failure: two checkpoints of the same 35B model couldn't be combined — one stored experts fused (…experts.gate_proj shape [E,I,H], mtp./top-level names), the other unfused (…experts.{e}.gate_proj shape [I,H], under a model.language_model. wrapper). Only ~2% of tensor names matched. TensorBridge normalizes both to one convention so they line up.

Design

 codecs (format)            canonical              adapters (architecture)
 ┌────────────┐          ┌───────────┐          ┌──────────────────────┐
 │safetensors │──read──► │TensorDesc │ ──plan──►│ rename / fuse / unfuse│──►stream out
 │   GGUF     │  (mmap)  │ dtype,shape│          │  (COPY/SLICE/FUSE)   │   (safetensors)
 └────────────┘          │ byte-span │          └──────────────────────┘
                         └───────────┘
  • Canonical TensorDesc — name, DType, shape, and a zero-copy [ptr,len) byte span into an mmap. Everything speaks this; nothing copies to translate.
  • Codecs parse a container's metadata into TensorDesc and stream bytes back out. Reads are mmap'd (bounded RSS — a 65 GB shard costs ~0 resident until touched); writes are two-pass (declare → stream), one tensor resident at a time.
  • Adapters emit a TransformPlan: an ordered list of output tensors, each a recipe over the inputs — COPY one, SLICE a sub-tensor out of a fused axis, or FUSE N tensors into one. The executor streams the plan with bounded memory.
  • dtype engine routes every float conversion through IEEE-754 fp32 with round-to-nearest-even; integer/bool tensors pass through byte-exact (never reinterpreted). Adding a float type is one <->fp32 pair.

Reliability

  • Every offset is bounds-checked against real file size; every dense tensor's byte length is checked against dtype × shape. Truncated/corrupt shards fail loudly at open (the exact bug that motivated this — a bad download).
  • make test (4 suites): dtype conversions validated bit-for-bit against PyTorch (100.0000% element match, bf16→fp16 on real weights); byte-exact MoE fuse→unfuse→fuse round-trip; GGUF read→write byte-identical round-trip; streaming cast/passthrough/offset-reorder.
  • Real-file validation: Q4_K/Q6_K dequant matches gguf-py element-for-element; GGUF writer reproduces a real 4.9 GB / 8B Q4_K_M file byte-for-byte; wire-stream output is byte-identical to batch convert on a 3 GB / 2603-tensor shard.
  • Portable, zero deps: builds and passes all tests on x86_64 (g++ 12) and aarch64 (g++ 13, DGX Spark) from identical source.

CLI

tbridge inspect <file>            [--full]                 # safetensors or GGUF (auto-detect)
tbridge convert <in> <out>        [--dtype F16|BF16|F32]   # any->safetensors; GGUF quant is dequantized
tbridge arch    <in> <out>        --op unfuse-experts|fuse-experts [--dtype ..]
tbridge rename  <in> <out>        --rule 'pat=>rep' [--rule ..]   # regex name remap
tbridge stream  [--dtype F16|BF16|F32]                     # safetensors stdin -> stdout (convert on the wire)
tbridge togguf  <in.gguf> <out.gguf>                       # GGUF read -> write (faithful re-encode)
tbridge dumptensor <file> <name>  [--n K]                  # dequant/load a tensor to fp32 (validation)

Convert during a network copy (bounded memory, one tensor resident, no temp file):

tbridge stream --dtype F16 < model.safetensors | ssh host 'cat > model_f16.safetensors'

Example — reconcile the 35B pair, then it can be merged:

tbridge arch   fused.safetensors  unfused.safetensors --op unfuse-experts
tbridge rename unfused.safetensors normed.safetensors  \
    --rule 'model\.language_model\.=>model.' --rule 'mtp\.=>model.'

Formats

Format Read Write Notes
safetensors ✅ ✅ mmap + streaming (file or stdin/stdout); F64/F32/F16/BF16, all int/bool
GGUF ✅ ✅ header+all-typed-metadata+tensors; float first-class; Q8_0/Q4_0/Q4_K/Q6_K dequantized to float, other quant types opaque passthrough

Build

make            # -> ./tbridge
make test       # dtype + adapter + gguf-writer + stream suites

Extension points

  • More quant types (Q5_K/Q3_K/Q2_K/…): add the block layout to quant.hpp; it plugs into the DType block path and the GGUF→float bridge automatically.
  • safetensors→GGUF (quantizing) writer: the GGUF writer + typed-KV model are in place; what remains is synthesizing arch metadata and a float→quant encoder.
  • New arch adapters: emit a TransformPlan; the streaming executor is generic.

About

Convert model tensors in real time across formats and architectures

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages