Skip to content

Latest commit

 

History

History
71 lines (58 loc) · 3.66 KB

README.md

File metadata and controls

71 lines (58 loc) · 3.66 KB

vitni-torch (experimental, opt-in)

The 0.1.0a2 preview also includes an opt-in resident full-model VitniModel wrapper for CPU and native Metal. See the full-model guide. Importing vitni_torch leaves normal PyTorch operations and backend choices unchanged. A full model runs only after explicit construction.

Explicit CPU and Apple MPS operators using Vitni's arithmetic contract:

  • canonical_matmul(a, b): contiguous rank-2 float32 matrices.
  • q4_k_linear(x, weights): float32 vector and raw GGUF Q4_K uint8 rows.

CPU computation calls the actual Rust engine through a small C ABI. MPS computation runs the engine's Metal kernels directly on MPS tensors through torch.mps.compile_shader; it does not copy the numerical operation to CPU. Import registers the vitni namespace without replacing PyTorch operators or changing PyTorch's global determinism settings.

Build/install from this repository with Rust/Cargo and Python 3.10+ available:

python -m pip install '.[torch]'
import torch
import vitni_torch

a = torch.randn(4, 256, dtype=torch.float32)
b = torch.randn(256, 8, dtype=torch.float32)
cpu = vitni_torch.canonical_matmul(a, b)
mps = vitni_torch.canonical_matmul(a.to('mps'), b.to('mps')).cpu()
assert torch.equal(cpu.view(torch.int32), mps.view(torch.int32))

This preview requires PyTorch 2.14+ and supports inference only. Detach inputs explicitly; there is no backward formula. Inputs must be contiguous, finite, and on the same supported device. Each floating input must be zero or have magnitude in [2**-32, 2**32], on both devices. Integer bit classification rejects subnormals and out-of-range inputs before arithmetic. Q4_K headers use the ordinary finite GGUF half-scale encodings; nonfinite results are rejected. The MPS kernel supports at most 64 chunks of 8192 input elements. Non-contiguous inputs, other dtypes, CUDA, batched matmul, and automatic replacement of an arbitrary PyTorch model are outside this preview's scope.

The package registers fake kernels for both operators. Local tests exercise torch.library.opcheck, torch.compile with eager/AOT-eager/Inductor backends, and export/reload on CPU and MPS. These registrations cover these operators; they do not automatically convert an existing model.

The canonical output is defined by Vitni's numerical rules and need not match torch.matmul's native arithmetic. Exact CPU/MPS equality must be tested with integer bit views, not allclose. Registration/FakeTensor/compilation checks are separate from those numerical checks. MPS compilation probes separate rounding, explicit FMA and signed zero and fails if the contract is not honored. It also reports a subnormal probe: the tested PyTorch MPS shader path flushes subnormals, which is why this preview enforces the restricted domain. CPU calls reject ambient flush-to-zero/non-nearest settings.

With these input bounds, matrix products lie between 2**-64 and 2**64. Rounded products share a grid no finer than 2**-87, so cancellation cannot create a nonzero subnormal intermediate; at most 2**19 terms cannot overflow. Finite Q4_K dequantization uses half values and bounded integer scales; its nonzero weight grid is at least 2**-24, and the same activation bounds keep its products and sums in the normal float32 range. This is a deliberately restricted operator contract, not support for arbitrary float32 bit patterns.

Finite-value validation can synchronize MPS work. Include this and transfer, allocation and first-call compilation costs when measuring an application. The project's approximately 2.47x TinyLlama/llama.cpp Metal forward-time ratio does not describe these general PyTorch operators; benchmark them separately.