What if an LLM's weights weren't stored, but generated by a function? Gradus replaces the weight tables of a transformer (Qwen2.5) with a compact generator: a frozen 152M LM (AILO) acting as a shared "weight factory", driven by a VQ codebook and semantic coordinates.
block index (9 bit) ──► codebook atom (64 floats)
│
context: tensor type (q/k/v/o/gate/up/down), layer, block position
│
▼
AILO 152M (FROZEN decoder)
│
▼
32×32 weight block of "Qwen"
| Experiment | Result |
|---|---|
| Pure function: coordinates → weight, no latent | cosine ≈ 0.05 ❌ weights are not a smooth function of coordinates |
| Weight copy (MSE), frozen AILO, 2× latent | cosine 0.996 ✅ AILO is a rich decoding basis |
| Free latents, 8×, wikitext held-out | ppl 47.6 → 85.1 ❌ free latents don't compress, they store |
| VQ codebook K=256 (274× latent-side) | ppl 47.6 → 70.6 — better while compressing 34× more |
| + gradient-accumulation batch fix | ppl 47.6 → 53.35 (+12%) ✅ GENERALIZES on unseen text |
| Full model (all attention+MLP projections) | run in progress (checkpointed, days-long) |
The key insight: don't train the generator to copy weights (information theory
forbids it at high ratios) — train it so the reconstructed model keeps its
perplexity (--objective task). Compression comes from cross-block redundancy
captured by the shared codebook, not from matrix smoothness.
Standard autograd crashes (device TDR) on AMD/DirectML. So Gradus ships its own engine: forward and backward written by hand for every op — Linear, LayerNorm, RMSNorm, causal attention + RoPE, GQA, SwiGLU, softmax, embeddings, cross-entropy, VQ with straight-through estimator, Adam with gradient clipping. Every backward is gradient-checked against PyTorch (errors ~1e-6 … 1e-9).
Result: a Radeon RX 6750 XT trains transformers with no CUDA, no ROCm, and no framework autograd. Both the generator (AILO) and the full target model (manual Qwen2: 24 layers, GQA 14:2, verified vs HF autograd at 4e-5) run on it.
python -m venv .venv
.venv\Scripts\activate # Windows
pip install -r requirements.txt
# AMD GPU (Windows): pip install torch-directml (then: python scripts/prepare_ailo_backbone.py)python run_gui.py"FWE Engine" section: configure → Start training (live log, checkpoint every 25 steps, resume after crash/reboot) → chat with the model from its latest checkpoint while it is still training.
# engine (sovereign, AMD-capable): task-fidelity training with checkpoints
python -m gradus engine-fwe --objective task --qwen-manual --dataset wikitext \
--include _proj --vq 512 --latent-dim 64 --steps 600 --lr 2e-4 --batch 8 \
--device dml --run-dir runs/engine-full --save-every 25 # --resume <ckpt> to continue
# chat with the reconstructed model from a checkpoint (works during training)
python -m gradus engine-chat --ckpt runs/engine-full/engine_ckpt.pt --prompt "Hello!"
# engine self-tests (gradient checks + training bricks 1..3)
python -m gradus engine-test --brick 3 --device dml
# classic PyTorch pipeline (CUDA/CPU): weight-copy & analysis
python -m gradus info|analyze|train|reconstruct|eval|chat|demo| Component | Size |
|---|---|
| AILO decoder (frozen, reusable across models) | 152.3M |
| Adapters (in_proj + head + type/layer embeddings) | 1.3M |
| VQ codebook + indices (the Qwen-specific part) | ~0.5 MB |
| Token embedding (not compressed in v1) | 136.1M |
| Deploy total | ~290M vs 494M original (1.7×) |
The per-model payload is ~0.5MB: the decoder is paid once. Scaling the same decoder: ~4× net on 1.5B, ~11× on 7B. Full architecture, training recipe, success criteria and licensing: docs/MODELLO.md.
Research prototype under active development. All numbers in this README are reproducible from the CLI above.
Code: MIT (see LICENSE). AILO: author's own model. Source model: Qwen2.5 (Alibaba, Apache 2.0) — the compressed codebook/indices are a derivative of Qwen weights. Dataset: wikitext-2.