Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion .github/workflows/exact-llm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,15 @@ jobs:
hashes:
name: hashes (${{ matrix.model }}, ${{ matrix.os }}${{ matrix.routes && format(', {0}', matrix.routes) || '' }})
runs-on: ${{ matrix.os }}
timeout-minutes: ${{ matrix.model == 'SmolLM2-135M' && 90 || 120 }}
timeout-minutes: ${{ (matrix.model == 'SmolLM2-135M' && matrix.routes != 't3') && 90 || 120 }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, ubuntu-24.04-arm, macos-latest, macos-15-intel, windows-latest, windows-11-arm]
model: [SmolLM2-135M, SmolLM2-360M]
# check's default routes. An include entry with a route of its own would overwrite this value, so it becomes
# a job of its own instead of being merged into the default job of the same model and runner
routes: ['']
# SmolLM2-1.7B, one route per job, each inside 120 minutes in run 43759765827: f64 on the six runners, float32
# beside it on ubuntu-24.04-arm alone (53 min; 86 to over 120 elsewhere); f32 and i8 on the non-Windows
# runners, except f32 on macos-latest and macos-15-intel (each ran past 120 minutes: macos-latest in run 43759765827,
Expand All @@ -35,6 +38,11 @@ jobs:
- {model: SmolLM2-1.7B, os: ubuntu-latest, routes: i8, args: "--routes i8 --no-float"}
- {model: SmolLM2-1.7B, os: ubuntu-24.04-arm, routes: i8, args: "--routes i8 --no-float"}
- {model: SmolLM2-1.7B, os: macos-15-intel, routes: i8, args: "--routes i8 --no-float"}
# the t3 route (t3.c, built by the runner's C compiler; not on Windows) on the 135M: Linux
# x86-64 and arm64 with gcc, macOS arm64 with Apple clang; then t3_nomul.py on the kernel check built
- {model: SmolLM2-135M, os: ubuntu-latest, routes: t3, args: "--routes t3 --no-float --no-onnx"}
- {model: SmolLM2-135M, os: ubuntu-24.04-arm, routes: t3, args: "--routes t3 --no-float --no-onnx"}
- {model: SmolLM2-135M, os: macos-latest, routes: t3, args: "--routes t3 --no-float --no-onnx"}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065
Expand All @@ -57,3 +65,8 @@ jobs:
shell: bash
working-directory: proofs/exact-llm
run: python -u exact_llm.py check --model ${{ matrix.model }} ${{ matrix.args }}
- name: The t3 kernel that check built, disassembled; fails on any multiply, multiply-add or divide instruction or library call
if: matrix.routes == 't3'
shell: bash
working-directory: proofs/exact-llm
run: python t3_nomul.py
29 changes: 28 additions & 1 deletion proofs/exact-llm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,31 @@ Each row narrows the exact model's product operands and is compared with the sam

The large difference at 16-bit inputs is one outlier per vector setting its shift: peel 1 brings it to 0.175, peel 2 to 0.03 to 0.06, peel 4 adds nothing; peel 2 also takes 14-bit inputs and weights from 411 to 470. `measure16.py` (rows 1 to 11, against saved float32 runs), `record/exact16-table.txt`; last row `exact_llm.py hash`, `text` and `distance.py`.

## t3: the products by adds and subtracts

A fourth route, `exact16-t3`, run where a C compiler is installed, not on Windows (`check --routes t3`; not in check's default three). Every product of exact-16's forward pass is computed by `t3.c`, whose arithmetic is add, subtract, shift by a constant and bitwise logic: the matrix products in `mm16` (q, k, v, o, gate, up, down, logits, the scores q16·k16, the context p·vh and p·vl), the peeled terms, and the element-wise products (the norm's squares and weight, rotary, the exp and SiLU interpolations, SiLU(gate)·up) through `Exact.mul`. Divisions (52,477 per token) and square roots (61) stay as on the other routes: NumPy's `//` and float `sqrt`, each checked by an integer bracket that NumPy multiplies (q·d ≤ n < (q + 1)·d, s² ≤ n < (s + 1)²). Shifts by a computed amount (the block shift, the row scale) stay NumPy's too.

| signed digits, `t3.c` | for m ≥ 0: `h = m >> 1`, `s = m + h`, `c = h ^ s`; plus `s & c`, minus `h & c`; m = plus − minus, non-adjacent form; a negative value by its magnitude with plus and minus exchanged |
| --- | --- |
| stream | a 16-bit word per instruction, `op << 14 \| i`; per output row, for digit position k = 15 down to 0: `ADD i` for each input with digit +1 at k, `SUB i` for each with −1, then `STEP` (`EMIT` at k = 0) |
| machine | per lane: `s` (int32), `acc` (int64). `ADD i`: s = s + x[i]. `SUB i`: s = s − x[i]. `STEP`: acc = acc + acc; acc = acc + s; s = 0. `EMIT`: `STEP`, then y[row] = acc, acc = 0. 8 lanes at a time |
| weights | each matrix compiled to its stream once (924,918,488 bytes for the 135M: 459,184,492 digits and 16 words for each of 204,672 output rows), kept in `$EXACT_LLM_T3` (default `.cache/t3`), one folder per `t3.c` and model |
| activations | the right operand recoded into digits at each product: k16 for the scores, vh and vl for the context, the smaller-magnitude operand element-wise and for the peeled terms |

| Apple M1, 64 prompts, batch 8, one `check` process | logits | continuations |
| --- | --- | --- |
| `exact16-f64` | `b0cc875f…0b4460` | `a800ebad…7613cc` |
| `exact16-t3`, 8 threads | `b0cc875f…0b4460` | `a800ebad…7613cc` |

MATCH on both, 0 clamped (`record/t3-m1.txt`). The M1 was shared with other jobs during this run, so its seconds are not a measure of either route. Rerun of this code on the M1, with the kernel built and every weight stream compiled afresh from the weights (`record/t3-m1-rerun.txt`):
- `routes_test.py`: 42 cases on four routes, 0 different;
- `check --routes t3`: exact MATCH, and the t3 route MATCH with 0 clamped;
- `t3_nomul.py` on that kernel: no multiply or divide in 517 instructions. Per token at the last position (context 64, `t3_count.py`, the kernel's counters): 459,184,492 digit adds and subtracts in the weight products, as the digit census predicts, and 6,549,504 Horner adds; 1,408,559 and 6,442,304 in the peeled terms; 11,875,560 and 1,658,880 in the attention (scores 5,007,459, p·vh 900,330, p·vl 5,967,771, Horner 552,960 each); 1,271,526 and 4,451,136 element-wise; 492,841,961 adds and subtracts in all. `t3_nomul.py`: the kernel's disassembly holds no multiply, multiply-add or divide instruction and calls no library multiply or divide (Apple clang, arm64: 0 of 517 instructions, 502 in the five functions of `t3.c` and 15 in its call stubs). It does report the `mul` in `t3.c` built without its `KEEP` barriers, and the calls `__mulsi3` (a multiply built for rv32i) and `___divti3` (a 128-bit divide on arm64 macOS). `t3_rv32.c`: the stream machine, freestanding, compiled for rv32i (no M extension) by LLVM: 45 instructions, none a multiply, no library multiply; compiled only, not run.

On the runners, the workflow runs `check --model SmolLM2-135M --routes t3 --no-float --no-onnx`, then `t3_nomul.py` on the kernel that check built, on three runners: ubuntu-latest (x86-64, gcc), ubuntu-24.04-arm (arm64, gcc) and macos-latest (arm64, Apple clang), 120 minutes each. On every Linux and macOS runner, `routes_test.py` also builds `t3.c` and tests it against int64 products.

Known limits of t3: `t3_run` does not check what `pthread_create` returns, so a thread that fails to start leaves its rows unset, which `check` prints as DIFFERENT; an input index is 14 bits, so a product has at most 16,383 inputs (for the context products p·vh and p·vl, 16,383 positions).

## Larger models

One code path: the sizes come from each model's pinned `config.json`, the clamps by rule from its own weights (above), the bound per route from the width. `python exact_llm.py check --model SmolLM2-360M` runs exact-16 on its three routes, float32 and onnxruntime.
Expand Down Expand Up @@ -360,7 +385,9 @@ Commands:
| `expected-16-SmolLM2-360M.sha256`, `record/exact16-360m-m1.txt`, `record/ci-43754428621.txt` | the 360M's pair; every line of its M1 run; every line of the twelve jobs (135M and 360M on six runners) |
| `expected-16-SmolLM2-1.7B.sha256`, `record/ci-43759765827.txt` | the 1.7B's pair; every line of its 14 jobs (one route each), with each job's end and wall time |
| `record/peaks-16.json`, `record/peaks-16-SmolLM2-360M.json` | each site's peak, clamp, margin and clamped count, from `check --peaks` |
| `routes_test.py` | the three routes against int64 products |
| `routes_test.py` | the three routes against int64 products; t3 too where a C compiler is installed, not on Windows, with every 16-bit value through the stream and 200,081 element-wise pairs |
| `t3.c`, `t3_nomul.py`, `t3_count.py`, `t3_rv32.c` | the t3 kernel; its disassembly checked for multipliers; its counters per token; the stream machine for rv32i |
| `record/t3-m1.txt`, `record/t3-m1-rerun.txt`, `record/t3-count.json` | every line of the t3 check on the M1, and of its rerun with the kernel and streams built afresh; the counters per token from `t3_count.py` |
| `spare_row.py`, `spare_row_test.py`, `record/spare-m1.txt`, `record/spare-campaign.json` | exact-16 with every product and look checked, the fault campaign; its test on synthetic operands; every line of the M1 runs; every fault of the campaign |
| `count_ops.py`, `report_census.py`; `count_spare.py`, `report_spare.py`, `record/census-spare.json`, `record/census-spare.md` | the census of every NumPy operation per token, sorted into steps and looks: of exact and exact-16; of exact-16 with and without the spare-row checks, and its tables |
| `measure16.py`, `record/exact16-table.txt` | the width table, one row per run |
Expand Down
Loading
Loading