S1-25: MUL_MAT_ID and ADD_ID backward wiring - #19
Open
dillon-blake wants to merge 1 commit into
Open
Conversation
MoE training is blocked by one missing backward case. MUL_MAT_ID has no case in
ggml_compute_backward and falls into the op-level default GGML_ABORT, so any MoE
graph dies the moment a gradient is requested.
Two new ops at the enum TAIL (rebase hygiene -- inserting renumbers every op after
it and conflicts across the whole backend matrix):
OUT_PROD_ID d(b) -- gather the expert matrix each (slot, token) used and
push the gradient back through it
OUT_PROD_ID_GRP d(as) -- outer product scattered into the expert slice each
(slot, token) selected, accumulated over all of them
Both are gather/scatter WITH accumulation, which is why neither is expressible in
existing ops: the expert axis is a selection, not a broadcast. Constructors and
shape contracts only -- the CPU kernels are S1-26 and S1-27.
The weight-grad half is not optional. build_lora_mm_id computes
mul_mat_id(B, mul_mat_id(A, cur, ids), ids), so the trainable LoRA A/B tensors ARE
the 3D expert operand -- an "activations only" backward would silently train
nothing.
Two things the ticket got wrong, both found by building it:
- ggml_out_prod_id(as, grad, ids) is UNDER-DETERMINED. d(b) has b's shape, and b's
middle dim is not recoverable from as/grad/ids -- the forward broadcasts b's
columns across slots whenever ids->ne[0] is a multiple of it. Both modes are live
in the LoRA MoE graph: the inner mul_mat_id has ne_b1 == 1, the outer has
ne_b1 == n_used. So ne_b1 is passed, and the forward's broadcast rule asserted.
- CPU supports_op ends in `default: return true`. A new op with no dispatch case is
therefore reported SUPPORTED, gets scheduled, and hits ggml_compute_forward's
`default: GGML_ABORT` -- it would look implemented right up until it killed the
process. Both new ops now return false explicitly; S1-26/S1-27 flip them.
test-backend-ops: test_add_id and test_mul_mat_id never called ggml_set_param, so
`grad -o ADD_ID` and `grad -o MUL_MAT_ID` requested no gradients and printed OK
while having no backward at all. Both now ask for them.
ADD_ID 16 grad cases now EXECUTE and pass. Verified non-vacuous: scaling the
VJP by 2 turns 26 cases red.
MUL_MAT_ID 516 grad cases register and report not-supported rather than
aborting -- the wiring is exercised now, and they turn on with no test
change the day a kernel lands.
This was referenced Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MoE training is blocked by one missing backward case:
MUL_MAT_IDhas no case inggml_compute_backwardand falls into the op-level defaultGGML_ABORT, so any MoE graph dies the moment a gradient is requested.The two new ops
At the enum tail (rebase hygiene — inserting renumbers every op after it and conflicts across the whole backend matrix):
OUT_PROD_IDd(b)— gather the expert matrix each(slot, token)used, push the gradient back through itOUT_PROD_ID_GRPd(as)— the outer product scattered into the expert slice each(slot, token)selected, accumulated over all of themBoth are gather/scatter with accumulation, which is why neither is expressible in existing ops: the expert axis is a selection, not a broadcast. Constructors and shape contracts only — the CPU kernels are S1-26 / S1-27.
The weight-grad half is not optional.
build_lora_mm_idcomputesmul_mat_id(B, mul_mat_id(A, cur, ids), ids), so the trainable LoRA A/B tensors are the 3D expert operand. An "activations only" backward would silently train nothing.Two things the ticket got wrong, both found by building it
ggml_out_prod_id(as, grad, ids)is under-determined.d(b)hasb's shape, andb's middle dim is not recoverable fromas/grad/ids— the forward broadcastsb's columns across slots wheneverids->ne[0]is a multiple of it. Both modes are live in the LoRA MoE graph: the innermul_mat_idhasne_b1 == 1, the outer hasne_b1 == n_used. Sone_b1is passed explicitly and the forward's broadcast rule is asserted, rather than assuming 1 and handing a broadcasting graph a wrong-shaped gradient.CPU
supports_opends indefault: return true. A new op with no dispatch case is therefore reported supported, gets scheduled, and hitsggml_compute_forward'sdefault: GGML_ABORT. It would look implemented right up until it killed the process. Both new ops nowreturn falseexplicitly; S1-26/S1-27 flip them.The tests were checking nothing
test_add_idandtest_mul_mat_idnever calledggml_set_param— sograd -o ADD_IDandgrad -o MUL_MAT_IDrequested no gradients, compared nothing, and printedOKwhile the ops had no backward at all. (grad -o MUL_MAT_IDreportedOKon 16829 cases.)Both now ask for them:
ADD_ID— 16 grad cases now execute and pass. Verified non-vacuous: scaling the VJP by 2 turns 26 cases red.MUL_MAT_ID— 516 grad cases register and report not-supported rather than aborting. The wiring is exercised now, and they turn on with no test change the day a kernel lands.