Skip to content

S1-25: MUL_MAT_ID and ADD_ID backward wiring - #19

Open
dillon-blake wants to merge 1 commit into
learning-llamas-basefrom
ticket/S1-25-mul-mat-id-add-id-backward
Open

S1-25: MUL_MAT_ID and ADD_ID backward wiring#19
dillon-blake wants to merge 1 commit into
learning-llamas-basefrom
ticket/S1-25-mul-mat-id-add-id-backward

Conversation

@dillon-blake

Copy link
Copy Markdown
Owner

MoE training is blocked by one missing backward case: MUL_MAT_ID has no case in ggml_compute_backward and falls into the op-level default GGML_ABORT, so any MoE graph dies the moment a gradient is requested.

The two new ops

At the enum tail (rebase hygiene — inserting renumbers every op after it and conflicts across the whole backend matrix):

op gradient
OUT_PROD_ID d(b) — gather the expert matrix each (slot, token) used, push the gradient back through it
OUT_PROD_ID_GRP d(as) — the outer product scattered into the expert slice each (slot, token) selected, accumulated over all of them

Both are gather/scatter with accumulation, which is why neither is expressible in existing ops: the expert axis is a selection, not a broadcast. Constructors and shape contracts only — the CPU kernels are S1-26 / S1-27.

The weight-grad half is not optional. build_lora_mm_id computes mul_mat_id(B, mul_mat_id(A, cur, ids), ids), so the trainable LoRA A/B tensors are the 3D expert operand. An "activations only" backward would silently train nothing.

Two things the ticket got wrong, both found by building it

ggml_out_prod_id(as, grad, ids) is under-determined. d(b) has b's shape, and b's middle dim is not recoverable from as/grad/ids — the forward broadcasts b's columns across slots whenever ids->ne[0] is a multiple of it. Both modes are live in the LoRA MoE graph: the inner mul_mat_id has ne_b1 == 1, the outer has ne_b1 == n_used. So ne_b1 is passed explicitly and the forward's broadcast rule is asserted, rather than assuming 1 and handing a broadcasting graph a wrong-shaped gradient.

CPU supports_op ends in default: return true. A new op with no dispatch case is therefore reported supported, gets scheduled, and hits ggml_compute_forward's default: GGML_ABORT. It would look implemented right up until it killed the process. Both new ops now return false explicitly; S1-26/S1-27 flip them.

The tests were checking nothing

test_add_id and test_mul_mat_id never called ggml_set_param — so grad -o ADD_ID and grad -o MUL_MAT_ID requested no gradients, compared nothing, and printed OK while the ops had no backward at all. (grad -o MUL_MAT_ID reported OK on 16829 cases.)

Both now ask for them:

  • ADD_ID — 16 grad cases now execute and pass. Verified non-vacuous: scaling the VJP by 2 turns 26 cases red.
  • MUL_MAT_ID — 516 grad cases register and report not-supported rather than aborting. The wiring is exercised now, and they turn on with no test change the day a kernel lands.

MoE training is blocked by one missing backward case. MUL_MAT_ID has no case in
ggml_compute_backward and falls into the op-level default GGML_ABORT, so any MoE
graph dies the moment a gradient is requested.

Two new ops at the enum TAIL (rebase hygiene -- inserting renumbers every op after
it and conflicts across the whole backend matrix):

  OUT_PROD_ID      d(b)  -- gather the expert matrix each (slot, token) used and
                            push the gradient back through it
  OUT_PROD_ID_GRP  d(as) -- outer product scattered into the expert slice each
                            (slot, token) selected, accumulated over all of them

Both are gather/scatter WITH accumulation, which is why neither is expressible in
existing ops: the expert axis is a selection, not a broadcast. Constructors and
shape contracts only -- the CPU kernels are S1-26 and S1-27.

The weight-grad half is not optional. build_lora_mm_id computes
mul_mat_id(B, mul_mat_id(A, cur, ids), ids), so the trainable LoRA A/B tensors ARE
the 3D expert operand -- an "activations only" backward would silently train
nothing.

Two things the ticket got wrong, both found by building it:

- ggml_out_prod_id(as, grad, ids) is UNDER-DETERMINED. d(b) has b's shape, and b's
  middle dim is not recoverable from as/grad/ids -- the forward broadcasts b's
  columns across slots whenever ids->ne[0] is a multiple of it. Both modes are live
  in the LoRA MoE graph: the inner mul_mat_id has ne_b1 == 1, the outer has
  ne_b1 == n_used. So ne_b1 is passed, and the forward's broadcast rule asserted.

- CPU supports_op ends in `default: return true`. A new op with no dispatch case is
  therefore reported SUPPORTED, gets scheduled, and hits ggml_compute_forward's
  `default: GGML_ABORT` -- it would look implemented right up until it killed the
  process. Both new ops now return false explicitly; S1-26/S1-27 flip them.

test-backend-ops: test_add_id and test_mul_mat_id never called ggml_set_param, so
`grad -o ADD_ID` and `grad -o MUL_MAT_ID` requested no gradients and printed OK
while having no backward at all. Both now ask for them.

  ADD_ID       16 grad cases now EXECUTE and pass. Verified non-vacuous: scaling the
               VJP by 2 turns 26 cases red.
  MUL_MAT_ID   516 grad cases register and report not-supported rather than
               aborting -- the wiring is exercised now, and they turn on with no test
               change the day a kernel lands.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant