Problems found while working on bloomery, by severity #637

These are problems I ran into while working on bloomery, collected in one place instead of an issue each. Each item stands on its own and can be split out, fixed or closed by itself. The line links in the first item point at 345ac8d9 and the others at be69eb0c, both on main.

How they are sorted. High gives people running bloomery a wrong or unrepeatable answer or a refused load, or blocks a build or contributor step with no workaround. Medium is a contributor tool or gate that stops or gives a wrong verdict every time under the conditions given, with a workaround. Low is everything else: test coverage, failures that need an unusual setup, and one-line fixes. An item that says I'm fixing it has a pull request on the way that links here; every other item is open for anyone.

High를 제외한 이슈들은 같은 파일을 건드리는 작업을 할 때 수정합니다. 혹시 그 전에 작업 하시는 분은 댓글 하나만 부탁드려용!

High

• main did not build at 345ac8d9: gemm_reads_blocks32 had no arm for GemmEntry::Q5_0 after raon1: the prompt in wide units on the common GEMM, KDA and route owners (#616) #627 landed on quant, card leg: Qwen3.5/3.6's placed card leg runs the common leg's Q8_0, Q6_K, IQ3_XXS and IQ4_XS experts (#75) #629. Fixed in a0d5dec43.

• With the default expert stream, the same greedy command gave different tokens after each restart of the same build. Fixed in gpu: a split pick is a function of the prompt: the expert stream prices its lane by where the probe's experts sit (lane_rate), its probe's timing printed beside it, and an admit's return by the walk's last card layer alone; gate_nvtier's sittings clause loads the paged plan twice, once beside a host copy load #647.

Medium

• With the MTP draft on, a Qwen3.8 prompt call can fill the draft's key and value store from hidden rows that never got the last layer's combine.

  The MTP draft (the small speculative head that proposes tokens for the target) keeps its own key and value store. While a prompt runs, it reads the target's hidden row for every prompt row. Only rows that the head's mix covered are final. In two cases some rows are not covered, so the same prompt leaves a different draft store than it does when its rows run as one ubatch walk with the head. The target is unaffected: next ids, logits, KV, GDN and conv state were bit identical in every failing run, and only the draft's store differs, at the positions that read the uncovered rows. Case 1 hits a default Qwen3.8 serve on a load with host-resident routed experts, because the draft is on by default there and the split walk (BLOOMERY_SPLIT_WALK unset) cuts a wide enough ubatch in two units. Loads with every routed expert on the card never split. Case 2 needs BLOOMERY_QWEN3_UBATCH below 4,096, or a caller that runs a prompt longer than one ubatch without checkpoint marks, since the server cuts prompt calls every 4,096 positions and the default ubatch is 4,096 rows. In my runs the four drafted passes after the call proposed and committed the same tokens either way, so I saw no acceptance change; acceptance and decode speed at scale were not measured.

  What the code does. The ubatch path walks each ubatch and runs the head only on the call's last one, last.then_some(head) (body38.rs:3951-3967). unit_widths cuts a walk of m rows into ⌈m/2⌉ and ⌊m/2⌋ when SplitWalk::Rule and splitwalk::pays say so, which needs at least one host-resident routed expert (wide38.rs:366-387). Gemm38::walk keeps only the last unit for the head (wide38.rs:2451, wide38.rs:2500), and Gemm38::head runs the head site's mix over that unit's columns with Before::Combine (wide38.rs:2507-2534). Before::Combine applies the previous site's combine to the streams in place, adding the weighted layer output to res (hc_gated.rs:1122-1124); for the head site that is the last layer's combine. The prompt call feeds the draft after every ubatch, whether or not that ubatch ran the head (body38.rs:3965-3967). Policy::unit and Policy::warm walk the draft over the unit's rows, each store row at p reading the target's hidden row at p - 1 (policy.rs:379-442). That read is the ubatch arena's final streams buffer (body38.rs:4067-4079), copied by the draft walk (mtp38.rs:1560-1563). TargetRows states the precondition the two cases break: the walk must have run its head, whose mix applies the last layer's combine (mtp38.rs:782-786).

  Why the stores differ. A row the head's mix did not cover keeps the streams from before the last layer's feed-forward output was combined in, and the draft reads it as if it were final. Rows a covered walk leaves are final, so only the store rows that read uncovered rows differ. I inferred this from the code; it matches both measured differences position for position, but I have no kernel-level trace of it. In the split case, a 1,699-row walk from position 301 splits into 850 and 849 rows. The first unit holds rows 301 to 1150, which store rows 302 to 1151 read, so the difference starts at position 302 and covers about 848 positions. In the earlier-ubatch case, 1,024-row ubatches run the same walk as 1,024 and 675 rows with the head only on the second, so the difference runs from position 302 to position 1325.

  Where it came from. The draft's store feed after every prompt unit came in with f1547441c ("app, gpu, gpu-gates, levers, model: Qwen3.8's MTP speculative window (mtpwin, MTP chain 4)", midagedev, 2026-09-30). The ubatch path already ran the head on the last ubatch only at that commit, so the earlier-ubatch case dates from there. The TargetRows precondition text came one commit before in a3453c2d1 (same author and day). The feed moved into mtp/policy.rs in 1d03c11fd (2026-10-08) with the same walks. The split walk is 275074ded (midagedev, 2026-10-11), whose subject begins "Qwen3.8's prompt walks a P ≥ the width rule's ubatch as two units". Its gates pin the target's bits under the split, and none compares the draft's store.

  Reproduction. Slot 1 holds a 300-id request. Its continuing call of 1,699 ids then runs twice: once as one call, and once as four calls of 566, 566, 566 and 1 ids, each of which runs one ubatch with the head. Slot 0's drafted pass runs between the parts. Compare next ids, drafted passes, the draft's store, the first step's logits, the live stores and the checkpoint positions. On the Qwen3.8 fixture with its Q8_0 MTP draft and host-resident experts, with a 3,072-position context so the one call is one ubatch, the ids, passes, step logits and live stores matched and the draft's store did not. The four stores had equal length, and the first differing key and value were both at index 154624 (position 302, at 512 values a position), with 434,128 differing key values and 434,137 differing value values. Under the default rule the one call's store differed from the other three, which equal each other: the one call with the split walk off, the parts with it off, and the parts under the rule. The 566-row parts are too narrow for the rule to split. For the earlier-ubatch case, run the same comparison with BLOOMERY_QWEN3_UBATCH=1024: the first differing key stays at index 154624, the last is at 678911 (position 1325, the last store row that reads the first ubatch), and 523,013 of the 524,288 key values differ. With the default ubatch and the split walk off, the comparison matches on the same card, and it matches on an RTX 5090 machine with the rule on, where the unit count was not printed.

  Fix direction: give the draft final rows for every row it reads. After a ubatch walk that feeds the draft, apply the last layer's combine to every column the head's mix did not cover: the earlier units of a split walk, and all columns of a ubatch that runs without the head. The standalone combine enqueue_combine (hc_gated.rs:1488) with each unit's combine weights would do it without the head's projection. Another option is to run the head site's mix over all columns and read only the last row for the head. Add a test that compares the draft's store after one prompt call walked as several ubatches, and under BLOOMERY_SPLIT_WALK=every, against the same call walked as one unit.

Low

• --n-l in tools/ref/window-union.py and router-coverage.py bursts accepts negative counts, a layer named twice, a negative spread total and a repeated option, and reports a placement it was not given.

  Someone sizing a card placement from router traces passes the plan's per-layer card counts as --n-l, and router-coverage.py bursts reads it the same way. Several malformed specs are taken without an error, and the tool prints numbers for a placement that is not the one typed. 0:-1 gives layer 0 a card count of -1; the report header says card -1 experts (n_l -1..0) and every ratio is the one for zero card experts, and --policy strata hands -1 to the residency replay as that layer's card size. 0-1:4,1:9 names layer 1 twice and the later value silently wins. spread:-5@0-1 gives every layer 0 instead of refusing the negative total. --n-l 0:1 --n-l 0:5 keeps only the last spec, and bursts stores its options in a dict and does the same. --k 2,2 prints the k = 2 row twice. Only a count above the expert total is refused, and that check sits in card_sets, not in the reader. These are analysis tools; no engine, gate or served path reads them.

  What the code does. The reader is window-union.py L69-L93. L76 converts the spread total with int(total), and the while left > 0 loop at L79 never runs for a negative total. L92 assigns n_l[l] = int(n) with no bound and no check that the layer was already named. The only bound is card_sets L96-L104, which refuses n > n_expert (L102) but not n < 0; set(range(-1)) is empty, so a negative count reads as zero. main L281-L294 overwrites spec on each --n-l (L283) and keeps --k values as given (L294). In router-coverage.py, bursts' option loop L1038-L1046 overwrites opts[a].

  Why it happens. parse_n_l was written as a plain parser for well-formed plan lines and left all range checking to card_sets, which checks only the upper bound. Both came in with 051d51e1c ("tools/ref: window-union.py", 2026-09-24). router-coverage.py bursts started reading --n-l through the same two functions in 68c60a6d6 ("gpu-gates, tools/ref, docs/cards: the PCIe and residency probes (probes38)", 2026-09-28).

  Reproduction, from a checkout of be69eb0c, with a two-layer, 16-expert, top-2 set built by the tool's own test helper:

  d=$(mktemp -d)
  python3 - "$d" <<'EOF'
  import importlib.util, sys
  spec = importlib.util.spec_from_file_location("wu", "tools/ref/window-union.py")
  wu = importlib.util.module_from_spec(spec); spec.loader.exec_module(wu)
  wu._fake_set(sys.argv[1], "set", {0: [(3, 7)] * 8, 1: [(3, 7)] * 8}, 16, 2)
  for s in ("0:-1", "0-1:4,1:9", "spread:-5@0-1"):
      print(f"parse_n_l({s!r}) -> {wu.parse_n_l(s, [0, 1])}")
  EOF
  python3 tools/ref/window-union.py "$d/set" --n-l 0:-1 --k 2 | sed -n 3p
  python3 tools/ref/window-union.py "$d/set" --n-l 0:-1 --k 2 --policy strata | sed -n 3p
  python3 tools/ref/window-union.py "$d/set" --n-l 0:1 --n-l 0:5 --k 2 | sed -n 1p
  python3 tools/ref/window-union.py "$d/set" --n-l 0:0 --k 2,2 | sed -n 5,7p

  Each command exits 0 and prints:

  parse_n_l('0:-1') -> {0: -1, 1: 0}
  parse_n_l('0-1:4,1:9') -> {0: 4, 1: 9}
  parse_n_l('spread:-5@0-1') -> {0: 0, 1: 0}
  ## set: 8 tokens, chunk -, 2 layers, card -1 experts (n_l -1..0), model m.gguf, build b0
  ## set: 8 tokens, chunk -, 2 layers, card -1 experts (n_l -1..0), model m.gguf, build b0
  # window-union — n_l 0:5, card list id prefix, k 2, policy static
  k	windows	predict	mean	p10	p90	pooled	union/window	slots/window
  2	7	0.500	0.500	0.500	0.500	0.500	8.0	16.0
  2	7	0.500	0.500	0.500	0.500	0.500	8.0	16.0
  

  Fix direction: give parse_n_l the expert count and make it the one place that checks a spec. It should refuse, by name, a count outside 0..n_expert, a layer named twice and a negative spread total. card_sets can then drop its own bound, since every card set comes from that reader. Refuse a repeated --n-l (and the other single-value options) in both command lines, and a repeated --k value. Add the refused specs to window-union.py --self-test beside the existing spread case.

• router-coverage.py bursts prints a Python traceback instead of its error line when --n-l is refused.

  bursts with a bad --n-l (a count above the expert total, or a layer the set did not trace) ends with a traceback and router_coverage.SetError: ... instead of the one-line router-coverage: ... message that every other refusal in that command prints. The exit code is 1 either way, so scripts are not affected, but the user sees a crash. A bad --layers in the same command is reported cleanly, which shows the intended behavior.

  What the code does. window_union() loads window-union.py by path (router-coverage.py L400-L406). That file loads router-coverage.py by path again, as a new module named router_coverage, and raises that module's rc.SetError (window-union.py L52-L54, raised at L91 and L102). bursts calls wu.parse_n_l and wu.card_sets (L464-L472), and main catches the SetError of the running __main__ module (L1057).

  Why it happens. When router-coverage.py runs as a script, its SetError is __main__.SetError. The copy that window-union.py loads defines a second, unrelated router_coverage.SetError. An error raised from the window-union side is an instance of the second class, so except SetError in main does not match it and it escapes as a traceback. The path load from bursts came with 68c60a6d6 (2026-09-28), which first called window-union's reader from bursts; window-union.py's own load of router-coverage.py dates from 051d51e1c (2026-09-24).

  Reproduction, with the same set as the item above:

  python3 tools/ref/router-coverage.py bursts "$d/set" --window 4 --n-l 0:17 2>&1 | sed -n '1p;$p'
  python3 tools/ref/router-coverage.py bursts "$d/set" --window 4 --n-l 5:1 2>&1 | sed -n '1p;$p'
  python3 tools/ref/router-coverage.py bursts "$d/set" --window 4 --layers 9 2>&1 | tail -1

  Traceback (most recent call last):
  router_coverage.SetError: layer 0: n_l 17 is above n_expert 16
  Traceback (most recent call last):
  router_coverage.SetError: --n-l names layer 5, which the set does not trace
  router-coverage: <tmp>/set: --layers [9] not in the set
  

  window-union.py itself reports the first case cleanly (window-union: layer 0: n_l 17 is above n_expert 16), because there both sides use the same loaded module.

  Fix direction: make the two files share one SetError. In window_union(), put the running module into sys.modules["router_coverage"] before loading window-union.py, and have window-union.py reuse an already loaded router_coverage. Alternatively catch the loaded module's wu.rc.SetError in bursts and re-raise it as the caller's SetError. A --self-test case that runs bursts with an out-of-range --n-l through main and checks for the one-line message would keep it fixed.

Hardware

• The build failure needs no particular hardware.
• The MTP draft store item: an NVIDIA GB10 (DGX Spark), driver 595.84, CUDA 13.3, with host-resident experts. The comparison matched on an RTX 5090 machine (driver 595.99.02, CUDA 13.2) with the rule on.
• The --n-l and traceback items need only Python 3 and a checkout.
• The token item: one RTX 5090 32 GB (driver 595.99.02), x86_64 Linux, the model on a local NVMe drive. A 28 GB host room makes the plan page routed experts through the NVMe tier.

AI help

AI 코딩 에이전트가 문제를 조사하고 재현해 이슈 초안을 썼습니다. 이슈 초안 리뷰는 제가 직접 했습니다.