Skip to content

Sync upstream b11887 - #327

Merged
danielhanchen merged 452 commits into
masterfrom
sync/upstream-b11887
Oct 25, 2026
Merged

danielhanchen merged 452 commits into
masterfrom
sync/upstream-b11887

Conversation

@danielhanchen

Copy link
Copy Markdown
Member

Syncs master with upstream ggml-org/llama.cpp at b11887 (6050b46), up from b11436 (b9a5a00, 2026-10-06). Must be merged with a merge commit, not squashed, so the sync point stays an ancestor of master (see scripts/unsloth/upstream-sync.json).

Why now: every pin has merged a newer upstream tag than master (#61, #264, #304 now carry b11879). With master this far behind, a pin PR targeting master would show its own change plus ~450 upstream commits, so the pins have been parked on per-pin base/upstream-<sha> branches. Once master contains b11887, each unslothai pin PR can target master directly and GitHub shows exactly its own change.

What the merge does:

  • git merge --no-ff b11887; the only conflicts were modify/delete.
  • sync_deletes.py kept the fork's deletion for all 19 upstream workflows (the fork owns no upstream CI). AGENTS.md and CODEOWNERS stay deleted (removed_ok).
  • Two workflows upstream added since the last sync, behavior-check.yml and build-wasi.yml, are dropped for the same reason (the 10-06 sync let 9 of these through and CI: drop stray upstream workflows and waive the deliberate metadata deletions in the sync guard #275 had to remove them).
  • upstream-sync.json: tag b11887, commit 6050b46, synced_at 2026-10-25. That is the only change to .github/ or scripts/unsloth/: every workflow and script is byte-identical to master.
  • Outside .github/ and scripts/unsloth/ the tree equals b11887 exactly, apart from the three removed_ok deletions.

Checks run locally:

  • verify_upstream_sync.py --merge HEAD --fork master --upstream b11887: history PASS (all 377 fork commits kept), tree PASS, deletions PASS (22, all upstream retiring its own files), ast PASS, c_enums PASS. content flags only upstream-sync.json, which the sync has to edit; the 10-06 sync shows the same line.
  • check_workflow_triggers.py: 14 workflows, no pull_request_target, no unjustified workflow_run.
  • test_sync_deletes.py, test_verify_upstream_sync.py (all negative controls caught), test_check_workflow_triggers.py: pass.

The nightly and the pin set are unaffected: the prebuilt builds an aged upstream tag plus pr-set.json and only reads scripts/unsloth/ from master, which is unchanged.

wanghqc and others added 30 commits October 14, 2026 13:02
The clamp kernel indexed source and destination as one flat contiguous array, so
a strided view as input (the new CLAMP v=1 case) read the wrong elements. Add a
strided kernel_clamp_nc, as the other unary ops have, and take it whenever the
source is not contiguous.
* spec : add lilicorr support

* convert : use convention for hparam assignment

* add todo for hparam use in llama-batch.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* typo fix

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…g#30498)

* server: save only restorable slot states to the prompt cache

The idle-slot save loop stores every idle slot whenever a task starts,
while only completion tasks restore from the prompt cache. Rerank,
embedding, decision and infill states fill the cache up to its limit
without ever being read back, and slots already cached are offered
again at every task start.

Each slot now tracks with cache_pending whether its state is restorable
and missing from the cache: a completion release or an explicit slot
restore sets it, prompt_save consumes it, prompt_clear resets it. Both
save paths share this rule, and the LRU path restores only for
completion tasks.

* server: test that decision slots skip the prompt cache

Decision tasks never mark their slot as cacheable, so starting new
tasks must not save their idle slots to the RAM prompt cache, with or
without a unified KV cache.

* server: restore any reusing task from the prompt cache

Embedding and rerank tasks never reuse their state, so their slots are
cleared on release like child slots and the cache only holds states
that a later task can restore. Infill and decision tasks now restore
from the cache too, which replaces the cache_pending flag.

Decision models disable the prompt cache unless --cache-ram is given,
since their states seldom repeat.
An embedding-only graph has no logits, and build_pooling() returns
early when embeddings are off, so since the models no longer expand
t_embd the graph ends up empty and its inputs are never allocated.
Expand t_embd in that case.
* llama: honor PARTIAL_ONLY in the plain KV cache

A non-SWA llama_kv_cache rolls back with seq_rm, so it holds no
partial state, yet state_write ignored the flag and serialized every
cell. A draft context made of a plain KV cache (MTP head, dense -md
draft) therefore copied its whole KV into each server checkpoint,
growing it by the draft KV size per token.

The cache now writes and reads nothing for PARTIAL_ONLY unless it is
SWA, and the iSWA wrapper no longer gates its base cache since it
filters itself.

* Update src/llama-kv-cache.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* llama: delegate PARTIAL_ONLY to the caches of every memory wrapper

The hybrid, hybrid-idx and dsa-iswa wrappers skipped their attention
caches on PARTIAL_ONLY, which also dropped an SWA window that is
partial state. Each cache now applies the flag itself, so the
wrappers only delegate. The indexer is marked stale only when a
restore rewrites cells.

test-llama-context gains a case checking that the partial state is
not larger than the full state and, without SWA, does not grow with
the tokens.

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…ml-org#30491)

* metal : add BF16 variant of the tensor API flash attention kernel

* cont : move tensor FA kernels to kernels/fa_tensor.metal
…-org#29200)

* cpu: add F16 SWIGLU_OAI reference implementation and F16 activation op test coverage

Adds ggml_compute_forward_swiglu_oai_f16() to the CPU backend, mirroring
the existing F32 path, and extends test-backend-ops with F16 test cases
for SWIGLU_OAI so downstream backends(e.g. Hexagon) can be CPU-diffed
for correctness.

* ggml-openvino: fix f16 SWIGLU_OAI type mismatch in alpha/one constants

translate_glu_swiglu_oai creates alpha_const and one as f32 constants
unconditionally, so f16 inputs hit a type mismatch in Swish/Add on GPU
("Swish inputs must have the same type but they are: f16 and f32").
translate_glu_swiglu_clamp already upcasts to f32 and converts back at
the end; apply the same pattern here.

---------

Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
…y channels (ggml-org#27703)

* vulkan : copy strided f16 KV once so reads spread across memory channels

Assisted-by: Claude (Opus 4.8 / Opus 5)

* tests : cover the f16 KV copy to scratch

Assisted-by: Claude (Opus 4.8 / Opus 5)

* vulkan : gate the f16 KV scratch copy to AMD

Assisted-by: Claude (Opus 4.8 / Opus 5)

* vulkan : reuse the generic copy shader for the f16 KV scratch fill

The f16 clause needs only a strided copy, not a dequant, so drop the
dedicated shader and dispatch the existing copy pipeline instead. The
copy is described in source memory order so the strided access lands on
the writes, which stream, instead of the reads, which would concentrate
on a few memory channels.

Assisted-by: Claude (Opus 4.8 / Opus 5 / Fable 5)

* vulkan : extend the strided KV scratch copy to bf16

Assisted-by: Claude Opus 5

* vulkan : clean up comments

Assisted-by: Claude Opus 5

* consolidate copy shaders

* disable on RDNA4

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
…g#30532)

* llama : split llama_prefetch() into merge and prefetch steps

Merge the ranges first, then call the system functions.

This fixes -Wunreachable-code-return on Windows.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Use const_cast

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
…#30485)

* unicode : reject overlong UTF-8 and out-of-range codepoints

* Update tests/test-unicode.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* tests: document byte-wise UTF-8 error recovery

* break out of first test on failure

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
…rg#30421)

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* replace oneAPI package by DLE+oneDNN package

* support to install oneDNN in windows, in release and docker

* for internal test

* remove test code

* Update intel.Dockerfile
* vulkan : fix concurrent first device init

* tests : load models in parallel in test-thread-safety

Loading them serially never exercised concurrent first backend init, so the crash reported in ggml-org#29764 was invisible to this test.

* vulkan : create fence before retaining device in buffer context
tojson wrote nan, inf and -inf for these floats, as values and as dict
keys; json.dumps writes NaN, Infinity and -Infinity. The string and safe
filters of a float keep nan and inf, as Python's str() does, so they no
longer use the tojson function. float_to_json is renamed float_repr, as
it now gives str(); float_json gives the json.dumps text.

Also say in a comment why a last digit 9 is not bumped at a power of 2.
* Add Support to maion-coder arch

* iAdd Support to support Draft models

* Address comments

* Remove maion context shift function

---------

Co-authored-by: Karthik Vijayan <kavijay@microsoft.com>
…gml-org#30582)

The registry hard-coded 10 for AGENTIC_MAX_TURNS while DEFAULT_AGENTIC_CONFIG
said 100. The registry seeds SETTING_CONFIG_DEFAULT, so the loop's
`|| DEFAULT_AGENTIC_CONFIG.maxTurns` fallback never fires and users got 10
turns. Reference the shared constant, as the adjacent MCP timeout does.

Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL)
* bump ty to 0.0.88

* arange -> linspace.tolist() to pass expected type
…ggml-org#30567)

Slot selection takes free slots by id or by last use only, so child
tasks often land on a slot holding a parent state while empty slots
are idle, and overwrite it without saving it to the prompt cache.

Free slots are now ordered empty first, then least recently used, for
both the LRU fallback and the child slots, and a child task saves the
state it overwrites. With 4 grouped decision requests in flight and
12 states in rotation, prompt cache reads go from 5 to 36.

[no release]
The slot file held the target context and the context checkpoints,
but never the draft context, so a restored slot drafted with an
empty or stale draft KV until the cache refilled.

The file now ends with the live draft state, the full draft context
and the speculative state, in the same appendix style as the
checkpoints. A draft state that does not fit the draft context is
dropped and the draft starts empty, while older files and older
servers keep working.

[no release]
…rg#30578)

* model : mellum - read MoE gating params from GGUF, unify RoPE

- read/write expert_weights_norm and expert_gating_func instead of hardcoded values (defaults unchanged)
- use per-layer rope params via get_rope_freq_base/scale
- check n_expert once, outside the layer loop

* Update conversion/mellum.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

[no release]
The first evaluated token attends to the positions above
pos - n_swa, so a SWA state starting at pos - n_swa + 1 holds the
whole window, but the reuse threshold asked for two more cells.
A saved SWA state keeps only the cells inside its window, so a
state read back from a slot file or from the RAM prompt cache was
always rejected and the slot rolled back to a checkpoint.

The SWA threshold now matches the window, while a state without SWA
keeps the exact bound. The test harness gains override_kv, used to
give the SWA test model a window smaller than the test context.

[no release]
Small helper around the GitHub Actions API: lists the workflow runs that are not
finished yet (in_progress, queued, pending, waiting, requested), grouped by status.

Supports selecting runs by status, by branch (server-side, to reduce API traffic)
or by a regex on the branch name, and can cancel the selected runs via --kill.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
…gml-org#30520)

* add support

* remove unneeded bos fix

* add video flag

* fix flake error

* remove video-specific budgeting

* convert : fix lilicorr-dflash bid clash

* spec : enforce lilicorr n_tokens condition

* spec : fix lilicorr walk

[no release]
ggerganov and others added 27 commits October 24, 2026 17:17
…gml-org#31100)

The two T4 runners in the ggml-ci group (ggml-6-x86-vulkan-t4,
ggml-8-x86-nvidia-a10) share the same labels and can both pick up the
same jobs, so extend the per-runner results dir from ggml-org#31046 to them.

Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
ggml-org#30978)

Use the dropdown menu surface (border, radius, background, shadow,
padding) for the hover card and the context gauge popup, and fix the
hover card open/close animation to use bits-ui data-state attributes.

Assisted-by: Claude (Opus 5.5 Medium)
* ui : move models manager to its own route, keep Discover as a dialog

Models now lives at #/models instead of a dialog. Discover opens as a
standalone dialog over it, and leaving the page returns to the chat it
was opened from.

Assisted by: Claude (Opus 5.5 Medium)

* ui : polish the models page layout and filters

Give the models page a page header with the Discover action, calmer
column labels, tinted section and family headers, and an empty state.
Load the model list on a direct visit, slide the detail pane over the
table below xl, and draw the filter toggles as one box matching the
context select, with room for their shadow.

Assisted by: Claude (Opus 5.5 Medium)

* ui : round the models table family headers

Assisted by: Claude (Opus 5.5 Medium)

* ui : fix models page navigation and Discover gating

Record the models page's way back after every navigation, so Back,
a typed URL or a reload no longer return to a stale chat. Escape
closes the detail pane, then leaves the page. Gate every Discover
entry point on one shared check, recheck the API key on /models,
skip the duplicate router fetch on a direct load, and share the
route load between pages.

Assisted by: Claude (Opus 5.5 Medium)

* ui : show model details in a drawer

Open the models manager's detail pane in a drawer, from the right on
desktop and from the bottom on a phone, instead of an inline side
panel and a full-screen cover. Pad the pane on both sides, drop its
close button on a phone, and assert that filtered rows are gone from
the page rather than merely hidden.

Assisted by: Claude (Opus 5.5 Medium)

* ui : make whole models table rows clickable

A click anywhere on a model or quant row opens it, unless it lands on
one of the row's own controls (load toggle, actions menu). The name
button stays the keyboard path to the same action.

Assisted by: Claude (Opus 5.5 Medium)
The src1 power-of-two rescale in kernel_mul_mm_id is superseded by the
per expert-token amax scaling done in the compute graph
(ref: ggml-org#30861), which keeps the
MoE down-projection activations within f16 range before the op runs.

Drop the two-stage reduction kernels, their scratch buffer and the
GGML_PREC_F32 dispatch added in ggml-org#26223,
as well as the test cases that drive MUL_MAT_ID src1 out of f16 range.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
…org#31055)

* hexagon: KV-head team split for HMX flash attention in multicore

* hexagon: merge FA_HEAD_SPLIT and FA_SPLIT into single GGML_HEXAGON_FA_SPLIT flag

0 = off (token-split only); 1 = head-split, plus team-split when
n_kv_heads < N_cores so stranded cores still participate (e.g. gemma-4
with 2 KV heads on 4 cores). Where heads divide evenly across cores,
plain head-split is used.
* jinja : fix select and selectattr argument handling

* jinja : address review feedback

* jinja : Simplified test attr
* model : add DeepSeek-V4.1-Flash (deepseek41)

* deepseek41 : remove MXFP8, convert the Engram table to BF16

* deepseek41 : compute the compressed-KV read layers once in the cache

* deepseek41 : use get_cells() for the shared-history check, drop llama_kv_cache::seq_pos_tok_le

* deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the host gather

* deepseek41 : keep the Engram hash constants in hparams.ple_*, one module per Engram layer

* deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle

* deepseek41 : keep il last in build_hc_mixes

* deepseek41 : sort the top-k rows before the sparse gather

* tests : add deepseek41 to the Metal fusion baseline

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
…l-org#30242)

* chat : reject JSON nested more than 128 levels before walking it

The schema builder, the jinja engine and the decision code walk request
JSON with recursion. A request nested deep enough overflows the stack of
the HTTP thread and llama-server dies (SIGBUS on macOS, SIGSEGV on Linux):
for a tool schema from a few hundred levels on macOS and about 10000 on
Linux.

Check the depth first and answer 400 past 128 levels: the chat inputs in
common_chat_templates_apply (tools, tool-call arguments,
chat_template_kwargs, json_schema), the json_schema of /completion, and the
whole /v1/systemone body. A gemma4 tool result that is too deep stays text.

* fix the root, not the call side

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* opencl: add `mul_mv_q2_k_f32`

* opencl: add q2_k SOA mv and tiled mm

* opencl: add q2_k noshuffle gemm/gemv for adreno

* opencl: add q2_k dp4a gemm for adreno

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>

* opencl: tune q2_k kernels for adreno

---------

Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add topk

* opencl: optimize topk

* opencl: add fp lightning indexer

* opencl: optimize lightning indexer

* opencl: refactor

* opencl: refactor tiled topk kernel

* opencl: adjust format
…ps (ggml-org#31071)

* hex-tools: support for --compare option in the profile.py

* hex-unary: improve collapse logic to also do splits if nrows is small

* hex-colapse: handle all colapsing at the host and make it generic

* hex-colapse: remove leftover unary helper
…rg#30106)

WM of the small tiles was 32, or 8 for 8-wide subgroups, whatever the
tile's BLOCK_SIZE and WARP. With 16-wide subgroups (Mali) the second
subgroup of a workgroup wrote into the next tile's columns. With 8-wide
subgroups some _id tiles left rows uncomputed. Derive WM so there is one
warp tile per subgroup.

Tested on my Pixel 9 Android phone. Before the change there were 20
MUL_MAT and 6 MUL_MAT_ID failures. After the change they all pass.

Fixes ggml-org#28637

Assisted-by: Claude Opus 5.5
…9920)

* vulkan: Add a larger tile size for conv3d (coopmat2 only)

This adds a larger tile size, in practice it is only for coopmat2 where
we don't need shared memory to store the output (would use more than 64KB).

Add some additional representative backend tests. At least, this is what codex
came up with from looking at sd.cpp Wan VAE.

* only apply large tile size for newer driver versions
…p rows (ggml-org#29948)

* cuda: fuse FFN gate/up and GLU in MMQ with interleaved gate/up rows

For prompt processing, run the gate and up matmuls of a dense FFN as one
mul_mat_q launch and apply the GLU in the write-back.

- The loaders take the gate weights as a second pointer and fill the weight
  tile with up and gate rows interleaved in chunks of 8, so each thread holds
  an up value and the matching gate value (rows r and r + 8 of an MMA tile).
- The MMA loop is unchanged: same tile, warp layout, vec_dot and one sum[]
  per thread. The fused write-back computes the GLU from registers and writes
  half the rows.
- The fused path uses the regular tiling (no stream-k) with the column tiles
  innermost so that neighbouring blocks share weight tiles.
- Enabled for Q4_K, Q5_K, Q6_K, Q8_0 and IQ4_XS on NVIDIA Turing and newer.

* cuda: assert the fused MMQ gate argument and trim redundant comments

* cuda: move the fused MMQ hint check to should_fuse and add TODOs

* cuda: pass prec_src1 to the fused MMQ write-back

* cuda: fuse only with efficient tiling and no fallback, merge the GLU write-back

* cuda: require has_fusion explicitly, drop the fusion-only src1 precision check

* cuda: launch the fused MMQ kernel through the stream-k kernel with one block per tile

* cuda: inline the MMQ tiling-efficiency check

* tests: one fused gate/up MMQ case per type and GLU op

* cuda: always launch the fused MMQ kernel with one block per tile

* tests: shorten the fused gate/up MMQ test comment

* cuda: select the fused MMQ loader and write-back in ggml_cuda_mmq_get_util_funcs

* cuda: run the fused MMQ kernel on the plain tiling path

* cuda: put the column tiles innermost in the MMQ plain tiling path
Co-authored-by: Shreya Agrawal <shreyaagrawal@Shreyas-MacBook-Air.local>
* llama : respect no_alloc in memory caches

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* llama : TODO: move no_alloc out of llama_hparams

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)

* llama : TODO: move no_alloc out of llama_hparams

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
…based expert detection (ggml-org#30820)

* llama-quant : fix quantizing 4D tensors

nrows_total was ne[1]*ne[2], so for a 4D tensor only the first ne[2]
slices were quantized and the rest of the rows were dropped. Use
ggml_nrows(tensor) to count all rows, including ne[3].

Fixes ggml-org#29916

* llama-quant : detect expert tensors by name and fix imatrix size for 4D tensors

The MXFP4_MOE path used ne[2] > 1 to detect MoE expert tensors. This is
wrong for non-expert tensors with 3 or more dims, e.g. MLA projections
(BailingMoE3) and 4D conv kernels. Use the tensor name instead: all
tensors used by GGML_OP_MUL_MAT_ID are named *_exps or *_chexps. This
also removes the BailingMoE3 special case.

The imatrix holds one vector of ne[0] values per 2D slice, so the
expected size is ne[0]*ne[2]*ne[3], not ne[0]*ne[2]. This matches how
the imatrix is indexed per row for 4D tensors in llama_tensor_quantize_impl.

Assisted-by: Claude Opus 5.5
* qwen: merge tensors on convert

* convert: use generator functions

* qwen4exp: remove conts and pre-scale
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-25T23:15:18.715811Z c3379b0 PR opened
🔒 Security Review ✅ Completed 2026-10-25T23:15:26.669721Z c3379b0 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c3379b02da

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/llama-batch.cpp
Comment on lines +136 to +138
if (has_state) {
state_vec = batch_inp.state;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor per-token offsets when flattening state embeddings

When a caller sets state embeddings in an order different from token-index order, this copies the states onto the wrong tokens. llama_batch_ext::set_token_state() appends each state in setter-call order and records its location in tokens[idx].state_off, but this assignment ignores those offsets, and ubatch_add() subsequently reads states by token index. For example, setting token 1's state before token 0's silently swaps their hidden states during MTP decoding. Build state_vec in token order using each token's state_off, as the mixed-embedding path does for embd_off.

Useful? React with 👍 / 👎.

Comment on lines +389 to +392
for t in flatten(suite):
tid = t.id()
try:
t.run(res)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Run the discovered test suite without bypassing fixtures

For Python agentic tasks whose hidden tests use setUpClass() or setUpModule(), flattening the discovered suite and calling each test's run() directly skips those fixtures, as well as their teardown and cleanup hooks. This incorrectly grades valid implementations as failures and can leak state between tests. I reproduced this with a test that reads an attribute initialized by setUpClass(): standard unittest discovery passes, while this runner reports error. Run the discovered suite through its normal TestSuite.run(res) lifecycle while retaining the custom result collector.

Useful? React with 👍 / 👎.

(int64_t) (beta->nb[2] / sizeof(float)),
(int64_t) (beta->nb[3] / sizeof(float)),
1.0f / sqrtf((float) S_v),
g->ne[0] == 1 && ggml_get_op_params_i32(dst, 0) < v->ne[2] && S_v == 128 && q->ne[0] == 128,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Exclude snapshot-producing batches from the MMA path

On an MMA-enabled GPU, this condition admits GDN batches with 1 < K < n_tokens when the other shape requirements are met, but gdn_single() only writes the final recurrent state and has no snapshot-count or snapshot-stride argument. For example, an eligible 128-token batch with K=17 leaves snapshot slots 1–16 unwritten. build_recurrent_attn() nevertheless copies all 17 slots into the recurrent cache, so a subsequent partial rollback can restore stale state and silently change inference results. Require K == 1 for this path until the MMA kernel implements the per-token snapshots.

Useful? React with 👍 / 👎.

@danielhanchen
danielhanchen merged commit 490b347 into master Oct 25, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.