the Vulkan tier keeps deltanet state per session and drops a replaced model's device state; the scheduler's row pointers die with the step

[borisbat]

Behavior change: two streams decoding turn about on the Vulkan tier of a deltanet model (Qwen3.5) each read their own recurrent state; a model loaded after another was deleted in the same process drops the earlier model's device state instead of decoding its weights; the scheduler no longer leaves session pointers behind a step. Single-stream, single-model outputs are unchanged.

A user running dasllama-server on Qwen3.5-9B-MTP over Vulkan hit two things: a GC failed on dangling pointers exception after two streams finished together, and on the next run one stream's answer carrying the other stream's prompt. Both reproduce on a Q5_K sibling of that model here and both are fixed.

The crash: a decode step gathers pointers at the decoding streams' sessions into batch_rows and never cleared them; the same step reaps finished streams, deleting their sessions, and the server's validating heap collect between steps walked the stale pointers. The step now clears its row arrays after the batched sample.

The contamination: the Vulkan deltanet decode step kept one device copy of each recurrent layer's state, keyed by layer with a valid flag and no session identity. The second stream found the first stream's state resident and ran on it; a prefill's flush pulled a foreign session's state into the caller's buffers; every session's position-zero reset invalidated the other sessions' device progress. The step now records each resident copy's owner (the session state's host address), sends a foreign session's dirty copy home before uploading its own, answers a flush only for the owner, and lets a session's reset release only its own copies. Every Session with deltanet state carries a DnOwner token whose finalizer, run by the Session's finalizer at every delete, releases that session's device copies unwritten, so a switch never flushes into freed memory and a new session at the same addresses never inherits a dead one's copy.

A third defect surfaced while running the module's test files with the tier armed, and it is the one that turns a second model in one process into garbage: deleting a model never drops its device state, and the next load only reset the routing marks, so the second model's stacks uploaded beside the first's and the offset-keyed stack lookup served whichever model registered that offset first. In one process that gave a garbage second model (test_parity's pinned-greedy cells: the first model passes, every later one decodes noise), an assert on the decode attention block's geometry (test_parity_pregate), and the descriptor cap after eight loads. The upload rail now drops a still-installed model's device state before uploading its own; the whole-model resident driver and the per-op marks both count as installed.

The same seam reached one more site the codex round caught: a rejected speculative round's rollback (mtp_state_restore) invalidated every session's device copy; it now releases the rolling-back session's own.

The step and its flush are now [hot_path], so they read their layer's entry through the file's pointer idiom (?[] after the key is made) instead of copying the step struct out and back per token; the lazy builders behind them are [cold_path].

Where to look: vk_moe_dn_step, vk_dn_step_flush, vk_dn_step_release in dasllama_vulkan_decode.das; DnOwner beside Session and dn_reset / mtp_state_restore in dasllama_common.das; the release seat in dasllama_gpu_tier.das; the step's tail in dasllama_scheduler.das. ARCHITECTURE_GPU_VULKAN_DECODE.md gains sec.2.2u for the ownership, ARCHITECTURE_GPU.md sec.1.5 names the state seams as a Vulkan-only seat, ARCHITECTURE_ENGINE.md sec.1.9 carries the step's clear.

Validation, claims, ledger

Validation

• test_scheduler.das gains a two-stream deltanet cell on Qwen3.5-0.8B-Q8_0 under the GPU tier: a one-stream scheduler and a two-stream scheduler both against the solo generate references; and the batching cell asserts the row array is empty after a drain whose last step reaped batched streams (the unfixed code fails it).
• The new cell on the 5060 Ti: both arms green - the one-stream scheduler reproduces generate on the tier, and the two-stream scheduler's A and B each reproduce their solo references while the tier logs a state switch per recurrent layer per step. Before the ownership fix the two-stream arm mismatched; with the token in the tier module (finalizer unseen by the scheduler) the one-stream arm crashed on the reap-then-switch.
• The scheduler file under the tier: the MTP self-spec cell went green with the model-swap fix (it was the third model in its process). The SmolLM cells (chunk sizes, batching, evict, media splice, mrope) were red because their bit-exact generate() references and the scheduler's streams rode different roundings of the tier - a logits probe put the resident batch decode and the device prefill within about a logit of the CPU path with every argmax agreeing over a few steps, so those cells now pin the CPU lane (moe_gpu_drop_model after the load) instead of asserting one backend's exactness across two.
• test_vulkan_tier 36/36 (the synthetic deltanet step and flush-seam cells) on the change.
• New test_gpu_model_swap.das (Qwen3-0.6B, SmolLM2-135M, DASLLAMA_GPU=1): each model reloaded behind the other decodes its own greedy tokens, exact, with the arm asserted (not skipped) when DASLLAMA_GPU=1 is set and model A's two loads pinned to the same driver. Green with the fix (three drops logged); the control with the drop disabled goes red - model A behind B decoded a repeating token run instead of its reference.
• New model-free cell in test_deltanet.das: a counting release hook installed through set_moe_gpu_dn_state_hooks sees one release per position-zero reset, per speculative rollback and per delete, each carrying the session's own state range - the ownership token, the finalizer's module-scope resolution and the registry seam, all in the CI lane.
• Parity evidence for the changed path: the deltanet decode step serves Qwen3.5; test_parity's pinned-greedy cell on Qwen3.5-0.8B-Q8_0 is green under DASLLAMA_GPU=1 with the log showing GPU MoE tier: deltanet triples of 18 layers resident and deltanet decode step engaged, and the two-stream scheduler cell on the same model logs the tier's state switches. No stocked kq (K-quant) Qwen3.5 carries a frozen fixture (test_parity skips Qwen3.5-4B Q4_K_M/Q5_K_M/Q6_K as not present), so the kq half of the served-path parity duty is not met here; the lcpp_bench --parity runs (Qwen3-0.6B-Q8_0, Qwen3-4B Q4_K_M) ran with DASLLAMA_GPU=1 and their logs show resident driver armed, covering the whole-model driver the swap fix touches, not the deltanet step.
• The five functions already citing ARCHITECTURE_GPU_VULKAN.md sec.2.2o were re-read against the amended section: gpu_slot_rearm_ restores a slot's marks and then calls the upload rail, whose new drop guard is inert there because the server's slot switch drops the outgoing model (marks installed) before the incoming slot's marks are restored.
• test_vulkan_kernels.das (the fused deltanet step cell among them) 92/92 under the tier on the rebuilt binary; the four reached files were rerun after the rebase's compiler rebuild.
• The server repro on Qwen3.5-9B-MTP-Q5_K_M over Vulkan (a 4017-token and a 16-token greedy chat request in flight together): before, the short stream answered with the long stream's content and doubled tokens ("one to ten ten in in words words"); after, each stream answers its own prompt, and the validating collects that run after both streams were reaped pass where the user's crashed.
• test_deltanet's chunked-vs-recurrent cell went red under the tier: its chunked arm became the GPU chain and its continuation the GPU decode step. A three-arm probe on the same 150-token prompt puts the GPU chain within 0.19 logits (of 13.5) of the CPU recurrent arm, the CPU chunked arm within 0.22, the recurrent state within 0.04 of 11.7, and both arms' eight-step continuations through the decode step in lockstep; a verbatim standalone replica of the cell passes. Under dastest the recurrent-prefilled arm flips a near-tie at step 2 (the prompt is synthetic ids), with or without the tier's deltanet rails. The cell was a freeform token-parity cell, the shape tests/REVIEW.md rules out; it is now the forced-feed logits-tolerance form (one arm's greedy token fed to both, logits within 4% of the step's scale, measured 1.5% apart), pins the chain and the decode step off the way it already pins the vector activation forms, and logs both arms per step.
• Runs on this box (RTX 5060 Ti, driver 616.56), -jit, DASLLAMA_GPU=1 where the tier is named: run.das --suite model-free in its CI shape (no models dir, no GPU) 1244 passed / 0 failed / 151 skipped; test_scheduler 24/24 under the tier; test_deltanet 6/6 under the tier and 5/6 model-free (the model cell skips); test_gpu_model_swap 2/2, its swap cell red under the control; test_parity 19 passed / 18 skipped (the large tier) / 0 failed with sixteen models through one process; test_parity_pregate 8/8; test_vulkan_tier 36/36; test_vulkan_dec_tail 4/4; test_mtp_snapshot 2/2; lcpp_bench --parity on Qwen3-0.6B-Q8_0 and Qwen3-4B-Instruct-2507-Q4_K_M, 40/40 token-for-token each.
• The evidence for every deltanet and release branch is this box's runs (RTX 5060 Ti, driver 616.56, the commands above): CI has no GPU and no models, so its green covers compile, link and the model-free suite only.
• Dupe triage: the new cells' compare helpers (max abs diff, max abs, id-list equality) had exact twins as private copies in five test and harness files; the new ones now live once in _model_tier.das (logits_maxdiff, logits_maxabs, ids_equal) and the two new files use them. The pre-existing per-file copies are a folder sweep of their own. The tier's three-assignment hook setters (set_moe_gpu_dn_state_hooks and the async-FFN twin) are exact by shape and stay: one setter per seat trio is the tier's registration form.
• Review rounds: eight checklist and audit agents plus two codex rounds; the first round's P1 (the rollback's global invalidate) is fixed above; the second round's P1 was real too - the pre-load drop reset the MoE layer request the load had just armed, so a replacement model whose only requested rail is resident expert layers would have run without them; the drop now carries the request across, and a model-free cell in test_gpu_model_swap.das pins it; its P2 asks for a CPU oracle and forced-feed logits in the two-stream cell, declined - the cell's oracle is the same implementation run solo, which isolates the ownership seam from GPU-vs-CPU numerics (test_vulkan_tier owns those), and same-implementation token equality is exact by construction; the decoded text now logs on both sides of each compare.
• Board rows this speedup-free change touches: none. The ledger commit's mirror pass writes a speed-up down (Q3_K_L 1.08, IQ3_M 1.06 of the reference on the debug rig); gen_bench_records's hardware stamp refuses this box while the remote-access daemon runs, so the ten 1B vehicles' pp512 rows are entered as owed in PERF_LEDGER.md's OWED ROWS and the board's rows stand until re-minted.
• Rebased onto dasLLAMA: sample-and-match speculative decoding, the nine-row Metal verify fix, and the verify-cost dig #3931 (the sampled speculative round) and LAWS.md deleted #3934 (LAWS.md retired): seven rule documents conflicted and took master's wording; SamplingParams and DnOwner now sit together in dasllama_common.das. After the rebase, on this box: run.das --suite model-free in CI shape 89 files, 1404 tests, 1251 passed, 152 skipped, one red - test_facade_docs, red on master since dasLLAMA: sample-and-match speculative decoding, the nine-row Metal verify fix, and the verify-cost dig #3931 (an engine sample_row_ with no facade twin), fixed here by renaming the helper sample_logits_row (7/7 after; test_mtp_sampled_walk 5/5); test_scheduler on the CPU with models 23 passed / 1 skipped; test_deltanet 6/6 and test_gpu_model_swap 4/4 under the tier; the lint gate clean on the 15 changed files over all three rails. Two extra commits carry that rename and the three CMake references to the retired LAWS.md (two install excludes and the bundle gate's name list).

Claims - stated, not tested

• Two streams decoding turn about now pay a flush and a cold upload per recurrent layer per switch; the single-stream path is unchanged. The per-session device slot that removes that cost is ledgered.
• A value-field finalizer runs with its holder's delete, two levels deep and through an array of holders, interpreted and under JIT (probed this session); the Session's generated finalizer therefore runs the token's - provided the deleting module can see the finalizer, since a struct's finalizer resolves in the scope of the module that deletes its holder. The token therefore lives beside Session in the engine module; in the tier module it was silently skipped for the default when the scheduler reaped a stream, which cost two crash rounds here and is a lint candidate.

Not done

• The comment skill now says a //! that restates its own [arch] citation's section is a duplicate; 50 files carry a //! block directly above an [arch] line today, and each is judged at that file's next harvest (the on-first-touch sweep), not here.
• Per-session device slots for the deltanet step (followup_vulkan): two streams on the tier should not round-trip their state per layer per switch.
• The user reported about 2 tok/s decode on a 16 GB card for the 9B hybrid (a report, not a measurement here); a hybrid on the per-op tier pays that tier's per-layer cost, and a whole-model resident driver for the hybrids is followup_vulkan 2.
• Coverage the tier's ownership seam still lacks in a model-free form: a synthetic two-owner arm in test_vulkan_tier's deltanet step cell (today the switch, the owner-only flush and the release predicate reach a test only through the scheduler's two-stream cell on a GPU box with Qwen3.5-0.8B). No cell asserts scheduler == generate on the tier for a non-recurrent model; the SmolLM cells pin the CPU lane and the deltanet cell covers the recurrent one.
• The reporter's exact file (Qwen3.5-9B-MTP-UD-Q5_K_XL) was benched on this box after the PR opened but the server scenario was not rerun on it; the Q5_K_M sibling above is the server repro. On either file the whole-model resident driver declines silently on four gates (K-quant serving mode, the in-file MTP block, gated attention, the recurrent layers), so both run on the per-op rails - the reported 2 tok/s is that tier's cost, ledgered as its own arc.

🤖 Generated with Claude Code

[Copilot]

🔵 Needs a closer look

It changes Vulkan-tier state ownership/lifetime and scheduler pointer lifetimes in hot paths, so it warrants careful human validation beyond automated review.

Pull request overview

This PR fixes correctness and lifetime issues in dasLLAMA’s Vulkan GPU tier when running multiple concurrent deltanet sessions and when loading multiple models sequentially in one process, and it hardens the scheduler’s per-step pointer bookkeeping to avoid dangling references during validation GC.

Changes:

• Make Vulkan deltanet decode-step device-resident state session-owned (flush-on-owner-switch; owner-only flush; per-session release on reset/rollback/delete).
• Drop stale installed GPU model device state before uploading a new model (preserving the new load’s MoE layer request across the drop).
• Clear scheduler batch-row pointer arrays within the step to prevent stale session pointers surviving past stream reaping; add/adjust tests and architecture/review docs accordingly.

File summaries

File	Description
utils/watchdog/REVIEW.md	Adds a folder-local REVIEW checklist routing consent-wording changes to the dasLLAMA performance checklist.
utils/REVIEW.md	Refines utils-wide REVIEW rules (tool definition + CI-row obligations + dlim routing wording).
utils/LAWS.md	Records Boris rulings for the REVIEW checklist splits/moves under utils/.
utils/dasllama-server/tests/REVIEW.md	Adds tests-subfolder REVIEW rules for Playwright fixtures/spec authoring constraints.
utils/dasllama-server/REVIEW.md	Routes Playwright/fixtures changes to the new tests/REVIEW.md; clarifies request-field change duties.
utils/dasllama-server/main.das	Suppresses the Metal-specific CPU-prefill log message when running in Vulkan mode.
utils/dasllama-server/LAWS.md	Records Boris rulings for dasllama-server REVIEW rule moves/deletions.
utils/common/REVIEW.md	Adds a utils/common checklist stating cross-tool review applicability for shared utils code.
tutorials/integration/cpp/class_adapters_module.das.inc	Regenerates/rewrites the embedded tutorial module blob (byte array).
skills/LAWS.md	Records Boris ruling tightening //! doc retention when an [arch] citation already carries the contract.
skills/comment_style_hygiene.md	Updates comment hygiene rules to treat //! that restates its own [arch] section as duplicate.
plans/kernel_parity_pass.md	Adds methodology + measurements notes for parity pass rows (pp512 vehicles) and rep-count rule.
modules/dasUnitTest/unit_test.das.inc	Regenerates/rewrites the embedded unit_test module blob (byte array).
modules/dasLLAMA/tests/test_scheduler.das	Adds a 2-stream Vulkan deltanet isolation test; drops GPU model state in CPU-bitexact scheduler tests; adds pointer-lifetime assertion.
modules/dasLLAMA/tests/test_gpu_model_swap.das	New tests covering “drop installed model before load” behavior and multi-model swap correctness under GPU tier.
modules/dasLLAMA/tests/test_deltanet.das	Converts deltanet parity check to forced-feed logits tolerance; adds model-free owner-release seam test; pins GPU tier routes off for CPU-claim cell.
modules/dasLLAMA/tests/run.das	Registers the new test_gpu_model_swap.das in suite selection.
modules/dasLLAMA/tests/REVIEW.md	Refines/clarifies review rules around suite inclusion, logging requirements, and forced-feed vs token-exact compares.
modules/dasLLAMA/tests/CLAUDE.md	Documents new/changed tests and their gating/pinning rationale.
modules/dasLLAMA/tests/_model_tier.das	Adds shared helper functions for logits/id comparisons used across tests.
modules/dasLLAMA/REVIEW.md	Clarifies routing rules and some review obligations (GPU/Vulkan and measurement-related phrasing).
modules/dasLLAMA/REVIEW_MEASUREMENT.md	Tightens measurement provenance/board-row requirements and clarifies board-cell definition.
modules/dasLLAMA/REVIEW_GPU.md	Refines Vulkan routing glob and moves Vulkan-only obligations into REVIEW_GPU_VULKAN.md.
modules/dasLLAMA/REVIEW_GPU_VULKAN.md	Consolidates Vulkan-specific review rules and clarifies/update obligations (prefill_ids gate, cm2 probes, K/V codec coverage, etc.).
modules/dasLLAMA/PERF_LEDGER.md	Records owed board-row remints for pp512 vehicles based on new mirror-pass measurements.
modules/dasLLAMA/LAWS.md	Records Boris rulings around measurement/GPU review checklist edits and Vulkan-rule relocations.
modules/dasLLAMA/followup_vulkan.md	Updates followup ledger entries to reflect the landed fixes and remaining work.
modules/dasLLAMA/dasllama/dasllama_vulkan_decode.das	Implements per-session deltanet state ownership, owner-only flush, per-session release, and refactors flush/drop helpers.
modules/dasLLAMA/dasllama/dasllama_vulkan_common.das	Extends DnStep with recorded owner host addresses for state and history.
modules/dasLLAMA/dasllama/dasllama_scheduler.das	Clears batch pointer arrays before prefill/reap to prevent dangling session pointers across steps.
modules/dasLLAMA/dasllama/dasllama_math_vulkan.das	Registers the new deltanet release hook with the GPU tier.
modules/dasLLAMA/dasllama/dasllama_gpu_tier.das	Adds deltanet routing getters; adds release hook type + storage; exposes installed-marks predicate; extends hooks API.
modules/dasLLAMA/dasllama/dasllama_gpu_resident.das	Drops installed GPU state before upload when needed, preserving the load’s MoE layer request.
modules/dasLLAMA/dasllama/dasllama_common.das	Adds DnOwner token with finalizer; uses per-session release on reset/rollback; stamps token bounds at run-state creation.
modules/dasLLAMA/dasllama/dasllama_arch_qwen35.das	Routes deltanet decode GPU path documentation to the new ownership architecture section.
modules/dasLLAMA/ARCHITECTURE.md	Updates architecture index to include the new Vulkan decode doc section range.
modules/dasLLAMA/ARCHITECTURE_GPU.md	Documents the deltanet state seams seat and clarifies Vulkan-only SPI placement.
modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN.md	Documents the “drop installed model before upload” guard and its failure mode.
modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_DECODE.md	Adds section 2.2u describing deltanet decode-step per-session state ownership and lifetime rules.
modules/dasLLAMA/ARCHITECTURE_ENGINE.md	Adds anchor + describes scheduler step clearing its gather arrays to prevent dangling pointers.

Review details

• Files reviewed: 38/40 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

One new GPU-tier swap test can fail instead of skipping on boxes where DASLLAMA_GPU=1 is set but the tier cannot actually arm/serve the model, due to a skip condition that checks only gpu_want_auto() rather than the computed armed state.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 36/38 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The new test_deltanet_owner_release calls mtp_state_restore without taking a snapshot first, which can crash due to copy_floats_par indexing src[0]/dst[0] even when n==0.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 36/38 changed files
• Comments generated: 1
• Review effort level: Lite