feat(qwen3): multi-position hedged verification for the DSpark lane - #958
feat(qwen3): multi-position hedged verification for the DSpark lane#958FeathBow wants to merge 2 commits into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3a81ba130d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| match ladder_chains { | ||
| 0 => 0, | ||
| c => scratch.markov_max_batch() / c, | ||
| }, |
There was a problem hiding this comment.
When the active batch approaches the 256-span verify limit, this bound only accounts for Markov scratch capacity, so the draft lane still generates up to hedge_cap × ladder_chains expensive ladder rows even though try_execute_hedged_verify stops adding chains once expanded.len() == max_batch. At batch 256 every generated hedge is discarded and verification falls back to the plain pass; fixed hedge mode therefore pays the full sequential Markov ladder cost for zero benefit, while auto mode misattributes that cost to C=0 after clear_round_chains. Include (max_verify_batch - requests.len()) / ladder_chains in the request-slot bound and verify the near-capacity case with a same-context A/B.
AGENTS.md reference: AGENTS.md:L108-L110
Useful? React with 👍 / 👎.
3a81ba1 to
9dbffd6
Compare
|
@codex check again please |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9dbffd6484
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| /// `PEGAINFER_SPEC_HEDGE=<H>` enables the parallel hedge and caps the number | ||
| /// of hedged requests per verify round at `H` (each hedged request costs one | ||
| /// extra verify span and two lane scratch KV pages per hedge chain). `0`/ | ||
| /// unset/unparsable = off. Read once. | ||
| pub(super) fn spec_hedge_cap() -> usize { | ||
| static CAP: std::sync::OnceLock<usize> = std::sync::OnceLock::new(); | ||
| *CAP.get_or_init(|| { | ||
| std::env::var("PEGAINFER_SPEC_HEDGE") |
There was a problem hiding this comment.
The feature is exposed only through new environment variables, but neither the root key-env list nor docs/models/qwen3/dspark-integration.md is updated; the latter still says DSpark changes only proposal and introduces no new verify/KV path, which is now actively misleading for operators debugging scratch-page allocation, graph behavior, or output differences. Record these controls, defaults, and the revised production invariant in the Qwen3 documentation as required by the repository workflow.
AGENTS.md reference: AGENTS.md:L172-L176
Useful? React with 👍 / 👎.
| if elapsed > ROUND_GAP_CUTOFF || executed_c >= self.stats.len() { | ||
| return; |
There was a problem hiding this comment.
When a prefill or admission stall exceeds ROUND_GAP_CUTOFF, this return excludes the elapsed interval but leaves cur_tokens, cur_secs, and rounds_in_window intact. The next verify rounds therefore continue the pre-stall window instead of starting the documented boundary, mixing measurements from potentially different workloads and allowing the auto controller to commit to the wrong chain count after traffic changes; clear the in-progress window state on this path.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
the long gap is now fixed by cutting off the active window
9dbffd6 to
61806d6
Compare
ef0671c to
b1bb301
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b1bb301af5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| # DSpark Integration (Qwen3-4B) | ||
|
|
||
| **TL;DR:** DeepSeek's **DSpark** (paper + DeepSpec repo, Jun 2026) is **our DFlash drafter plus two bolt-ons**: (1) a *semi-autoregressive* **Markov head** — a rank-256 logit bias added in a short sequential loop over the block so each draft token conditions on the previously sampled one; this is the whole draft-quality win (+16–18% accepted length over DFlash in their offline table, paper Table 1), and (2) a **confidence head** (tiny linear → per-position survival probability) feeding a **hardware-aware prefix scheduler** that trims verify length under load — the production throughput/Pareto win. The released checkpoint `dspark_qwen3_4b_block7` has an **identical backbone to our `Qwen3-4B-DFlash-b16`** (same hidden 2560 / 5 layers / `target_layer_ids=[1,9,17,25,33]` / KV-injection) — it differs only by `block_size=7`, +4 tensors (`markov_w1 [V,256]`, `markov_w2 [V,256]`, `confidence_head.proj.weight [1,2816]`, `.bias [1]`), and an "anchor-is-the-first-prediction" block-layout tweak. **Our verify/accept/KV-transaction stack is reused unchanged** — DSpark only changes the *propose* step. On the 5090 greedy sweep, DSpark beats the matched block7 DFlash baseline by **+3.6% geomean output tok/s** overall, with the expected wins on text/code (+3.1% to +15.8%; random is the synthetic exception) and a better accepted-draft distribution (**2.52 vs 2.30** accepted draft tokens/round, full-7 accept **17.4% vs 13.9%**). Decisions settled: **block_size = 7**, **reuse the target head** (DSpark's `embed_tokens`/`lm_head` are byte-identical to target Qwen3-4B's — skip loading them). **Phase 1 delta over DFlash is small: config + 2 tensor loads + one sequential Markov loop in `execute_dflash_draft`; no new verify/KV/graph.** | ||
| **TL;DR:** DeepSeek's **DSpark** (paper + DeepSpec repo, Jun 2026) is **our DFlash drafter plus two bolt-ons**: (1) a *semi-autoregressive* **Markov head** — a rank-256 logit bias added in a short sequential loop over the block so each draft token conditions on the previously sampled one; this is the whole draft-quality win (+16–18% accepted length over DFlash in their offline table, paper Table 1), and (2) a **confidence head** (tiny linear → per-position survival probability) feeding a **hardware-aware prefix scheduler** that trims verify length under load — the production throughput/Pareto win. The released checkpoint `dspark_qwen3_4b_block7` has an **identical backbone to our `Qwen3-4B-DFlash-b16`** (same hidden 2560 / 5 layers / `target_layer_ids=[1,9,17,25,33]` / KV-injection) — it differs only by `block_size=7`, +4 tensors (`markov_w1 [V,256]`, `markov_w2 [V,256]`, `confidence_head.proj.weight [1,2816]`, `.bias [1]`), and an "anchor-is-the-first-prediction" block-layout tweak. **Our verify/accept/KV-transaction stack is reused unchanged** — DSpark only changes the *propose* step (the later env-gated hedge, `PEGAINFER_SPEC_HEDGE*`, additionally expands verify with extra chain spans over scratch KV pages; off by default). On the 5090 greedy sweep, DSpark beats the matched block7 DFlash baseline by **+3.6% geomean output tok/s** overall, with the expected wins on text/code (+3.1% to +15.8%; random is the synthetic exception) and a better accepted-draft distribution (**2.52 vs 2.30** accepted draft tokens/round, full-7 accept **17.4% vs 13.9%**). Decisions settled: **block_size = 7**, **reuse the target head** (DSpark's `embed_tokens`/`lm_head` are byte-identical to target Qwen3-4B's — skip loading them). **Phase 1 delta over DFlash is small: config + 2 tensor loads + one sequential Markov loop in `execute_dflash_draft`; no new verify/KV/graph** *(true of Phase 1 as landed — the later opt-in hedge adds expanded verify spans, scratch KV pages, and per-shape Markov graphs, all behind `PEGAINFER_SPEC_HEDGE*`)*. |
There was a problem hiding this comment.
This August commit meaningfully rewrites the TL;DR and adds hedge verification guidance, but the document still reports Last touched: 2026-06 on line 5. Bump it to 2026-08 so readers do not treat the newly revised production guidance as two months older than it is.
AGENTS.md reference: AGENTS.md:L150-L152
Useful? React with 👍 / 👎.
Signed-off-by: FeathBow <feathbow@gmail.com>
b1bb301 to
f670b1a
Compare
Signed-off-by: FeathBow <feathbow@gmail.com>
1 participant
Description
A third of DSpark's speculative rounds accept zero draft tokens on agentic traffic — a full target forward buys one token — yet at the exact position where the chain breaks, the Markov head's runner-up is the committed token 29.6% of the time. Single-chain decoding throws that capacity away. This change verifies it instead: each draft round also proposes one alternative chain per configured branch position (
PEGAINFER_SPEC_HEDGE=<H>,PEGAINFER_SPEC_HEDGE_POSITIONS=0,1,2) — the exact Markov top-2 at the branch, taken from a device-side snapshot, with greedy continuation over the same backbone logits — and all chains ride the existing verify pass as extra spans on the bucket-aligned CUDA graphs. The longest-matching chain wins and its scratch KV pages copy back; greedy output stays byte-identical, and the KV transaction interface and the draft/verify token-span seam are unchanged. Branch positions come from an offline counterfactual study over 10,801 logged break events: branching at the next position outprices deeper candidates at the same position.Hedge eligibility travels the draft seam as a remaining-output budget: only a greedy request whose earliest branch position survives the verify-side span clamp can consume a hedge slot, and draft and verify select from the same list, so batch order never changes whether hedging runs. An opt-in controller (
PEGAINFER_SPEC_HEDGE_AUTO=1) picks the chain count at runtime by measured tokens/sec — median window rate with an ascending margin, so flat traffic settles at zero chains. Kernel surface: a two-stage stridedmarkov_step_top2,hedge_ladder_force, and achainsdivisor plus an optional request map on the markov step kernels (the mapped variants areunsafewith documented device-index preconditions). KV surface: scratch pages outside the allocatable pool (new_with_scratch_pages) and a rawcopy_pageprimitive.Test Env
Verification
cargo check -p pegainfer-k3green on the shared-kernel signature change. glm52's two identical call-site updates do not compile on this host (its kernels require sm_100+) — please re-verify in a cu132 environment.dflash_speculative_gate5/5 on the exact basebdbce8f4, withPEGAINFER_REQUIRE_HEDGE_GATE=1: the hedged execution gate re-runs the sequential and concurrent-heterogeneous lossless suites in a child process with the hedge enabled, and its pass requires parsed per-round counters — hedged rounds > 0, at least one chain win (exercising winner copy-back and hidden-row compaction), and at least one loss round. The draft-headroom case is fixed in this PR (constants read from both checkpoint configs; unconstructable pairings skip explicitly).