Skip to content

speculative: keep DFlash draft positions contiguous across mtmd vision chunks - #5

Open
wangcc57 wants to merge 3 commits into
z-lab:dflash2from
wangcc57:codex/fix-dflash2-mtmd-vision-position-v2
Open

speculative: keep DFlash draft positions contiguous across mtmd vision chunks#5
wangcc57 wants to merge 3 commits into
z-lab:dflash2from
wangcc57:codex/fix-dflash2-mtmd-vision-position-v2

Conversation

@wangcc57

Copy link
Copy Markdown

Problem

With DFlash2 enabled, multimodal requests could fail with:

  • find_slot: non-consecutive token position
  • last position stored in KV cache; required Y = X + 1
  • failed to process mtmd chunk

Text-only requests worked; the failure was reproducible after an image entered the prompt.

Cause

The target context and the independent DFlash context do not share the same position space around mtmd image chunks. DFlash was copying target positions into its own one-axis RoPE KV cache, so the next draft batch could start at a non-consecutive position.

Fix

Keep DFlash draft positions in the draft-local consecutive position space:

  • mtmd feature injection uses the next position from the independent DFlash KV memory
  • draft noise blocks use the draft KV position instead of dp.n_past

This keeps speculative decoding valid across image chunks.

Validation

Validated with production Qwen3.8-27B Q4_K_M + DFlash2 on 3x Tesla V100:

  • text request: HTTP 200
  • single image request: HTTP 200
  • approximately 22K text plus image: HTTP 200; draft_n=63, draft_n_accepted=42
  • two images plus following text: HTTP 200, response OK
  • public nginx/Bearer image request: HTTP 200
  • no failed to process mtmd chunk, KV assertion, CUDA error, OOM, or process crash

git diff --check passes.

Related: ggml-org#25144

Keep DFlash draft positions in the independent draft KV context's consecutive position space across multimodal mtmd chunks.
Normalize indentation in the DFlash2 mtmd position fix.
Align the draft position declaration with the surrounding code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant