Speculative decoding routes every step to the greedy-only RejectionSampler with no temperature guard, so a temperature-1.0 request decodes greedily and vt::RandomSample is never called

[localai-bot]

With speculative decoding configured, the runner routes every step to the greedy-only RejectionSampler regardless of temperature, so a temperature: 1.0 request decodes GREEDILY and Sampler::forward — the whole sampling pipeline — is never called.

src/vllm/v1/worker/gpu/runner.cpp::GPUModelRunner::sample_tokens branches on drafts alone:

if (exec_state_.step.num_draft_tokens > 0) {
  ModelRunnerOutput out_rej = sample_tokens_with_rejection(logits);
  ...
  return out_rej;
}
const SamplingMetadata sm = input_batch_.make_sampling_metadata();
const SamplerOutput sampler_output = sampler_.forward(queue_, logits, sm);

There is no temperature predicate on that branch, and RejectionSampler::forward (src/vllm/v1/spec_decode/rejection_sampler.cpp) has no refusal either — its argument checks cover rank, dtype and the cu_num_logits shape and nothing else. include/vllm/v1/spec_decode/rejection_sampler.h states the intended contract in its own deferral list: "The STOCHASTIC path ... and the residual-distribution _resample_kernel Gumbel draw ... Everything below keys off temperature == 0; a temperature > 0 request must NOT be routed here yet." Nothing enforces that sentence.

Two consequences, and the second is what makes it urgent.

1. A correctness divergence. Upstream's rejection_sample (vllm/v1/worker/gpu/spec_decode/rejection_sampler_utils.py) carries both branches and picks between them per request; ours implements only is_greedy. So with --speculative-config set and a request at the default temperature 1.0, we emit the target model's argmax while vLLM emits a draw from the residual distribution. Same silent shape as #1985: the request asks to sample, and we do not.

2. It silently voids any sampler measurement taken on a speculative recipe. #1984 is about RandomSampleKernel costing milliseconds per decode step. On a run configured with --speculative-config '{"method":"dflash",...}' that kernel is never launched at all, so a before/after on such a recipe returns two identical numbers — and two identical numbers read exactly like "the change did nothing" rather than like "the code under test never ran". This was caught while writing the acceptance measurement for RandomSampleKernel is a single-thread serial scan of a 248,320-wide vocab, the exact shape this file already measured at ~7.5 ms/token and rewrote for greedy argmax #1984 against a baseline recipe that carries --speculative-config; it would otherwise have produced a confident null result.

A cheap check that separates the two states before any fix, on one build: run the same prompt at --temperature 0 and at the server default, both WITHOUT --speculative-config. The default-temperature arm takes Sampler::forward -> vt::RandomSample; the temperature-0 arm short-circuits to GreedyArgmax at src/vllm/v1/sample/sampler.cpp and never calls it. The per-token difference is the serial Gumbel scan's cost, measured directly and without a branch to attribute it to.

Not designed here, and the two halves are separable: refusing a temperature > 0 request under spec decode by name is a small change that makes the deferral honest, while porting the stochastic accept and the residual-distribution resample is the real work and needs its own row and its own gate. apply_sampling_params over the expanded batch (rejection_sampler.py:113-120) is named in the same deferral list and is a prerequisite for either.