feat(SAMPLE-CORE): the checkpoint's sampling defaults, and the Gumbel draw that scanned 248,320 tokens on one thread (#1984, #1985)

.agents/issue-index.md

@@ -761,3 +761,6 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#2040](https://github.com/mudler/vllm.cpp/issues/2040) | `ENG-HYBRID-PLACEMENT` | W3c: replace W3b's hand-written per-model round trip with ONE shared seam every architecture routes through, exploiting that every MoE block already has the same `(Dev, weights, params, [T,H] dh, T)` shape — and correct `docs/ENVIRONMENT.md` and `docs/FEATURES.md`, which W3b made false by leaving them saying the knobs place nothing | bug |
 | [#2046](https://github.com/mudler/vllm.cpp/issues/2046) | `ENG-HYBRID-PLACEMENT` | `qwen3_5.cpp` kept private `Dev`/`DBuf`/`MakeTensor`/`Reshape` copies instead of the shared `dense_device_glue.h` set — the off-framework divergence its own `ResidentWeight` comment records, where a repair reached 25 model files and not this one. The private types also had INTERNAL LINKAGE, which is what forced the MoE placement seam to carry a glue-templated second spelling; migrating collapses it back to one | bug |
 | [#2050](https://github.com/mudler/vllm.cpp/issues/2050) | `ENG-HYBRID-PLACEMENT` | Laguna's FFN is host-orchestrated token-at-a-time — per-token host rows, the router on the host through `MatmulNK`, and a host scalar combine loop — so a device-shaped MoE entry wrapping those loops would put it in the placement seam's wired list while moving nothing and adding a round trip: supported to read, a regression to measure. The real repair is a device-resident batched FFN, which is a model rework with a performance gate | gap |
+| [#1984](https://github.com/mudler/vllm.cpp/issues/1984) | `SAMPLE-CORE` | `RandomSampleKernel` is launched `<<<n, 1>>>` and scans a 248,320-wide vocab on one thread per row, computing two `SplitMix64` rounds and an f64 `log` per element. Eleven lines above it the same file records that a single-block single-thread scan of a ~151k vocab cost ~7.5 ms/token, which is why greedy argmax was rewritten into `ArgmaxPartialKernel`/`ArgmaxFinalKernel`; the Gumbel draw never got that treatment. Upstream is whole-tensor (`vllm/v1/sample/ops/topk_topp_sampler.py::sample_with_exponential_noise`), so this is a mirror obligation. Reached by every non-greedy row through `ModelRunner::execute_model` -> `Sampler::forward` -> `vt::RandomSample`. Spec: [sample-gen-config-and-parallel-gumbel.md](specs/sample-gen-config-and-parallel-gumbel.md) | perf |
+| [#1985](https://github.com/mudler/vllm.cpp/issues/1985) | `SAMPLE-CORE` | `generation_config.json` is read for `eos_token_id` only (`hf_config.cpp::ReadGenerationConfigEosIds`), so `Qwen/Qwen3.8-27B`'s shipped `top_k: 20` / `top_p: 0.95` never reach `SamplingParams` and `to_sampling_params` resolves omitted knobs straight to the neutral OpenAI defaults, which disable both filters. vLLM applies them through `ModelConfig.get_diff_sampling_param` -> `OpenAIServing*.default_sampling_params` -> `to_sampling_params`. Since `vllm bench serve` stopped sending `--temperature`, both engines sample at temperature 1.0 and vLLM draws from 20 candidates while we draw from 248,320: different sampling on two sides of a parity benchmark. Spec: [sample-gen-config-and-parallel-gumbel.md](specs/sample-gen-config-and-parallel-gumbel.md) | bug |
+| [#2002](https://github.com/mudler/vllm.cpp/issues/2002) | `SAMPLE-CORE` | With `--speculative-config` set, `GPUModelRunner::sample_tokens` branches on `num_draft_tokens > 0` alone and returns the greedy-only `RejectionSampler`'s output, so `Sampler::forward` and `vt::RandomSample` are never called and a `temperature: 1.0` request decodes GREEDILY. `include/vllm/v1/spec_decode/rejection_sampler.h` states the contract it violates in its own deferral list ("a temperature > 0 request must NOT be routed here yet"); neither the runner nor `RejectionSampler::forward` enforces it. Found while writing #1984's acceptance measurement against a baseline recipe carrying `--speculative-config`, where the sampler under test would never have been launched and the null result would have read as "the change did nothing" | bug |

CMakeLists.txt

@@ -737,6 +737,7 @@ add_library(vllm STATIC
   src/vllm/config/cache.cpp
   src/vllm/config/scheduler.cpp
   src/vllm/config/device.cpp
+  src/vllm/config/generation.cpp
   src/vllm/config/kv_transfer.cpp
   src/vllm/config/offload.cpp
   src/vllm/config/weight_residency.cpp

docs/reference/server.md

@@ -200,6 +200,7 @@ a stop token early.
 | `--enable-radix-attention` / `--disable-radix-attention` | model default | SGLang-named alias for the prefix-cache toggle |
 | `--enable-jump-forward` | off | Jump-forward decoding for structured output (token-unique subset) |
 | `--enable-force-include-usage` | off | Force the usage block in responses |
+| `--generation-config auto\|vllm\|<dir>` | `auto` | Where the server's default sampling parameters come from. `auto` takes `temperature`, `top_k`, `top_p`, `min_p` and `repetition_penalty` from the checkpoint's own `generation_config.json`, and a request that OMITS one of those knobs then gets the checkpoint's value; a request that sends one still wins. `vllm` loads no file and keeps the neutral OpenAI defaults. A directory path reads a `generation_config.json` from there instead. `Qwen/Qwen3.8-27B` ships `top_k: 20` and `top_p: 0.95`, so under `auto` an unparameterised request samples from 20 candidates rather than all 248,320. The server prints what it resolved at startup. `--override-generation-config` is unavailable ([#1985](https://github.com/mudler/vllm.cpp/issues/1985)). |
 | `--tool-call-parser <name>` | `hermes` | Select one of 42 registered names across 38 dialect families. `auto` detects the dialect from the chat template, and `none` disables parsing. Gemma-4 accepts wrapped or bare text calls. Inkling requests require `"skip_special_tokens": false`. The `inkling` reasoning parser is unavailable. |
 | `--reasoning-parser <name>` | `none` | Select `think_auto`, `deepseek_r1`, `deepseek_v3`, `holo2`, `mistral`, `minimax_m2`, `minimax_m2_append_think`, `step3`, `olmo3`, `muse_glimmer`, `qwen3`, or `mimo`. `auto` detects from the template. The `qwen3` and `mimo` names share an adapter. |
 | `--kv-transfer-config '<json>'` | (unset) | External KV connector, same JSON as vLLM's flag. See [the KV offload guide](../KV-OFFLOAD.md) |

include/vllm/config/generation.h

@@ -0,0 +1,54 @@
+// Ported from: vllm/config/model.py @ 5559679229bc961848b121ccdeaa8fa5d79bec98
+// (ModelConfig.generation_config, ModelConfig.try_get_generation_config,
+// ModelConfig.get_diff_sampling_param).
+//
+// SCOPE: turning a checkpoint's generation_config.json into the server-wide
+// sampling defaults an OpenAI request resolves against. Upstream owns this on
+// ModelConfig; this tree has no ModelConfig, so the narrowing is a free
+// function over the already-parsed HfConfig, which is where the file's keys
+// land (transformers_utils/hf_config.h).
+//
+// DEFERRED, and tracked under `## Owed` in
+// .agents/specs/sample-gen-config-and-parallel-gumbel.md:
+//   - --override-generation-config (config/model.py:305), the dict that is
+//     merged over the file's values before the narrowing.
+//   - the server-wide output cap upstream derives from max_new_tokens
+//     (completion/serving.py:81-86). max_tokens is CARRIED here so a later row
+//     has it, and no caller reads it yet.
+#ifndef VLLM_CONFIG_GENERATION_H_
+#define VLLM_CONFIG_GENERATION_H_
+
+#include <string>
+
+#include "vllm/sampling_params.h"
+#include "vllm/transformers_utils/hf_config.h"
+
+namespace vllm {
+
+// The three forms vLLM's --generation-config takes (config/model.py:298-304).
+// "auto" is the DEFAULT and means the checkpoint's own file.
+inline constexpr const char* kGenerationConfigAuto = "auto";
+// "vllm" means load no file at all and use vLLM's neutral defaults.
+inline constexpr const char* kGenerationConfigNone = "vllm";
+
+// ModelConfig.get_diff_sampling_param (config/model.py). `generation_config` is
+// the --generation-config selector: "auto" (the checkpoint's own sibling file,
+// already parsed into `config`), "vllm" (no file), or a DIRECTORY path holding
+// a generation_config.json.
+//
+// A directory that does not exist, or holds no readable generation_config.json,
+// yields an empty result rather than throwing, because upstream's loader
+// returns {} on every failure path and never raises.
+DefaultSamplingParams GetDiffSamplingParam(
+    const HfConfig& config,
+    const std::string& generation_config = kGenerationConfigAuto);
+
+// The same narrowing over an already-read file, so a caller that has the six
+// keys in hand does not have to reconstruct an HfConfig around them. This is
+// where max_new_tokens becomes max_tokens.
+DefaultSamplingParams NarrowGenerationConfigSampling(
+    const GenerationConfigSampling& sampling);
+
+}  // namespace vllm
+
+#endif  // VLLM_CONFIG_GENERATION_H_

include/vllm/entrypoints/openai/protocol.h

@@ -266,12 +266,23 @@ struct CompletionRequest {
   // PARAMS), then runs PostInit(). `default_max_tokens` is the serving-resolved
   // fallback used only when the request omits max_tokens (Task 2 supplies the
   // model-derived value; unset => our SamplingParams default).
+  // `defaults` is the server-wide DefaultSamplingParams derived from the
+  // checkpoint's generation_config.json (config/generation.h). Upstream's
+  // resolution order, mirrored exactly: an explicitly sent request field wins;
+  // an omitted one takes the checkpoint value; only if the checkpoint declares
+  // nothing does the neutral OpenAI default apply
+  // (CompletionRequest._DEFAULT_SAMPLING_PARAMS). nullptr means no
+  // server-provided defaults, and reproduces the pre-#1985 behaviour byte for
+  // byte, which is what keeps every existing caller unchanged.
   SamplingParams to_sampling_params(
-      std::optional<int> default_max_tokens = std::nullopt) const;
+      std::optional<int> default_max_tokens = std::nullopt,
+      const DefaultSamplingParams* defaults = nullptr) const;
 
   // to_beam_search_params — completion/protocol.py:260. beam_width == n, the
-  // resolved max_tokens, ignore_eos, temperature (None => 1.0) and length_penalty.
-  vllm::BeamSearchParams to_beam_search_params(int max_tokens) const;
+  // resolved max_tokens, ignore_eos, temperature (None => the checkpoint's, then
+  // 1.0) and length_penalty.
+  vllm::BeamSearchParams to_beam_search_params(
+      int max_tokens, const DefaultSamplingParams* defaults = nullptr) const;
 };
 
 // Ported from: vllm/entrypoints/openai/completion/protocol.py:580-584
@@ -511,10 +522,12 @@ struct ChatCompletionRequest {
 
   // to_sampling_params — chat_completion/protocol.py:585. See CompletionRequest.
   SamplingParams to_sampling_params(
-      std::optional<int> default_max_tokens = std::nullopt) const;
+      std::optional<int> default_max_tokens = std::nullopt,
+      const DefaultSamplingParams* defaults = nullptr) const;
 
   // to_beam_search_params — chat_completion/protocol.py:589. See CompletionRequest.
-  vllm::BeamSearchParams to_beam_search_params(int max_tokens) const;
+  vllm::BeamSearchParams to_beam_search_params(
+      int max_tokens, const DefaultSamplingParams* defaults = nullptr) const;
 };
 
 // Ported from: vllm/entrypoints/openai/chat_completion/protocol.py:94

include/vllm/entrypoints/openai/serving_chat.h

@@ -245,6 +245,14 @@ class OpenAIServingChat {
   // the IDENTICAL chat template as chat-completions instead of reinventing it.
   const ChatPromptFn& prompt_fn() const { return prompt_fn_; }
 
+  // See OpenAIServingCompletion::set_default_sampling_params (#1985).
+  void set_default_sampling_params(vllm::DefaultSamplingParams defaults) {
+    default_sampling_params_ = std::move(defaults);
+  }
+  const vllm::DefaultSamplingParams& default_sampling_params() const {
+    return default_sampling_params_;
+  }
+
  private:
   // Build the per-request tool parser (get_tool_parser) when ToolsEnabled and a
   // parser name is configured; else nullptr. ONE instance per request (the
@@ -279,6 +287,9 @@ class OpenAIServingChat {
   // unavailable on this handler.
   const vllm::tok::Tokenizer* beam_tokenizer_ = nullptr;
   std::optional<int32_t> beam_eos_token_id_;
+  // See set_default_sampling_params. Empty => every knob falls to the neutral
+  // OpenAI default, byte-identical to the behaviour before #1985.
+  vllm::DefaultSamplingParams default_sampling_params_;
   // Multimodal chat seam (see set_multimodal_chat_fn). Null => the text-only
   // path runs unchanged (mm parts drop to the joined-text content).
   MultiModalChatFn mm_chat_fn_;

include/vllm/entrypoints/openai/serving_completion.h

@@ -96,6 +96,21 @@ class OpenAIServingCompletion {
     beam_eos_token_id_ = eos_token_id;
   }
 
+  // The server-wide sampling defaults from the checkpoint's
+  // generation_config.json (#1985), mirroring
+  // `self.default_sampling_params = self.model_config.get_diff_sampling_param()`
+  // in {completion,chat_completion}/serving.py. Unset (the default) means no
+  // server-provided defaults, which is the pre-#1985 resolution and is exactly
+  // what `--generation-config vllm` resolves to. Called from server_main once at
+  // startup; this handler is the ONLY place the value is applied, so deleting
+  // that call makes the whole feature unreachable and the reachability gate red.
+  void set_default_sampling_params(vllm::DefaultSamplingParams defaults) {
+    default_sampling_params_ = std::move(defaults);
+  }
+  const vllm::DefaultSamplingParams& default_sampling_params() const {
+    return default_sampling_params_;
+  }
+
  private:
   v1::LLMEngine* sync_engine_ = nullptr;
   v1::AsyncLLM* async_engine_ = nullptr;
@@ -105,6 +120,9 @@ class OpenAIServingCompletion {
   // unavailable on this handler.
   const vllm::tok::Tokenizer* beam_tokenizer_ = nullptr;
   std::optional<int32_t> beam_eos_token_id_;
+  // See set_default_sampling_params. Empty => every knob falls to the neutral
+  // OpenAI default, byte-identical to the behaviour before #1985.
+  vllm::DefaultSamplingParams default_sampling_params_;
   // Monotonic request counter — the request_id is "cmpl-<counter>". Upstream
   // uses random_uuid() (serving/engine/serving.py:_base_request_id); no
   // random/uuid is wired at T0, so a counter stands in (id uniqueness only).

include/vllm/sampling_params.h

@@ -132,6 +132,40 @@ struct StructuredOutputsParams {
 
 // Sampling parameters for text generation (T0 field subset). Defaults match
 // upstream SamplingParams exactly.
+// Ported from: the dict vllm/config/model.py::ModelConfig.get_diff_sampling_param
+// returns, and that vllm/entrypoints/openai/*/serving.py stores as
+// `self.default_sampling_params`.
+//
+// The SERVER-WIDE sampling defaults a checkpoint's own generation_config.json
+// asks for. Every field is optional because "the checkpoint declared nothing"
+// and "the checkpoint declared the neutral value" resolve differently: an unset
+// field falls through to the OpenAI neutral default, a set one does not. That
+// distinction is the whole rule, so it is carried in the type rather than in a
+// sentinel.
+//
+// `max_tokens` is already renamed from the file's `max_new_tokens`, exactly
+// where upstream renames it (get_diff_sampling_param, "Huggingface definition
+// of max_new_tokens is equivalent to vLLM's max_tokens").
+struct DefaultSamplingParams {
+  std::optional<double> repetition_penalty;
+  std::optional<double> temperature;
+  std::optional<int> top_k;
+  std::optional<double> top_p;
+  std::optional<double> min_p;
+  std::optional<int> max_tokens;
+
+  bool empty() const {
+    return !repetition_penalty.has_value() && !temperature.has_value() &&
+           !top_k.has_value() && !top_p.has_value() && !min_p.has_value() &&
+           !max_tokens.has_value();
+  }
+
+  // The set fields as `{key: value, ...}`, for the startup line upstream logs
+  // ("Default vLLM sampling parameters have been overridden by ..."). Empty
+  // string when nothing is set.
+  std::string ToString() const;
+};
+
 struct SamplingParams {
   // Number of outputs to return for the given prompt request.
   int n = 1;

include/vllm/transformers_utils/hf_config.h

@@ -66,6 +66,29 @@ struct RopeParameters {
 // `text_config` sub-dict; LoadHfConfig resolves that nested object as the source
 // of the text fields, mirroring upstream PretrainedConfig.get_text_config().
 // `model_type` and `architectures` are always read from the top-level wrapper.
+// The sampling keys of ONE generation_config.json, as
+// ModelConfig.try_get_generation_config (vllm/config/model.py) surfaces them.
+//
+// Upstream returns `GenerationConfig.to_diff_dict()`, i.e. only the keys that
+// differ from a bare `GenerationConfig()`. The pinned floor is
+// `transformers >= 5.5.3` (vLLM requirements/common.txt:10), and at 5.x every
+// sampling field of a bare GenerationConfig is None, so every key the JSON
+// declares survives the diff and reading the file literally is exact. Measured
+// against transformers 5.3.0 on a Qwen-shaped file: repetition_penalty 1.05,
+// temperature 1.0, top_k 20 and top_p 0.95 all survive. If the pin's floor ever
+// moves back to transformers 4.x -- whose defaults were temperature 1.0,
+// top_k 50, top_p 1.0 -- this parse becomes wrong and has to grow that table.
+struct GenerationConfigSampling {
+  std::optional<double> repetition_penalty;
+  std::optional<double> temperature;
+  std::optional<int> top_k;
+  std::optional<double> top_p;
+  std::optional<double> min_p;
+  // Upstream renames this to max_tokens when it narrows; kept under its HF name
+  // here because this struct is the FILE, not the narrowed result.
+  std::optional<int> max_new_tokens;
+};
+
 struct HfConfig {
   std::string model_type;
   std::vector<std::string> architectures;
@@ -126,12 +149,32 @@ struct HfConfig {
   // secondary stop ids gated on ignore_eos. Empty when the file is absent,
   // unparseable, or carries no eos_token_id.
   std::vector<int32_t> generation_config_eos_ids;
+  // The six SAMPLING keys of the same sibling generation_config.json, which
+  // upstream reads through the same one file read and then narrows in
+  // ModelConfig.get_diff_sampling_param (config/model.py). Every field is unset
+  // when the file is absent, unparseable, or does not declare that key, which
+  // is what makes "the checkpoint said nothing" distinguishable from "the
+  // checkpoint said the neutral value" -- the distinction the whole defaulting
+  // rule turns on. See include/vllm/config/generation.h for the narrowing and
+  // for the --generation-config selector that can point somewhere else.
+  GenerationConfigSampling generation_config_sampling;
 };
 
 // Loads and parses `path`. Throws std::runtime_error (message includes the
 // path) on missing file, malformed JSON, or missing required fields.
 HfConfig LoadHfConfig(const std::string& path);
 
+// Reads the sampling keys of the generation_config.json at `path`. Mirrors
+// ModelConfig.try_get_generation_config's failure polarity exactly: a missing
+// file, unparseable JSON, a non-object document, a null value and a
+// wrong-typed value each leave the corresponding field unset rather than
+// throwing, because upstream's loader returns {} and never raises.
+//
+// Exposed because --generation-config can name a DIRECTORY other than the
+// checkpoint's, which is the one case the sibling read on HfConfig cannot
+// serve.
+GenerationConfigSampling ReadGenerationConfigSamplingFile(const std::string& path);
+
 // The same parse, from a config object already in hand. `source` appears in
 // every error message exactly where the path would, so a refusal still names
 // where the config came from.