feat(server): structured output with grammar-constrained decoding

[davide221]

response_format was ignored, so a request for JSON got the model's habits instead: JSON in a code fence, or JSON after a sentence. Apps that parse the reply broke. tool_choice: "required" was only a hint. With this PR the reply follows the schema exactly, on every model.

What changes

• common/gbnf: llama.cpp's GBNF parser and pushdown matcher (b6500, MIT), matching tokens by their decoded text.
• server/json_schema_grammar: llama.cpp's JSON Schema to GBNF converter. The schema is read as ordered JSON, so properties keep their order.
• common/output_constraint: one per request, shared through SamplerCfg::constraint.
  • The sampler draws as usual. When the grammar rejects the draw, it draws again from the tokens the grammar allows (llama.cpp's grammar rejection sampling). Otherwise it costs one token check per step. With temperature sampling this is exactly the masked distribution; with top_k/top_p, tokens the unmasked draw could reach weigh slightly more, which saves masking the whole vocabulary every step.
  • A redraw at the start of the reply skips blank tokens and tool-call openers: the rejected draw (a code fence, a sentence) meant to write the reply. Without this rule, models drifted into calling tools again and again.
  • It waits for </think> when the request starts in a thinking block.
  • It lets go once the reply opens a tool call. The openers come from the tool parser (tool_call_openers), so both always agree.
  • It never allows a token whose text holds a NUL byte. Seen live: "answer_cent\0s".
  • If a token is injected past the sampler, it stops constraining instead of leaving the request with no legal token.
• Commit points: the scheduler's advance_slot for paged engines, the token callback for the single-request path. A mask holds for one position only, so a constrained request never speculates (no DFlash, MTP or DSpark for it).
• Request fields: response_format (chat completions), text.format (responses) and output_format (messages), each with json_object or json_schema. An unsupported or non-string type is a 400. tool_choice required, any or a named function forces a tool call.
• server/docs/API.md marks these as supported.

Requests without a format or a forced tool call run the same code as before.

Measured

Measured with tool-eval-bench: 69 scenarios, seed 42, temperature 0, each model with its qualified launch flags. "Before" is engine 5740ef0a; "after" is this PR on 5f402c61. Qwen 3.8 27B Vision's "before" is its own pin, 3d8da899.

Model	Before	After	Structured Output
Qwen 3.8 27B (R9700)	89	93, 94 (two runs)	33% → 83%
Qwen 3.8 27B Vision (R9700)	90	91	50% → 83%
DeepSeek V4.1 Flash (192 GB layout)	90	92	50% → 83%
Qwen3.8-Flash-Next (R9700 + Strix Halo)	90	92	50% → 75%
Qwen3.8-Flash-Next (Strix Halo)	—	93	75%

• TC-45 (tool_choice: required) used to be excluded from the score, since the endpoint did not enforce it. It now passes on every model.
• What still fails is model behavior. TC-68 sends no response_format, so the model wraps its JSON in prose. Flash-Next calls extra tools before answering TC-67.
• DeepSeek V4.1's TC-03 goes from pass to fail, but a build of 5f402c61 without this PR fails it the same way: it comes from main since the pin, not from this change.
• Cost: a constrained request decodes without speculation. A Flash-Next JSON reply took 18.2 s instead of 14.6 s, and a thinking + schema request on Qwen 27B decodes at about 40 tok/s. Qwen 27B's median turn over the whole bench was unchanged at about 3.0 s.

Example, Qwen 3.8 27B, json_schema with an enum:

• before: ```json\n{"title": "2001: A Space Odyssey", "year": 1968, ...}\n```
• after: {"title": "2001: A Space Odyssey", "year": 1968, "genre": "sci-fi"}

Tests

• test_output_constraint.cpp: 18 cases. They cover the GBNF matcher, schema grammars, the sampler redraw (greedy and sampled), the start-of-reply redraw rule, end tokens, tool-call release, waiting for thinking, NUL bytes, the format parsing of all three APIs and forced tool calls.
• test_server_unit: 678 passed, 0 failed (HIP build). The change is host C++ only; the CUDA build is not checked locally.

🤖 Generated with Claude Code

[davide221]

Code review

Found 2 issues.

Copy to agent

Use the copy button in the top-right corner of the block, then paste into your agent.

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.

Repository: Luce-Org/lucebox
Reviewed commit: 8416c69e0e1374951b2fd52a84b5a9d59aac962e
Verify the findings against the current code, run relevant checks for any fixes, and explain any findings you reject.

<file name="server/src/server/http_server.cpp">

<violation number="1" line="2234" end_line="2234">
Non-string response_format `type` throws json::type_error instead of returning 400

`output_format_schema` reads the user-controlled format object with `format.value("type", std::string())`. nlohmann's `value()` calls `get&lt;std::string&gt;()` when the key is present, which throws `json::type_error.302` for any non-string value. A plain client mistake such as `{"response_format": {"type": null}}` or `{"type": 1}` therefore raises an exception. `attach_output_constraint` is called inside the `try` in `handle_model_request` whose only `catch` shown is `catch (const json::parse_error &amp;)` (line 2924), which does not catch `type_error`, so the request is not answered with the promised 400 — it either surfaces as a 500 from an outer handler or escapes entirely. The same pattern exists at line 2264 (`tool_choice.value("type", std::string())` in `tool_choice_requires_call`) and line 2330 (`req.tool_choice.value("type", "")`), reachable whenever `tool_choice` is an object with a non-string `type`. Fix: read the value with `format.contains("type") &amp;&amp; format["type"].is_string()` and treat anything else as the unsupported-type 400 path. The fact I could not settle: whether `parse_endpoint_request` pre-validates `response_format`/`tool_choice` types, and whether an outer `catch (const std::exception &amp;)` in the caller of `handle_model_request` converts this to a 500 rather than a crash.
</violation>

</file>

<file name="server/src/common/sampler.cpp">

<violation number="1" line="271" end_line="285">
Constraint redraw is not the masked distribution when top_k/top_p is active

`sample_logits` first draws from the full chain applied to the *unmasked* logits, and only on a grammar rejection redraws from the chain applied to the masked logits. The comment claims this "gives exactly the masked distribution", and the mixture arithmetic does cancel out when the chain is mask-invariant (temperature only). But `top_k`/`top_p` are not mask-invariant: an allowed token that falls outside the unmasked top-k gets probability 0 in the first draw, while it has positive probability under the masked chain, so the mixture `C(logits)(t) + (1−C(logits)(A))·C(masked)(t)` ≠ `C(masked)(t)`. Concretely, with `top_k=40`, temperature &gt; 0, and a grammar that rejects the argmax draw, allowed tokens that ranked below the unmasked top-40 are systematically underweighted and in-top-k allowed tokens overweighted relative to the distribution llama.cpp-style constrained decoding (mask, then run the chain) produces. Every sampled constrained request with `top_k`/`top_p` set (a normal chat request) gets a biased distribution whenever a redraw triggers. Fix: mask the logits *before* running the chain (as `sampler_distribution` already does), or rejection-sample by redrawing from the unmasked chain until the draw is allowed.
</violation>

</file>

Issue overview

1. Non-string response_format type throws json::type_error instead of returning 400 · confirmed (server/src/server/http_server.cpp:2234)

2. Constraint redraw is not the masked distribution when top_k/top_p is active · likely (server/src/common/sampler.cpp:271-285)

_{Reviewed 8416c69e · 3 candidate issues checked, kept at ≥70% confidence · add the code-review label to review again}

[davide221]

Non-string response_format type throws json::type_error instead of returning 400 · confirmed

output_format_schema reads the user-controlled format object with format.value("type", std::string()). nlohmann's value() calls get<std::string>() when the key is present, which throws json::type_error.302 for any non-string value. A plain client mistake such as {"response_format": {"type": null}} or {"type": 1} therefore raises an exception. attach_output_constraint is called inside the try in handle_model_request whose only catch shown is catch (const json::parse_error &) (line 2924), which does not catch type_error, so the request is not answered with the promised 400 — it either surfaces as a 500 from an outer handler or escapes entirely. The same pattern exists at line 2264 (tool_choice.value("type", std::string()) in tool_choice_requires_call) and line 2330 (req.tool_choice.value("type", "")), reachable whenever tool_choice is an object with a non-string type. Fix: read the value with format.contains("type") && format["type"].is_string() and treat anything else as the unsupported-type 400 path. The fact I could not settle: whether parse_endpoint_request pre-validates response_format/tool_choice types, and whether an outer catch (const std::exception &) in the caller of handle_model_request converts this to a 500 rather than a crash.

Checked (90%): At server/src/server/http_server.cpp:2234 format.value("type", std::string()) throws json::type_error when response_format.type is present but null/non-string, and attach_output_constraint only returns 400 on boolean false rather than catching that exception; the shown handle_model_request catch at server/src/server/http_server.cpp:2924 handles only json::parse_error, so this malformed but plausible client input bypasses the promised 400 path. The same unchecked string read pattern appears at server/src/server/http_server.cpp:2264 and :2330 for tool_choice.type.

[davide221]

Constraint redraw is not the masked distribution when top_k/top_p is active · likely

sample_logits first draws from the full chain applied to the unmasked logits, and only on a grammar rejection redraws from the chain applied to the masked logits. The comment claims this "gives exactly the masked distribution", and the mixture arithmetic does cancel out when the chain is mask-invariant (temperature only). But top_k/top_p are not mask-invariant: an allowed token that falls outside the unmasked top-k gets probability 0 in the first draw, while it has positive probability under the masked chain, so the mixture C(logits)(t) + (1−C(logits)(A))·C(masked)(t) ≠ C(masked)(t). Concretely, with top_k=40, temperature > 0, and a grammar that rejects the argmax draw, allowed tokens that ranked below the unmasked top-40 are systematically underweighted and in-top-k allowed tokens overweighted relative to the distribution llama.cpp-style constrained decoding (mask, then run the chain) produces. Every sampled constrained request with top_k/top_p set (a normal chat request) gets a biased distribution whenever a redraw triggers. Fix: mask the logits before running the chain (as sampler_distribution already does), or rejection-sample by redrawing from the unmasked chain until the draw is allowed.

Checked (85%): At sampler.cpp:271-285 the first draw runs the full chain (including top_k truncation at lines 210-214 and top_p cuts at 223-254) on unmasked logits, and only on grammar rejection redraws from the chain over masked logits. The draw-then-redraw mixture equals the masked distribution only if the chain is mask-invariant; top_k/top_p are not — an allowed token outside the unmasked top-k has probability 0 in the first draw but positive probability in the masked chain, so in-top-k allowed tokens are overweighted versus mask-first constrained decoding (which sampler_distribution at line 302 correctly does). This triggers on any normal sampled request with top_k/top_p and a response_format whenever a redraw occurs.