feat(serve): typed decisions through /v1/systemone and /v1/decisions

README.md

@@ -309,6 +309,11 @@ Prefill of a 4,463-token prompt, in tok/s:
   `/tokenize`, `/detokenize`, `/apply-template`, `/v1/rerank`, `/metrics`, `/slots` and `/props`,
   streaming, tool calls, structured output through xgrammar (`--structured-output`), token log
   probabilities and prompt grafts ([HTTP serving](docs/serving.md)).
+- **Typed decisions.** `/v1/systemone` (TypeSafe's System One API, as llama.cpp, vLLM and Ollama
+  serve it) and `/v1/decisions` (OpenAI's Decisions API) answer choice, yes/no and score questions
+  about a state with the served chat model: one prompt per question, the answer read from the
+  first token's exact label log probabilities, the state prefilled once
+  ([typed decisions](docs/serving.md#typed-decisions)).
 - **Model suspend.** `--model-suspend` lets an idle server give its device memory back without
   exiting and take it again on the next request, retained conversations included. Ternary Bonsai 2
   27B on an RTX 3090: 7.9 GiB down to 0.3 GiB in 0.33 s, back in 1.1 s

docs/README.md

@@ -11,7 +11,7 @@ spelling and defaults.
 | [Linux build](build-linux.md) | the container image, native Ubuntu builds for every target (`70`, `75`, `80`, `86`, `89`, `120a`), build options and the Bash launchers |
 | [Windows build](build-windows.md) | native Windows builds, their toolchain constraints and the launcher scripts |
 | [CLI](cli.md) | text, chat-history, image/video input, output streams, sampling, speculative decoding, and the runtime options |
-| [HTTP serving](serving.md) | OpenAI Responses/Chat Completions, Anthropic Messages, llama.cpp endpoints, the router, model suspend, state, streaming, token counting, authentication, tool calls, and the server options |
+| [HTTP serving](serving.md) | OpenAI Responses/Chat Completions, Anthropic Messages, llama.cpp endpoints, rerank and typed decisions, the router, model suspend, state, streaming, token counting, authentication, tool calls, and the server options |
 | [Ngram copy proposals](ngram.md) | copy acceleration alongside MTP, DFlash or DFlash2, on by default with a drafter |
 | [Launch recipes](recipes.md) | the `ninfer-serve` commands for every model and memory size, from an 8 GB card to the largest context, and the flags they use |
 | [Supported GPUs](gpus.md) | architectures and their builds, every card of 8 GB and more with what has run on it, and what fits in a given amount of GPU memory |

docs/serving.md

@@ -461,6 +461,8 @@ up. A router instead unloads the failed model once its requests are gone and rep
 | `POST /tokenize`, `POST /detokenize` | the model's tokenizer, in llama.cpp's shape |
 | `POST /apply-template` | the prompt a chat request renders to, as text, without generation |
 | `POST /v1/rerank`, `POST /rerank` (and `/reranking`) | documents ranked by relevance to a query, scored by the model (see [Rerank](#rerank)) |
+| `POST /v1/systemone` | typed questions about a state answered by the model, TypeSafe's System One API (see [Typed decisions](#typed-decisions)) |
+| `POST /v1/decisions` | the same reads in OpenAI's Decisions API shape |
 | `GET /models`, `POST /models/load\|unload\|sleep`, `GET /models/sse` | llama.cpp's router API: the catalog and its lifecycle (see [Several models](#several-models-router)) |
 | `POST /v1/responses` | OpenAI Responses Core generation, state, typed Items, and SSE |
 | `POST /v1/responses/input_tokens` | Responses prompt-token count without generation |
@@ -1379,6 +1381,72 @@ array of `{"index", "score"}`, with `text` when `return_text` is set. The judgem
 worth (`--max-concurrency`) at a time and share the cached judging prefix; on an RTX 3090 with a 27B
 model and two lanes, four short documents took 0.4 s.
 
+## Typed decisions
+
+`POST /v1/systemone` answers typed questions about a `state` in the shape of TypeSafe's System One
+API, as llama.cpp, vLLM and Ollama serve it, and `POST /v1/decisions` takes the same reads in the
+shape of OpenAI's Decisions API. The served chat model is the decision model: each question becomes
+one prompt -- a fixed system turn asking for one label, then the state, the question and its
+options under one-token labels (`A`.. for a choice, `yes`/`no` for a noul, `0`.. for a score),
+thinking off -- and its answer is read from the first reply token's exact log probabilities of
+those labels (a noul's `yes` and `Yes`, `no` and `No` added up), without sampling past it. No
+startup option is needed; a Qwen3.5-core model reads the labels exactly, Flash-Next and Gemma 4
+read them from the token's 20 most likely alternatives (`diagnostics.<id>.readout` says which).
+
+```bash
+curl http://127.0.0.1:8080/v1/systemone -d '{
+  "state": {"ticket": "My card was charged twice for one order."},
+  "questions": {
+    "team": {"type": "choice", "instructions": "Which team should handle this?",
+             "criteria": {"billing": "payments and refunds", "shipping": "deliveries"}},
+    "urgent": {"type": "noul", "instructions": "Does the customer need a reply within the hour?"},
+    "tone": {"type": "score", "instructions": "How upset is the customer?",
+             "criteria": ["calm", "annoyed", "furious"]}}}'
+```
+
+The request takes `state` (a string, or JSON that the prompt shows as its compact text),
+`questions` (an object of 1 to 64 question ids, each `{type, instructions, criteria}`, asked in
+request order) and optionally `instructions` (context placed ahead of the state) and
+`chat_template_kwargs` (accepted for compatibility; thinking stays off). A `choice` lists 2 to 26
+options as `{name: description or null}`, a `score` 2 to 10 level descriptions lowest first, and a
+`noul` takes no criteria or `{"true": ..., "false": ...}` descriptions. Unknown fields, `joint`
+reads and a thinking request are refused with 400; `images`, `files`, `audio` and `videos` with 501.
+
+The response holds `answers` by question id: a `choice` with the most likely option, its
+`probabilities` by option (summing to 1) and `confidence`; a `noul` with `noul` (the probability of
+yes) and the same `probabilities` and `confidence`; a `score` with `score` (the expected level index,
+weighted by probability), the `legend` and `probabilities` by level index, and `confidence`.
+Probabilities are the softmax over the labels; `confidence` is the chosen label's probability over
+the whole vocabulary, its share of the labels times `diagnostics.<id>.label_mass`, the probability
+the model put on the labels at all. `diagnostics.<id>.argmax_is_label` says whether the model's most
+likely token was one of the labels, and `readout` whether the labels were read exactly (`exact`) or
+from the top alternatives (`top_k`, where a label outside them counts as likely as the least likely
+alternative, its upper bound). `usage.input_tokens` sums the prompts of every read and
+`output_tokens` counts one token per question. The probabilities are those of a general chat model
+under this prompt, not calibrated to any data, and the head's logits are BF16, so two options the
+model scores within one BF16 step (about 0.1 nat at these magnitudes) share one probability; a label
+the tokenizer does not hold as one token is refused with 400.
+
+`POST /v1/decisions` takes OpenAI's request -- `input` as a string or user messages with string
+content or `input_text` parts, `questions` as an array of `{type, name, instructions}` with
+`choices: [{value, description}]` (string or boolean values, echoed with their type) for a `choice`
+and `levels: [{label, description}]` for a `score`; a `predicate` is a noul -- within the same
+limits, and answers with `answers` in question order: a `predicate` with its `probability`, a
+`choice` with `choice`, `probabilities: [{value, probability}]` and `confidence`, a `score` with
+`score`, `probabilities: [{value, label, probability}]` and `confidence`; `name` is echoed, null for
+an unnamed question. `safety_identifier` is accepted and ignored; streaming, tools, images and
+refusal scoring are not implemented.
+
+The reads of one request run a lane's worth (`--max-concurrency`) at a time, so a request leaves
+room for other clients' generation between its windows, and every read starts with the state, which
+the prefix cache prefills once: a question costs its own text and one token. On an RTX 3090 with
+Ternary Bonsai 2 27B and four lanes, a request about a 370-token state took 17 ms for one question
+and 235 ms for sixteen once the state was cached (150 ms and 0.8 s with the state's prefill); about
+a 13.5K-token state, 39 ms and 0.33 s (5.3 s and 6.1 s cold). Sixteen questions beside a streaming
+256-token chat completion took 0.42 s instead of 0.26 s and slowed the chat's decode from 90 to
+83 tokens/s. With the system turn the labels held 98-99% of the first token's probability on that
+model, where the user turn alone left 60-90%.
+
 ## OpenAI Responses Core
 
 NInfer implements the typed-Item and semantic-event core of the OpenAI

include/ninfer/ops/readout_logprobs.h

@@ -0,0 +1,55 @@
+#pragma once
+
+#include "core/tensor.h"
+
+#include <cstdint>
+
+#include <cuda_runtime.h> // cudaStream_t
+
+namespace ninfer::ops {
+
+// The most tokens one readout_logprobs() call reads.
+inline constexpr std::int32_t kReadoutLogprobsMaximum = 32;
+
+/**
+ * Op: Readout token log-probabilities
+ *
+ * The log probabilities of a caller-chosen set of tokens under one logits column, the typed
+ * decision readout: a request names the tokens its answer can be and reads their probabilities
+ * at the first generated position, without sampling them.
+ *
+ * Math / indexing:
+ *   Let l[r] be the exact real value represented by logits[r]. For every k in [0,count),
+ *
+ *     ideal[k] = l[token_ids[k]] - log(sum_{r=0..valid_rows-1} exp(l[r])).
+ *
+ * Logical shapes:
+ *   logits is [physical_rows] or [physical_rows,1], token_ids is [capacity] and output is
+ *   [capacity], with 1<=count<=capacity<=kReadoutLogprobsMaximum and
+ *   1<=valid_rows<=physical_rows. The first `count` values of token_ids are in [0,valid_rows);
+ *   the rest of token_ids and output are not read or written. Physical rows
+ *   [valid_rows,physical_rows) take no part in the denominator.
+ *
+ * Supported domain:
+ *   logits is contiguous finite BF16, token_ids is contiguous I32, and output is contiguous
+ *   FP32. Storage has its dtype's natural alignment.
+ *
+ * Numeric:
+ *   output is the FP32 numerical approximation of ideal. Reduction association and private
+ *   accumulator precision are implementation choices; the independent oracle evaluates the full
+ *   formula in FP64 from the represented BF16 inputs.
+ *
+ * Effects:
+ *   Writes output[0..count) and preserves both inputs. Output must not overlap logits or
+ *   token_ids.
+ *
+ * Workspace:
+ *   None.
+ *
+ * Execution:
+ *   Enqueues work on stream and owns no persistent state.
+ */
+void readout_logprobs(const Tensor& logits, const Tensor& token_ids, std::int32_t count,
+                      std::int32_t valid_rows, Tensor& output, cudaStream_t stream);
+
+} // namespace ninfer::ops

include/ninfer/types.h

@@ -861,6 +861,9 @@ struct StructuredOutputOptions {
     std::vector<std::string> choices;
 };
 
+// The most token ids one request may name for the typed readout (ExecutionOptions::readout_tokens).
+inline constexpr std::size_t kMaximumReadoutTokens = 32;
+
 struct ExecutionOptions {
     StructuredOutputOptions structured_output;
     SamplingOverrides sampling;
@@ -875,6 +878,12 @@ struct ExecutionOptions {
     // kMaximumTokenLogprobs most likely tokens under the distribution the sampler draws from
     // (see GenerationResult::content_logprobs). Sampling is unaffected.
     bool logprobs = false;
+    // Typed readout, with logprobs: the exact log probability of each of these token ids at the
+    // first generated token, under the softmax of the head's logits (the distribution of a greedy
+    // request without penalties), whether or not the id is among the top alternatives
+    // (TokenLogprob::readout). At most kMaximumReadoutTokens ids, each within the model's public
+    // token domain. A core without the readout leaves TokenLogprob::readout empty.
+    std::vector<TokenId> readout_tokens;
 };
 
 struct OutputOptions {
@@ -1275,6 +1284,10 @@ struct TokenLogprob {
     std::array<float, kMaximumTokenLogprobs> top_values{};
     // Decoded byte string for each top-k alternative, aligned with top_ids.
     std::array<std::string, kMaximumTokenLogprobs> top_bytes;
+    // The log probability of each ExecutionOptions::readout_tokens id, in the request's order,
+    // under the softmax of the head's logits at the first generated token; empty on every other
+    // token and when the request named no readout tokens or the core has no typed readout.
+    std::vector<float> readout;
 };
 
 struct OutputDelta {