Proposal: vLLM scoring fields on /v1/completions (allowed_token_ids, return_tokens_as_token_ids, token-id prompts) for classifier-style clients #768

Proposal

Some clients use a vLLM server as a classifier or scorer, for example AnyJev's VLLMBackend and Tacit's ServerEngine. They send /v1/completions with max_tokens: 1, temperature: 0, logprobs: K, allowed_token_ids: [...] (the label tokens), return_tokens_as_token_ids: true, and often a token-id prompt (list[int] or list[list[int]]). They read the label from the sampled token, or from the token_id:N keys of top_logprobs.

On FreeToken today, logprobs and token-id prompts get a 400, which is correct. allowed_token_ids and return_tokens_as_token_ids are accepted and then ignored (extra="allow", as in #693). Once #224 adds logprobs, these requests with a text prompt will get HTTP 200 with a wrong answer (token-id prompts still get the 400): the sampled token is not restricted to the labels, and top_logprobs cover the whole vocabulary under text keys. The client has no error to notice.

Minimal reproduction (CPU only, PYTHONPATH=python). It shows what main does with these fields today; the fake engine always answers " Paris":

import asyncio, json
from types import SimpleNamespace
from freetoken.message import UserReply
from freetoken.server.openai_api import CompletionRequest, handle_completion

class State:
    config = SimpleNamespace(served_model_name="m", max_output_tokens=None)
    def __init__(self): self.sent = []
    def new_user(self): return 1
    async def send_one(self, msg): self.sent.append(msg)
    async def wait_for_ack(self, uid):
        yield UserReply(uid=1, incremental_output=" Paris", finished=True, finish_reason="length")

async def show(**kw):
    s = State()
    r = await handle_completion(CompletionRequest(model="m", max_tokens=1, temperature=0, **kw), None, s, {})
    out = f"{r.status_code} {json.loads(r.body)['error']['message']}" if hasattr(r, "body") else f"200 text={r['choices'][0]['text']!r}"
    sent = [k for k in vars(s.sent[0].sampling_params) if "allow" in k or "token_id" in k] if s.sent else None
    print(kw, "->", out, "| allowed/token_id fields sent:", sent)

async def main():
    await show(prompt="Is the sky blue? Answer Yes or No:", allowed_token_ids=[9454, 2753])
    await show(prompt="Q:", allowed_token_ids=[9454, 2753], return_tokens_as_token_ids=True)
    await show(prompt="Q:", allowed_token_ids=[9454, 2753], logprobs=2)
    await show(prompt=[[1, 2, 3]])
asyncio.run(main())

{'prompt': 'Is the sky blue? Answer Yes or No:', 'allowed_token_ids': [9454, 2753]} -> 200 text=' Paris' | allowed/token_id fields sent: []
{'prompt': 'Q:', 'allowed_token_ids': [9454, 2753], 'return_tokens_as_token_ids': True} -> 200 text=' Paris' | allowed/token_id fields sent: []
{'prompt': 'Q:', 'allowed_token_ids': [9454, 2753], 'logprobs': 2} -> 400 logprobs is not supported | allowed/token_id fields sent: None
{'prompt': [[1, 2, 3]]} -> 400 OpenAI token-id prompt inputs are not supported; pass text prompt strings instead | allowed/token_id fields sent: None

Cause (current main, cc56b6b)

• python/freetoken/server/api_models.py:101-102: CompletionRequest has extra="allow" and does not declare allowed_token_ids or return_tokens_as_token_ids. _completion_unsupported_reason (openai_api.py:655-668) checks only the fields it declares.
• openai_api.py:656-657 rejects token-id prompts, and CompletionRequest.prompt already types them (list[int] | list[list[int]]).

Proposed design (on top of #224)

1. allowed_token_ids: a per-row vocabulary mask applied to the logits before sampling, validated as 0 <= id < vocab_size and non-empty (400 otherwise).
2. Logprobs on a masked row: if a row sends both allowed_token_ids and logprobs, compute its logprobs over the allowed set (renormalized), so the label scores sum to 1. vLLM gives this only with --logprobs-mode processed_logprobs, which applies to the whole server. Deciding per row keeps feat(server): OpenAI-compatible logprobs for chat and legacy completions #224's raw logprobs for every other request.
3. return_tokens_as_token_ids: return token and the top_logprobs keys as "token_id:N", as vLLM does. This avoids collisions when several ids decode to the same text.
4. Token-id prompts: accept list[int] / list[list[int]] and send the ids straight to the scheduler. Reject an empty list and ids outside [0, vocab_size) with a 400.
5. Logprobs cap: feat(server): OpenAI-compatible logprobs for chat and legacy completions #224 caps completions logprobs at 5 (the OpenAI legacy limit). Scoring clients ask for one entry per label. A server cap like vLLM's --max-logprobs (default 20; 32 would cover most label sets) would serve them.
6. Fields that are set but not implemented: return 400 for vLLM sampling or extension fields that change the result and are not implemented (guided_*, structured_outputs, prompt_logprobs, best_of, use_beam_search, min_tokens, ...), not accept them silently. This adds to the deny-list proposed in seed and chat logprobs are silently ignored (200 with no effect); proposal for per-request seeded sampling #693.

Items 1-4 are what the clients above need to get correct answers; 5 and 6 make the remaining gaps visible.

Environment

Serving nvidia/Qwen3.8-Flash-Next-NVFP4 on an RTX 5080 16 GB (sm_120, driver 615.78.08), torch 2.14.1+cu130; behaviour above checked CPU-only on main cc56b6b.