Repository navigation
Conversation
_run_scheduler wraps startup and pushes an ("error", reason) ack before dying,
so the supervisor can report the real cause. _run_tokenize_worker did not, so
every tokenizer-side startup failure reached the parent only as a dead process
and surfaced as the generic "backend worker freetoken-detokenizer-0 exited
during load" — the actionable reason (unreadable tokenizer.json, missing
checkpoint file, ZMQ bind clash) was lost.
Wrap tokenize_worker in the same try/report/re-raise the scheduler already uses.
The traceback still prints and the process still exits non-zero; only the ack is
added. No ack_queue (direct call) stays a plain propagate.
Adds tests/server/test_launch_worker_errors.py, which fails before this change.
_schedule_next_batch gives prefill unconditional priority (or-short-circuit), so a long prompt prefilled in chunks occupies that many consecutive scheduler steps during which in-flight decodes are never scheduled. Measured on the live service: four cold prefills of 680/797/506/250 chunks over 267/317/190/93 s, a prefill-burst share of 70% in a 15-minute window, at 0.388 s per 384-token chunk. An already-decoding request therefore waited up to ~5 minutes for its next token. Reversing the order outright would starve prefill, and a request that never prefills never starts. The scheduler already carried the marker for this: a TODO reading "support other policies: e.g. DECODE first". Add --decode-interleave-every N: prefill keeps priority, but after N consecutive prefill steps one decode step is taken if a decode is runnable. At N=8 that bounds the stall to ~3 s instead of ~300 s, for about 1% of prefill wall time. Unset keeps the historical order bit-for-bit, and a Scheduler constructed without the policy (as the accounting tests build it) still works. Assisted-by: Claude Code
…of it Review of the previous commit returned "merge with 2 test nits" and no correctness defect, and named the gap precisely: several tests drove _decide, a pure re-implementation of the decision written inside the test file. That is a model of the code, not the code, so it can drift and the tests keep passing while the real _schedule_next_batch diverges. Adds coverage against the real method via Scheduler.__new__ with stub managers: a truth table over policy=None/1/3/8 asserting the exact slot sequence, strict alternation at every=1, the latch case (decode due but unavailable, prefills continue, then decode fires exactly once), the pure-decode and no-work paths, a policy that exists but is disabled, decode-manager call counts, and the config -> Scheduler.__init__ -> CLI wiring. Two expectations written from reasoning were wrong and the real scheduler corrected them (every=1 starts with prefill, because the streak begins at 0), which is the value this coverage adds. Assisted-by: Claude Code
/v1/models published `config.max_seq_len` -- the model's positional ceiling -- while the engine clamps its own `max_seq_len` to `min(config.max_seq_len, num_pages * page_size)` at startup and again in `_refresh_seq_state` after a rebuild, and the scheduler admits against that clamped value. The frontend process holds an unclamped ServerArgs copy, so the two disagreed about one quantity: on a 12 GiB card, 262144 advertised against a 178176-token pool. `ft launch` reads this number to size each agent's context window (opencode `limit.context`, codex `context_window`/`max_context_window`, CLAUDE_CODE_MAX_CONTEXT_TOKENS), so agents set their compaction threshold past what the server can hold and hit `context_length_exceeded` instead of compacting in time. Split the pool resolution out of `cache_geometry` into `kv_pool_geometry` so /v1/models and /v1/cache/status cannot drift apart, and clamp the advertised value with it. Falls back to the model ceiling while `num_pages` is still 0 (loading, or an engine build that sends no meta ack). The existing route test passed either way -- FakeState carries no pool geometry, so it fell through to the ceiling. It now asserts that fallback explicitly, alongside new cases for the clamp, a pool larger than the ceiling, and a rebuild superseding the load-time allocation. Fixes FlashML-org#448 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Declare requires_sm90 on the fa BackendInfo entry so an explicit --attention-backend fa fails during config validation on Ada and consumer Blackwell instead of after model load, during CUDA-graph capture. Fixes FlashML-org#442
STREAM_KEEPALIVE_SECONDS exists, so the tests set it without raising=False: a renamed constant now fails them instead of leaving them on the 15 s default. The manager helper drops the abort list no test read.
…ease request buffers
…o the allocated KV pool
…r worker died during load
… sm_90 at config time
…tore, add --dist-port # Conflicts: # python/freetoken/engine/config.py
…ping its socket TCPStore owns master_listen_fd and closes it with the store; FlashML-org#461 also kept the Python socket on the engine, whose finalizer then closed whatever file reused that fd number after destroy_process_group at shutdown. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit b54cced)
…long prefill runs
…ental decoding # Conflicts: # python/freetoken/tokenizer/detokenize.py
…limit FlashML-org#464 used `not req.can_decode`, but since FlashML-org#546 overlap runs device_len a step ahead of the host, so a non-terminal EOS was dropped from the matched text one step early. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 1814a3a)
…han once Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 577427a)
…cial_tokens decodes FlashML-org#464's scheduler-side DecodeStatus decoded with special tokens while FlashML-org#393 lets a request skip them on the frontend, so a stop spanning a special token matched different text. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit fc2392b)
…semantic responses # Conflicts: # python/freetoken/server/generation.py
…tice a client that left # Conflicts: # python/freetoken/server/api_server.py
This was referenced Oct 2, 2026
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
TCPStore compatibility, port validation, and streaming event-order defects remain unresolved.
Review effort: Balanced
Findings: 1
Open (6)
What changed in this PR
This deployment merge combines upstream reliability, scheduling, cache, streaming, and API metadata improvements.
Changes:
- Adds stream keep-alives, Qwen marker filtering, incremental stop matching, and startup diagnostics.
- Adds decode interleaving, KV-aware context reporting, and compact radix-key storage.
- Secures/configures TP rendezvous and validates attention backend architecture support.
| File | Description |
|---|---|
tests/tokenizer/test_detokenize_extras.py |
Tests ordered same-UID decoding. |
tests/server/test_stream_keepalive.py |
Tests SSE keep-alives and disconnects. |
tests/server/test_qwen_semantic_special_tokens.py |
Tests Qwen marker filtering. |
tests/server/test_openai_api.py |
Tests KV-clamped model metadata. |
tests/server/test_launch_worker_errors.py |
Tests tokenizer startup diagnostics. |
tests/server/test_dist_port_arg.py |
Tests distributed-port parsing. |
tests/scheduler/test_match_stop_str.py |
Tests incremental stop matching. |
tests/scheduler/test_interleave.py |
Tests decode interleaving policy. |
tests/engine/test_distributed_store.py |
Tests loopback rendezvous storage. |
tests/engine/test_attention_backend_arch.py |
Tests FA architecture guards. |
python/freetoken/tokenizer/detokenize.py |
Refactors incremental decoding state. |
python/freetoken/server/openai_api.py |
Clamps advertised context capacity. |
python/freetoken/server/launch.py |
Reports tokenizer worker failures. |
python/freetoken/server/generation.py |
Filters Qwen transport markers. |
python/freetoken/server/args.py |
Adds distributed-port and interleave options. |
python/freetoken/server/api_server.py |
Adds keep-alives and KV geometry. |
python/freetoken/scheduler/scheduler.py |
Integrates stop decoding and interleaving. |
python/freetoken/scheduler/interleave.py |
Defines interleaving policy. |
python/freetoken/kvcache/swa_radix_cache.py |
Clones stored SWA radix keys. |
python/freetoken/kvcache/radix_cache.py |
Clones stored radix keys. |
python/freetoken/kvcache/hybrid_radix_cache.py |
Clones stored hybrid radix keys. |
python/freetoken/engine/engine.py |
Adds architecture checks and loopback TCPStore. |
python/freetoken/engine/config.py |
Adds interleave and rendezvous configuration. |
python/freetoken/core.py |
Stores per-request stop decode state. |
python/freetoken/attention/__init__.py |
Declares FA architecture requirements. |
docs/cli.md |
Documents --dist-port. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+643
to
+646
| return torch.distributed.TCPStore( | ||
| "127.0.0.1", config.distributed_port, config.tp_info.size, | ||
| is_master=True, timeout=timeout, multi_tenant=True, master_listen_fd=sock.detach(), | ||
| ) |
Comment on lines
+170
to
+173
| # A view would keep the request's full output-budget token buffer alive. | ||
| new_node.set_key_value( | ||
| input_ids[prefix_len:].clone(), indices[prefix_len:].clone() | ||
| ) |
Comment on lines
+173
to
+174
| if not 0 <= n <= 65535: | ||
| raise argparse.ArgumentTypeError("must be between 0 and 65535") |
| reasoning_delta, content_delta = reasoning_parser.parse_stream_chunk(content_delta) | ||
| if reasoning_delta: | ||
| stripped_reasoning = strip_special_tokens(reasoning_delta, specials) | ||
| stripped_reasoning = _filter_reasoning(reasoning_delta) |
Comment on lines
+15
to
+16
| four long cold prefills: 680/797/506/250 chunks, 267/317/190/93 s | ||
| prefill-burst share of a 15-minute window: 70% |
Comment on lines
+448
to
+451
| parser.add_argument( | ||
| "--decode-interleave-every", | ||
| type=int, | ||
| default=ServerArgs.decode_interleave_every, |
Owner
Author
|
Superseded in part: upstream merged FlashML-org#559 as 🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #75 (the rebuild). This merges 9 open upstream PRs that are generic or matter for this deployment, one merge each, at their current heads:
/v1/modelscontext_length to the KV poolfabelow sm_90 at config timeb54cced--dist-port*:1920(all interfaces), now on127.0.0.1:1920; b54cced hands the fd to TCPStore--decode-interleave-every1814a3a,577427a,fc2392bskip_special_tokens<|im_start|>is fed back to the agent as a control tokenConflict resolutions:
engine/config.py(fix(server): loopback-bind the TP rendezvous store, add --dist-port FlashML-org/FreeToken#461): keep ourdistributed_timeout = 1800.0and adddistributed_port.tokenizer/detokenize.py(fix(scheduler): match stop strings with incremental decoding FlashML-org/FreeToken#464): keepdetokenize()→detokenize_with_meta(). The same-uid guard goes at the top ofdetokenize_with_meta(1080deploy's form).server/generation.py(fix(server): hide qwen transport markers from semantic responses FlashML-org/FreeToken#266 over feat(server): OpenAI-compatible logprobs for chat and legacy completions FlashML-org/FreeToken#224's logprobs):GenerationErrorstays._content_delta. That includes fix(server): hide qwen transport markers from semantic responses FlashML-org/FreeToken#266's new_semantic_content_events, which is outside the conflict markers. Otherwise that path drops logprobs.server/api_server.pystream_with_cancellation(fix(server): keep a silent stream alive and notice a client that left FlashML-org/FreeToken#572 over fix(server): guarantee AbortMsg delivery on stream cancellation FlashML-org/FreeToken#222):asyncio.waitloop and keep-alive, plus fix(server): guarantee AbortMsg delivery on stream cancellation FlashML-org/FreeToken#222'suidslist and shielded per-uid abort infinally.finishedis set only onStopAsyncIteration, and the pending__anext__is cancelled infinally.Live checks on the new build:
max_tokens2/8/64 return exactly 2/8/64 tokens.127.0.0.1:1919and127.0.0.1:1920./v1/modelscontext_length 262144;/v1/models/→ 200.Aborting requestin the log.🤖 Generated with Claude Code
https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt