Repository navigation
Conversation
…sing an evictable cache entry The eager warmup forward right before capture leaves the decode (req, local) pair in the 64-entry FIFO memo; the capture hit it and baked its addresses into the graph without a reference. After ~64 distinct prefill shapes the entry is evicted, the allocator reuses the storage, and every decode replay feeds garbage indices to the fused hash kernel (out-of- bounds reads). A capture now always builds its own pair, which the graph recomputes into its pool on each replay. Assisted-by: Claude Opus 5.5
… snapshot HostPrefixTier.lookup is longest-prefix, and _rehydrate_snapshot attached whatever it returned to the node at the node's full depth. When a prompt diverges inside a node (the Qwen template re-rendering earlier turns without their thinking) the split node found an earlier, shorter snapshot and the request resumed the GDN recurrence with the tokens in between missing -- silently wrong output, and the bad state was kept on the node. Assisted-by: Claude Opus 5.5
…ed after the engine stream Two defects in upstream FlashML-org#525's HostPrefixTier: - it kept the LinearStatePool tensors from construction; a runtime rebuild with num_mamba_slots replaces them, so snapshots were read from / restored into orphaned tensors (wrong state, IndexError when the pool grew, and the old pool never freed); - its private copy stream never waited on the engine stream, so a restore into a slot freed while overlap's already-launched extra decode step still writes it could be overwritten. Assisted-by: Claude Opus 5.5
…age after mapping them FlashML-org#487 hoists and merges system messages in the frontend, but FlashML-org#391 maps developer -> system later, in the tokenizer worker. [system, developer, user] (or a developer message mid- conversation) reached the Qwen template as a non-first system message and failed with "System message must be at the beginning". The mapping now re-applies the merge. Assisted-by: Claude Opus 5.5
…tent The qwen3 reasoning parser starts inside reasoning unless thinking is off, but the Qwen template renders the final assistant turn as <think>...</think> + content, so a continue_final_message continuation (fork #47) never emits </think> and the whole reply came back under reasoning_content with empty content. Assisted-by: Claude Opus 5.5
The check was `<= 0`, which NaN passes; NaN or inf then poisons the logits of every token seen so far (fork #47's sampler plan) and the request silently returns garbage. Assisted-by: Claude Opus 5.5
asyncio.gather passes the first GenerationError up without cancelling its siblings, and the aborts that follow drop the events those siblings wait on, so they stayed pending forever (no accounting row, 'Task was destroyed but it is pending' at shutdown). Assisted-by: Claude Opus 5.5
…es and /v1/responses FlashML-org#222 only wrapped the OpenAI handlers, so an abandoned non-streaming Anthropic or Responses request kept decoding to max_tokens, blocking a single-slot server meanwhile. Both now use the same watcher and answer 499. Assisted-by: Claude Opus 5.5
…nalty prompt mask Fork #47's sampler plan scattered the raw prompt ids into a [rows, vocab+1] mask; image tokens carry pseudo-ids >= 1,000,000, so repetition_penalty with an image indexed out of bounds -- a device-side assert that takes the server down. They now land in the scratch column. Assisted-by: Claude Opus 5.5
…npacker introduced msgpack.Unpacker defaults to max_buffer_size=100 MiB where the unpackb it replaced sized the buffer to the message, so a larger message (a big image's pixel values) raised BufferFull and killed the receiving worker. Assisted-by: Claude Opus 5.5
… stream too Same race the GDN copies had (the KV-page tier is off here and refused on nvfp4 pools); also corrects the ordering comment and the msgpack buffer-cap comment. Assisted-by: Claude Opus 5.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #63. Fixes for 11 bugs found by a full correctness re-review of the
5080deploystack (every merged upstream PR and fork fix, reviewed where they interact). Each commit states its bug and adds a test that fails without the fix and passes with it; one commit (d0bafe4) covers two related bugs.Active in production before this PR:
9458d50lookupis longest-prefix). When a conversation diverged inside a cached node (the Qwen template re-rendering earlier turns without their thinking), the request resumed the GDN recurrence with thousands of tokens missing from its state: silently wrong output, kept on the node. Now exact matches only.3cd6c746d1f0ad[system, developer, user](or a mid-conversation developer message) failed with the template's "System message must be at the beginning": the frontend hoists system messages before the tokenizer maps developer -> system.59df75acontinue_final_messagecontinuation came back entirely inreasoning_content(the template closes</think>before the assistant content).63c1812repetition_penalty+ an image scattered multimodal placeholder ids (>= 1,000,000) out of bounds: a device-side assert that kills the server.1ac87c7Unpackercaps messages at 100 MiB (unpackbdid not); a larger message (a big image) killed the receiving worker.655d491/v1/messagesand/v1/responsesnever watched for disconnects, so an abandoned request decoded tomax_tokens.5c4be62repetition_penaltyNaN/inf passed validation and poisoned logits.dc060d9n > 1chat choice left its sibling generations pending forever.d0bafe4(part)Latent (needs a runtime
num_mamba_slotsrebuild):d0bafe4(part): the tier kept the GDN pool tensors a rebuild replaces (wrong restores, IndexError, the old pool never freed).d14f06dgives the dormant KV-page tier copies the same stream ordering.Found and not fixed (reported, low impact here):
use_fp8_lmhead/dense-FP8 predicates evaluated withoutconfig.quantin the reader vs the builder (only matters for checkpoints declaring their own lm_head/attention scheme); a dead duplicateParallelLMHead._logits; hostile-input gaps on a loopback server (huge prompt lists, >2^64 ints inlogit_bias,/tokenizeedge cases); per-step host syncs when logprobs/penalties are requested;min_tokensblocking EOS one step longer than asked.Testing (RTX 5080, production flags incl.
FT_GDN_HOST_TIER=1):pytest tests/ -m "not slow"at the merged tip8975217: 2462 passed, 206 skipped; the same 4 environment-only failures (3xtest_muse_glimmer_vision.pyon transformers 5.15.1,test_async_pull_coalescedneedingpytest-asyncio). feat(kvcache): host-resident prefix tier for hybrid GDN models (snapshots + KV pages) FlashML-org/FreeToken#525's script-style tier tests: 5/4/6 checks pass.[system, developer, user]-> 200 ('Blau');continue_final_message-> content' four, five, six', no reasoning;repetition_penalty: NaN-> 400; image +repetition_penalty: 1.1-> 200 (' red circle blue square'), server healthy; 70 distinct prompt lengths then a decode -> correct answer, no fault; a 13K-token prompt repeated -> 13.4 s then 1.7 s (the tier and prefix cache still hit on exact matches).🤖 Generated with Claude Code
https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt