fix - #64
Closed
CrimsonDump wants to merge 1 commit into
Closed
fix#64CrimsonDump wants to merge 1 commit into
CrimsonDump wants to merge 1 commit into
Conversation
…ition, ROCm GLM-5.2/5.3 profile
Sync the PD serving layer (tilert.pd_vllm) with the downstream serve tree.
Admission bundle
- wire: PROTOCOL_VERSION 2 (hello refuses a v1 peer), accept/reject/done carry
a tenancy generation, wants_prompt_token_ids().
- receive_server: explicit tenancy state machine (FREE/RESERVED/TRANSFERRING/
COMPLETE/CANCELLING), monotonic generation, live-writer count, rid-scoped
release(), tombstones for late ranks, expect(), state_snapshot().
- prefill_connector: sender waits to be admitted before any RDMA write,
_send_with_retry with exponential backoff (tilert_admission_attempts),
admission_window_s, rank-0 prompt ids only when a penalty is requested.
Capabilities bundle
- capabilities.py + engine_iface (prepare_grammar, supports_{logprobs,
penalties,ignore_eos}; fail-closed defaults; StubEngine grows the same).
- decode_server: GET /capabilities, pre-flight validate_generation_request +
prepare_grammar before the wire-wait, _drain_own_kv/_abandon_pending_kv,
top_logprobs / grammar_spec / enable_thinking, TILERT_DECODE_POLL_MS.
Router decomposition
- pd_router split into request_gate, decode_pool (NodeLease, --queue-timeout,
honest 429), decode_response (DecodeReader), reply, presentation
(OpenAI envelope, usage gating, --force-include-usage), stop_strings (vLLM
detokenizer port + StopWindow), logprobs (top_logprobs <= 5), sampling,
generation_defaults (--model / --generation-config / --default-*), grammar_spec,
grammar_backend; openai_params vendored as tilert.pd_vllm.openai_params.
- oai_parser: IncrementalDetok(skip_special_tokens), .holding, .finish() for
multi-byte tails; glm47 remains the only family and the --parser default.
Profiles
- mla_nsa: top_p/top_k resolved the vLLM way (fixes the hard-coded 0.95/256),
ignore_eos clears the stop set, grammar hooks, declares_penalties = False,
sparse-KI layer expansion with an optional strict ki_layer_ids check.
- profiles/glm5_2.py: GLM-5.2/5.3 profile (79 layers, layout_version 12,
sparse KI full-layer set); on a ROCm torch build or
TILERT_PD_ENGINE_BACKEND=rocm it uses profiles/glm5_rocm_engine.py, otherwise
it lazily imports tilert.models.glm_5_2 and raises a clear ImportError.
- profiles/base.py: glm5_2 / glm5_3 aliases; build_engine signature unchanged.
Tier 3 decisions
- dsv32 stays at NUM_LAYERS = 62 (this tree is newer).
- glm5 keeps 79 layers / layout 10; no TILERT_PD_NO_MTP / layout 1010.
- tilert_sync_send is removed (breaking config change): sending inside the
forward window cannot coexist with admission retries, which must outlast the
prefill response. The key now logs a warning and is ignored.
- ToolCall.call_id stays public; an `id` property aliases it.
- dsv32 NUM_LAYERS, glm5 layout, --parser default glm47 and decode_server
--model default glm5 all keep this tree's values.
Tests: CPU-only subset under tests/pd_vllm (798 passed, 2 skipped without
vllm / a tokenizer on disk); flake8/mypy overrides for the test doubles.
README: PD section documents the admission version, the bf16 KV path for
builds without fp8_ds_mla (prefill --kv-cache-dtype auto, decode bf16), the
glm5_2/glm5_3 profile and the new router flags.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
CrimsonDump
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.