Repository navigation
docs(casebook): three cases from a Qwen3.8 dedicated-engine campaign - #90
jiangkuaixue123 wants to merge 8 commits into
Conversation
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
Reviewed head 86f0033. Negative/INCONCLUSIVE diagnostic cases are valid casebook material; no campaign PASS or positive gain is required. Two delivery gaps remain: the packaged knowledge projection is stale and each case lacks a resolvable pinned measurement trail.
Ready CI independently corroborates the packaging problem: full-check job119809826280 fails test_generated_index_matches_current_sources_without_historical_git_objects; installed-package job119809825962 rejects the casebook SKILL.md digest. The knowledge-model-policy context is also currently red and should be resolved truthfully, separately from content quality.
Please regenerate the projection and bind each cited observation to retained evidence, then rerun ready/package checks. No GPU rerun or new measurement is requested. Review covered changed content, adjacent provenance conventions and CI logs; raw campaign execution was not repeated.
| - [Silent capacity and error failures](cases/qwen38-silent-capacity-and-error-failures.md): | ||
| HTTP 200 with an error frame, HTTP 400 with an empty body, failure at token 993. | ||
| - [Upstream FP8 kernels vs own Triton](cases/qwen38-upstream-fp8-kernels-vs-own-triton.md): | ||
| a numerically cautious kernel ran 100x slower than the borrowed library. |
There was a problem hiding this comment.
[P1] Regenerate the packaged index for the changed skill bytes
This edits an indexed SKILL.md without updating the generated knowledge projection. Both ready and installed-package checks fail on the source digest, so the shipped index does not match the packaged owner. Use the stock generator on the final source, retain its source revision, and rerun generation --check plus clean installed-reader checks.
| { | ||
| "type": "local", | ||
| "note": "Qwen3.8 dedicated-engine campaign, run records for the CUDA Graph round", | ||
| "paths": ["evidence/"] |
There was a problem hiding this comment.
[P2] Identify immutable evidence for each recorded observation
A generic 'evidence/' directory and prose note cannot locate the run supporting the 0.70→0.00 logits result. The other two new cases likewise cite no concrete source for token993 or 2.8/273/279 TFLOPS. Bind each case to specific retained file/run IDs and an immutable revision (or supported versioned resource and digest), with the relevant limits. Existing neighboring cases demonstrate a pinned ledger convention. This requests traceability, not a new hardware experiment.
There was a problem hiding this comment.
The new source pins at c0992c9e locate public records, but this provenance finding is not fully cleared: the unchanged FP8 narrative still contradicts its new evidence annotations.
- P2 — Retain the withdrawal in the claim itself.
qwen38-upstream-fp8-kernels-vs-own-triton.md:5still presents the 145 s prefill as an unqualified performance observation. The newly cited campaign record explicitly withdraws 145.3 s as first-call/JIT-confounded evidence. The new sidecar limitation acknowledges this, but the main explanation does not. - The same paragraph at lines 15–16 calls an abandon-below-250 threshold pre-registered, while the new sidecar and footer explicitly say that threshold is absent from the located records. Keep the documented 280 go line and observed 273/279 figures; withdraw the unsupported 250 historical assertion or label it as an unverified proposal with its own source.
- Minimal next step for the existing author, @jiangkuaixue123: align the body and sidecar, label the 145 s report as withdrawn/JIT-confounded rather than timing evidence, and preserve the useful source-bound microbenchmark account. No new hardware or model run is requested.
Current-version evidence
Assessed head: c0992c9e0e0a63f6ed61b753d5fec16928beb74a; API base: a1ee3266466194c16f4e2159ab845b758ed9561b; observed main: ae0d32218949fd6b906c3e7819ba0d830955c196.
Current body and its sibling JSON were compared with both cited ledger comments. Their retrieved body SHA-256 values match the new pins: 37b84900b48fc5da201792c280519969514d2bbed0afe75a3c4ef56329077201 and aedccaca1c8fbdb8c09eac5a79f9a48f43ec960ce33f4513822648c10b98a3dd.
This is a scoped source-fidelity follow-up in the original thread. It does not reuse old-head CI failures, repeat the separate SSE finding, resolve this thread, or approve the current PR. Source/readback inspection only; no local tests or experiments.
Dong1017
added
the
area:knowledge
ChatGPT Codex Connector
|
The retained SkillEval report records actual cloud generation and independent isolated replay: 19/19 → 19/19, with no observed gain on this fixture. The frozen baseline/candidate,
|
Skill evaluationPLANNED: not executed; there are no task outcomes yet.
Author verification steps (nonpaid; not yet run)
python tools/build_skill_index.py --source-revision 405033073d1140084da654c3701ac32cc4d1e4a1 --checkThe declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy. Evidence and logs · Retention: 30 days. Plan bindings, budget, frozen exam and advisory detailsAdvisory: obtain missing evidenceRetained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Model/settings:
Automatic execution / cumulative PR budget: {"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}Projection check selection (not an executed result): {"declared_in":"src/inferrsi/data/repo_packs/inferrsi/knowledge/README.md","source_revision":"405033073d1140084da654c3701ac32cc4d1e4a1","status":"ready"}Provider/settings: Uncovered / unsupported paths (89)
|
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
**范围：**复审当前 head 的 Qwen3.8 casebook 诊断建议与其新增来源绑定；此前索引再生成和不可追溯原始测量的意见仍留在原线程。本轮新增 1 个与 SSE 完成语义有关的可行动 P2（见 inline），不重发既有 provenance 反馈。仅静态检查文档、现有 review 与 API 契约；未运行代码、服务或 benchmark。
|
|
||
| | Symptom | What actually happened | Detection that would have caught it | | ||
| | --- | --- | --- | | ||
| | HTTP 200 with an SSE `error` frame mid-stream | Decode ran past the configured graph bucket, the engine fell back or failed, and the wrapper kept the stream open | Assert `completion_tokens` equals the requested count and `finish_reason == length` on every request | |
There was a problem hiding this comment.
P2｜不要把 max_tokens 上限当成所有请求的实际完成长度。 这里建议「每个请求都要求 completion_tokens 等于请求数且 finish_reason == length」，但 OpenAI-compatible Chat Completions 中请求的 token 数是上限；模型正常 EOS/stop sequence 会提前终止并返回 finish_reason=stop、更少的 completion tokens（API streaming 语义）。把这个规则作为通用检测会将合法成功流误报为故障，甚至鼓励服务层强行续写。
建议拆开两种断言：一般请求直接解析每个 SSE frame、拒绝内嵌 error/意外断流、要求正常终止帧和合法 finish_reason，使用可用的 usage 信息核对输出；特设的强制满长/容量边界 fixture（明确排除正常提前 stop）才断言输出达到冻结的 token 预算并以 length 结束。这样既能检测原案中的 HTTP 200 错误帧，也不会把正常 EOS 当回归。
910ebc5 to
1f8f9f1
Compare
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
增量复审 e9f95f6c：三项修正已核实；仍需完成原 provenance 线程中的一处 P2 文案修正。 不要求新增模型或硬件实验，既有作者继续处理即可。
- 已修正： SSE 指导现在允许正常提前
stop，只在明确禁用提前停止的 fixture 中要求完整长度；FP8 后续意见涉及的 145s/JIT 混淆和无依据的 250 阈值已明确撤回。 - 已修正，限定包装层面： 旧 source-digest finding中涉及的 7 个 canonical 文件，其实际 SHA-256 均与当前 packaged record 匹配，也与声明 source pin 的字节相同。没有复用旧 head 的 57-record 检查收据。
- P2，原 provenance 线程的剩余项： SKILL 入口、case 表格和 PR 摘要仍把 token 993 当作已确定的失败位置，但 case footer/sidecar 明确说明没有找到包含 993 的 artifact，所定位 job 1951 是 876 tokens。只读入口或表格的使用者仍会把未经支持的精确数字当成测量证据。**最小修正：**在断言位置直接标明它是未核实的历史回忆，或删除无依据的精确 token 数；若采用 876，应保留对应 job 身份，不混同两次运行。同步摘要，并在 canonical 修改后更新 projection。
- **独立前提仍未通过：**当前 declaration check 失败；如实保留作者/model/effort 信息，不能只改声明以迎合规则。软件 CI 与 PLANNED 均不证明来源测量真实或 skill 有增益。
检查版本、证据与限制
Head: e9f95f6c2156482b4a09537423455bdb5bafa4aa；前次检查 head: c0992c9e0e0a63f6ed61b753d5fec16928beb74a。PR API 当前返回 base 218565698d850e955f85d0d394a5f0e5e3619812；另已观测 main 50c19da43e0b358e6f990c00719891ad308a8ac1，本次不声称完成针对后者的整合测试。比较时区分了 rebase 带来的 main 变化与本 PR 自有增量。
Declared source pin: 161fac03e65a04461d643fdf57cbc60148414c5b。7 个 canonical 文件与 packaged record/source pin 的字节绑定已独立核对。4 个公开 ledger comment 的正文哈希与 sidecar 一致，这证明可追溯到这些公开陈述，不是原始测量重放。
Ready CI的 full-check / installed-package Python 3.10、3.11 及 qualification 通过；declaration job失败；SkillEval仅 plan/publish 成功，execute skipped、0 comparisons。
本次只做源码、文本差分与独立 hash 核对，未执行 PR 代码、完整 generator、模型或设备。保留旧缺失 raw logs 等已明确披露的覆盖限制；没有要求重新制造历史实验或解决他人的线程。
Local-author SkillEval evidence for the HTTP-200 error-stream slice (Refs #253)Result: both arms PASS 19/19 (
The candidate author read this PR's Setup
Limits
Evidence location: the retained evidence directory (about 0.4 MB: frozen plan, both arms' logs, artifacts and author-reported execution files, Independent replay (reviewer's copy; Linux host with bubblewrap ≥ 0.5 and system Python ≥ 3.10): git clone https://github.com/InferMatrixLab/InferRSI && cd InferRSI
git checkout 50c19da43e0b358e6f990c00719891ad308a8ac1
PYTHONPATH=src:. python3 tools/skill_eval_qualify.py --output "$QUAL" \
--revision 50c19da43e0b358e6f990c00719891ad308a8ac1 --runner-image "<host description>"
# Copy the retained evidence directory to $E unchanged, then:
sha256sum "$E/plan/freeze.json" # expect 6ba64278…fd03
PYTHONPATH=src:. python3 tools/skill_eval_compare.py report --output "$E"
Disposition requested: an independent reviewer should check access, log completeness, the current head and the existing review, then record a scoped disposition. This comment claims no skill benefit and no approval; #253 stays open. |
Independent review of the local-author SkillEval evidence (refs #253)Disposition: the retained evidence supports the posted result within its stated scope. Both arms pass 19/19 on the reconstructed CPU parser task at head The review was done by a separate agent that took no part in producing the evidence, working only from the evidence comment and the retained files. Checked, read-only:
Not checked: no trusted replay was run. Environment, access and usage remain author-reported. Corrections to the evidence comment:
Author note: the #253 requester approved this run on 2026-11-01 outside GitHub. It used a flat-rate subscription login, so no per-run dollar bound was set and per-run cost was not observed. Still open: this task does not exercise the token-993 wording in |
- CUDA Graph replay read capture-time state: greedy ids matched while per-step logits diverged by 0.70, so token-id equality is not a correctness oracle. - Silent capacity and error failures: HTTP 200 with an SSE error frame, HTTP 400 with an empty body and no log line, and a bucket-headroom failure at token 993. - Upstream FP8 kernels vs a self-written Triton GEMM: 2.8 vs 273 TFLOPS on the target card, with the pre-registered go/abandon thresholds and the two installation traps (placeholder PyPI package, vendored extension pinned to a torch minor version). Each entry records the observation, the disposition, a suggested revisit check, its applicability and its limitations; none makes a performance claim. Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com> Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Point each case at the issue ledger, engine blob, or retained evidence file that carries the observation, including the digest and the figures those records do not contain. Agent-assisted-by: Cursor / grok-4.7 [effort=unknown] (author) Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com> Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Split stream checks so a normal stop finish is success, and reserve an exact length assertion for a capacity fixture. Label the 145 s prefill as withdrawn JIT-confounded timing and drop the unsupported 250 threshold. Agent-assisted-by: Cursor / grok-4.7 [effort=unknown] (author) Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com> Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Refresh the packaged index from the rebased casebook source so the installed digest matches the current skill bytes. Agent-assisted-by: Cursor / grok-4.7 [effort=unknown] (author) Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com> Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Link each Qwen3.8 case to its typed record, keep digests in that record, and refresh the coverage view for the new case bytes. Agent-assisted-by: Cursor / grok-4.7 [effort=unknown] (author) Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com> Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Point the packaged index at the rebased casebook source after the SSE and FP8 wording alignment. Agent-assisted-by: Cursor / grok-4.7 [effort=unknown] (author) Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com> Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
The retained run is job 1951, which failed after 876 tokens. Drop the unverified token-993 figure from the skill summary and the case. Agent-assisted-by: Cursor / grok-4.7 [effort=unknown] (author) Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com> Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Refresh the packaged index and coverage view from the job 1951 wording. Agent-assisted-by: Cursor / grok-4.7 [effort=unknown] (author) Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com> Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
e9f95f6 to
dc5a0a3
Compare
![agent-ci-infra-bot[bot]](https://avatars.githubusercontent.com/u/343315635?s=60&v=4){kind=small}
There was a problem hiding this comment.
Reviewed commit: dc5a0a3
Coverage: Snapshot: 2643 files, 42171503 bytes; skipped 0 entries; Initial model context: 28 files; other accepted files available through tools; Git workspace contains a fixed snapshot, not full upstream history
| | --- | --- | --- | | ||
| | HTTP 200 with an SSE `error` frame mid-stream | Decode ran past the configured graph bucket, the engine fell back or failed, and the wrapper kept the stream open | On every request, parse each SSE frame, reject an embedded `error` or a truncated stream, and require a terminal frame whose `finish_reason` is `stop` or `length`. When usage is present, check `completion_tokens` against it. A normal EOS/`stop` finish below `max_tokens` is success. Assert `completion_tokens` equals a frozen budget and `finish_reason == length` only on a capacity fixture that disables early stop | | ||
| | HTTP 400, empty body, 0-byte server log | The error handler deliberately dropped all exceptions ("client addresses and arbitrary exceptions do not belong in service logs") | Log a structured line with exception type and a truncated message; never return an error status without a body | | ||
| | Job 1951 failed after 876 tokens with no message | The graph bucket was chosen from `prompt + headroom` with `headroom = 256`, while the request asked for 2048 output tokens | Validate bucket coverage against maximum output length, not against prompt slack, before starting the run | |
There was a problem hiding this comment.
F1 — medium, must-fix, evidence provenance — Job 1951's output length is taken from a different run
- Where:
src/inferrsi/data/skill_library/methods/inference-optimization-casebook/cases/qwen38-silent-capacity-and-error-failures.md:15 - Problem: the third row now attributes "the request asked for 2048 output tokens" to job 1951. The record this case cites for job 1951 says a different number: nano-vllm PR 8 (body digest
b4c5b6c2…1684, the digest stored in the sibling JSON) reads "then errored after 876 tokens in the 1024-output request at the Graph capacity boundary". Issue acceptance: Qwen3.8 FP8 vs vLLM 0.32.0 — ≥25% TTFT or TPOT gain, ≤5% regression #67 comment 6346532120 agrees ("errored after 876 output tokens exactly at the 1024 Graph capacity boundary"), and the follow-up 6346928151 records the same request as changing "from 876-token error to 1024-token completion". The 2048 figure belongs to the separate, later 8192-input/2048-output demotion note inpreparation.md, which this file's own footer calls "the later headroom note". - Conditions: any reader using this row to derive the bucket-coverage rule; the number is the row's only quantitative evidence.
- Impact: the entry mixes two runs and states a measurement the cited job did not make, which is exactly the conflation the earlier provenance thread asked to avoid ("若采用 876，应保留对应 job 身份，不混同两次运行"). A reader checking 2048-output coverage would be validating against a run that only exercised 1024.
- Fix: state job 1951's request as recorded (1024 output tokens at the 1024 capacity boundary, 876 tokens emitted), and if the 8192/2048 demotion is kept, present it separately as the later
preparation.mdobservation rather than as job 1951.
|
This PR adds three Qwen3.8 diagnostic cases to the inference-optimization casebook, one SKILL.md line per case, and the regenerated knowledge projection. Packaging is now sound — rendering the index against the declared source pin 4050330 reproduces the committed projection byte-for-byte, F2 — medium, must-fix, admission policy — the knowledge-model-policy declaration cannot pass as written
F3 — medium, suggestion, consistency — the PR summary still asserts the withdrawn token 993
Reviewed commit: dc5a0a3 |
agent-ci-infra-bot
Bot
added
bot-reviewed
kind:docs
|
@Dong1017 requesting a Why: material authorship used Cursor with Head: The exception request (paths + reason) is already in the PR description declaration. If you approve, please post a top-level conversation comment with a |
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
Content review of dc5a0a3e, read-only. Three findings on the case files, all in the provenance class the earlier threads cover. None needs a new measurement, a GPU run or a paid model call. The admission decision on the declaration is separate and is not made here.
Verified first. The source binding on this head holds up. I recomputed every digest the three sidecars record, against the live objects, and all ten match:
- Issue #67 comment bodies 6337245823, 6331682106, 6336720892, 6336567189
nano-vllmgraph.pyandrunner.pyat3f7516a7,server.pyatf2d1adc9preparation.mdatd5e17195nano-vllmpull requests 3 and 8 bodies
graph_headroom = 256 and bucket_for(len(tokens) + self.graph_headroom) sit at runner.py lines 45, 144 and 155, as the sidecar states. The FP8 sidecar's statement that no raw microbenchmark file exists and that the digests are public ledger bodies is the right way to record that limit rather than imply a measurement trail that is not there.
The declaration's exception scope also lines up: the 87 listed paths are exactly the protected changed paths under src/inferrsi/data/skill_library/** and src/inferrsi/data/repo_packs/**/knowledge/**, with docs/skill-coverage.md and tests/test_knowledge_contract.py correctly outside it.
C1 — medium, must-fix — the job 1951 row attributes a 2048-token request to a 1024-token failure
-
Where:
cases/qwen38-silent-capacity-and-error-failures.mdline 15, third table row. -
Problem: the row reads "Job 1951 failed after 876 tokens with no message", with the cause "the graph bucket was chosen from
prompt + headroomwithheadroom = 256, while the request asked for 2048 output tokens". The located records describe a different request:- Issue #67 comment 6346532120: "The fourth ignore-EOS request errored after 876 output tokens exactly at the 1024 Graph capacity boundary."
- nano-vllm pull request 8 body: "errored after 876 tokens in the 1024-output request at the Graph capacity boundary."
The 2048 figure belongs to a different observation.
preparation.md, cited in the same sidecar, records that an8192-input/2048-outputrequest demotes to eager after its reserved decode window. That is a demotion, not a failure, and not job 1951."with no message" disagrees with the same records: the failure was emitted as a streamed error frame, and the official client reported
success=trueover it. That is row 1 of this table, so rows 1 and 3 currently describe one incident with two incompatible signatures. -
Impact: the round that withdrew token 993 put a number from an unrelated record in its place, at the same assertion site. A reader taking this row as the capacity lesson gets the wrong output length and the wrong failure signature.
-
Fix: state job 1951 against the 1024 capacity boundary and the 1024-output request. If the
prompt + headroomversus maximum-output-length lesson should keep the 8192/2048 illustration, give it its own row attributed topreparation.mdand label it a demotion. The detection column is sound either way.
C2 — low — "three successive regression gates" has no located source
- Where:
cases/qwen38-cuda-graph-state-not-accumulating.mdline 7, and the case table in the PR description ("Three regression gates passed on ids"). - Problem: Issue #67 records that greedy-id equality served as the correctness statement — PR #2's required description wording was "top1 与 greedy id 全同" — and that it was later falsified and demoted to a smoke check (comment 6337245823, section 5). Reading all 100 comments on #67 I found no record of three gates. Of every precise figure in the three cases, this count is the only one I could not bind to a record.
- Fix: drop the count, or cite whatever states it. The surrounding sentence carries the lesson without it.
C3 — low — 0.00e+00 is qualified everywhere except where it is asserted
- Where:
cases/qwen38-cuda-graph-state-not-accumulating.mdline 17. - Problem: the observation table gives
0.00e+00as a value. The footer and the sidecar's limitations both state that the ledger records a bitwise pass and that0.00e+00is this case's notation for it, so the disclosure is complete — but not at the place a reader takes the number from. This is the placement the token-993 item asked about, at much lower stakes:0.00e+00follows from "bitwise", where 993 had no support at all. - Fix: a short qualifier in the row would close it, for example "bitwise pass (
0.00e+00by notation)". Leaving it as is, with the sidecar carrying the caveat, is also defensible.
Sequencing. C1 is a canonical edit, so the projection regenerates and the head moves. Per docs/knowledge-model-policy.md, new commits require a renewed approval and the declaration's head_sha must track the head. Landing these corrections before the declaration is settled avoids spending an approval on a head that is about to change.
Checked, not checked. Static review of the three case files, their sidecars, SKILL.md, the PR description and the cited records; independent recomputation of the ten source digests above; the exception scope against the protected changed paths. I did not run the generator, the test suite or any repository tool, did not reproduce any measurement, and used no device. CI at dc5a0a3e stands on its own: full-check and installed-package pass on both Python versions, which covers the generated-index and packaged-digest tests that the first review's packaging finding failed on.
This comment is content review only. It is not merge clearance, and it records no admission decision.
What this adds
Three cases to the inference-optimization casebook, taken from a dedicated-engine
campaign on one H20-3e card. Each is a failure or decision that cost a round and
that transfers to other models and other stacks.
qwen38-cuda-graph-state-not-accumulatingqwen38-silent-capacity-and-error-failuresqwen38-upstream-fp8-kernels-vs-own-tritonSKILL.mdgains one line per case under "Recorded diagnostic options".Evidence state
Honest edges, all written into the entries:
marked INCONCLUSIVE because the logits comparison ran after it.
was not characterised beyond the failure itself.
gain over a reference server.
Format
Each case keeps the existing pattern: a
.mdsibling for the observation and thesuggested revisit check, a
.jsonsibling for identity, origin, applicability,limitations and attribution. JSON validated against the existing files' fields.
knowledge-model-policy