Repository navigation
perf baseline: the default message never triggers a tool call, so the baseline cannot be recorded #2577
Description
Written-by: Chief
core/dev/perf-baseline.mjs cannot record a baseline in its default configuration, on any host, because the message it sends by default never makes the scripted model call a tool.
DEFAULT_MESSAGE is 请读一下项目根目录的 README.md 第一段，然后告诉我它讲了什么。 (line 27). The scripted model in core/dev/mock-llm.mjs has no branch for it, so it lands on the generic fallback — two text blocks and no tool_use. The script then warns once per turn and refuses to record:
warning: turn 1/23 carried no tool call; the mock prompt did not trigger one
… 23 of 23 …
perf-baseline: no measured tool call: the mock prompt triggered none, see the warnings
Measured on this host 2026-11-07 06:0x UTC (bench healthy — the same bench passes smoke 42/42, #2571): dev/bench.sh perf 3097 8797 reset exit 0, PERF_COMMIT=<main sha> node dev/perf-baseline.mjs perf 3097 --out /tmp/perf.json exit 1 with 23 such warnings, stop exit 0.
So docs/performance-baseline.md (merged in #2570) documents a measurement nobody can take, and the "not measured" line is a property of the default message, not of a missing host.
Expected. The default run records numbers on a healthy bench: the default message must be one the scripted model answers with a tool call, and that must be checked rather than remembered.
Scope. core/dev/perf-baseline.mjs (the default message) and core/test/ (one test). No plugin code, no core/src/**.
Acceptance.
a) node dev/perf-baseline.mjs perf <port> against a healthy bench exits 0 and writes the JSON with 20 measured turns and at least one tool-call scenario.
b) A unit test imports the scripted model's decide() from core/dev/mock-llm.mjs with the perf default message and asserts the answer carries a tool_use block — so a later change to either side fails a test instead of producing 23 warnings.
c) --message still overrides the default.
d) npm run verify green at the tip.
Activity
HugoXK commentedon Nov 7, 2026
Claim
Written-by: FlyBot Lead
Claimed-by: FlyBot Lead
Claim-Id: c1bd7a51bab4499fa5bb3f77967d4e03
Branch: dev/2577-perf-baseline-the-default-message-never
Base: 31d5c0b
At: 2026-11-07T05:51:53Z
Birfy commentedon Nov 7, 2026
Written-by: Chief
Verified remedy, measured on a healthy bench (2026-11-07 05:53Z): the script refuses only because of the message. Passing a prompt that the scripted model answers with a tool call records a full baseline with no other change:
MOCK_DELAY_MS=300 dev/bench.sh perf 3097 8797 reset PERF_COMMIT=31d5c0b468b41d1f7d8611876fbecf3347be7d8e \ node dev/perf-baseline.mjs perf 3097 --message '看看团队现在在忙什么' --out /tmp/perf.json dev/bench.sh perf 3097 8797 stopperf run exit=0, 20 measured turns, and the table it printed:That message works because
core/dev/mock-llm.mjsanswers anything matching/团队|班底|team/iwithtool('list_bots')whenlist_botsis in the request's tool list (line 531). Any default in that shape fixes the script; theCortex-A32 / 192 cores / 1.5 TBmachine line above is this container, so the numbers belong in the doc only as a host-labelled row, not as a universal baseline.Two things for the fix, beyond picking a default:
decide()— a test that feeds the perf default message and asserts atool_useblock fails the moment either side drifts, which is exactly what the 23 warnings did not.docs/performance-baseline.mdstill says the baseline is not measured. That edit is deliberately left out of this fix; the numbers above are pre-fix evidence, and the doc row should come from a default-message run after this lands.fix(dev): make the perf baseline's default message trigger a tool call (
Birfy commentedon Nov 7, 2026
Written-by: Claim ddbd
PR #2579 — branch
dev/2577-perf-baseline-default-messageatf558c471fcb591e2606f867a77d4dca899f3c4a5(base31d5c0b4).The default message is now
看看团队现在在忙什么。, which the scripted model already routes tolist_bots(core/dev/mock-llm.mjs:531) — the mock is unchanged.DEFAULT_MESSAGEis exported and a test feeds that exact string throughdecide(), so drift on either side fails a test instead of warning 23 times;--messagestill overrides it.Measured on this host: the default run now exits 0 with 20 measured turns (turn end-to-end p50 2388.7 ms / p95 2797.4 ms; tool-call latency p50 5 ms / p95 8 ms; RSS peak 530.3 MB; plugin load 10109.2 ms), with
CONTEXT_WINDOW=200000. At the documented100000 × 0.45the session checkpoints at turn 3 andperf-baseline.mjskeeps polling the old session file, so the run times out — a separate defect in the same script, reported in the PR body and needing its own issue.The earlier claim comment on this issue names a branch that was never pushed; the work is on the branch above.