Performance baseline: measure current smoothness (help wanted)

Why

We do not know how smooth FlyBot actually is. A repo search found no performance baseline or perf issues (only E2E-related #204 and #324). Before optimizing anything, we need measured numbers on real machines. This issue asks for volunteers to measure the current state and report it.

What to measure (personal mode)

Record machine (OS, CPU, RAM), FlyBot version/commit, and model used, then:

Cold start: launch to the first message being sendable.
First-message latency: send to first streamed token (with a fast model, to separate FlyBot overhead from model time).
Streaming: does the input box stay responsive while text streams; does scroll jump?
Long session: a chat with 500+ messages and many tool calls: scroll smoothness, memory after 30 min, time to reopen it.
Background tab: hide the page for 5 min, return: does it recover (cf. Timeline panel stays on Loading… in a background tab (hidden page never does the first fetch) #259)?
Concurrency: run 3 and 6 Bots at once (tree example): total time, CPU/RAM, any stalls.
Timeline on: enable the plugin and repeat 4 and 6; note the overhead.
Slow or failing network: throttle or drop the model API: are waiting, timeout and retry messages clear, or is it a blank screen?

What to measure (team mode, two or more instances once available, #352)

Approval latency: from trigger to the card appearing at the other person.
Many tasks: 200+ cards on the board ([Feature RFC] Agent inbox board: one place for tasks, approvals and failures (personal first, team second) #349): load time and scroll.
Cross-instance retry: kill one side mid-task: is work duplicated or lost ([Team] Recover task handoffs without duplicate work #55)?

How to report

Comment with a small table: scenario, number (ms, fps, MB or subjective), machine, and a screen recording or trace for anything that feels slow. Mark each item fine / noticeable / bad. No tokens or personal content in recordings.

Output of this issue

A baseline table in the issue body, updated from the comments.
4-5 key metrics chosen from it, to record on each daily tag (Release cadence: at least one tag per day, small and stable #351) so regressions show up.
Follow-up issues only for items marked bad.

Process (2026-10-21)

Implemented by an agent from the spec in the triage comment of 2026-10-21; that comment is the spec of record and wins where it differs from the text above. Reviewed per .github/agent/review-policy.json — no Codex review. The earlier sentence about volunteers measuring manually and "cloud OpenCode with Codex review" is void; it predates the owner decision of 2026-10-19.

Triage on main 7b9e90b, 2026-10-19 ~12:10 Asia/Shanghai: needs-human, not agent-ready. Question for the owner: which metrics and which machine define the baseline? Without that a measurement cannot be checked.

Triage (2026-10-21): agent-ready. The question that parked this ("which metrics, which machine") is now decided by the owner: measure on the maintenance box itself (Node 24, same container), four metrics only, and ship a reproducible script plus one baseline table. One item at a time — this is the first.

Metrics (all on a mock bench via core/dev/bench.sh, DEEPSEEK_API_KEY=mock-key, no real model)

turn end-to-end p50/p95 — message POST → that turn's turn/end, from the same session events the activity feed folds (includes the mock delay; MOCK_DELAY_MS fixed).
tool-call latency p50/p95 — tool/call → tool/result per call.
resident RSS — VmRSS of the dsh web process (pid file in .dsh-test-<name>/web.pid): after ready, after warm-up, at the end; report the peak.
plugin load time — process start → the dsh web: http line in web.log (covers every plugin's install).

Acceptance criteria

node core/dev/perf-baseline.mjs <bench-name> <port> --out <file.json> on a freshly reset bench, TZ=UTC, 3 warm-up + 20 measured turns, each turn one message to the Chief's own chat that triggers one tool call. Exits non-zero if the bench is down or a turn times out, and leaves no process of its own behind.
Output is JSON { machine, commit, node, scenarios: [{ metric, unit, samples, p50, p95, peak }] } plus one markdown table row, so two runs diff cleanly.
docs/performance-baseline.md holds each metric's one-sentence definition, the exact command, and one real run on the maintenance box with machine line (OS, CPU, RAM), commit SHA, Node version, date. No thresholds yet: a baseline, not a gate.
core/test/perf-baseline.test.js covers nearest-rank p50/p95 (n=1, odd and even n), the JSON schema and the markdown row, so the numbers are not untested code.
Tooling only: no change under core/src/**, no DSH patch, writes only to .dsh-test-<name>/ and the paths below.

Change scope

new core/dev/perf-baseline.mjs
new docs/performance-baseline.md
docs/subsystems/ds-bot-core.md (owns core/dev/, required by the arch-doc check)
new core/test/perf-baseline.test.js
no changelog fragment (not user-visible)

Out of scope for this pass: the UI smoothness items (items 1–8) and team mode (9–11) — they stay follow-ups until there is a baseline.

Reproduce

dev/bench.sh perf 3097 8797 reset
node core/dev/perf-baseline.mjs perf 3097 --out /tmp/perf.json
dev/bench.sh perf 3097 8797 stop

Triage spec (2026-10-21, main 16f50984).

Acceptance for a PR

New core/dev/perf-baseline.mjs runs the measurable scenarios on the existing bench (core/dev/bench.sh + the scripted model): cold start, first-message latency (model isolated), streaming responsiveness, a 500+ message session reopen, background-tab recovery.
Emits a markdown table plus JSON: commit, machine, per-scenario ms / MB / fps, and a fine / noticeable / bad mark.
New docs/performance.md records the machine, the first baseline numbers, and how to rerun.
Four or five key metrics are named there for recording on each daily tag (Release cadence: at least one tag per day, small and stable #351).

Scope: core/dev/perf-baseline.mjs (new), docs/performance.md (new), one root package.json script. No product code.

Gate: NOT agent-ready yet — the issue asks for numbers on real machines, and a container + mock-model run is a different baseline. Blocking: the owner decides whether the container + mock-model baseline counts as the engineering baseline (real-machine numbers stay welcome as comments). If yes, this becomes the first item for Dev.

Correction to the gate line above (concurrent maintenance call): the maintenance round recorded the owner decision at 09:53 — maintenance box / same container, Node 24, four metrics, reproducible script plus one baseline table. So this is agent-ready and ordered first for Dev; the "NOT agent-ready yet" sentence is superseded. Keep the acceptance and scope above, run it on the bench, and write the numbers into docs/performance.md without claiming a real-machine baseline.

汇总（2026-10-21）：本 issue 现在保持 needs-human，不是 agent-ready。

规格以 6313125530 为准：维护轮这台机器（Node 24、同一容器）、四项指标（turn 端到端 p50/p95、工具调用延迟、常驻 RSS、插件加载耗时）、验收 5 条、改动 5 个文件（core/dev/perf-baseline.mjs、docs/performance-baseline.md、core/test/perf-baseline.test.js、docs/subsystems/ds-bot-core.md）。
09:55 那条 "Gate: NOT agent-ready yet" 是对放行状态的更正，不改变上面的规格内容。
卡点只有一个，是用户的一次二选一：(a) 容器内 mock 基线算不算工程基线（可复现脚本 + 表，本规格）；(b) 必须真机人工测量（人会占用机器，脚本就不必交付）。Chief 不降门槛，走用户点选。
拍板前 needs-human；拍板后要么按 (a) 转 agent-ready，要么按 (b) 改写测量方案。Dev 侧本期全机 shell 不可用，不催它。

Final triage state (Chief decision this round, 2026-10-21): agent-ready, first in the Dev order. The earlier "needs-human pending the user's a/b choice" is superseded: the Chief fixed option (a) — run on the maintenance box (Node 24, same container) and deliver a reproducible script plus one baseline table. Metrics fixed by the Chief: turn end-to-end p50/p95, tool-call latency, resident RSS, plugin load time. The 09:53 spec is the spec of record; keep the acceptance/scope there. One item at a time: #358, #471, #355 follow.

State aligned with the Chief: needs-human while the user's sandbox authorization card is open — the spec and acceptance above stand. Blocking: the user (approve danger-full-access, or install bubblewrap); the moment the card is approved this flips back to agent-ready and Dev claims it. This is an environment block, not missing information.

Written-by: Chief

沙箱已放开（这台机器现在的文件策略是 danger-full-access，workflow 节点同样能开 shell）。按 2026-10-21 10:11 那条写下的承诺——"the moment the card is approved this flips back to agent-ready"——本单现在转 agent-ready 并派活。

规格照旧，以 6313125530 为准：

四个指标：turn 端到端 p50/p95、工具调用延迟 p50/p95、常驻 RSS、插件加载耗时；基线表不设阈值（是基线，不是门）。
交付文件：core/dev/perf-baseline.mjs（新）、docs/performance-baseline.md（新：机器、数字、复跑方法）、core/test/perf-baseline.test.js（新：脚本自身可跑、输出形状稳定）。
跑法：core/dev/bench.sh + scripted model，DEEPSEEK_API_KEY=mock-key，Node 24；不碰 ~/.dsh。
真机人工项（滚动手感、后台标签恢复）只作为文档里的手动行，脚本不假装测得到。

已派 issue 快线 i356-impl。

Claim

Written-by: FlyBot Lead
Claimed-by: FlyBot Lead
Claim-Id: f1238b006c1f45d89804b46d14754031
Branch: dev/356-performance-baseline-measure-current-smo
Base: 5d9e301
At: 2026-11-06T16:11:53Z

Written-by: Dev

Implementing on the claimed branch dev/356-performance-baseline-measure-current-smo — tip 70a72fae04bec767a99e37e4bc1b6e0df4849f3c, base 5d9e3013. PR #2570 (4 files: core/dev/perf-baseline.mjs, core/test/perf-baseline.test.js, docs/performance-baseline.md, docs/subsystems/ds-bot-core.md).

Spec followed as the 09:53 comment states it: four metrics only (turn end-to-end p50/p95, tool-call latency p50/p95, resident RSS peak, plugin load time), tooling only, no thresholds.

One gap to know before review: this host has no deepseek-harness checkout and no pnpm, so core/dev/bench.sh cannot boot and I could not produce the first real run. docs/performance-baseline.md records the command and leaves the four rows as — with the reason, rather than numbers I did not measure. A follow-up pass on a bench host fills the table; the script and its 11 unit tests are in the tip.

Written-by: Chief

Decision for this pass (2026-11-06 17:10Z): acceptance 3 (one real measured run) is deferred; the script, the doc and the test ship now.

Why: I tried the fix myself first. The only harness checkout here is unbuilt, and building it needs two tools this host does not have — git (repo build metadata) and cc (the native/system addon). DSH_CLIENT_COMMIT_HASH covers the first; nothing covers the second. Filed as #2571 with both unlock options (install a toolchain, or provide a prebuilt harness).

So PR #2570 must ship with docs/performance-baseline.md:68-86 reading "not measured — this host cannot build the deepseek-harness checkout; see #2571" and the four metric rows left as —. No invented numbers: the script is the deliverable of this pass, the numbers are #2571's.
I am not relaxing the rest: the script runs the four metrics, the red-first test stays, the doc keeps the machine line and the rerun recipe. When #2571 is resolved, a follow-up fills the table and this issue closes with real numbers.

Record why the baseline table is empty, and where the run is tracked

Navigation Menu

Repository navigation

Performance baseline: measure current smoothness (help wanted) #356

Why

What to measure (personal mode)

What to measure (team mode, two or more instances once available, #352)

How to report

Output of this issue

Process (2026-10-21)

3 remaining items

Claim

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Performance baseline: measure current smoothness (help wanted) #356

Description

Why

What to measure (personal mode)

What to measure (team mode, two or more instances once available, #352)

How to report

Output of this issue

Process (2026-10-21)

Activity

Birfy commented on Oct 19, 2026

Birfy commented on Oct 21, 2026

Birfy commented on Oct 21, 2026

Birfy commented on Oct 21, 2026

Birfy commented on Oct 21, 2026

Birfy commented on Oct 21, 2026

3 remaining items

Birfy commented on Oct 21, 2026

Birfy commented on Nov 6, 2026

HugoXK commented on Nov 6, 2026

Claim

Birfy commented on Nov 6, 2026

Birfy commented on Nov 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions