From 0bda3d551e279436e7041d7564f0f2424fb33ed7 Mon Sep 17 00:00:00 2001 From: Anthony Ronning <101225832+AnthonyRonning@users.noreply.github.com> Date: Wed, 7 Oct 2026 15:34:52 +0000 Subject: [PATCH 1/2] maple-agent: add the phase 0 scenario checklist and a performance check Two documents and one script for the trimmed build. The scenario checklist has one scenario per feature that carries over to the next runtime, marks the ones the runtime tests already cover, and keeps a results table; it ran on this build and runs again on the new runtime before that merges. The performance check is a smoke and sanity check, not a gate: cold start to an interactive window from the app's own startup markers, resident memory at idle and after a long task, and a by-hand note on whether a long streamed answer stays smooth. The numbers recorded here come from a release build on the machine named in the document and are to be repeated on the same machine before the swap merges. Co-Authored-By: Claude Fable 5.1 --- apps/maple-agent/README.md | 6 ++ apps/maple-agent/docs/performance-check.md | 47 +++++++++ .../docs/phase0-scenario-checklist.md | 95 +++++++++++++++++++ apps/maple-agent/scripts/perf-check.sh | 41 ++++++++ 4 files changed, 189 insertions(+) create mode 100644 apps/maple-agent/docs/performance-check.md create mode 100644 apps/maple-agent/docs/phase0-scenario-checklist.md create mode 100755 apps/maple-agent/scripts/perf-check.sh diff --git a/apps/maple-agent/README.md b/apps/maple-agent/README.md index caf04a752..6418218ef 100644 --- a/apps/maple-agent/README.md +++ b/apps/maple-agent/README.md @@ -495,6 +495,12 @@ a release build. `just ci` is the full local format, lint, build and test gate; `just release` separately validates the optimized binary. PR jobs have no signing or publishing credentials. See the root agent guide for shared checks. +Two by-hand checks complement the automated ones: the +[scenario checklist](docs/phase0-scenario-checklist.md), one scenario per +feature with the results of each run, and the +[performance check](docs/performance-check.md), a short release-build smoke +test of cold start, memory and streaming smoothness. + ## Update and release boundary Prod and unpackaged builds accept stable `maple-agent-vX.Y.Z` releases. Dev diff --git a/apps/maple-agent/docs/performance-check.md b/apps/maple-agent/docs/performance-check.md new file mode 100644 index 000000000..ebe7d75d1 --- /dev/null +++ b/apps/maple-agent/docs/performance-check.md @@ -0,0 +1,47 @@ +# Performance check + +A short smoke and sanity check for a release build, not a gate. It covers +what the runtime plan asks for before the runtime swap: cold start to an +interactive window, memory at idle and after a long task, and whether a long +streamed answer stays smooth. Time to first token depends on the network, so +only Maple's own startup is timed. Record the numbers with the machine they +were taken on, and repeat them on the same machine before the swap merges. + +## Recipe + +1. Build the release binary from the component directory: + + ```sh + just release + ``` + +2. Run the script with the environment the app normally gets (the API + origins and `XDG_*` roots of the account you test with), pointing it at + the release binary. The second argument is the idle period in seconds: + + ```sh + scripts/perf-check.sh target/release/maple-agent 30 + ``` + + The script starts the app with `RUST_LOG=warn,maple_agent=debug`, waits + for the `startup: runtime started` marker, prints every `startup:` marker + with its millisecond offset from process start, samples resident memory + when the window is open and again after the idle period, and leaves the + app running. + +3. In the running app, start a task and ask for a long streamed answer, for + example a 1500-word essay. Watch the transcript while it streams and note + whether it scrolls smoothly or stalls. When the answer ends, sample memory + once more with the `ps` line the script prints, then quit the app. + +The startup markers come from the app itself (`startup: logging ready`, +`settings loaded`, `backend ready`, `gpui app ready`, `window open`, `first +render`, `chat screen open`, `local bootstrap applied`, `credential +validation done`, `runtime started`). "Window open" is the number to watch +for cold start; "runtime started" is when the task list is live. + +## Recorded runs + +| Date | Build | Machine | Window open | Runtime started | Memory, window open | Memory after idle | Memory after long task | Long answer | +|---|---|---|---|---|---|---|---|---| +| 2026-10-07 | 93a235d2 (phase 0 trimmed build), `just release` | Apple M3 Max virtual machine, 8 cores, 48 GB, macOS 27.0.1 | 729 ms | 1956 ms | 113.7 MB | 114.0 MB | 186.8 MB | smooth: six samples 8-10 s apart over an 85 s stream each showed the transcript further along, following the tail; CPU 9-10 % | diff --git a/apps/maple-agent/docs/phase0-scenario-checklist.md b/apps/maple-agent/docs/phase0-scenario-checklist.md new file mode 100644 index 000000000..902845f9d --- /dev/null +++ b/apps/maple-agent/docs/phase0-scenario-checklist.md @@ -0,0 +1,95 @@ +# Phase 0 scenario checklist + +One scenario per feature that carries over to the next runtime. Run it on the +trimmed build at the end of phase 0 and again on the new runtime before it +merges. Scenarios the runtime tests cover are marked **automated** with the +test that covers them; the rest are run by hand and the result recorded in the +table at the end. + +Setup: a signed-in account on a local or development stack, a project folder +with a few files, the built-in computer-use backend set up (macOS or Linux), +Codex and Claude Code installed and signed in on the machine, and one custom +stdio MCP server configured in Settings > Integrations. + +## Tasks + +| # | Scenario | How | Pass when | +|---|---|---|---| +| T1 | Automatic then semantic title | Start a task with a one-line prompt | The sidebar shows a generated title at once and a semantic one after the first reply | +| T2 | Rename | Rename the task from its sidebar menu | The new name survives a relaunch | +| T3 | Settle and archive | Settle a finished task, archive it, then restore it | It moves between Active, Settled and Archived and comes back with its history | +| T4 | Delete | Delete a task | It disappears from every list and its folder of attachments is gone | +| T5 | Pin | `/pin` a task, then unpin it | The pin shows in the sidebar and clears | +| T6 | Projects and recent folders | Create a task in a second folder | The project switcher lists both roots; the task runs in its own root | + +## Composer + +| # | Scenario | How | Pass when | +|---|---|---|---| +| C1 | Queued follow-up | Press Enter with a message while a run is active | The message waits in the queue and runs as the next turn (**automated**: `desktop_send_during_active_run_stages_a_native_queue`) | +| C2 | Steering | Ctrl/Cmd+Enter during a run | The message lands inside the current turn (**automated**: `background_completion_steers_active_task_without_new_run`) | +| C3 | Stop and resume | Stop a long shell command, then send a follow-up | The stopped notice shows, no orphaned declined pair, and the follow-up runs | +| C4 | Attachments | Attach an image by picker and by paste, then send | The model sees the image (vision model) or a `read_image` reference (text model) | +| C5 | Voice input | Dictate a message | The transcript of the dictation lands in the composer | +| C6 | Per-task pickers | Switch the model, Web and an integration on the draft, then send | The created task carries the three choices | + +## Slash commands and skills + +| # | Scenario | How | Pass when | +|---|---|---|---| +| S1 | `/new`, `/pin`, `/web`, `/model`, `/help` | Run each | Each does what `/help` says | +| S2 | `/compact` | After a few turns | "Compaction completed." and the context ring drops | +| S3 | Skill command | Run a bundled skill such as `/handoff` | The skill's instructions load and the task follows them | +| S4 | Project skill behind trust | Open an untrusted project with `.agents/skills` | The trust prompt appears; after trusting, the skill is offered | +| S5 | Instruction files and system prompt | A project with `AGENTS.md`; a custom system prompt in Settings | Both reach the model (ask it to quote them) | + +## Tools + +| # | Scenario | How | Pass when | +|---|---|---|---| +| L1 | `read`, `shell`, `edit`, `write` | One prompt that uses all four | Each runs without a card and the files match | +| L2 | `read_image` | A local and a public https image | Both are described | +| L3 | `todo_write` | Ask for a multi-step plan | The plan pins above the composer and updates | +| L4 | `web_search` and `open_url` | Ask for a page title | Web results and the title come back | +| L5 | `request_user_input` | Ask the model to ask you a question | The card shows; the answer reaches the model; Escape skips | + +## External agents + +| # | Scenario | How | Pass when | +|---|---|---|---| +| E1 | Codex start, send, status, cancel | Blocking `agent_start`, then `agent_send`, `agent_status`, and Stop from the row | Rows stream activity; cancel kills the process (**automated**: `external_agents::tests`) | +| E2 | Claude Code | The same with `provider: claude` | Same | +| E3 | Background agent | `agent_start` with `background: true` | The row stays after the turn; the completion notice and the model's follow-up arrive | +| E4 | Questions on Maple's cards | Codex `request_user_input`; Claude `AskUserQuestion` | Both render as Maple's question card and the answers reach the agent | +| E5 | Approvals | A Codex configuration that asks on request; a Claude `Write` | Accepted at once, no card (**automated**: `external_agent_approvals_are_accepted_without_a_card`) | + +## Computer use, MCP, ACP + +| # | Scenario | How | Pass when | +|---|---|---|---| +| U1 | Built-in CUA | Enable it in Settings, ask a task to list open windows | The task sees the windows; the per-task switch works | +| U2 | Old computer-use task | Open a task saved by a build with the standalone driver | The driver does not start; the task's Cua row reads unconfigured (**automated**: `persisted_cua_driver_stdio_entry_is_stripped_before_any_agent_starts`) | +| M1 | Custom stdio and HTTP MCP servers | Enable one of each on a task | Their tools are offered directly and run | +| A1 | ACP agent mode | A stdio ACP client: `session/new`, a tool prompt, `session/list`, `session/load`, cancel, `/compact` | No modes advertised, zero `session/request_permission` for tools, the trust chooser on an untrusted project (**automated**: `acp::tests`) | +| A2 | Proxy, `login`, `--version` | Run each mode | Each works as the README says | + +## Side models and the rest + +| # | Scenario | How | Pass when | +|---|---|---|---| +| D1 | Tool and thinking summaries | Setting on; run a tool | One-line summaries appear and turn off with the setting | +| D2 | Image descriptions for text models | Attach an image on a text-only model | The description reaches the model | +| D3 | Context ring and compaction | A long task | The ring fills; `/compact` empties it | +| D4 | Plan meter | Settings > Usage | The plan, percent used and reset date show, nothing else | +| D5 | Notifications | Finish a task and ask a question with the window in the background | Two notifications, none for permissions | +| D6 | Appearance, fonts, motion, Vim | Change each setting | Each applies live | +| D7 | Old task with `delegate` history | Open one | Its old calls render as plain rows and new runs offer no `delegate` (**automated**: `persisted_summon_extension_is_removed_on_the_next_run`) | + +## Results + +Record each run here: date, build (commit), machine, lane, and any scenario +that did not pass with a note. + +| Date | Build | Machine | Lane | Result | +|---|---|---|---|---| +| 2026-10-07 | phase 0 trimmed tree (stack #1092 top), `just release` | Apple M3 Max virtual machine, 8 cores, 48 GB, macOS 27.0.1 | Local | Passed on this build: T1-T5, S1, L1, L3, C1, A1, A2, D1, D3, D4, D6. Passed earlier the same day on debug builds of the same stack: C3, C6, S2, S4, L2, L4, L5, E1-E5, U2, D7, M1 (stdio). Not run: C2 by hand (Cmd+Enter not deliverable by the test harness; automated test covers it), C4, C5, T6, S3, S5, U1 (the VM cannot grant the macOS permissions), M1 HTTP, D2, D5. No scenario failed. | diff --git a/apps/maple-agent/scripts/perf-check.sh b/apps/maple-agent/scripts/perf-check.sh new file mode 100755 index 000000000..1b6fff060 --- /dev/null +++ b/apps/maple-agent/scripts/perf-check.sh @@ -0,0 +1,41 @@ +#!/bin/sh +# Smoke-level performance check for a Maple Agent release build. +# +# Measures what the plan asks for, on this machine, as a sanity check rather +# than a gate: cold start to an interactive window, memory at idle and after a +# task, and whether a long streamed answer stays smooth (recorded by hand, +# see the prompt below). Time to first token depends on the network, so only +# Maple's own startup is timed. +# +# Usage: perf-check.sh [idle seconds] +# Run it with the same environment the app normally gets (MAPLE_API_URL, +# XDG_* and so on). It launches the binary with RUST_LOG=warn,maple_agent=debug, +# reads the `startup:` markers from its stderr, samples resident memory once +# the window is open and again after the idle period, then leaves the app +# running so a long task can be driven by hand. Stop the app yourself. +set -eu +binary=${1:?usage: perf-check.sh [idle seconds]} +idle=${2:-30} +log=$(mktemp "${TMPDIR:-/tmp}/maple-perf.XXXXXX") +RUST_LOG=warn,maple_agent=debug "$binary" >"$log" 2>&1 & +pid=$! +echo "pid $pid, log $log" +for _ in $(seq 1 600); do + if grep -q 'startup: runtime started' "$log"; then break; fi + sleep 0.1 +done +echo "--- startup markers (ms since process start)" +grep -o 'startup: .* at [0-9]* ms' "$log" | sed 's/startup: //; s/ from [^ ]* at / at /' +rss() { ps -o rss= -p "$pid" | awk '{printf "%.1f MB", $1/1024}'; } +echo "--- memory" +echo "window open: $(rss)" +sleep "$idle" +echo "after ${idle}s idle: $(rss)" +cat < Date: Wed, 7 Oct 2026 19:20:57 +0000 Subject: [PATCH 2/2] maple-agent: shorten the performance check and fix the checklist The performance check now asks for three things on a release build: the time to an open window, memory after a long streamed answer, and whether streaming stays smooth. The script and the extra startup and idle numbers go. The scenario checklist drops its planning-stage name, cites the build it ran on, and no longer claims that a background-completion test covers Ctrl/Cmd+Enter steering. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01STEaSNTNUiiNNLXcKidE2d --- apps/maple-agent/README.md | 8 +-- apps/maple-agent/docs/performance-check.md | 50 +++++-------------- ...rio-checklist.md => scenario-checklist.md} | 15 +++--- apps/maple-agent/scripts/perf-check.sh | 41 --------------- 4 files changed, 24 insertions(+), 90 deletions(-) rename apps/maple-agent/docs/{phase0-scenario-checklist.md => scenario-checklist.md} (86%) delete mode 100755 apps/maple-agent/scripts/perf-check.sh diff --git a/apps/maple-agent/README.md b/apps/maple-agent/README.md index 6418218ef..6b4126e75 100644 --- a/apps/maple-agent/README.md +++ b/apps/maple-agent/README.md @@ -496,10 +496,10 @@ a release build. `just ci` is the full local format, lint, build and test gate; signing or publishing credentials. See the root agent guide for shared checks. Two by-hand checks complement the automated ones: the -[scenario checklist](docs/phase0-scenario-checklist.md), one scenario per -feature with the results of each run, and the -[performance check](docs/performance-check.md), a short release-build smoke -test of cold start, memory and streaming smoothness. +[scenario checklist](docs/scenario-checklist.md), one scenario per feature +with the results of each run, and the +[performance check](docs/performance-check.md), a quick release-build sanity +check. ## Update and release boundary diff --git a/apps/maple-agent/docs/performance-check.md b/apps/maple-agent/docs/performance-check.md index ebe7d75d1..6357fd0c5 100644 --- a/apps/maple-agent/docs/performance-check.md +++ b/apps/maple-agent/docs/performance-check.md @@ -1,47 +1,23 @@ # Performance check -A short smoke and sanity check for a release build, not a gate. It covers -what the runtime plan asks for before the runtime swap: cold start to an -interactive window, memory at idle and after a long task, and whether a long -streamed answer stays smooth. Time to first token depends on the network, so -only Maple's own startup is timed. Record the numbers with the machine they -were taken on, and repeat them on the same machine before the swap merges. +A quick sanity check on a release build, not a gate. Run it on the same +machine before and after a change to the Agent's runtime, and compare. -## Recipe - -1. Build the release binary from the component directory: +1. Build the release binary and start it with startup logging: ```sh just release + RUST_LOG=warn,maple_agent=debug target/release/maple-agent 2>startup.log & + pid=$! ``` -2. Run the script with the environment the app normally gets (the API - origins and `XDG_*` roots of the account you test with), pointing it at - the release binary. The second argument is the idle period in seconds: - - ```sh - scripts/perf-check.sh target/release/maple-agent 30 - ``` - - The script starts the app with `RUST_LOG=warn,maple_agent=debug`, waits - for the `startup: runtime started` marker, prints every `startup:` marker - with its millisecond offset from process start, samples resident memory - when the window is open and again after the idle period, and leaves the - app running. - -3. In the running app, start a task and ask for a long streamed answer, for - example a 1500-word essay. Watch the transcript while it streams and note - whether it scrolls smoothly or stalls. When the answer ends, sample memory - once more with the `ps` line the script prints, then quit the app. - -The startup markers come from the app itself (`startup: logging ready`, -`settings loaded`, `backend ready`, `gpui app ready`, `window open`, `first -render`, `chat screen open`, `local bootstrap applied`, `credential -validation done`, `runtime started`). "Window open" is the number to watch -for cold start; "runtime started" is when the task list is live. + Once the window is open, `grep 'startup: window open' startup.log` gives + the time from process start to the window. -## Recorded runs +2. Ask for a long streamed answer, such as a 1500-word essay, and note + whether the transcript scrolls smoothly. Then read the memory in KB with + `ps -o rss= -p "$pid"`. -| Date | Build | Machine | Window open | Runtime started | Memory, window open | Memory after idle | Memory after long task | Long answer | -|---|---|---|---|---|---|---|---|---| -| 2026-10-07 | 93a235d2 (phase 0 trimmed build), `just release` | Apple M3 Max virtual machine, 8 cores, 48 GB, macOS 27.0.1 | 729 ms | 1956 ms | 113.7 MB | 114.0 MB | 186.8 MB | smooth: six samples 8-10 s apart over an 85 s stream each showed the transcript further along, following the tail; CPU 9-10 % | +| Date | Build | Machine | Window open | Memory after a long answer | Streaming | +|---|---|---|---|---|---| +| 2026-10-07 | `93a235d2` | Apple M3 Max virtual machine, macOS 27.0.1 | 729 ms | 187 MB | Smooth | diff --git a/apps/maple-agent/docs/phase0-scenario-checklist.md b/apps/maple-agent/docs/scenario-checklist.md similarity index 86% rename from apps/maple-agent/docs/phase0-scenario-checklist.md rename to apps/maple-agent/docs/scenario-checklist.md index 902845f9d..d12efe8be 100644 --- a/apps/maple-agent/docs/phase0-scenario-checklist.md +++ b/apps/maple-agent/docs/scenario-checklist.md @@ -1,10 +1,9 @@ -# Phase 0 scenario checklist +# Scenario checklist -One scenario per feature that carries over to the next runtime. Run it on the -trimmed build at the end of phase 0 and again on the new runtime before it -merges. Scenarios the runtime tests cover are marked **automated** with the -test that covers them; the rest are run by hand and the result recorded in the -table at the end. +One scenario per user-facing feature of Maple Agent. Run it before and after a +change to the Agent's runtime. Scenarios the runtime tests cover are marked +**automated** with the test that covers them; the rest are run by hand and the +result recorded in the table at the end. Setup: a signed-in account on a local or development stack, a project folder with a few files, the built-in computer-use backend set up (macOS or Linux), @@ -27,7 +26,7 @@ stdio MCP server configured in Settings > Integrations. | # | Scenario | How | Pass when | |---|---|---|---| | C1 | Queued follow-up | Press Enter with a message while a run is active | The message waits in the queue and runs as the next turn (**automated**: `desktop_send_during_active_run_stages_a_native_queue`) | -| C2 | Steering | Ctrl/Cmd+Enter during a run | The message lands inside the current turn (**automated**: `background_completion_steers_active_task_without_new_run`) | +| C2 | Steering | Ctrl/Cmd+Enter during a run | The message lands inside the current turn | | C3 | Stop and resume | Stop a long shell command, then send a follow-up | The stopped notice shows, no orphaned declined pair, and the follow-up runs | | C4 | Attachments | Attach an image by picker and by paste, then send | The model sees the image (vision model) or a `read_image` reference (text model) | | C5 | Voice input | Dictate a message | The transcript of the dictation lands in the composer | @@ -92,4 +91,4 @@ that did not pass with a note. | Date | Build | Machine | Lane | Result | |---|---|---|---|---| -| 2026-10-07 | phase 0 trimmed tree (stack #1092 top), `just release` | Apple M3 Max virtual machine, 8 cores, 48 GB, macOS 27.0.1 | Local | Passed on this build: T1-T5, S1, L1, L3, C1, A1, A2, D1, D3, D4, D6. Passed earlier the same day on debug builds of the same stack: C3, C6, S2, S4, L2, L4, L5, E1-E5, U2, D7, M1 (stdio). Not run: C2 by hand (Cmd+Enter not deliverable by the test harness; automated test covers it), C4, C5, T6, S3, S5, U1 (the VM cannot grant the macOS permissions), M1 HTTP, D2, D5. No scenario failed. | +| 2026-10-07 | `93a235d2`, `just release` | Apple M3 Max virtual machine, 8 cores, 48 GB, macOS 27.0.1 | Local | Passed on this build: T1-T5, S1, L1, L3, C1, A1, A2, D1, D3, D4, D6. Passed earlier the same day on debug builds of the same stack: C3, C6, S2, S4, L2, L4, L5, E1-E5, U2, D7, M1 (stdio). Not run: C2 (the test harness cannot deliver Cmd+Enter), C4, C5, T6, S3, S5, U1 (the VM cannot grant the macOS permissions), M1 HTTP, D2, D5. No scenario failed. | diff --git a/apps/maple-agent/scripts/perf-check.sh b/apps/maple-agent/scripts/perf-check.sh deleted file mode 100755 index 1b6fff060..000000000 --- a/apps/maple-agent/scripts/perf-check.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/bin/sh -# Smoke-level performance check for a Maple Agent release build. -# -# Measures what the plan asks for, on this machine, as a sanity check rather -# than a gate: cold start to an interactive window, memory at idle and after a -# task, and whether a long streamed answer stays smooth (recorded by hand, -# see the prompt below). Time to first token depends on the network, so only -# Maple's own startup is timed. -# -# Usage: perf-check.sh [idle seconds] -# Run it with the same environment the app normally gets (MAPLE_API_URL, -# XDG_* and so on). It launches the binary with RUST_LOG=warn,maple_agent=debug, -# reads the `startup:` markers from its stderr, samples resident memory once -# the window is open and again after the idle period, then leaves the app -# running so a long task can be driven by hand. Stop the app yourself. -set -eu -binary=${1:?usage: perf-check.sh [idle seconds]} -idle=${2:-30} -log=$(mktemp "${TMPDIR:-/tmp}/maple-perf.XXXXXX") -RUST_LOG=warn,maple_agent=debug "$binary" >"$log" 2>&1 & -pid=$! -echo "pid $pid, log $log" -for _ in $(seq 1 600); do - if grep -q 'startup: runtime started' "$log"; then break; fi - sleep 0.1 -done -echo "--- startup markers (ms since process start)" -grep -o 'startup: .* at [0-9]* ms' "$log" | sed 's/startup: //; s/ from [^ ]* at / at /' -rss() { ps -o rss= -p "$pid" | awk '{printf "%.1f MB", $1/1024}'; } -echo "--- memory" -echo "window open: $(rss)" -sleep "$idle" -echo "after ${idle}s idle: $(rss)" -cat <