Skip to content

Skip planning on error and not-found pages - #159

Open
DavertMik wants to merge 6 commits into
mainfrom
fix/skip-planning-on-error-pages
Open

Skip planning on error and not-found pages#159
DavertMik wants to merge 6 commits into
mainfrom
fix/skip-planning-on-error-pages

Conversation

@DavertMik

@DavertMik DavertMik commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Exploration burned a full planning round on pages with nothing to test — including the case in the reported trace, a 200 response whose only sign of being dead is its own copy.

Changes

explore-command.ts — sub-page skip. discoverNewSubPages checks the page right after visiting it. If getStateErrorPageError reports an error page: warn, mark it failed, move to the next candidate. No research, no planning, no AI call spent on it. This covers server-rendered errors — anything with a 4xx/5xx document status or a standard HTTP error title.

planner.ts — one prompt line. That check only sees HTTP status and standard error titles, so a page returning 200 with app-specific "doesn't exist" copy slips past it. Reading that from arbitrary wording is AI judgment, so the planning prompt carries the rule (general, no wording from any specific page): a page that reports a missing resource, shows a failure state, or holds no content and no controls gets an empty scenarios list, and nothing is invented for it.

Empty output is a skip, not a failure. Zero scenarios with no plan to expand used to throw the same generic planning error as a failed call, so exploration retried the page and then spent two more calls per remaining planning style on it — the prompt rule above bought nothing. An empty scenario list is a value now, not an exception: the planner returns the plan it built, and runAllStyles reads its length. No tests means nothing to test here, so it stops trying further styles and the next sub-page candidate is picked; the empty plan is not carried into the saved plans. An expanding plan is untouched — a later style returning nothing keeps the tests already planned.

Tests

tests/integration/planner.test.ts asserts the prompt carries the rule, that empty output yields an empty plan rather than an error, and that an existing plan survives an empty style.

bun test tests/integration/ and bun test tests/unit/ pass, format and lint clean.

The prompt half can only be confirmed in a live run — worth the regression label if you want it proven against the page in the trace.

🤖 Generated with Claude Code

https://claude.ai/code/session_013TYaLWuuLjj5mb26NuYjoj

DavertMik and others added 3 commits August 29, 2026 14:34
Exploration spent a full round of planning on pages that had nothing to
test. A sub-page that turned out to be an error page was researched and
planned anyway, and a page that reports a missing record with a normal
200 response — no HTTP status to detect it by — came back with a set of
invented scenarios written against a page with nothing to click, once
per planning style.

Sub-page exploration now checks the page it just visited and moves on to
the next candidate when it is an error page, before any research runs.
For pages that only their own copy identifies as dead, the planner is
told to return no scenarios, and no scenarios for a page nothing has
been planned for is reported as an error page — which lands in the same
skip.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TYaLWuuLjj5mb26NuYjoj
Zero scenarios meant the model returned nothing, which is not the same
as the page being dead — the research can be thin, the call can fail,
the model can decline. Reporting that as an error page put a diagnosis
on the page that nothing had established, and the !feature gate was
only there to hide how often the guess would be wrong.

Empty output is a planning failure again, as before. The prompt rule
stands on its own: a page with nothing to exercise gets no scenarios,
so nothing is invented for it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TYaLWuuLjj5mb26NuYjoj
Comment thread src/ai/planner.ts
<task>
Based on the page research, create ${this.MIN_TASKS}-${this.MAX_TASKS} exploratory testing scenarios.
For each scenario provide specific steps and expected outcomes.
Exception: if the page reports the requested resource is missing, shows a failure state, or holds no content and no controls, return an empty scenarios list. Never invent tests for a page with nothing to exercise.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plan() still throws when scenarios is empty and there is no current plan. So this turns the intended skip into a planning error. Should an empty list be handled as a valid skip?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworked in 64ad7e7 — the earlier EmptyPlanError is gone. An empty scenario list is a value, not an exception: plan() returns the plan it built and runAllStyles checks plan.tests.length === 0, which means nothing to test here, so it stops trying further planning styles and the loop picks the next sub-page candidate. An empty plan is not pushed into the saved plans either.

Expanding an existing plan is unchanged, and a genuine planning failure still retries as before.

DavertMik and others added 3 commits September 1, 2026 03:04
Zero scenarios with no plan to expand threw the same generic planning
error as a failed call, so exploration retried the page and then spent
another two calls per remaining planning style before giving up on it.

The planner now throws EmptyPlanError, which says only what happened to
the output and nothing about the page. Explore skips straight to the next
sub-page candidate on it, with no retry and no further styles.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NrPgHXFYH6PzRksorVUGFC
An empty scenario list is a value, not an exception. The planner returns
the plan it built, empty or not, and explore reads its length: no tests
means nothing to test here, so it stops trying further planning styles
and the next sub-page candidate is picked. An empty plan is not carried
into the saved plans either.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NrPgHXFYH6PzRksorVUGFC
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants