Skip planning on error and not-found pages - #159
Open
DavertMik wants to merge 6 commits into
Open
Conversation
Exploration spent a full round of planning on pages that had nothing to test. A sub-page that turned out to be an error page was researched and planned anyway, and a page that reports a missing record with a normal 200 response — no HTTP status to detect it by — came back with a set of invented scenarios written against a page with nothing to click, once per planning style. Sub-page exploration now checks the page it just visited and moves on to the next candidate when it is an error page, before any research runs. For pages that only their own copy identifies as dead, the planner is told to return no scenarios, and no scenarios for a page nothing has been planned for is reported as an error page — which lands in the same skip. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013TYaLWuuLjj5mb26NuYjoj
Zero scenarios meant the model returned nothing, which is not the same as the page being dead — the research can be thin, the call can fail, the model can decline. Reporting that as an error page put a diagnosis on the page that nothing had established, and the !feature gate was only there to hide how often the guess would be wrong. Empty output is a planning failure again, as before. The prompt rule stands on its own: a page with nothing to exercise gets no scenarios, so nothing is invented for it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013TYaLWuuLjj5mb26NuYjoj
DenysKuchma
requested changes
Aug 31, 2026
| <task> | ||
| Based on the page research, create ${this.MIN_TASKS}-${this.MAX_TASKS} exploratory testing scenarios. | ||
| For each scenario provide specific steps and expected outcomes. | ||
| Exception: if the page reports the requested resource is missing, shows a failure state, or holds no content and no controls, return an empty scenarios list. Never invent tests for a page with nothing to exercise. |
Collaborator
There was a problem hiding this comment.
plan() still throws when scenarios is empty and there is no current plan. So this turns the intended skip into a planning error. Should an empty list be handled as a valid skip?
Contributor
Author
There was a problem hiding this comment.
Reworked in 64ad7e7 — the earlier EmptyPlanError is gone. An empty scenario list is a value, not an exception: plan() returns the plan it built and runAllStyles checks plan.tests.length === 0, which means nothing to test here, so it stops trying further planning styles and the loop picks the next sub-page candidate. An empty plan is not pushed into the saved plans either.
Expanding an existing plan is unchanged, and a genuine planning failure still retries as before.
Zero scenarios with no plan to expand threw the same generic planning error as a failed call, so exploration retried the page and then spent another two calls per remaining planning style before giving up on it. The planner now throws EmptyPlanError, which says only what happened to the output and nothing about the page. Explore skips straight to the next sub-page candidate on it, with no retry and no further styles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrPgHXFYH6PzRksorVUGFC
…error-pages # Conflicts: # CHANGELOG.md
An empty scenario list is a value, not an exception. The planner returns the plan it built, empty or not, and explore reads its length: no tests means nothing to test here, so it stops trying further planning styles and the next sub-page candidate is picked. An empty plan is not carried into the saved plans either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrPgHXFYH6PzRksorVUGFC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Exploration burned a full planning round on pages with nothing to test — including the case in the reported trace, a 200 response whose only sign of being dead is its own copy.
Changes
explore-command.ts— sub-page skip.discoverNewSubPageschecks the page right after visiting it. IfgetStateErrorPageErrorreports an error page: warn, mark it failed, move to the next candidate. No research, no planning, no AI call spent on it. This covers server-rendered errors — anything with a 4xx/5xx document status or a standard HTTP error title.planner.ts— one prompt line. That check only sees HTTP status and standard error titles, so a page returning 200 with app-specific "doesn't exist" copy slips past it. Reading that from arbitrary wording is AI judgment, so the planning prompt carries the rule (general, no wording from any specific page): a page that reports a missing resource, shows a failure state, or holds no content and no controls gets an empty scenarios list, and nothing is invented for it.Empty output is a skip, not a failure. Zero scenarios with no plan to expand used to throw the same generic planning error as a failed call, so exploration retried the page and then spent two more calls per remaining planning style on it — the prompt rule above bought nothing. An empty scenario list is a value now, not an exception: the planner returns the plan it built, and
runAllStylesreads its length. No tests means nothing to test here, so it stops trying further styles and the next sub-page candidate is picked; the empty plan is not carried into the saved plans. An expanding plan is untouched — a later style returning nothing keeps the tests already planned.Tests
tests/integration/planner.test.tsasserts the prompt carries the rule, that empty output yields an empty plan rather than an error, and that an existing plan survives an empty style.bun test tests/integration/andbun test tests/unit/pass, format and lint clean.The prompt half can only be confirmed in a live run — worth the
regressionlabel if you want it proven against the page in the trace.🤖 Generated with Claude Code
https://claude.ai/code/session_013TYaLWuuLjj5mb26NuYjoj