Skip planning on error and not-found pages - #159
Open
DavertMik wants to merge 3 commits into
Open
Conversation
Exploration spent a full round of planning on pages that had nothing to test. A sub-page that turned out to be an error page was researched and planned anyway, and a page that reports a missing record with a normal 200 response — no HTTP status to detect it by — came back with a set of invented scenarios written against a page with nothing to click, once per planning style. Sub-page exploration now checks the page it just visited and moves on to the next candidate when it is an error page, before any research runs. For pages that only their own copy identifies as dead, the planner is told to return no scenarios, and no scenarios for a page nothing has been planned for is reported as an error page — which lands in the same skip. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013TYaLWuuLjj5mb26NuYjoj
Zero scenarios meant the model returned nothing, which is not the same as the page being dead — the research can be thin, the call can fail, the model can decline. Reporting that as an error page put a diagnosis on the page that nothing had established, and the !feature gate was only there to hide how often the guess would be wrong. Empty output is a planning failure again, as before. The prompt rule stands on its own: a page with nothing to exercise gets no scenarios, so nothing is invented for it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013TYaLWuuLjj5mb26NuYjoj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Exploration burned a full planning round on pages with nothing to test — including the case in the reported trace, a 200 response whose only sign of being dead is its own copy.
Changes
explore-command.ts— sub-page skip.discoverNewSubPageschecks the page right after visiting it. IfgetStateErrorPageErrorreports an error page: warn, mark it failed, move to the next candidate. No research, no planning, no AI call spent on it. This covers server-rendered errors — anything with a 4xx/5xx document status or a standard HTTP error title.planner.ts— one prompt line. That check only sees HTTP status and standard error titles, so a page returning 200 with app-specific "doesn't exist" copy slips past it. Reading that from arbitrary wording is AI judgment, so the planning prompt carries the rule (general, no wording from any specific page): a page that reports a missing resource, shows a failure state, or holds no content and no controls gets an empty scenarios list, and nothing is invented for it.Empty planner output stays a planning failure, exactly as before — zero scenarios means the model returned nothing, which is not evidence about the page.
Tests
tests/integration/planner.test.tsasserts the prompt carries the rule.bun test tests/integration/andbun test tests/unit/pass, format and lint clean.The prompt half can only be confirmed in a live run — worth the
regressionlabel if you want it proven against the page in the trace.🤖 Generated with Claude Code
https://claude.ai/code/session_013TYaLWuuLjj5mb26NuYjoj