Conversation
…s on
The system prompt told the model to call getEditorState first and to reach
for takeScreenshot / execJsInLivePreview "listed below", but none of the
phoenix-editor tools were in its tool list: they sat behind ToolSearch. It
was being told to use tools it could not see, so it fell back to what it
could — an Explore subagent grepping the project — and only discovered the
editor tools later, usually to verify work already finished.
- alwaysLoad on the four "look at what the user is looking at" tools:
getEditorState, takeScreenshot, execJsInLivePreview, controlEditor.
~1.2k tokens on the cached system block; one ToolSearch round-trip costs
a whole model turn, so it pays for itself immediately.
- searchHint on all ten, worded in the terms someone would search with
(browser, rendered page, DOM, screenshot, responsive) rather than the
"live preview" jargon, so the six deferred ones stay findable.
- Prepend one line naming the active file, the unsaved buffers and what the
live preview renders. The panel assembles it, since the data and the
context chips both live there; this side only renders the prose. Says
plainly when a list is complete, because told only that a list *can* be
truncated the model re-checks with getEditorState — the exact lookup this
is meant to save.
Measured on the same prompt against a page with a runtime-only bug:
baseline +alwaysLoad +both
ToolSearch calls 2 0 0
first move Explore↴ Explore↴ the right file
model turns 10 9 6-9
wall time 85.5s 76.1s 17-27s
cost $0.2335 $0.2960 $0.08-0.11
The line itself costs ~106 tokens typically, ~195 worst case.
Two measured changes and one wording correction, all from the A/B runs recorded in phoenix-pro/unshipped/ai-panel-efficiency-notes.md. - searchEditorBuffers: regex search over the UNSAVED open files only. Those are the only files where the model's Grep is wrong — Grep reads disk, and disk is stale for a buffer the user has edited but not saved. Read/Edit are already buffer-safe (the agent flushes first); search was the one gap. The response names the files it covered and tells the model to Grep the rest, and when nothing is unsaved it says Grep is authoritative and does no work. Deliberately not a project scan: that would run on the UI thread and re-do what ripgrep already does off-thread. - resizeLivePreview is alwaysLoad. Responsive checks are a core use case and it cost two ToolSearch round-trips every time: same fix, 3 fewer turns, ~10s faster per responsive task. - Context line names which tools the "unsaved" note applies to. "Unsaved, so stale on disk" made the model refuse Edit and drive DocumentManager by hand through execJsInEditor — no edit card, no undo, 4x the cost. Now: "Read and Edit see the unsaved text as normal; Grep does not, so use searchEditorBuffers to search these."
Four phoenix-builder tools so a running Claude session can be asked "run
the AI test suite" and do the rest. The session is the runner and the judge;
the tools do the deterministic parts.
- run_ai_test_suite: installs the fixture, records git revs / CLI version /
connected instance, opens a run record, and returns the runner briefing
plus the documents for exactly the tests in scope — quick (default), all,
a suite name, explicit test ids, or resumeRunId for a stopped run.
- ai_test_progress: called after every test; answers "how far along is it",
lists runs, and stops a run on request. Progress lives in reports/runs/.
- save_ai_test_report: writes reports/latest.md, overwriting the previous
run — git history is the archive, baseline.md is never touched. A stopped
run is saved with a Partial section; a completed run's progress record
is removed.
- compare_ai_test_reports: diffs latest.md against baseline.md by default,
or against the last committed latest.md ("previous"), flagging
REGRESSION / quality drop / slower per the suite's thresholds. Reads the
merged "PASS · poor" Result column and the older two-column form.
The suite itself lives in phoenix-pro under unit-tests/ai_model_tests/.
CLAUDE.md tells any session what to do when asked; the MCP README lists
the tools. .eslintignore excludes the suite's fixture pages — they are
deliberately broken and are test data, not code.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The AI panel rarely used the Phoenix editor tools. It would answer a question about the page on screen by grepping the whole project with an Explore subagent, and only reach for the live preview later, if at all.
The cause was not reluctance, it was visibility. The system prompt already told the model to call
getEditorStatefirst and to reach fortakeScreenshot/execJsInLivePreview"listed below" — but none of thephoenix-editortools were in its tool list. They sat behindToolSearch. The model was being told to use tools it could not see, so it fell back to what it could see, and typically discovered the editor tools only near the end of a turn, to verify work it had already finished.alwaysLoadon the four "look at what the user is looking at" tools:getEditorState,takeScreenshot,execJsInLivePreview,controlEditor. A singleToolSearchround-trip costs a whole model turn, so having them present pays for itself.searchHinton all ten, worded in the terms someone would actually search with (browser, rendered page, DOM, screenshot, responsive) instead of the "live preview" jargon, so the six that stay deferred remain findable.The panel-side half is in phoenix-pro (already merged) and honours the context chips: dismiss the selection or cursor chip and the file is no longer named, dismiss the live preview chip and neither is what it renders, dismiss both and nothing is attached. Paths are project-relative inside the project and absolute outside it, the unsaved list is bounded by length, and the line states plainly when a list is complete — told only that a list can be truncated, the model calls
getEditorStateto check, which is the exact lookup this is meant to save.Without this, a question like "what does this page do?" could be answered with a request for the user to paste a URL or file path, because nothing told the model a file was open in front of them.
Test plan
Verified in the desktop app against Claude CLI 2.1.259:
"resize live preview", foundresizeLivePreviewthrough its hint, resized and screenshotted