Repository navigation
[Tests] Add deterministic local browser search scenario - #63
Conversation
b272336 to
b09c784
Compare
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Birfy
left a comment
Birfy
left a comment
There was a problem hiding this comment.
Self-review (Birfy) — merge readiness
Status: ready-for-review · MERGEABLE · CI green on default jobs; the browser job is skipped in Actions (opt-in local fixture).
What looks solid
- Deterministic local search fixture with scripted decisions and an independent verifier on the result page — good #13 slice with no live model/API.
- Unit tests cover missing result text and Runner startup/shutdown cleanup.
Non-blocking
- Repository Actions has historically skipped the
browserjob; local opt-in remains the proof path — fine if documented in the PR (already noted). - Does not complete #13; keep stacked with #64.
Not approving (own PR). Merge-ready pending human approval.
hsliuustc0106
left a comment
hsliuustc0106
left a comment
There was a problem hiding this comment.
Review (hsliuustc0106) — approve
Light-impact pass on head 4b072fa7851c766254686e1261ce134a49c7f81a.
Impact
- changed:
tests/search_task_fixture.py,tests/system/test_search_task_fixture.py,tests/test_browser_search_fixture_units.py - callers/dependents: none outside the new opt-in harness (uses existing
browse+flights.SPEC) - related tests: unit verifier + Runner start/stop cleanup; system test gated on
S1A_BROWSER_TESTS - not expanded: production browser agents beyond the fixture reuse of
browse
Risk: low (test-only; scripted decisions; no live model/API). Missing tests: none obvious.
Readiness
- CI green on exact head (core + full); browser job skipped in Actions (expected — opt-in local fixture)
- MERGEABLE; independent verifier checks submitted query and page text, not just the scripted final answer
Approving this #13 fixture slice. Does not complete the broader issue.
Related to #13; this is one repeatable fixture slice and does not complete the issue.
Adds an opt-in local browser scenario that submits a fixed search form through the existing Playwright MCP harness. The browser decision model and final answer are scripted, so no live model or paid API is called. The verifier checks the recorded query and the actual result text in the terminal page snapshot. Fast unit checks reject missing result text and ensure the temporary server/thread is cleaned up when Runner startup or shutdown fails.
Local verification on the Mac, for head ca88c17:
CI diagnosis (read-only, 2026-10-09):
The evidence indicates that Actions are currently disabled at the repository level, rather than this head awaiting an approval. No repository settings or credentials were changed. CI remains unverified until Actions are enabled through the repository's normal process and checks run on this head.
This is test coverage only, not a live-model baseline or a completion claim for #13 or #54.
Cloud validation (2026-10-10)
Independent run on exact head
ca88c17d3e5bcc6651d4128adbb3d6eecfe8edcdon the assistant's cloud computer (Linux, CPython 3.12.14), using the frozen core + dev install:pytest -q -rs): 788 passed, 47 skipped, 4 warnings, 154 subtests passed.ruff format --check .,ruff check .,ty check,uv lock --check, andgit diff --check: passed.scripts/smoke.sh: passed (smoke: ok), with no Jev key and no live model call.uv build: passed for source distribution and wheel.The 47 skips include optional game/report extras and intentionally unsupported backend contract cases; this is the full default suite on the core install, not all optional configurations. Writable cloud caches and
socksio==1.0.0were used only in the isolated test environment so HTTPX could honor the existing cloud SOCKS proxy. No source change was needed.The opt-in browser fixture was also attempted on this exact head using Node 24.19.0, Playwright MCP 0.0.78 and installed Chromium 154.0.8037.57. MCP startup succeeded after passing the existing cloud CA/cache configuration through the harness's supported options. Chromium then aborted before the local fixture received any page request:
socket() failed: Operation not permittedatchrome/browser/process_singleton_posix.cc:297. A supported elevated-execution review allowed a retry, but the same runtime restriction remained. The cloud browser check is blocked, not passed; the earlier Mac browser pass above is separate evidence. No TLS verification, proxy settings or browser sandbox flags were weakened.These results do not establish GitHub Actions success or a live-model baseline. No Actions settings, credentials, PR readiness/state, merges or releases were changed.