Make Fisherman's replicate mode trustworthy

[DavertMik]

Fixes the five root causes behind every failed Fisherman episode in the April/June traces: the agent poisoning its own request store, first-match spec lookups, a scope filter that never matched, an unconditionally successful finish, and silent exhaustion.

Spec: docs/superpowers/specs/2026-08-29-fisherman-reliability-design.md · Plan: docs/superpowers/plans/2026-08-29-fisherman-reliability.md

What changed

• Replicate mode no longer eats its own rejects. loadFromDisk() admits only xhr_* files — provenance was already encoded in the id prefix (browser captures xhr_, Fisherman's own calls unprefixed), so no new envelope key and poisoned directories migrate for free. Behavioral change: a output/requests/ dir containing only Fisherman-made files now reports replicate mode unavailable — correct, there was never real browser traffic to replicate.
• Ranked example lookup. findCapturedRequest prefers exact endpoint over deeper sub-path, then 2xx over rejection, then newest. Paths are normalized with the existing isDynamicSegment(), so {id} patterns and concrete ids both match.
• Scope that actually matches. The endpoint list is scoped by the most selective non-id path segment shared between the page URL and captured request paths (the project slug wins over generic literals). When scoping degrades to the global list, the prompt says so instead of silently listing every project's endpoints.
• finish gated by the request ledger. Success requires at least one real 2xx write in this run; claimed created ids are verified against the requests that actually returned them (RequestMap in src/utils/request-map.ts), unverifiable claims are dropped, and each verified item names its creating request (request: "POST /api/…", logged by Pilot as via POST /api/…). A run that ends without finish synthesizes an honest summary from the ledger instead of returning summary: ''.
• Bounded guessing. Four consecutive failures against one endpoint end the run; one general prompt rule forbids creating a substitute resource type when the requested one has no endpoint.
• New integration test (tests/integration/fisherman.test.ts, aimock) covers the scoped prompt and the ledger gate end-to-end; unit coverage for ranking, scoping, load filtering, and ledger results.

Live-session auth (2026-08-30)

Langfuse trace de95bd1cffce09169599d99d1bee56cd (beta.testomat.io) showed every Fisherman write rejected 403 {"error":"Unauthorized"} in ~8ms while the same browser session performed successful writes to the same project minutes earlier. The requests were doomed by Fisherman's own credential assembly: an unfiltered cookie-jar dump with a dead localhost session pair ahead of the valid one, plus an x-csrf-token scraped from a 7-week-old capture of another project (the store's "newest" iteration was really alphabetical filename order).

Principle now enforced (spec: docs/superpowers/specs/2026-08-30-fisherman-live-session-auth-design.md): captures are a source of API shape, never credentials. Credentials come from the live browser session or explicit api.headers config.

• extractAuthHeaders is session-gated: only captures made during this session, newest first by timestamp; cookie is never scraped from captures; capture files without timestamps read as stale.
• refreshAuth precedence is captured < live browser < config, and the browser-derived layers apply only in replicate mode — achieve mode authenticates solely through api.headers. Behavioral change: Authorization/x-api-key values from previous sessions' captures are no longer sent; long-lived tokens belong in api.headers.
• The cookie jar is filtered to the API origin via context.cookies(baseEndpoint); the live CSRF token is read from the page's meta[name="csrf-token"] (provider renamed to browserHeaderProvider). A failed CSRF read degrades to no token without discarding the cookies.
• isDynamicSegment gained a version-segment carve-out: v1/v2-style segments are never treated as ids, so POST /api/v1/suites stays literal in endpoint lists.
• A turn without a tool call is now treated as finish (tool choice no longer forced): gpt-oss-style models that close with prose or a json pseudo-tool no longer 400 the whole invocation — which previously discarded the completed write roundtrips and re-created the same data on every retry (4 duplicate suite+test pairs in trace 7d5ddba053a054f6a3725503c9569340).

Review follow-ups deliberately left out of this PR: dedupe of the run-ledger watermark (fisherman.ts / fisherman-tools.ts), RequestMap's last-write-wins attribution when two responses return the same id (report text only), redacting auth headers in the persisted output/requests/*.yaml files (pre-existing behavior), and oauth2/hex-word segments still classifying as dynamic.

Regression fixture: deliberately NOT enabled

The plan's rollout step called for flipping fisherman: { enabled: true } in tests/regression/fixture/explorbot.config.js, gated on verifying the fixture can feed replicate mode. It cannot: the Trackly fixture has zero fetch/XHR — all UI writes are native form posts (form.requestSubmit() in client.js), which XhrCapture ignores by resource type and content type. Replicate mode would see zero captures, so the flip would prove nothing. Options:

• (a) Add an api block to the regression config — the fixture already serves /api/openapi.json, so achieve mode works today, but that exercises a different code path than these fixes target.
• (b) Change the fixture client to submit forms via fetch with JSON — exercises replicate mode, but alters fixture behavior for every regression scenario.
• (c) Leave Fisherman disabled in regression for now.

Until one of those lands, the plan's acceptance criterion (a trace where precondition() ids appear on the page the Tester acts on) is blocked on that decision. Regression runs remain user-triggered only — apply the regression label when you want one.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MHhNTXzihYU8f8mqNdjrVP
https://claude.ai/code/session_019JP2duAZr2ppH436viAJYs