feat(web-search): pool Tavily API keys with failover and cooldowns

.memory/mobile-stream-recovery.md

@@ -174,3 +174,9 @@ Cache model after the merge: `saveChats` keeps #242's committed-write ordering (
 Dropped as duplicates of #242: 37b5920a's load() stream-before-draft reorder and the create() `!saveChats` fallback (#242's older-list rule admits the create's own committed row); d41ad06c's cancelledControls enum case, URLProtocol branch and restorePendingApproval delivery. The carried cancelled-Stop test now follows #242's silent-cancellation rule.
 
 Verified: iOS build-for-testing after every pick; Android testDebugUnitTest 206/206 and lint; test:ci-policy 45/45. Simulator XCTest pending (Mac authorization prompt).
+
+# PR #278 iOS removal-test synchronization follow-up (2026-09-27)
+
+The CI failure in `testRemovalCancelsAdmittedConsumerBeforeHeldEventsPublish` was not accompanied by an XCTest result bundle. Inspection found a test-harness race: its `/events` suffix gate could hold either the intended `/streams/stream-recovery/events` consumer or the unrelated `/progress/events` observer, depending on request timing. The test now holds the unique stream-consumer path and deliberately runs the denied progress observer while that hold is armed, waiting for it to finish before sending. This verifies that unrelated progress traffic cannot steal the held consumer request; no production behavior changed.
+
+On iPhone 17 Pro Max / iOS 27 simulator, the focused test passed 10 consecutive repetitions. The six neighboring accepted-turn/removal lifecycle tests also passed. Evidence: `/Users/sambitbiswas/Library/Developer/Xcode/DerivedData/AidenOnTheGo-dsoibpylxjwthkgezllxaeiietwn/Logs/Test/Test-AidenOnTheGo-2026.09.27_16-47-08--0400.xcresult` and `/Users/sambitbiswas/Library/Developer/Xcode/DerivedData/AidenOnTheGo-dsoibpylxjwthkgezllxaeiietwn/Logs/Test/Test-AidenOnTheGo-2026.09.27_16-48-31--0400.xcresult`.

.memory/web-search-key-pool.md

@@ -0,0 +1,32 @@
+# Web Search API key pool — 2026-09-27
+
+- Plan: `docs/plans/web-search-key-pool-plan.md`. Source idea: pi-web-access #453. No code was copied.
+- Pool-capable providers are listed in `WEB_SEARCH_KEY_POOL_PROVIDER_IDS` in `renderer/shared/web-search-key-pool.ts`. Only `tavily` is listed today.
+  - Add a provider only after its adapter maps 401/403 to `auth` and its quota statuses to `quota`.
+- Storage lives in `main/services/web-search-credential-core.ts`:
+  - The `primary` entry is the legacy single-key slot.
+  - Other keys are stored in `…:pool:<id>`.
+  - An encrypted index at `…:pool-index` holds the order, labels and strategy.
+  - With no index, a saved key is treated as `[primary]`.
+  - `read()` returns the first pool key, so `has()` and readiness already understand pools.
+  - `remove()` wipes every slot.
+  - Pool mutations are serialized through a promise chain.
+- Provider-wide removal deletes the credential's colon-delimited secret family in one encrypted-map write; cleanup does not rely on a readable index, so orphaned slots from a corrupt index or interrupted add/remove are erased.
+- Runtime lives in `main/services/web-search-key-pool-core.ts`:
+  - `runWithWebSearchKeyPool` plus `WebSearchKeyPoolTracker`. The tracker's cooldowns and round-robin cursor are in memory only.
+  - The service reads `getCredentialPool` (from `web-search-main.ts`) and uses the shared `webSearchKeyPoolTracker` singleton in `web-search-credentials.ts`.
+  - `beforeProviderAttempt` runs once per keyed request.
+  - When every key is cooling, no request is sent: the pool throws `quota` if any key was quota-limited, otherwise `auth`.
+- IPC lives in `main/handlers/web-search-key-pool.ts`:
+  - It has injected dependencies and is registered from `phase2.ts`.
+  - Tests register it and invoke it directly (register-and-invoke).
+  - `webSearch:removeCredential` also clears the tracker.
+- UI: `renderer/components/settings/web-search-key-pool.tsx` replaces the single API-key field in `ProviderSetupDialog` for pool providers.
+  - It adds a `warning` Badge color that uses the status-warning tokens.
+- `renderer/components/settings/web-search-settings.test.tsx` greps the source. Do not grow it. New UI coverage goes in `web-search-key-pool.test.tsx`, which uses renderToStaticMarkup.
+- Formatting: the repo uses `oxfmt`, not prettier. Running `oxfmt` on `ui.tsx`, `ipc.ts` or `package.json` reformats unrelated code, so format only new files.
+- Validation:
+  - `npm run test:web-search`: 175 tests pass.
+  - `npm run test:settings-design`: 60 tests pass.
+  - `npm run type-check` and scoped ESLint pass.
+  - No live provider requests were made.

.papercuts/troubleshooting.md

@@ -1407,6 +1407,11 @@ because their native file-mutator test binary had not been built. Run
 ## 2026-09-26 PR #121 merge of #251 (Remote contract revision 14)
 - A PR that adds to the Remote contract has to renumber when main bumps `contractRevision`. The conflicts show up in 7 files: both fixtures, the TS/iOS/Android fixture assertions and the iOS fixture CodingKeys. After resolving, `cmp` the Android copy against the shared fixture. Plan docs that name the revision also go stale.
 
+## 2026-09-27 Web Search key pool (feature/web-search-key-pool)
+- The worktree guard is inconsistent about heredocs. `cd … && python3 - <<EOF` sometimes runs and sometimes gets refused as "too complex", and `cat >> file <<EOF` appends are refused. What works reliably: write the script with the Write tool into the scratchpad, then run `python3 /abs/script.py` as a plain command.
+- `WebSearchService`'s `adapterFactories` override replaces the whole registry and does not merge into it. A service test that mixes a real adapter (such as Tavily with a fake `fetch`) and stubs has to list the real factory explicitly. Otherwise the route is quietly not ready, and automatic routing falls back.
+- `oxfmt` on `renderer/components/ui.tsx`, `renderer/lib/ipc.ts` and `package.json` reformats code that has nothing to do with the change (main is not oxfmt-clean). Format only new files, and reapply small edits to existing ones by hand.
+
 ## 2026-09-27 timed ask-user (feature/timed-ask-user)
 - The worktree-isolation guard refuses compound shell commands (python heredoc plus a runner, `cat >> <<EOF` then npx, and `$HOME` inside gradlew env). Write scripts to the scratchpad and run them as a separate plain command, and spell out absolute SDK paths.
 - The Android gradle run printed only "Unable to locate a Java Runtime" to the log, and its background task still reported exit 0. Check the log, not the task status.

docs/plans/README.md

@@ -14,6 +14,7 @@ This directory is the source of truth for Aiden's implementation plans. The engi
 | Plan                                                                                               | Status  | Current state                                                                                                                                                                                                             |
 | -------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | [Aiden CLI](aiden-cli-plan.md) | Active | Phases 0–5 implementation complete; macOS/Linux CLI (57 tests each), Linux native helpers, complete shared subagent suites, root TypeScript/lint, and Android client checks pass. Physical iPhone acceptance is pending an unlocked device. Phase 6 adds QR remote pairing, scheduled-run notifications (mobile push + desktop), shared desktop memory, daemon autostart, and prebuilt binaries; see the [checklist](aiden-cli-parity-checklist.md). |
+| [Web Search API key pool](web-search-key-pool-plan.md) | Implemented for review | Tavily accepts up to 8 encrypted keys with ordered or round-robin use. Rejected keys (401/403) and quota-limited keys (429/432/433) cool down in memory with backoff and fail over to the next key. When every key is cooling, no request is sent and automatic routing falls back. Settings can add, remove, reorder and retry keys, and the renderer never sees a key. Other providers and CLI parity remain. |
 | [Timed ask-user waits](timed-ask-user-plan.md) | Implemented for review | Optional `timeoutSeconds` on `ask_user_question`; unattended (Remote) runs always expire within 5 min and resolve with an explicit best-judgement result. Late desktop answers become a Send/Queue follow-up offer; Remote `expiresAt` carries the real deadline; iOS/Android say when a question expired. PR CI pending. |
 | [Rich link previews for Chats and Bots](rich-link-previews-plan.md) | Implemented for review | Shared regular Chat and Bot transcripts now show provider-aware inline icons and bounded hover/focus cards. User HTTP(S) text is autolinked, assistant streaming and persisted Markdown share the opt-in renderer, and URL-derived previews perform no network requests. |
 | [Simulator Devices](simulator-devices-plan.md) | Partial | Environment **Simulator** tab plus `device_*` agent tools, ported from T3 Code (MIT, `1c127066`). iOS only; consent-gated pinned `expo-device-hub@0.12.0` + `agent-device@0.21.12`; token-authenticated loopback proxy. Phases 0–3 done (spike, flagged tab shell, main-process toolchain/host/proxy/service/IPC, live stream viewer with controls and screenshot-to-chat; `test:devices` 96 pass; fake-hub Electron E2E; real-Mac acceptance passed). Phase 4 (agent `device_*` tools), Phase 5 (simulators on paired Macs over Aiden Remote; [plan](simulator-devices-phase-5-peers.md)) and Phase 6 (procedural 3D device frames; [plan](simulator-devices-phase-6-3d.md)) done. Phase 7 done: Settings → **Simulator** (consent switches, pinned/installed helper versions, prune, remove installed tools) and [`docs/devices.md`](../devices.md); onboarding skipped while the flag is off. Agent guidance now prefers `device_*`/`agent-device` for the watched device but allows shell `xcrun simctl`/`xcodebuild`/`adb` for builds, installs, logs, port forwarding, and diagnostics (T3 #13908). Remaining: real-Mac acceptance for Phases 4–6 before the flag defaults on; SSH hosts are a later follow-up. |

docs/plans/web-access-rehaul-plan.md

@@ -386,6 +386,11 @@ whole redacted durable snapshot. Provider credential removal affects only that
 provider. If it makes the current route unusable, Settings shows the invalid
 route and requires a user choice rather than silently changing recipients.
 
+Pool-capable providers (Tavily first) add the `webSearch:keyPool:*` channels
+(`get`, `add`, `remove`, `reorder`, `setStrategy`, `resetCooldown`). They return
+only the redacted pool projection. See the
+[Web Search API key pool plan](web-search-key-pool-plan.md).
+
 ## UI plan
 
 Before implementation, re-review

docs/plans/web-search-key-pool-plan.md

@@ -0,0 +1,76 @@
+# Web Search API key pool
+
+Status: Implemented for review (Tavily).
+
+## Goal
+
+Let a Web Search provider hold several API keys. When one key is rejected or
+runs out of quota, the search moves to the next key without the user noticing.
+The idea comes from pi-web-access #453. Tavily is the first provider because
+its adapter already maps 401/403 to `auth` and 429/432/433 to `quota`.
+
+## Design
+
+- **Storage.** Each key is its own encrypted secret with the provider's
+  endpoint binding.
+  - The first key stays in the existing single-key slot
+    (`web-search:<provider>:api-key`), which the pool calls the `primary`
+    entry.
+  - Other keys use `…:pool:<entryId>`.
+  - An encrypted index at `…:pool-index` stores the order, labels, `addedAt`
+    and the strategy. It never holds key material.
+  - A key saved before this change, when no index exists yet, is read as a
+    one-entry pool. No migration step is needed.
+  - Provider-wide removal deletes the credential's colon-delimited secret
+    family in one encrypted-map write, including secondary slots omitted from
+    a corrupt or stale index.
+- **Selection.** Two strategies:
+  - `ordered` always starts with the first key.
+  - `round-robin` starts each search one key further along.
+  - With either strategy, keys that are cooling down are skipped.
+- **Failover.** What happens depends on the error:
+
+  | Error | Result |
+  | --- | --- |
+  | `auth` (401/403) | The key cools down for 15 minutes, doubling on each repeat up to 24 hours. The next key is tried. |
+  | `quota` (429/432/433) | The key cools down for 1 minute, doubling on each repeat up to 1 hour. The next key is tried. |
+  | Any other error, including cancellation and timeout | The search stops immediately. |
+
+  - Cooldowns live only in memory, so restarting Aiden clears them.
+  - If every key is cooling down, no request is sent. The search fails with
+    `quota` if any key hit a quota limit, and with `auth` otherwise. Automatic
+    routing then falls back to the next provider as usual.
+- **Budget.** The subagent `beforeProviderAttempt` fence runs before every
+  keyed request, so each key tried counts against the child's network budget.
+- **IPC.** The `webSearch:keyPool:get|add|remove|reorder|setStrategy|resetCooldown`
+  channels:
+  - check the owner document;
+  - enforce the rollout mutation fence;
+  - return only IDs, labels, order, strategy and cooldown state.
+
+  Keys can be written but never read back from the renderer.
+- **Settings.** In Settings → Web Search → provider setup, Tavily replaces its
+  single API-key field with a pool editor. The editor offers:
+  - an ordered key list with a position, label and status badge;
+  - move up and down buttons, plus Alt+Arrow keyboard reordering;
+  - Remove, and Retry now for a key that is cooling down;
+  - an optional label and a password field for adding a key;
+  - a strategy radio group with no card borders.
+- **Logging.** Keys never appear in errors, logs or the renderer state.
+
+## Tests
+
+| Test file | Covers |
+| --- | --- |
+| `main/services/web-search-key-pool-core.test.ts` | Rotation and failover through the real Tavily adapter with a fake `fetch`, cooldown timing, and the index parser. |
+| `main/services/web-search-credential-core.test.ts` | Pool storage, legacy primary synthesis, duplicate keys and the maximum pool size, serialization, corrupt-index cleanup, and removal of orphaned secondary slots. |
+| `main/services/web-search.test.ts` | Service-level failover, charging each key attempt, and falling back to the next route. |
+| `main/handlers/web-search-key-pool.test.ts` | Register-and-invoke IPC. |
+| `renderer/components/settings/web-search-key-pool.test.tsx` | The rendered list's status, order and disabled states. |
+
+## Follow-ups
+
+- Extend the pool to other keyed providers once their adapters map
+  quota/auth statuses reliably.
+- Optionally persist cooldowns across restarts.
+- CLI parity for `packages/cli`, which still uses one key per provider.

ios/AidenOnTheGoTests/AidenChatTests.swift

@@ -1172,12 +1172,19 @@ final class AidenChatTests: XCTestCase {
             }
             return fixture.response(request)
         }
-        await model.load()
+        await model.load(observeProgress: false)
         model.draft = "Hello"
         XCTAssertTrue(model.canSend)
         let arrived = expectation(description: "consumer events held")
-        AidenChatProgressLifecycleURLProtocol.holdNextRequest(endingIn: "/events") { arrived.fulfill() }
+        AidenChatProgressLifecycleURLProtocol.holdNextRequest(endingIn: "/streams/stream-recovery/events") { arrived.fulfill() }
         defer { AidenChatProgressLifecycleURLProtocol.releaseHeldRequest() }
+        // The chat's unrelated progress stream also ends in `/events`. Exercise
+        // it while the stream-consumer hold is armed to prove it cannot steal
+        // the intended gate.
+        model.startProgressObservation()
+        try await waitForProgressRequestCount(1)
+        try await waitForProgressObservationToStop(model)
+        XCTAssertEqual(AidenChatProgressLifecycleURLProtocol.progressRequestCount, 1)
         await model.send()
         await fulfillment(of: [arrived], timeout: 2)
         let admitted = await cache.loadChat(instanceId: "instance-progress-lifecycle", chatId: model.chat.id)

main/handlers/phase2.ts

@@ -39,7 +39,11 @@ import { rendererDocumentOwner } from "../services/renderer-document-owner.js";
 import type { RendererDocumentOwner } from "../services/renderer-document-owner.js";
 import { mutatePortableConfigAndSync } from "../services/portable-credential-snapshot.js";
 import { withMcpConfigurationPublication } from "../services/mcp-config-lease.js";
-import { webSearchCredentials } from "../services/web-search-credentials.js";
+import {
+  webSearchCredentials,
+  webSearchKeyPoolTracker,
+} from "../services/web-search-credentials.js";
+import { registerWebSearchKeyPoolHandlers } from "./web-search-key-pool.js";
 import { webSearchExistingAuthReuse } from "../services/web-search-auth-reuse-main.js";
 import {
   DEFAULT_WEB_SEARCH_FALLBACK_ON,
@@ -549,9 +553,20 @@ export function registerPhase2Handlers(): void {
       settings.providerConfig[providerId],
     );
     await webSearchCredentials.remove(reference, () => !owner.isDestroyed());
+    webSearchKeyPoolTracker.clear(providerId);
     if (owner.isDestroyed()) throw new Error("The renderer document is no longer active.");
     return readWebSearchSnapshot();
   });
+  registerWebSearchKeyPoolHandlers<Electron.IpcMainInvokeEvent>({
+    handle: (channel, handler) => ipcMain.handle(channel, handler),
+    credentials: webSearchCredentials,
+    tracker: webSearchKeyPoolTracker,
+    providerConfig: async (providerId) =>
+      (await configStore.getWebSearchSettings()).providerConfig[providerId],
+    owner: webSearchMutationOwner,
+    assertMutationAllowed: (providerId) =>
+      assertWebSearchRolloutMutationAllowed("set-credential", providerId, webSearchRollout),
+  });
 
   // Legacy Exa aliases remain for one rollback window. They use the same
   // fenced v2 credential path and never expose the plaintext key.