feat(web-search): pool Tavily API keys with failover and cooldowns

[sambitcreate]

Summary

Tavily can now hold up to 8 encrypted API keys. When one key is rejected or runs out of quota, the search moves to the next key.

• Order: keys are used in order or round-robin. The strategy is set per provider.
• Cooldowns: a rejected key (401/403) cools down for 15 minutes, doubling on each repeat up to 24 hours. A quota-limited key (429, or Tavily's 432/433) cools down for 1 minute, doubling up to 1 hour. Then the next key is tried.
• Other errors: any other failure, including cancellation and timeout, stops the search immediately.
• All keys cooling: no request is sent. The pool fails with quota (or auth if every key was rejected), so automatic routing falls back as usual.
• Keys stay hidden: keys never reach the renderer, error messages or logs.

Source: pi-web-access #453 (idea only; no code copied).

Changes by surface

• Credential storage (main/services/web-search-credential-core.ts)
  • The existing single-key slot is now the pool's primary entry. Keys saved before this change keep working with no migration.
  • Additional keys live in …:pool:<id> slots with the same endpoint binding.
  • An encrypted …:pool-index stores the order, labels and strategy.
  • Pool mutations run one at a time. Duplicate keys and a ninth key are rejected, and a corrupt index falls back to the primary key.
• Runtime (main/services/web-search-key-pool-core.ts, web-search.ts)
  • runWithWebSearchKeyPool plus an in-memory WebSearchKeyPoolTracker for cooldowns and the round-robin cursor.
  • The subagent beforeProviderAttempt budget check runs once per keyed request.
• IPC (main/handlers/web-search-key-pool.ts)
  • New webSearch:keyPool:get|add|remove|reorder|setStrategy|resetCooldown channels.
  • Each channel checks the sending document and the rollout mutation fence. It returns only entry IDs, labels, order, strategy and cooldown state.
  • webSearch:removeCredential now also clears pool cooldowns.
• Settings (renderer/components/settings/web-search-key-pool.tsx)
  • Tavily's setup dialog replaces the single key field with a pool editor.
  • The editor shows an ordered list with a position, label and a status badge (Active, Rate limited, or Rejected, each with the time left).
  • Keys can be moved with the up/down buttons or Alt+Arrow, removed, or retried with Retry now.
  • Adding a key takes an optional label and a password field.
  • The strategy radio group has no card borders. It follows docs/settings-design-system.md and the AGENTS UI rules: semantic tokens, the shared Button, and focus-visible rings kept.
  • Badge gains a warning color built from the existing status-warning tokens.
• Docs
  • New plan: docs/plans/web-search-key-pool-plan.md, with an index row in docs/plans/README.md.
  • The IPC list in the web access rehaul plan now includes the new channels.
  • New .memory/web-search-key-pool.md and entries in .papercuts/troubleshooting.md.

This does not change onboarding, since the feature is an optional extra for an existing provider, or the Aiden Remote contract, so there are no mobile changes.

Tests run

• npm run test:web-search: 175 pass. It includes these new files, now registered in the script:
  • main/services/web-search-key-pool-core.test.ts
  • main/handlers/web-search-key-pool.test.ts
  • renderer/components/settings/web-search-key-pool.test.tsx
• npm run test:settings-design: 60 pass.
• npm run type-check: pass.
• Scoped eslint on the touched files: pass.

What the tests cover:

• Rotation and failover use the real Tavily adapter with a fake fetch and check which bearer key each request carried:
  • 401, 429, 432 and 433 each move to the next key;
  • a 500 stops without trying another key;
  • round-robin spreads searches across keys, and ordered always starts with the first key;
  • a pool where every key is cooling sends no request;
  • cancellation stops failover.
• Service-level check: a pooled Tavily route charges its budget once per key, then falls back to the next provider.
• Credential pool storage and register-and-invoke IPC.
• Rendered list status and disabled states.

No live provider requests were made.

Follow-ups

• Enable the pool for other keyed providers once their quota and auth status mapping is verified.
• Optionally persist cooldowns across restarts. Today they reset when Aiden restarts, which the UI says.
• CLI parity in packages/cli, which still supports one key per provider.

🤖 Generated with Claude Code

[pullfrog]

Important

Provider-wide key removal can leave orphaned encrypted API keys behind.

Reviewed changes This review covers the initial implementation of Tavily key pooling across encrypted storage, routing, IPC, Settings, and tests.

• Credential pool: Keeps the legacy key slot as primary and stores additional keys with encrypted metadata.
• Failover and cooldowns: Adds ordered or round-robin selection, auth/quota backoff, and per-key request budget checks.
• IPC and Settings: Adds redacted pool operations and a key editor for adding, labeling, reordering, removing, and retrying keys.
• Coverage and docs: Adds storage, runtime, service, IPC, and rendered UI tests, plus the implementation plan and project memory.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using GPT Luna ｜ 𝕏}

[very-hermes-bot]

Hermes Review Bot

Confidence: 5

Engine: agy/gemini-3.8-flash-high
Review mode: full
Head: aa9704f5293b33095d456c6491602a836590737f
Generated: 2026-09-29T16:44:54+00:00
Reviews: 1

Summary

Enables multiple API keys for Tavily Web Search (up to 8 keys) with automated in-memory rotation, failover, and exponential backoff cooldowns. Keys are partitioned into dedicated encrypted secret slots with an encrypted index, preserving the single primary slot for backward compatibility without migration. Runtime search execution charges subagent network attempt budgets per key attempt, skips cooling keys, and falls back to subsequent routes only when all keys are exhausted or limited. The renderer UI replaces the single-field setup with a dedicated pool editor adhering to semantic tokens and accessibility guidelines, never receiving key material over IPC. Maintainers should double-check that the in-memory cooldown tracker's process-local lifecycle matches expectations, as restarting the app resets all cooldown states to active.

Confidence Score: 5/5

Fully traced and verified across secret storage encryption, IPC message boundaries, rotation and cooldown algorithms, search failover orchestration, and UI accessibility.

📁 Important Files Changed

• main/services/web-search-key-pool-core.ts: Implements pure key pool rotation, exponential backoff tracking, Tavily quota/auth mapping, and failover runner logic.
• main/services/web-search-credential-core.ts: Manages encrypted credential storage for pool entries, serialized index read-modify-write mutations, duplicate prevention, and family-scoped secret deletion.
• main/services/web-search.ts: Integrates key pool failover into route execution, charging per-key provider attempts against caller budgets and handling route fallback.
• main/handlers/web-search-key-pool.ts: Exposes register-and-invoke IPC channels with sender document liveness checks and rollout mutation gating, returning strictly redacted pool metadata.
• renderer/components/settings/web-search-key-pool.tsx: Renders the settings pool editor with key reordering, status badges, cooldown countdowns, and accessible radio choices.
• main/services/secret-map-core.ts: Adds deleteSecretKeyFamily to atomically clean up secret entries and their associated bindings by prefix.
• ios/AidenOnTheGoTests/AidenChatTests.swift: Stabilizes an iOS test-harness race condition by scoping the URL protocol hold to stream recovery endpoints.

Findings

No findings.

📊 Sequence Diagram

sequenceDiagram
  autonumber
  participant Caller as Caller / Subagent
  participant Service as WebSearchService
  participant Tracker as WebSearchKeyPoolTracker
  participant Adapter as TavilyAdapter
  participant Tavily as Tavily API

  Caller->>Service: search(query)
  Service->>Tracker: order(keys, strategy)
  Tracker-->>Service: [key1, key2]
  Service->>Service: beforeKeyAttempt(key1)
  Service->>Adapter: search(key1)
  Adapter->>Tavily: POST /search (key1)
  Tavily-->>Adapter: 429 Rate Limit
  Adapter-->>Service: WebSearchError("quota")
  Service->>Tracker: recordFailure(key1, "quota")
  Service->>Service: beforeKeyAttempt(key2)
  Service->>Adapter: search(key2)
  Adapter->>Tavily: POST /search (key2)
  Tavily-->>Adapter: 200 OK
  Adapter-->>Service: WebSearchResultSet
  Service->>Tracker: recordSuccess(key2)
  Service-->>Caller: WebSearchResultSet

Loading

Machine-Readable Findings

[]

---

Last reviewed commit: aa9704f5293b
Reviews (1) · Comment /hermes review to trigger a new review · /hermes review full for full re-review

[pullfrog]

✅ No new issues found.

Reviewed changes This incremental review covers the commit landed since the prior pullfrog review, which resolves that review's orphaned-secret finding.

• Family-scoped secret deletion: Added deleteSecretKeyFamily to the encrypted secret map and secrets.deleteKeyFamily, deleting a root ID, every colon-delimited descendant, and matching binding entries in one map write.
• Index-independent provider removal: Provider-wide credential removal now sweeps the credential's secret family directly instead of walking the parsed pool index, erasing orphaned slots left by a corrupt index or an interrupted add/remove mutation.
• Regression coverage and docs: Added family-deletion and stale-guard tests to the secret-map and credential suites, updated the plan and project memory, and registered the new test in the CI registry (verified with test:ci:registry and the policy check).

Verified locally: test:web-search passes (177 tests including the touched suites), npm run type-check passes, and the CI registry/policy validators accept the re-sorted registry. A namespace-collision audit of deleteKeyFamily found no surface that stores foreign credentials under a web-search:<provider>:api-key descendant: the family root is closed registry (tavily only via webSearchKeyPoolSupported), the web-search: prefix has exactly one producer, and custom model providers are custom:-rooted and removed via single-entry deleteKey.

  ^{｜ View workflow run ｜ Using GLM Flash ｜ 𝕏}

[pullfrog]

✅ No new issues found.

Reviewed changes This incremental review covers the iOS lifecycle-test synchronization follow-up and its project note.

• Separated the competing event streams. Updated the removal lifecycle test to hold only /streams/stream-recovery/events, run the denied /progress/events observer independently, and wait for it to stop before sending the turn.

  ^{｜ View workflow run ｜ Using GPT Luna ｜ 𝕏}

[pullfrog]

✅ No new issues found.

Reviewed changes This incremental review covers the two origin/main sync merges added since the prior Pullfrog review.

• Reconciled CI inventory. Kept the key-pool test suites registered while incorporating the current main test list.
• Synchronized project notes. Preserved Web Search pool documentation and troubleshooting entries alongside current main notes.

  ^{｜ View workflow run ｜ Using GPT Luna ｜ 𝕏}