[sunwrobert]

throw-to-result: src/cli/index.ts (vendored in openrouter-web as packages/bench-harness/src/cli/index.ts)
remaining: 13 → 0 in this module (git grep -E "^\s*throw new Error\(" -- src/cli/index.ts | grep -vE "// invariant:" | wc -l); openrouter-web inventory git grep -E "^\s*throw new Error\(" -- 'packages/**/*.ts' 'services/**/*.ts' ':!*.test.ts' ':!*.integration.test.ts' | grep -vE "// invariant:" | wc -l is 1298 on main and becomes 1285 once this lands upstream and scripts/subtree-pull-bench-harness.sh vendors it (the subtree is read-only in the monorepo, so the fix has to land here first).

TL;DR

Every expected user-input / environment failure in the Kepler CLI (src/cli/index.ts) is now a CliError literal-union value carried in an effect/Result instead of a throw new Error(msg); main() renders it once with formatCliError, writes error: <msg> to stderr and exits 1, so the user-visible message line and exit code are unchanged.

Convention choice: effect/Result (Result.fail / Result.succeed / Result.gen / Result.isFailure), the package's existing convention (src/internal/zod.ts, src/internal/result.ts, src/internal/testing.ts); @openrouter-monorepo/lib-result is not a dependency of this standalone repo.

What changed?

• type CliError — one kind per distinct failure, carrying the interpolated data, never the message string:

export type CliError =
  | { readonly kind: "invalid_image_detail"; readonly raw: string }
  | { readonly kind: "invalid_cost_tier"; readonly raw: string }
  | { readonly kind: "invalid_reasoning_effort"; readonly raw: string }
  | { readonly kind: "adaptive_reasoning_effort_unsupported"; readonly model: string | undefined }
  | { readonly kind: "unknown_benchmark"; readonly benchmarkId: string }
  | { readonly kind: "api_key_missing" }
  | { readonly kind: "unsafe_session_id"; readonly sessionId: string }
  | { readonly kind: "invalid_solver_config_json"; readonly cause: unknown }
  | { readonly kind: "model_required"; readonly benchmarkId: string }
  | { readonly kind: "unknown_solver_config_options"; readonly benchmarkId: string; readonly options: readonly string[] }
  | { readonly kind: "invalid_benchmark_config"; readonly benchmarkId: string; readonly cause: ZodError }
  | { readonly kind: "invalid_draco_panel_config"; readonly cause: ZodError }
  | { readonly kind: "unsupported_benchmark"; readonly benchmarkId: string };

• formatCliError(error): string — exhaustive switch ending in error satisfies never; produces the exact message text each throw produced before.
• validateImageDetail, validateCostTier, validateReasoningEffort, requireModel, resolveApiKey, resolveSessionId, parseSolverConfig, buildSchemaValidatedConfig, buildBenchmarkConfig, parseArgs now return Result.Result<T, CliError>; parseArgs, buildSchemaValidatedConfig and the new prepareRun compose with Result.gen so the first failure short-circuits.
• The three Result→throw bridges (parseSolverConfig JSON, BenchmarkRunConfigSchema, DracoPanelConfigSchema) now propagate the ZodError / parse cause inside the CliError instead of re-throwing its message.
• main() is async; prepareRun(argv) collects args, benchmark, API key, session id and solver config; reportCliError is the single boundary (stderr + process.exitCode = 1), the same pattern the existing Benchmark failed: branch already uses.
• Callers (complete list): main() in src/cli/index.ts; src/cli/index.test.ts. No other importer of parseArgs / buildBenchmarkConfig exists (git grep -n "from \"../cli\"\|cli/index" → tests only).
• // invariant: markers added: none.
• src/cli/index.test.ts: every toThrow case is now an assertFailure + expect(failure).toEqual({ kind, ... }) / formatCliError assertion with the same inputs; new cases for invalid_image_detail, model_required, unsupported_benchmark, invalid_draco_panel_config (every kind reachable through the exported functions now has a test; unknown_benchmark, api_key_missing, unsafe_session_id, invalid_solver_config_json are exercised by the CLI smoke below since they read process.env / the registry in main()).

Why?

Code-scan finding sfind-0fd69853fe6b48d5829462cec63a8681 (throw-to-result): these 13 sites are expected failures, not invariants, and the module already had Result values in hand that it converted back into throws.

How to test

bun test src/cli/index.test.ts   # 29 pass
for c in "--cost-tier bogus" "--image-detail ultra" "--reasoning-effort nope" "--model openai/gpt-5 --reasoning-effort auto" "--benchmark nope --model m" "--model m --solver-config {bad" "--benchmark gpqa_diamond --limit 1" "--model m --solver-config {\"bogus\":1} --limit 1"; do OPENROUTER_API_KEY=x bun src/cli/index.ts $c; echo "exit=$?"; done
env -u OPENROUTER_API_KEY -u BENCHMARKING_OPENROUTER_API_KEY bun src/cli/index.ts --model m; echo "exit=$?"
OPENROUTER_API_KEY=x BENCH_CHILD_WORKFLOW_ID=$'a\tb' bun src/cli/index.ts --model m; echo "exit=$?"

Expected: each prints exactly one error: <same message as on main> line to stderr and exits 1.

Benchmark impact

None: no solver, scorer, prompt, dataset or config-schema change; successful invocations build the identical BenchmarkRunConfig.

Reviewer focus

• purely mechanical: no
• Behavior change (the only one): a failing invocation no longer prints Bun's uncaught-exception source excerpt and stack trace under the error: line; the error: <message> line and exit status 1 are byte-identical for all 10 smoke cases (base-vs-head diff of exit= + error: lines is empty).
• Ordering of checks is preserved (image-detail → cost-tier → reasoning-effort; unknown benchmark → API key → session id → solver-config JSON; --model required before unknown-option / schema checks).
• Open PR fix(cli): forward solver config to all model benchmarks and improve error resilience #77 also edits src/cli/index.ts (solver-config forwarding, num() TypeError); it does not carry this marker and is a different change, but will conflict textually with whichever lands second.
• Prod impact: CLI-only (Temporal/mission-control call the harness through runBenchmarkById, not main()); worst case is a wrong message text for one failure kind, caught by the per-kind tests.

Show me

• Base vs head CLI smoke (10 failure cases, message line + exit code): identical.
• Head index.test.ts on base source: 0 pass / 1 fail (compile error — parseArgs on base is not a Result), so every converted assertion is proven to depend on the change; on head: 29 pass.
• bun run format:check, bun run check (no findings in src/cli/; pre-existing warnings elsewhere unchanged), bun run typecheck, bun test (1833 pass), bun run build: green.

Proof matrix

Behavior	Unit	Integration	E2E
Each CliError kind reachable via parseArgs / buildBenchmarkConfig (9 kinds)	index.test.ts per-kind assertFailure + toEqual({ kind, ... }); base fails (compile), head passes	n/a (pure functions)	CLI smoke, same message + exit 1 as main
unknown_benchmark, api_key_missing, unsafe_session_id, invalid_solver_config_json (env / registry, inside main())	n/a (read process.env; same shape as the 9 above)	n/a	CLI smoke, same message + exit 1 as main
Unchanged inputs (all documented options, hand-built configs, tau3 defaults, ori reasoning effort)	20 existing tests now unwrap assertSuccess, pass on head	n/a	n/a (success path reaches OpenRouter; not exercised without a key)

Thermo-nuclear fix loop

2 passes. Pass 1 findings fixed: explicit isFailure/fail(x.failure) chains replaced by Result.gen in parseArgs, prepareRun, buildSchemaValidatedConfig; NonNullable<ReturnType<typeof getBenchmark>> replaced by the Benchmark type; error.error field renamed cause; failCli → reportCliError; main() made async; test helpers typed with CliArgs / BenchmarkRunConfig instead of ReturnType<…>["success"]. Pass 2: clean. deslop-micro: no invented terminology left after pass 1 renames (prepareRun, RunSetup, reportCliError, CliError, formatCliError).

Visualization

Checklist

• Tests cover changed behavior
• Public API or configuration changes are backward compatible, or the break is documented (parseArgs / buildBenchmarkConfig return Result; only consumers are main() and the colocated test)
• Benchmark changes document dataset provenance and licensing (n/a)
• No credentials, private results, or restricted dataset contents are included
• Documentation is updated where needed (n/a)

Link to Devin session: https://openrouter.devinenterprise.com/sessions/4e5e881590874d60a3a5cf4a79a9be6e
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/4e5e881590874d60a3a5cf4a79a9be6e?variant=devin
Requested by: @sunwrobert

---

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

• Disable automatic comment, CI, and merge conflict monitoring

---

Original prompt from Devin Bot

Please remediate these findings together. --code-scan-step fix:sfind-0fd69853fe6b48d5829462cec63a8681
Shared root cause: packages/bench-harness/src/cli/index.ts in OpenRouterTeam/openrouter-web has 13 throw new Error sites that are all expected user-input/environment failures (invalid --image-detail/--cost-tier/--reasoning-effort, missing --model or API key, unsafe BENCH_CHILD_WORKFLOW_ID, bad solver-config, unsupported benchmark), plus three Result-to-throw bridges (lines ~407-408, 417-419, 479-480). Make the validators, parseArgs, parseSolverConfig, buildSchemaValidatedConfig, buildBenchmarkConfig, resolveApiKey and resolveSessionId return Results and propagate; the import.meta.main boundary in main() prints the message to stderr and exits 1 so CLI behavior is unchanged. Only callers are main() and the colocated index.test.ts (convert toThrow to failure assertions). Note the finding guidance says this package follows effect/Result conventions; the profile's conversion rule names ``@openrouter-monorepo/lib-result — pick the package's existing convention and state the choice in the PR body.

@playbook:playbook-063fbf6c0b3945bf948a331c5e4f011d

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[perry-v2]

✅ Approved. Expected CLI validation failures now flow through typed CliError Results to a single reporting boundary. No actionable regressions found.

Scope and evidence

Reviewed both changed files (src/cli/index.ts and src/cli/index.test.ts, 461 additions / 209 deletions) at 9d0b5919e1ab4a2e4a8981d658ecbf214b7f7adc against the actual-base diff for base c6a05aa324948d4fd66b89c2c7e8b9a3065ee01a. Two independent correctness passes covered both files, with additional checks of error paths, security, callers, dependencies and reuse. Examined Result short-circuiting, validation order, formatted failures, config construction and the async CLI reporting boundary.

Risk

The principal compatibility change is the Result-returning parser/config-builder API; the repository callers are updated with it. Expected input failures deliberately lose the uncaught-exception stack trace. No benchmark implementation or configuration schema changes are introduced. Static inspection found no defect in the successful-config or validation-failure paths.

Verification and CI

This was a static review: no PR code, tests, installs or builds were run, and the author's smoke-test claims were not independently reproduced. Inspected the converted validation assertions and added failure-kind cases. At the reviewed head, validate, CodeQL and both analysis checks were successful; the commit-status API also reported success.