feat(llm): claude-cli provider: run the Q&A track on a logged-in Claude Code CLI

dokimi/benchmarks/spec.v1.yaml

@@ -72,15 +72,11 @@ future:
     enabled: false
     min_score: 0.85
 
-# Bounds on the document set the static track scores. They are a sanity check
-# on the include/exclude globs, not a target: min_docs catches a glob that
-# matched almost nothing, max_docs catches one that escaped into the tree (8
-# markdown files live outside docs/). Measured selection at the time of
-# writing: 35 files. Adding a doc past the ceiling is meant to be a deliberate
-# edit here, so raise max_docs in the same commit that adds the doc.
+# Sanity bounds on the static track's doc set, not targets: min_docs catches a glob
+# that matched almost nothing, max_docs one that escaped; raise it with each new doc.
 golden_manifest:
   min_docs: 10
-  max_docs: 35
+  max_docs: 36
   include:
     - docs/**/*.md
   exclude:

dokimi/changelog.d/20261026_150000_napetrov_claude-cli-backend.md

@@ -0,0 +1,35 @@
+### Added
+
+- **`--provider claude-cli` / `--judge-provider claude-cli`**: answers, judging,
+  question generation, personas and novelty can run through a logged-in Claude
+  Code CLI (`claude -p`) instead of an API key. Each call runs with no tools, no
+  user or project settings and no MCP servers, in one fixed empty directory, with
+  the prompt on stdin. A requested `temperature` or `seed` is refused, since
+  the CLI cannot honour them. `cost_usd` is `None`, because the CLI's figure is
+  list price. Rate limits, overload and timeouts are retried; an unknown model or
+  a missing login is not. See `docs/subscription-backends.md`.
+- **`backends`**: lists the CLI backends with install, version and login status;
+  calls no model. `benchmark preflight` checks a CLI backend's login instead
+  of an API key.
+
+### Changed
+
+- The `--provider` choices of every command come from one tuple,
+  `dokimi.llm.ANSWER_PROVIDERS`. `tools` keeps API providers only, since it needs
+  tool calling.
+- A CLI provider refuses an agentic arm (`mcp_agent`, `skill_agent`, `skill_catalog`) and
+  `--session-turns` in `arms run` and `arms matrix-run`, before any answer is paid for: the CLI
+  runs one prompt with no tools and no conversation.
+- A CLI reply records the model the CLI resolved its alias to: `resolved_model` on each usage
+  row (answers, judge calls), `judge_resolved_models` in the judge record. The response-cache
+  key takes the CLI version, so a new release that moves `sonnet` does not replay the old
+  model's replies. The resolved id is known only after a call, so a move the CLI does not
+  ship with is not caught.
+- Independence checks treat an alias and its full id as one model (`opus` and
+  `claude-opus-…`). The novelty default judge on `claude-cli` is another Claude family than the answerer.
+- The CLI child process drops `ANTHROPIC_*`, `CLAUDE_*` and `CLAUDECODE` variables from the
+  environment (except `CLAUDE_CONFIG_DIR`, `CLAUDE_CODE_OAUTH_TOKEN` and
+  `CLAUDE_CODE_GIT_BASH_PATH` (Windows)), so a billing or
+  session setting in the caller does not reroute the call. Git is stopped at the work dir's parent. A Bedrock or Vertex route set
+  in that environment is dropped too: the CLI then uses its own login, and `backends` reports
+  that login, not the route.

dokimi/docs/README.md

@@ -313,6 +313,8 @@ code.
   verifier pass rate against a baseline.
 - [answer-formats.md](answer-formats.md) — run Inspect AI, OpenAI evals,
   skill-creator and promptfoo answer-only sets on the answers track
+- [subscription-backends.md](subscription-backends.md) — run the Q&A track through a
+  logged-in coding-agent CLI (`--provider claude-cli`) instead of an API key
 - [context-window.md](context-window.md) — read a run's context window from
   its Claude Code trace with `cli.py context trace`: start-up load, growth,
   compactions, tool-result weight. No LLM call.

dokimi/docs/architecture.md

@@ -411,6 +411,10 @@ Guide: [run-consistency.md](run-consistency.md).
   adapter seam: `LiteLLMAdapter` in production, `FakeAdapter` in tests (the
   `fake_llm` fixture). `ask_parsed` is the redraw-until-it-parses loop.
   `llm_call`, `chat_completion` and `utils.get_llm` delegate to `LLM`.
+  `dokimi/llm_cli.py` serves the CLI providers (`claude-cli`): `LiteLLMAdapter`
+  hands their requests to it, and it runs the logged-in CLI through `process.run`
+  ([subscription-backends.md](subscription-backends.md)). `dokimi/llm_json.py`
+  holds the JSON extraction helpers `llm.py` re-exports.
 - `dokimi/caching.py` — the two caches that hang off `llm.py`: provider prompt
   caching (the stable part of a prompt goes first, so the provider can reuse it)
   and an opt-in response cache for byte-identical single-shot requests

dokimi/docs/reviewing.md

@@ -20,7 +20,9 @@ pip install -e .                 # installs the `dokimi` command
 The `novelty` and `arms` steps take `--model` and `--provider`; `arms run` also
 needs `--judge-provider` (it does not fall back to `--provider` for the judge);
 `tasks run` and `repo` take `--model` only. The provider is one of `openai`,
-`anthropic`, `amazon-bedrock`, `google-vertex`, `openrouter` or `openai-codex`;
+`anthropic`, `amazon-bedrock`, `google-vertex`, `openrouter`, `openai-codex` or
+`claude-cli` (a logged-in Claude Code CLI, no key; see
+[subscription backends](subscription-backends.md));
 `arms run --judge-provider` takes `google`, an alias of `google-vertex`, but not
 `openai-codex`.
 Credentials are read from the environment: `OPENAI_API_KEY`,

dokimi/docs/subscription-backends.md

@@ -0,0 +1,107 @@
+---
+depends_on:
+  - dokimi/dokimi/llm_cli.py
+  - dokimi/dokimi/commands/backends.py
+---
+<!--
+Copyright (C) 2026 Intel Corporation
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Subscription backends: a logged-in CLI as the LLM provider
+
+The Q&A track (answers, judging, question generation, personas, novelty) can call
+a model through a coding-agent CLI the machine is already signed in to, instead
+of an API key. Today one backend exists, `claude-cli` (Claude Code's `claude -p`).
+Pass it wherever a command takes `--provider` or `--judge-provider`, except `tools`, which needs tool calling.
+
+## Check what this machine can use
+
+```bash
+python cli.py backends          # one row per backend: installed, version, signed in, how
+python cli.py backends --json
+```
+
+This calls no model. A backend that is not signed in prints how to fix it. On a
+machine without a browser, `claude setup-token` prints a token; export it as
+`CLAUDE_CODE_OAUTH_TOKEN`. `benchmark preflight` checks the login the same way,
+instead of looking for an API key.
+
+## Run
+
+```bash
+python cli.py answers generate --product oneTBB --questions questions/oneTBB.json \
+  --output answers/oneTBB.json --provider claude-cli --model haiku
+python cli.py eval score --product oneTBB --answers answers/oneTBB.json \
+  --output eval/oneTBB.json --judge-provider claude-cli --judge-model sonnet
+```
+
+`--model` takes what `claude --model` takes: an alias (`haiku`, `sonnet`,
+`opus`) or a full id (`claude-sonnet-5-5`). The default judge model works on
+`claude-cli` as it does on `anthropic`. When the answerer and the judge are the
+same model, the judge grades its own answers.
+
+## How it differs from an API provider
+
+| | API provider | `claude-cli` |
+|---|---|---|
+| Credential | `*_API_KEY` | the CLI's own login |
+| `--temperature`, `--seed` | sent when set | refused before the run (`unsupported_sampling_params`) |
+| `max_tokens` | output ceiling | ignored: the CLI has no flag for it |
+| Hidden prompt | none | fixed CLI preamble (≈0.5k–1.3k tokens, by CLI version), plus working directory, git status, platform and date |
+| Tools | `chat(tools=...)` | refused: one prompt, one reply |
+| `cost_usd` | litellm price | `None`: the CLI's `total_cost_usd` is list price, not what a subscription is charged |
+| Latency | model only | plus ≈2–3 s CLI start-up per call |
+
+What this means for results:
+
+- **Don't pool scores across transports.** `claude-cli:sonnet` and
+  `anthropic:claude-sonnet-5-5` get different prompts (the CLI adds its own text)
+  and different sampling. Compare arms within one backend.
+- **The CLI states its environment, and nothing turns this off.** It does so
+  with no tools and with its own system prompt replaced;
+  `--exclude-dynamic-system-prompt-sections` was tried and does not remove it.
+  Tested and not working:
+  `CLAUDE_CODE_HIDE_CWD`, `CLAUDE_CODE_DISABLE_GIT_INSTRUCTIONS` (Claude Code
+  2.1.304). dokimi runs every call in one fixed empty directory
+  (`$XDG_CACHE_HOME/dokimi/cli/claude-cli`, default
+  `~/.cache/dokimi/cli/claude-cli`), so the stated environment is the same in the
+  with-docs and the without-docs arm, and the response cache still hits. A
+  without-docs answer may still mention it ("the working directory isn't a git
+  repository").
+- **Repeats differ.** With no temperature or seed, the same prompt gives a
+  different answer each time; measure variance with repeats, not a pinned seed.
+
+## Isolation
+
+Each call is `claude -p --input-format text --output-format json --model <m>
+--tools "" --setting-sources "" --strict-mcp-config --no-session-persistence
+--system-prompt <s>`. The prompt goes on stdin, so its size is not bounded by
+the argument limit; the system prompt is an argument and is. Without these flags, a plain `claude -p` in a repository
+carries about 37k tokens of agent prompt, tool schemas and `CLAUDE.md`.
+
+- `--bare` is never used: it reads only `ANTHROPIC_API_KEY`, so a subscription
+  login fails with "Authentication error".
+- The child gets the caller's environment minus every variable starting with
+  `ANTHROPIC_`, `CLAUDE_` or `CLAUDECODE`, except `CLAUDE_CONFIG_DIR`,
+  `CLAUDE_CODE_OAUTH_TOKEN` and `CLAUDE_CODE_GIT_BASH_PATH` (Windows). That
+  drops the session variables, an API key or
+  token that would win over the login, and a base URL, model alias or
+  Bedrock/Vertex switch, so the CLI runs on its own login. If dokimi itself
+  runs inside Claude Code and the session variables were passed on, the child
+  would join the parent's session. Git is stopped at the work dir's parent
+  (`GIT_CEILING_DIRECTORIES`), so it finds no repository above it.
+- The CLI keeps its own config directory, because that is where the login is
+  stored. User settings and `CLAUDE.md` stay out because of
+  `--setting-sources ""`.
+
+## Errors and retries
+
+Rate limits, overload, 5xx and a timeout are retried with the same backoff as
+an API provider. An unknown model, a missing login, a missing binary, an
+unusable work directory and a subscription usage limit fail on the first attempt.
+
+## Terms of use
+
+A consumer subscription is for personal use. Keep CI and shared benchmark
+runs on API keys.

dokimi/dokimi/cli.py

@@ -14,6 +14,7 @@
 from dokimi.commands import (
     answers,
     arms,
+    backends,
     baseline,
     benchmark,
     catalog,
@@ -35,7 +36,7 @@
     tasks,
     tools,
 )
-from dokimi import __version__, process
+from dokimi import __version__, llm_cli, process
 from dokimi.console import tolerate_console_encoding
 from dokimi.defaults import JUDGE_MODEL_FOR_PROVIDER, default_judge_model
 
@@ -81,6 +82,7 @@ def positive_int(value: str) -> int:
     repo.register(sub, positive_int)            # repo (repositories as subjects)
     tools.register(sub, positive_int)           # tools (tool-choice probes)
     context.register(sub, positive_int)         # context (context window of a run)
+    backends.register(sub, positive_int)        # backends (CLI providers)
 
     return p
 
@@ -99,6 +101,11 @@ def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace:
             parser.error(f"--judge-provider {args.judge_provider} has no default judge model; "
                          f"pass --judge-model")
         args.judge_model = model
+    provider, model = getattr(args, "provider", None), getattr(args, "model", None)
+    if provider in llm_cli.BACKENDS and model is not None and not llm_cli.accepts_model(provider, model):
+        # The default answer model is not a Claude model: left out, every call would fail.
+        parser.error(f"--provider {provider} takes a Claude model (sonnet, haiku, opus or a claude-* id), "
+                     f"not {model!r}: pass --model")
     return args
 
 
@@ -113,7 +120,7 @@ def main() -> None:
 
     try:
         args.func(args)
-    except (EmptyQuestionError, InputError) as exc:  # a bad input record, not a bug: no traceback
+    except (EmptyQuestionError, InputError, llm_cli.SamplingError) as exc:  # bad input, not a bug: no traceback
         print(f"Error: {exc}", file=sys.stderr)
         # 2, as the commands' other input errors: 1 is the verdict of `eval judge-validity`
         # and of the --strict gates, and a malformed file must not read as a failed gate.

dokimi/dokimi/commands/answers.py

@@ -9,6 +9,7 @@
 import sys
 from pathlib import Path
 
+from dokimi.llm import ANSWER_PROVIDERS
 from dokimi.commands.evaluate import _require_ragas, _run_ragas_eval
 from dokimi.defaults import (
     DEFAULT_ANSWER_MODEL,
@@ -158,7 +159,7 @@ def register(sub, positive_int) -> None:
                               "Sets output to results/{product}_{run_id}/answers/{product}.json")
     gen_a_p.add_argument("--output", default=None, help="Output file (default: answers/{product}.json)")
     gen_a_p.add_argument("--model", default=DEFAULT_ANSWER_MODEL, help="LLM model for answering")
-    gen_a_p.add_argument("--provider", default=DEFAULT_ANSWER_PROVIDER, choices=["openai", "anthropic", "amazon-bedrock", "google-vertex", "openrouter", "openai-codex"])
+    gen_a_p.add_argument("--provider", default=DEFAULT_ANSWER_PROVIDER, choices=ANSWER_PROVIDERS)
     gen_a_p.add_argument("--max-tokens", type=int, default=DEFAULT_MAX_TOKENS_PER_QUESTION, help="Max tokens to retrieve per question")
     gen_a_p.add_argument("--top-k", type=int, default=None, help=f"Number of docs to retrieve before reranking (default from config/products.yaml retrieval.top_k or {DEFAULT_RETRIEVAL_TOP_K})")
     gen_a_p.add_argument("--rerank-threshold", type=float, default=0.3, help="Min relevance score (0-1) to keep docs")

dokimi/dokimi/commands/arms.py

@@ -11,6 +11,8 @@
 import sys
 from pathlib import Path
 
+from dokimi.commands.arms_refusals import refuse_cli_arms, refuse_cli_cell, refuse_cli_cell_model
+from dokimi.llm import ANSWER_PROVIDERS
 from dokimi.utils import question_id
 from dokimi.eval.judge import JUDGE_PROVIDERS
 from dokimi.defaults import (
@@ -123,6 +125,7 @@ def cmd_arms_matrix_run(args: argparse.Namespace) -> None:
             cell_args = argparse.Namespace(**vars(args))
             _apply_matrix_cell(cell_args, cell)
             _check_sampling_support(cell_args)
+            refuse_cli_cell(cell_args, specs)
     except (ValueError, OSError, RecursionError) as exc:
         print(f"Error loading matrix run inputs: {exc}", file=sys.stderr)
         sys.exit(1)
@@ -233,7 +236,7 @@ def register(sub, positive_int) -> None:
     )
     arms_run_p.add_argument("--model", default=DEFAULT_ANSWER_MODEL, help="LLM for answering")
     arms_run_p.add_argument("--provider", default=DEFAULT_ANSWER_PROVIDER,
-                            choices=["openai", "anthropic", "amazon-bedrock", "google-vertex", "openrouter", "openai-codex"])
+                            choices=ANSWER_PROVIDERS)
     arms_run_p.add_argument("--harness", default="arms-runner",
                             help="Execution harness label stamped into output (default: arms-runner)")
     arms_run_p.add_argument("--plugins", default="",
@@ -379,7 +382,7 @@ def _add_common_run_args(parser, positive_int, *, matrix_required: bool = False)
     )
     parser.add_argument("--model", default=DEFAULT_ANSWER_MODEL, help="LLM for answering")
     parser.add_argument("--provider", default=DEFAULT_ANSWER_PROVIDER,
-                        choices=["openai", "anthropic", "amazon-bedrock", "google-vertex", "openrouter", "openai-codex"])
+                        choices=ANSWER_PROVIDERS)
     parser.add_argument("--harness", default="arms-runner",
                         help="Execution harness label stamped into output (default: arms-runner)")
     parser.add_argument("--plugins", default="",
@@ -552,6 +555,7 @@ def _apply_named_matrix_cell(args: argparse.Namespace) -> None:
 
 
 def _apply_matrix_cell(args: argparse.Namespace, matrix_cell) -> None:
+    refuse_cli_cell_model(matrix_cell)
     args.matrix_cell = matrix_cell.name
     args.model = matrix_cell.model
     args.provider = matrix_cell.provider
@@ -625,6 +629,7 @@ def _build_arms_output(args: argparse.Namespace, specs: list[str], questions: li
     plugins = create_plugins([s.strip() for s in args.plugins.split(",") if s.strip()])
     validate_plugins_for_harness(plugins, args.harness)
     treatments = wrap_treatments(treatments, plugins)
+    refuse_cli_arms(args, treatments)
     plugin_set = plugin_set_metadata(plugins)
     print(f"Arms: {', '.join(t.name for t in treatments)}")
     if args.matrix_cell:

dokimi/dokimi/commands/arms_refusals.py

@@ -0,0 +1,62 @@
+# Copyright (C) 2026 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+"""Refusals for the arms commands on a CLI provider, raised before any answer is paid for."""
+
+from __future__ import annotations
+
+import argparse
+
+
+def _is_agentic_arm(treatment) -> bool:
+    """True when the arm offers tools, so it runs the agentic loop (see ``AgentConfig.tools``).
+
+    The tools are added in ``prepare()``, which needs a question and may retrieve, so the
+    arm types that offer tools are named here instead of probed per question.
+    """
+    from dokimi.plugins import PluginWrappedTreatment
+    from dokimi.treatments.arms import (
+        CombinedTreatment,
+        MCPAgentTreatment,
+        SkillAgentTreatment,
+        SkillCatalogTreatment,
+    )
+
+    inner = treatment.inner if isinstance(treatment, PluginWrappedTreatment) else treatment
+    if isinstance(inner, CombinedTreatment):
+        return any(_is_agentic_arm(sub) for sub in inner.treatments)
+    return isinstance(inner, (MCPAgentTreatment, SkillAgentTreatment, SkillCatalogTreatment))
+
+
+def refuse_cli_arms(args: argparse.Namespace, treatments: list) -> None:
+    """Refused before any answer: a CLI provider runs one prompt, with no tools and no conversation."""
+    from dokimi import llm_cli
+
+    if llm_cli.backend(args.provider) is None:
+        return
+    agentic = [t.name for t in treatments if _is_agentic_arm(t)]
+    if agentic:
+        raise ValueError(f"--provider {args.provider} runs one prompt with no tools, so it cannot run "
+                         f"the agentic arm(s) {', '.join(agentic)}: use an API provider for them")
+    if getattr(args, "session_turns", None) is not None:
+        raise ValueError(f"--session-turns needs a conversation, which --provider {args.provider} "
+                         "does not have: use an API provider for a session")
+
+
+def refuse_cli_cell_model(matrix_cell) -> None:
+    """A matrix cell sets provider and model after the parse-time model gate has read only --model."""
+    from dokimi import llm_cli
+
+    if llm_cli.backend(matrix_cell.provider) and not llm_cli.accepts_model(matrix_cell.provider,
+                                                                          matrix_cell.model):
+        raise ValueError(f"matrix cell {matrix_cell.name!r}: provider {matrix_cell.provider} takes a "
+                         f"Claude model (sonnet, haiku, opus or a claude-* id), not {matrix_cell.model!r}")
+
+
+def refuse_cli_cell(args: argparse.Namespace, specs: list[str]) -> None:
+    """A matrix cell on a CLI provider with an agentic arm is refused before the first cell is answered."""
+    from dokimi import llm_cli
+    from dokimi.treatments import create_treatments
+
+    if llm_cli.backend(args.provider) is not None:
+        refuse_cli_arms(args, create_treatments(specs, top_k=args.top_k, rerank_threshold=args.rerank_threshold))