Answering stack: one implementation for evaluate/benchmark/answers and arms/subjects #228

Split out of #94 (§5 row X), as @antonwolfy asked in his review: merging the two answering stacks is the largest refactor in the CLI plan, and "blocked on" hid its size. Facts below are from main @ 85b8d84a; paths are under dokimi/.

Problem

dokimi has two code paths that answer a question with an LLM, with and without docs, and judge the answers. They were written separately, and they give different numbers for the same question set.

No CI workflow runs either stack today, so the drift has gone unnoticed.

Differences that change scores

1. Retrieval request.
   • Stack P sends max_tokens=4000 and leaves max_results at its default (answerer.py:308). top_k is stored (:102) and never read; a TODO at :307 says so.
   • Stack A sends max_results=top_k and leaves max_tokens at the client default of 8000 (treatments/arms.py:81-83).
   • The clients cut context at max_tokens*4 characters, so that is 16k vs 32k. Context7's tokens parameter and cache key differ too (mcp/context7.py:119, :144), so the two stacks retrieve different docs for the same question.
2. No docs found.
   • Stack P judges the literal string "[FALLBACK: No relevant documentation found]" as the with-docs answer, with no model call (answerer.py:248-260).
   • Stack A makes a real call with the plain prompt, which is the baseline.
3. Rerank exception. Stack P falls back to the placeholder (answerer.py:337). Stack A uses the raw docs (treatments/arms.py:93-98).
4. Answer call.
   • Stack P calls llm_call_with_usage with no system prompt, max_tokens, temperature, seed or cache prefix (answerer.py:379, :417).
   • Stack A passes all five (arm_runner.py:377).
   • The prompts differ as well. Stack P puts the question first and says "use ONLY the documentation". Stack A puts the context first and says "where relevant".
5. What the judge sees.
   • Stack P shows 200-character Doc i: snippets (eval/judge.py:381, :606), and its baseline gets "(No documentation provided)".
   • Stack A shows the full material through eval/judge_context.py:278, stamped treatment_material.v1.
6. Judge temperature. benchmark does not pass it (commands/benchmark.py:52); evaluate does.
7. --max-tokens means opposite things. In Stack P it is the retrieval budget (benchmark.py:358). In Stack A it is the output ceiling (arms.py:270). This is the same drift that CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94 §2.4 lists.
8. Library id. The pipeline resolves it without the registry (pipeline.py:622); answers generate and arms use the registry.
9. Statistics. Stack P reports unpaired means and deltas. Stack A reports paired tests with holm/bh and interactions.

Only in one stack

• Only in Stack A:
  • N arms, agentic arms, skills, sessions
  • --judge-panel
  • cost_summary (USD, cache hits, latency)
  • --grounded-from-topics
• Only in Stack P: personas, question generation and the question-set hash.
• Neither has: a budget cap, or resume. Only answers generate and eval score save incrementally.

Already shared

llm.py, the Judge class (Stack P calls evaluate_answers at judge.py:267; Stack A calls score_answer at :586), reranker.py, the MCP clients and factory, metrics/usage.py, records.py (it reads both shapes), save_artifact and defaults.py.

About ten consumers read answers.v1/eval.v1, including trust_gate, the dashboard, report, model_compare and ragas.

Dead code in Stack P

Answerer.llm (answerer.py:89-101) is never used. The empty-docs branch at :348-363 cannot be reached.

Drift is happening now

#214 adds a verbatim prompt style to both answerer.py and arm_runner.py, in two different shapes. #217 touches commands/arms.py, and #166 touches llm.py.

Why it blocks #94

#94 turns evaluate, benchmark and subjects into the pipeline, benchmark and subject recipes of plan/run (§2.2). With two stacks, a recipe would run on one stack while run arms runs on the other, and compare would put numbers side by side that were measured in different ways.

Proposal: keep Stack A

Stack A already covers Stack P's two arms as a special case: DocTreatment is with_docs and Baseline is without_docs. It also has paired statistics, cost accounting and panels. Stack P keeps what it does upstream of answering: personas, question generation and the answers.v1/eval.v1 consumers.

Steps. Each is its own PR, and a PR that changes numbers is never mixed with a refactor (#4 conventions):

1. Characterization test, no behaviour change. Run both stacks on one fixture and record, for each stack:

   • the retrieval arguments
   • the prompts
   • the provider call counts
   • the judge context
   • the scores
   • the cost

   It uses fake_llm and provider_calls (tests/conftest.py:53), plus a fake MCP client that records max_results/max_tokens, a stub reranker and frozen timestamps. Cases: docs found, no docs, rerank exception, a ground-truth chunk.

   The test pins today's differences as expected values. Later PRs flip them one at a time, so each change in numbers is visible in the diff.

   The tests that exist today do not cover this:

   • test_pipeline_question_set stubs whole pipeline steps.
   • test_eval_answerer patches llm_call_with_usage instead of using fake_llm.
   • test_provider_call_budgets counts calls but never compares the two stacks.

2. Fix Stack P in place. This changes numbers.

   • Honour top_k.
   • Pass the judge temperature in benchmark.
   • Resolve the library id through the registry.
   • Put placeholder judging behind an explicit flag, so a missing doc is no longer scored as an answer by default.
   • Delete the dead code.

   Stamp a method_version in the artifacts so that old and new runs are not compared silently.

3. One retrieval path. Stack P retrieves through DocTreatment. After this, the test from step 1 shows identical retrieval arguments.

4. The pipeline delegates answering and judging to ArmRunner.

   • Add an adapter from arms.v1 to answers.v1/eval.v1, so the ten consumers keep working.
   • Add incremental save to ArmRunner first. Without it, evaluate would lose the partial results it saves today.

5. Flag.

   • In the Stack P commands, --max-tokens becomes --retrieval-max-tokens, with a tombstone (CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94 §2.4/F1b, concept "tokens").
   • In Stack A, --max-tokens keeps its meaning as the output ceiling.

6. Remove the rest.

   • The consumers read through records.py.
   • Answerer and the adapter are deprecated, then removed.
   • answers generate becomes run answers on the shared stack.

Before step 3, decide how #214 adds verbatim: either as one style in a shared prompt builder, or by landing it in Stack A only.

Risks

• Historical comparability. Steps 2–4 change scores, so old and new runs are not comparable. The method_version stamp lets compare refuse to put them side by side, or mark the comparison as exploratory.
• The trust gate. Its thresholds were calibrated on rows where the placeholder was judged. Re-check them after step 2.
• Old eval.v1 files have no judge-material stamp. records.py must read them as "material unknown".
• Scripts. The meaning of --max-tokens flips for existing scripts. The tombstone in step 5 has to catch the old use.
• Cache. The Context7 cache key changes, so the first runs after step 3 are slower and cost more.
• Complexity baselines. scripts/growth_baseline.json has entries for ArmRunner.judge and EvaluationPipeline.run. Update them in the PR that moves the code.

Out of scope

• The other duplicated subsystems: comparison statistics, artifact writers and so on. They are tracked in CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94 §5.
• New features in ArmRunner.

Links: #94 (§2.2, §2.4, §5 X), #4 (waves 4 and 5), #121 (user loop), #214, #217, #166.