Architecture cleanup: fix live bugs, then deepen the LLM, eval-record, novelty, arms and process modules #4

Why

An architecture review of dokimi found several concepts implemented in more than one place: the LLM call path, reading answer/eval records, spawning subprocesses, parsing arm specs, novelty coverage. The copies have drifted apart, and some of that drift is live bugs on main. This issue lists the fix series and the order it lands in, so reviewers can see the whole plan and contributors know which files will be busy.

Wave 1: bug fixes

Each PR carries a regression test that fails on main.

• fix(tasks): resolve --output-dir before handing it to the agent, so a failed cell can't read as passed #8 tasks run with the default relative --output-dir can score a failed cell as passed. The agent writes its reward relative to the task checkout, the framework reads it relative to its own cwd, finds nothing, and falls back to returncode == 0.

• fix(panel): judge answers on their real question and context, and make parse retries real draws #5 eval panel-score judges every answer with an empty question and empty context: it reads question/content, but the answerer writes question_text/snippet. Separately, the panel's parse retries replay the same cached reply.

• fix(trust-gate): measure retrieval hit rate from a recorded retrieval outcome #10 The trust gate's retrieval_hit_rate always passes, because nothing writes retrieval_hit. Fix: the judge and panel stamp it per evaluation, and older eval files report "not measured".

• fix(cli): offer only the judge providers the Judge accepts #11 The CLI offers openai-codex as a judge provider. Judge rejects it, but only after the answers have been paid for.

• fix(tasks): error the cell when its stale run dir can't be cleared #24 Follow-up to fix(tasks): resolve --output-dir before handing it to the agent, so a failed cell can't read as passed #8: error the cell when its stale run dir can't be cleared.

Wave 2: current model defaults

• feat(defaults)!: one current answer/judge/generator default on every surface (gpt-6.1-sol / claude-sonnet-5-5) #28 One default set on every surface: gpt-6.1-sol for answers and generation, claude-sonnet-5-5 for the judge. Drop gpt-4o-mini. Require litellm ≥ 1.105.0 for prices. Results will not be comparable with older baselines.

Wave 3: low-conflict refactors

• refactor(artifacts): write answers, eval and arms through save_artifact #27 Write answers, eval and arms through artifacts.save_artifact (validated, atomic).
• refactor(gate): merge soft_gate and hard_gate into one score_gate #22 Merge gate/soft_gate.py and gate/hard_gate.py.
• feat(questions): write questions.v2 (question), keep reading v1 (text) #43 Add questions.v2 with a canonical question field; v1 text files are still read. Generated sets currently fail questions.v1.
• refactor(treatments): one arm-spec grammar module for split, parse and naming #26 One arm-spec grammar module covering + and how names relate to specs.
• refactor(novelty): the bundle record owns its validity #44 Novelty: the bundle record owns exclusion classes, coverage and the answer-key verdict (today the gate and the analyzer disagree on the unmeasured share). Follow-up: RunSettings and a thin CLI. Coverage basis: fix(novelty-gate): measure the unmeasured share over readable words, and stamp the basis #101. settings: refactor(novelty): one flag-to-setting resolver (RunSettings); thin the novelty CLI #105

Wave 4: central modules

Busy files during this wave: llm.py, utils.py, eval/*, report/*.

• One LLM-call module for provider routing, retry, draws, cache and usage, replacing utils.get_llm. About 5 PRs. Step 1, the module with old entry points delegating: refactor(llm): one LLM module behind every call; old entry points delegate #78.
• refactor(eval): give answer and eval records one reader (eval/records.py) #80 eval/records.py: a single reader for answer, eval and arms records.
• eval/compare.py: pairing, summaries and significance. This changes published numbers: report generate and report eval currently disagree on the same file.

Wave 5

• Shared CLI model/provider flags, and an arms run module behind a typed run config instead of an argparse.Namespace hand-off.
• refactor(process): one process runner for git and misc spawn sites #29 (git and misc sites; harnesses follow) One process runner: kills the whole process tree on timeout, decodes UTF-8, takes an explicit env, keeps partial output. Rebuilt on the current docker_solver. harnesses: refactor(harnesses): command, explorer, BenchFlow and repo-mode docker on the process runner #79 docker solver: refactor(harnesses): docker-* solver, sandbox and probe on the process runner #104

Wave 6

• Design issue: a harness = a profile + where the runner runs + a transcript parser.

How these land

• Each PR has one purpose. Fixes, behaviour-preserving refactors, and changes to numbers or defaults are never mixed in one PR.
• At most two open PRs per hot area. The next one is rebased after the previous one merges.
• Wave 4 lands back-to-back. Until it is through, please hold PRs that touch llm.py, eval/* or report/*, or mention them here.