fix(answers,pipeline,judge,gate): Stack P fixes, step 2 of #228

dokimi/changelog.d/20261028_120000_napetrov_stack-p-step2.md

@@ -0,0 +1,48 @@
+### Added
+
+- **`benchmark run --judge-temperature`** — a new flag: it sets the judge's sampling
+  temperature and is recorded as `judge_temperature` in the eval output. Before this change the
+  flag did not exist, and `benchmark run` rejected it as an unknown argument.
+
+### Changed
+
+- **Stack P scores move (#228 step 2)** — the context arm now asks the doc source for the
+  `top_k` docs the command was given: `benchmark run` asks for 3 (`DEFAULT_RETRIEVAL_TOP_K`; it
+  has no `--top-k`), `dokimi evaluate` and `answers generate` for their `--top-k` (default 3).
+  Before, they left `max_results` at the client default of 5. The doc library id resolves
+  through the registry's `context7_id` before the client's own resolver, in the `--registry`
+  that was loaded. A question with no document (no hit, or the reranker raised) is no longer judged on
+  its `[FALLBACK: ...]` placeholder by default, so it drops out of the paired statistics; the
+  record and its `retrieval_hit` stay, its diagnosis stays `empty_retrieval`, and the run
+  summary says how many questions it left out. Pass `--judge-placeholder` to judge the
+  placeholder again (this restores only that part of the old method). `benchmark run`,
+  `evaluate` and `eval score` take the same flag; `benchmark batch` has none and never judges
+  the placeholder. Scores from before this change are not comparable with new ones.
+- **Trust gate** — `retrieval_hit_rate` counts every evaluation's recorded retrieval outcome,
+  not only the ones scored in both arms, so unjudged misses still count against the rate, and a
+  judge-error row keeps the outcome of its answer. The rate is shown even when no question is
+  scored in both arms. The thresholds are unchanged.
+- **Zero-score check** — a judge-error row on a retrieval miss now counts as unusable in
+  `zero_score_fraction` (a judge error on a retrieval hit is still left out), so a run can fail
+  that fail-severity check where intel/main passed it.
+- **Stamped artifacts** — the answers and eval artifacts carry `method_version` (2). Artifacts
+  from before this change have no stamp and represent the legacy method (version 1); readers do
+  not yet normalize or enforce the version, so it does not yet stop a comparison of mixed methods.
+  `eval score` stamps its eval with the version of the answers file it judges, so legacy answers
+  re-judged today stay version 1. An eval artifact also carries `judge_placeholder`, so two files
+  of one version that differ in it are told apart.
+
+### Fixed
+
+- **Stack P no longer ignores `top_k`** — `Answerer` passes `max_results=top_k` to the doc
+  source; the context arm's retrieval request is now the one the arms stack makes.
+- **Library id from the registry** — `EvaluationPipeline` resolves the library id through
+  `resolve_library_id_for_product` (explicit id, then an `mcp:` source's `;id=`, then `context7_id`,
+  then the client), in the
+  registry that `benchmark run` and `benchmark batch` loaded from `--registry`. A registered
+  product without `context7_id` now fails for a client that needs a declared id.
+
+### Removed
+
+- Dead code in the answering path: the unused `Answerer.llm` client and the unreachable
+  "no documents" branch of `_generate_with_docs`.

dokimi/docs/methodology.md

@@ -199,11 +199,11 @@ RAGAS meta-evaluation can be run after answer generation to add faithfulness and
 
 Every serious run should answer "Can we trust this run?" before interpreting deltas. The trust gate checks:
 
-- minimum evaluated question count,
-- zero-score fraction,
-- minimum `with_docs` average,
-- average delta,
-- score coefficient of variation,
+- minimum evaluated question count (both arms scored; a question with no document is not judged by default, so it is not counted here),
+- zero-score fraction (computed when at least one question is scored in both arms; over those paired questions plus retrieval misses with a scored baseline or a judge error, which count as unusable),
+- minimum `with_docs` average (both-arm-scored questions only),
+- average delta (both-arm-scored questions only),
+- score coefficient of variation (both-arm-scored questions only),
 - inter-rater agreement when panel data exists,
 - retrieval hit rate — the share of questions where at least one document reached the `with_docs` prompt, from the per-evaluation `retrieval_hit` the judge and panel record; eval files judged before that field existed show it as not measured, and so do files where fewer than 80% of evaluations record it (`min_retrieval_coverage`), since the rate would then describe a subset,
 - multi-run variance when `--multi-run N` is used.

dokimi/docs/reference/cli-interface.json

@@ -2250,6 +2250,22 @@
               "required": false,
               "type": null
             },
+            {
+              "action": "_StoreTrueAction",
+              "choices": null,
+              "const": true,
+              "default": false,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": true,
+              "key": "--judge-placeholder",
+              "kind": "option",
+              "nargs": 0,
+              "position": null,
+              "required": false,
+              "type": null
+            },
             {
               "action": "_StoreAction",
               "choices": [
@@ -2273,6 +2289,22 @@
               "required": false,
               "type": null
             },
+            {
+              "action": "_StoreAction",
+              "choices": null,
+              "const": null,
+              "default": null,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": false,
+              "key": "--judge-temperature",
+              "kind": "option",
+              "nargs": null,
+              "position": null,
+              "required": false,
+              "type": "float"
+            },
             {
               "action": "_StoreAction",
               "choices": null,
@@ -5746,6 +5778,22 @@
               "required": false,
               "type": null
             },
+            {
+              "action": "_StoreTrueAction",
+              "choices": null,
+              "const": true,
+              "default": false,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": true,
+              "key": "--judge-placeholder",
+              "kind": "option",
+              "nargs": 0,
+              "position": null,
+              "required": false,
+              "type": null
+            },
             {
               "action": "_StoreAction",
               "choices": [
@@ -5955,6 +6003,22 @@
           "required": false,
           "type": null
         },
+        {
+          "action": "_StoreTrueAction",
+          "choices": null,
+          "const": true,
+          "default": false,
+          "group_key": null,
+          "group_required": null,
+          "hidden": false,
+          "is_flag": true,
+          "key": "--judge-placeholder",
+          "kind": "option",
+          "nargs": 0,
+          "position": null,
+          "required": false,
+          "type": null
+        },
         {
           "action": "_StoreAction",
           "choices": [

dokimi/dokimi/commands/benchmark.py

@@ -27,7 +27,7 @@
 from dokimi.commands.library import _load_registry
 
 
-def _run_single_library(entry, output_dir: str, model: str, provider: str, judge_model: str, judge_provider: str = DEFAULT_JUDGE_PROVIDER, doc_source_override=None, max_tokens_per_question: int = DEFAULT_MAX_TOKENS_PER_QUESTION, force_regen: bool = False, concurrency: int = DEFAULT_CONCURRENCY, questions_from=None, questions_root=None) -> dict:
+def _run_single_library(entry, output_dir: str, model: str, provider: str, judge_model: str, judge_provider: str = DEFAULT_JUDGE_PROVIDER, doc_source_override=None, max_tokens_per_question: int = DEFAULT_MAX_TOKENS_PER_QUESTION, force_regen: bool = False, concurrency: int = DEFAULT_CONCURRENCY, questions_from=None, questions_root=None, judge_temperature=None, judge_placeholder=False, registry=None) -> dict:
     """Run full evaluation pipeline for one ProductEntry. Returns result dict."""
     from dokimi.orchestrator import EvaluationPipeline
     from pathlib import Path as _Path
@@ -58,12 +58,15 @@ def _run_single_library(entry, output_dir: str, model: str, provider: str, judge
         provider=provider,
         judge_model=judge_model,
         judge_provider=judge_provider,
+        judge_temperature=judge_temperature,
+        judge_placeholder=judge_placeholder,
         context7_id=entry.context7_id,
         doc_source=doc_source,
         max_tokens_per_question=max_tokens_per_question,
         force_regen=force_regen,
         questions_from=questions_from,
         product_key=entry.key,
+        registry=registry,
         questions_root=_Path(questions_root) if questions_root else None,
     )
     result = pipeline.run(concurrency=concurrency)
@@ -247,6 +250,9 @@ def cmd_benchmark_run(args: argparse.Namespace) -> None:
             concurrency=concurrency,
             questions_from=questions_from,
             questions_root=questions_root,
+            judge_temperature=getattr(args, "judge_temperature", None),
+            judge_placeholder=getattr(args, "judge_placeholder", False),
+            registry=registry,
         )
     else:
         print(f"\n🔁 Multi-run mode: {n_runs} evaluation passes")
@@ -269,6 +275,9 @@ def cmd_benchmark_run(args: argparse.Namespace) -> None:
                 concurrency=concurrency,
                 questions_from=qfrom,
                 questions_root=questions_root,
+                judge_temperature=getattr(args, "judge_temperature", None),
+                judge_placeholder=getattr(args, "judge_placeholder", False),
+                registry=registry,
             )
             try:
                 eval_summary = r["result"]["steps"]["evaluation"]["summary"]
@@ -323,6 +332,7 @@ def cmd_benchmark_batch(args: argparse.Namespace) -> None:
                 force_regen=getattr(args, "force_regen", False),
                 concurrency=getattr(args, "concurrency", DEFAULT_CONCURRENCY),
                 questions_root=getattr(args, "questions_root", None),
+                registry=registry,
             )
             results.append(r)
         except Exception as exc:
@@ -369,6 +379,10 @@ def register(sub, positive_int) -> None:
                              help=JUDGE_MODEL_HELP)
     bench_run_p.add_argument("--judge-provider", default=DEFAULT_JUDGE_PROVIDER, dest="judge_provider",
                              choices=JUDGE_PROVIDERS)
+    bench_run_p.add_argument("--judge-temperature", type=float, default=None, dest="judge_temperature",
+                             help="Judge sampling temperature, recorded as judge_temperature in the eval output (default: provider default)")
+    bench_run_p.add_argument("--judge-placeholder", action="store_true", dest="judge_placeholder",
+                             help="Judge the no-retrieval placeholder answers too (default: they are not scored)")
     bench_run_p.add_argument("--registry", default=None, help="Path to custom products.yaml")
     bench_run_p.add_argument("--max-tokens", type=positive_int, default=DEFAULT_MAX_TOKENS_PER_QUESTION, dest="max_tokens",
                              help="Max tokens to retrieve per question from doc source (default: 4000)")

dokimi/dokimi/commands/evaluate.py

@@ -17,6 +17,7 @@
     DEFAULT_JUDGE_PROVIDER,
     JUDGE_MODEL_FOR_PROVIDER,
     JUDGE_MODEL_HELP,
+    LEGACY_ANSWERING_METHOD_VERSION,
 )
 from dokimi.eval.judge import JUDGE_PROVIDERS
 from dokimi.eval.judge_validity import DEFAULT_MIN_ACCURACY, NoUsablePairs
@@ -203,6 +204,9 @@ def cmd_eval_score(args: argparse.Namespace) -> None:
         model=args.judge_model,
         provider=args.judge_provider,
         temperature=args.judge_temperature,
+        judge_placeholder=getattr(args, "judge_placeholder", False),
+        # The eval says which answering method made the answers it judges; an unstamped file is legacy.
+        answers_method_version=run.meta.get("method_version") or LEGACY_ANSWERING_METHOD_VERSION,
         run_metadata={
             "question_model": run.meta.get("question_model", "unknown"),
             "question_provider": run.meta.get("question_provider", "unknown"),
@@ -478,6 +482,8 @@ def register(sub, positive_int) -> None:
     score_p.add_argument("--concurrency", type=positive_int, default=DEFAULT_CONCURRENCY, help="Parallel judge calls (default: 5)")
     score_p.add_argument("--judge-temperature", type=float, default=None, dest="judge_temperature",
                          help="Judge sampling temperature, recorded as judge_temperature in the output")
+    score_p.add_argument("--judge-placeholder", action="store_true", dest="judge_placeholder",
+                         help="Judge the no-retrieval placeholder answers too (default: they are not scored)")
     score_p.set_defaults(func=cmd_eval_score)
 
     # eval ragas

dokimi/dokimi/commands/orchestrate.py

@@ -60,6 +60,7 @@ def cmd_evaluate(args: argparse.Namespace) -> None:
             judge_model=args.judge_model,
             judge_provider=args.judge_provider,
             judge_temperature=args.judge_temperature,
+            judge_placeholder=getattr(args, "judge_placeholder", False),
             personas_count=args.personas_count,
             questions_per_topic=args.questions_per_topic,
             # Only pass top_k when explicitly set; otherwise let pipeline use config default
@@ -122,6 +123,8 @@ def register(sub, positive_int) -> None:
     eval_p.add_argument("--judge-provider", default=DEFAULT_JUDGE_PROVIDER, choices=JUDGE_PROVIDERS)
     eval_p.add_argument("--judge-temperature", type=float, default=None, dest="judge_temperature",
                         help="Judge sampling temperature, recorded as judge_temperature in the eval output")
+    eval_p.add_argument("--judge-placeholder", action="store_true", dest="judge_placeholder",
+                        help="Judge the no-retrieval placeholder answers too (default: they are not scored)")
     eval_p.add_argument("--personas-count", type=positive_int, default=5, help="Target number of personas")
     eval_p.add_argument("--questions-per-topic", type=positive_int, default=2, help="Questions per topic per persona")
     eval_p.add_argument("--top-k", type=positive_int, default=None, help="Docs to retrieve before reranking (default from config/products.yaml retrieval.top_k)")

dokimi/dokimi/defaults.py

@@ -90,6 +90,12 @@ class _JudgeModelForProvider(str):
 
 # Decision (2026-03-04): one fixed semantic retrieval config.
 DEFAULT_RETRIEVAL_TOP_K = 3
+
+# Answering/judging method version, stamped into answers and eval artifacts; bump it when a change moves scores.
+# A later compare is meant to refuse mixed versions; an unstamped artifact predates it (#228 step 2) and is v1.
+ANSWERING_METHOD_VERSION = 2
+#: The version of an artifact that carries no stamp (written before ANSWERING_METHOD_VERSION existed).
+LEGACY_ANSWERING_METHOD_VERSION = 1
 DEFAULT_MAX_TOKENS_PER_QUESTION = 4000
 DEFAULT_CONCURRENCY = 5
 DEFAULT_RESULTS_SUFFIX = "_final"

dokimi/dokimi/eval/answerer.py

@@ -15,9 +15,13 @@
 
 from dokimi.utils import question_id
 from dokimi.artifacts import save_artifact
-from dokimi.llm import ChatModel
-from dokimi.llm import llm_call_with_usage, ChatOpenAI, ChatAnthropic, LANGCHAIN_AVAILABLE
-from dokimi.defaults import DEFAULT_ANSWER_MODEL, DEFAULT_ANSWER_PROVIDER
+from dokimi.defaults import (
+    ANSWERING_METHOD_VERSION,
+    DEFAULT_ANSWER_MODEL,
+    DEFAULT_ANSWER_PROVIDER,
+    DEFAULT_RETRIEVAL_TOP_K,
+)
+from dokimi.llm import llm_call_with_usage, LANGCHAIN_AVAILABLE
 from dokimi.metrics.usage import UsageRecord, to_record
 
 from .reranker import SimpleReranker, SentenceTransformerReranker, SENTENCE_TRANSFORMERS_AVAILABLE
@@ -70,7 +74,7 @@ def __init__(
         model: str = DEFAULT_ANSWER_MODEL,
         provider: str = DEFAULT_ANSWER_PROVIDER,
         api_key: Optional[str] = None,
-        top_k: int = 5,
+        top_k: Optional[int] = None,
         rerank_threshold: float = 0.3,
         debug_retrieval: bool = False
     ):
@@ -80,7 +84,7 @@ def __init__(
             model: LLM model name
             provider: "openai" or "anthropic"
             api_key: Optional API key
-            top_k: Number of docs to retrieve before reranking
+            top_k: Number of docs to retrieve before reranking (default DEFAULT_RETRIEVAL_TOP_K)
             rerank_threshold: Minimum relevance score (0-1) to keep docs
             debug_retrieval: If True, include detailed retrieval metadata in output
         """
@@ -95,20 +99,15 @@ def __init__(
         self.provider = provider
         self.api_key = api_key
 
-        self.llm: Optional[ChatModel]
-        if provider == "openai":
-            self.llm = ChatOpenAI(model=model, api_key=api_key)
-        elif provider == "anthropic":
-            self.llm = ChatAnthropic(model=model, api_key=api_key)
-        elif provider in ("google", "gemini", "google-vertex", "openrouter",
-                          "amazon-bedrock", "vertex_ai"):
-            self.llm = None  # llm_call_with_usage handles these natively
-        else:
+        # The answer call goes through llm_call_with_usage, which picks the provider's client itself.
+        if provider not in ("openai", "anthropic", "google", "gemini", "google-vertex",
+                            "openrouter", "amazon-bedrock", "vertex_ai"):
             raise ValueError(
                 f"Unsupported provider: '{provider}'. "
                 "Use 'openai', 'anthropic', 'google', 'openrouter', or 'amazon-bedrock'."
             )
-        self.top_k = top_k
+        # None means the default: the doc source must never get max_results=None (it would return every chunk).
+        self.top_k = DEFAULT_RETRIEVAL_TOP_K if top_k is None else top_k
         self.debug_retrieval = debug_retrieval
 
         # Initialize reranker (prefer semantic, fallback to lexical)
@@ -315,12 +314,11 @@ def _retrieve_docs(
         }
 
         try:
-            # Retrieve top-k docs
-            # Note: Context7 client returns single doc, so we call it once for now
-            # TODO: Update Context7 client to support top_k natively
+            # Retrieve top-k docs, the same request the arms stack sends (max_results=top_k)
             raw_docs = self.mcp_client.get_library_docs(
                 library_id=library_id,
                 query=question,
+                max_results=self.top_k,
                 max_tokens=max_tokens
             )
             metadata["raw_count"] = len(raw_docs)
@@ -358,24 +356,7 @@ def _generate_with_docs(
         retrieval_metadata: Optional[Dict[str, Any]] = None,
         verbatim: bool = False,
     ) -> Dict[str, Any]:
-        """Generate context-arm answer."""
-        if not docs:
-            # No LLM call is made — emit a zero-cost/zero-token metrics block so
-            # this row carries the same telemetry shape as answered rows.
-            empty = UsageRecord(model=self.model, provider=self.provider,
-                                cost_usd=0.0, n_calls=0)
-            token_usage = empty.as_token_usage_dict()
-            token_usage["context_chars"] = 0
-            return {
-                "answer": "[No documentation retrieved]",
-                "retrieved_docs": [],
-                "model": self.model,
-                "doc_source": "none",
-                "retrieval_metadata": retrieval_metadata if self.debug_retrieval else None,
-                "token_usage": token_usage,
-                "metrics": empty.as_metrics_dict(answer_chars=0),
-            }
-
+        """Generate context-arm answer. Called only with a non-empty *docs* (see ``_generate_answer_pair``)."""
         # ~4k tokens at ~4 chars/token; keeps prompt within most models' context window
         MAX_DOC_CHARS = 15_000
 
@@ -458,6 +439,7 @@ def _build_output(self, answers: List[Dict[str, Any]]) -> Dict[str, Any]:
 
         return {
             "generated_at": self._get_timestamp(),
+            "method_version": ANSWERING_METHOD_VERSION,
             "model": self.model,
             "provider": self.provider,
             "total_questions": len(answers),