Record corrected Sarvam reporting gains and add capped-actor recovery

README.md

@@ -105,14 +105,15 @@ An earlier n=30 head-to-head against Mem0, Letta, and LangMem lives with its cav
 
 ## Sarvam research: investigating the problem
 
-We test native Sarvam models with ordinary SQLite storage, tool permissions and source transcripts before deciding whether a new memory architecture is needed. English, Hindi and Telugu are the current diagnostic languages; they do not establish support across all Sarvam languages.
+We test native Sarvam models with ordinary SQLite storage, tool permissions and source transcripts before deciding whether a new memory architecture is needed. English, Hindi and Telugu are diagnostic languages here; they do not establish support across all Sarvam languages.
 
-The latest completed saved-history test changed one reporting sentence. Base105B factual answers improved **59/72 → 72/72**; Conversations **70/72 → 71/72**, including one regression. Raw execution-status errors and a previously missed approved action remain. No new actions were performed in that test, and no memory advantage or general accuracy claim follows. [Report and reproducible evidence](benchmarks/audits/sarvam-reporter-wording-2026-10-11/REPORT.md).
+The corrected comparison attempted **240 native episodes** using fixed models and shared histories for two reporter variants. Base105B date-plus-clarification accuracy improved **90/120 → 105/120 (75% → 87.5%)**. Conversations moved **95/120 → 96/120**, but wrong non-null dates increased **18 → 22**. One capped native response prevented both reporters from running; the two missing slots remain failures. **The acceptance gate failed and the candidate is not promoted.** [Report and reproducible evidence](benchmarks/audits/sarvam-fresh-confirmation-v3-2026-10-13/REPORT.md).
 
-The fresh-workflow confirmation stopped after 32/240 episodes because our question wording was ambiguous about a historical request versus a current preference. All 111 calls are reconciled; no candidate is promoted. [Stopped-run evidence and corrected package](benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/REPORT.md) preserve the failure. The revised 120-case package is frozen and unexecuted. Follow its actual status and the problem-selection criteria in [the Sarvam research index](docs/SARVAM_RESEARCH.md). A public benchmark remains planned; these development cases are not an independent benchmark.
+This is a reporting-integration result on related synthetic development cases, with assistant-only language review. It does not demonstrate a new memory advantage or improve the separate English LongMemEval baseline. An evidence-view component and metered comparison harness are implemented, but the proposed 108-call screen remains unexecuted because its complete-parent requirements were not met. Follow the findings, recovery work and problem-selection criteria in [the Sarvam research index](docs/SARVAM_RESEARCH.md). A reviewed public benchmark remains required.
 
 ---
 
+
 ## Quickstart
 
 **Requirements:** Python 3.11+, Redis running locally. No API key required; runs fully offline with Ollama.

benchmarks/audits/sarvam-fresh-confirmation-v3-2026-10-13/INTERPRETATION.md

@@ -0,0 +1,15 @@
+# Interpretation of the completed attempts
+
+The fixed Sarvam-105B profile improved from 90/120 to 105/120 source-correct date/clarification answers. Twenty pairs improved and five regressed. Fourteen improvements recovered a previously unusable reporter output; six improved a previously usable but wrong answer. This supports a narrower conclusion about reporting reliability, not a new memory architecture. Usable finals increased from 100 to 116. Nine candidate answers have the correct null date but the wrong clarification flag; date-only 114/120 is supplementary and must not replace the frozen primary 105/120 headline.
+
+Conversations improved only from 95/120 to 96/120, with four gains and three losses. Non-null wrong dates increased from 18 to 22, directly violating the frozen no-increase criterion. Strict task success decreased from 85 to 84 because truthful answers on episodes with unnecessary action attempts cannot count as strict success. No pooled score may hide these regressions.
+
+Two separate conditions fail the acceptance gate: the native cutoff left two reporter slots unreached, and Conversations made more non-null wrong date assertions. The candidate is not promoted. Completing all 240 native attempts is not the same as completing all 480 reporter calls: 478 were issued, with 2 unreached slots retained as failures.
+
+There are concrete remaining source-present failures. Candidate Conversations misses all 12 preferred-date questions that involve two booking references, despite the unchanged target preference being supplied explicitly. It also misses 7 of 12 unresolved-condition questions and 2 of 12 resolved-condition questions. Across models, some null/clarification misses are contract disagreements, not invented dates; these are counted and described separately. The review packet records the exact source, not just a category label. These outcomes implicate question/source interpretation in this integration, but do not by themselves identify model reasoning, prior actor prose or input layout as the cause.
+
+The actor-stage gap is directly traceable: one successful HTTP response had `finish_reason=length`, no content and no tool calls, so the actor validator returned `invalid_action_response` before the reporting branch. No backend action occurred. A separate no-tools reporting handoff is a reasonable bounded recovery hypothesis; no live gain follows until it is measured. The original episode remains failed and unchanged.
+
+Backend enforcement prevented forbidden state changes in this test. Of 48 required new writes, 47 completed; Conversations missed one legitimate request near the synthetic approval boundary. Conversations made 11 unnecessary attempts, which the host rejected; the base model made none. Frozen server time and synthetic approval identities are experiment controls, not production authentication or real-time expiry validation.
+
+All relevant source information was supplied in these cases. We have not demonstrated a missing-persistence problem or a unique multilingual memory advantage. This is a useful integration finding with a stronger measured base-model result and clear remaining failures. It is not proof of product novelty, coverage of all Sarvam languages, a production-ready system, or improvement on the separate English LongMemEval baseline.

benchmarks/audits/sarvam-fresh-confirmation-v3-2026-10-13/NEXT_STEP.md

@@ -0,0 +1,10 @@
+# Next step after the corrected comparison
+
+**Do not promote the candidate or rerun the current manifest.** The frozen gate failed: two reporter slots were not reached, and Conversations wrong non-null dates increased from 18 to 22. All 240 native episodes and their failures remain in the record.
+
+1. Completed offline: implemented and independently reviewed an opt-in reporting handoff for the exact observed capped-actor class. It must use verified prior source/receipts, execute no new tool, retry no actor call, preserve the cap as an explicit failure marker, and reject uncertain tool outcomes. Commit `b956fb8` and 234 related tests pass; exact capped-input preparation makes no provider call and leaves the old result hash unchanged. Software tests and saved-input replay establish routing integrity only.
+2. Preserve the source-only 108-call evidence-view selection as **blocked and unexecuted**. Its selected capped episode has no candidate reporter request. Do not silently drop it, generate a replacement history, or relax the complete-parent prerequisite. A future comparison would need a new concrete protocol and launch notification under the remaining total research budget.
+3. Before another large run, separate preferred/requested dates from confirmed backend dates and distinguish known from unresolved conditions in a matched source-interpretation test. The present data show a source-present problem, not a proven retrieval or persistent-memory deficit. Keep models fixed, include strong ordinary controls, count added calls/tokens, preserve regressions and define the stopping rule before spending.
+4. Require fresh cases, independent Hindi/Telugu review and a reproducible public benchmark before generalizing or claiming a novel product advantage. Assistant review is sufficient for this explicitly limited development diagnosis, not for broad language-quality claims.
+
+This turn executes only the corrected v3 run. The 108-call screen and any recovery comparison remain unpaid/unexecuted. The English baseline is separate and unchanged.

benchmarks/audits/sarvam-fresh-confirmation-v3-2026-10-13/README.md

@@ -0,0 +1,15 @@
+# Corrected fresh Sarvam comparison v3
+
+Read [VERIFICATION.md](VERIFICATION.md) for the completed clean-export check. Read [REPORT.md](REPORT.md), [FAILURE_REVIEW.md](FAILURE_REVIEW.md) and [NEXT_STEP.md](NEXT_STEP.md). This archive preserves the completed or stopped outcome, all attempted calls and missing outputs; never restart its manifest. The primary gate is unchanged by supplementary metrics. The bank has120related EN/HI/TE renderings of40scenarios and assistant-only language review. It is development research, not a held-out industry benchmark or a demonstrated memory-product advantage.
+
+Verify the archive SHA256 and every extracted file against artifact-manifest.json. Extract to an empty directory. The following commands are offline; they neither read credentials nor call a model. Use new output paths.
+
+```sh
+mkdir /tmp/sarvam-v3-evidence
+tar -xzf benchmarks/audits/sarvam-fresh-confirmation-v3-2026-10-13/evidence.tar.gz -C /tmp/sarvam-v3-evidence
+python -m benchmarks.sarvam_fresh_confirmation.audit /tmp/sarvam-v3-evidence/phase --output /tmp/sarvam-v3-replay.json
+python /tmp/sarvam-v3-evidence/phase/analysis/independent_check.py /tmp/sarvam-v3-evidence/phase /tmp/sarvam-v3-independent.json
+python -m benchmarks.sarvam_fresh_cases_v3 /tmp/sarvam-v3-regenerated
+```
+
+The two audit outputs must equal phase/verification.json and phase/independent-aggregation.json. Generated runtime/gold JSON values must equal the frozen bank. Raw grade changes or interpreting assistant review as independent human validation are not part of this replay. The `unexecuted-screen/` directory preserves the blocked prospective comparison; it contains no live request manifest or results. Supplemental research/prototypes are offline only; no architectural gain follows from software tests.

benchmarks/audits/sarvam-fresh-confirmation-v3-2026-10-13/REPORT.md

@@ -0,0 +1,52 @@
+# Corrected fresh-workflow comparison v3 — October 13, 2026
+
+All 240 native episodes were attempted once. The table counts all 120 episodes per model, including failed or unreached final answers. This is a development comparison of one reporting sentence using fixed model profiles and shared native histories; it is not an English baseline, independent benchmark or new memory result.
+
+The frozen mechanical gate is **FAIL**. The complete-pair requirement is **not met**: 478 reporter calls were issued across 480 planned reporter slots; 2 slots were not reached. No candidate is promoted. Conversations non-null wrong dates also increased from 18 to 22, violating a second frozen requirement. See [INTERPRETATION.md](INTERPRETATION.md) for the gain and failure breakdown.
+
+| Model | Control: correct date + clarification | Candidate | Paired gains / losses |
+|---|---:|---:|---:|
+| sarvam-105b | 90/120 (75.0%) | 105/120 (87.5%) | 20 / 5 |
+| sarvam-105b-conversations | 95/120 (79.2%) | 96/120 (80.0%) | 4 / 3 |
+
+## What the scores do and do not mean
+
+The reporters differ only in their first system sentence. Both share the same original customer records, native actor calls, tool receipts and backend state. The host projects execution status from actual receipts, but never changes the model’s answer or clarification flag. Empty/capped responses fail. Null with clarification=false can preserve the correct absence of a date while still failing the frozen date-plus-clarification contract; date-only totals are shown below.
+
+| Model / arm | Usable final | Date only | Clarification | Raw status | Safe fact + state | Strict task | Non-null date mismatch |
+|---|---:|---:|---:|---:|---:|---:|
+| sarvam-105b / control | 100/120 | 94/120 | 96/120 | 76/120 | 90/120 | 90/120 | 5 |
+| sarvam-105b / candidate | 116/120 | 114/120 | 107/120 | 87/120 | 105/120 | 105/120 | 2 |
+| sarvam-105b-conversations / control | 120/120 | 95/120 | 106/120 | 106/120 | 94/120 | 85/120 | 18 |
+| sarvam-105b-conversations / candidate | 120/120 | 96/120 | 108/120 | 107/120 | 95/120 | 84/120 | 22 |
+
+## Actions, evidence and the blocked follow-up
+
+Across 240 shared native episodes: 48 required new writes, 1 missed required writes, 11 unnecessary update attempts and 0 forbidden state changes. These are shared action outcomes, not gains caused by the reporter sentence.
+
+`confirm-fresh-13-en-m0` returned HTTP 200 with finish_reason `length`, 8,192 completion tokens (8,191 reported reasoning tokens), null content and no tool calls on its first native call. The integration classified this as `invalid_action_response` and never issued either reporter. No state changed. This is not exhaustion of three actor calls. The original result remains unchanged.
+
+The separately proposed 108-call evidence-layout/prose comparison includes that history and requires a complete parent comparison plus an existing candidate request. It cannot pass preparation. It remains **unexecuted**; no case was dropped or substituted, and no gate was weakened. An opt-in recovery route is engineering work only and must be tested prospectively before any accuracy claim.
+
+## Language and failure review
+
+Assistant reviewers checked all 240 native episodes and 480 reporter slots, including absent outputs. Actual outputs and source interpretations were cross-checked against saved records; no independent native-speaking human review was performed. The original v1 had 32 development-exposed episodes. V2 fixed 12 historical/current questions; v3 additionally fixed 12 preference/request questions before execution. Gold dates, sources, backend contracts, actor and reporter prompts were not revised after outcomes. This is 120 renderings of 40 related scenarios from ten inherited families, not 120 independent workflows.
+
+| Model / language | Control correct | Candidate correct |
+|---|---:|---:|
+| sarvam-105b / en | 28/40 | 33/40 |
+| sarvam-105b / hi | 31/40 | 36/40 |
+| sarvam-105b / te | 31/40 | 36/40 |
+| sarvam-105b-conversations / en | 34/40 | 34/40 |
+| sarvam-105b-conversations / hi | 32/40 | 31/40 |
+| sarvam-105b-conversations / te | 29/40 | 31/40 |
+
+All family/language metrics are in `slice-summary.json`; every miss, changed output and raw-status mismatch is enumerated in [FAILURE_REVIEW.md](FAILURE_REVIEW.md). Full language judgments and receipt hashes are in the archive. Related cases limit generalization; no independent-binomial confidence interval is asserted.
+
+## Cost and verification
+
+This phase made 844 calls; known token-cost estimate ₹90.677925, retained amount with headroom ₹113.347725, unknown-usage calls 0. Native research cumulative: 7035 calls, ₹270.657780 known / ₹370.580775 retained under the original total ₹6,000 authority. These are usage estimates at published ₹15/₹60 input/output per million tokens, with cache discounts ignored and 25% headroom; they are not an invoice or balance check.
+
+Primary audit replays all requests, receipts and local SQLite transitions without network calls. A separate raw parser and arithmetic check agrees. Runtime/gold files regenerate from the committed v3 builder. Archive checksums and clean-export verification are published with the evidence. The earlier stopped run and unexecuted v2 remain preserved.
+
+See [NEXT_STEP.md](NEXT_STEP.md) for the bounded next task.

benchmarks/audits/sarvam-fresh-confirmation-v3-2026-10-13/VERIFICATION.md

@@ -0,0 +1,12 @@
+# Clean-export verification
+
+Verified from a fresh `git archive` of commit `058c39f21cb6f6031fd2dc3dd7e431399ae6a15a`, without depending on the original experiment directories. [Machine-readable record](clean-export-verification.json).
+
+- Archive SHA-256: `b0378b68360ca06901e9c0b0d864d1fc4151e739dcc1b44a6c3dfa5545f7dfc4`; all 3,348 extracted file hashes match.
+- Primary request/receipt/SQLite replay exactly reproduces `verification.json`, including the failed gate and missing reporter slots.
+- Separate raw-response parser and accounting reproduce `independent-aggregation.json`.
+- Committed v3 builder regenerates the frozen runtime and gold values.
+- Capped-actor handoff preparation reproduces the saved prospective request hashes without dispatch. The original failed result remains unchanged.
+- All 234 focused Sarvam tests pass from the clean export. No provider request or credential read is part of these checks.
+
+These checks establish reproducibility and software behavior. They do not turn assistant semantic review into independent human review, promote the candidate, establish a recovery accuracy gain, or validate a novel memory product. The subsequent publication commit adds this verification record only; CI checks the final PR head separately.