Study Sarvam source interpretation with all 240 histories

README.md

@@ -105,11 +105,11 @@ An earlier n=30 head-to-head against Mem0, Letta, and LangMem lives with its cav
 
 ## Sarvam research: investigating the problem
 
-We test native Sarvam models with ordinary SQLite storage, tool permissions and source transcripts before deciding whether a new memory architecture is needed. English, Hindi and Telugu are diagnostic languages here; they do not establish support across all Sarvam languages.
+We test fixed Sarvam models with ordinary storage and preserved source evidence before deciding whether a new memory product is needed. The latest comparison completed **720 reporter calls across all 240 saved English/Hindi/Telugu histories**, comparing chronology, lossless grouping, and grouping without prior agent prose.
 
-The corrected comparison attempted **240 native episodes** using fixed models and shared histories for two reporter variants. Base105B date-plus-clarification accuracy improved **90/120 → 105/120 (75% → 87.5%)**. Conversations moved **95/120 → 96/120**, but wrong non-null dates increased **18 → 22**. One capped native response prevented both reporters from running; the two missing slots remain failures. **The acceptance gate failed and the candidate is not promoted.** [Report and reproducible evidence](benchmarks/audits/sarvam-fresh-confirmation-v3-2026-10-13/REPORT.md).
+Removing prose moved date-plus-clarification accuracy **104→105/120 on base105B** and **96→103/120 on Conversations** versus the contemporaneous chronological control. Conversations wrong definite dates fell **22→8**, but false abstentions rose **2→9**, raw status accuracy fell **109→97/120**, and the **predeclared acceptance gate failed**. The candidate is not promoted. Paired uncertainty intervals include zero. [Full result and raw replay package](benchmarks/audits/sarvam-source-interpretation-2026-10-13/REPORT.md).
 
-This is a reporting-integration result on related synthetic development cases, with assistant-only language review. It does not demonstrate a new memory advantage or improve the separate English LongMemEval baseline. An evidence-view component and metered comparison harness are implemented, but the proposed 108-call screen remains unexecuted because its complete-parent requirements were not met. A subsequent two-call diagnostic recovered that one skipped English report with the same model and no action retry; both reporter variants were correct. This is one observed recovery, with added compute, and does not rewrite the study scores. [Recovery result](benchmarks/audits/sarvam-capped-recovery-2026-10-13/REPORT.md). Follow the findings, recovery work and problem-selection criteria in [the Sarvam research index](docs/SARVAM_RESEARCH.md). A reviewed public benchmark remains required.
+These are source-present reporting failures and synthetic development results with assistant-only language review. They establish neither a unique multilingual memory advantage nor a newer English LongMemEval score. The earlier corrected v3 study and one-case recovery remain unchanged. The next proposed task is an ordinary question-specific evidence contract separating requested facts, conditional preferences and confirmed state; it is not a completed fix. [Research history and product criteria](docs/SARVAM_RESEARCH.md). A reviewed public multilingual benchmark remains required.
 
 ---
 

benchmarks/audits/sarvam-source-interpretation-2026-10-13/NEXT_STEP.md

@@ -0,0 +1,27 @@
+# Next task — proposed offline work, not another paid run
+
+The 720-call phase is closed. C improves some factual counts but fails the frozen reliability gate; do not promote it, retry failures, replace cases, or run another wording/prose-removal sweep. Historical v3, the blocked 108-call screen and the one-case recovery remain unchanged.
+
+## The narrower problem to solve
+
+The same record can contain a customer's requested date, a condition-dependent preference and the backend's confirmed date. Those facts can legitimately differ. The current integration sometimes picks the wrong fact, or abstains despite a known answer. Removing all prior prose helps some questions and harms others. Original evidence is present, so adding persistence alone is not justified by these results.
+
+Prepare an ordinary, explicit evidence contract before inventing a new memory architecture:
+
+1. Define separate records for a user assertion/request/preference, its booking reference and any correction or unresolved condition. Keep exact source spans and references. Keep confirmed state and action receipts in their own fields.
+2. Define a query plan that identifies the requested fact and booking from the actual user question. No case IDs, family tags, gold targets or per-language case exceptions may enter the runtime. Do not assume the model can extract this plan correctly; extraction failures must count.
+3. Require any proposed answer to identify its supporting record and any condition it depends on. A quote match checks provenance only; semantic support must still be tested. A missing condition can justify clarification, but lack of execution approval does not erase a known customer preference.
+4. Keep authoritative action status owned by the existing receipt layer. Do not use model prose as proof of a write, and do not claim this existing guard as novel.
+5. Create an offline case-to-contract review that includes every observed gain, loss, cap and no-treatment mismatch. Prove source preservation and expose ambiguity before an implementation or paid request is approved as ready.
+
+This is a proposed design and test specification, not a completed feature or predicted gain. Use simple typed storage/source scoping as a strong ordinary control. Current [Mem0, Zep/Graphiti and Supermemory documentation](PRIOR_ART_UPDATE.md) already covers related temporal, source and schema capabilities. We have not tested these competitors on this dataset or established a feature absent from them.
+
+## Before a future paid comparison
+
+- Freeze a candidate and a contemporaneous ordinary control with the same Sarvam profiles. If extraction/verification adds calls, provide an equal-call/equal-budget control; show actual tokens, latency and cost separately.
+- Use fresh workflow instances with independent wording and source review, including known and unknown conditions, clear corrections, multiple bookings and contradictory assistant claims. Do not call reused templates an independent market sample. Seek native human review before a public language-quality claim; current assistant-only review remains explicitly limited.
+- Predeclare date-plus-clarification, false abstention, wrong definite dates, incomplete outputs and receipt-owned status, with paired gains/losses and scenario-level uncertainty. A net score rise cannot hide new reliability failures.
+- Publish an auditable benchmark package as required by the founder, but distinguish this development bank from a validated public benchmark. Expand language coverage by evidence and review capacity, rather than claiming all-language support from Hindi/Telugu.
+- Reproduce a useful gain on fresh cases, then test an actual user workflow at acceptable latency. Only then decide whether the method merits a product or paper claim.
+
+No further paid run was launched after this 720-call phase. The next paid package must be concrete and notified before dispatch under the standing total budget. The founder does not need to provide another key. No external outreach or daily automation is authorized by this proposal.

benchmarks/audits/sarvam-source-interpretation-2026-10-13/PRIOR_ART_UPDATE.md

@@ -0,0 +1,16 @@
+# Product differentiation check — October 13, 2026
+
+Checked October 14 UTC, October 13 America/New_York, while the frozen 720-call campaign was running. This targeted primary-source check does not change the protocol, inputs, acceptance rules or paid requests. It supplements the earlier product decision frame; it is not an exhaustive literature review or a competitor evaluation.
+
+| Primary source | Documented capability | Consequence for our claim |
+| --- | --- | --- |
+| [Mem0 temporal reasoning](https://mem0.ai/blog/introducing-temporal-reasoning-in-mem0), updated September 28, 2026 | Temporal enrichment, distinctions between plans and states, evolving state keys, and time-aware retrieval. | Storing dates or distinguishing an old fact from a current one is already established product work. |
+| [Zep Context Graph overview](https://help.getzep.com/graph-overview), current v3 documentation | Facts retain source-episode links and validity times. The documentation explicitly distinguishes provenance from correctness. | Source preservation alone is neither a new feature nor proof of reliable interpretation. |
+| [Graphiti custom entity and edge types](https://help.getzep.com/graphiti/core-concepts/custom-entity-and-edge-types) | Domain-specific schemas for extracted entities and relationships. | A typed fact representation is an ordinary baseline we must compare against. |
+| [Supermemory relationship semantics](https://supermemory.ai/blog/memory-graph-relationship-semantics/), July 23, 2026 | Distinguishes updates, extensions and uncertain inferences, with source and authority considerations. | Separating a derived belief from an authoritative account state is also prior art. |
+
+These are vendor descriptions of capabilities, not measured results on our cases. We have not run these competing systems, established equal-compute comparisons with them, or established their Hindi/Telugu accuracy. Missing documentation would not establish that a capability is absent.
+
+The credible near-term output is a reproducible failure analysis and, only if measured, a bounded improvement with fixed Sarvam models. Product differentiation requires a fresh, independently reviewed comparison showing that our method prevents meaningful user-facing errors better than ordinary source scoping, typed records and existing frameworks at an acceptable cost and latency. A benchmark contribution also requires dataset quality, public artifacts, clear denominators and an honest account of development exposure; it does not require pretending that the current intervention succeeded.
+
+The present test can discriminate between these specific evidence-presentation choices. It cannot establish a missing persistent-memory layer because the relevant original evidence is already supplied. The final product decision must follow the complete results and their regressions.

benchmarks/audits/sarvam-source-interpretation-2026-10-13/PROTOCOL.md

@@ -0,0 +1,44 @@
+# Prospective source-interpretation comparison — October 13, 2026 (New York)
+
+Status: preparing before any new response. User authorizes the next proposed phase under the original total INR6,000 research ceiling. New scope; do not resume, alter or replace the closed v3 cohort, capped recovery diagnostic, or blocked 108-call screen.
+
+## Question and fixed intervention
+
+Does hiding the actor's prior prose, while preserving original source evidence and actual execution receipts, improve the final answer? Primary intervention is C versus B. Secondary organization comparison is B versus A; total operational comparison is C versus A. Report all three, not just the winner. The preselected forward candidate is C. Positive B-only results remain exploratory and require a separately declared confirmation; no retroactive primary change.
+
+Use every one of the 240 saved v3 native histories: 40 related source scenarios, rendered in English/Hindi/Telugu, for each of two fixed Sarvam profiles. For 239 histories use the saved candidate reporter request and actual tool trace. For confirm-fresh-13-en-m0 reconstruct from the original actor request and capped native receipt using the reviewed handoff helper; verify no action, unchanged backend state and exact scope. Do not use either new diagnostic answer as source evidence or impute an old cohort result.
+
+A: ordinary chronological evidence, complete, under the common new reporter instruction. This is a fresh control, not the unchanged v3 reporter prompt or a reused historical answer. B: lossless source/execution/actor/protocol groups, mechanically reconstructable to A's evidence. C: B with only nonempty actor content/reasoning_content/refusal strings omitted. Keep exact source text, references, dates, conditions, question, scope, tool proposals and actual receipts. Keep a local omission ledger; do not send hidden labels, review annotations, gold, inferred facts or the duplicate audit sidecar. No native actor call, tool execution, backend update or answer extraction is permitted in this phase.
+
+All arms use the same reporter instruction/schema: existing sarvam_evidence_ablation.SYSTEM. Sarvam-105B reasoning low; Sarvam-105B-Conversations reasoning null; temperature 0; max_tokens 8192; one request per history/arm. Same call counts and ceilings, not equal actual tokens. C intentionally removes actor prose strings; actor tool proposals remain. Do not attribute an effect specifically to authority confusion rather than compression/context change; this experiment cannot distinguish those mechanisms.
+
+## Pre-response source census and allocation
+
+Independent source checks found nonempty omissions in 184/240 histories: base 98/120, Conversations 86/120. Remaining 56 are B=C byte-identical inputs, including the capped episode. Keep them in all-episode results and explicitly report them as a negative-control/no-treatment slice. Differences there cannot demonstrate a removal effect. The treated slice is predefined solely from input content before new outputs, not by correctness. Verify and freeze exact IDs.
+
+720 prospective slots: 3 per history. Assign each scenario's six language/model histories the six arm permutations once, rotating across the 40 scenarios. Seed 202610134 for deterministic history ordering; keep each history's assigned arm order. Audit planned arm-position counts by model/language/family. Launch at most four requests per batch with four workers, globally paced at 90 RPM (user-reported account limit 120). Concurrency means observed dispatch order within a batch can vary; retain local pacer-release timestamps and report allocation versus locally observed order separately. These timestamps precede request reservation/transport and do not establish provider receipt order. Do not change order after responses.
+
+## Metrics and advancement criterion
+
+Primary metric: exact source-target date AND needs_clarification, with invalid, missing, refused or truncated finals counted as failures in the planned denominator. Sealed existing labels are used only in offline scoring; preparation and execution may copy/hash opaque label bytes. Independent raw-response aggregation and assistant source review must verify results. Human/native independent review is absent.
+
+Primary treatment effect: paired C minus B on the 184 treated histories, reported separately for each model (98 and 86). Also report all-240 C-B, A-B and A-C, and the 56 no-treatment slice. For each show correct counts, gains/losses, valid-output recovery versus fixes between usable answers, wrong non-null dates, raw action-status accuracy, receipt-projected status accuracy, fully correct reports, unknown-condition guesses, false abstentions (valid output with null answer or clarification=true when the source target is determinate), invalid outputs, language/family slices, input/output tokens, cost and request latency.
+
+Screen passes for a subsequent fresh confirmation only if all conditions hold:
+1. All720 slots attempted once with complete auditable receipts, no infrastructure error or unknown usage; scoring/source integrity verified. Model output caps remain accuracy failures, not excluded rows.
+2. Treated C-B date+clarification gain is strictly positive on each fixed model and at least two distinct workflow IDs per model have a positive net treated gain across their translations.
+3. All-episode C-A date+clarification gain is strictly positive on each fixed model.
+4. On all episodes for each model, C has no more wrong non-null dates or false abstentions on determinate targets than either A or B, and neither raw nor receipt-projected status accuracy is lower than either control. Invalid finals are reported separately and remain primary-metric failures.
+5. Independent semantic review finds no unresolved source/label ambiguity affecting these comparisons. Preserve original scores and publish any sensitivity calculation separately; do not repair gold after seeing results to pass.
+
+This is a development screening gate, not production, statistical-significance or novelty certification. Report 40 scenario clusters, each containing related language/model observations. Publish fixed-seed 10,000-draw paired scenario-cluster bootstrap percentile95% intervals for primary and operational contrasts, with seed202610134; repeated languages/model/arms are not independent samples. For each model, resample40workflow IDs with replacement; primary treated C-B rate is sum(paired factual deltas)/sum(treated histories) in the sampled clusters, while all-history C-A rate uses sum(paired factual deltas)/sum(all histories). Report intervals as percentage-point differences, with no unplanned weighting. If a bootstrap draw has no treated histories, omit that draw and report its count (expected zero here). Do not convert a positive point estimate with an interval crossing zero into conclusive efficacy. Report model-specific clusters and whole-study clusters without pooling models into a primary score.
+
+## Meter and stop rules
+
+Original research carry: 7,037 calls, INR271.224630 known token-cost estimate and INR371.289339 retained with headroom/unknown reservations. New phase maximum720 calls, INR1,550 phase cap, full-context worst-case reservation INR1,548.288 (360 base at3.072 +360 Conversations at1.2288). Campaign total retained ceiling remains INR6,000. Prices rechecked October14UTC/October13NewYork: input15/output60 INR per million for both profiles; ignore cache discounts conservatively. Token estimates are not invoice verification.
+
+Freeze code, source snapshots, protocol, budgets, pacing, all inputs, request hashes, omission ledgers, order and prior carry. Independent preflight plus focused tests before launch. Durable exclusive started marker and per-call meter reservation before dispatch. No request retries or manifest restart. On at least2 cumulative infrastructure errors, finish already started batch calls then stop before any new batch. Budget/source integrity failure stops immediately; retain already started receipts. Known cap/invalid model responses do not trigger infrastructure stop and remain failures. Retain all720 planned denominators if early stop; a stopped campaign cannot pass. Root notifies user before paid dispatch, then runs exactly this campaign once and stops regardless of result. No auto-repeat or second paid phase.
+
+## Product interpretation
+
+A gain would support a tested reporting-boundary improvement under these fixed models and synthetic development workflows. It would not prove long-term memory is necessary, that the original AgentMem English architecture is multilingual, that Sarvam lacks ordinary storage/approval tools, or that grouping/provenance is novel. A negative result rejects these interventions on this bank; it does not identify the underlying cause. Stronger ordinary controls, fresh tasks, independent language review, broader languages, action-level and cost/latency validation are required before a unique-product claim. This phase should decide whether there is enough measured benefit to warrant that work.