Audit separated Sarvam finalization and preserve failed transport repair

benchmarks/audits/sarvam-separated-finalization-2026-10-11/README.md

@@ -0,0 +1,17 @@
+# Native Sarvam separated execution/finalization
+
+**Candidate failed; larger288-answer comparison closed with zero calls.**26new requests preserve six integration HTTP400s, an actual pending-approval write in isolated SQLite, and a separate four-call repair. The repair admits requests, but base105B returns no final content; Conversations returns the correct receipt-grounded date in both formats. These are setup diagnostics, not general accuracy or a new memory result. Read [REPORT.md](REPORT.md).
+
+Implementation commits: b73e659 and ee89633. Archive SHA256: `d46971387dc11b8ceeb394e11e31b65d9866c081c20411e61ef142445e40d238`. The artifact manifest lists all190files, including frozen code/protocols, separate ledgers, actual SQLite, raw requests/replies, strict grades, failed gate and closed comparison. No keys, account balance observations or real customer data are included.
+
+From a checkout containing those implementations, extract into an unused directory and replay without paid calls:
+
+```sh
+mkdir /tmp/sarvam-separated-evidence
+tar -xzf benchmarks/audits/sarvam-separated-finalization-2026-10-11/evidence.tar.gz -C /tmp/sarvam-separated-evidence
+PYTHONPATH=. python /tmp/sarvam-separated-evidence/audit_all.py /tmp/sarvam-separated-evidence /tmp/sarvam-separated-replay.json
+```
+
+Use a new output path. Verify extracted hashes against artifact-manifest.json before replay. Network access is disabled by the verifier. It reconstructs26calls, actual state, scores and cumulative accounting; same-code replay is not independent native-language annotation. Prior evidence remains in sarvam-native-study-2026-10-11 and sarvam-structured-control-2026-10-11.
+
+New known estimate INR0.658995; cumulative native research known INR59.345550/retained INR106.439732. Retained includes headroom and full unknown reservations and is not an invoice. Do not run or resume these frozen paid manifests. NEXT_PHASE.md in the archive is a proposal, not completed work. English score and default application runtime are unchanged.

benchmarks/audits/sarvam-separated-finalization-2026-10-11/REPORT.md

@@ -0,0 +1,47 @@
+# Separated Sarvam execution/finalization — October 11, 2026
+
+**The full candidate has not passed. The larger 288-answer comparison remains closed.** We found a request-construction error in our integration, preserved a genuine pending-approval action failure in isolated SQLite, and tested a bounded transport repair. No AgentMem memory architecture, model upgrade or broad accuracy improvement is demonstrated.
+
+## What was tested
+
+Execute native tool actions first, then generate two final answers from exactly the same actual history: ordinary JSON versus strict schema. The only within-pair difference is response_format. Base sarvam-105b stays at low reasoning; sarvam-105b-conversations stays at null. Both have temperature0 and8192 output caps. Shared action calls are charged once in actual spending; each arm's attributed path includes its full shared prefix. This cannot measure independent action-policy improvement.
+
+Seven excluded setup situations ×two models were frozen before calls. A complete14/14candidate pass, correct backend, permitted actions and known usage were required before the larger comparison. Setup stopped after the first8pairs because3pairs encountered infrastructure errors. Six other pairs were never attempted. All histories, receipts, labels and scores remain unchanged.
+
+## Findings, separated by cause
+
+| Observation | Evidence and interpretation |
+|---|---|
+| Six HTTP400 final calls | All occurred in base105B histories containing native tool messages. Our caller omitted tool definitions. API explicitly rejected that contract. This is our integration error, not evidence that the model answered incorrectly. |
+| Pending approval caused an actual write | Base105B changed SETUP-BOOK-203 from2026-11-23/v3 to2026-12-23/v4 despite “Approval is still pending. Do not change yet.” This happened before either final HTTP400. One actual action, shared by two final arms—not two independent violations. Backend intentionally lacked a server approval guard. This is a synthetic application failure, not a claim about Sarvam's production authorization. |
+| Approved and duplicate base actions | The approved action updated real SQLite correctly. The duplicate returned already_applied without another write. Their final requests then failed validation. |
+| One Conversations schema gain | On the already-applied Telugu request, ordinary final output incorrectly said the answer was unknown. Schema returned the recorded date and already_applied. Both outputs were structurally valid. This is one selected setup gain, not a general accuracy result. |
+| Correct denial but strict status miss | Conversations left the Hindi denied booking unchanged and returned its correct date. It used not_requested where the frozen gold expected not_performed. “Only report the current date” makes the distinction ambiguous. Preserve the score, but do not describe this as an unsafe action or factual date error. |
+| Closed-booking receipt miss | Conversations did not attempt the explicitly requested rejected operation. State stayed correct. Schema reported the correct current date; ordinary output said unknown. Both failed the artificial requirement to obtain a rejection receipt. A production system need not perform a pointless closed-booking write merely to satisfy this diagnostic. |
+| Copy and unknown-value controls | Both Conversations final formats copied the Hindi string and correctly represented the missing English telephone as null with clarification. Other model renderings were unattempted. |
+
+Within the incomplete setup, Conversations control passed2/5strict tasks versus schema3/5; base0/3versus0/3because all six final calls failed at the API boundary. These are diagnostic counts, not estimated language/model accuracy. Original gate remains false.
+
+## Separate four-call repair
+
+Sarvam's official guide retains tool definitions after tool execution and documents tool_choice=none. We froze a new four-call diagnostic with those definitions retained, actual role-tagged histories unchanged and no executable backend. We reused one previously successful native action history per model; no action was repeated. Histories differ across models, so comparison is only within each model.
+
+All4requests returned HTTP200 with valid usage and no requested tools. Conversations returned the correct actual confirmed date in both formats (2/2). Base105B returned content=null in both formats (0/2), with finish_reason=stop,32completion tokens and31reported reasoning tokens each, despite the8192cap. No answer was hidden in reasoning_content; that field was null too. The diagnostic therefore fixes HTTP admission but fails complete final-answer compatibility. It is one observed native-history path; do not infer a universal API bug or infer the provider's internal cause.
+
+The two saved databases stayed byte-identical. No final tool call was executed. Original400s were not retried: new request structure, new manifest and separate ledger. The original failed setup and288answer comparison remain closed. We did not keep changing the test until it passed.
+
+## Accounting and verification
+
+- This iteration:26APIrequests,20HTTP200responses,6HTTP400responses,0automatic retries. Of these,22belong to setup and4to the new transport diagnostic.
+- Known usage estimate INR0.658995; retained accounting INR19.255748. The latter includes full reservations for6errors with no usage, plus25%headroom on known usage; it is not an invoice or a claim that those errors were billed.
+- Native-research cumulative:4,974requests; known INR59.345550; retained INR106.439732 against the existing INR6,000ceiling. Earlier rate-limit errors remain in their original ledgers.
+-37focused tests pass; Ruff and diff checks pass. Offline replay reconstructs actual tool traces, SQLite outcomes, requests, receipts, strict grades and metered usage. The repair also verifies that definitions/history were retained, only response_format differs within pairs, and source databases did not change.
+- Assistant-authored synthetic cases; no independent native review. Correlated translated situations and selected successful histories cannot establish population performance, business benefit or novelty. English benchmark scores are unaffected.
+
+See probe/verification.json, probe/gate.json, comparison/CLOSED_UNATTEMPTED.json and transport-repair/verification.json. Source review covers every attempted setup pair and every repair response. Previous reports remain immutable.
+
+## Research decision
+
+The useful result is a narrower diagnosis: formatting, backend permission and final answer content are separate failure surfaces. Strong ordinary controls must be established before claiming a memory product is needed. Next work is specified in NEXT_PHASE.md. The finalizer is not promoted and no broad accuracy claim is made.
+
+Official reference checked October11: https://docs.sarvam.ai/api/api-guides-tutorials/chat-completion/overview . Raw observed behavior takes precedence over assuming the documented interface always succeeds. Published prices used in our estimate remain INR15/Minput and60/Moutput; dashboard invoice has not been reconciled.