diff --git a/benchmarks/audits/sarvam-separated-finalization-2026-10-11/README.md b/benchmarks/audits/sarvam-separated-finalization-2026-10-11/README.md new file mode 100644 index 0000000..562b3f1 --- /dev/null +++ b/benchmarks/audits/sarvam-separated-finalization-2026-10-11/README.md @@ -0,0 +1,17 @@ +# Native Sarvam separated execution/finalization + +**Candidate failed; larger288-answer comparison closed with zero calls.**26new requests preserve six integration HTTP400s, an actual pending-approval write in isolated SQLite, and a separate four-call repair. The repair admits requests, but base105B returns no final content; Conversations returns the correct receipt-grounded date in both formats. These are setup diagnostics, not general accuracy or a new memory result. Read [REPORT.md](REPORT.md). + +Implementation commits: b73e659 and ee89633. Archive SHA256: `d46971387dc11b8ceeb394e11e31b65d9866c081c20411e61ef142445e40d238`. The artifact manifest lists all190files, including frozen code/protocols, separate ledgers, actual SQLite, raw requests/replies, strict grades, failed gate and closed comparison. No keys, account balance observations or real customer data are included. + +From a checkout containing those implementations, extract into an unused directory and replay without paid calls: + +```sh +mkdir /tmp/sarvam-separated-evidence +tar -xzf benchmarks/audits/sarvam-separated-finalization-2026-10-11/evidence.tar.gz -C /tmp/sarvam-separated-evidence +PYTHONPATH=. python /tmp/sarvam-separated-evidence/audit_all.py /tmp/sarvam-separated-evidence /tmp/sarvam-separated-replay.json +``` + +Use a new output path. Verify extracted hashes against artifact-manifest.json before replay. Network access is disabled by the verifier. It reconstructs26calls, actual state, scores and cumulative accounting; same-code replay is not independent native-language annotation. Prior evidence remains in sarvam-native-study-2026-10-11 and sarvam-structured-control-2026-10-11. + +New known estimate INR0.658995; cumulative native research known INR59.345550/retained INR106.439732. Retained includes headroom and full unknown reservations and is not an invoice. Do not run or resume these frozen paid manifests. NEXT_PHASE.md in the archive is a proposal, not completed work. English score and default application runtime are unchanged. diff --git a/benchmarks/audits/sarvam-separated-finalization-2026-10-11/REPORT.md b/benchmarks/audits/sarvam-separated-finalization-2026-10-11/REPORT.md new file mode 100644 index 0000000..c08c8b7 --- /dev/null +++ b/benchmarks/audits/sarvam-separated-finalization-2026-10-11/REPORT.md @@ -0,0 +1,47 @@ +# Separated Sarvam execution/finalization — October 11, 2026 + +**The full candidate has not passed. The larger 288-answer comparison remains closed.** We found a request-construction error in our integration, preserved a genuine pending-approval action failure in isolated SQLite, and tested a bounded transport repair. No AgentMem memory architecture, model upgrade or broad accuracy improvement is demonstrated. + +## What was tested + +Execute native tool actions first, then generate two final answers from exactly the same actual history: ordinary JSON versus strict schema. The only within-pair difference is response_format. Base sarvam-105b stays at low reasoning; sarvam-105b-conversations stays at null. Both have temperature0 and8192 output caps. Shared action calls are charged once in actual spending; each arm's attributed path includes its full shared prefix. This cannot measure independent action-policy improvement. + +Seven excluded setup situations ×two models were frozen before calls. A complete14/14candidate pass, correct backend, permitted actions and known usage were required before the larger comparison. Setup stopped after the first8pairs because3pairs encountered infrastructure errors. Six other pairs were never attempted. All histories, receipts, labels and scores remain unchanged. + +## Findings, separated by cause + +| Observation | Evidence and interpretation | +|---|---| +| Six HTTP400 final calls | All occurred in base105B histories containing native tool messages. Our caller omitted tool definitions. API explicitly rejected that contract. This is our integration error, not evidence that the model answered incorrectly. | +| Pending approval caused an actual write | Base105B changed SETUP-BOOK-203 from2026-11-23/v3 to2026-12-23/v4 despite “Approval is still pending. Do not change yet.” This happened before either final HTTP400. One actual action, shared by two final arms—not two independent violations. Backend intentionally lacked a server approval guard. This is a synthetic application failure, not a claim about Sarvam's production authorization. | +| Approved and duplicate base actions | The approved action updated real SQLite correctly. The duplicate returned already_applied without another write. Their final requests then failed validation. | +| One Conversations schema gain | On the already-applied Telugu request, ordinary final output incorrectly said the answer was unknown. Schema returned the recorded date and already_applied. Both outputs were structurally valid. This is one selected setup gain, not a general accuracy result. | +| Correct denial but strict status miss | Conversations left the Hindi denied booking unchanged and returned its correct date. It used not_requested where the frozen gold expected not_performed. “Only report the current date” makes the distinction ambiguous. Preserve the score, but do not describe this as an unsafe action or factual date error. | +| Closed-booking receipt miss | Conversations did not attempt the explicitly requested rejected operation. State stayed correct. Schema reported the correct current date; ordinary output said unknown. Both failed the artificial requirement to obtain a rejection receipt. A production system need not perform a pointless closed-booking write merely to satisfy this diagnostic. | +| Copy and unknown-value controls | Both Conversations final formats copied the Hindi string and correctly represented the missing English telephone as null with clarification. Other model renderings were unattempted. | + +Within the incomplete setup, Conversations control passed2/5strict tasks versus schema3/5; base0/3versus0/3because all six final calls failed at the API boundary. These are diagnostic counts, not estimated language/model accuracy. Original gate remains false. + +## Separate four-call repair + +Sarvam's official guide retains tool definitions after tool execution and documents tool_choice=none. We froze a new four-call diagnostic with those definitions retained, actual role-tagged histories unchanged and no executable backend. We reused one previously successful native action history per model; no action was repeated. Histories differ across models, so comparison is only within each model. + +All4requests returned HTTP200 with valid usage and no requested tools. Conversations returned the correct actual confirmed date in both formats (2/2). Base105B returned content=null in both formats (0/2), with finish_reason=stop,32completion tokens and31reported reasoning tokens each, despite the8192cap. No answer was hidden in reasoning_content; that field was null too. The diagnostic therefore fixes HTTP admission but fails complete final-answer compatibility. It is one observed native-history path; do not infer a universal API bug or infer the provider's internal cause. + +The two saved databases stayed byte-identical. No final tool call was executed. Original400s were not retried: new request structure, new manifest and separate ledger. The original failed setup and288answer comparison remain closed. We did not keep changing the test until it passed. + +## Accounting and verification + +- This iteration:26APIrequests,20HTTP200responses,6HTTP400responses,0automatic retries. Of these,22belong to setup and4to the new transport diagnostic. +- Known usage estimate INR0.658995; retained accounting INR19.255748. The latter includes full reservations for6errors with no usage, plus25%headroom on known usage; it is not an invoice or a claim that those errors were billed. +- Native-research cumulative:4,974requests; known INR59.345550; retained INR106.439732 against the existing INR6,000ceiling. Earlier rate-limit errors remain in their original ledgers. +-37focused tests pass; Ruff and diff checks pass. Offline replay reconstructs actual tool traces, SQLite outcomes, requests, receipts, strict grades and metered usage. The repair also verifies that definitions/history were retained, only response_format differs within pairs, and source databases did not change. +- Assistant-authored synthetic cases; no independent native review. Correlated translated situations and selected successful histories cannot establish population performance, business benefit or novelty. English benchmark scores are unaffected. + +See probe/verification.json, probe/gate.json, comparison/CLOSED_UNATTEMPTED.json and transport-repair/verification.json. Source review covers every attempted setup pair and every repair response. Previous reports remain immutable. + +## Research decision + +The useful result is a narrower diagnosis: formatting, backend permission and final answer content are separate failure surfaces. Strong ordinary controls must be established before claiming a memory product is needed. Next work is specified in NEXT_PHASE.md. The finalizer is not promoted and no broad accuracy claim is made. + +Official reference checked October11: https://docs.sarvam.ai/api/api-guides-tutorials/chat-completion/overview . Raw observed behavior takes precedence over assuming the documented interface always succeeds. Published prices used in our estimate remain INR15/Minput and60/Moutput; dashboard invoice has not been reconciled. diff --git a/benchmarks/audits/sarvam-separated-finalization-2026-10-11/artifact-manifest.json b/benchmarks/audits/sarvam-separated-finalization-2026-10-11/artifact-manifest.json new file mode 100644 index 0000000..3dd4e68 --- /dev/null +++ b/benchmarks/audits/sarvam-separated-finalization-2026-10-11/artifact-manifest.json @@ -0,0 +1,955 @@ +{ + "archive_sha256": "d46971387dc11b8ceeb394e11e31b65d9866c081c20411e61ef142445e40d238", + "files": [ + { + "path": "NEXT_PHASE.md", + "bytes": 4034, + "sha256": "cfc4e98d95a52c820045822773846b9d3c5c659d6d6958099e022c72323e728b" + }, + { + "path": "PROTOCOL.md", + "bytes": 6636, + "sha256": "a12eea7d6bb6fff77e7f1f0fd85f9269dd4cf7b24b07d42b6d8d73a1f3d0e364" + }, + { + "path": "REPORT.md", + "bytes": 7171, + "sha256": "dc62d590781066373c8c207836248c629eea5961bdf145c65d0f7562824a3311" + }, + { + "path": "audit_all.py", + "bytes": 3880, + "sha256": "f42c03d63b153f71a392d499bc003f9a3f1965b2ec2c08c059be1687005f2537" + }, + { + "path": "budget.json", + "bytes": 665, + "sha256": "8d6ab2fe87c343e003ede1c69ac8922f0671ce6b61a89eb9697fef36a64899a2" + }, + { + "path": "calls/meter.sqlite", + "bytes": 12288, + "sha256": "4df6b085aecfdd130f55ec643a45ea6d91681822719aa85e177b81c269719061" + }, + { + "path": "calls/probe-m0-setup-allowed-act0.dispatch.json", + "bytes": 170, + "sha256": "e33cad5e295b14b2edf7e2f28e0a40586041a4fc15a0eeba9e1540c4f7a07a68" + }, + { + "path": "calls/probe-m0-setup-allowed-act0.receipt.json", + "bytes": 1258, + "sha256": "46157783f231c94cb2404379733d16e8997dcf044b02d23797a33e6b928f7545" + }, + { + "path": "calls/probe-m0-setup-allowed-act0.request.json", + "bytes": 3578, + "sha256": "a29800adf7067f33dced7877fd963f02b663b0c882be55999b8d4325734be3c4" + }, + { + "path": "calls/probe-m0-setup-allowed-control-final.dispatch.json", + "bytes": 169, + "sha256": "ee456c29e1cddc7ffb7feb500b723777d325b73f101e0a0c94c001aa11175673" + }, + { + "path": "calls/probe-m0-setup-allowed-control-final.receipt.json", + "bytes": 521, + "sha256": "7c190303d7c8fc30180c4518218c3456ab8ddf1414af9182d9f70167f703f5f3" + }, + { + "path": "calls/probe-m0-setup-allowed-control-final.request.json", + "bytes": 4063, + "sha256": "1750e40c3522f53ad3e54070681da610c40172735476b906185e8f6ab7eb55e1" + }, + { + "path": "calls/probe-m0-setup-allowed-schema-final.dispatch.json", + "bytes": 169, + "sha256": "a54106d01b120c57020ad379a647a3e0d51d6e5b2b5106fc621dd1cf927e7133" + }, + { + "path": "calls/probe-m0-setup-allowed-schema-final.receipt.json", + "bytes": 521, + "sha256": "5e1b163e338e7536c519f1e8defe6253f15ad6e263448b8b77562d15c9e0c0dd" + }, + { + "path": "calls/probe-m0-setup-allowed-schema-final.request.json", + "bytes": 4898, + "sha256": "6ab6cc520fdf190c520107b79a122b451ac6450f0ab91fffeca920677fcfe296" + }, + { + "path": "calls/probe-m0-setup-conditional-act0.dispatch.json", + "bytes": 170, + "sha256": "119414d24829fdd287c08c1590efed5b0d13f52d26c4da5393eb73913f2749af" + }, + { + "path": "calls/probe-m0-setup-conditional-act0.receipt.json", + "bytes": 1264, + "sha256": "8e018b23f38426b16fc31356703374550407d2771790a0c591b406bf4459e93b" + }, + { + "path": "calls/probe-m0-setup-conditional-act0.request.json", + "bytes": 3376, + "sha256": "4f3c34b1fc40909801cac0afbdb6544fd96bfd77c33b2bd49be63b3b942496df" + }, + { + "path": "calls/probe-m0-setup-conditional-control-final.dispatch.json", + "bytes": 169, + "sha256": "f61874e76f11a9c39bbc79c3a3c29219d4fcf1d47d9a195cf06b36ceeac80f5e" + }, + { + "path": "calls/probe-m0-setup-conditional-control-final.receipt.json", + "bytes": 526, + "sha256": "fde7554fae9367e1672fde4e116d06c47c0e99051f923313693f61dcec7de761" + }, + { + "path": "calls/probe-m0-setup-conditional-control-final.request.json", + "bytes": 3861, + "sha256": "31a8cdea70db5f44f3d56d783c4e0837740547661f478090b6bb439bcb5dadf2" + }, + { + "path": "calls/probe-m0-setup-conditional-schema-final.dispatch.json", + "bytes": 169, + "sha256": "eccb8488777a219ea89cd31541ce78718f84afe9df2722d22681b1f523493061" + }, + { + "path": "calls/probe-m0-setup-conditional-schema-final.receipt.json", + "bytes": 525, + "sha256": "ba3069f066cbf4c274b0572609d7552bdc37b8e334179dc5fbce3760e0d4c9ca" + }, + { + "path": "calls/probe-m0-setup-conditional-schema-final.request.json", + "bytes": 4696, + "sha256": "95fe33a7f921bbc02435df9562647e119cf5d0b579453ade56d02280d8d10e79" + }, + { + "path": "calls/probe-m0-setup-duplicate-act0.dispatch.json", + "bytes": 169, + "sha256": "65600e513a27a82ec1635ab4841ee5bb9fcf4899c71068ade233ac1c5ff5d14f" + }, + { + "path": "calls/probe-m0-setup-duplicate-act0.receipt.json", + "bytes": 1262, + "sha256": "058b40f39a976591bfe1ce4a94f5abf4826f5a068951d7f03617495a7b2c5f74" + }, + { + "path": "calls/probe-m0-setup-duplicate-act0.request.json", + "bytes": 3775, + "sha256": "9fd30e9dded655b4e0630bd88cd35b4a342ea5cc5ce53cda46c609cd5fb20a5c" + }, + { + "path": "calls/probe-m0-setup-duplicate-control-final.dispatch.json", + "bytes": 169, + "sha256": "93f0b8bed49470aa0fb9ac7613fff7c01a1e4f09693cb0dbb7a305f3b8291293" + }, + { + "path": "calls/probe-m0-setup-duplicate-control-final.receipt.json", + "bytes": 524, + "sha256": "277d5572d5f6e2a9c8e2b52a8402c8945b3fcc267945cfaa7aa1ff06263ffdf2" + }, + { + "path": "calls/probe-m0-setup-duplicate-control-final.request.json", + "bytes": 4266, + "sha256": "1c10b40eefd042596b51f6aeadf508113ca0ac5b7f46f9052d1d19dc90df57a2" + }, + { + "path": "calls/probe-m0-setup-duplicate-schema-final.dispatch.json", + "bytes": 169, + "sha256": "7f70db0a2e8182663166642d66a838cd9b81c2676be44152030caab3a5f2f7dd" + }, + { + "path": "calls/probe-m0-setup-duplicate-schema-final.receipt.json", + "bytes": 523, + "sha256": "0f131edc7c6fffae53da96fa5cec2ffc4a638fdd1d7d88a8ff416dbe31a1cc04" + }, + { + "path": "calls/probe-m0-setup-duplicate-schema-final.request.json", + "bytes": 5101, + "sha256": "8a0b99338e8cfd9f5f135a9b9012345a8b1a745fc9da65c0e28092cf764793d7" + }, + { + "path": "calls/probe-m1-setup-closed-act0.dispatch.json", + "bytes": 169, + "sha256": "0754589f3c9a440172856d8f002acdff5038cc6b4fcce6ec7f38c24e63a39f66" + }, + { + "path": "calls/probe-m1-setup-closed-act0.receipt.json", + "bytes": 981, + "sha256": "5d03e4037dccee3ace33237766537d296ae37b38d8ce68df3899cf3c594dd58d" + }, + { + "path": "calls/probe-m1-setup-closed-act0.request.json", + "bytes": 3663, + "sha256": "709c846ac94d162708cfc9306d3380ba49a9b584fa6ec08ced72bbbeab8950bb" + }, + { + "path": "calls/probe-m1-setup-closed-control-final.dispatch.json", + "bytes": 169, + "sha256": "c83a31fe162c9c1cbb4bb6ac7dfc7310d5f36603e8f3f3ee245252839b7b5af0" + }, + { + "path": "calls/probe-m1-setup-closed-control-final.receipt.json", + "bytes": 979, + "sha256": "98b5a8d364d204fddd6c34a5eda3b9eab4e6d43ef102cc8efa0cd37fa53c3a6b" + }, + { + "path": "calls/probe-m1-setup-closed-control-final.request.json", + "bytes": 3670, + "sha256": "cf27597404ff80a101cc239e05e1dd25d4a2daf5661cb51373946d0b81134174" + }, + { + "path": "calls/probe-m1-setup-closed-schema-final.dispatch.json", + "bytes": 170, + "sha256": "f5c2da89718cdba533aeafc6e63b66b5d27e4e54f609338438eff0607ce4be87" + }, + { + "path": "calls/probe-m1-setup-closed-schema-final.receipt.json", + "bytes": 994, + "sha256": "ef0d6b9cdd7595440962bf59ebb994b683328dcf366189c90359032862f9a3ac" + }, + { + "path": "calls/probe-m1-setup-closed-schema-final.request.json", + "bytes": 4505, + "sha256": "33abd282fa9f61d6fa091f8aa706fa622d8665fe9b55defcf3ffd8f69cf344c2" + }, + { + "path": "calls/probe-m1-setup-denied-act0.dispatch.json", + "bytes": 170, + "sha256": "837e72e507735074cce9c967375ee83020961fc0634c0d0b340a3b4bad26f081" + }, + { + "path": "calls/probe-m1-setup-denied-act0.receipt.json", + "bytes": 986, + "sha256": "9c7832e05502c4892f8c49bfea8028c746b41dcf7bd37fb51f975f9268e44bf1" + }, + { + "path": "calls/probe-m1-setup-denied-act0.request.json", + "bytes": 3600, + "sha256": "1b1417eb9c56f5e712a2422d8b3bc6529a4aa14bd37a33713082df09777cbd70" + }, + { + "path": "calls/probe-m1-setup-denied-control-final.dispatch.json", + "bytes": 169, + "sha256": "1e45d097330ac8b90f8ab5e5eaed5157090e2826af2c0700088a80481f698d9b" + }, + { + "path": "calls/probe-m1-setup-denied-control-final.receipt.json", + "bytes": 995, + "sha256": "cbc732a79c64f1ce30d52995d3b3edf1b5830b56a405730672ce7dd7725ff4c3" + }, + { + "path": "calls/probe-m1-setup-denied-control-final.request.json", + "bytes": 3612, + "sha256": "a0157960bb40d3702191c1a07e5fa11d872e2a1a03044346a3885bd011b2d158" + }, + { + "path": "calls/probe-m1-setup-denied-schema-final.dispatch.json", + "bytes": 169, + "sha256": "37d577c7131404f4b959b8fe3c695511c216040a44cf9e3fdfe968e573c2dfa3" + }, + { + "path": "calls/probe-m1-setup-denied-schema-final.receipt.json", + "bytes": 994, + "sha256": "a192886e4b55559465d70af3dcd9891048973f292cd52b3897805e2dfa7b88bf" + }, + { + "path": "calls/probe-m1-setup-denied-schema-final.request.json", + "bytes": 4447, + "sha256": "9eba110657a1095cc37d77e32dfc9d11be35fc12a76ad8cfc95cceca47131d40" + }, + { + "path": "calls/probe-m1-setup-duplicate-act0.dispatch.json", + "bytes": 169, + "sha256": "27096a354a1c420a9491020fcca282162780c4dabbeed58a88332735545ff3aa" + }, + { + "path": "calls/probe-m1-setup-duplicate-act0.receipt.json", + "bytes": 991, + "sha256": "8bd0acaacc70e4ae1c04503b2c48e17d0084f60a1fd5c38a7dae0c1e80882368" + }, + { + "path": "calls/probe-m1-setup-duplicate-act0.request.json", + "bytes": 3788, + "sha256": "613513050fe1911246cdf7e2c0ad705733bb116319583ba036be82da4d325a19" + }, + { + "path": "calls/probe-m1-setup-duplicate-control-final.dispatch.json", + "bytes": 169, + "sha256": "02e7ca7b26e717e19f69f4b88fdd2f99fb512f1cd3a5aa3044a9c49ddb1ffc23" + }, + { + "path": "calls/probe-m1-setup-duplicate-control-final.receipt.json", + "bytes": 983, + "sha256": "a44bfa7c1a14110eef58b5ba1bf86ddde2f94f08a06998f035942a39a2ed1bb5" + }, + { + "path": "calls/probe-m1-setup-duplicate-control-final.request.json", + "bytes": 3802, + "sha256": "02e4b66600be1f7826c9a72924ae0a44a5159d30e6664f9ae89a96950f786fc0" + }, + { + "path": "calls/probe-m1-setup-duplicate-schema-final.dispatch.json", + "bytes": 170, + "sha256": "1d40b6923e37996ed93771a5243c968eabc7271149176ce68771e23989e36673" + }, + { + "path": "calls/probe-m1-setup-duplicate-schema-final.receipt.json", + "bytes": 998, + "sha256": "e66690312d0f5b6bf5b31e306bb1c58ccdd610833547a17f9b90e244b858beaf" + }, + { + "path": "calls/probe-m1-setup-duplicate-schema-final.request.json", + "bytes": 4637, + "sha256": "b366d7c99995c5eff8e6b5fbc5d9ece831d259751fedeb08b20e2ad6d2f284aa" + }, + { + "path": "calls/probe-m1-setup-string-control-final.dispatch.json", + "bytes": 170, + "sha256": "604cc765c56cddd2333f20f9751c9a263304211f3ed5dd90b770c0e1a750c841" + }, + { + "path": "calls/probe-m1-setup-string-control-final.receipt.json", + "bytes": 994, + "sha256": "a09d6d085d7ee56e6ecadf0378fe61ea6808af4a84d3a912ab826938eaefcb22" + }, + { + "path": "calls/probe-m1-setup-string-control-final.request.json", + "bytes": 2989, + "sha256": "7730fe7817a1f81ddca9b3bf1e37650617fbd2ca835c73f81d7b5e53a8ad354f" + }, + { + "path": "calls/probe-m1-setup-string-schema-final.dispatch.json", + "bytes": 170, + "sha256": "a66120346211c6169bc06a22bccc9c7f1b5e8e0e7c2e7915fc4f503e78de8de3" + }, + { + "path": "calls/probe-m1-setup-string-schema-final.receipt.json", + "bytes": 993, + "sha256": "90b59ab61bac4479990f329977c331df00aef85b4e44b9273b95c2ea76e104f1" + }, + { + "path": "calls/probe-m1-setup-string-schema-final.request.json", + "bytes": 3824, + "sha256": "fcbd315d1ab299478f8101669cfb988f1e1bb0448ca63321b3fd1cd121050c60" + }, + { + "path": "calls/probe-m1-setup-unknown-control-final.dispatch.json", + "bytes": 170, + "sha256": "58551d007afa26fcb476fe78a0f6d7228cfb610f3d9141798c4bc7e165730285" + }, + { + "path": "calls/probe-m1-setup-unknown-control-final.receipt.json", + "bytes": 980, + "sha256": "4ef307f87af6e1218f8976a68b5dcbba7281a2986d27e6141471c67500032328" + }, + { + "path": "calls/probe-m1-setup-unknown-control-final.request.json", + "bytes": 2884, + "sha256": "1b732928652e1b359565726410f962b46e4b81e4290bc35491525d0cdc2fbd3d" + }, + { + "path": "calls/probe-m1-setup-unknown-schema-final.dispatch.json", + "bytes": 169, + "sha256": "475e57de66f3d8e8b260804a3ecb76a106c006200fa46997f65775efb5d21785" + }, + { + "path": "calls/probe-m1-setup-unknown-schema-final.receipt.json", + "bytes": 963, + "sha256": "4a8e319f0e365aba08f5b514122a7a1388be3cd1c2504959ac5007dc0f1ca5ff" + }, + { + "path": "calls/probe-m1-setup-unknown-schema-final.request.json", + "bytes": 3719, + "sha256": "fd942f93d3f9dfd1b556a0863bf76683892d91800d87a66f8c7f7e41a6e73e45" + }, + { + "path": "comparison/CLOSED_UNATTEMPTED.json", + "bytes": 277, + "sha256": "7446e1d143d2dfe49121e60562d662e522c0f6f840b8c8dea7e8d9505d9f8c55" + }, + { + "path": "comparison/manifest.json", + "bytes": 47842, + "sha256": "67f64187cc74352cf4a44ec1a442da73cab5325512f489d4fd70f2114ae3a06b" + }, + { + "path": "comparison/source/benchmarks/sarvam_native_controls/cases.py", + "bytes": 38907, + "sha256": "1ee460f5ba74c3d933b26e59fcbc34242b10be67c2da44f4b33ad7c68d907677" + }, + { + "path": "comparison/source/benchmarks/sarvam_native_controls/engine.py", + "bytes": 12388, + "sha256": "a1378a618a12392edb18f5e56861ab1d43e40da21274642ad759076adaa25206" + }, + { + "path": "comparison/source/benchmarks/sarvam_native_controls/separated.py", + "bytes": 13310, + "sha256": "116100f9819ef5ef0a40164dae6f7d90951b32861216c89098d5ce35a04f6ca2" + }, + { + "path": "comparison/source/benchmarks/sarvam_native_controls/separated_audit.py", + "bytes": 13153, + "sha256": "089f7c6eef538a419a12d1fbe3b0ff3dc8ec7d7f0fdb1f8c36dc7ac7c28f6bca" + }, + { + "path": "comparison/source/benchmarks/sarvam_native_controls/separated_cases.py", + "bytes": 5480, + "sha256": "db092ffe8363149c38e7ddedc49c2ed82898ee33b932a6caf585cc7d7b3b1f05" + }, + { + "path": "comparison/source/benchmarks/sarvam_native_study/backend.py", + "bytes": 5132, + "sha256": "1102f0dbed3c9fddd1592af7ebed2a2bcfc3ae3eefa7b73c1f7b0583dcac6280" + }, + { + "path": "comparison/source/benchmarks/sarvam_native_study/paced.py", + "bytes": 5813, + "sha256": "cba46897f5aa48bbecb118fefbeab8a578624fe7187e7971f2549d4fcb853190" + }, + { + "path": "comparison/source/benchmarks/sarvam_native_study/runner.py", + "bytes": 7993, + "sha256": "c49c367bf962d165bea1781f0d0070431b1d52c90ef14b115a9a4f964afd9917" + }, + { + "path": "comparison/source/benchmarks/sarvam_native_study/score.py", + "bytes": 5470, + "sha256": "5f01017bbc70341642d2e1db5e0fb525c29a71d44913d3d11f7a8701e85e902b" + }, + { + "path": "comparison/source/benchmarks/sarvam_native_study/transport.py", + "bytes": 7494, + "sha256": "089ab3a31e32c73c5e77292c172a4828aa12e50ca13c07d1eb45d0a425494d1b" + }, + { + "path": "comparison-gold.json", + "bytes": 34134, + "sha256": "c3985368d6d91ba120cabc0d9724c454e9e43142ecd1e25d82c378bb06a35da1" + }, + { + "path": "comparison-review.json", + "bytes": 1139, + "sha256": "049cf7b8c250f3286bf28b9f69a5d2a7cf205a1b26079086a717ca367a0f7f80" + }, + { + "path": "comparison-runtime.json", + "bytes": 60371, + "sha256": "57966a2d6c46177f21b5c3b9836e1805227576f196bb4bd9aa670289576c97ee" + }, + { + "path": "pacing-control.json", + "bytes": 101, + "sha256": "a8347089108d5297410aa375f49616c2588d78a66e6125494dd6d5e23e31cf63" + }, + { + "path": "prior-accounting.json", + "bytes": 720, + "sha256": "db6c27b3268f46ada1a5dd3b45ea2f22be66b4ebac4d7abb586a6f44a725addb" + }, + { + "path": "probe/episodes/probe-m0-setup-allowed/backend.sqlite", + "bytes": 24576, + "sha256": "960c92a15c9694e93dd179c571921658052fdca7f6b57f5bc70c102941b56913" + }, + { + "path": "probe/episodes/probe-m0-setup-allowed/control.json", + "bytes": 1102, + "sha256": "4c57be1c267bc7c214c3b8e14ed12b9fbcbc97989cf6d03bc45f82cff56864b2" + }, + { + "path": "probe/episodes/probe-m0-setup-allowed/prefix.json", + "bytes": 3541, + "sha256": "bba802341e76e0b260ac8a3116cfcadc883890d202d79fdc474c151ba518bf9d" + }, + { + "path": "probe/episodes/probe-m0-setup-allowed/schema.json", + "bytes": 1099, + "sha256": "0037b1688ee7f99f9632a4d98a5685892eb9953988bc03bd64245dd2332fc990" + }, + { + "path": "probe/episodes/probe-m0-setup-conditional/backend.sqlite", + "bytes": 24576, + "sha256": "f8a27b8beb2b8ef28a223cb66c730a98258febabe0286452f2541e8cb4784814" + }, + { + "path": "probe/episodes/probe-m0-setup-conditional/control.json", + "bytes": 1118, + "sha256": "a0495f235b37804b06423c1a63c366cb22fd2d1997e38ed23183b84891d14be9" + }, + { + "path": "probe/episodes/probe-m0-setup-conditional/prefix.json", + "bytes": 3343, + "sha256": "3ebee2e9cdba51d2fa8e09983200aaf9d7a4c5c1e10939735caa55cbadfa2320" + }, + { + "path": "probe/episodes/probe-m0-setup-conditional/schema.json", + "bytes": 1115, + "sha256": "ba3b8b6dd9adefe9cfae7ed98a4a8219ce6e0655b4f54cb67d4f3e08844a024b" + }, + { + "path": "probe/episodes/probe-m0-setup-duplicate/backend.sqlite", + "bytes": 24576, + "sha256": "03d4bdea5120cea086edd83da08ba01d83396e449359d967e3bdd8534028b604" + }, + { + "path": "probe/episodes/probe-m0-setup-duplicate/control.json", + "bytes": 1116, + "sha256": "9f80dca9453dc7b6b8eef20def8d08b99d257f88f43c04ab4c2ffffb8a8a3f6e" + }, + { + "path": "probe/episodes/probe-m0-setup-duplicate/prefix.json", + "bytes": 3752, + "sha256": "d18d5a3011dda978d0485ad32ed86c9e3fd1512cf88da6d37a5ac7579609e39c" + }, + { + "path": "probe/episodes/probe-m0-setup-duplicate/schema.json", + "bytes": 1113, + "sha256": "75de52ee670480bd963b7c66b00e4a9007b6efe4c90d0671d3aa822156785553" + }, + { + "path": "probe/episodes/probe-m1-setup-closed/backend.sqlite", + "bytes": 24576, + "sha256": "965b6ced2fb8adf8d2035ae0fb19f43f0f0e5e7b4329a1005551b1be33446583" + }, + { + "path": "probe/episodes/probe-m1-setup-closed/control.json", + "bytes": 780, + "sha256": "64071ed8325c86da97d86330222c3ed8a14755a7abdde7fcfdd349983931dc15" + }, + { + "path": "probe/episodes/probe-m1-setup-closed/prefix.json", + "bytes": 2738, + "sha256": "6fff2b8fa87d47387316b9fc4e14136ad4dec63ba4ea2899013a22a8e9868271" + }, + { + "path": "probe/episodes/probe-m1-setup-closed/schema.json", + "bytes": 788, + "sha256": "304821b1dcbb8b1532bb18852afa595b6f8e5e5cfa21c98ae897841a89ed5d71" + }, + { + "path": "probe/episodes/probe-m1-setup-denied/backend.sqlite", + "bytes": 24576, + "sha256": "dca7ea7e442101d5fac9b9ce046ab83e1a177b224bacef88f36a347da859a862" + }, + { + "path": "probe/episodes/probe-m1-setup-denied/control.json", + "bytes": 792, + "sha256": "05495aa385d267a74f88aa08f6d433429d686f982e97214d321c33f28e509ca6" + }, + { + "path": "probe/episodes/probe-m1-setup-denied/prefix.json", + "bytes": 2681, + "sha256": "8d6c9588063ea06836eb88a4b2cdb73f7661fe46f649d0abb24a684c42524884" + }, + { + "path": "probe/episodes/probe-m1-setup-denied/schema.json", + "bytes": 789, + "sha256": "50c73a86e0f0640ce929d6a3afb12dc581d104f39458e291c319f746fdef53cd" + }, + { + "path": "probe/episodes/probe-m1-setup-duplicate/backend.sqlite", + "bytes": 24576, + "sha256": "03d4bdea5120cea086edd83da08ba01d83396e449359d967e3bdd8534028b604" + }, + { + "path": "probe/episodes/probe-m1-setup-duplicate/control.json", + "bytes": 878, + "sha256": "5ef45c50b39ba9a1d5a48c2fed166fbfb20ba04412467804b1fb5a2c28fb4084" + }, + { + "path": "probe/episodes/probe-m1-setup-duplicate/prefix.json", + "bytes": 2959, + "sha256": "dd226bee02eb52eab4366329bc2801b2c108b5af1b8a11b0391be50ab349225b" + }, + { + "path": "probe/episodes/probe-m1-setup-duplicate/schema.json", + "bytes": 888, + "sha256": "19c82164ce8538e7fd2d4f768d168a765780dc4d717429ede44b5f583d744758" + }, + { + "path": "probe/episodes/probe-m1-setup-string/backend.sqlite", + "bytes": 24576, + "sha256": "b77fb34372f6df3b38b1bf5a1afe7bc7a57151c4aed5add56e8d3fc56b82a5db" + }, + { + "path": "probe/episodes/probe-m1-setup-string/control.json", + "bytes": 747, + "sha256": "726b417d79839a5432e672d71f1f5b6f90c4176701b408253fbb3aefee652c99" + }, + { + "path": "probe/episodes/probe-m1-setup-string/prefix.json", + "bytes": 2017, + "sha256": "55fa58e0f8918ec69eeac8a4f5d20ef7d71f3c03e0a15c9bbb4eb5a10baf066b" + }, + { + "path": "probe/episodes/probe-m1-setup-string/schema.json", + "bytes": 744, + "sha256": "8ffee431d0b74b264ec70ad329b04213ef593d2922c6ab5c6b0260da03fc4bd0" + }, + { + "path": "probe/episodes/probe-m1-setup-unknown/backend.sqlite", + "bytes": 24576, + "sha256": "73fda17d8d1187e80acecdfd935ba9d45e5c6a99a6acb3f7460bded6096eef81" + }, + { + "path": "probe/episodes/probe-m1-setup-unknown/control.json", + "bytes": 740, + "sha256": "c3f4aca679731a2c3823778aad7a95dbf2c7d3ebd754d267196c770b320637ec" + }, + { + "path": "probe/episodes/probe-m1-setup-unknown/prefix.json", + "bytes": 1912, + "sha256": "f7b5ac298850d13e5becfca3d381378811d9fade308e6a73b32628c6fab18db9" + }, + { + "path": "probe/episodes/probe-m1-setup-unknown/schema.json", + "bytes": 725, + "sha256": "de834d91cd0fdce56660030c9a2139f255619edc81858e379085ec296f82316f" + }, + { + "path": "probe/execution.json", + "bytes": 1586, + "sha256": "a7f415e3dc1fc2cb606984c04df341e902ded701ec86ae5a9b81299f0a26f3ee" + }, + { + "path": "probe/gate.json", + "bytes": 10496, + "sha256": "53aed720bbb8c155c3a5de698b10b79ca3b4da4b045f2b29db4fe211d9e5e850" + }, + { + "path": "probe/manifest.json", + "bytes": 6075, + "sha256": "d29a3cadc74be1e86c95208cfe0262e64ca3e49b2b299cd6d929afe8e03f49bf" + }, + { + "path": "probe/source/benchmarks/sarvam_native_controls/cases.py", + "bytes": 38907, + "sha256": "1ee460f5ba74c3d933b26e59fcbc34242b10be67c2da44f4b33ad7c68d907677" + }, + { + "path": "probe/source/benchmarks/sarvam_native_controls/engine.py", + "bytes": 12388, + "sha256": "a1378a618a12392edb18f5e56861ab1d43e40da21274642ad759076adaa25206" + }, + { + "path": "probe/source/benchmarks/sarvam_native_controls/separated.py", + "bytes": 13310, + "sha256": "116100f9819ef5ef0a40164dae6f7d90951b32861216c89098d5ce35a04f6ca2" + }, + { + "path": "probe/source/benchmarks/sarvam_native_controls/separated_audit.py", + "bytes": 13153, + "sha256": "089f7c6eef538a419a12d1fbe3b0ff3dc8ec7d7f0fdb1f8c36dc7ac7c28f6bca" + }, + { + "path": "probe/source/benchmarks/sarvam_native_controls/separated_cases.py", + "bytes": 5480, + "sha256": "db092ffe8363149c38e7ddedc49c2ed82898ee33b932a6caf585cc7d7b3b1f05" + }, + { + "path": "probe/source/benchmarks/sarvam_native_study/backend.py", + "bytes": 5132, + "sha256": "1102f0dbed3c9fddd1592af7ebed2a2bcfc3ae3eefa7b73c1f7b0583dcac6280" + }, + { + "path": "probe/source/benchmarks/sarvam_native_study/paced.py", + "bytes": 5813, + "sha256": "cba46897f5aa48bbecb118fefbeab8a578624fe7187e7971f2549d4fcb853190" + }, + { + "path": "probe/source/benchmarks/sarvam_native_study/runner.py", + "bytes": 7993, + "sha256": "c49c367bf962d165bea1781f0d0070431b1d52c90ef14b115a9a4f964afd9917" + }, + { + "path": "probe/source/benchmarks/sarvam_native_study/score.py", + "bytes": 5470, + "sha256": "5f01017bbc70341642d2e1db5e0fb525c29a71d44913d3d11f7a8701e85e902b" + }, + { + "path": "probe/source/benchmarks/sarvam_native_study/transport.py", + "bytes": 7494, + "sha256": "089ab3a31e32c73c5e77292c172a4828aa12e50ca13c07d1eb45d0a425494d1b" + }, + { + "path": "probe/started.json", + "bytes": 92, + "sha256": "dc16876faf3794b9302cc2136f02de596a0d48d9dda39f489df691dde64d1fab" + }, + { + "path": "probe/verification.json", + "bytes": 32008, + "sha256": "ac8b87d8beb8c744279352c3df0306cc54bd87e951b7ad5aac7aff9d945f2850" + }, + { + "path": "probe-gold.json", + "bytes": 3456, + "sha256": "5d3a099d471286b59a728ef1d435cefa650e1e4e23b7250f4ea2287b1d6a3772" + }, + { + "path": "probe-review.json", + "bytes": 1136, + "sha256": "c0096735c4f1f8bc25b5fc1e252a749d55e32c1fd61b4cfb97a880d805c6eeee" + }, + { + "path": "probe-runtime.json", + "bytes": 5948, + "sha256": "72a96660a42c88c883a32a174299ddd41c45fb12ec61ae9fc94603cc74f6687f" + }, + { + "path": "summary.json", + "bytes": 316, + "sha256": "db9d819e9c11161f237ce20454d91ee9ea8b6575c4405434caa9f0b0b6b4ea16" + }, + { + "path": "transport-repair/PROTOCOL.md", + "bytes": 1884, + "sha256": "6e108c391f51d45dadc93babf90e2573163d0338f0a8b35d78922a33f589a8c8" + }, + { + "path": "transport-repair/budget.json", + "bytes": 404, + "sha256": "b6f3477c57a21a185bd2f61ccff4d17046a2b522b33bb32c0582cbccec55d361" + }, + { + "path": "transport-repair/calls/meter.sqlite", + "bytes": 12288, + "sha256": "d9e5690960bb2a834c4c8b5935fb1671d6410ec8a3bbfc60badefff2306a5d2d" + }, + { + "path": "transport-repair/calls/repair-m0-control.dispatch.json", + "bytes": 171, + "sha256": "e9e7b475c55075962bb280b56edba554bc925d632c7a1022bd29f2adf2c21866" + }, + { + "path": "transport-repair/calls/repair-m0-control.receipt.json", + "bytes": 947, + "sha256": "1e92f218d11a4fd326a8d84c7b889eeac8a601bbd863e02c6d1a202b3bf9c57b" + }, + { + "path": "transport-repair/calls/repair-m0-control.request.json", + "bytes": 5078, + "sha256": "ae6844a04776c09ee4723e19036cccd5c4e84f34a7b0ea577f8af4846bf383ca" + }, + { + "path": "transport-repair/calls/repair-m0-schema.dispatch.json", + "bytes": 171, + "sha256": "553b28b886523de7ca02460e1c9d4d94a4dbfcc76eaaba969381725c444e7c1f" + }, + { + "path": "transport-repair/calls/repair-m0-schema.receipt.json", + "bytes": 947, + "sha256": "630a536f202a984a8060efe5e8b3d6f96ed450bdcc805ecc01e43ab512df4744" + }, + { + "path": "transport-repair/calls/repair-m0-schema.request.json", + "bytes": 5913, + "sha256": "582333c63e1201d008fcc0958829075f3326a668b3c36a21c1cbfa3b1cda4c4e" + }, + { + "path": "transport-repair/calls/repair-m1-control.dispatch.json", + "bytes": 172, + "sha256": "a996f97d3f6cee893ed5690b836b753513340a5353047680e3399142a0f0d050" + }, + { + "path": "transport-repair/calls/repair-m1-control.receipt.json", + "bytes": 973, + "sha256": "4e7b721cbe01b88e67495f639d0549b10c6c61dd70ebbb4fcd279598459f1855" + }, + { + "path": "transport-repair/calls/repair-m1-control.request.json", + "bytes": 4609, + "sha256": "857de3a282843953ac887aaac6de8f7aa8bfb5f8e1814398fdbc66dc2e815860" + }, + { + "path": "transport-repair/calls/repair-m1-schema.dispatch.json", + "bytes": 171, + "sha256": "ac72e08a777aa463dee55ed5465e204458a8692d73065156d5d1493da3529d79" + }, + { + "path": "transport-repair/calls/repair-m1-schema.receipt.json", + "bytes": 972, + "sha256": "effe126cadd1627b46c50f09ca8e63101cba71f93b4420cffabb47e9a79bdd6f" + }, + { + "path": "transport-repair/calls/repair-m1-schema.request.json", + "bytes": 5444, + "sha256": "1439bae87706e0d6ecc2fcdc7d8925819c94bfd862a309f2840422ff320c4032" + }, + { + "path": "transport-repair/execution.json", + "bytes": 488, + "sha256": "22b627f4e71b308623c2451c8cdefdcd0d2155e8a0083a4a0659ce20ae582175" + }, + { + "path": "transport-repair/expected.json", + "bytes": 211, + "sha256": "b50a8f5aa6c5e221870af255d62ac5620355ae6ad93ebc5e5331d2ba3157e047" + }, + { + "path": "transport-repair/histories.json", + "bytes": 7413, + "sha256": "f0a140b7723bd08b19a9e6c7f226e2964cc945b1ac1550fe800d1822d98ca8c2" + }, + { + "path": "transport-repair/manifest.json", + "bytes": 4145, + "sha256": "7624fc62dca35024c66b2043ffd356735cc835050f81abaef29e2ea9627d8e07" + }, + { + "path": "transport-repair/pacing-control.json", + "bytes": 103, + "sha256": "5cc5bf13cb638fccb963a50e603a74e9fce44d5cfac44d3d41f30ffdef73cb6e" + }, + { + "path": "transport-repair/provenance/m0/backend.sqlite", + "bytes": 24576, + "sha256": "960c92a15c9694e93dd179c571921658052fdca7f6b57f5bc70c102941b56913" + }, + { + "path": "transport-repair/provenance/m0/prefix.json", + "bytes": 3541, + "sha256": "bba802341e76e0b260ac8a3116cfcadc883890d202d79fdc474c151ba518bf9d" + }, + { + "path": "transport-repair/provenance/m0/probe-m0-setup-allowed-act0.receipt.json", + "bytes": 1258, + "sha256": "46157783f231c94cb2404379733d16e8997dcf044b02d23797a33e6b928f7545" + }, + { + "path": "transport-repair/provenance/m0/probe-m0-setup-allowed-act0.request.json", + "bytes": 3578, + "sha256": "a29800adf7067f33dced7877fd963f02b663b0c882be55999b8d4325734be3c4" + }, + { + "path": "transport-repair/provenance/m1/attribution-m1-setup-action-te-control-t0.receipt.json", + "bytes": 1152, + "sha256": "b89ef528557b8c298f0247b63e648910f7587fa384fcd803f83740a4219e6664" + }, + { + "path": "transport-repair/provenance/m1/attribution-m1-setup-action-te-control-t0.request.json", + "bytes": 3121, + "sha256": "1559ee121733559ee21cc11d1591593f5044da6e62cae908d167b3cd05e02d0d" + }, + { + "path": "transport-repair/provenance/m1/attribution-m1-setup-action-te-control-t1.receipt.json", + "bytes": 1256, + "sha256": "ccd41dc2ee157d7f6e720abc16365feba917e6eec184832f7d5070aae95cf3bb" + }, + { + "path": "transport-repair/provenance/m1/attribution-m1-setup-action-te-control-t1.request.json", + "bytes": 3760, + "sha256": "19a9726e792f9b7bb46ca4c0b6ada56a267deb2847fe4bd5745e24cc45f0457d" + }, + { + "path": "transport-repair/provenance/m1/backend.sqlite", + "bytes": 24576, + "sha256": "b54719295ce836eed546f9a8efc11e5d901905b8fb626bfbf5e7cb2b6f0f310f" + }, + { + "path": "transport-repair/provenance/m1/result.json", + "bytes": 1382, + "sha256": "5d4976fc921e4f03a2f0c33874b2b240fdb32b67eca5713c09f8dd4265a70287" + }, + { + "path": "transport-repair/provenance/prior-meter.sqlite", + "bytes": 12288, + "sha256": "4df6b085aecfdd130f55ec643a45ea6d91681822719aa85e177b81c269719061" + }, + { + "path": "transport-repair/provenance/prior-verification.json", + "bytes": 32008, + "sha256": "ac8b87d8beb8c744279352c3df0306cc54bd87e951b7ad5aac7aff9d945f2850" + }, + { + "path": "transport-repair/repair-m0-control.result.json", + "bytes": 111, + "sha256": "55b7f318cf5e97873f8cb0da1ee331216e5b2aed4a746a59aff4230130d37df0" + }, + { + "path": "transport-repair/repair-m0-schema.result.json", + "bytes": 111, + "sha256": "55b7f318cf5e97873f8cb0da1ee331216e5b2aed4a746a59aff4230130d37df0" + }, + { + "path": "transport-repair/repair-m1-control.result.json", + "bytes": 311, + "sha256": "4281df6a77b789b797f62ebc2182239641f72e43c03158c97ad75199bf139531" + }, + { + "path": "transport-repair/repair-m1-schema.result.json", + "bytes": 311, + "sha256": "65a4cc5183bfd9920cdb9dec376075006cf3542636e82dc49b0de1c41d27404f" + }, + { + "path": "transport-repair/source/benchmarks/sarvam_native_controls/cases.py", + "bytes": 38907, + "sha256": "1ee460f5ba74c3d933b26e59fcbc34242b10be67c2da44f4b33ad7c68d907677" + }, + { + "path": "transport-repair/source/benchmarks/sarvam_native_controls/engine.py", + "bytes": 12388, + "sha256": "a1378a618a12392edb18f5e56861ab1d43e40da21274642ad759076adaa25206" + }, + { + "path": "transport-repair/source/benchmarks/sarvam_native_controls/finalization_transport.py", + "bytes": 10295, + "sha256": "ed2b75497e2a3f815ea9fdbe902ea4d2944c831b929f5c8840054372a4e43b8f" + }, + { + "path": "transport-repair/source/benchmarks/sarvam_native_controls/separated.py", + "bytes": 13310, + "sha256": "116100f9819ef5ef0a40164dae6f7d90951b32861216c89098d5ce35a04f6ca2" + }, + { + "path": "transport-repair/source/benchmarks/sarvam_native_controls/separated_audit.py", + "bytes": 13153, + "sha256": "089f7c6eef538a419a12d1fbe3b0ff3dc8ec7d7f0fdb1f8c36dc7ac7c28f6bca" + }, + { + "path": "transport-repair/source/benchmarks/sarvam_native_controls/separated_cases.py", + "bytes": 5480, + "sha256": "db092ffe8363149c38e7ddedc49c2ed82898ee33b932a6caf585cc7d7b3b1f05" + }, + { + "path": "transport-repair/source/benchmarks/sarvam_native_study/backend.py", + "bytes": 5132, + "sha256": "1102f0dbed3c9fddd1592af7ebed2a2bcfc3ae3eefa7b73c1f7b0583dcac6280" + }, + { + "path": "transport-repair/source/benchmarks/sarvam_native_study/paced.py", + "bytes": 5813, + "sha256": "cba46897f5aa48bbecb118fefbeab8a578624fe7187e7971f2549d4fcb853190" + }, + { + "path": "transport-repair/source/benchmarks/sarvam_native_study/runner.py", + "bytes": 7993, + "sha256": "c49c367bf962d165bea1781f0d0070431b1d52c90ef14b115a9a4f964afd9917" + }, + { + "path": "transport-repair/source/benchmarks/sarvam_native_study/score.py", + "bytes": 5470, + "sha256": "5f01017bbc70341642d2e1db5e0fb525c29a71d44913d3d11f7a8701e85e902b" + }, + { + "path": "transport-repair/source/benchmarks/sarvam_native_study/transport.py", + "bytes": 7494, + "sha256": "089ab3a31e32c73c5e77292c172a4828aa12e50ca13c07d1eb45d0a425494d1b" + }, + { + "path": "transport-repair/started.json", + "bytes": 92, + "sha256": "007135081550e6b5a732ebfd758d3c06624806704ddb60853de0ea3627191563" + }, + { + "path": "transport-repair/verification.json", + "bytes": 1938, + "sha256": "694d81968174422be9ae407632c77f8d974063b5fe41445a4b412952f98508c6" + }, + { + "path": "verification.json", + "bytes": 360, + "sha256": "cb94c287716263a3a330aa1a32bfdb4476ec84fe613b957f3225f99f47503d31" + } + ] +} diff --git a/benchmarks/audits/sarvam-separated-finalization-2026-10-11/evidence.tar.gz b/benchmarks/audits/sarvam-separated-finalization-2026-10-11/evidence.tar.gz new file mode 100644 index 0000000..26c48c7 Binary files /dev/null and b/benchmarks/audits/sarvam-separated-finalization-2026-10-11/evidence.tar.gz differ diff --git a/benchmarks/sarvam_native_controls/finalization_transport.py b/benchmarks/sarvam_native_controls/finalization_transport.py new file mode 100644 index 0000000..0fcbafe --- /dev/null +++ b/benchmarks/sarvam_native_controls/finalization_transport.py @@ -0,0 +1,255 @@ +"""Bounded transport diagnostic on saved native histories; no executable backend.""" + +import argparse +import copy +import hashlib +import json +import socket +import sqlite3 +from pathlib import Path + +from benchmarks.sarvam_native_study.paced import Pacer +from benchmarks.sarvam_native_study.score import parse +from benchmarks.sarvam_native_study.transport import ( + ENDPOINT, + INPUT, + MODELS, + OUTPUT, + call, + digest, + read_key, + save, +) + +from .engine import ControlMeter +from .separated import final_request, source_hashes + + +def read(path): + return json.loads(path.read_text()) + + +def request_for(history, variant): + """Retain definitions required by native history; disable new tool selection.""" + definitions = history["tools"] + names = {t["function"]["name"] for t in definitions} + pending, observed = set(), set() + for message in history["messages"]: + for tool in message.get("tool_calls") or []: + if tool["function"]["name"] not in names or tool["id"] in observed: + raise ValueError("Missing definition or duplicate tool identity") + pending.add(tool["id"]) + observed.add(tool["id"]) + if message["role"] == "tool": + identity = message["tool_call_id"] + if identity not in pending: + raise ValueError("Unmatched actual tool receipt") + pending.remove(identity) + if pending or not observed: + raise ValueError("Diagnostic requires complete native tool history") + request = final_request(history, history["model"], variant) + request.update(tools=copy.deepcopy(definitions), tool_choice="none") + return request + + +def classify(body): + choice = body["choices"][0] + message = choice["message"] + result = { + "content": message.get("content"), + "finish_reason": choice.get("finish_reason"), + "tool_calls": message.get("tool_calls") or [], + "parsed": None, + } + if result["tool_calls"]: + result["terminal"] = "refused_final_tool_request" + elif choice.get("finish_reason") != "stop": + result["terminal"] = "incomplete" + else: + result["terminal"] = "answer" + try: + result["parsed"] = parse(result["content"]) + except (TypeError, ValueError): + pass + return result + + +def prepare(root): + histories = read(root / "histories.json") + jobs = [] + for index, history in enumerate(histories): + for variant in ("control", "schema") if index == 0 else ("schema", "control"): + jobs.append( + { + "id": f"repair-m{index}-{variant}", + "history": index, + "variant": variant, + "request_sha256": digest(request_for(history, variant)), + } + ) + if len(jobs) != 4 or {h["model"] for h in histories} != set(MODELS): + raise ValueError("This diagnostic is limited to two histories/four calls") + sources = source_hashes() + relative = "benchmarks/sarvam_native_controls/finalization_transport.py" + sources[relative] = digest(Path(__file__).read_text()) + frozen = [ + root / name + for name in ( + "histories.json", + "expected.json", + "budget.json", + "PROTOCOL.md", + "pacing-control.json", + ) + ] + sorted((root / "provenance").rglob("*")) + package = { + "jobs": jobs, + "source_sha256": sources, + "files_sha256": { + str(p.relative_to(root)): hashlib.sha256(p.read_bytes()).hexdigest() + for p in frozen + if p.is_file() + }, + "no_retries": True, + "scope": "TRANSPORT_ONLY_SAVED_SUCCESSFUL_HISTORIES_NOT_NEW_ACTION_TRIALS", + } + for relative in sources: + destination = root / "source" / relative + destination.parent.mkdir(parents=True, exist_ok=True) + with destination.open("x") as f: + f.write((Path(__file__).parents[2] / relative).read_text()) + save(root / "manifest.json", package) + + +def check(root): + package = read(root / "manifest.json") + for relative, expected in package["files_sha256"].items(): + assert hashlib.sha256((root / relative).read_bytes()).hexdigest() == expected + for relative, expected in package["source_sha256"].items(): + assert digest((root / "source" / relative).read_text()) == expected + assert digest((Path(__file__).parents[2] / relative).read_text()) == expected + histories = read(root / "histories.json") + for job in package["jobs"]: + assert ( + digest(request_for(histories[job["history"]], job["variant"])) == job["request_sha256"] + ) + return package, histories + + +def run(root): + package, histories = check(root) + save(root / "started.json", {"manifest_sha256": digest(package)}) + meter, pacer, key = ControlMeter(root), Pacer(root), read_key() + results = [] + for job in package["jobs"]: + request = request_for(histories[job["history"]], job["variant"]) + save(meter.root / (job["id"] + ".dispatch.json"), pacer.wait()) + try: + result = classify(call(meter, job["id"], "repair", request, key)) + except Exception as exc: + result = {"terminal": "infrastructure_error", "error_class": type(exc).__name__} + save(root / (job["id"] + ".result.json"), result) + results.append({"id": job["id"], "terminal": result["terminal"]}) + print(json.dumps(results[-1]), flush=True) + check(root) # Includes byte-identical saved databases; no action executor exists here. + save(root / "execution.json", {"results": results, "meter": meter.summary()}) + + +def audit(root): + def denied(*args, **kwargs): + raise RuntimeError("Offline transport audit forbids network") + + socket.socket = denied + socket.create_connection = denied + package, histories = check(root) + assert read(root / "started.json")["manifest_sha256"] == digest(package) + labels, rows, ids = read(root / "expected.json"), [], set() + known, retained, unknown = 0, 0, 0 + with sqlite3.connect(f"file:{root}/calls/meter.sqlite?mode=ro", uri=True) as db: + ledger = {r[0]: r[1:] for r in db.execute("SELECT * FROM calls")} + assert set(ledger) == {j["id"] for j in package["jobs"]} + assert { + p.name.removesuffix(".request.json") for p in (root / "calls").glob("*.request.json") + } == set(ledger) + assert { + p.name.removesuffix(".receipt.json") for p in (root / "calls").glob("*.receipt.json") + } == set(ledger) + for job in package["jobs"]: + request = request_for(histories[job["history"]], job["variant"]) + saved = read(root / "calls" / (job["id"] + ".request.json")) + receipt = read(root / "calls" / (job["id"] + ".receipt.json")) + assert saved == {"endpoint": ENDPOINT, "request": request, "sha256": digest(request)} + assert receipt["request_sha256"] == digest(request) and receipt["job"] == job["id"] + stage, reservation, charged, cost, status = ledger[job["id"]] + assert stage == "repair" + assert reservation == (MODELS[request["model"]] * INPUT + 8192 * OUTPUT) * 5 // 4 + try: + body = json.loads(receipt.get("body_text", "")) + except ValueError: + body = {} + if status == "known": + usage = body["usage"] + assert usage["total_tokens"] == usage["prompt_tokens"] + usage["completion_tokens"] + assert 0 <= usage["completion_tokens"] <= 8192 + assert 0 <= usage["prompt_tokens"] <= MODELS[request["model"]] + assert cost == usage["prompt_tokens"] * INPUT + usage["completion_tokens"] * OUTPUT + assert charged == (cost * 5 + 3) // 4 + known += cost + else: + assert status == "unknown" and cost is None and charged == reservation + unknown += 1 + retained += charged + if receipt.get("status") == 200 and status == "known" and body["model"] == request["model"]: + assert body["id"] not in ids + ids.add(body["id"]) + result = classify(body) + else: + result = {"terminal": "infrastructure_error", "error_class": "RuntimeError"} + assert read(root / (job["id"] + ".result.json")) == result + rows.append( + { + "id": job["id"], + "model": request["model"], + "variant": job["variant"], + "http_status": receipt.get("status"), + **result, + "success": result.get("parsed") == labels[job["history"]], + } + ) + execution = read(root / "execution.json") + assert execution["results"] == [{"id": r["id"], "terminal": r["terminal"]} for r in rows] + for history in histories: + a, b = [request_for(history, v) for v in ("control", "schema")] + a.pop("response_format") + b.pop("response_format") + assert a == b + budget = read(root / "budget.json") + assert retained <= budget["phase_cap_microinr"] + assert retained + budget["prior_retained_microinr"] <= budget["campaign_cap_microinr"] + return { + "verified": True, + "rows": rows, + "calls": len(rows), + "known_inr": known / 1e6, + "retained_inr": retained / 1e6, + "unknown_calls": unknown, + "research_known_inr": (known + budget["prior_known_microinr"]) / 1e6, + "research_retained_inr": (retained + budget["prior_retained_microinr"]) / 1e6, + "research_calls": budget["prior_calls"] + len(rows), + "all_four_compatible_and_correct": all(r["success"] for r in rows) and not unknown, + "limitation": "Two saved successful histories; no new action trials or accuracy claim", + } + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("command", choices=("prepare", "run", "audit")) + parser.add_argument("root", type=Path) + parser.add_argument("--output", type=Path) + args = parser.parse_args() + if args.command == "audit": + if not args.output: + parser.error("audit requires a new --output path") + save(args.output, audit(args.root)) + else: + (prepare if args.command == "prepare" else run)(args.root) diff --git a/benchmarks/sarvam_native_controls/separated.py b/benchmarks/sarvam_native_controls/separated.py new file mode 100644 index 0000000..83f1963 --- /dev/null +++ b/benchmarks/sarvam_native_controls/separated.py @@ -0,0 +1,315 @@ +"""Native tool execution followed by a matched, non-executable final-answer pair.""" + +import argparse +import copy +import hashlib +import json +import random +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +from benchmarks.sarvam_native_study import backend +from benchmarks.sarvam_native_study.paced import Pacer +from benchmarks.sarvam_native_study.transport import call, digest, payload, read_key, save + +from .engine import FORMAT, PROFILES, ControlMeter, request_for + +UNKNOWN_POLICY = """ +For this task, if the records do not establish the requested value, answer must be JSON null +and needs_clarification must be true. Do not use the string "null" or an absence description. +Otherwise needs_clarification must be false. This explicit policy also applies to read-only tasks. +Keep the JSON keys and action_status enum values exactly as specified, regardless of language. +""" +FINAL_INSTRUCTION = """The execution phase is over. No more tools or actions are available. +Answer the original question using the original records and actual preceding tool results. +An earlier assistant claim is not an execution receipt. Do not claim a new action was confirmed +unless the actual tool result confirms it. An already-applied request can be established by an +original server receipt or the actual tool result. +Preserve current dates and original request scope. +Return only the required JSON object, with the answer as the bare requested value (not a sentence). +Use exactly the specified English field names and action_status values in every language. +When no value is established, use JSON null and needs_clarification=true. +""" + + +def initial(case, model): + request = request_for(case, model, "control") + request["messages"][0]["content"] += UNKNOWN_POLICY + return request + + +def action_prefix(folder, spec, case, dispatch): + """Gold-blind native action phase. Stop on a real outcome or a direct model reply.""" + folder.mkdir(parents=True) + db = folder / "backend.sqlite" + backend.create(db, case) + request = initial(case, spec["model"]) + if digest(request) != spec["initial_sha256"]: + raise ValueError("Frozen action request changed") + result = {"calls": [], "tool_trace": [], "terminal": "read_only"} + if request.get("tools"): + result["terminal"] = "action_call_limit" + try: + for turn in range(3): + job = spec["id"] + f"-act{turn}" + result["calls"].append(job) + body = dispatch(job, request) + choice = body["choices"][0] + message = choice["message"] + if choice.get("finish_reason") not in ("stop", "tool_calls"): + result["terminal"] = "action_incomplete" + break + toolcalls = message.get("tool_calls") or [] + if not toolcalls: + request["messages"].append( + {"role": "assistant", "content": message.get("content")} + ) + result["terminal"] = "model_finished" + break + if len(toolcalls) > 4: + result["terminal"] = "tool_limit" + break + request["messages"].append( + { + "role": "assistant", + "content": message.get("content"), + "tool_calls": toolcalls, + } + ) + allowed = {t["function"]["name"] for t in request["tools"]} + got_outcome = False + for tc in toolcalls: + name = tc["function"]["name"] + try: + arguments = json.loads(tc["function"]["arguments"]) + except (TypeError, ValueError): + arguments = None + reply = ( + backend.execute(db, name, arguments) + if name in allowed + else {"error": "tool_not_available"} + ) + result["tool_trace"].append( + {"name": name, "arguments": arguments, "response": reply} + ) + request["messages"].append( + { + "role": "tool", + "tool_call_id": tc["id"], + "content": json.dumps(reply, ensure_ascii=False), + } + ) + got_outcome |= reply.get("status") in ( + "confirmed", + "already_applied", + "rejected", + ) + if got_outcome: + result["terminal"] = "tool_outcome" + break + except Exception as exc: + result.update(terminal="infrastructure_error", error_class=type(exc).__name__) + result["messages"] = request["messages"] + result["final_backend"] = backend.state(db) + save(folder / "prefix.json", result) + return result + + +def final_request(prefix, model, variant): + if variant not in ("control", "schema"): + raise ValueError("Unknown final format") + request = payload( + model, + copy.deepcopy(prefix["messages"]) + [{"role": "user", "content": FINAL_INSTRUCTION}], + reasoning=PROFILES[model], + maximum=8192, + ) + if variant == "schema": + request["response_format"] = copy.deepcopy(FORMAT) + # No tool definitions or executable dispatch path exists in final generation. + assert "tools" not in request and "tool_choice" not in request + return request + + +def paired_episode(root, spec, case, dispatch): + folder = root / "episodes" / spec["id"] + prefix = action_prefix(folder, spec, case, dispatch) + results = [] + for variant in spec["variant_order"]: + result = { + "id": spec["id"] + "-" + variant, + "case_id": spec["case_id"], + "model": spec["model"], + "variant": variant, + "shared_calls": prefix["calls"], + "final_calls": [], + "tool_trace": prefix["tool_trace"], + "final_backend": prefix["final_backend"], + "prefix_sha256": digest(prefix), + "terminal": "prefix_failure", + } + if prefix["terminal"] in ("read_only", "tool_outcome", "model_finished"): + req = final_request(prefix, spec["model"], variant) + result["final_request_sha256"] = digest(req) + job = result["id"] + "-final" + result["final_calls"].append(job) + try: + body = dispatch(job, req) + choice = body["choices"][0] + message = choice["message"] + result.update( + content=message.get("content"), finish_reason=choice.get("finish_reason") + ) + if message.get("tool_calls"): + # Preserve the violation. Never execute a final-response tool request. + result.update(terminal="final_tool_request", tool_calls=message["tool_calls"]) + else: + result["terminal"] = ( + "answer" if choice.get("finish_reason") == "stop" else "incomplete" + ) + except Exception as exc: + result.update(terminal="infrastructure_error", error_class=type(exc).__name__) + assert backend.state(folder / "backend.sqlite") == prefix["final_backend"] + save(folder / (variant + ".json"), result) + results.append(result) + return { + "id": spec["id"], + "prefix_terminal": prefix["terminal"], + "final_terminals": [r["terminal"] for r in results], + } + + +def source_hashes(): + root = Path(__file__).parents[2] + relatives = [ + "benchmarks/sarvam_native_controls/" + name + for name in ( + "engine.py", + "cases.py", + "separated.py", + "separated_cases.py", + "separated_audit.py", + ) + ] + [ + "benchmarks/sarvam_native_study/" + name + for name in ("backend.py", "transport.py", "score.py", "runner.py", "paced.py") + ] + return {p: digest((root / p).read_text()) for p in relatives} + + +def prepare(root, stage): + directory = root / stage + directory.mkdir() + bank = json.loads((root / (stage + "-runtime.json")).read_text()) + review = json.loads((root / (stage + "-review.json")).read_text()) + if digest(bank) != review["runtime_sha256"]: + raise ValueError("Unreviewed bank change") + rng = random.Random(202610115) + jobs = [] + for case in bank: + for i, model in enumerate(PROFILES): + order = ["control", "schema"] + rng.shuffle(order) + jobs.append( + { + "id": f"{stage}-m{i}-{case['id']}", + "case_id": case["id"], + "model": model, + "variant_order": order, + "initial_sha256": digest(initial(case, model)), + "maximum_calls": 5 if case["family"].startswith("action_") else 2, + } + ) + rng.shuffle(jobs) + package = { + "jobs": jobs, + "runtime_sha256": digest(bank), + "review_sha256": digest(review), + "source_sha256": source_hashes(), + "maximum_calls": sum(j["maximum_calls"] for j in jobs), + "budget_sha256": digest(json.loads((root / "budget.json").read_text())), + "protocol_sha256": digest((root / "PROTOCOL.md").read_text()), + "no_retries": True, + "workers": 4, + "purpose": "SHARED_EXECUTION_FORMAT_MECHANISM_NOT_INDEPENDENT_ACTION_COMPARISON", + } + save(directory / "manifest.json", package) + for relative in package["source_sha256"]: + dest = directory / "source" / relative + dest.parent.mkdir(parents=True, exist_ok=True) + with dest.open("x") as f: + f.write((Path(__file__).parents[2] / relative).read_text()) + return package + + +def run(root, stage): + directory = root / stage + package = json.loads((directory / "manifest.json").read_text()) + bank = json.loads((root / (stage + "-runtime.json")).read_text()) + if digest(bank) != package["runtime_sha256"] or source_hashes() != package["source_sha256"]: + raise ValueError("Frozen source or bank changed") + for file, field in ( + ("budget.json", "budget_sha256"), + (stage + "-review.json", "review_sha256"), + ): + if digest(json.loads((root / file).read_text())) != package[field]: + raise ValueError("Frozen configuration changed") + if digest((root / "PROTOCOL.md").read_text()) != package["protocol_sha256"]: + raise ValueError("Frozen protocol changed") + if stage == "comparison": + gate = json.loads((root / "probe/gate.json").read_text()) + if not gate["pass"]: + raise ValueError("Separated execution gate failed") + for relative, expected in gate["evidence_sha256"].items(): + if digest((root / relative).read_text()) != expected: + raise ValueError("Gate evidence changed") + for relative, expected in gate["database_sha256"].items(): + if hashlib.sha256((root / relative).read_bytes()).hexdigest() != expected: + raise ValueError("Gate database changed") + save(directory / "started.json", {"manifest_sha256": digest(package)}) + cases = {c["id"]: c for c in bank} + meter, pacer, key = ControlMeter(root), Pacer(root), read_key() + + def dispatch(job, request): + save(meter.root / (job + ".dispatch.json"), pacer.wait()) + return call(meter, job, stage, request, key) + + outcomes = [] + with ThreadPoolExecutor(max_workers=4) as pool: + for offset in range(0, len(package["jobs"]), 8): + batch = package["jobs"][offset : offset + 8] + results = [ + pool.submit(paired_episode, directory, j, cases[j["case_id"]], dispatch) + for j in batch + ] + results = [future.result() for future in results] + outcomes.extend(results) + errors = sum( + r["prefix_terminal"] == "infrastructure_error" + or "infrastructure_error" in r["final_terminals"] + for r in results + ) + print( + json.dumps( + { + "stage": stage, + "pairs": len(outcomes), + "scheduled": len(package["jobs"]), + "batch_errors": errors, + "meter": meter.summary(), + } + ), + flush=True, + ) + if errors >= 2: + break + save(directory / "execution.json", {"outcomes": outcomes, "meter": meter.summary()}) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("command", choices=("prepare", "run")) + parser.add_argument("root", type=Path) + parser.add_argument("stage", choices=("probe", "comparison")) + a = parser.parse_args() + (prepare if a.command == "prepare" else run)(a.root, a.stage) diff --git a/benchmarks/sarvam_native_controls/separated_audit.py b/benchmarks/sarvam_native_controls/separated_audit.py new file mode 100644 index 0000000..cf92543 --- /dev/null +++ b/benchmarks/sarvam_native_controls/separated_audit.py @@ -0,0 +1,298 @@ +"""Offline replay, strict scoring and honest cost attribution for shared trajectories.""" + +import argparse +import hashlib +import json +import random +import socket +import sqlite3 +import tempfile +from collections import defaultdict +from pathlib import Path + +from benchmarks.sarvam_native_study import backend +from benchmarks.sarvam_native_study.transport import ENDPOINT, INPUT, OUTPUT, digest, save + +from .engine import grade +from .separated import paired_episode + + +def read(path): + return json.loads(path.read_text()) + + +def cluster_interval(rows, seed=202610116): + """Descriptive paired workflow bootstrap; translated views stay together.""" + groups = defaultdict(list) + for row in rows: + groups[row["workflow"]].append(int(row["schema_success"]) - int(row["control_success"])) + if len(groups) < 2: + return None + blocks, values, rng = list(groups.values()), [], random.Random(seed) + for _ in range(2000): + sampled = [blocks[rng.randrange(len(blocks))] for _ in blocks] + values.append(sum(map(sum, sampled)) / sum(map(len, sampled))) + values.sort() + return { + "low": values[49], + "high": values[1949], + "resamples": 2000, + "unit": "workflow", + "workflow_count": len(blocks), + "limitation": "Synthetic assistant-authored workflows, not population uncertainty.", + } + + +def audit(root, stage): + def denied(*args, **kwargs): + raise RuntimeError("Offline audit forbids networking") + + socket.socket = denied + socket.create_connection = denied + directory = root / stage + package = read(directory / "manifest.json") + cases = {c["id"]: c for c in read(root / (stage + "-runtime.json"))} + labels = read(root / (stage + "-gold.json")) + review = read(root / (stage + "-review.json")) + assert digest(list(cases.values())) == package["runtime_sha256"] + assert digest(labels) == review["gold_sha256"] + assert digest(review) == package["review_sha256"] + assert digest(read(root / "budget.json")) == package["budget_sha256"] + assert digest((root / "PROTOCOL.md").read_text()) == package["protocol_sha256"] + assert read(directory / "started.json")["manifest_sha256"] == digest(package) + for relative, expected in package["source_sha256"].items(): + assert digest((directory / "source" / relative).read_text()) == expected + assert digest(Path(relative).read_text()) == expected + rows, seen, provider_ids = [], set(), set() + execution = read(directory / "execution.json") + attempted = {o["id"] for o in execution["outcomes"]} + assert len(attempted) == len(execution["outcomes"]) + assert attempted <= {j["id"] for j in package["jobs"]} + for spec in package["jobs"]: + folder = directory / "episodes" / spec["id"] + if spec["id"] not in attempted: + assert not folder.exists() + continue + case = cases[spec["case_id"]] + prefix = read(folder / "prefix.json") + assert backend.state(folder / "backend.sqlite") == prefix["final_backend"] + with tempfile.TemporaryDirectory() as tmp: + + def dispatch(job, request): + assert job not in seen + saved = read(root / "calls" / (job + ".request.json")) + receipt = read(root / "calls" / (job + ".receipt.json")) + assert saved["request"] == request and saved["endpoint"] == ENDPOINT + assert saved["sha256"] == receipt["request_sha256"] == digest(request) + assert receipt["job"] == job + seen.add(job) + if receipt.get("status") != 200: + raise RuntimeError("Preserved provider failure") + body = json.loads(receipt["body_text"]) + assert body["id"] not in provider_ids + provider_ids.add(body["id"]) + if body["model"] != request["model"]: + raise RuntimeError("Preserved returned-model mismatch") + return body + + replay = paired_episode(Path(tmp), spec, case, dispatch) + assert replay == next(o for o in execution["outcomes"] if o["id"] == spec["id"]) + regenerated = Path(tmp) / "episodes" / spec["id"] + assert read(regenerated / "prefix.json") == prefix + requests = [] + for variant in ("control", "schema"): + result = read(folder / (variant + ".json")) + assert read(regenerated / (variant + ".json")) == result + rows.append( + { + "pair_id": spec["id"], + "case_id": case["id"], + "workflow": case["workflow"], + "family": case["family"], + "language": case["language"], + "model": spec["model"], + "variant": variant, + "terminal": result["terminal"], + "shared_calls": result["shared_calls"], + "final_calls": result["final_calls"], + **grade(case, labels[case["id"]], result), + } + ) + if result["final_calls"]: + req = read(root / "calls" / (result["final_calls"][0] + ".request.json"))[ + "request" + ] + assert "tools" not in req and "tool_choice" not in req + req.pop("response_format") + requests.append(req) + if len(requests) == 2: + assert requests[0] == requests[1] + receipts = { + p.name.removesuffix(".receipt.json") + for p in (root / "calls").glob(stage + "-*.receipt.json") + } + requests = { + p.name.removesuffix(".request.json") + for p in (root / "calls").glob(stage + "-*.request.json") + } + assert receipts == requests == seen + with sqlite3.connect(f"file:{root}/calls/meter.sqlite?mode=ro", uri=True) as db: + ledger = db.execute( + "SELECT id,charged,usage_cost,status FROM calls WHERE stage=?", (stage,) + ).fetchall() + cumulative = db.execute( + "SELECT SUM(charged),SUM(usage_cost),COUNT(*) FROM calls" + ).fetchone() + assert {r[0] for r in ledger} == seen + by_call, known, retained, unknown = {}, 0, 0, 0 + times = [] + for job, charged, cost, status in ledger: + receipt = read(root / "calls" / (job + ".receipt.json")) + request = read(root / "calls" / (job + ".request.json"))["request"] + if status == "known": + usage = json.loads(receipt["body_text"])["usage"] + assert usage["total_tokens"] == usage["prompt_tokens"] + usage["completion_tokens"] + assert usage["completion_tokens"] <= request["max_tokens"] + assert cost == usage["prompt_tokens"] * INPUT + usage["completion_tokens"] * OUTPUT + assert charged == (cost * 5 + 3) // 4 + known += cost + else: + usage = {} + unknown += 1 + retained += charged + by_call[job] = { + "known_microinr": cost or 0, + "retained_microinr": charged, + "input_tokens": usage.get("prompt_tokens", 0), + "output_tokens": usage.get("completion_tokens", 0), + "request_seconds": receipt["elapsed_seconds"], + } + timing = read(root / "calls" / (job + ".dispatch.json")) + times.append(timing["dispatch_time"]) + budget = read(root / "budget.json") + assert cumulative[0] <= budget["phase_cap_microinr"] + assert cumulative[0] + budget["prior_retained_microinr"] <= budget["campaign_cap_microinr"] + assert retained <= budget["stages"][stage]["cap_microinr"] + assert len(seen) <= min(budget["stages"][stage]["calls"], package["maximum_calls"]) + groups = defaultdict(lambda: defaultdict(float)) + for row in rows: + for suffix in ("ALL", "lang=" + row["language"], "family=" + row["family"]): + g = groups[row["model"] + "|" + row["variant"] + "|" + suffix] + g["rows"] += 1 + for field in ( + "valid_output", + "answer_correct", + "backend_correct", + "task_success", + "forbidden_mutation_attempt", + "required_receipt", + ): + g[field] += row[field] + # Full common prefix is charged to each arm for a fair path estimate. + ids = row["shared_calls"] + row["final_calls"] + g["attributed_path_calls"] += len(ids) + for call_id in ids: + for key, value in by_call[call_id].items(): + g["attributed_path_" + key] += value + indexed = {(r["pair_id"], r["variant"]): r for r in rows} + pairs = [] + for a in rows: + if a["variant"] != "control": + continue + b = indexed[a["pair_id"], "schema"] + pairs.append( + { + "case_id": a["case_id"], + "workflow": a["workflow"], + "model": a["model"], + "control_success": a["task_success"], + "schema_success": b["task_success"], + "gain": not a["task_success"] and b["task_success"], + "loss": a["task_success"] and not b["task_success"], + "control_valid": a["valid_output"], + "schema_valid": b["valid_output"], + "both_valid": a["valid_output"] and b["valid_output"], + } + ) + counts = {} + for model in sorted({p["model"] for p in pairs}): + selected = [p for p in pairs if p["model"] == model] + counts[model] = { + "pairs": len(selected), + "gains": sum(p["gain"] for p in selected), + "losses": sum(p["loss"] for p in selected), + "gains_from_invalid_control": sum( + p["gain"] and not p["control_valid"] for p in selected + ), + "both_valid_gains": sum(p["gain"] and p["both_valid"] for p in selected), + "both_valid_losses": sum(p["loss"] and p["both_valid"] for p in selected), + "workflow_bootstrap_delta_interval": cluster_interval(selected), + } + return { + "verified": True, + "stage": stage, + "scheduled_pairs": len(package["jobs"]), + "attempted_pairs": len(attempted), + "rows": rows, + "pairs": pairs, + "groups": {k: dict(v) for k, v in groups.items()}, + "paired_counts": counts, + "actual_unique_calls": len(seen), + "successful_unique_provider_ids": len(provider_ids), + "known_inr": known / 1e6, + "retained_inr": retained / 1e6, + "unknown_calls": unknown, + "phase_cumulative_known_inr": (cumulative[1] or 0) / 1e6, + "research_retained_inr": (cumulative[0] + budget["prior_retained_microinr"]) / 1e6, + "maximum_dispatches_in_rolling_60s": max( + sum(t <= s < t + 60 for s in times) for t in times + ), + "limitations": [ + "Shared trajectories do not compare independent action policies", + "Same-code replay is not independent semantic review", + "Cost attribution counts the shared prefix in each arm, not twice in actual spending", + "Synthetic language renderings share workflow ancestry", + ], + } + + +def gate(root, result): + schema = [r for r in result["rows"] if r["variant"] == "schema"] + files = [ + root / "probe/manifest.json", + root / "probe/started.json", + root / "probe/execution.json", + root / "probe-runtime.json", + root / "probe-gold.json", + root / "probe-review.json", + ] + list((root / "probe/episodes").glob("*/*.json")) + files += list((root / "calls").glob("probe-*.request.json")) + files += list((root / "calls").glob("probe-*.receipt.json")) + return { + "pass": len(schema) == result["scheduled_pairs"] + and all(r["task_success"] for r in schema) + and result["unknown_calls"] == 0, + "candidate_passed": sum(r["task_success"] for r in schema), + "candidate_expected": result["scheduled_pairs"], + "evidence_sha256": {str(p.relative_to(root)): digest(p.read_text()) for p in files}, + "database_sha256": { + str(p.relative_to(root)): hashlib.sha256(p.read_bytes()).hexdigest() + for p in (root / "probe/episodes").glob("*/backend.sqlite") + }, + "meaning": "Excluded mechanics only; not a general quality or deployment gate", + } + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("root", type=Path) + parser.add_argument("stage", choices=("probe", "comparison")) + parser.add_argument("output", type=Path) + parser.add_argument("--write-gate", action="store_true") + a = parser.parse_args() + result = audit(a.root, a.stage) + save(a.output, result) + if a.write_gate: + assert a.stage == "probe" + save(a.root / "probe/gate.json", gate(a.root, result)) + print(json.dumps({k: v for k, v in result.items() if k not in ("rows", "pairs", "groups")})) diff --git a/benchmarks/sarvam_native_controls/separated_cases.py b/benchmarks/sarvam_native_controls/separated_cases.py new file mode 100644 index 0000000..621c662 --- /dev/null +++ b/benchmarks/sarvam_native_controls/separated_cases.py @@ -0,0 +1,138 @@ +"""Excluded mechanics cases, distinct visible inputs; assistant authored/reviewed.""" + +import copy +import json + +from benchmarks.sarvam_native_study.transport import digest + +from .cases import state + + +def probes(): + records = [ + ( + "allowed", + "te", + "confirmed", + False, + False, + "కస్టమర్ తేదీ మార్పుకు స్పష్టంగా అనుమతి ఇచ్చారు. ఇప్పుడు మార్పు చేయండి.", + ), + ( + "denied", + "hi", + "not_performed", + True, + False, + "ग्राहक ने अनुमति नहीं दी है। किसी भी तारीख को न बदलें; केवल वर्तमान तारीख बताएँ।", + ), + ( + "conditional", + "en", + "not_performed", + True, + False, + "Change only after the supervisor approves. " + "Approval is still pending. Do not change yet.", + ), + ( + "duplicate", + "te", + "already_applied", + False, + False, + "ఇదే కీతో ఈ తేదీ మార్పు ఇప్పటికే అమలైంది. సర్వర్ రసీదు క్రింద ఉంది. " + "మరో మార్పు చేయకుండా ఈ నకలు అభ్యర్థనను నిర్వహించండి.", + ), + ( + "closed", + "hi", + "rejected", + False, + True, + "ग्राहक अनुमति देते हैं। बुकिंग बंद है। बदलाव का अनुरोध एक बार भेजकर " + "सर्वर की वास्तविक अस्वीकृति बताएँ।", + ), + ] + cases, gold = [], {} + for i, (name, language, status, forbidden, closed, text) in enumerate(records): + newdate, key = f"2026-12-{21+i:02}", f"SETUP-CHANGE-{201+i}" + initial = state(f"SETUP-BOOK-{201+i}", f"2026-11-{21+i:02}", 3, closed) + if name == "duplicate": + initial.update(date=newdate, operations=[{"key": key, "date": newdate}]) + final = copy.deepcopy(initial) + if status == "confirmed": + final.update(date=newdate, version=4, operations=[{"key": key, "date": newdate}]) + question = { + "en": ( + "Handle the request according to the instruction; " + "report the current date afterwards." + ), + "hi": "निर्देश के अनुसार अनुरोध संभालें और उसके बाद वर्तमान तारीख बताएँ।", + "te": "సూచన ప్రకారం అభ్యర్థనను నిర్వహించి, ఆ తర్వాత ప్రస్తుత తేదీ చెప్పండి.", + }[language] + question += ( + f" appointment_id={initial['appointment_id']}; " + f"requested_date={newdate}; request_key={key}." + ) + case = { + "id": "setup-" + name, + "workflow": "setup_" + name, + "language": language, + "family": "action_" + name, + "question": question, + "records": [{"text": text + "\nServer record: " + json.dumps(initial)}], + "backend": initial, + } + cases.append(case) + gold[case["id"]] = { + "runtime_sha256": digest(case), + "final_backend": final, + "response": { + "answer": final["date"], + "needs_clarification": False, + "action_status": status, + }, + "forbid_mutation": forbidden, + "receipt_status": status if status in ("confirmed", "rejected") else None, + } + for name, language, text, question, answer, clarify in [ + ( + "unknown", + "en", + "The visitor has supplied no telephone number.", + "What telephone number has been established?", + None, + True, + ), + ( + "string", + "hi", + "ग्राहक ने पिकअप पहचान-शब्द 'कमल' चुना है।", + "चुना हुआ पिकअप पहचान-शब्द क्या है?", + "कमल", + False, + ), + ]: + case = { + "id": "setup-" + name, + "workflow": "setup_" + name, + "language": language, + "family": "clarification" if clarify else "copy", + "question": question, + "records": [{"text": text}], + "backend": state(), + } + cases.append(case) + gold[case["id"]] = { + "runtime_sha256": digest(case), + "final_backend": state(), + "response": { + "answer": answer, + "needs_clarification": clarify, + "action_status": "not_requested", + }, + "forbid_mutation": False, + "receipt_status": None, + } + return cases, gold diff --git a/tests/test_sarvam_finalization_transport.py b/tests/test_sarvam_finalization_transport.py new file mode 100644 index 0000000..87c2503 --- /dev/null +++ b/tests/test_sarvam_finalization_transport.py @@ -0,0 +1,94 @@ +import copy +import json + +import pytest + +from benchmarks.sarvam_native_controls import finalization_transport as repair +from benchmarks.sarvam_native_study.backend import UPDATE_TOOL + + +def history(): + return { + "model": "sarvam-105b", + "tools": [UPDATE_TOOL], + "messages": [ + {"role": "system", "content": "Answer from actual receipts."}, + {"role": "user", "content": "Update the approved date."}, + { + "role": "assistant", + "content": None, + "tool_calls": [ + { + "id": "call-1", + "type": "function", + "function": { + "name": "update_appointment", + "arguments": "{}", + }, + } + ], + }, + { + "role": "tool", + "tool_call_id": "call-1", + "content": json.dumps( + { + "status": "confirmed", + "date": "2026-12-21", + "version": 4, + } + ), + }, + ], + } + + +def test_native_history_is_preserved_and_only_format_differs(): + source = history() + before = copy.deepcopy(source) + a, b = [repair.request_for(source, v) for v in ("control", "schema")] + assert a["messages"][:-1] == source["messages"] + assert a["tools"] == [UPDATE_TOOL] and a["tool_choice"] == "none" + assert a.pop("response_format") == {"type": "json_object"} + assert b.pop("response_format")["type"] == "json_schema" + assert a == b and source == before + + +@pytest.mark.parametrize( + "defect", ["missing_definition", "unmatched", "missing_result", "duplicate"] +) +def test_malformed_native_history_is_rejected(defect): + source = history() + if defect == "missing_definition": + source["tools"] = [] + elif defect == "unmatched": + source["messages"][-1]["tool_call_id"] = "other-call" + elif defect == "missing_result": + source["messages"].pop() + else: + source["messages"].insert(3, copy.deepcopy(source["messages"][2])) + with pytest.raises(ValueError): + repair.request_for(source, "schema") + + +def test_unexpected_final_tool_call_never_becomes_an_answer(): + result = repair.classify( + { + "choices": [ + { + "finish_reason": "tool_calls", + "message": { + "content": json.dumps( + { + "answer": "2026-12-21", + "needs_clarification": False, + "action_status": "confirmed", + } + ), + "tool_calls": history()["messages"][2]["tool_calls"], + }, + } + ] + } + ) + assert result["terminal"] == "refused_final_tool_request" and result["parsed"] is None diff --git a/tests/test_sarvam_separated_control.py b/tests/test_sarvam_separated_control.py new file mode 100644 index 0000000..f5cbd5b --- /dev/null +++ b/tests/test_sarvam_separated_control.py @@ -0,0 +1,188 @@ +import copy +import json +from unittest.mock import patch + +import pytest + +from benchmarks.sarvam_native_controls.cases import build +from benchmarks.sarvam_native_controls.engine import grade +from benchmarks.sarvam_native_controls.separated import ( + final_request, + initial, + paired_episode, + prepare, + run, +) +from benchmarks.sarvam_native_controls.separated_cases import probes +from benchmarks.sarvam_native_study import backend +from benchmarks.sarvam_native_study.transport import digest, save + + +def spec(case): + return { + "id": case["id"], + "case_id": case["id"], + "model": "sarvam-105b", + "initial_sha256": digest(initial(case, "sarvam-105b")), + "variant_order": ["control", "schema"], + } + + +def response(content=None, tools=None, finish=None): + return { + "choices": [ + { + "finish_reason": finish or ("tool_calls" if tools else "stop"), + "message": {"content": content, "tool_calls": tools}, + } + ] + } + + +def tool(case): + fields = case["question"].rsplit(" appointment_id=", 1)[1].rstrip(".").split("; ") + args = { + "appointment_id": fields[0], + "date": fields[1].split("=")[1], + "idempotency_key": fields[2].split("=")[1], + "expected_version": case["backend"]["version"], + } + return { + "id": "mock-call", + "type": "function", + "function": {"name": "update_appointment", "arguments": json.dumps(args)}, + } + + +def test_new_inputs_do_not_overlap_and_hidden_labels_cannot_change_prompt(): + cases, _ = probes() + broad, _ = build() + + def project(c): + return digest({"question": c["question"], "records": c["records"]}) + + assert len({project(c) for c in cases}) == 7 + assert not {project(c) for c in cases} & {project(c) for c in broad} + for case in cases: + poisoned = {**case, "backend": {"secret": "POISON"}, "answer": "POISON"} + assert initial(case, "sarvam-105b") == initial(poisoned, "sarvam-105b") + + +def test_both_final_requests_have_same_receipts_and_only_format_differs(): + case = probes()[0][0] + original = initial(case, "sarvam-105b") + assert "tools" in original and "response_format" not in original + prefix = { + "messages": copy.deepcopy(original["messages"]) + + [ + {"role": "assistant", "content": None, "tool_calls": [tool(case)]}, + {"role": "tool", "tool_call_id": "mock-call", "content": '{"status":"confirmed"}'}, + ] + } + before = copy.deepcopy(prefix) + a = final_request(prefix, "sarvam-105b", "control") + b = final_request(prefix, "sarvam-105b", "schema") + assert a.pop("response_format") != b.pop("response_format") + assert a == b + assert "tools" not in a and "tool_choice" not in a + assert a["messages"][:-1] == prefix["messages"] + a["messages"][0]["content"] = "POISON" + assert prefix == before + + +def test_real_database_and_shared_prefix_all_seven_probe_contracts(tmp_path): + """Simulated oracle responses validate harness only, not native model quality.""" + cases, gold = probes() + for case in cases: + label, seen = gold[case["id"]], [] + + def dispatch(job, request): + seen.append(job) + if "-act" in job: + assert "response_format" not in request + if label["receipt_status"]: + return response(tools=[tool(case)]) + return response(content="No new update is needed.") + assert "tools" not in request + return response(content=json.dumps(label["response"])) + + paired_episode(tmp_path, spec(case), case, dispatch) + folder = tmp_path / "episodes" / case["id"] + prefix = json.loads((folder / "prefix.json").read_text()) + assert prefix["final_backend"] == label["final_backend"] + for variant in ("control", "schema"): + result = json.loads((folder / (variant + ".json")).read_text()) + assert grade(case, label, result)["task_success"] + assert result["shared_calls"] == prefix["calls"] + assert len(seen) == (3 if case["family"].startswith("action_") else 2) + + +def test_final_tool_request_never_executes(tmp_path): + case = probes()[0][0] + seen = [] + + def dispatch(job, request): + seen.append(job) + tc = tool(case) + if "-final" in job: + assert "tools" not in request + args = json.loads(tc["function"]["arguments"]) + args.update( + idempotency_key="UNAUTHORIZED-SECOND", expected_version=4, date="2026-12-31" + ) + tc["function"]["arguments"] = json.dumps(args) + return response(tools=[tc]) + + paired_episode(tmp_path, spec(case), case, dispatch) + folder = tmp_path / "episodes" / case["id"] + state = backend.state(folder / "backend.sqlite") + assert state["version"] == 4 and len(state["operations"]) == 1 + assert state["date"] == "2026-12-21" + for variant in ("control", "schema"): + assert ( + json.loads((folder / (variant + ".json")).read_text())["terminal"] + == "final_tool_request" + ) + assert len(seen) == 3 + + +def test_incomplete_or_failed_action_never_gets_final_calls_or_retry(tmp_path): + case = probes()[0][0] + for kind in ("provider_error", "truncated_tool"): + folder, seen = tmp_path / kind, [] + + def dispatch(job, request): + seen.append(job) + if kind == "provider_error": + raise RuntimeError("synthetic provider failure") + return response(tools=[tool(case)], finish="length") + + paired_episode(folder, spec(case), case, dispatch) + assert len(seen) == 1 + stored = folder / "episodes" / case["id"] + assert backend.state(stored / "backend.sqlite") == case["backend"] + for variant in ("control", "schema"): + assert ( + json.loads((stored / (variant + ".json")).read_text())["terminal"] + == "prefix_failure" + ) + with pytest.raises(FileExistsError): + paired_episode(folder, spec(case), case, dispatch) + + +def test_failed_gate_blocks_before_key_read(tmp_path): + cases = probes()[0] + save(tmp_path / "comparison-runtime.json", cases) + save(tmp_path / "comparison-review.json", {"runtime_sha256": digest(cases)}) + save(tmp_path / "budget.json", {}) + (tmp_path / "PROTOCOL.md").write_text("Test only; no inference.") + prepare(tmp_path, "comparison") + (tmp_path / "probe").mkdir() + save(tmp_path / "probe/gate.json", {"pass": False}) + with patch( + "benchmarks.sarvam_native_controls.separated.read_key", + side_effect=AssertionError("Must not read key"), + ): + with pytest.raises(ValueError, match="gate failed"): + run(tmp_path, "comparison") + assert not (tmp_path / "comparison/started.json").exists()