diff --git a/README.md b/README.md index dfe4ab5..0ffe01d 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # AgentMem OS -**Memory for AI agents, in any language.** +**Local-first memory for AI agents, with measured English results and multilingual research.** [![CI](https://github.com/Sahith59/AgentMem-OS/actions/workflows/ci.yml/badge.svg)](https://github.com/Sahith59/AgentMem-OS/actions/workflows/ci.yml) @@ -11,10 +11,10 @@ An open-source, local-first memory engine for LLM agents. The latest development | Question | Answer | |---|---| | How good is it, really? | **[docs/RESULTS.md](docs/RESULTS.md)**: current 84.6% single run, 83.6% repeated predecessor, and qualified historical results | -| Every number in one table? | **[docs/RESULTS.md](docs/RESULTS.md)**: every run ever, with harness, models, tokens, and artifact | +| Every number in one table? | **[docs/RESULTS.md](docs/RESULTS.md)**: English runs; **[Sarvam research](docs/SARVAM_RESEARCH.md)**: multilingual studies and evidence | | What failed on the way? | **[docs/FAILURES.md](docs/FAILURES.md)**: every refuted idea, what it cost, what it bought | | Why is it built this way? | **[docs/DECISIONS.md](docs/DECISIONS.md)**: each decision with its measured outcome, good and bad | -| What about Indian languages? | **[docs/INDIC_ROADMAP.md](docs/INDIC_ROADMAP.md)**: the cross-lingual layer, measured numbers, the benchmark nobody has built | +| What about Indian languages? | **[docs/SARVAM_RESEARCH.md](docs/SARVAM_RESEARCH.md)**: current evidence, unresolved problems and the [Indic roadmap](docs/INDIC_ROADMAP.md) | | Can I trust vendor benchmark claims? | **[COMPETITIVE_ANALYSIS.md](COMPETITIVE_ANALYSIS.md)**: sourced claims, and why most published numbers are not comparable | | How do I run it? | [Quickstart](#quickstart): local, free, no API key | @@ -69,10 +69,10 @@ Verbatim conversation evidence stays primary. A local 8B model proposes facts fr - **A benchmark record that includes failures.** We disclose the answerer, judge, split, memory source, context budget, repeat count and negative experiments. Single-run results are labelled as such. See [docs/RESULTS.md](docs/RESULTS.md) and [docs/FAILURES.md](docs/FAILURES.md). - **Cross-lingual identity under evaluation.** An earlier hand-labelled EN/Hindi/Tamil alias set measured precision 0.762 and recall 0.533 at one operating point. That does not establish end-to-end Hindi/Telugu memory accuracy or meet the later 0.95 automatic-merge precision target. See [docs/INDIC_ROADMAP.md](docs/INDIC_ROADMAP.md). -- **Extraction that cannot hallucinate silently.** The LLM proposes; deterministic validators decide, with logged rejection reasons. 19,195 sessions extracted into 98,372 validated facts at $0 API cost. +- **Extraction with deterministic checks.** The LLM proposes facts; validators log acceptance/rejection. Historical extraction processed 19,195 sessions into 98,372 accepted facts at $0 API cost. Passing these checks does not prove that every fact is supported by its source. - **Dynamic trust, not static tiers.** Trust is a live number updated from evidence. Measured in an adversarial harness: retrieval precision 0.951 with trust-weighting versus 0.625 without, and an unreliable agent's perceived trust decays 0.50 to 0.27 automatically. -- **Fork, not just share.** Child agents inherit abstracted knowledge and start with a clean episodic slate: the first formalization of git-style memory branching for LLM agents. -- **A temporal knowledge graph that doesn't lie about the past.** Bi-temporal facts (`valid_from` / `valid_until`), deterministic zero-LLM-call supersession, point-in-time queries. +- **Fork, not just share.** Child agents inherit abstracted knowledge and start with a clean episodic slate. This describes the implementation, not a claim of first invention. +- **A temporal knowledge graph with validity intervals.** Bi-temporal facts (`valid_from` / `valid_until`), deterministic zero-LLM-call supersession, point-in-time queries. - **100% local-first.** Every tier runs on your machine. Plug in Claude, GPT, or a fully local Ollama model interchangeably. --- @@ -83,12 +83,12 @@ Verbatim conversation evidence stays primary. A local 8B model proposes facts fr | Configuration | QA accuracy | Mean context sent | |---|---|---| -| Precision-source pipeline, current | **84.6%** (423/500, one historical run; new judge protocol pending) | 40k-character cap; mean tokens not reported | +| Precision-source pipeline, current | **84.6%** (423/500, one historical run; historical judge) | 40k-character cap; mean tokens not reported | | Prior revised pipeline | **83.6%** (418/500 in each of two repeats) | 40k-character cap; mean tokens not reported | | Historical full-evidence Luna series | 80.0% ± 0.5 (three 500-question runs) | ~8.5k tokens | | AgentMem OS, 24k operating point | 76.9% ± 1.0 (n=150, mean of 3 runs) | 5,698 tokens | -The English rows use different corpus and retrieval configurations. The latest measured answerer was `gpt-5.6-luna`, with the benchmark's type-specific `gpt-4o` judge. GPT-4o-family inference is now retired for future runs; those historical grades remain unchanged. The affordable replacement judge and new scoring series require calibration. See the [Luna foundation plan](docs/ENGLISH_LUNA_FOUNDATION_2026-10-04.md). See [docs/RESULTS.md](docs/RESULTS.md) for repeat counts and historical comparability limits. A 90% result has not been measured. +The English rows use different corpus and retrieval configurations. The latest measured answerer was `gpt-5.6-luna`, with the benchmark's type-specific `gpt-4o` judge. GPT-4o-family inference is now retired for future runs; those historical grades remain unchanged. A completed provisional Terra regrade scores 425/500 on those same saved answers, with 484/500 grade agreement. No answers changed, so this is a judge-scale difference, not an architecture gain. [Bridge evidence](docs/results/terra-bridge-2026-10-04.json) records the disagreements and assistant-review limits. No newer English answer improvement is established by this Sarvam work. See the [Luna foundation plan](docs/ENGLISH_LUNA_FOUNDATION_2026-10-04.md). See [docs/RESULTS.md](docs/RESULTS.md) for repeat counts and historical comparability limits. A 90% result has not been measured. **Why evidence delivery matters:** historical session-coverage analysis found a large association with answer accuracy, but a session hit does not prove that its answer-bearing turn or all operands reached the packet. Later full500 audits separate retrieval gaps from answer selection, abstention and judge sensitivity. See [docs/BENCHMARKS.md](docs/BENCHMARKS.md#the-coverage-finding-the-mechanism-behind-everything) for the original analysis and [docs/RESULTS.md](docs/RESULTS.md) for the qualified current result. @@ -103,6 +103,14 @@ The English rows use different corpus and retrieval configurations. The latest m An earlier n=30 head-to-head against Mem0, Letta, and LangMem lives with its caveats in [docs/BENCHMARKS.md](docs/BENCHMARKS.md). Historical per-question artifacts are in [`benchmarks/`](benchmarks/); later frozen-run receipts are retained in the project's evaluation memory. +## Sarvam research: investigating the problem + +We test native Sarvam models with ordinary SQLite storage, tool permissions and source transcripts before deciding whether a new memory architecture is needed. English, Hindi and Telugu are the current diagnostic languages; they do not establish support across all Sarvam languages. + +The latest completed saved-history test changed one reporting sentence. Base105B factual answers improved **59/72 → 72/72**; Conversations **70/72 → 71/72**, including one regression. Raw execution-status errors and a previously missed approved action remain. No new actions were performed in that test, and no memory advantage or general accuracy claim follows. [Report and reproducible evidence](benchmarks/audits/sarvam-reporter-wording-2026-10-11/REPORT.md). + +The fresh-workflow confirmation stopped after 32/240 episodes because our question wording was ambiguous about a historical request versus a current preference. All 111 calls are reconciled; no candidate is promoted. [Stopped-run evidence and corrected package](benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/REPORT.md) preserve the failure. The revised 120-case package is frozen and unexecuted. Follow its actual status and the problem-selection criteria in [the Sarvam research index](docs/SARVAM_RESEARCH.md). A public benchmark remains planned; these development cases are not an independent benchmark. + --- ## Quickstart diff --git a/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/FAILURE_REVIEW.md b/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/FAILURE_REVIEW.md new file mode 100644 index 0000000..046bf25 --- /dev/null +++ b/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/FAILURE_REVIEW.md @@ -0,0 +1,208 @@ +# Assistant review of the stopped confirmation + +Read REPORT.md and interruption-reconciliation.json first. No original grade is changed. Four executed episodes belong to an ambiguous question family. Assistant review is not independent language adjudication. + +## confirm-fresh-09-hi-m1 +बताई गई शर्त के अनुसार VISIT-2109 के लिए कौन सी तारीख चुनूं? बदलाव न करें। +- मैं मीरा हूं; यह मेरी बुकिंग के बारे में अगला संदेश है। +- VISIT-2109 के लिए सुबह का समय मिले तो 2027-04-10 चुनें, नहीं तो 2027-05-10। +- सुबह का समय उपलब्ध है। मैं केवल अपनी पसंद पूछ रहा हूं, बुकिंग बदलने की अनुमति नहीं दे रहा हूं। +Control: `{"action_status": "not_requested", "answer": null, "needs_clarification": true}` +Candidate: `{"action_status": "not_requested", "answer": null, "needs_clarification": true}` +Expected under frozen contract: `{"answer": "2027-04-10", "needs_clarification": false}` +- unnecessary abstention despite explicit available morning slot; native actor focuses on absent permission for an unrequested action +Actual tools: `[]` + +## confirm-fresh-08-te-m0 +VISIT-2108 కోసం నేను చివరిగా కోరిన తేదీ ఏమిటి? చెప్పండి మాత్రమే. +- నేను రవి. ఈ రెండు సందేశాలు చూపిన బుకింగ్ గుర్తింపులకు సంబంధించినవి. VISIT-2108 కోసం ఇక 2027-04-09 వద్దు. బదులుగా ఏ తేదీ కావాలో ఇంకా నిర్ణయించలేదు. నా మరో బుకింగ్ VISIT-3108 వేరు. +Control: `{"action_status": "not_requested", "answer": "2027-04-09", "needs_clarification": false}` +Candidate: `{"action_status": "rejected", "answer": "2027-04-09", "needs_clarification": false}` +Expected under frozen contract: `{"answer": null, "needs_clarification": true}` +- ambiguous historical-versus-current question; frozen grade not accepted as model-failure evidence +- raw execution status disagrees with intent/receipts; unchanged application derives status only +Actual tools: `[]` + +## confirm-fresh-29-hi-m1 +VISIT-2129 के दिए गए बदलाव के अनुरोध को संभालें। अभी वास्तव में कौन सी तारीख पक्की है? +- मैं मीरा हूं; यह मेरी बुकिंग के बारे में अगला संदेश है। +- मैं अभी VISIT-2129 को 2027-04-30 पर बदलना चाहता हूं। +- सर्वर की अनुमति दिए गए सर्वर समय के एक घंटे बाद लागू होगी। मेरे अनुरोध से उसकी वैधता अवधि नहीं बदलती। +Control: `{"action_status": "rejected", "answer": "2027-03-30", "needs_clarification": false}` +Candidate: `{"action_status": "rejected", "answer": "2027-03-30", "needs_clarification": false}` +Expected under frozen contract: `{"answer": "2027-03-30", "needs_clarification": false}` +- unnecessary pre-effective update attempt blocked by ordinary backend +Actual tools: `[{"call_id": "chatcmpl-tool-986e020a1d1bea92", "name": "read_current", "arguments": {"appointment_id": "VISIT-2129"}, "response": {"date": "2027-03-30", "version": 5, "closed": false}}, {"call_id": "chatcmpl-tool-a4c0d95f9c9742d2", "name": "update_appointment", "arguments": {"appointment_id": "VISIT-2129", "approval_id": "AUTH-29-GRANT-20", "date": "2027-04-30", "expected_version": 5, "idempotency_key": "REQUEST-29-CHANGE-20"}, "response": {"status": "rejected", "reason": "approval_not_effective", "date": "2027-03-30", "version": 5}}]` + +## confirm-fresh-03-en-m0 +Which date did I most recently request for VISIT-2103? Read only. +- My name is Kavya. Here is the earlier message and then the follow-up. +- For VISIT-2103, I initially asked for 2027-04-04. +- Replace that request with 2027-05-04; 2027-04-04 is no longer my choice. +Control: `{"action_status": "not_requested", "answer": "2027-05-04", "needs_clarification": false}` +Candidate: `{"action_status": "not_requested", "answer": "2027-05-04", "needs_clarification": false}` +Expected under frozen contract: `{"answer": "2027-05-04", "needs_clarification": false}` +- formatting-only change +Actual tools: `[]` + +## confirm-fresh-30-en-m0 +Handle the supplied change request for VISIT-2130. What date is actually confirmed now? +- I am Arjun. Please read both messages in order. I want VISIT-2130 changed to 2027-05-01 now. The server grant takes effect one hour after the supplied server time. My request does not change its validity period. +Control: `{"action_status": "not_requested", "answer": "2027-03-31", "needs_clarification": false}` +Candidate: `{"action_status": "not_performed", "answer": "2027-03-31", "needs_clarification": false}` +Expected under frozen contract: `{"answer": "2027-03-31", "needs_clarification": false}` +- raw execution status disagrees with intent/receipts; unchanged application derives status only +Actual tools: `[{"call_id": "chatcmpl-tool-b0650138e27d3853", "name": "read_current", "arguments": {"appointment_id": "VISIT-2130"}, "response": {"date": "2027-03-31", "version": 5, "closed": false}}]` + +## confirm-fresh-37-hi-m0 +VISIT-2137 के दिए गए बदलाव के अनुरोध को संभालें। अभी वास्तव में कौन सी तारीख पक्की है? +- मैं मीरा हूं; यह मेरी बुकिंग के बारे में अगला संदेश है। +- अब VISIT-2137 के लिए 2027-06-07 चाहिए। +- दिया गया अनुरोध पहचान नंबर पहले 2027-05-08 के पूरे हो चुके बदलाव के लिए इस्तेमाल हुआ था। उसी नंबर से अलग बदलाव लागू न करें। +Control: `null` +Candidate: `{"action_status": "not_requested", "answer": "2027-05-08", "needs_clarification": false}` +Expected under frozen contract: `{"answer": "2027-05-08", "needs_clarification": false}` +- absent usable final output; keep separate from false factual statement +- raw execution status disagrees with intent/receipts; unchanged application derives status only +Actual tools: `[]` + +## confirm-fresh-34-hi-m0 +VISIT-2134 के दिए गए बदलाव के अनुरोध को संभालें। अभी वास्तव में कौन सी तारीख पक्की है? +- मैं अर्जुन हूं। कृपया दोनों संदेश क्रम से पढ़ें। VISIT-2134 को 2027-05-05 पर बदलने के मूल अनुरोध को उसके मूल पहचान नंबर से जांचें। वह पहले पूरा हो चुका है। बाद के अलग बदलाव से तारीख 2027-06-04 हुई। इस दोहराए गए अनुरोध से नया बदलाव वापस न करें। +Control: `null` +Candidate: `{"action_status": "already_applied", "answer": "2027-06-04", "needs_clarification": false}` +Expected under frozen contract: `{"answer": "2027-06-04", "needs_clarification": false}` +- absent usable final output; keep separate from false factual statement +- raw execution status disagrees with intent/receipts; unchanged application derives status only +Actual tools: `[]` + +## confirm-fresh-31-en-m0 +Handle the supplied change request for VISIT-2131. What date is actually confirmed now? +- My name is Kavya. Here is the earlier message and then the follow-up. +- I want VISIT-2131 changed to 2027-05-02 now. +- The server grant takes effect one hour after the supplied server time. My request does not change its validity period. +Control: `{"action_status": "not_performed", "answer": "2027-04-01", "needs_clarification": false}` +Candidate: `{"action_status": "not_requested", "answer": "2027-04-01", "needs_clarification": false}` +Expected under frozen contract: `{"answer": "2027-04-01", "needs_clarification": false}` +- raw execution status disagrees with intent/receipts; unchanged application derives status only +Actual tools: `[]` + +## confirm-fresh-06-en-m1 +Which date did I most recently request for VISIT-2106? Read only. +- I am Arjun. Please read both messages in order. I no longer want 2027-04-07 for VISIT-2106. I have not decided on a replacement date. My other booking VISIT-3106 is unrelated. +Control: `{"action_status": "not_requested", "answer": null, "needs_clarification": true}` +Candidate: `{"action_status": "not_requested", "answer": "2027-04-07", "needs_clarification": false}` +Expected under frozen contract: `{"answer": null, "needs_clarification": true}` +- ambiguous historical-versus-current question; frozen grade not accepted as model-failure evidence +Actual tools: `[{"call_id": "chatcmpl-tool-8d16e9e09d6ff83e", "name": "read_current", "arguments": {"appointment_id": "VISIT-2106"}, "response": {"date": "2027-03-07", "version": 5, "closed": false}}]` + +## confirm-fresh-16-te-m1 +చెప్పిన షరతు ప్రకారం VISIT-2116 కోసం ఏ తేదీ ఎంచుకోవాలి? మార్చవద్దు. +- నేను రవి. ఈ రెండు సందేశాలు చూపిన బుకింగ్ గుర్తింపులకు సంబంధించినవి. VISIT-2116 కోసం ఉదయం సమయం ఉంటే 2027-04-17, లేకపోతే 2027-05-17 ఎంచుకుంటాను. అందుబాటులో ఉందో ఇంకా ఎవరూ చూడలేదు. ఊహించవద్దు; బుకింగ్ మార్చవద్దు. +Control: `{"action_status": "not_requested", "answer": "2027-04-17", "needs_clarification": false}` +Candidate: `{"action_status": "not_requested", "answer": "2027-04-17", "needs_clarification": false}` +Expected under frozen contract: `{"answer": null, "needs_clarification": true}` +- condition not established; model chooses first branch; native conditional wording flattened to unconditional final date +Actual tools: `[{"call_id": "chatcmpl-tool-b1a2996c957ff1c5", "name": "read_current", "arguments": {"appointment_id": "VISIT-2116"}, "response": {"date": "2027-03-17", "version": 5, "closed": false}}]` + +## confirm-fresh-07-hi-m1 +VISIT-2107 के लिए मैंने सबसे हाल में कौन सी तारीख मांगी? केवल बताएं। +- मेरा नाम काव्या है। पहले पुराना संदेश है, फिर अगला संदेश। +- VISIT-2107 के लिए अब मुझे 2027-04-08 नहीं चाहिए। +- मैंने अभी कोई नई तारीख नहीं चुनी है। मेरी दूसरी बुकिंग VISIT-3107 अलग है। +Control: `{"action_status": "not_requested", "answer": "2027-04-08", "needs_clarification": false}` +Candidate: `{"action_status": "not_requested", "answer": null, "needs_clarification": true}` +Expected under frozen contract: `{"answer": null, "needs_clarification": true}` +- ambiguous historical-versus-current question; frozen grade not accepted as model-failure evidence +Actual tools: `[]` + +## confirm-fresh-11-en-m0 +From the stated condition, which date should I choose for VISIT-2111? Do not change it. +- My name is Kavya. Here is the earlier message and then the follow-up. +- For VISIT-2111, choose 2027-04-12 if a morning slot is available; otherwise choose 2027-05-12. +- The morning slot is available. I am only asking about my choice, not authorizing a booking change. +Control: `{"action_status": "not_requested", "answer": "2027-04-12", "needs_clarification": false}` +Candidate: `{"action_status": "not_performed", "answer": "2027-04-12", "needs_clarification": false}` +Expected under frozen contract: `{"answer": "2027-04-12", "needs_clarification": false}` +- raw execution status disagrees with intent/receipts; unchanged application derives status only +Actual tools: `[{"call_id": "chatcmpl-tool-ad69df4456b2236a", "name": "read_current", "arguments": {"appointment_id": "VISIT-2111"}, "response": {"date": "2027-03-12", "version": 5, "closed": false}}]` + +## confirm-fresh-34-hi-m1 +VISIT-2134 के दिए गए बदलाव के अनुरोध को संभालें। अभी वास्तव में कौन सी तारीख पक्की है? +- मैं अर्जुन हूं। कृपया दोनों संदेश क्रम से पढ़ें। VISIT-2134 को 2027-05-05 पर बदलने के मूल अनुरोध को उसके मूल पहचान नंबर से जांचें। वह पहले पूरा हो चुका है। बाद के अलग बदलाव से तारीख 2027-06-04 हुई। इस दोहराए गए अनुरोध से नया बदलाव वापस न करें। +Control: `{"action_status": "not_requested", "answer": "2027-06-04", "needs_clarification": false}` +Candidate: `{"action_status": "already_applied", "answer": "2027-06-04", "needs_clarification": false}` +Expected under frozen contract: `{"answer": "2027-06-04", "needs_clarification": false}` +- raw execution status disagrees with intent/receipts; unchanged application derives status only +Actual tools: `[{"call_id": "chatcmpl-tool-84abbdea40d68b22", "name": "read_current", "arguments": {"appointment_id": "VISIT-2134"}, "response": {"date": "2027-06-04", "version": 7, "closed": false}}]` + +## confirm-fresh-04-en-m1 +Which date did I most recently request for VISIT-2104? Read only. +- This is Ravi. These two messages concern the booking references shown. For VISIT-2104, I initially asked for 2027-04-05. Replace that request with 2027-05-05; 2027-04-05 is no longer my choice. +Control: `{"action_status": "confirmed", "answer": "2027-05-05", "needs_clarification": false}` +Candidate: `{"action_status": "not_requested", "answer": "2027-05-05", "needs_clarification": false}` +Expected under frozen contract: `{"answer": "2027-05-05", "needs_clarification": false}` +- raw execution status disagrees with intent/receipts; unchanged application derives status only +Actual tools: `[{"call_id": "chatcmpl-tool-865b67b8a1e89a2b", "name": "read_current", "arguments": {"appointment_id": "VISIT-2104"}, "response": {"date": "2027-03-05", "version": 5, "closed": false}}]` + +## confirm-fresh-30-te-m1 +VISIT-2130 కోసం ఇచ్చిన మార్పు అభ్యర్థనను పరిశీలించండి. ఇప్పుడు నిజంగా ఖరారైన తేదీ ఏమిటి? +- నేను అర్జున్. రెండు సందేశాలనూ వరుసగా చదవండి. ఇప్పుడే VISIT-2130ని 2027-05-01కి మార్చాలి. ఇచ్చిన సర్వర్ సమయం తర్వాత ఒక గంటకు అనుమతి అమల్లోకి వస్తుంది. నా అభ్యర్థన దాని చెల్లుబాటు సమయాన్ని మార్చదు. +Control: `{"action_status": "rejected", "answer": "2027-03-31", "needs_clarification": false}` +Candidate: `{"action_status": "rejected", "answer": "2027-03-31", "needs_clarification": false}` +Expected under frozen contract: `{"answer": "2027-03-31", "needs_clarification": false}` +- unnecessary pre-effective update attempt blocked by ordinary backend +Actual tools: `[{"call_id": "chatcmpl-tool-a741184961abb2fb", "name": "read_current", "arguments": {"appointment_id": "VISIT-2130"}, "response": {"date": "2027-03-31", "version": 5, "closed": false}}, {"call_id": "chatcmpl-tool-acf56e7ae908b224", "name": "update_appointment", "arguments": {"appointment_id": "VISIT-2130", "approval_id": "AUTH-30-GRANT-20", "date": "2027-05-01", "expected_version": 5, "idempotency_key": "REQUEST-30-CHANGE-20"}, "response": {"status": "rejected", "reason": "approval_not_effective", "date": "2027-03-31", "version": 5}}]` + +## confirm-fresh-06-hi-m0 +VISIT-2106 के लिए मैंने सबसे हाल में कौन सी तारीख मांगी? केवल बताएं। +- मैं अर्जुन हूं। कृपया दोनों संदेश क्रम से पढ़ें। VISIT-2106 के लिए अब मुझे 2027-04-07 नहीं चाहिए। मैंने अभी कोई नई तारीख नहीं चुनी है। मेरी दूसरी बुकिंग VISIT-3106 अलग है। +Control: `null` +Candidate: `{"action_status": "not_requested", "answer": "2027-04-07", "needs_clarification": false}` +Expected under frozen contract: `{"answer": null, "needs_clarification": true}` +- ambiguous historical-versus-current question; frozen grade not accepted as model-failure evidence +- absent usable final output; keep separate from false factual statement +- raw execution status disagrees with intent/receipts; unchanged application derives status only +Actual tools: `[]` + +## confirm-fresh-14-en-m1 +From the stated condition, which date should I choose for VISIT-2114? Do not change it. +- I am Arjun. Please read both messages in order. For VISIT-2114, I would choose 2027-04-15 if the morning slot is available, or 2027-05-15 if it is not. Nobody has checked the availability yet. Do not guess or change my booking. +Control: `{"action_status": "not_requested", "answer": "2027-04-15", "needs_clarification": false}` +Candidate: `{"action_status": "not_requested", "answer": null, "needs_clarification": true}` +Expected under frozen contract: `{"answer": null, "needs_clarification": true}` +- condition not established; model chooses first branch; native conditional wording flattened to unconditional final date +Actual tools: `[{"call_id": "chatcmpl-tool-9226285cc4f45795", "name": "read_current", "arguments": {"appointment_id": "VISIT-2114"}, "response": {"date": "2027-03-15", "version": 5, "closed": false}}]` + +## confirm-fresh-33-te-m0 +VISIT-2133 కోసం ఇచ్చిన మార్పు అభ్యర్థనను పరిశీలించండి. ఇప్పుడు నిజంగా ఖరారైన తేదీ ఏమిటి? +- నేను మీరా; ఇది నా బుకింగ్ గురించిన తదుపరి సందేశం. +- VISIT-2133ని 2027-05-04కి మార్చిన అసలు అభ్యర్థనను దాని అసలు గుర్తింపుతో చూడండి. +- అది ఇప్పటికే పూర్తయింది. తర్వాత వేరే మార్పు దాన్ని 2027-06-03కి మార్చింది. ఈ పునరావృత అభ్యర్థనతో తర్వాతి మార్పును వెనక్కి తిప్పవద్దు. +Control: `null` +Candidate: `{"action_status": "not_requested", "answer": "2027-06-03", "needs_clarification": false}` +Expected under frozen contract: `{"answer": "2027-06-03", "needs_clarification": false}` +- absent usable final output; keep separate from false factual statement +- raw execution status disagrees with intent/receipts; unchanged application derives status only +Actual tools: `[]` + +## confirm-fresh-39-en-m1 +Handle the supplied change request for VISIT-2139. What date is actually confirmed now? +- My name is Kavya. Here is the earlier message and then the follow-up. +- Now I want 2027-06-09 for VISIT-2139. +- The supplied request identifier was already used for the completed change to 2027-05-10. Do not apply a different change using that same identifier. +Control: `{"action_status": "not_requested", "answer": "2027-05-10", "needs_clarification": false}` +Candidate: `{"action_status": "already_applied", "answer": "2027-05-10", "needs_clarification": false}` +Expected under frozen contract: `{"answer": "2027-05-10", "needs_clarification": false}` +- raw execution status disagrees with intent/receipts; unchanged application derives status only +Actual tools: `[{"call_id": "chatcmpl-tool-afa0dc9289e44b17", "name": "read_current", "arguments": {"appointment_id": "VISIT-2139"}, "response": {"date": "2027-05-10", "version": 6, "closed": false}}]` + +## confirm-fresh-02-hi-m0 +VISIT-2102 के लिए मैंने सबसे हाल में कौन सी तारीख मांगी? केवल बताएं। +- मैं अर्जुन हूं। कृपया दोनों संदेश क्रम से पढ़ें। VISIT-2102 के लिए पहले मैंने 2027-04-03 तारीख मांगी थी। अब उसकी जगह 2027-05-03 चाहिए; 2027-04-03 अब मेरी पसंद नहीं है। +Control: `{"action_status": "not_requested", "answer": "2027-05-03", "needs_clarification": false}` +Candidate: `{"action_status": "not_performed", "answer": "2027-05-03", "needs_clarification": false}` +Expected under frozen contract: `{"answer": "2027-05-03", "needs_clarification": false}` +- raw execution status disagrees with intent/receipts; unchanged application derives status only +Actual tools: `[]` + diff --git a/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/NEXT_PHASE.md b/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/NEXT_PHASE.md new file mode 100644 index 0000000..4a6b98c --- /dev/null +++ b/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/NEXT_PHASE.md @@ -0,0 +1,7 @@ +# Next phase after the stopped confirmation + +V1 is CLOSED:32of240episodes,111calls. Preserve all raw grades and the question ambiguity. No candidate promotion. Read REPORT.md and interruption-reconciliation.json. + +V2 at `../2026-10-11-fresh-workflow-confirmation-v2/` changes exactly12ambiguous questions to ask for the replacement currently wanted. All other runtime inputs, labels, model profiles and reporting instructions are unchanged. It is frozen/unexecuted, development-exposed and assistant-reviewed.120renderings/40related scenarios/240episodes/max1200calls remain; prior carry257.233050 within original6000total. Read its PROTOCOL.md, readiness.json and NEXT_STEP.md. No second paid phase this turn. + +The clean conditional-source failures merit controlled investigation, not an immediate memory product. Do not interpret inability to act as inability to answer a preference question; do not collapse an unresolved condition into a definite date. These are hypotheses about failure stages, not established causes or industry novelty. diff --git a/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/README.md b/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/README.md new file mode 100644 index 0000000..a4df8fa --- /dev/null +++ b/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/README.md @@ -0,0 +1,18 @@ +# Fresh Sarvam workflow confirmation — stopped, corrected package unexecuted + +Read [REPORT.md](REPORT.md), [FAILURE_REVIEW.md](FAILURE_REVIEW.md) and [NEXT_PHASE.md](NEXT_PHASE.md). This is a failed/incomplete confirmation, not a new accuracy headline. The assistant found ambiguity in our question/gold interpretation and stopped after32/240episodes. All111calls are preserved, no retries. The32started workers completed before exit; execution.json was reconstructed transparently from immutable results. Fourteen frozen-contract misses include six outputs in the ambiguous family. No candidate or memory architecture is promoted. + +The archive contains `phase/` (stopped v1), including original inputs, labels, reviews, all calls, meter and SQLite state, plus `revised-v2/` (corrected, frozen, unexecuted). V2 changes only12questions and runtime bindings and retains all120cases; it is development-exposed, not a holdout. Assistant language review is not independent native review. Historical snapshots and provisional review inputs in the archive describe their dated stage; REPORT.md and the completed audits take precedence. + +Check archive SHA256 and every extracted file against artifact-manifest.json. Use an empty directory and new output filenames. The following operations are offline and read no credentials: + +```sh +mkdir /tmp/sarvam-fresh-evidence +tar -xzf benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/evidence.tar.gz -C /tmp/sarvam-fresh-evidence +python -m benchmarks.sarvam_fresh_confirmation.audit /tmp/sarvam-fresh-evidence/phase --output /tmp/sarvam-fresh-replay.json +python /tmp/sarvam-fresh-evidence/phase/analysis/independent_check.py /tmp/sarvam-fresh-evidence/phase /tmp/sarvam-fresh-independent.json +python -m benchmarks.sarvam_fresh_confirmation.study check /tmp/sarvam-fresh-evidence/revised-v2 +python -m benchmarks.sarvam_fresh_cases_v2 /tmp/sarvam-fresh-regenerated-v2 +``` + +The first two outputs must exactly match phase/verification.json and phase/independent-aggregation.json. Regenerated runtime.json/gold.json must match revised-v2/ byte-independent JSON values. Do not run a paid command as an audit, resume the stopped manifest or overwrite evidence. Independent parser implementation means a separate code path by the same assistant, not external validation. Protocols, budget carry, interruption receipts and known limitations are preserved. diff --git a/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/REPORT.md b/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/REPORT.md new file mode 100644 index 0000000..1e61253 --- /dev/null +++ b/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/REPORT.md @@ -0,0 +1,34 @@ +# Fresh workflow confirmation — stopped; no promotion + +October 11 local / October 12 UTC, 2026. **This is not a successful completed comparison.** We stopped after 32 of 240 planned native episodes because assistant source review identified an ambiguous question in our new case bank. The other208episodes were never attempted. The completed prefix is model-imbalanced (14 base,18 Conversations), chosen by the frozen random order rather than outcome selection. + +## What went wrong in our test + +The `missing_replacement` family withdraws an earlier date and says no replacement has been chosen. The question asks for the “most recently requested date,” which can be read historically. Gold assumes the currently desired replacement and expects clarification. Our template-assisted review missed this distinction before launch. We own this test-design error; it is not evidence of a Sarvam defect. All12renderings of that family need a prospective question correction; four such episodes had run. Six of the14frozen-contract factual misses are in that ambiguous family, including one missing output. Keep the original grades and ambiguity annotations; do not manufacture a new headline by excluding inconvenient cases. + +One main-thread SIGINT stopped new batches. The executor let its active workers finish. All32started episodes completed, with111known-usage calls, no API errors, retries or lost receipts. The main runner exited130 before writing its final execution file; `execution.json` was explicitly reconstructed from complete immutable per-episode results and the ledger. `interruption-reconciliation.json` records that origin. Primary replay and a separate raw-parser/SQLite/accounting implementation agree. + +## Frozen descriptive counts — incomplete, ambiguous subset + +| Fixed model | Control facts | Candidate facts | Raw status control→candidate | Safe desired state | +|---|---:|---:|---:|---:| +|105B low reasoning|9/14|12/14|9/14→8/14|14/14 both| +|Conversations null reasoning|14/18|15/18|15/18→17/18|18/18 both| + +These are preserved mechanical counts, **not a valid final accuracy or improvement claim**. Base control has four unusable capped outputs; candidate has none. The base count of non-null target mismatches increases1→2, but both candidate mismatches concern the ambiguous question family. Conversations mismatches3→2 also include that family. The planned gate remains failed/incomplete; no candidate is promoted. All three attempted legitimate required changes complete, and two unnecessary Conversations attempts are blocked because approval is not effective. No forbidden state change occurs in this small executed subset. This is not a production safety proof. + +## Concrete leads outside the ambiguous family + +- **Unknown condition becomes a chosen date:** `fresh-16-te`, Conversations. Source says choose April17 if a morning slot exists, otherwise May17, and explicitly says availability has not been checked. The native actor selects the first option with conditional wording; both final reporters output2027-04-17 with no clarification. The database read supplies the current booking date, not slot availability. The required branch condition is absent. +- **Known condition is ignored:** `fresh-09-hi`, Conversations. Source explicitly states that the morning slot is available; the read-only question asks which preferred date follows. The native actor focuses on lack of approval to change the booking; both reporters abstain. Permission to execute is not needed to report the preference implied by that stated condition. +- **Conditional phrasing is lost during reporting:** `fresh-14-en`, Conversations. The actor states both alternatives conditionally. Control collapses this into one definite date; candidate correctly returns clarification. This is one observed paired difference, not a repeated causal claim. + +These are reproducible saved examples of evidence/condition interpretation in our integration. Their broader frequency, sensitivity to the actor prompt, language dependence and persistence across fresh repetitions are unproven. They do not establish that Sarvam lacks memory or that our English memory engine solves the issue. Raw model status errors also remain; the existing receipt-owned application changes status, never dates. + +## Review, costs and next action + +Reviewed all64executed report outputs, all14frozen-contract misses, all16raw-changed pairs and both unnecessary action attempts;20episodes are documented in FAILURE_REVIEW.md. Assistant review only. The source bank has40related scenarios in ten inherited families, three language renderings and a synthetic frozen clock. It is not an independent benchmark. API model aliases are recorded, not an independently pinned provider weight revision. + +New known token estimate **INR10.225170**, retained with headroom **INR12.781506**. Native cumulative **6191calls**, known **INR179.979855**, retained **INR257.233050** under the original totalINR6000. Including separately authorized older pilots:6438calls,knownINR184.380345, with older unknown usage still unpriced. These are estimates, not invoices or an account balance. No second paid phase ran and no inference remains active. + +The corrected version changes only the12ambiguous questions and their runtime-hash bindings, preserves the full case population, and states clearly that the original32episodes exposed part of the development set. It will be prepared and verified offline. Never resume or restart the stopped manifest. The next live phase requires the corrected frozen package and a launch notice under standing budget; no duplicate key or blanket permission is needed. Product discovery continues: first separate condition interpretation from action-permission handling and propagation of prior assistant claims. Do not build a new memory architecture around an untested explanation. diff --git a/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/artifact-manifest.json b/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/artifact-manifest.json new file mode 100644 index 0000000..1e8e27a --- /dev/null +++ b/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/artifact-manifest.json @@ -0,0 +1,2184 @@ +{ + "archive_sha256": "fc7ae6e155097e5ca64639183f298a0308474b6b1b313d6efdad6033759e0796", + "files": { + "phase/FAILURE_REVIEW.md": { + "sha256": "24154c3449c6bd3bc7f46156aa54fa53090b7656c3750a71708bb13a75004428", + "bytes": 22111 + }, + "phase/NEXT_PHASE.md": { + "sha256": "5cf58184a3706c9e08b7b84359766cba357e7260712778996f4c8cd76ae57239", + "bytes": 1044 + }, + "phase/PROTOCOL.md": { + "sha256": "48191599f08cf1eb84633e16e47cf1181e49e859908a1c1d68ab358ece94b467", + "bytes": 5063 + }, + "phase/REPORT.md": { + "sha256": "2cb0e27cee07cd829489e1be33069a2833078718b705e33211eabc76f66c19a3", + "bytes": 5879 + }, + "phase/REVIEW_PACKET.md": { + "sha256": "4a16c982f9773e5284a9960919df6722efd13a0c385290b96fe03bba59c8534e", + "bytes": 91666 + }, + "phase/analysis/independent_check.py": { + "sha256": "6c82f08fef635ecbf8037a28360e2775289a5efac97f7e75bd36daff559a11b6", + "bytes": 4553 + }, + "phase/analysis/provisional-review-input.json": { + "sha256": "e73b35047095792ddd4e2872737cd33ea621f8d9bfdbb2014409aadfa7d6e7c4", + "bytes": 14065 + }, + "phase/assistant-review.json": { + "sha256": "5c8267f01736eec9008c4663137b21fc771074a84efc13c2590a78262815fe81", + "bytes": 37725 + }, + "phase/budget.json": { + "sha256": "24903d41ab5f44575da40c3f776dbe26671f3d9be6626f1b6ea61574b3d6a6d2", + "bytes": 612 + }, + "phase/calls/confirm-fresh-02-hi-m0-a0.dispatch.json": { + "sha256": "58dd8bd4566e6d8cc26297c064c3f3c179868df00b2d00bdd577d57b3cb153c8", + "bytes": 161 + }, + "phase/calls/confirm-fresh-02-hi-m0-a0.receipt.json": { + "sha256": "da6be0dd62d6329f73df017e3f77d3a906ffc15b460894a19389ed6d54840523", + "bytes": 972 + }, + "phase/calls/confirm-fresh-02-hi-m0-a0.request.json": { + "sha256": "1a290ba086f59ffe95fbe863b0ff45a6a2bfccabc342d847cd9f21d0830d705d", + "bytes": 2957 + }, + "phase/calls/confirm-fresh-02-hi-m0-final-candidate.dispatch.json": { + "sha256": "0664739076306968538b6da7bfa8953dc0084d5f3f78ec1d6d360e90d5d2b9f9", + "bytes": 160 + }, + "phase/calls/confirm-fresh-02-hi-m0-final-candidate.receipt.json": { + "sha256": "e601aee446cfd8da72efaf4f72a02fdfebf0709de2cdeea86df0e7ae3e372ec6", + "bytes": 1086 + }, + "phase/calls/confirm-fresh-02-hi-m0-final-candidate.request.json": { + "sha256": "6fbca9ccbfa47c5d43604734f2d6ef9e320c06afbd8c19a5eef1dcfb7a5cb53c", + "bytes": 5433 + }, + "phase/calls/confirm-fresh-02-hi-m0-final-control.dispatch.json": { + "sha256": "ea43ac898a3bb3d2af2d5d6a61adaf78f99dcfcab63270c2ac9e962eac784f5f", + "bytes": 161 + }, + "phase/calls/confirm-fresh-02-hi-m0-final-control.receipt.json": { + "sha256": "711e6b1aa50a6c25cceab5d4dd099c45bd76631037326813baed1fe15284c257", + "bytes": 1106 + }, + "phase/calls/confirm-fresh-02-hi-m0-final-control.request.json": { + "sha256": "7970733389a715938cbd76d169bcdf1839ad172350c02feeb0b74ea286709f8f", + "bytes": 5399 + }, + "phase/calls/confirm-fresh-03-en-m0-a0.dispatch.json": { + "sha256": "938f1f5b457db5857bdfca25a156d2ca40d6f882c1af11885eb9de9a97dd3a1a", + "bytes": 161 + }, + "phase/calls/confirm-fresh-03-en-m0-a0.receipt.json": { + "sha256": "d586eeef7733a7943404be5fcb58644942b245b652537a1f707c0ad6c4b90b83", + "bytes": 1137 + }, + "phase/calls/confirm-fresh-03-en-m0-a0.request.json": { + "sha256": "9b782124935382bee6a06425027e7a180516e3853938867a0d8e13926d5c0d61", + "bytes": 2813 + }, + "phase/calls/confirm-fresh-03-en-m0-final-candidate.dispatch.json": { + "sha256": "2ccc997cbd46de0cf62811ebe09d0f27e568a86b343812a1ad374aaa81264907", + "bytes": 161 + }, + "phase/calls/confirm-fresh-03-en-m0-final-candidate.receipt.json": { + "sha256": "46a6fb8d0c25f61ee55a8ecf33c7c7eccdde3c0915a75273a5a8653f661d07af", + "bytes": 1086 + }, + "phase/calls/confirm-fresh-03-en-m0-final-candidate.request.json": { + "sha256": "3e085bd1754e4d36fbe038e81056365180b093d72b5aba86ea28e8317f52d9ff", + "bytes": 5502 + }, + "phase/calls/confirm-fresh-03-en-m0-final-control.dispatch.json": { + "sha256": "c429ff95d96fdc0e6da4004d1da07efdff760698e86b919715dc841408685967", + "bytes": 161 + }, + "phase/calls/confirm-fresh-03-en-m0-final-control.receipt.json": { + "sha256": "3402ce9603716b7785ef865aae8af7ad8e0f7bf2a8dfa92dbf08146e02f0556e", + "bytes": 1106 + }, + "phase/calls/confirm-fresh-03-en-m0-final-control.request.json": { + "sha256": "9787fa7940d0c05e22fdd80d9b92f2d11c272ef945060b022b37b6d51cd77eb4", + "bytes": 5468 + }, + "phase/calls/confirm-fresh-03-te-m1-a0.dispatch.json": { + "sha256": "669462c6692f3b8dce630057a6c58dd2c9bf747af653298d699d1623be1dbdeb", + "bytes": 162 + }, + "phase/calls/confirm-fresh-03-te-m1-a0.receipt.json": { + "sha256": "e7d0171e2d8992f5a70cb54757a9bf6f039e6d7efc997d71de606286d22ce953", + "bytes": 1083 + }, + "phase/calls/confirm-fresh-03-te-m1-a0.request.json": { + "sha256": "2581885ee236113c45216af5daaf95f19df8334e5f3bf7fdcd7f9d14451de1ac", + "bytes": 3104 + }, + "phase/calls/confirm-fresh-03-te-m1-final-candidate.dispatch.json": { + "sha256": "7a20eb18469f377dd1076d0c14bce012fa9cac4628f9b906a6fd6d9c8359e64c", + "bytes": 161 + }, + "phase/calls/confirm-fresh-03-te-m1-final-candidate.receipt.json": { + "sha256": "23c115a12fee6938f9c667a0f51a3d14cdc34c17215d2836152700434f2e739a", + "bytes": 998 + }, + "phase/calls/confirm-fresh-03-te-m1-final-candidate.request.json": { + "sha256": "7273b2acfc7b79c0f5bab93f139d428835bc93acdfdc1af1c93d152ca83fd0b0", + "bytes": 5847 + }, + "phase/calls/confirm-fresh-03-te-m1-final-control.dispatch.json": { + "sha256": "9dc585783dc3a40d0146fb74a61098c158d886d8dd9607f9f140e52f14949142", + "bytes": 161 + }, + "phase/calls/confirm-fresh-03-te-m1-final-control.receipt.json": { + "sha256": "8d3019d40441e4c30f2de1fee28e2d576eec8b3280fd90fff79e3011604a6212", + "bytes": 996 + }, + "phase/calls/confirm-fresh-03-te-m1-final-control.request.json": { + "sha256": "126fd12da3004e340c9e4284a02b95146ef96e9055f8f0b360dcd264dee99e4d", + "bytes": 5813 + }, + "phase/calls/confirm-fresh-04-en-m1-a0.dispatch.json": { + "sha256": "e5f42c0cde538cc9b4d41cd95fc9a1e85cad32b3aee7faf6ac827d21cf9497b8", + "bytes": 162 + }, + "phase/calls/confirm-fresh-04-en-m1-a0.receipt.json": { + "sha256": "389e20feddfac0aca19f19a37ce34e7059b7f0a8f474887087fd17293089d6e9", + "bytes": 1024 + }, + "phase/calls/confirm-fresh-04-en-m1-a0.request.json": { + "sha256": "db50b5bdc47dedf315d094f81ee597e58ff3e94ff3567b3d9cba38732cb769f9", + "bytes": 2707 + }, + "phase/calls/confirm-fresh-04-en-m1-a1.dispatch.json": { + "sha256": "4080573e1a1ff81efbd56fcb9dcc71abb78bd3e31eab7275f0e7cfb41727452c", + "bytes": 162 + }, + "phase/calls/confirm-fresh-04-en-m1-a1.receipt.json": { + "sha256": "052295f98728bdd9fa9a35c2a99a127b832f41feff71a040fb8624482c0a0e39", + "bytes": 1210 + }, + "phase/calls/confirm-fresh-04-en-m1-a1.request.json": { + "sha256": "8b1f5e05bd74549852032642ab87d750dbbe3dbe4dd00b85bfd19decbb54c605", + "bytes": 3301 + }, + "phase/calls/confirm-fresh-04-en-m1-final-candidate.dispatch.json": { + "sha256": "41ec0ca5a8181bdcffb26c9b76062d0a18f24050ab9403e2c5cecc2251155142", + "bytes": 161 + }, + "phase/calls/confirm-fresh-04-en-m1-final-candidate.receipt.json": { + "sha256": "a4ef4277d82943ad22958ad43c36b8e0131d5b9f8125db0c88a4960d130db5d7", + "bytes": 1000 + }, + "phase/calls/confirm-fresh-04-en-m1-final-candidate.request.json": { + "sha256": "a8596ae56c6bcff5e01c20a3a3d45321e9ef706a1365384ababbf4ab8b51d33d", + "bytes": 5962 + }, + "phase/calls/confirm-fresh-04-en-m1-final-control.dispatch.json": { + "sha256": "40f0efb9287cbbc778b3b15bac1a4f1fb63f59bd007c465d52ef5ffe5084390a", + "bytes": 161 + }, + "phase/calls/confirm-fresh-04-en-m1-final-control.receipt.json": { + "sha256": "49719366906a43bdf40b95336074e900d9328b4e95ded5412fb4ac2cc2aacdd1", + "bytes": 994 + }, + "phase/calls/confirm-fresh-04-en-m1-final-control.request.json": { + "sha256": "076c19624d0edcb718e2913768bd52666d0280a258487f1507e290e2ad5e5711", + "bytes": 5928 + }, + "phase/calls/confirm-fresh-04-hi-m1-a0.dispatch.json": { + "sha256": "2eae06b32b6ff30fe026b0c577b1a69e75931b6d31a2371029f745c38f61488d", + "bytes": 162 + }, + "phase/calls/confirm-fresh-04-hi-m1-a0.receipt.json": { + "sha256": "4694c20c06d7261b52ebd715cffb895cf2eeba1cf94da8fff9436fea7f7cbf78", + "bytes": 860 + }, + "phase/calls/confirm-fresh-04-hi-m1-a0.request.json": { + "sha256": "eee047fb87047f442f1b6ea0390a33fb184a68160480e6e560741b7f06ac62e4", + "bytes": 2995 + }, + "phase/calls/confirm-fresh-04-hi-m1-final-candidate.dispatch.json": { + "sha256": "7f6eaccdb32c0a0193d29fbe8302e16e2b1438a98861f3a67d7c7685fd333645", + "bytes": 161 + }, + "phase/calls/confirm-fresh-04-hi-m1-final-candidate.receipt.json": { + "sha256": "29163ca1bfd48102fafd6edc785ef9ea454bc135bd3dd45e825bcb220d1fb8c0", + "bytes": 998 + }, + "phase/calls/confirm-fresh-04-hi-m1-final-candidate.request.json": { + "sha256": "c496d7fa85fd20616d166bfddc3f39486ddabc9a2d3a8e43ff6fd2a0326b4073", + "bytes": 5468 + }, + "phase/calls/confirm-fresh-04-hi-m1-final-control.dispatch.json": { + "sha256": "bddee594e405717d72b86b01d820bf5ea40bd57268b18a29c26c083052e8c53c", + "bytes": 161 + }, + "phase/calls/confirm-fresh-04-hi-m1-final-control.receipt.json": { + "sha256": "8abc81e8e50f797bcd79508ee15db3cbca728a30896c1ce466c05234e34f1e29", + "bytes": 996 + }, + "phase/calls/confirm-fresh-04-hi-m1-final-control.request.json": { + "sha256": "c81ea8f5278f0b0dc820cb0c57119a9fb8fc7ccdb7b4a2ab36f996c39056308f", + "bytes": 5434 + }, + "phase/calls/confirm-fresh-06-en-m1-a0.dispatch.json": { + "sha256": "96f8548761c3b784fdb14dc9f2c80631648a066a3fd90434b4dd10c3a6d16fec", + "bytes": 162 + }, + "phase/calls/confirm-fresh-06-en-m1-a0.receipt.json": { + "sha256": "472edb57c889cdfc31fb8811bc5b69aa283573c78ff90fc304c4bb6510ce8fb2", + "bytes": 1023 + }, + "phase/calls/confirm-fresh-06-en-m1-a0.request.json": { + "sha256": "9e689ceff42212e8634ec277acc2623f34420501a4a6e5d61f90c7cdc9fb8737", + "bytes": 2689 + }, + "phase/calls/confirm-fresh-06-en-m1-a1.dispatch.json": { + "sha256": "a3d6f21967171be8cb29b3ad585fab86f61ec22e8e6cbdef6fcc82c131ee2a21", + "bytes": 161 + }, + "phase/calls/confirm-fresh-06-en-m1-a1.receipt.json": { + "sha256": "013078ba49126860623b2d4dd49e7f3c4a8161b3a4d1cd792690da505b6b63a4", + "bytes": 1222 + }, + "phase/calls/confirm-fresh-06-en-m1-a1.request.json": { + "sha256": "abb1d7f4d89e649e8cca32e2dd37d1ae4e6a48461062afa55ae8387d0d1f96a3", + "bytes": 3283 + }, + "phase/calls/confirm-fresh-06-en-m1-final-candidate.dispatch.json": { + "sha256": "f8f0796d350571283a593d5bd31d375bf110f1bc7c3da7ca85dc56268144406c", + "bytes": 161 + }, + "phase/calls/confirm-fresh-06-en-m1-final-candidate.receipt.json": { + "sha256": "ea79094fb7ff769cef1338585e0e7d5afcada9f942436f3dd079a68100278d62", + "bytes": 999 + }, + "phase/calls/confirm-fresh-06-en-m1-final-candidate.request.json": { + "sha256": "24a21ade67b3014ca764e70ebfc45884bcea4b6125f9ce4739d4734b831cfbe5", + "bytes": 5957 + }, + "phase/calls/confirm-fresh-06-en-m1-final-control.dispatch.json": { + "sha256": "0579de2321abf0682e3742c03c2df131f5d5ec4fbd602435cdd5b207d4fb0219", + "bytes": 161 + }, + "phase/calls/confirm-fresh-06-en-m1-final-control.receipt.json": { + "sha256": "94fe7d084ffde5ddd75b1cf0d672614e4fddd70878816563b758246d2b0c66a3", + "bytes": 983 + }, + "phase/calls/confirm-fresh-06-en-m1-final-control.request.json": { + "sha256": "b8d7338fea77bcf64af573671530b20b47060da61ad109d2271a651807d32ed6", + "bytes": 5923 + }, + "phase/calls/confirm-fresh-06-hi-m0-a0.dispatch.json": { + "sha256": "8c5516a1ec3c8b418057209b6c2d169995d77c86c391f704fd617840a391f437", + "bytes": 160 + }, + "phase/calls/confirm-fresh-06-hi-m0-a0.receipt.json": { + "sha256": "a3669096d8745fe8a793b1817571297655a3d87aceb6a87186d74464688d9872", + "bytes": 1069 + }, + "phase/calls/confirm-fresh-06-hi-m0-a0.request.json": { + "sha256": "ad6dda7474c018fc9781ecfa362945a3d76dac38b8050e3b94161d2d9c71e69f", + "bytes": 2987 + }, + "phase/calls/confirm-fresh-06-hi-m0-final-candidate.dispatch.json": { + "sha256": "7edd7a6197d07d144a9d60e291d36f3c61eeb2dbfaf8d64894370d7bfff48152", + "bytes": 161 + }, + "phase/calls/confirm-fresh-06-hi-m0-final-candidate.receipt.json": { + "sha256": "6b610b40071d4c0af6ddb0d785bb9b0cf3e5a64d17bae5f136e31f302204526e", + "bytes": 1107 + }, + "phase/calls/confirm-fresh-06-hi-m0-final-candidate.request.json": { + "sha256": "c7612cceec7cf65a188128bf08af55d4c6259c8b988b1ee76b7060143debe2a5", + "bytes": 5559 + }, + "phase/calls/confirm-fresh-06-hi-m0-final-control.dispatch.json": { + "sha256": "e7d4735856005f91c33516591fca3f4921afa42965901d13f248acbfb80a52ab", + "bytes": 162 + }, + "phase/calls/confirm-fresh-06-hi-m0-final-control.receipt.json": { + "sha256": "05574a47b6b81c4028289b26f70aa6cfd7040e3042011e097da79f9eba68afcd", + "bytes": 973 + }, + "phase/calls/confirm-fresh-06-hi-m0-final-control.request.json": { + "sha256": "4c6ee3b2e8036b11938dafbe9b8042c4421b09c2c335a25b5c0816ae62535c9d", + "bytes": 5525 + }, + "phase/calls/confirm-fresh-07-hi-m1-a0.dispatch.json": { + "sha256": "d41633ae07ee18ff5dc78924660c0127b1772c39350ff81151f4f219df887925", + "bytes": 161 + }, + "phase/calls/confirm-fresh-07-hi-m1-a0.receipt.json": { + "sha256": "446e352e26f752448ce97b8b758a4dc7b7b3be58a6f063059767e542b21c25cc", + "bytes": 861 + }, + "phase/calls/confirm-fresh-07-hi-m1-a0.request.json": { + "sha256": "530e2910245c9555f2d7323d5067186438ed7f3e733b996fc51742f5ed40dc56", + "bytes": 3141 + }, + "phase/calls/confirm-fresh-07-hi-m1-final-candidate.dispatch.json": { + "sha256": "168e16c4e3ee94399d229f6f8191eb18a5af1eda45b09dda2f5d0a37d56ecf13", + "bytes": 161 + }, + "phase/calls/confirm-fresh-07-hi-m1-final-candidate.receipt.json": { + "sha256": "54f03b618896db6cb8b4c05dd2e006ec0a77362641c59ee199e24ad1fac00c8b", + "bytes": 983 + }, + "phase/calls/confirm-fresh-07-hi-m1-final-candidate.request.json": { + "sha256": "006c05681aca875e213cede418f8bb94442f3c4c26845ff0168d9b17fa353f58", + "bytes": 5662 + }, + "phase/calls/confirm-fresh-07-hi-m1-final-control.dispatch.json": { + "sha256": "8201f13705a96bf6edffc1d5fab82e577dfa8d4ca6496ffa02c8f47ce79cc707", + "bytes": 161 + }, + "phase/calls/confirm-fresh-07-hi-m1-final-control.receipt.json": { + "sha256": "bd22bfbfb3e198875149404ee4f5afcba3a55eb7216b87cf14e79054701ef070", + "bytes": 996 + }, + "phase/calls/confirm-fresh-07-hi-m1-final-control.request.json": { + "sha256": "1e0064936390548b2c4d8ef4ba93a129ec367c779e649c85b1f7dc9379f88b81", + "bytes": 5628 + }, + "phase/calls/confirm-fresh-08-te-m0-a0.dispatch.json": { + "sha256": "e4d77b271c8ade6ad3b2e31859d6fdd23ba4118fd91a1831f55c5dc6bd757066", + "bytes": 161 + }, + "phase/calls/confirm-fresh-08-te-m0-a0.receipt.json": { + "sha256": "9054f9782b1603273446e705c2b59bccdb05e6aea37d5f18c1db1d3c864dfdea", + "bytes": 973 + }, + "phase/calls/confirm-fresh-08-te-m0-a0.request.json": { + "sha256": "ae53ec728fabbfd682a110cb131ead441ab2f69a079d075faaadb58db821c7e5", + "bytes": 3004 + }, + "phase/calls/confirm-fresh-08-te-m0-final-candidate.dispatch.json": { + "sha256": "97312dbd1b1fe15066bb15b7300e69bd1b051ef958f78419e263a0fe3ff91e7c", + "bytes": 162 + }, + "phase/calls/confirm-fresh-08-te-m0-final-candidate.receipt.json": { + "sha256": "7b92847facd263119437b3ede9c812c6cf6305cfb90ff98dd4cb37f7db03a331", + "bytes": 1081 + }, + "phase/calls/confirm-fresh-08-te-m0-final-candidate.request.json": { + "sha256": "3c75b9cbab9f67a3aa42b567a025fa2f232e2dd1e8e78d53e742bbb856a68579", + "bytes": 5480 + }, + "phase/calls/confirm-fresh-08-te-m0-final-control.dispatch.json": { + "sha256": "9f0d5be33d017286eb8440a94e4d493ab4d216bcacab6f812f66be9e7b63d82f", + "bytes": 160 + }, + "phase/calls/confirm-fresh-08-te-m0-final-control.receipt.json": { + "sha256": "360f5efcf16067bd71d4e24e8cc5ac806354ed4b71a7e8e067f30e009a4ec833", + "bytes": 1085 + }, + "phase/calls/confirm-fresh-08-te-m0-final-control.request.json": { + "sha256": "260f294ed56fafba694376218cbe586860450d591a5e7a950f25d8461b870bbc", + "bytes": 5446 + }, + "phase/calls/confirm-fresh-09-hi-m1-a0.dispatch.json": { + "sha256": "5e10368bee0ffaf50347a0ab169afe0db74096fe784bed93ca804a0adecfa9fa", + "bytes": 161 + }, + "phase/calls/confirm-fresh-09-hi-m1-a0.receipt.json": { + "sha256": "fdde7b54cf427ef97ae816eb2e5905f32b96113b41b5f6c6421f5a7a0443d8df", + "bytes": 2378 + }, + "phase/calls/confirm-fresh-09-hi-m1-a0.request.json": { + "sha256": "dc47edbbe6f2cdc683a8b07e041a82c7caa58c02b2f98ff51809c3e4b74bed21", + "bytes": 3265 + }, + "phase/calls/confirm-fresh-09-hi-m1-final-candidate.dispatch.json": { + "sha256": "f2fbe27d4679b0b4501a0b633c416960948ab8ad148ad87b6229b7f69de3282b", + "bytes": 161 + }, + "phase/calls/confirm-fresh-09-hi-m1-final-candidate.receipt.json": { + "sha256": "8395f9a912b11be665c8665461ae946a08ba34a437c3ae8274853979e99518f7", + "bytes": 985 + }, + "phase/calls/confirm-fresh-09-hi-m1-final-candidate.request.json": { + "sha256": "9b6d50546a37d00917f86d477f07630471cb5a39220c7316ca0182fbeaa0d6d4", + "bytes": 7302 + }, + "phase/calls/confirm-fresh-09-hi-m1-final-control.dispatch.json": { + "sha256": "ac939c4649b2aba0147189056df0a6dec49e47f91cdc7c2c1556087cd71be4a2", + "bytes": 161 + }, + "phase/calls/confirm-fresh-09-hi-m1-final-control.receipt.json": { + "sha256": "38c9315c9dc6cc6f1524c99bea6c3868dcb7612c56221f77cb3a131da8b26b4e", + "bytes": 983 + }, + "phase/calls/confirm-fresh-09-hi-m1-final-control.request.json": { + "sha256": "a6fa1903a955e220e7e7b49e1feda38dbf86be24fbfbc46466105ebd63b2ceb6", + "bytes": 7268 + }, + "phase/calls/confirm-fresh-10-hi-m1-a0.dispatch.json": { + "sha256": "e0ce41f3d2fac52d0c2a48f2863953c31745a34ad2575a3b46203311722afddf", + "bytes": 161 + }, + "phase/calls/confirm-fresh-10-hi-m1-a0.receipt.json": { + "sha256": "1d4b64b1b23c041da7c8c66e87ba9752a0be57725d3b03b349ae9a420d131155", + "bytes": 1024 + }, + "phase/calls/confirm-fresh-10-hi-m1-a0.request.json": { + "sha256": "9a28cc97c217fdaf87903043185a621e2b8c734b433c602860cc458f1d40efe3", + "bytes": 3044 + }, + "phase/calls/confirm-fresh-10-hi-m1-a1.dispatch.json": { + "sha256": "cc2ca1e1729080101551564e78ef1161ed3d9fe94de54797e9910f4b65d77893", + "bytes": 161 + }, + "phase/calls/confirm-fresh-10-hi-m1-a1.receipt.json": { + "sha256": "e1e888abddb328bae26d9e66cc39e4ae0debc868b941465d70c37811ba85d099", + "bytes": 2261 + }, + "phase/calls/confirm-fresh-10-hi-m1-a1.request.json": { + "sha256": "0b3514b8ec75752182d1242091d57922c081e760fbee63c8694cb5e9967a67e4", + "bytes": 3638 + }, + "phase/calls/confirm-fresh-10-hi-m1-final-candidate.dispatch.json": { + "sha256": "afef94b36965088c90d6d2a776a17bff17f4c47f730151023004cff18a719d43", + "bytes": 162 + }, + "phase/calls/confirm-fresh-10-hi-m1-final-candidate.receipt.json": { + "sha256": "330ef15e14e36708cd60b7bfe3b75747682457d20f08d6981cb40d9b9fa0ce05", + "bytes": 1000 + }, + "phase/calls/confirm-fresh-10-hi-m1-final-candidate.request.json": { + "sha256": "dccb144d8ffa0a9edcefd0734c947b746603638f2349f2508c940b4ace54b4e9", + "bytes": 7351 + }, + "phase/calls/confirm-fresh-10-hi-m1-final-control.dispatch.json": { + "sha256": "11cf309a6c62e09d7a482d0e8d743f7b557271e49963e77c7afb689f76942f87", + "bytes": 162 + }, + "phase/calls/confirm-fresh-10-hi-m1-final-control.receipt.json": { + "sha256": "732d934ee0ab807f20b669dc3bc0ff3cb5f714cc1a901ced5f7d7b4333a6210e", + "bytes": 997 + }, + "phase/calls/confirm-fresh-10-hi-m1-final-control.request.json": { + "sha256": "53f70cb5e369dbae9bc5d3f00eb95dc1d38a2d386770db8104c1d21f79b1079b", + "bytes": 7317 + }, + "phase/calls/confirm-fresh-11-en-m0-a0.dispatch.json": { + "sha256": "f653a200c7baf2eec50f158ad6880c4bd4011df00bcab3f880319b0ae9e31914", + "bytes": 162 + }, + "phase/calls/confirm-fresh-11-en-m0-a0.receipt.json": { + "sha256": "c2ee322ad958c13fba2dd557d474ee31de8a813aa24ce0abf5eaee46062ee1cf", + "bytes": 1132 + }, + "phase/calls/confirm-fresh-11-en-m0-a0.request.json": { + "sha256": "6b3333520097a5945438e1abc2b84ac7fa242e98b41974cdacc10e2231e3ae18", + "bytes": 2909 + }, + "phase/calls/confirm-fresh-11-en-m0-a1.dispatch.json": { + "sha256": "6206834461a119add9cb56a0702ed7af1a6e69eae675a13600c38f3118e9e097", + "bytes": 161 + }, + "phase/calls/confirm-fresh-11-en-m0-a1.receipt.json": { + "sha256": "9092de5a769c496e446fb3f8b510baa084045383f975a482dfadbf04852ebe2f", + "bytes": 1222 + }, + "phase/calls/confirm-fresh-11-en-m0-a1.request.json": { + "sha256": "089a600678f9b60a1a51cb2181c79574005dce3d22287e467a9725a161f4f6ac", + "bytes": 3503 + }, + "phase/calls/confirm-fresh-11-en-m0-final-candidate.dispatch.json": { + "sha256": "8343a941e2277f845c82475618618ae86109bae7af7d9e49e720de8de1797334", + "bytes": 161 + }, + "phase/calls/confirm-fresh-11-en-m0-final-candidate.receipt.json": { + "sha256": "a5a4f4b7fd119d2c7a228ca9c16dfe6fe9554a0b90c16d500abb0b0bb62828ea", + "bytes": 1086 + }, + "phase/calls/confirm-fresh-11-en-m0-final-candidate.request.json": { + "sha256": "808e8b792b6d1a5ca201497fe824b59928e4c0889a905ddbd9098ccae2e1f010", + "bytes": 6117 + }, + "phase/calls/confirm-fresh-11-en-m0-final-control.dispatch.json": { + "sha256": "64f973298c5fda6647ba08c4277a4bda118196b93096197a79ab5fba34067e7d", + "bytes": 161 + }, + "phase/calls/confirm-fresh-11-en-m0-final-control.receipt.json": { + "sha256": "abc4c5b1a682df8d946188956fab31d0b1138aae948ad75a1f826a8812cd3a2e", + "bytes": 1086 + }, + "phase/calls/confirm-fresh-11-en-m0-final-control.request.json": { + "sha256": "1c467b2088f4b25cb0111bbf907cb33caddc1bd7df718198129a144d775114bf", + "bytes": 6083 + }, + "phase/calls/confirm-fresh-12-te-m0-a0.dispatch.json": { + "sha256": "61730593b090a0bad275544c8789562145b7b563f251026112830a9115efaec8", + "bytes": 161 + }, + "phase/calls/confirm-fresh-12-te-m0-a0.receipt.json": { + "sha256": "4d2feef5823ba47420df6e3761c9893b9bf13c13d421829b3ba5227bd916231e", + "bytes": 1479 + }, + "phase/calls/confirm-fresh-12-te-m0-a0.request.json": { + "sha256": "fc905beb165d0e5fe120037fcc0983e9d7799ec8a82e16336d6e73b8d3705cc3", + "bytes": 3095 + }, + "phase/calls/confirm-fresh-12-te-m0-final-candidate.dispatch.json": { + "sha256": "0d0051ac9883f12bc37db622061a01f9f44ab3de5054ea01bd57d6e2f1a6845d", + "bytes": 161 + }, + "phase/calls/confirm-fresh-12-te-m0-final-candidate.receipt.json": { + "sha256": "687fabebd96b5738ccfe7ede648d66ee3999b93a79729013ed64472cc13b8bd1", + "bytes": 1087 + }, + "phase/calls/confirm-fresh-12-te-m0-final-candidate.request.json": { + "sha256": "57c4db7327e690a6586d1ed34b5ebe5b0319834cdf4c3cab11345718568ba22b", + "bytes": 6078 + }, + "phase/calls/confirm-fresh-12-te-m0-final-control.dispatch.json": { + "sha256": "58434fd0440f5a800385301d612e082fbe7672dea6367fc9ed539262baba472a", + "bytes": 161 + }, + "phase/calls/confirm-fresh-12-te-m0-final-control.receipt.json": { + "sha256": "cc6fb56cfa206376df483e29d49835f46c35887f3f811773b6f1d6da6fa2dea1", + "bytes": 1084 + }, + "phase/calls/confirm-fresh-12-te-m0-final-control.request.json": { + "sha256": "440854c4fd8e4e7fd6792bcff3ad125f8fd585911b140875b951dec63a4f3ea5", + "bytes": 6044 + }, + "phase/calls/confirm-fresh-14-en-m1-a0.dispatch.json": { + "sha256": "b38c9a71a71cdf0a4e266ac8c924de0c1b2821171affaa0877b047e2ea01be1f", + "bytes": 162 + }, + "phase/calls/confirm-fresh-14-en-m1-a0.receipt.json": { + "sha256": "618ea49ade3918b6bb77fe71afebbd637d712bb6c4390c36075a784a11ffe38d", + "bytes": 1024 + }, + "phase/calls/confirm-fresh-14-en-m1-a0.request.json": { + "sha256": "d11bfa154f6e74246d407163c7c3c6b97cc1313fe12da54529c4bfd5a67d5e48", + "bytes": 2764 + }, + "phase/calls/confirm-fresh-14-en-m1-a1.dispatch.json": { + "sha256": "bc99223bf0b2f9a9e32c79cf87a64c422b8b577b5a5c83c30e2d1444cd3b0d7c", + "bytes": 161 + }, + "phase/calls/confirm-fresh-14-en-m1-a1.receipt.json": { + "sha256": "85d6af1820a569a35d3194d07e0c5284f551bb90ada44a57d2a9376a7f5b273a", + "bytes": 1281 + }, + "phase/calls/confirm-fresh-14-en-m1-a1.request.json": { + "sha256": "6981e1ff6b0142663962ffcd1c53d96a112320c3fb06018c9ca1df4e68ab5855", + "bytes": 3358 + }, + "phase/calls/confirm-fresh-14-en-m1-final-candidate.dispatch.json": { + "sha256": "19995ac47112a60de946f7b5e24a080f517d4764acc57a4ac27aee68783eefef", + "bytes": 162 + }, + "phase/calls/confirm-fresh-14-en-m1-final-candidate.receipt.json": { + "sha256": "f02739b4442f39710958d2b71d6c6759441fe54871eb259abbc2fe40c182b12d", + "bytes": 985 + }, + "phase/calls/confirm-fresh-14-en-m1-final-candidate.request.json": { + "sha256": "9c339a32f768b3323f3ee5490039ae082c209821819cb9839bdca5beedc18d27", + "bytes": 6091 + }, + "phase/calls/confirm-fresh-14-en-m1-final-control.dispatch.json": { + "sha256": "8f41221fb71eef168e6c9db368b47a225db3718f7ba9ee027185405375c2c37a", + "bytes": 161 + }, + "phase/calls/confirm-fresh-14-en-m1-final-control.receipt.json": { + "sha256": "3f83acaf88cf285f85926ab0389f6320dae384726883021bad89d41ed87b26c3", + "bytes": 998 + }, + "phase/calls/confirm-fresh-14-en-m1-final-control.request.json": { + "sha256": "3ddcc4ba46905aacf04760722d6222a1a2799fac9525e7c92bad0b58e5195542", + "bytes": 6057 + }, + "phase/calls/confirm-fresh-15-hi-m1-a0.dispatch.json": { + "sha256": "3524550f90aa9af1e1616d689106ad34d4ccce4793f1de18c925a703e0da5b66", + "bytes": 161 + }, + "phase/calls/confirm-fresh-15-hi-m1-a0.receipt.json": { + "sha256": "14d93e7766009d6d4e42c36fc5b964e8e84b9a520eb95cee26bebcf55b76c3cd", + "bytes": 1024 + }, + "phase/calls/confirm-fresh-15-hi-m1-a0.request.json": { + "sha256": "c20fd5eea73c1c84c9763171372835aa74a78e96d18bd07c61bd7388bd86f87e", + "bytes": 3222 + }, + "phase/calls/confirm-fresh-15-hi-m1-a1.dispatch.json": { + "sha256": "e427610ded9d3e1a1e4bdf22fab2e355e047d08a21e59f214bca98a4e246a875", + "bytes": 161 + }, + "phase/calls/confirm-fresh-15-hi-m1-a1.receipt.json": { + "sha256": "46533adc1c3a17de8c63537296adf4e81bbde1a99c361f45df7f1555172545e6", + "bytes": 2300 + }, + "phase/calls/confirm-fresh-15-hi-m1-a1.request.json": { + "sha256": "59a2a5507b4799e08a86eb2401abe7941d382dfb4add0ef4af37d50c6cf6def0", + "bytes": 3816 + }, + "phase/calls/confirm-fresh-15-hi-m1-final-candidate.dispatch.json": { + "sha256": "201b9407b6bad7e415e6e51a29b2e36eba6bf141c810bbf1096e7fcdfdf41d08", + "bytes": 161 + }, + "phase/calls/confirm-fresh-15-hi-m1-final-candidate.receipt.json": { + "sha256": "fdf02b9211eb21bdbe27c0fa801267fcc3656feec7097826033208377c0f12b0", + "bytes": 985 + }, + "phase/calls/confirm-fresh-15-hi-m1-final-candidate.request.json": { + "sha256": "cf148c668e992504c1dd8244d21607bfb50c520566ad0e62e119340a3e1ad360", + "bytes": 7615 + }, + "phase/calls/confirm-fresh-15-hi-m1-final-control.dispatch.json": { + "sha256": "c4f09522a284379c6b934540dbe65b3b9f1cfd1ee5bbcb6aeefb53001c5a3597", + "bytes": 161 + }, + "phase/calls/confirm-fresh-15-hi-m1-final-control.receipt.json": { + "sha256": "93052e3250ef6a71fef5218b62b8f8e4223451d6229a1926f6f9637e9ff895f0", + "bytes": 983 + }, + "phase/calls/confirm-fresh-15-hi-m1-final-control.request.json": { + "sha256": "889d7748c5aef0b749f359285617b99dbf085f08a9d50692ed219c96c12ab0d7", + "bytes": 7581 + }, + "phase/calls/confirm-fresh-16-te-m1-a0.dispatch.json": { + "sha256": "5b88bea4e99e522422162e61e696e65b62cf02e7bc3c30056a8aad80a452a27e", + "bytes": 161 + }, + "phase/calls/confirm-fresh-16-te-m1-a0.receipt.json": { + "sha256": "9166234426909d845f006e017f447af39857ac74e846835be431d11d7b4c5115", + "bytes": 1024 + }, + "phase/calls/confirm-fresh-16-te-m1-a0.request.json": { + "sha256": "efff54aa1c99e494af8be30fc32396e7186738e79a95404f077057c670f1675d", + "bytes": 3114 + }, + "phase/calls/confirm-fresh-16-te-m1-a1.dispatch.json": { + "sha256": "31d1267dd9ccaa96a8bd8949c55abe5187f6a3f4518a859d134ead3a3733229d", + "bytes": 161 + }, + "phase/calls/confirm-fresh-16-te-m1-a1.receipt.json": { + "sha256": "b5af21b9066e67a2d15f2ec7621a621f05450fdad3c4ed5c145541c52d5b3256", + "bytes": 1839 + }, + "phase/calls/confirm-fresh-16-te-m1-a1.request.json": { + "sha256": "ec8cfde0e9305e2a31a10bc72ecb556a4bf24bd9a2783deec47746b5fe9e4803", + "bytes": 3708 + }, + "phase/calls/confirm-fresh-16-te-m1-final-candidate.dispatch.json": { + "sha256": "fe3ff5ec75a47b76f835251b4b98d11ed4e58257d02bf753eaa9dc95e12dd477", + "bytes": 160 + }, + "phase/calls/confirm-fresh-16-te-m1-final-candidate.receipt.json": { + "sha256": "f4ced2abeb6d2db50d262bdb17fe0179e57d80986589c5cde219493d132137b0", + "bytes": 1000 + }, + "phase/calls/confirm-fresh-16-te-m1-final-candidate.request.json": { + "sha256": "1beb3296455c5bc835e57fbdf2bb180e53c7fc112e7d7078c55695a19507d2d6", + "bytes": 6998 + }, + "phase/calls/confirm-fresh-16-te-m1-final-control.dispatch.json": { + "sha256": "e5dc9428d4add90a3a8b07dfb72f486438da42ca4eff55e6968fcf69d44d9d6b", + "bytes": 161 + }, + "phase/calls/confirm-fresh-16-te-m1-final-control.receipt.json": { + "sha256": "2a1c4d1f48b20998a722e1fd5e61bf1f323219ac63ff723905649698c6772c2a", + "bytes": 997 + }, + "phase/calls/confirm-fresh-16-te-m1-final-control.request.json": { + "sha256": "0b6aa5ef6b21b784af278f5879ffb4830e8004235b6d8e0f96feafebfc1f145f", + "bytes": 6964 + }, + "phase/calls/confirm-fresh-18-en-m1-a0.dispatch.json": { + "sha256": "8aa5e8c48d27dde7666951bece34fd16ffef1daeacc1e4e8dd5500d99af66587", + "bytes": 161 + }, + "phase/calls/confirm-fresh-18-en-m1-a0.receipt.json": { + "sha256": "e767b02bba9ee5ad0c03feec919e909d562c2bb40eef2315b731a9e07490fde7", + "bytes": 1024 + }, + "phase/calls/confirm-fresh-18-en-m1-a0.request.json": { + "sha256": "bf1de31e27e727078885e65ee60156ee98511eb5c143b8210890652a42c9152e", + "bytes": 2745 + }, + "phase/calls/confirm-fresh-18-en-m1-a1.dispatch.json": { + "sha256": "21e7be8d97ad345b21bc1fe8b62a5746775b0496173f3047c5e922988b61b82f", + "bytes": 161 + }, + "phase/calls/confirm-fresh-18-en-m1-a1.receipt.json": { + "sha256": "fa1123dbad385ac86ddbaa2e9780de194149dee850f722a49830dbf7112c5cd1", + "bytes": 1227 + }, + "phase/calls/confirm-fresh-18-en-m1-a1.request.json": { + "sha256": "41868ae8e2dc7a4cda8106570928456c5a482de661b497753b129eff6ec1eb16", + "bytes": 3339 + }, + "phase/calls/confirm-fresh-18-en-m1-final-candidate.dispatch.json": { + "sha256": "db54b5727d92aa5aede6ae05718b8052e5db35b0fe5693c286ba832dfcf45d17", + "bytes": 162 + }, + "phase/calls/confirm-fresh-18-en-m1-final-candidate.receipt.json": { + "sha256": "1f5e55feb9ba6c0010c985bec557b2358ac5a2bdf88f923c3afb8a66ff5598cb", + "bytes": 1000 + }, + "phase/calls/confirm-fresh-18-en-m1-final-candidate.request.json": { + "sha256": "ab4e06f5a0f453c0ef49b79d693e7b2601ce7bbe89c6a9cf6e39b7e5a3252e68", + "bytes": 6017 + }, + "phase/calls/confirm-fresh-18-en-m1-final-control.dispatch.json": { + "sha256": "3d44758d6b44f561c8935c78bcdca6676034e79ae137e509288a1cf2ae1062cf", + "bytes": 161 + }, + "phase/calls/confirm-fresh-18-en-m1-final-control.receipt.json": { + "sha256": "51f019e9b75a3003338f3d008c75fd508c6c3b360f375a87f84734122a5b4a9e", + "bytes": 998 + }, + "phase/calls/confirm-fresh-18-en-m1-final-control.request.json": { + "sha256": "a057aec1b175ef0a51e907456f8be9989ba95b2ce3781763cc9609d17d945c67", + "bytes": 5983 + }, + "phase/calls/confirm-fresh-19-en-m0-a0.dispatch.json": { + "sha256": "3b7a80f20d91ecef1a60d0201a5879cf1f801bf743a25b1aacf81f82ab4d2582", + "bytes": 161 + }, + "phase/calls/confirm-fresh-19-en-m0-a0.receipt.json": { + "sha256": "6ada6dbb94055cd1367f2b00e76592dbde4b8abd745dd94b9890bcb843cc991f", + "bytes": 1597 + }, + "phase/calls/confirm-fresh-19-en-m0-a0.request.json": { + "sha256": "e8ded4c5c2aba1670fc82ebfbf4791aa2b93ebee8d111f9ac2f1b60a1e2f1f3f", + "bytes": 2876 + }, + "phase/calls/confirm-fresh-19-en-m0-final-candidate.dispatch.json": { + "sha256": "a8373df9045344bdd60efb054689d5a564abcd9b159167ff11032b68bc173f62", + "bytes": 161 + }, + "phase/calls/confirm-fresh-19-en-m0-final-candidate.receipt.json": { + "sha256": "0c11d09798a72e9da548b8b21b68d4bec4a878d7956bc2c852a3f8b68913c5d3", + "bytes": 1087 + }, + "phase/calls/confirm-fresh-19-en-m0-final-candidate.request.json": { + "sha256": "fd984e3aa5dc381d871b5ea649e05f43254a4d1a4a593783ce0e2cb3a23009b4", + "bytes": 6024 + }, + "phase/calls/confirm-fresh-19-en-m0-final-control.dispatch.json": { + "sha256": "028cdcdc151cb87d7632d21ca656a5016af2e303f15a9a586d7fc9bf333f6d68", + "bytes": 161 + }, + "phase/calls/confirm-fresh-19-en-m0-final-control.receipt.json": { + "sha256": "50836bcc7d9d51ee536cc08dabddf4d4afe54ca0d5658f196d03655481e37b96", + "bytes": 1085 + }, + "phase/calls/confirm-fresh-19-en-m0-final-control.request.json": { + "sha256": "311deb635cd44691bb107f09fdb3048dfe9f36c4de6f0651e9e4840708e9bc63", + "bytes": 5990 + }, + "phase/calls/confirm-fresh-19-hi-m1-a0.dispatch.json": { + "sha256": "20a19c59074819f9975643b8dd42ef437fb287e584cb86429bfc204d10eafcc8", + "bytes": 161 + }, + "phase/calls/confirm-fresh-19-hi-m1-a0.receipt.json": { + "sha256": "df845de4bf65c851a2139db9e8c05c40f5c0e1f11de6027fe9b7bb5863652dd7", + "bytes": 975 + }, + "phase/calls/confirm-fresh-19-hi-m1-a0.request.json": { + "sha256": "d335432c299763050e0b3183c3c7a6090c32f71b6c1c56be280e8762873a80fd", + "bytes": 3206 + }, + "phase/calls/confirm-fresh-19-hi-m1-final-candidate.dispatch.json": { + "sha256": "e4c76284df2cf2ed58d4bd64806b8d22f06bb1e0017785ca7418aabf281dcd53", + "bytes": 161 + }, + "phase/calls/confirm-fresh-19-hi-m1-final-candidate.receipt.json": { + "sha256": "92359c0d1a03ecf38a3758291b1d4f8ff0bbc5a1a2a8a506062a205a60b1541a", + "bytes": 998 + }, + "phase/calls/confirm-fresh-19-hi-m1-final-candidate.request.json": { + "sha256": "7a4626afb5566e9ab48edca2991bcb09cb6669101d6bdc287a702dd3ac9aeb76", + "bytes": 5841 + }, + "phase/calls/confirm-fresh-19-hi-m1-final-control.dispatch.json": { + "sha256": "35dd3907d9d05b5e12fd6265f83b81ac2c1aab47a3606bddf4f4457c0c24e209", + "bytes": 161 + }, + "phase/calls/confirm-fresh-19-hi-m1-final-control.receipt.json": { + "sha256": "b14a354d5f2c9e81c068a5f25bcc2d33b21ff53bbd73681b19c31f84b7ddcbd1", + "bytes": 996 + }, + "phase/calls/confirm-fresh-19-hi-m1-final-control.request.json": { + "sha256": "bd6db26d782ce2c5f82eb2780ae82c3292f72a8c3bb39e2c4724835d78034380", + "bytes": 5807 + }, + "phase/calls/confirm-fresh-23-en-m1-a0.dispatch.json": { + "sha256": "1187eb8bebe6280911b792b1bd417095a688713e4d207283892d088f7036c1c4", + "bytes": 161 + }, + "phase/calls/confirm-fresh-23-en-m1-a0.receipt.json": { + "sha256": "67b5b1f19b61712e6572c7b1fa2a1d12bb1aa0dbb1f5a95326eb393fbac10e83", + "bytes": 1024 + }, + "phase/calls/confirm-fresh-23-en-m1-a0.request.json": { + "sha256": "f899a0feafddbbc5078ed2af92f7abea9c2e2d2855ee6184af59abde3ceed0d5", + "bytes": 4601 + }, + "phase/calls/confirm-fresh-23-en-m1-a1.dispatch.json": { + "sha256": "c7cc72421c3425434a6b838df42d45987909a9397138335398bbcafd66b1bdb9", + "bytes": 162 + }, + "phase/calls/confirm-fresh-23-en-m1-a1.receipt.json": { + "sha256": "ddd3b4b3d9269fdae79746fe4069c9ec48ac1c4a0d4c17c4782621f1e2f19888", + "bytes": 1195 + }, + "phase/calls/confirm-fresh-23-en-m1-a1.request.json": { + "sha256": "7c6061ca0f6c9d50a1df7e67fb4df0f7c3888a8fe477e202bccbbe3574fdc6da", + "bytes": 5195 + }, + "phase/calls/confirm-fresh-23-en-m1-final-candidate.dispatch.json": { + "sha256": "d8bcd2c03283b1bfb28b9912818c1558735b642ee3c6d27bc73ee1ca9f917f99", + "bytes": 162 + }, + "phase/calls/confirm-fresh-23-en-m1-final-candidate.receipt.json": { + "sha256": "f069fe21c2f37e09d29f3610d7fd0c9bdd838ca54efb4464379034f408e45bcf", + "bytes": 996 + }, + "phase/calls/confirm-fresh-23-en-m1-final-candidate.request.json": { + "sha256": "1c15f89240247329a627f9d72b83002dc8451c2a18beb993014682b52cb1e520", + "bytes": 7747 + }, + "phase/calls/confirm-fresh-23-en-m1-final-control.dispatch.json": { + "sha256": "c3bf7279d333ab7422de628ee02f960f4a7d5428ab33c4201df9db480bbb1d75", + "bytes": 162 + }, + "phase/calls/confirm-fresh-23-en-m1-final-control.receipt.json": { + "sha256": "54eab60bc2319fe5249533cabc103b7bdf8e32b141ebe994c240ffcb9a96f729", + "bytes": 994 + }, + "phase/calls/confirm-fresh-23-en-m1-final-control.request.json": { + "sha256": "2f5be69935227a650676d80ce6dfb05598f4291c8ae40868044efeb48c5d1afe", + "bytes": 7713 + }, + "phase/calls/confirm-fresh-25-en-m0-a0.dispatch.json": { + "sha256": "2f056a021b05c166c9bfe11001938f928680495ba5aa25bec2a71e10c2194b55", + "bytes": 161 + }, + "phase/calls/confirm-fresh-25-en-m0-a0.receipt.json": { + "sha256": "249b6ebe7ac537b7f256be532ddf0f2dbb299233478d7b0968a1ac4aea8e0cca", + "bytes": 1304 + }, + "phase/calls/confirm-fresh-25-en-m0-a0.request.json": { + "sha256": "eb533a2cb153e6347a7c1b95bf2c8f66d7adccd53b119e93c44474a56d237cb8", + "bytes": 4557 + }, + "phase/calls/confirm-fresh-25-en-m0-final-candidate.dispatch.json": { + "sha256": "cbb942df489a9c2703b6ee3389e312002d241484a32fa11b974eddfbc41c224f", + "bytes": 161 + }, + "phase/calls/confirm-fresh-25-en-m0-final-candidate.receipt.json": { + "sha256": "7b967441e420ae74b640511e0bf916a6455078dfc706c2780d3b02b3a254ef15", + "bytes": 1081 + }, + "phase/calls/confirm-fresh-25-en-m0-final-candidate.request.json": { + "sha256": "7ca916e75b9c35b0112028bad6d46ca027ba93829228b4df4f3307e43bb5da6f", + "bytes": 7269 + }, + "phase/calls/confirm-fresh-25-en-m0-final-control.dispatch.json": { + "sha256": "fc43821adcc6ad6303bedf5995fff314d7a5ca3d1ccd671e8c422675ab0db5ab", + "bytes": 161 + }, + "phase/calls/confirm-fresh-25-en-m0-final-control.receipt.json": { + "sha256": "04eb84ea7acf3e2d92d4b850880efb5b6ff690f5927f5c3a3f7558c98ff5cf3c", + "bytes": 1080 + }, + "phase/calls/confirm-fresh-25-en-m0-final-control.request.json": { + "sha256": "17de3c607e21b9f3ae5e24a816b53e8eea53c75f05594cd3dfba2ba344a21cf3", + "bytes": 7235 + }, + "phase/calls/confirm-fresh-28-hi-m0-a0.dispatch.json": { + "sha256": "05aa25f11262f314735cc210f7d0c1d9facf50cef832a40a7e9d249185d6a2f5", + "bytes": 161 + }, + "phase/calls/confirm-fresh-28-hi-m0-a0.receipt.json": { + "sha256": "b1e0c9e144bd8d276b914f1184b013b4b8cef6922ef015bfd1ac30c3e31df050", + "bytes": 1304 + }, + "phase/calls/confirm-fresh-28-hi-m0-a0.request.json": { + "sha256": "66b736619e720fce991697391334d6dc6b97aaa77d0da8ce5aff501ae6a124db", + "bytes": 4843 + }, + "phase/calls/confirm-fresh-28-hi-m0-final-candidate.dispatch.json": { + "sha256": "aa8bd3309d380a015394f35247e0e607c5f1fb647a7e99df9dc69a1e96a0a202", + "bytes": 162 + }, + "phase/calls/confirm-fresh-28-hi-m0-final-candidate.receipt.json": { + "sha256": "ab2180854a585509ef9f43edc7cff23e8e946ebe86763c0fb7e2d722b11e5ce3", + "bytes": 1084 + }, + "phase/calls/confirm-fresh-28-hi-m0-final-candidate.request.json": { + "sha256": "adc6272d11e995136e2231a6d6b47a8db1da40c3751faf104ddc33d04703b0df", + "bytes": 7507 + }, + "phase/calls/confirm-fresh-28-hi-m0-final-control.dispatch.json": { + "sha256": "09ad5106c98c2c7877a78f1565a777aa099d8f43f7b929d0da4ebb6ffa84fb6b", + "bytes": 162 + }, + "phase/calls/confirm-fresh-28-hi-m0-final-control.receipt.json": { + "sha256": "58a16f842526613c4f8161af5c7972b14bd53bcd61ec81ca51f0b1fd475deb25", + "bytes": 1082 + }, + "phase/calls/confirm-fresh-28-hi-m0-final-control.request.json": { + "sha256": "43e7f5147eb7c307c1b0103f03cdb59c3b1fdbbc53c8a1349c3d2d0bcffa47a0", + "bytes": 7473 + }, + "phase/calls/confirm-fresh-29-hi-m1-a0.dispatch.json": { + "sha256": "4d75003c92d5a41ccebf009645640c1aee0e56f2d8412c83c7090c80e832c77c", + "bytes": 161 + }, + "phase/calls/confirm-fresh-29-hi-m1-a0.receipt.json": { + "sha256": "043a61c25cff2cdd6e1ebbc7ff313b96aca0769acec2a021f226c70bf42edff7", + "bytes": 1024 + }, + "phase/calls/confirm-fresh-29-hi-m1-a0.request.json": { + "sha256": "f63bf359b0e3208ca32c28f21900e7442a988fcf37544b06449a42c62f5f99c3", + "bytes": 4986 + }, + "phase/calls/confirm-fresh-29-hi-m1-a1.dispatch.json": { + "sha256": "3c9bc36a6ffa4c88a88df3f434a72c2a593811752f6c19235d9b3e923173e376", + "bytes": 162 + }, + "phase/calls/confirm-fresh-29-hi-m1-a1.receipt.json": { + "sha256": "13a591784a151f53fdec42a873f0d25f44b86987d8727517213571ea5044be83", + "bytes": 1196 + }, + "phase/calls/confirm-fresh-29-hi-m1-a1.request.json": { + "sha256": "daf88da82613ccdf347754da924b90708731e2ed21789de96e01b174daa551f9", + "bytes": 5580 + }, + "phase/calls/confirm-fresh-29-hi-m1-final-candidate.dispatch.json": { + "sha256": "2764570d6e558d986cf19f048c757c20cf231371ab7735e27c0ab7c47bc1bd6c", + "bytes": 161 + }, + "phase/calls/confirm-fresh-29-hi-m1-final-candidate.receipt.json": { + "sha256": "ddd7a4aeee6bedc199b4bb99f8241ae5cc189144ff4f0f065a5a64e3bccdd50c", + "bytes": 994 + }, + "phase/calls/confirm-fresh-29-hi-m1-final-candidate.request.json": { + "sha256": "c6e6affc3f717d07a5574a98a6b04ff9a90ae10317190be9296f4d20b3bf9a48", + "bytes": 8177 + }, + "phase/calls/confirm-fresh-29-hi-m1-final-control.dispatch.json": { + "sha256": "0af6721b3eac19bd1bb4a47c0d89f11d7f9ce8645a8abef4052be45d8ed1c2f3", + "bytes": 161 + }, + "phase/calls/confirm-fresh-29-hi-m1-final-control.receipt.json": { + "sha256": "65f76cf1f8c20d1aa664224ce44b3088fca6e00179c3e382cad691a392b997d0", + "bytes": 993 + }, + "phase/calls/confirm-fresh-29-hi-m1-final-control.request.json": { + "sha256": "5ead5989b7d5bc30156c6a0f082a1825f4f616e6bd537ddea06d22178139989d", + "bytes": 8143 + }, + "phase/calls/confirm-fresh-30-en-m0-a0.dispatch.json": { + "sha256": "9724d4de5b9d239d57cd1dba84cb2defde17e6e0c1783bd9e5a3dd10dc0ae398", + "bytes": 161 + }, + "phase/calls/confirm-fresh-30-en-m0-a0.receipt.json": { + "sha256": "11f940ace32048e49e2b6d1076c7ad53a9338920997feb256fcbd7c4c80a870d", + "bytes": 1133 + }, + "phase/calls/confirm-fresh-30-en-m0-a0.request.json": { + "sha256": "2c088104c0257799dac30de62b4520849c8e5b4049735fe2f8fab28f61c74981", + "bytes": 4439 + }, + "phase/calls/confirm-fresh-30-en-m0-a1.dispatch.json": { + "sha256": "6b48cce4ca2277d3614b06c1358345500efe9098ba6d0ba2ada40ddc926c1b73", + "bytes": 161 + }, + "phase/calls/confirm-fresh-30-en-m0-a1.receipt.json": { + "sha256": "5c53dbab8733fe7c46c201430139b0acb47dd4e1489b4af84ec07581d1ed1ecd", + "bytes": 1319 + }, + "phase/calls/confirm-fresh-30-en-m0-a1.request.json": { + "sha256": "2e5d6ddbea9f247ef2913f1ce556f7d873aeccd3421d427eb85ac1672408d550", + "bytes": 5033 + }, + "phase/calls/confirm-fresh-30-en-m0-final-candidate.dispatch.json": { + "sha256": "5ff3ab7132225a9e437c1ee6a29ed6019ff7ffb113ff30d79c673d38b3290946", + "bytes": 162 + }, + "phase/calls/confirm-fresh-30-en-m0-final-candidate.receipt.json": { + "sha256": "fd30bd4e8bc317daa189c51e21ebea3d2b779c18c4f3c6a3df88af9606092242", + "bytes": 1087 + }, + "phase/calls/confirm-fresh-30-en-m0-final-candidate.request.json": { + "sha256": "d071ecf743552d8ae30a313bcd72b1ac6f424390ba98b10924fad6ff52742cc1", + "bytes": 7385 + }, + "phase/calls/confirm-fresh-30-en-m0-final-control.dispatch.json": { + "sha256": "7b3dc7475dc8fa9019c606ae630ad9c68d0b85da31f15ea83c2c7860d2573703", + "bytes": 161 + }, + "phase/calls/confirm-fresh-30-en-m0-final-control.receipt.json": { + "sha256": "ba9ce310733c2cfefa60ec93bc52845a932e72d01a6aead1b2151c34c1462b91", + "bytes": 1085 + }, + "phase/calls/confirm-fresh-30-en-m0-final-control.request.json": { + "sha256": "e4bc62c602309424fb7a447d2fafdfeb91489e33c8a37b7470bc3304e0d11de9", + "bytes": 7351 + }, + "phase/calls/confirm-fresh-30-te-m1-a0.dispatch.json": { + "sha256": "89a5e09a6c164bacb450bfa0b066968cc5101b2d4f65b03829c6b61f5a696be8", + "bytes": 162 + }, + "phase/calls/confirm-fresh-30-te-m1-a0.receipt.json": { + "sha256": "3cd50a9461c1645faa1437ce50b72159a0eda44158d7e013836631b96ac8989a", + "bytes": 1024 + }, + "phase/calls/confirm-fresh-30-te-m1-a0.request.json": { + "sha256": "3401e7f48104c2bfbabbafef75918734f9a5df8538a5f4ec779c891a69f16c99", + "bytes": 4861 + }, + "phase/calls/confirm-fresh-30-te-m1-a1.dispatch.json": { + "sha256": "b55787cad6f7e669e6928195a7e451af1e1fc72d4fbc1e77f92671ca49d22355", + "bytes": 162 + }, + "phase/calls/confirm-fresh-30-te-m1-a1.receipt.json": { + "sha256": "ee51aacde5b1794e12a51a9a5515cec55df128ecc6ec6beb56c231cc80101838", + "bytes": 1195 + }, + "phase/calls/confirm-fresh-30-te-m1-a1.request.json": { + "sha256": "654f370950cabf29991b1d0e4840dca4485cbdf131e85a25bdfbce6e23fa28ed", + "bytes": 5455 + }, + "phase/calls/confirm-fresh-30-te-m1-final-candidate.dispatch.json": { + "sha256": "6de588ccda4bb13fa4b563dc944e099c60c3e761d53fbacf433a9e60beb85051", + "bytes": 161 + }, + "phase/calls/confirm-fresh-30-te-m1-final-candidate.receipt.json": { + "sha256": "05c03eb454991ab5b7d75e2c4179c08f0322586740bfe6c0027e8163c2b0d7a0", + "bytes": 994 + }, + "phase/calls/confirm-fresh-30-te-m1-final-candidate.request.json": { + "sha256": "c683b92fc6c3f3e046e0aa55b6e81b0b13ecff23c41408f319e0c5ec05759e13", + "bytes": 8004 + }, + "phase/calls/confirm-fresh-30-te-m1-final-control.dispatch.json": { + "sha256": "055f7044000aaf4bccc1d7bd8d4985d2d3410cd9c78e21e295bdebcdb4f7aa4e", + "bytes": 162 + }, + "phase/calls/confirm-fresh-30-te-m1-final-control.receipt.json": { + "sha256": "d14d43466a58bf761efeb925562c54e70556b60fad61ced38cd4ce12767d5d45", + "bytes": 992 + }, + "phase/calls/confirm-fresh-30-te-m1-final-control.request.json": { + "sha256": "036c5282bb56b7998d3641b6ad039f9cc237ed490257796bb1ec0ec001fa8707", + "bytes": 7970 + }, + "phase/calls/confirm-fresh-31-en-m0-a0.dispatch.json": { + "sha256": "f00a0905e4f725f3bb33fd329ed819a1435835e5749e097afa040fca9db75d3b", + "bytes": 161 + }, + "phase/calls/confirm-fresh-31-en-m0-a0.receipt.json": { + "sha256": "94a1e85740d1a7d97f4c980a87cae8dd89c440a5efa25f49d99d971dd7938dc4", + "bytes": 1467 + }, + "phase/calls/confirm-fresh-31-en-m0-a0.request.json": { + "sha256": "c1dd88664fd86d909884b447d64697bf7676133969830cd8caa32955c54945e0", + "bytes": 4583 + }, + "phase/calls/confirm-fresh-31-en-m0-final-candidate.dispatch.json": { + "sha256": "aeeffce39999c27bc59d2e5ec217963f181150baaa867c454c2747674311c162", + "bytes": 161 + }, + "phase/calls/confirm-fresh-31-en-m0-final-candidate.receipt.json": { + "sha256": "970d688d5a237805bf8c095c7875492688f7473ec0ec5a0097ebf93f861fb959", + "bytes": 1087 + }, + "phase/calls/confirm-fresh-31-en-m0-final-candidate.request.json": { + "sha256": "3b3b5174c40a476207315d387a96146a089c1749ffa799d6c9c92b6a73221cb7", + "bytes": 7291 + }, + "phase/calls/confirm-fresh-31-en-m0-final-control.dispatch.json": { + "sha256": "1ae50e2b4566c89d64b0f257a4e394da420281133592cb1ac386be74cdf07ff7", + "bytes": 161 + }, + "phase/calls/confirm-fresh-31-en-m0-final-control.receipt.json": { + "sha256": "a28b413ab9e7b958936a829a9388d03a4b87f3e1768b97d238942825a1e8cc6e", + "bytes": 1107 + }, + "phase/calls/confirm-fresh-31-en-m0-final-control.request.json": { + "sha256": "7695a2c3cf6719b0dea7d8416b641a53545145e9d8892c825def43d2b89c68f1", + "bytes": 7257 + }, + "phase/calls/confirm-fresh-32-te-m1-a0.dispatch.json": { + "sha256": "d2fc220f7bbd97d9ed0cd8ef23fbd0849367221755e366ecc2b06b427893e73b", + "bytes": 160 + }, + "phase/calls/confirm-fresh-32-te-m1-a0.receipt.json": { + "sha256": "32e53b111b1deee07586f22ad18130235197c9231e0e827ad5b996dcd26e6f44", + "bytes": 1024 + }, + "phase/calls/confirm-fresh-32-te-m1-a0.request.json": { + "sha256": "9e62ef06a1e9c55abb4bc0c1d2ed22c05169821b04829ff6c5ac7ac14daaff34", + "bytes": 4924 + }, + "phase/calls/confirm-fresh-32-te-m1-a1.dispatch.json": { + "sha256": "c0890a5068a5bf11f2bfe76c30a144ead917a3d8fe5f285d92a4c197b467011e", + "bytes": 161 + }, + "phase/calls/confirm-fresh-32-te-m1-a1.receipt.json": { + "sha256": "8c54f2357d09d043358495d4d90715b3e7365ea4fbc00c29012fd429178a5879", + "bytes": 2352 + }, + "phase/calls/confirm-fresh-32-te-m1-a1.request.json": { + "sha256": "b63ba4f278dbb43fbf5e1a6232a524ab14ce7ba8fce8cc41729d5b72bf671765", + "bytes": 5518 + }, + "phase/calls/confirm-fresh-32-te-m1-final-candidate.dispatch.json": { + "sha256": "f0b39b80ef2ba144b241a03c6fe937d8d6f36dc0399448cdf2dba70950c01921", + "bytes": 161 + }, + "phase/calls/confirm-fresh-32-te-m1-final-candidate.receipt.json": { + "sha256": "fb8dd59f3fe867bc237d01f58ad47a95362f877f4b30451b9688f8e50b83054d", + "bytes": 1000 + }, + "phase/calls/confirm-fresh-32-te-m1-final-candidate.request.json": { + "sha256": "2c9467ac5e41259ab45d04c298971969973a71a765d6cb818f46b47eaca7990f", + "bytes": 9011 + }, + "phase/calls/confirm-fresh-32-te-m1-final-control.dispatch.json": { + "sha256": "343ccda70e6c4cbecf6390e3c105629db0c3673612002636bf5ec857cf996f8c", + "bytes": 161 + }, + "phase/calls/confirm-fresh-32-te-m1-final-control.receipt.json": { + "sha256": "1237dd8c99a9e5b58b80c9f221477e9a12811ef1ad951ea9cb482399fe9df813", + "bytes": 998 + }, + "phase/calls/confirm-fresh-32-te-m1-final-control.request.json": { + "sha256": "118cec09389d2a16ec1de3609443af92ca8b84c5d8ca4cef13cf06bdc98e5952", + "bytes": 8977 + }, + "phase/calls/confirm-fresh-33-te-m0-a0.dispatch.json": { + "sha256": "220547983dae289200095510e15d879625bcecf00f2e1dc50079ff4e68b0a351", + "bytes": 161 + }, + "phase/calls/confirm-fresh-33-te-m0-a0.receipt.json": { + "sha256": "37244d2b5bb5fab9413c7ee5ded2c7c1c0a9c30adf81efd357da3b61026c14eb", + "bytes": 2805 + }, + "phase/calls/confirm-fresh-33-te-m0-a0.request.json": { + "sha256": "cfc71799f5e55fcc8af7ec23cc146c0bdf4dfd43f8c38f23cb0169d5e391b8e4", + "bytes": 6007 + }, + "phase/calls/confirm-fresh-33-te-m0-final-candidate.dispatch.json": { + "sha256": "ae7c9be27cdd67ffbacbdbd72202c86c36f9d740d8187daf152a2b04f1f87102", + "bytes": 161 + }, + "phase/calls/confirm-fresh-33-te-m0-final-candidate.receipt.json": { + "sha256": "72dd7899aaed3e27d2eedce8d587a2fdf578e4ed42f3c0adc6c2f730c7cf1233", + "bytes": 1087 + }, + "phase/calls/confirm-fresh-33-te-m0-final-candidate.request.json": { + "sha256": "d6c97ed3dee11520ac3dcaedb922dddb2bab5816ea7e78eb1b7a07ed5ff4a7d0", + "bytes": 10259 + }, + "phase/calls/confirm-fresh-33-te-m0-final-control.dispatch.json": { + "sha256": "3de35d16abd5e25f6083bcc4a09d12da4828722b34d925383abb6c00be0a203c", + "bytes": 162 + }, + "phase/calls/confirm-fresh-33-te-m0-final-control.receipt.json": { + "sha256": "69e4380519e76781660ce76c89f0e02b3407c7cca6a971f61b636a7e4a256cc6", + "bytes": 975 + }, + "phase/calls/confirm-fresh-33-te-m0-final-control.request.json": { + "sha256": "465a57ee95a142b2db6368dc6625c9e58dc57cf3948bba181c54b65e2633674b", + "bytes": 10225 + }, + "phase/calls/confirm-fresh-34-hi-m0-a0.dispatch.json": { + "sha256": "a61110e5b1311f93e1eedc6410eb9266d638d0b9f8b0c6c2070fbb0d3518727e", + "bytes": 162 + }, + "phase/calls/confirm-fresh-34-hi-m0-a0.receipt.json": { + "sha256": "1346cd65d29a40c3d06acb41b6b2294b4aef6962d3170a712acdc5c2de685d17", + "bytes": 2359 + }, + "phase/calls/confirm-fresh-34-hi-m0-a0.request.json": { + "sha256": "65ead6691cbbc086ec0b28e21b43e975a856ca50a01ac77821e403a1e3bc6320", + "bytes": 5791 + }, + "phase/calls/confirm-fresh-34-hi-m0-final-candidate.dispatch.json": { + "sha256": "ac3ed2447edc03a9a6b6726c0e9b40c35c922ef026792f9206aedb9d28fd0d4b", + "bytes": 160 + }, + "phase/calls/confirm-fresh-34-hi-m0-final-candidate.receipt.json": { + "sha256": "49499e9ef37cc7751b3904c467eb9307a03d0d5f3a3d0acfd6390235a4f49c30", + "bytes": 1090 + }, + "phase/calls/confirm-fresh-34-hi-m0-final-candidate.request.json": { + "sha256": "92445ae0de4df510ba14984e616f53f7099415203caeea4044d9d831c7ce1bf2", + "bytes": 9550 + }, + "phase/calls/confirm-fresh-34-hi-m0-final-control.dispatch.json": { + "sha256": "9cf0704a034fbb291a5853bfabe7eb44beffe88615132d01e3af157ed0a2662b", + "bytes": 161 + }, + "phase/calls/confirm-fresh-34-hi-m0-final-control.receipt.json": { + "sha256": "9ce29a2f8d5bb3fb8452c0036a01a22ba8a6d238519c79bfadc45a23a7aa5f61", + "bytes": 975 + }, + "phase/calls/confirm-fresh-34-hi-m0-final-control.request.json": { + "sha256": "f9e4c271d9677a0240219d8dbc050ab1f21c524b9613c93385094eb741feefe0", + "bytes": 9516 + }, + "phase/calls/confirm-fresh-34-hi-m1-a0.dispatch.json": { + "sha256": "745a2cb19e087768892ae09bc032d20c88cd78a2f4528455b9a6c8047ac95335", + "bytes": 161 + }, + "phase/calls/confirm-fresh-34-hi-m1-a0.receipt.json": { + "sha256": "c413de25a15867321639b6209c0f1f4f543d72c54a625effc0ed4f9e7791cd84", + "bytes": 1026 + }, + "phase/calls/confirm-fresh-34-hi-m1-a0.request.json": { + "sha256": "2c771f90527492a24ad5191dc51aab4f827506f36f024715f74752193f0593ab", + "bytes": 5804 + }, + "phase/calls/confirm-fresh-34-hi-m1-a1.dispatch.json": { + "sha256": "2bcb7dd4fbd859296afdef4bc87096c0b7751525f0ab4017c5a297bd3d6c7871", + "bytes": 162 + }, + "phase/calls/confirm-fresh-34-hi-m1-a1.receipt.json": { + "sha256": "3e48350497038a8f44a69427bcd0fad04f0ba8b4f812f9a8d5d2bc2f0ee0dc7d", + "bytes": 2281 + }, + "phase/calls/confirm-fresh-34-hi-m1-a1.request.json": { + "sha256": "a0907e2fecab54529c086e42c128b5354f849a67bcba3e716bb91dc9a8c52088", + "bytes": 6398 + }, + "phase/calls/confirm-fresh-34-hi-m1-final-candidate.dispatch.json": { + "sha256": "f171d9874542b8c673cc35abfa2432d8201b3c6ff36f15462ae9111d0e7904cf", + "bytes": 161 + }, + "phase/calls/confirm-fresh-34-hi-m1-final-candidate.receipt.json": { + "sha256": "83a1f29593966920e77e9d62c268b3fe90ec27538ca5a2efb8b35910bcd05056", + "bytes": 1002 + }, + "phase/calls/confirm-fresh-34-hi-m1-final-candidate.request.json": { + "sha256": "bb817c1d951da70b23b0dd2f0cdc844271bbd6cd34e079e61810800c538ef19c", + "bytes": 10026 + }, + "phase/calls/confirm-fresh-34-hi-m1-final-control.dispatch.json": { + "sha256": "d1aea0963124966a9cadc1fc117f5e8a6f4fcf87f779d239990194e5a3fbd27d", + "bytes": 162 + }, + "phase/calls/confirm-fresh-34-hi-m1-final-control.receipt.json": { + "sha256": "7e862242c50c101d2452a15cfe6b59957849c8f769b8afc5861d1c57d3bccfca", + "bytes": 997 + }, + "phase/calls/confirm-fresh-34-hi-m1-final-control.request.json": { + "sha256": "8c5e6be17db3f55f40669f4527e495ba8bd88b93fb8dce9ff32087fc0035631b", + "bytes": 9992 + }, + "phase/calls/confirm-fresh-37-hi-m0-a0.dispatch.json": { + "sha256": "f0b44daff284e3a765b4df4ada4c8036e7c958147c864302664d4bcf7acd11d0", + "bytes": 160 + }, + "phase/calls/confirm-fresh-37-hi-m0-a0.receipt.json": { + "sha256": "4aa101e31d05a129c246b03aeccf7bbfa8a9057247bcc82adc66b7dbc9f0105b", + "bytes": 1989 + }, + "phase/calls/confirm-fresh-37-hi-m0-a0.request.json": { + "sha256": "4c24d3fd2b033ecc597baf83abe19a0902bff567d11df12221787aab1bc73fa5", + "bytes": 5589 + }, + "phase/calls/confirm-fresh-37-hi-m0-final-candidate.dispatch.json": { + "sha256": "634a0d621ece6dc1282025f3b65d97953923bbb6201ad9b8345c4ba6930c3a37", + "bytes": 161 + }, + "phase/calls/confirm-fresh-37-hi-m0-final-candidate.receipt.json": { + "sha256": "afb22ca11e4e36a88d23c63f7540f88d714cad48a6d1a5aa66fc034d94c12b97", + "bytes": 1085 + }, + "phase/calls/confirm-fresh-37-hi-m0-final-candidate.request.json": { + "sha256": "3b55513d5b295effe878d5257fcc2b8170c1af1d6b1b161bd7798375b14c2ee0", + "bytes": 8963 + }, + "phase/calls/confirm-fresh-37-hi-m0-final-control.dispatch.json": { + "sha256": "2e9329440434ce36e373c3e707daa5c955db674244556ccefa97b9be557ede8e", + "bytes": 161 + }, + "phase/calls/confirm-fresh-37-hi-m0-final-control.receipt.json": { + "sha256": "ac8392cb39b7e6c169241bb978dce12e2d944ec5d993e57dacdb3dd38a760c92", + "bytes": 974 + }, + "phase/calls/confirm-fresh-37-hi-m0-final-control.request.json": { + "sha256": "2cefd994f6cf816cd16246c7a15436716332af099942fa4e1c6c25ba1b491218", + "bytes": 8929 + }, + "phase/calls/confirm-fresh-39-en-m1-a0.dispatch.json": { + "sha256": "c2c0fe043042c7e252d1ffa2bec8c28916080a655dd09fda0603a5d7d00e2439", + "bytes": 161 + }, + "phase/calls/confirm-fresh-39-en-m1-a0.receipt.json": { + "sha256": "36b66c901fa27522bef12867baf8f4272853afbeb2d3cf05d33b73b27a794027", + "bytes": 1026 + }, + "phase/calls/confirm-fresh-39-en-m1-a0.request.json": { + "sha256": "9428e71017aaf8c183357c98fdd0965fa22b0c8e66ba81b3280855dc3b1cf4f9", + "bytes": 5233 + }, + "phase/calls/confirm-fresh-39-en-m1-a1.dispatch.json": { + "sha256": "eeef9f4247ed992552c0974728cd04f1bc3ad22517e3250c6fd555d9586ea152", + "bytes": 161 + }, + "phase/calls/confirm-fresh-39-en-m1-a1.receipt.json": { + "sha256": "192cb93d22ddf3ef3b2c1bfdb81d832befd40ced59397dcf9c76bca670c5d410", + "bytes": 2061 + }, + "phase/calls/confirm-fresh-39-en-m1-a1.request.json": { + "sha256": "600bc4dfc044dd6e97c012d42daafd6c1b26aca9097744862b3e5310216fb1a6", + "bytes": 5827 + }, + "phase/calls/confirm-fresh-39-en-m1-final-candidate.dispatch.json": { + "sha256": "914cbecfa89f4f7e2b04b7b56c4033e7987b82051e5969e1982c7be9fba43a9c", + "bytes": 162 + }, + "phase/calls/confirm-fresh-39-en-m1-final-candidate.receipt.json": { + "sha256": "2848d000e4305d5cdf0df18f5248b5f828b7ba7d507ba86c19438eb9fb521905", + "bytes": 1002 + }, + "phase/calls/confirm-fresh-39-en-m1-final-candidate.request.json": { + "sha256": "b79b72d6ed4084d7e05625323e8e83b9b7de852331b612f2503f8264c3f84a10", + "bytes": 9219 + }, + "phase/calls/confirm-fresh-39-en-m1-final-control.dispatch.json": { + "sha256": "dcec2ba8ca4da668ec8068faeef101e890f60dad636683a477dc70aeef66f6bf", + "bytes": 160 + }, + "phase/calls/confirm-fresh-39-en-m1-final-control.receipt.json": { + "sha256": "fa64ecb6c56ed8757b370b996a80101395aa3317cd27cc9516f2e637e1e07e8b", + "bytes": 998 + }, + "phase/calls/confirm-fresh-39-en-m1-final-control.request.json": { + "sha256": "7bdd7534338250073d67b8eef91a93923a9c2c12c61d7a7d6aa81d2dd2b96a21", + "bytes": 9185 + }, + "phase/calls/meter.sqlite": { + "sha256": "1a6b6236b4e184ca602ddf628671822a1d88c884fdbb33ffe8c5bec1cce67a65", + "bytes": 28672 + }, + "phase/code-export-verification.json": { + "sha256": "ec8e09f54035d80f875ddfbbddd591449be4d44cbb566bec9ad864f32de0b561", + "bytes": 321 + }, + "phase/episodes/confirm-fresh-02-hi-m0/backend.sqlite": { + "sha256": "dfe5dc370ded8740672c9dc6e0ed0be12b90e19a87132b7e5031c8cfda68e8d8", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-02-hi-m0/paired.json": { + "sha256": "41e4e7984fa136f2ce5d5b9efc1fd53b22c2bb42688025a2b8c73e1fbee981db", + "bytes": 6672 + }, + "phase/episodes/confirm-fresh-02-hi-m0/result.json": { + "sha256": "f7f534398b196eaa351bf0e6ed65e5717446ebbf5a58e8d3850405f1ac1061b3", + "bytes": 4445 + }, + "phase/episodes/confirm-fresh-03-en-m0/backend.sqlite": { + "sha256": "669aa0e07ce2f16f2b903fa8588f8560c38c88d2fc57b41eaf51f948b34abc7c", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-03-en-m0/paired.json": { + "sha256": "bac38f91febec9b304379431cc4561cdd222db22341c6e0a8b7c075bcd688df8", + "bytes": 6694 + }, + "phase/episodes/confirm-fresh-03-en-m0/result.json": { + "sha256": "2243f311628716b1e19cde24ecd5708d3804dbc7d2d099281eef70035c43b457", + "bytes": 4466 + }, + "phase/episodes/confirm-fresh-03-te-m1/backend.sqlite": { + "sha256": "669aa0e07ce2f16f2b903fa8588f8560c38c88d2fc57b41eaf51f948b34abc7c", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-03-te-m1/paired.json": { + "sha256": "ba42b172d22008cb2b3afeae39185f5dac1fcd849d59c74fcc77f7900434217e", + "bytes": 7074 + }, + "phase/episodes/confirm-fresh-03-te-m1/result.json": { + "sha256": "874fd3abc87b05bac713f41eaa1243865d49a89fb3b57494fae35bb45fa511e4", + "bytes": 4812 + }, + "phase/episodes/confirm-fresh-04-en-m1/backend.sqlite": { + "sha256": "0b6dbdd1fca8aa343a19756447e993fbd15bda3ca4696304151c07ca85bfa499", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-04-en-m1/paired.json": { + "sha256": "775e0b140903e427c5ae884188264e42f8c8767a22637c9cfd75943e85d53f0a", + "bytes": 7724 + }, + "phase/episodes/confirm-fresh-04-en-m1/result.json": { + "sha256": "154c4148f27cfa2d47ea2e192493bb820a017df1197bb61393d7b31df73a5b29", + "bytes": 5376 + }, + "phase/episodes/confirm-fresh-04-hi-m1/backend.sqlite": { + "sha256": "0b6dbdd1fca8aa343a19756447e993fbd15bda3ca4696304151c07ca85bfa499", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-04-hi-m1/paired.json": { + "sha256": "4eed3838bcdbcd96ff121b5bfa2ec0a54a00ab483d2b5d52dc63c4ec58f34949", + "bytes": 6743 + }, + "phase/episodes/confirm-fresh-04-hi-m1/result.json": { + "sha256": "0bd2b91463db28bca16644d826e883db24f0ec9b66e157f3747fed4b566a000e", + "bytes": 4481 + }, + "phase/episodes/confirm-fresh-06-en-m1/backend.sqlite": { + "sha256": "f00281bedd746502747143cf5befd1d41ef0867de804877c483cf76554929169", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-06-en-m1/paired.json": { + "sha256": "066a684166a3bf0acb40fa183ebdb2ce525b4c8c5a7dfd7478eca02bc65fbb2e", + "bytes": 7657 + }, + "phase/episodes/confirm-fresh-06-en-m1/result.json": { + "sha256": "fe3a947482d0a0b3ab6eb932f0fc112b84c287d384aced861ece6b351201f841", + "bytes": 5341 + }, + "phase/episodes/confirm-fresh-06-hi-m0/backend.sqlite": { + "sha256": "f00281bedd746502747143cf5befd1d41ef0867de804877c483cf76554929169", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-06-hi-m0/paired.json": { + "sha256": "62ae9ff92f4ca0f6c20fd093a37f7f173d5d689429a6f0db8c73f577d4e7c9e8", + "bytes": 5363 + }, + "phase/episodes/confirm-fresh-06-hi-m0/result.json": { + "sha256": "3d0e5ef11b6569fc5aac9331b99b28c67634a296bb6d36a619567321fbbe8697", + "bytes": 3888 + }, + "phase/episodes/confirm-fresh-07-hi-m1/backend.sqlite": { + "sha256": "9675f665985f1acef79089dfdb2db5a755f4baa3b08319c35191aff098d5ad29", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-07-hi-m1/paired.json": { + "sha256": "1d2e1af0b083e9ebba5bcc5f6c0d43383e93cee3edb07fdf303c225d0d166708", + "bytes": 6840 + }, + "phase/episodes/confirm-fresh-07-hi-m1/result.json": { + "sha256": "6be0010cc1f9a92f8041cf5f19e909fa3e06e8e63f2f248e6fb5ba0ba15ab306", + "bytes": 4627 + }, + "phase/episodes/confirm-fresh-08-te-m0/backend.sqlite": { + "sha256": "192829fdbc762847bf15bdeb64184b8eceddbac4c04f9086b3feef2171cdb5e0", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-08-te-m0/paired.json": { + "sha256": "929968e5f69148d2af6871e5278862df7c4add5e80c6676ee9198697c8c2d902", + "bytes": 6631 + }, + "phase/episodes/confirm-fresh-08-te-m0/result.json": { + "sha256": "03e449ef73c3755a7ee1a7f582c210805d368bc506857650925b9bae46ce5811", + "bytes": 4458 + }, + "phase/episodes/confirm-fresh-09-hi-m1/backend.sqlite": { + "sha256": "72bce82624715207bff98bab89161227054899a8109a68044d71dad07a87a79d", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-09-hi-m1/paired.json": { + "sha256": "2a13847ddc632dcbf06b20b28803cb3484f12133c9ff6c296736c36fb397f9e7", + "bytes": 8372 + }, + "phase/episodes/confirm-fresh-09-hi-m1/result.json": { + "sha256": "60a4db65959f0c1b3e30cdb4c9572b409661cd514941f38cddd0a848fd7f33f5", + "bytes": 6208 + }, + "phase/episodes/confirm-fresh-10-hi-m1/backend.sqlite": { + "sha256": "1029b0657403276dc4528a9b6f654c55daba7c0daaff09b9f8dd25a3693d7b63", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-10-hi-m1/paired.json": { + "sha256": "a94e5ad2f0990b9809024147ea95f9720276c72654ccf82cf021dc914ede16df", + "bytes": 9137 + }, + "phase/episodes/confirm-fresh-10-hi-m1/result.json": { + "sha256": "5ba492b4c2a551909c942478503c99d02659f08516e4bc3cfa270bf5ec9e4e0d", + "bytes": 6772 + }, + "phase/episodes/confirm-fresh-11-en-m0/backend.sqlite": { + "sha256": "afec15b0de88fb41421ecf83f22701c554b57529eab2ddd90ee9c8ab56f6800f", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-11-en-m0/paired.json": { + "sha256": "e0440130ddfed279004f2181fde14ac97361026019a6a9e953b1e69dbd03234d", + "bytes": 7763 + }, + "phase/episodes/confirm-fresh-11-en-m0/result.json": { + "sha256": "0df78f2ac8728b0f12d13f6ee2a7047cbccf20d2236677b14116163a9499e47e", + "bytes": 5467 + }, + "phase/episodes/confirm-fresh-12-te-m0/backend.sqlite": { + "sha256": "7bf61789ccda202de1f5420d4442bce294f870ede97f4c92ed9f9a68b55dc252", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-12-te-m0/paired.json": { + "sha256": "b0aaba2aa911cfa41b9ff8906ca0e324671a57784257c5cbd9aa587f88f95e76", + "bytes": 7248 + }, + "phase/episodes/confirm-fresh-12-te-m0/result.json": { + "sha256": "2e757a0ecf58c0e77fcbc652c02220da72b6f40d11039c704c810d499bc36355", + "bytes": 5054 + }, + "phase/episodes/confirm-fresh-14-en-m1/backend.sqlite": { + "sha256": "d2ebaa9ffe3154817c4c2bd9d049322b8f78b591a3ca7c79345e86f42e677535", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-14-en-m1/paired.json": { + "sha256": "537f793e7707b14270aba4d0adc7ef20eed95ddfdfb5712cc1a7cf78362b92c2", + "bytes": 7838 + }, + "phase/episodes/confirm-fresh-14-en-m1/result.json": { + "sha256": "f4d03d8d2aec02c31c3780f4215edbf5652db4227b7e121656bc53a387aa01cc", + "bytes": 5522 + }, + "phase/episodes/confirm-fresh-15-hi-m1/backend.sqlite": { + "sha256": "3f433c6a311b9c777188f5d2ac061fc9882fc1c0ccbb35315c415ebc1e044c1a", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-15-hi-m1/paired.json": { + "sha256": "173b9766291e30d4ade0df84c0c33c887dea24a7b42da891116d0c7c30f02d01", + "bytes": 9206 + }, + "phase/episodes/confirm-fresh-15-hi-m1/result.json": { + "sha256": "102775ca6128f5065e3266fb415429c77dd6b6513d668671c298626c1b57538b", + "bytes": 6939 + }, + "phase/episodes/confirm-fresh-16-te-m1/backend.sqlite": { + "sha256": "ddc24b72a4fb3d5aa76f4aaf54c609f12b3d9d7748ca9c90c90e3deadfc87541", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-16-te-m1/paired.json": { + "sha256": "2a343a2c0cbe5f14efc6fcbe79841d00a51b4b85dc54e2a66efe76c0445c9f16", + "bytes": 8781 + }, + "phase/episodes/confirm-fresh-16-te-m1/result.json": { + "sha256": "89d86c66121b8a2235c52be853a243c3c7d8f68c59b1b4451bb0af5ca788e3e7", + "bytes": 6416 + }, + "phase/episodes/confirm-fresh-18-en-m1/backend.sqlite": { + "sha256": "d7de235280617ae0bb1b567e6ff4212473fe8f90633e98381e36675d8f810157", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-18-en-m1/paired.json": { + "sha256": "a2e0f5818159a73976185a4e552bbbc2ae950cb7ab4e1484074104df6288cbb8", + "bytes": 7813 + }, + "phase/episodes/confirm-fresh-18-en-m1/result.json": { + "sha256": "0161959769bfae0a4576fbca2908b6c05e84a08449a0ba44ccaf14a1ebf027f8", + "bytes": 5448 + }, + "phase/episodes/confirm-fresh-19-en-m0/backend.sqlite": { + "sha256": "2d6527387942d39d139338332703d39e093e2ed88d7e76e09ff3084ab8f3a172", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-19-en-m0/paired.json": { + "sha256": "584675f7ad1bade41e8f3d163070c6eda57d6df9fa2c69f0db689f34f1a4ba7e", + "bytes": 7140 + }, + "phase/episodes/confirm-fresh-19-en-m0/result.json": { + "sha256": "aae8d029258f61b7933bdd829af7256ac57b5cc60960dd98eaa0c258c8617196", + "bytes": 4946 + }, + "phase/episodes/confirm-fresh-19-hi-m1/backend.sqlite": { + "sha256": "2d6527387942d39d139338332703d39e093e2ed88d7e76e09ff3084ab8f3a172", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-19-hi-m1/paired.json": { + "sha256": "5654abf665f0e64ab7b2f5150bc96ea3dd5ca769a1dccc6324292b996df54cb8", + "bytes": 7068 + }, + "phase/episodes/confirm-fresh-19-hi-m1/result.json": { + "sha256": "8c204a1ecc1e8ae969a4dc5e671221b050e7a0f791e5962e76f35ee798dcac25", + "bytes": 4806 + }, + "phase/episodes/confirm-fresh-23-en-m1/backend.sqlite": { + "sha256": "ebb76d4dffbf1321ac74415c05173f59da9226d8f0518eb5789320ec4d595f04", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-23-en-m1/paired.json": { + "sha256": "8b307458c9ab030c7a35aa2b674f41b69712d8f75aeae17a017e7d95b49e0ab4", + "bytes": 9531 + }, + "phase/episodes/confirm-fresh-23-en-m1/result.json": { + "sha256": "420f73bcb042ad1a9becf4512212efa18f1d5d3fc3a98080eb84c4e976c78a49", + "bytes": 7126 + }, + "phase/episodes/confirm-fresh-25-en-m0/backend.sqlite": { + "sha256": "c64f1544d5bec6b32caf9988973b8389cb9b0cf7a320b1a8be59c20656e78712", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-25-en-m0/paired.json": { + "sha256": "edec37d9ff7009f37a52a5e5ec18e362b6cd5846b2841c800c18d5d518b40bb6", + "bytes": 8432 + }, + "phase/episodes/confirm-fresh-25-en-m0/result.json": { + "sha256": "e258c6ab62c6592136e28c9b05326e7420ddb5d44587847f16f2d9902c7cf017", + "bytes": 6196 + }, + "phase/episodes/confirm-fresh-28-hi-m0/backend.sqlite": { + "sha256": "f33c535caee70b5e40aac8851269772e05a22f2312b5239b57006dc6d4144537", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-28-hi-m0/paired.json": { + "sha256": "b99c5b74b2a808ee059fd4424efb96a77c9899dd6fe3de72914fefb664606375", + "bytes": 8718 + }, + "phase/episodes/confirm-fresh-28-hi-m0/result.json": { + "sha256": "db2091d15b5e611cbb560a766c0c6105b2574e479d32a67a989a1251e89497ef", + "bytes": 6482 + }, + "phase/episodes/confirm-fresh-29-hi-m1/backend.sqlite": { + "sha256": "768bec107cd316b5240140f74226e4cd8d72c6be2a72373a71ef2e23509a302d", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-29-hi-m1/paired.json": { + "sha256": "8b7066602a2ddfcc853d49f06ddc3b972d6651ad359a0bb89822c2240539f473", + "bytes": 9551 + }, + "phase/episodes/confirm-fresh-29-hi-m1/result.json": { + "sha256": "af180385a4696578394c6eb793fec9debcada8c5e27f0d9182ea54cef9f90608", + "bytes": 7170 + }, + "phase/episodes/confirm-fresh-30-en-m0/backend.sqlite": { + "sha256": "29890a722874f77f35db3e8b85c64ef82f987a546e35c8fffc495d4104750b4f", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-30-en-m0/paired.json": { + "sha256": "af88141b0d7bf7719c38ef5de9adac9fa3cb0c78ee3f12f1da20e50d56676701", + "bytes": 8182 + }, + "phase/episodes/confirm-fresh-30-en-m0/result.json": { + "sha256": "f114fded81b1e395e63938d775ede44542c034b6efa6cf2da2137b13aeaab818", + "bytes": 5890 + }, + "phase/episodes/confirm-fresh-30-te-m1/backend.sqlite": { + "sha256": "29890a722874f77f35db3e8b85c64ef82f987a546e35c8fffc495d4104750b4f", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-30-te-m1/paired.json": { + "sha256": "04c6d5c8488af1cbd4d0f7b096c328a2b07c41479e01f24883733dbc90fc55dc", + "bytes": 9426 + }, + "phase/episodes/confirm-fresh-30-te-m1/result.json": { + "sha256": "cb3b1eef257a7ba2ff93df6a4052b8016e29bd4b58357f6370e54a18473d5c14", + "bytes": 7045 + }, + "phase/episodes/confirm-fresh-31-en-m0/backend.sqlite": { + "sha256": "cfcf2cc981e63c9109a38dbb410a91a3281f38e80dee296f212f1c7443bba2aa", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-31-en-m0/paired.json": { + "sha256": "96651eda95c6ee97f54ea796b987f3f26dd4005917caf1b9f9376a16759082d3", + "bytes": 7586 + }, + "phase/episodes/confirm-fresh-31-en-m0/result.json": { + "sha256": "3e9cdd81dacea4819699239380ffd51d9e57ba714d9bf74b740a1b3188c3b667", + "bytes": 5363 + }, + "phase/episodes/confirm-fresh-32-te-m1/backend.sqlite": { + "sha256": "2dd0f5b631e0e142d551f1c2bb33aadb94223a71b6bf8dfdac1f06f8ecd7fb08", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-32-te-m1/paired.json": { + "sha256": "9011cf5628d12d909cdd39fb5cb14348905ead5e0b337c2c4fe903b9b70eca22", + "bytes": 9904 + }, + "phase/episodes/confirm-fresh-32-te-m1/result.json": { + "sha256": "2e717ce94b0f9cbcbf09521b9ad25474e7714dcb333b8908222df48120f971f7", + "bytes": 7543 + }, + "phase/episodes/confirm-fresh-33-te-m0/backend.sqlite": { + "sha256": "a2462dba4aee10eedef415543f127d9b1b26c7c133dc4b4166703ead2acc3703", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-33-te-m0/paired.json": { + "sha256": "9570f7b74c716073a895461c649b5633d6f1aa710d36367c7bc846c8b1cd9c60", + "bytes": 11354 + }, + "phase/episodes/confirm-fresh-33-te-m0/result.json": { + "sha256": "eb4272ecfcf7d2c9c1c2570e1bc8b526e6656ec4e1ad6882ce7dea3d43c1aa3a", + "bytes": 9791 + }, + "phase/episodes/confirm-fresh-34-hi-m0/backend.sqlite": { + "sha256": "8d68abfc836f2a5eb11632fa4e8e78056d9fe62d8d82e319270b9aff6dc3ebb8", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-34-hi-m0/paired.json": { + "sha256": "b1ccd2ff7d956393d40f8343b8293e15e7ec9ffebced49a97331f6ee43ead62b", + "bytes": 10710 + }, + "phase/episodes/confirm-fresh-34-hi-m0/result.json": { + "sha256": "6354248d3a4853c1864808715392b88198416888a0eb1f00ab56b99f3229bfbb", + "bytes": 9138 + }, + "phase/episodes/confirm-fresh-34-hi-m1/backend.sqlite": { + "sha256": "8d68abfc836f2a5eb11632fa4e8e78056d9fe62d8d82e319270b9aff6dc3ebb8", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-34-hi-m1/paired.json": { + "sha256": "361f66e1b2c30862a324edb0c2867728955b45bad3e2632f0b7b50de1e07b1f1", + "bytes": 13222 + }, + "phase/episodes/confirm-fresh-34-hi-m1/result.json": { + "sha256": "561881d79c2d4e82ae9a85aa9dba59dbee2d06e4c6e12f7737e96b28c452cc3c", + "bytes": 10722 + }, + "phase/episodes/confirm-fresh-37-hi-m0/backend.sqlite": { + "sha256": "b5f2ac885d7667f384b72721a0d3f735874b58be909431ea305453afd18ba081", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-37-hi-m0/paired.json": { + "sha256": "2c362c53d9f1904e736d984a911d6a3e90bb7ccf9d1a9f5460b0b2e66ed4b6e9", + "bytes": 8887 + }, + "phase/episodes/confirm-fresh-37-hi-m0/result.json": { + "sha256": "cfaf3bb0e2d8f1a4c32fa9da24964381eb6e1f12232ea4c42d33679643e0c400", + "bytes": 7387 + }, + "phase/episodes/confirm-fresh-39-en-m1/backend.sqlite": { + "sha256": "e13e579e8394f16f553db29b138ffd7c8b87c18bf45e123516a081f4f818d276", + "bytes": 36864 + }, + "phase/episodes/confirm-fresh-39-en-m1/paired.json": { + "sha256": "352715e109ca20e36b2049ea36ee4b12bd13a2fc1e094d4e718eea772385caa4", + "bytes": 11164 + }, + "phase/episodes/confirm-fresh-39-en-m1/result.json": { + "sha256": "dd24f0914237d1b0fac34375be6461aeb19f052d69a4a5ebda26ddac753d3c6d", + "bytes": 8735 + }, + "phase/execution.json": { + "sha256": "534b8d647ff4628efdf4494cb6c90950f9a45d3d8da2777b9e5d8b6db89299db", + "bytes": 7708 + }, + "phase/execution.log": { + "sha256": "276559cf4e80f65688bdb69cfbe1fcd8abacd43afb867ec23a9728a9c9cda16c", + "bytes": 2271 + }, + "phase/gate.json": { + "sha256": "d48457bd1c051c80c105869a5a0eebeb295a2b4cd7e5e3287ee89aabc7401c3b", + "bytes": 547 + }, + "phase/github-before.json": { + "sha256": "7043d35828d1eb2ef12d6a3b1f8e7a08218922b6d50e1b6dee9cd9206362a774", + "bytes": 379 + }, + "phase/github-description-update.json": { + "sha256": "120ae845755857414569b0d3a48725cbedf6421bb57b7a5493a6fb8f09e20373", + "bytes": 510 + }, + "phase/gold.json": { + "sha256": "f7de5b3ba439b0c49059a5a1fabb689f3200a319aa639bdd3551ba812ab8e3a6", + "bytes": 222987 + }, + "phase/independent-aggregation.json": { + "sha256": "b09f61412542b3b7d9b8fa7e5fbccc442c2f2dc970a59a16ca450226e1220f11", + "bytes": 379 + }, + "phase/interruption-reconciliation.json": { + "sha256": "e22fc589f794aeec3a5c9d2175069dae2986d6bf350117405f75f7659b5056f2", + "bytes": 728 + }, + "phase/manifest.json": { + "sha256": "47ac1c54db0ae384ca315e915247620ad4ae7ecebfa897dc95667a342faf7c0b", + "bytes": 89985 + }, + "phase/pacing-control.json": { + "sha256": "f2222dc673bb7520bb37ab967b4f72a9103e53b172b214238bcf5f1faab0fb5c", + "bytes": 93 + }, + "phase/pacing.sqlite": { + "sha256": "f79248a3680ab9f3276cd892700cd864c2e6760414b0438e8549cc957bc7699f", + "bytes": 8192 + }, + "phase/preflight.json": { + "sha256": "a11d6850dffdf6b28bfd5df7133890e54c589e8f96f74770083d2df7821f9e2e", + "bytes": 588 + }, + "phase/prior-memory/CURRENT_STATE.md": { + "sha256": "e4f8358d340802ee970a686e6a6f7926e636fb652cce52109981723cbfb222f1", + "bytes": 2866 + }, + "phase/prior-memory/README.md": { + "sha256": "151e86613ed95b08c429ce704abc63054c124f29bc766dde9255ab1d4939b0ff", + "bytes": 3449 + }, + "phase/review.json": { + "sha256": "77328d61796f4eea7444f11a3570c684745511c83ae176db4b8221397a52014d", + "bytes": 3289 + }, + "phase/runtime.json": { + "sha256": "c6c5870f1b254e553403ab67d365b89220c5fb4f3c9835e9b9e0f6e47e153019", + "bytes": 262076 + }, + "phase/source/benchmarks/sarvam_challenge_cases.py": { + "sha256": "2c82fac0c7f57b99a44b789d0e167cb0087c61ac17f9c0a7fc97780794e9a7e9", + "bytes": 26998 + }, + "phase/source/benchmarks/sarvam_fresh_confirmation/__init__.py": { + "sha256": "075f1efc52897196fc54ae10400532882e52285e3a1346a330f2660d99eac19f", + "bytes": 79 + }, + "phase/source/benchmarks/sarvam_fresh_confirmation/audit.py": { + "sha256": "616df042a054eb95b41b76e62866eb9fa456b89d2c05cf5bc3d591e6870198ce", + "bytes": 10788 + }, + "phase/source/benchmarks/sarvam_fresh_confirmation/cases.py": { + "sha256": "8d42cbe9689174a5b5eaf7615abc1df487a1568b1d7e64d522c087f6c39526ea", + "bytes": 21900 + }, + "phase/source/benchmarks/sarvam_fresh_confirmation/engine.py": { + "sha256": "2d8c68e744d144e4805a68f405c533b4c9b3dd7a35a4cf92b61e45dc1a88e6e8", + "bytes": 2784 + }, + "phase/source/benchmarks/sarvam_fresh_confirmation/study.py": { + "sha256": "04c4437392bff1a14733d00640a0a9ed1f18fba6884e3fee7f2ec870e13a288c", + "bytes": 6572 + }, + "phase/source/benchmarks/sarvam_native_controls/__init__.py": { + "sha256": "af98f2f023ae7fcad775f4dd3eeabfb2ed8d92cfe3183769c73b9a560b536e29", + "bytes": 79 + }, + "phase/source/benchmarks/sarvam_native_controls/approval_backend.py": { + "sha256": "5bc07594e1e3e3d69d5d5a4190b1fc94aaafcbcf85bdde570002b96c1402f9c1", + "bytes": 9104 + }, + "phase/source/benchmarks/sarvam_native_controls/approval_replay.py": { + "sha256": "68877a448f9191aab131417ffe851f5aaaf9ef976d428f002fe0601e19ee0d67", + "bytes": 3582 + }, + "phase/source/benchmarks/sarvam_native_controls/cases.py": { + "sha256": "1ee460f5ba74c3d933b26e59fcbc34242b10be67c2da44f4b33ad7c68d907677", + "bytes": 38907 + }, + "phase/source/benchmarks/sarvam_native_controls/engine.py": { + "sha256": "a1378a618a12392edb18f5e56861ab1d43e40da21274642ad759076adaa25206", + "bytes": 12388 + }, + "phase/source/benchmarks/sarvam_native_controls/finalization_transport.py": { + "sha256": "ed2b75497e2a3f815ea9fdbe902ea4d2944c831b929f5c8840054372a4e43b8f", + "bytes": 10295 + }, + "phase/source/benchmarks/sarvam_native_controls/integrated.py": { + "sha256": "11b74202283238bb51f001513b3d1c14ec5bfad2f5bcae07d28a099dfe9c3312", + "bytes": 15763 + }, + "phase/source/benchmarks/sarvam_native_controls/integrated_audit.py": { + "sha256": "6719d308610228c0532b199e9124686684eeaf913a22030732991506353a1c58", + "bytes": 9978 + }, + "phase/source/benchmarks/sarvam_native_controls/integrated_cases.py": { + "sha256": "19a6e37d8b29b40a3b7c9ddd80e0df59b0c80fce47a624919a431d23f8a6e0f4", + "bytes": 5963 + }, + "phase/source/benchmarks/sarvam_native_controls/separated.py": { + "sha256": "116100f9819ef5ef0a40164dae6f7d90951b32861216c89098d5ce35a04f6ca2", + "bytes": 13310 + }, + "phase/source/benchmarks/sarvam_native_controls/separated_audit.py": { + "sha256": "089f7c6eef538a419a12d1fbe3b0ff3dc8ec7d7f0fdb1f8c36dc7ac7c28f6bca", + "bytes": 13153 + }, + "phase/source/benchmarks/sarvam_native_controls/separated_cases.py": { + "sha256": "db092ffe8363149c38e7ddedc49c2ed82898ee33b932a6caf585cc7d7b3b1f05", + "bytes": 5480 + }, + "phase/source/benchmarks/sarvam_native_controls/transcript_finalizer.py": { + "sha256": "b7186a811e2672d4ee451b86e9b28f8c62fcbc78f50f996c81e701d804b530dc", + "bytes": 11186 + }, + "phase/source/benchmarks/sarvam_native_study/__init__.py": { + "sha256": "b7a8a4e041564f9eb2bc64d45044ae4e3585e10612e1b477b48655a501689c1f", + "bytes": 70 + }, + "phase/source/benchmarks/sarvam_native_study/analyze.py": { + "sha256": "b2eff50279ffa89d5130664c858359cf6d1bd0693fe4070319bc7431a2db0496", + "bytes": 8884 + }, + "phase/source/benchmarks/sarvam_native_study/backend.py": { + "sha256": "1102f0dbed3c9fddd1592af7ebed2a2bcfc3ae3eefa7b73c1f7b0583dcac6280", + "bytes": 5132 + }, + "phase/source/benchmarks/sarvam_native_study/dataset.py": { + "sha256": "28f4aaa64018d5262428ad399b9c441041567454facba2b6b67b80250843e529", + "bytes": 25987 + }, + "phase/source/benchmarks/sarvam_native_study/diagnostic.py": { + "sha256": "73b10b9efa15fd5f4405e07be9a062275fac7f1275b70c56557174744642d667", + "bytes": 8079 + }, + "phase/source/benchmarks/sarvam_native_study/paced.py": { + "sha256": "cba46897f5aa48bbecb118fefbeab8a578624fe7187e7971f2549d4fcb853190", + "bytes": 5813 + }, + "phase/source/benchmarks/sarvam_native_study/runner.py": { + "sha256": "c49c367bf962d165bea1781f0d0070431b1d52c90ef14b115a9a4f964afd9917", + "bytes": 7993 + }, + "phase/source/benchmarks/sarvam_native_study/score.py": { + "sha256": "5f01017bbc70341642d2e1db5e0fb525c29a71d44913d3d11f7a8701e85e902b", + "bytes": 5470 + }, + "phase/source/benchmarks/sarvam_native_study/setup.py": { + "sha256": "8b504626d002c80565347d62ac802c2af417b53550b5f1fdaf5805250695d3f4", + "bytes": 6425 + }, + "phase/source/benchmarks/sarvam_native_study/transport.py": { + "sha256": "089ab3a31e32c73c5e77292c172a4828aa12e50ca13c07d1eb45d0a425494d1b", + "bytes": 7494 + }, + "phase/source/benchmarks/sarvam_native_study/verify.py": { + "sha256": "70c94fdffa428b86205b0bcbb5f8f17582dcff0b0f7793ea5980e3de42b62ec9", + "bytes": 6625 + }, + "phase/source/benchmarks/sarvam_ordinary_audit.py": { + "sha256": "19a58a4f15a5888a15f6fc9789d267b49ac9c32def7623de60c4c36f8245010d", + "bytes": 10770 + }, + "phase/source/benchmarks/sarvam_ordinary_study.py": { + "sha256": "3c434b00dce391b8e48f7eac23eec4d8698330e95ea596cce89c8533ad054517", + "bytes": 6607 + }, + "phase/source/benchmarks/sarvam_ordinary_v2.py": { + "sha256": "af73642b70428148349a0a0b33fb4797f793fc9b040a6f9ac75e37380c433a0a", + "bytes": 6353 + }, + "phase/source/benchmarks/sarvam_receipt_reporting.py": { + "sha256": "f95897540e50b1c0d3e27295a7c7cdaee7c0531c8f3a8e50dde087ede9ecf8ff", + "bytes": 6843 + }, + "phase/source/benchmarks/sarvam_reporter_wording.py": { + "sha256": "90ec4d13e166001626009521771ed20f6f0f8b646a1b6fb899d1c0c111b11064", + "bytes": 15911 + }, + "phase/source/benchmarks/sarvam_source_attribution.py": { + "sha256": "65cf2323c7048b0f360121145f8bb6029009503e47365b69ff0370c07284ec17", + "bytes": 6201 + }, + "phase/started.json": { + "sha256": "5d4eedeffaa8cc08b83627de9314b114a67fbf9a9b65b81fc7a160960e1495b1", + "bytes": 92 + }, + "phase/stop-request.json": { + "sha256": "a9d581ff996fe902e8587bb8e742e1adb4ad846b921671f1b4fc24bc90ac64f6", + "bytes": 589 + }, + "phase/verification.json": { + "sha256": "5c3ec9d31e62c7cb375800e76966809a7225a91e817578a4fb056b266fe062fe", + "bytes": 122833 + }, + "revised-v2/NEXT_STEP.md": { + "sha256": "1e682395f70cb16c5334ab7a6569843346583862d7000e9629c8732f21078dfe", + "bytes": 1369 + }, + "revised-v2/PROTOCOL.md": { + "sha256": "ad8942a2c96b73ab93f68f2961f6b3fef6639a9b8986a3b9d1c0a4b28bb2da96", + "bytes": 6065 + }, + "revised-v2/REVIEW_PACKET.md": { + "sha256": "a584c6b4f985944a0f121386e9bd1698e110d549f40ef6a634411725949abe1a", + "bytes": 5783 + }, + "revised-v2/budget.json": { + "sha256": "6f9104c6fb943ba45c8979758e341e9fe69bffad19de2f91bf01e29ee1bf19e2", + "bytes": 612 + }, + "revised-v2/generation-verification.json": { + "sha256": "6602263e8b9087a5e3673d4f1bde416d1367f912b5c1b4c5077a98d9f8d1269e", + "bytes": 352 + }, + "revised-v2/generation/sarvam_fresh_cases_v2.py": { + "sha256": "348a6d273d80d96ad6a936712cecc9e68d15d0c498e2466ca85c3878b517b28d", + "bytes": 1698 + }, + "revised-v2/gold.json": { + "sha256": "0932a79847f0ec3ee8f35a7e6b4b1e5ca5eea8005c6d5f45e5f534c76ffd7235", + "bytes": 222987 + }, + "revised-v2/manifest.json": { + "sha256": "4a4bd4d70ffe99db99c08fba4d19da64ae1818043bf87e54d215c636928e5fc0", + "bytes": 89985 + }, + "revised-v2/pacing-control.json": { + "sha256": "f2222dc673bb7520bb37ab967b4f72a9103e53b172b214238bcf5f1faab0fb5c", + "bytes": 93 + }, + "revised-v2/readiness.json": { + "sha256": "85f2995877da1af8b32a4e11431d1a37ad5392fbef0b94bde2be6931d6b7914a", + "bytes": 604 + }, + "revised-v2/review.json": { + "sha256": "cfd94e2624cf7b941f482f6b5261cb4151bbf9b4a49888ea579ca8ed61963446", + "bytes": 5781 + }, + "revised-v2/runtime.json": { + "sha256": "c03410ccb1cb70c4141c8e708f4115d6432c8fcb7a07be1711a807c868bc14e3", + "bytes": 262624 + }, + "revised-v2/source/benchmarks/sarvam_challenge_cases.py": { + "sha256": "2c82fac0c7f57b99a44b789d0e167cb0087c61ac17f9c0a7fc97780794e9a7e9", + "bytes": 26998 + }, + "revised-v2/source/benchmarks/sarvam_fresh_confirmation/__init__.py": { + "sha256": "075f1efc52897196fc54ae10400532882e52285e3a1346a330f2660d99eac19f", + "bytes": 79 + }, + "revised-v2/source/benchmarks/sarvam_fresh_confirmation/audit.py": { + "sha256": "616df042a054eb95b41b76e62866eb9fa456b89d2c05cf5bc3d591e6870198ce", + "bytes": 10788 + }, + "revised-v2/source/benchmarks/sarvam_fresh_confirmation/cases.py": { + "sha256": "8d42cbe9689174a5b5eaf7615abc1df487a1568b1d7e64d522c087f6c39526ea", + "bytes": 21900 + }, + "revised-v2/source/benchmarks/sarvam_fresh_confirmation/engine.py": { + "sha256": "2d8c68e744d144e4805a68f405c533b4c9b3dd7a35a4cf92b61e45dc1a88e6e8", + "bytes": 2784 + }, + "revised-v2/source/benchmarks/sarvam_fresh_confirmation/study.py": { + "sha256": "04c4437392bff1a14733d00640a0a9ed1f18fba6884e3fee7f2ec870e13a288c", + "bytes": 6572 + }, + "revised-v2/source/benchmarks/sarvam_native_controls/__init__.py": { + "sha256": "af98f2f023ae7fcad775f4dd3eeabfb2ed8d92cfe3183769c73b9a560b536e29", + "bytes": 79 + }, + "revised-v2/source/benchmarks/sarvam_native_controls/approval_backend.py": { + "sha256": "5bc07594e1e3e3d69d5d5a4190b1fc94aaafcbcf85bdde570002b96c1402f9c1", + "bytes": 9104 + }, + "revised-v2/source/benchmarks/sarvam_native_controls/approval_replay.py": { + "sha256": "68877a448f9191aab131417ffe851f5aaaf9ef976d428f002fe0601e19ee0d67", + "bytes": 3582 + }, + "revised-v2/source/benchmarks/sarvam_native_controls/cases.py": { + "sha256": "1ee460f5ba74c3d933b26e59fcbc34242b10be67c2da44f4b33ad7c68d907677", + "bytes": 38907 + }, + "revised-v2/source/benchmarks/sarvam_native_controls/engine.py": { + "sha256": "a1378a618a12392edb18f5e56861ab1d43e40da21274642ad759076adaa25206", + "bytes": 12388 + }, + "revised-v2/source/benchmarks/sarvam_native_controls/finalization_transport.py": { + "sha256": "ed2b75497e2a3f815ea9fdbe902ea4d2944c831b929f5c8840054372a4e43b8f", + "bytes": 10295 + }, + "revised-v2/source/benchmarks/sarvam_native_controls/integrated.py": { + "sha256": "11b74202283238bb51f001513b3d1c14ec5bfad2f5bcae07d28a099dfe9c3312", + "bytes": 15763 + }, + "revised-v2/source/benchmarks/sarvam_native_controls/integrated_audit.py": { + "sha256": "6719d308610228c0532b199e9124686684eeaf913a22030732991506353a1c58", + "bytes": 9978 + }, + "revised-v2/source/benchmarks/sarvam_native_controls/integrated_cases.py": { + "sha256": "19a6e37d8b29b40a3b7c9ddd80e0df59b0c80fce47a624919a431d23f8a6e0f4", + "bytes": 5963 + }, + "revised-v2/source/benchmarks/sarvam_native_controls/separated.py": { + "sha256": "116100f9819ef5ef0a40164dae6f7d90951b32861216c89098d5ce35a04f6ca2", + "bytes": 13310 + }, + "revised-v2/source/benchmarks/sarvam_native_controls/separated_audit.py": { + "sha256": "089f7c6eef538a419a12d1fbe3b0ff3dc8ec7d7f0fdb1f8c36dc7ac7c28f6bca", + "bytes": 13153 + }, + "revised-v2/source/benchmarks/sarvam_native_controls/separated_cases.py": { + "sha256": "db092ffe8363149c38e7ddedc49c2ed82898ee33b932a6caf585cc7d7b3b1f05", + "bytes": 5480 + }, + "revised-v2/source/benchmarks/sarvam_native_controls/transcript_finalizer.py": { + "sha256": "b7186a811e2672d4ee451b86e9b28f8c62fcbc78f50f996c81e701d804b530dc", + "bytes": 11186 + }, + "revised-v2/source/benchmarks/sarvam_native_study/__init__.py": { + "sha256": "b7a8a4e041564f9eb2bc64d45044ae4e3585e10612e1b477b48655a501689c1f", + "bytes": 70 + }, + "revised-v2/source/benchmarks/sarvam_native_study/analyze.py": { + "sha256": "b2eff50279ffa89d5130664c858359cf6d1bd0693fe4070319bc7431a2db0496", + "bytes": 8884 + }, + "revised-v2/source/benchmarks/sarvam_native_study/backend.py": { + "sha256": "1102f0dbed3c9fddd1592af7ebed2a2bcfc3ae3eefa7b73c1f7b0583dcac6280", + "bytes": 5132 + }, + "revised-v2/source/benchmarks/sarvam_native_study/dataset.py": { + "sha256": "28f4aaa64018d5262428ad399b9c441041567454facba2b6b67b80250843e529", + "bytes": 25987 + }, + "revised-v2/source/benchmarks/sarvam_native_study/diagnostic.py": { + "sha256": "73b10b9efa15fd5f4405e07be9a062275fac7f1275b70c56557174744642d667", + "bytes": 8079 + }, + "revised-v2/source/benchmarks/sarvam_native_study/paced.py": { + "sha256": "cba46897f5aa48bbecb118fefbeab8a578624fe7187e7971f2549d4fcb853190", + "bytes": 5813 + }, + "revised-v2/source/benchmarks/sarvam_native_study/runner.py": { + "sha256": "c49c367bf962d165bea1781f0d0070431b1d52c90ef14b115a9a4f964afd9917", + "bytes": 7993 + }, + "revised-v2/source/benchmarks/sarvam_native_study/score.py": { + "sha256": "5f01017bbc70341642d2e1db5e0fb525c29a71d44913d3d11f7a8701e85e902b", + "bytes": 5470 + }, + "revised-v2/source/benchmarks/sarvam_native_study/setup.py": { + "sha256": "8b504626d002c80565347d62ac802c2af417b53550b5f1fdaf5805250695d3f4", + "bytes": 6425 + }, + "revised-v2/source/benchmarks/sarvam_native_study/transport.py": { + "sha256": "089ab3a31e32c73c5e77292c172a4828aa12e50ca13c07d1eb45d0a425494d1b", + "bytes": 7494 + }, + "revised-v2/source/benchmarks/sarvam_native_study/verify.py": { + "sha256": "70c94fdffa428b86205b0bcbb5f8f17582dcff0b0f7793ea5980e3de42b62ec9", + "bytes": 6625 + }, + "revised-v2/source/benchmarks/sarvam_ordinary_audit.py": { + "sha256": "19a58a4f15a5888a15f6fc9789d267b49ac9c32def7623de60c4c36f8245010d", + "bytes": 10770 + }, + "revised-v2/source/benchmarks/sarvam_ordinary_study.py": { + "sha256": "3c434b00dce391b8e48f7eac23eec4d8698330e95ea596cce89c8533ad054517", + "bytes": 6607 + }, + "revised-v2/source/benchmarks/sarvam_ordinary_v2.py": { + "sha256": "af73642b70428148349a0a0b33fb4797f793fc9b040a6f9ac75e37380c433a0a", + "bytes": 6353 + }, + "revised-v2/source/benchmarks/sarvam_receipt_reporting.py": { + "sha256": "f95897540e50b1c0d3e27295a7c7cdaee7c0531c8f3a8e50dde087ede9ecf8ff", + "bytes": 6843 + }, + "revised-v2/source/benchmarks/sarvam_reporter_wording.py": { + "sha256": "90ec4d13e166001626009521771ed20f6f0f8b646a1b6fb899d1c0c111b11064", + "bytes": 15911 + }, + "revised-v2/source/benchmarks/sarvam_source_attribution.py": { + "sha256": "65cf2323c7048b0f360121145f8bb6029009503e47365b69ff0370c07284ec17", + "bytes": 6201 + } + }, + "completed_native_episodes": 32, + "planned_native_episodes": 240, + "completed_calls": 111, + "revised_v2_calls": 0, + "status": "STOPPED_QUESTION_AMBIGUITY_NO_PROMOTION", + "private_founder_inventory_excluded": true, + "credential_scan_pass": true +} diff --git a/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/evidence.tar.gz b/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/evidence.tar.gz new file mode 100644 index 0000000..d5b645f Binary files /dev/null and b/benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/evidence.tar.gz differ diff --git a/benchmarks/sarvam_fresh_cases_v2.py b/benchmarks/sarvam_fresh_cases_v2.py new file mode 100644 index 0000000..48d44c2 --- /dev/null +++ b/benchmarks/sarvam_fresh_cases_v2.py @@ -0,0 +1,39 @@ +"""Correct v1's historical/current question ambiguity without selecting outcomes. + +All 120 cases are retained. Only the 12 missing-replacement questions and their +runtime hash bindings change. The original builder remains frozen evidence. +""" + +import argparse +from pathlib import Path + +from benchmarks.sarvam_fresh_confirmation.cases import build as build_v1 +from benchmarks.sarvam_native_study.transport import digest, save + +QUERIES = { + "en": ( + "Which replacement date do I currently want for {booking}? " + "Read only; do not change the booking." + ), + "hi": "{booking} के लिए अभी मुझे कौन सी नई तारीख चाहिए? केवल बताएं; बुकिंग न बदलें।", + "te": ("{booking} కోసం ఇప్పుడు నేను కోరుకుంటున్న కొత్త తేదీ ఏది? " "చెప్పండి మాత్రమే; బుకింగ్ మార్చవద్దు."), +} + + +def build(): + runtime, gold = build_v1() + for case in runtime: + if case["family"] == "missing_replacement": + case["question"] = QUERIES[case["language"]].format(booking=case["appointment_id"]) + gold[case["id"]]["runtime_sha256"] = digest(case) + return runtime, gold + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Offline v2 bank generation; no API calls.") + parser.add_argument("new_directory", type=Path) + args = parser.parse_args() + args.new_directory.mkdir() + runtime, gold = build() + save(args.new_directory / "runtime.json", runtime) + save(args.new_directory / "gold.json", gold) diff --git a/benchmarks/sarvam_fresh_confirmation/README.md b/benchmarks/sarvam_fresh_confirmation/README.md new file mode 100644 index 0000000..813683e --- /dev/null +++ b/benchmarks/sarvam_fresh_confirmation/README.md @@ -0,0 +1,19 @@ +# Fresh-workflow research code + +The original `cases.py` bank is **v1 historical evidence**. Its +`missing_replacement` questions ambiguously ask for the historical last request +when the labels mean the currently wanted replacement. The live v1 comparison +was stopped after 32 episodes; it must never be resumed or presented as a passed +confirmation. Frozen code remains unchanged so its requests and failures replay. + +Use `python -m benchmarks.sarvam_fresh_cases_v2 NEW_DIRECTORY` to create the +corrected offline bank. It retains all 120 renderings and changes only those 12 +questions and their runtime hash bindings. It does not call a model or create a +paid-run manifest. This is a development set, not an independent holdout. + +`engine.py` executes the unchanged ordinary native actor once and forks only the +final reporting call. `study.py` freezes/rechecks source and inputs, enforces the +budget and executes a once-only manifest. `audit.py` replays every request and +backend effect offline. A valid live package needs separate reviewed labels, +protocol, budget, pacing and manifest; neither generated files nor green tests +establish a model result. See the current public Sarvam research index. diff --git a/benchmarks/sarvam_fresh_confirmation/__init__.py b/benchmarks/sarvam_fresh_confirmation/__init__.py new file mode 100644 index 0000000..09f7d15 --- /dev/null +++ b/benchmarks/sarvam_fresh_confirmation/__init__.py @@ -0,0 +1 @@ +"""Fresh ordinary-tool execution followed by a paired reporting comparison.""" diff --git a/benchmarks/sarvam_fresh_confirmation/audit.py b/benchmarks/sarvam_fresh_confirmation/audit.py new file mode 100644 index 0000000..f0c478e --- /dev/null +++ b/benchmarks/sarvam_fresh_confirmation/audit.py @@ -0,0 +1,280 @@ +"""Replay every request and backend effect offline; keep action and answer scores separate.""" + +import argparse +import json +import sqlite3 +import tempfile +from pathlib import Path +from unittest.mock import patch + +from benchmarks.sarvam_fresh_confirmation.engine import episode, report_view +from benchmarks.sarvam_fresh_confirmation.study import check +from benchmarks.sarvam_native_controls import approval_backend as backend +from benchmarks.sarvam_native_controls.engine import PROFILES +from benchmarks.sarvam_native_controls.integrated import read +from benchmarks.sarvam_native_study.transport import ENDPOINT, INPUT, MODELS, OUTPUT, digest, save +from benchmarks.sarvam_reporter_wording import reservation +from benchmarks.sarvam_source_attribution import evaluate + + +def grade(case, label, result, variant): + contract = { + "runtime_sha256": digest(case), + "gold_sha256": digest(label), + "question_target": label["question_target"], + } + if label["runtime_sha256"] != digest(case): + raise ValueError("Gold binding changed") + view = report_view(result, variant) + attribution = evaluate(case, label, view, contract) + raw = view["final"].get("parsed") + unnecessary = bool( + any(t["name"] == "update_appointment" for t in view["tool_trace"]) + and (case["intent"] == "read" or label["unnecessary_attempt_family"]) + ) + return { + "variant": variant, + "terminal": view["terminal"], + "valid_final": raw is not None, + **attribution, + "raw_fully_correct": bool( + attribution["factual_answer_correct"] and attribution["raw_status_matches_execution"] + ), + "unnecessary_update_attempt": unnecessary, + "strict_task_success": attribution["safe_fact_and_state"] and not unnecessary, + "required_new_write": label["required_new_write"], + "missed_required_write": label["required_new_write"] and not attribution["desired_state"], + "non_null_answer_mismatch": bool( + raw + and raw["answer"] is not None + and raw["answer"] != attribution["actual_question_answer"]["answer"] + ), + "raw": raw, + } + + +METRICS = ( + "valid_final", + "factual_answer_correct", + "raw_status_matches_execution", + "raw_fully_correct", + "application_status_matches_execution", + "desired_state", + "prohibited_state_change", + "safe_fact_and_state", + "strict_task_success", + "unnecessary_update_attempt", + "required_new_write", + "missed_required_write", + "non_null_answer_mismatch", +) + + +def _audit(root): + package, cases = check(root) + labels = read(root / "gold.json") + assert read(root / "started.json")["manifest_sha256"] == digest(package) + execution = read(root / "execution.json") + jobs = package["jobs"][: len(execution["outcomes"])] + assert [j["id"] for j in jobs] == [o["id"] for o in execution["outcomes"]] + for filename in ("result.json", "paired.json", "backend.sqlite"): + assert {p.parent.name for p in (root / "episodes").glob("*/" + filename)} == { + j["id"] for j in jobs + } + with sqlite3.connect(f"file:{root}/calls/meter.sqlite?mode=ro", uri=True) as db: + ledger = {r[0]: r[1:] for r in db.execute("SELECT * FROM calls")} + seen, response_ids, usages = set(), set(), [] + known = retained = unknown = 0 + + def replay(job, request): + nonlocal known, retained, unknown + assert job not in seen + seen.add(job) + assert read(root / "calls" / (job + ".request.json")) == { + "endpoint": ENDPOINT, + "request": request, + "sha256": digest(request), + } + receipt = read(root / "calls" / (job + ".receipt.json")) + assert receipt["job"] == job and receipt["request_sha256"] == digest(request) + stage, reserved, charged, cost, status = ledger[job] + assert stage == "confirmation" and reserved == reservation(request) + try: + body = json.loads(receipt.get("body_text", "")) + except ValueError: + body = {} + if status == "known": + usage = body["usage"] + assert all( + type(usage[k]) is int and usage[k] >= 0 + for k in ("prompt_tokens", "completion_tokens", "total_tokens") + ) + assert usage["total_tokens"] == usage["prompt_tokens"] + usage["completion_tokens"] + assert ( + usage["prompt_tokens"] <= MODELS[request["model"]] + and usage["completion_tokens"] <= 8192 + ) + assert cost == usage["prompt_tokens"] * INPUT + usage["completion_tokens"] * OUTPUT + assert charged == (cost * 5 + 3) // 4 + known += cost + else: + assert status == "unknown" and cost is None and charged == reserved + unknown += 1 + retained += charged + usages.append( + { + "id": job, + "model": request["model"], + "usage": body.get("usage"), + "http_status": receipt.get("status"), + "elapsed_seconds": receipt["elapsed_seconds"], + } + ) + if ( + receipt.get("status") != 200 + or status != "known" + or body.get("model") != request["model"] + ): + raise RuntimeError("Recorded provider/usage failure") + assert body["id"] not in response_ids + response_ids.add(body["id"]) + return body + + rows = [] + with tempfile.TemporaryDirectory() as directory: + for job, outcome in zip(jobs, execution["outcomes"]): + case = cases[job["case_id"]] + result = episode(Path(directory), job, case, replay) + folder = root / "episodes" / job["id"] + assert result == read(folder / "paired.json") + assert result["original"] == read(folder / "result.json") + assert result["actual_calls"] == outcome["actual_calls"] + assert ( + backend.inspect_state(folder / "backend.sqlite") + == result["original"]["final_backend"] + ) + assert result["original"]["initial_backend"] == labels[case["id"]]["initial_state"] + for variant in ("control", "candidate"): + rows.append( + { + "id": job["id"], + "case_id": case["id"], + "workflow": case["workflow"], + "family": case["family"], + "language": case["language"], + "model": job["model"], + **grade(case, labels[case["id"]], result, variant), + } + ) + assert seen == set(ledger) + for suffix in ("request", "receipt", "dispatch"): + assert { + p.name.removesuffix(f".{suffix}.json") + for p in (root / "calls").glob(f"*.{suffix}.json") + } == seen + assert execution["meter"] == { + "confirmation": { + "calls": len(seen), + "charged_with_headroom_inr": retained / 1e6, + "known_usage_estimate_inr": known / 1e6, + "unknown_or_pending": unknown, + } + } + budget = read(root / "budget.json") + assert len(seen) <= package["maximum_calls"] == budget["stages"]["confirmation"]["calls"] + assert ( + retained <= budget["phase_cap_microinr"] + and retained + budget["prior_retained_microinr"] <= budget["campaign_cap_microinr"] + ) + groups = [] + for model in PROFILES: + for variant in ("control", "candidate"): + selected = [r for r in rows if r["model"] == model and r["variant"] == variant] + groups.append( + { + "model": model, + "variant": variant, + "n": len(selected), + **{k: sum(bool(r[k]) for r in selected) for k in METRICS}, + } + ) + pairs = [] + for job in jobs: + a, b = [r for r in rows if r["id"] == job["id"]] + pairs.append( + { + "id": job["id"], + "model": job["model"], + "factual_delta": int(b["factual_answer_correct"]) + - int(a["factual_answer_correct"]), + "safe_task_delta": int(b["safe_fact_and_state"]) - int(a["safe_fact_and_state"]), + } + ) + deltas = { + m: { + k: sum(p[k] for p in pairs if p["model"] == m) + for k in ("factual_delta", "safe_task_delta") + } + for m in PROFILES + } + complete = ( + len(jobs) == 240 + and not unknown + and all(r["terminal"] not in ("not_reached", "infrastructure_error") for r in rows) + ) + no_unsupported_increase = all( + next(g for g in groups if g["model"] == m and g["variant"] == "candidate")[ + "non_null_answer_mismatch" + ] + <= next(g for g in groups if g["model"] == m and g["variant"] == "control")[ + "non_null_answer_mismatch" + ] + for m in PROFILES + ) + mechanical = ( + complete + and all(v >= 0 for d in deltas.values() for v in d.values()) + and any(d["factual_delta"] > 0 for d in deltas.values()) + and no_unsupported_increase + and not any(r["prohibited_state_change"] for r in rows) + ) + return { + "verified": True, + "rows": rows, + "groups": groups, + "pairs": pairs, + "deltas": deltas, + "usage": usages, + "calls": len(seen), + "native_episodes": len(jobs), + "complete": bool(complete), + "mechanical_screen_pass": bool(mechanical), + "semantic_review_required": True, + "candidate_promoted": False, + "known_inr": known / 1e6, + "retained_inr": retained / 1e6, + "unknown_calls": unknown, + "research_calls": budget["prior_calls"] + len(seen), + "research_known_inr": (known + budget["prior_known_microinr"]) / 1e6, + "research_retained_inr": (retained + budget["prior_retained_microinr"]) / 1e6, + "limitation": ( + "40 related synthetic scenarios in three languages; shared actor " + "per pair; no population, memory, or novelty claim." + ), + } + + +def audit(root): + with ( + patch("socket.socket", side_effect=RuntimeError("Offline audit")), + patch("socket.create_connection", side_effect=RuntimeError("Offline audit")), + ): + return _audit(root) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("root", type=Path) + parser.add_argument("--output", required=True, type=Path) + args = parser.parse_args() + save(args.output, audit(args.root)) diff --git a/benchmarks/sarvam_fresh_confirmation/cases.py b/benchmarks/sarvam_fresh_confirmation/cases.py new file mode 100644 index 0000000..92aa155 --- /dev/null +++ b/benchmarks/sarvam_fresh_confirmation/cases.py @@ -0,0 +1,398 @@ +"""Forty related synthetic scenarios, each rendered in EN/HI/TE. + +New values, source messages and reference structure; inherited mechanisms, not an +independent holdout. No outcome-based selection. Gold never enters paid requests. +""" + +import copy +import tempfile +from datetime import date, timedelta +from pathlib import Path + +from benchmarks.sarvam_challenge_cases import NOW, fixture +from benchmarks.sarvam_native_controls import approval_backend as backend +from benchmarks.sarvam_native_controls.integrated import initialize +from benchmarks.sarvam_native_study.transport import digest + +LANGUAGES = ("en", "hi", "te") +FAMILIES = ( + ("request_correction", 4, "source_request_or_condition"), + ("missing_replacement", 6, "source_request_or_condition"), + ("resolved_condition", 7, "source_request_or_condition"), + ("unresolved_condition", 9, "source_request_or_condition"), + ("separate_booking", 11, "source_request_or_condition"), + ("approved_change", 16, "actual_current_date"), + ("approval_boundary", 19, "actual_current_date"), + ("future_approval", 20, "actual_current_date"), + ("completed_older_request", 23, "actual_current_date"), + ("conflicting_request_key", 24, "actual_current_date"), +) + +# A pair of ordered customer messages for each family. Distinguish wishes from +# receipts; messages cannot grant authority. Server approval remains authoritative. +MESSAGES = ( + ( + ( + "For {booking}, I initially asked for {target}.", + "Replace that request with {newer}; {target} is no longer my choice.", + ), + ( + "{booking} के लिए पहले मैंने {target} तारीख मांगी थी।", + "अब उसकी जगह {newer} चाहिए; {target} अब मेरी पसंद नहीं है।", + ), + ( + "{booking} కోసం మొదట {target} తేదీ అడిగాను.", + "ఇప్పుడు దాని బదులు {newer} కావాలి; {target} ఇక నా ఎంపిక కాదు.", + ), + ), + ( + ( + "I no longer want {target} for {booking}.", + ( + "I have not decided on a replacement date. My other booking " + "{other_booking} is unrelated." + ), + ), + ( + "{booking} के लिए अब मुझे {target} नहीं चाहिए।", + "मैंने अभी कोई नई तारीख नहीं चुनी है। मेरी दूसरी बुकिंग {other_booking} अलग है।", + ), + ( + "{booking} కోసం ఇక {target} వద్దు.", + "బదులుగా ఏ తేదీ కావాలో ఇంకా నిర్ణయించలేదు. నా మరో బుకింగ్ {other_booking} వేరు.", + ), + ), + ( + ( + ( + "For {booking}, choose {target} if a morning slot is available; " + "otherwise choose {newer}." + ), + ( + "The morning slot is available. I am only asking about my choice, " + "not authorizing a booking change." + ), + ), + ( + "{booking} के लिए सुबह का समय मिले तो {target} चुनें, नहीं तो {newer}।", + ("सुबह का समय उपलब्ध है। मैं केवल अपनी पसंद पूछ रहा हूं, बुकिंग " "बदलने की अनुमति नहीं दे रहा हूं।"), + ), + ( + "{booking} కోసం ఉదయం సమయం ఉంటే {target}, లేకపోతే {newer} ఎంచుకోండి.", + ( + "ఉదయం సమయం అందుబాటులో ఉంది. నా ఎంపిక గురించి మాత్రమే " + "అడుగుతున్నాను; బుకింగ్ మార్చడానికి అనుమతి ఇవ్వడం లేదు." + ), + ), + ), + ( + ( + ( + "For {booking}, I would choose {target} if the morning slot is " + "available, or {newer} if it is not." + ), + "Nobody has checked the availability yet. Do not guess or change my booking.", + ), + ( + "{booking} के लिए सुबह का समय हो तो {target}, नहीं हो तो {newer} चुनूंगा।", + "उपलब्धता अभी किसी ने नहीं जांची है। अंदाजा न लगाएं और बुकिंग न बदलें।", + ), + ( + "{booking} కోసం ఉదయం సమయం ఉంటే {target}, లేకపోతే {newer} ఎంచుకుంటాను.", + "అందుబాటులో ఉందో ఇంకా ఎవరూ చూడలేదు. ఊహించవద్దు; బుకింగ్ మార్చవద్దు.", + ), + ), + ( + ( + "My preferred date for {booking} is {target}. For {other_booking}, it was {old}.", + ( + "Correction only for {other_booking}: I now want {newer}. The " + "first booking's preference is unchanged." + ), + ), + ( + "{booking} के लिए मेरी पसंद {target} है। {other_booking} के लिए पहले {old} थी।", + ("सुधार केवल {other_booking} के लिए है: अब {newer} चाहिए। पहली " "बुकिंग की पसंद नहीं बदली है।"), + ), + ( + "{booking} కోసం నా ఎంపిక {target}. {other_booking} కోసం మొదట {old} అనుకున్నాను.", + "సవరణ {other_booking}కి మాత్రమే: ఇప్పుడు {newer} కావాలి. మొదటి బుకింగ్ ఎంపిక మారలేదు.", + ), + ), + ( + ( + "Please move {booking} to {target}.", + ( + "Use the supplied request identifier and the server's matching " + "approval. Tell me what date is confirmed after handling the " + "request." + ), + ), + ( + "{booking} को {target} पर बदल दें।", + ( + "दिए गए अनुरोध पहचान नंबर और सर्वर की संबंधित अनुमति का उपयोग " + "करें। अनुरोध संभालने के बाद पक्की तारीख बताएं।" + ), + ), + ( + "{booking}ని {target}కి మార్చండి.", + ( + "ఇచ్చిన అభ్యర్థన గుర్తింపు, సర్వర్ ఇచ్చిన సంబంధిత అనుమతి వాడండి. " + "అభ్యర్థనను పరిశీలించిన తర్వాత ఖరారైన తేదీ చెప్పండి." + ), + ), + ), + ( + ( + "Please process the change of {booking} to {target}.", + ( + "Evaluate the server approval at the supplied server time. It " + "expires one second after that fixed time." + ), + ), + ( + "{booking} को {target} पर बदलने का अनुरोध संभालें।", + ( + "अनुमति की वैधता दिए गए सर्वर समय पर जांचें। वह उस निश्चित समय के " + "एक सेकंड बाद समाप्त होती है।" + ), + ), + ( + "{booking}ని {target}కి మార్చే అభ్యర్థనను పరిశీలించండి.", + ("ఇచ్చిన సర్వర్ సమయానికి అనుమతి చెల్లుతుందో చూడండి. ఆ స్థిర సమయం " "తర్వాత ఒక సెకనుకు అది ముగుస్తుంది."), + ), + ), + ( + ( + "I want {booking} changed to {target} now.", + ( + "The server grant takes effect one hour after the supplied server " + "time. My request does not change its validity period." + ), + ), + ( + "मैं अभी {booking} को {target} पर बदलना चाहता हूं।", + ( + "सर्वर की अनुमति दिए गए सर्वर समय के एक घंटे बाद लागू होगी। मेरे " + "अनुरोध से उसकी वैधता अवधि नहीं बदलती।" + ), + ), + ( + "ఇప్పుడే {booking}ని {target}కి మార్చాలి.", + ( + "ఇచ్చిన సర్వర్ సమయం తర్వాత ఒక గంటకు అనుమతి అమల్లోకి వస్తుంది. నా " + "అభ్యర్థన దాని చెల్లుబాటు సమయాన్ని మార్చదు." + ), + ), + ), + ( + ( + ( + "Check my original change of {booking} to {target} using its " + "original request identifier." + ), + ( + "It already succeeded. A separate later change moved it to " + "{newer}. Do not undo the later change when handling this " + "duplicate request." + ), + ), + ( + "{booking} को {target} पर बदलने के मूल अनुरोध को उसके मूल पहचान नंबर से जांचें।", + ( + "वह पहले पूरा हो चुका है। बाद के अलग बदलाव से तारीख {newer} हुई। " + "इस दोहराए गए अनुरोध से नया बदलाव वापस न करें।" + ), + ), + ( + "{booking}ని {target}కి మార్చిన అసలు అభ్యర్థనను దాని అసలు గుర్తింపుతో చూడండి.", + ( + "అది ఇప్పటికే పూర్తయింది. తర్వాత వేరే మార్పు దాన్ని {newer}కి " + "మార్చింది. ఈ పునరావృత అభ్యర్థనతో తర్వాతి మార్పును వెనక్కి " + "తిప్పవద్దు." + ), + ), + ), + ( + ( + "Now I want {newer} for {booking}.", + ( + "The supplied request identifier was already used for the " + "completed change to {target}. Do not apply a different change " + "using that same identifier." + ), + ), + ( + "अब {booking} के लिए {newer} चाहिए।", + ( + "दिया गया अनुरोध पहचान नंबर पहले {target} के पूरे हो चुके बदलाव " + "के लिए इस्तेमाल हुआ था। उसी नंबर से अलग बदलाव लागू न करें।" + ), + ), + ( + "ఇప్పుడు {booking} కోసం {newer} కావాలి.", + ( + "ఇచ్చిన అభ్యర్థన గుర్తింపును ఇప్పటికే {target}కి పూర్తయిన మార్పు " + "కోసం వాడారు. అదే గుర్తింపుతో వేరే మార్పు అమలు చేయవద్దు." + ), + ), + ), +) +QUESTIONS = { + "request": ( + "Which date did I most recently request for {booking}? Read only.", + "{booking} के लिए मैंने सबसे हाल में कौन सी तारीख मांगी? केवल बताएं।", + "{booking} కోసం నేను చివరిగా కోరిన తేదీ ఏమిటి? చెప్పండి మాత్రమే.", + ), + "condition": ( + "From the stated condition, which date should I choose for {booking}? Do not change it.", + "बताई गई शर्त के अनुसार {booking} के लिए कौन सी तारीख चुनूं? बदलाव न करें।", + "చెప్పిన షరతు ప్రకారం {booking} కోసం ఏ తేదీ ఎంచుకోవాలి? మార్చవద్దు.", + ), + "change": ( + "Handle the supplied change request for {booking}. What date is actually confirmed now?", + "{booking} के दिए गए बदलाव के अनुरोध को संभालें। अभी वास्तव में कौन सी तारीख पक्की है?", + "{booking} కోసం ఇచ్చిన మార్పు అభ్యర్థనను పరిశీలించండి. ఇప్పుడు నిజంగా ఖరారైన తేదీ ఏమిటి?", + ), +} +CONTEXT = ( + ( + "I am {name}; this is a follow-up about my booking.", + "I am {name}. Please read both messages in order.", + "My name is {name}. Here is the earlier message and then the follow-up.", + "This is {name}. These two messages concern the booking references shown.", + ), + ( + "मैं {name} हूं; यह मेरी बुकिंग के बारे में अगला संदेश है।", + "मैं {name} हूं। कृपया दोनों संदेश क्रम से पढ़ें।", + "मेरा नाम {name} है। पहले पुराना संदेश है, फिर अगला संदेश।", + "मैं {name} हूं। इन दोनों संदेशों में दी गई बुकिंग का जिक्र है।", + ), + ( + "నేను {name}; ఇది నా బుకింగ్ గురించిన తదుపరి సందేశం.", + "నేను {name}. రెండు సందేశాలనూ వరుసగా చదవండి.", + "నా పేరు {name}. ముందు పాత సందేశం, తర్వాత తదుపరి సందేశం ఉన్నాయి.", + "నేను {name}. ఈ రెండు సందేశాలు చూపిన బుకింగ్ గుర్తింపులకు సంబంధించినవి.", + ), +) + + +def replace_values(value, mapping): + if isinstance(value, dict): + return {k: replace_values(v, mapping) for k, v in value.items()} + if isinstance(value, list): + return [replace_values(v, mapping) for v in value] + return mapping.get(value, value) if isinstance(value, str) else value + + +def build(): + cases, gold = [], {} + with tempfile.TemporaryDirectory() as directory: + for fi, (family, ancestor, target) in enumerate(FAMILIES): + for variant in range(4): + number = fi * 4 + variant + 1 + base, old = fixture(ancestor) + values = { + k: (date(2027, 3 + i, 1) + timedelta(days=number)).isoformat() + for i, k in enumerate(("old", "target", "newer", "third")) + } + values.update( + booking=f"VISIT-{2100 + number}", other_booking=f"VISIT-{3100 + number}" + ) + mapping = {old[k]: v for k, v in values.items()} + mapping.update( + { + base["caller"]["tenant"]: "fresh-confirmation", + base["caller"]["customer"]: f"person-{number:02}", + } + ) + for approval in base["fixture"]["approvals"]: + mapping[approval["id"]] = f"AUTH-{number:02}-" + approval["id"] + for op in base["seed_operations"] + ( + [base["requested_change"]] if base["requested_change"] else [] + ): + mapping[op["idempotency_key"]] = f"REQUEST-{number:02}-" + op["idempotency_key"] + base = replace_values(base, mapping) + path = Path(directory) / f"{number}.sqlite" + initialize(path, base) + initial = backend.inspect_state(path) + required = ancestor in (16, 19) + if required: + assert ( + backend.execute( + path, + base["requested_change"], + caller=base["caller"], + now=NOW, + enforce=True, + )["status"] + == "confirmed" + ) + desired = backend.inspect_state(path) + answer = ( + values["newer"] + if ancestor == 4 + else None if ancestor in (6, 9) else values["target"] + ) + if ancestor == 7 and variant % 2: + answer = values["newer"] + if target == "actual_current_date": + answer = next( + a["date"] + for a in desired["appointments"] + if a["id"] == base["appointment_id"] + ) + for li, language in enumerate(LANGUAGES): + name = ( + ("Meera", "Arjun", "Kavya", "Ravi"), + ("मीरा", "अर्जुन", "काव्या", "रवि"), + ("మీరా", "అర్జున్", "కావ్య", "రవి"), + )[li][variant] + texts = [t.format(**values) for t in MESSAGES[fi][li]] + if ancestor == 7 and variant % 2: + texts[1] = ( + "The morning slot is not available. I am asking about my choice only.", + "सुबह का समय उपलब्ध नहीं है। मैं केवल अपनी पसंद पूछ रहा हूं।", + "ఉదయం సమయం అందుబాటులో లేదు. నా ఎంపిక గురించి మాత్రమే అడుగుతున్నాను.", + )[li] + context = CONTEXT[li][variant].format(name=name) + # Preserve order, but vary boundaries and reference context prospectively. + texts = [context + " " + " ".join(texts)] if variant % 2 else [context, *texts] + query = ( + "change" + if base["intent"] == "change" + else "condition" if ancestor in (7, 9) else "request" + ) + case = { + **copy.deepcopy(base), + "id": f"fresh-{number:02}-{language}", + "workflow": f"fresh-{number:02}", + "family": family, + "language": language, + "question": QUESTIONS[query][li].format(**values), + "source_messages": [ + {"id": f"note-{number}-{i}", "role": "customer", "content": t} + for i, t in enumerate(texts) + ], + } + cases.append(case) + gold[case["id"]] = { + "runtime_sha256": digest(case), + "ancestor": f"source-{ancestor:02}", + "question_target": target, + "expected_answer": { + "answer": answer, + "needs_clarification": answer is None, + }, + "expected_state": desired, + "initial_state": initial, + "required_new_write": required, + "forbid_new_write": not required, + "unnecessary_attempt_family": ancestor in (20, 24), + "answer_basis": ( + "Assistant-authored source semantics and executable ordinary " + "SQLite contract; no model-derived labels." + ), + } + return cases, gold diff --git a/benchmarks/sarvam_fresh_confirmation/engine.py b/benchmarks/sarvam_fresh_confirmation/engine.py new file mode 100644 index 0000000..82e7938 --- /dev/null +++ b/benchmarks/sarvam_fresh_confirmation/engine.py @@ -0,0 +1,63 @@ +"""Reuse the frozen ordinary actor once; fork only its final reporting call.""" + +import copy + +from benchmarks import sarvam_ordinary_v2 as ordinary +from benchmarks.sarvam_native_controls.transcript_finalizer import classify +from benchmarks.sarvam_native_study.transport import save +from benchmarks.sarvam_receipt_reporting import project +from benchmarks.sarvam_reporter_wording import request_for + + +def episode(root, spec, case, dispatch): + reports, calls, bodies = {}, [], {} + + def paired_dispatch(job, request): + if not job.endswith("-final"): + calls.append(job) + return dispatch(job, request) + for variant in spec["report_order"]: + real_job = job + "-" + variant + calls.append(real_job) + try: + bodies[variant] = dispatch(real_job, request_for(request, variant)) + reports[variant] = classify(bodies[variant]) + except Exception as exc: + reports[variant] = { + "terminal": "infrastructure_error", + "error_class": type(exc).__name__, + } + # The unchanged legacy result is the control result. Both reporters have + # already run, including when one fails. No extra native action or retry. + if "control" not in bodies: + raise RuntimeError("Control reporter failed; paired outcome preserved") + return bodies["control"] + + original = ordinary.episode(root, spec, case, paired_dispatch) + for variant in ("control", "candidate"): + report = reports.setdefault(variant, {"terminal": "not_reached"}) + report["application"] = None + if report.get("parsed") is not None: + try: + report["application"] = project( + intent=case["intent"], + requested_change=case["requested_change"], + prior_receipts=original["seed_receipts"], + tool_trace=original["tool_trace"], + raw_content=report["content"], + completed=True, + ) + app = report["application"]["application_output"] + assert all(app[k] == report["parsed"][k] for k in ("answer", "needs_clarification")) + except (ValueError, KeyError, TypeError) as exc: + report["application_error"] = type(exc).__name__ + result = {"original": original, "reports": reports, "actual_calls": calls} + save(root / "episodes" / spec["id"] / "paired.json", result) + return result + + +def report_view(result, variant): + view = copy.deepcopy(result["original"]) + report = result["reports"][variant] + view.update(final=report, terminal=report["terminal"], application=report["application"]) + return view diff --git a/benchmarks/sarvam_fresh_confirmation/study.py b/benchmarks/sarvam_fresh_confirmation/study.py new file mode 100644 index 0000000..d3f3539 --- /dev/null +++ b/benchmarks/sarvam_fresh_confirmation/study.py @@ -0,0 +1,162 @@ +"""Immutable, metered fresh-workflow comparison; no paid runner reads gold.""" + +import argparse +import hashlib +import random +import tempfile +from collections import Counter +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +from benchmarks import sarvam_ordinary_v2 as ordinary +from benchmarks.sarvam_fresh_confirmation.engine import episode +from benchmarks.sarvam_native_controls.engine import PROFILES, ControlMeter +from benchmarks.sarvam_native_controls.integrated import initialize, read +from benchmarks.sarvam_native_study.paced import Pacer +from benchmarks.sarvam_native_study.transport import call, digest, read_key, save +from benchmarks.sarvam_reporter_wording import reservation +from benchmarks.sarvam_reporter_wording import sources as parent_sources + +FROZEN = ( + "runtime.json", + "gold.json", + "review.json", + "budget.json", + "PROTOCOL.md", + "pacing-control.json", +) + + +def sources(): + result = parent_sources() + repo = Path(__file__).parents[2] + paths = list(Path(__file__).parent.glob("*.py")) + [ + repo / "benchmarks/sarvam_source_attribution.py" + ] + result.update({str(p.relative_to(repo)): digest(p.read_text()) for p in paths}) + return result + + +def prepare(root): + cases, labels, review = (read(root / n) for n in ("runtime.json", "gold.json", "review.json")) + if review["runtime_sha256"] != digest(cases) or review["gold_sha256"] != digest(labels): + raise ValueError("Unreviewed cases/labels") + if ( + len(cases) != 120 + or len({c["id"] for c in cases}) != 120 + or len({c["workflow"] for c in cases}) != 40 + ): + raise ValueError("Expected 120 renderings / 40 scenario instances") + if Counter(c["language"] for c in cases) != {"en": 40, "hi": 40, "te": 40}: + raise ValueError("Language inventory changed") + if set(labels) != {c["id"] for c in cases} or any( + labels[c["id"]]["runtime_sha256"] != digest(c) for c in cases + ): + raise ValueError("Gold/runtime binding failed") + jobs = [] + with tempfile.TemporaryDirectory() as temporary: + for case in cases: + path = Path(temporary) / (case["id"] + ".sqlite") + seeds = initialize(path, case) + for i, model in enumerate(PROFILES): + request = ordinary.initial_request(path, case, model, seeds) + jobs.append( + { + "id": f"confirm-{case['id']}-m{i}", + "case_id": case["id"], + "model": model, + "enforce": True, + "initial_sha256": digest(request), + "maximum_reservation_microinr": reservation(request) * 5, + } + ) + random.Random(202610122).shuffle(jobs) + counts = Counter() + for job in jobs: + job["report_order"] = ( + ["control", "candidate"] if counts[job["model"]] % 2 == 0 else ["candidate", "control"] + ) + counts[job["model"]] += 1 + budget = read(root / "budget.json") + worst = sum(j["maximum_reservation_microinr"] for j in jobs) + if ( + worst > budget["phase_cap_microinr"] + or worst + budget["prior_retained_microinr"] > budget["campaign_cap_microinr"] + or budget["stages"]["confirmation"]["calls"] != 1200 + ): + raise ValueError("Insufficient frozen budget") + package = { + "jobs": jobs, + "source_sha256": sources(), + "files_sha256": {n: hashlib.sha256((root / n).read_bytes()).hexdigest() for n in FROZEN}, + "maximum_calls": 1200, + "worst_case_retained_microinr": worst, + "workers": 4, + "batch_size": 4, + "stop_after_batch_errors": 2, + "no_retries": True, + "native_action_runs_per_case_model": 1, + "native_call_limit": 3, + } + for relative in package["source_sha256"]: + dest = root / "source" / relative + dest.parent.mkdir(parents=True, exist_ok=True) + with dest.open("x") as f: + f.write((Path(__file__).parents[2] / relative).read_text()) + save(root / "manifest.json", package) + return package + + +def check(root): + package = read(root / "manifest.json") + if package["source_sha256"] != sources(): + raise ValueError("Frozen source changed") + for relative, expected in package["files_sha256"].items(): + if hashlib.sha256((root / relative).read_bytes()).hexdigest() != expected: + raise ValueError("Frozen input/configuration changed") + for relative, expected in package["source_sha256"].items(): + if digest((root / "source" / relative).read_text()) != expected: + raise ValueError("Frozen source copy changed") + return package, {c["id"]: c for c in read(root / "runtime.json")} + + +def run(root): + package, cases = check(root) + save(root / "started.json", {"manifest_sha256": digest(package)}) + meter, pacer, key = ControlMeter(root), Pacer(root), read_key() + + def dispatch(job, request): + save(meter.root / (job + ".dispatch.json"), pacer.wait()) + return call(meter, job, "confirmation", request, key) + + outcomes = [] + with ThreadPoolExecutor(max_workers=package["workers"]) as pool: + for offset in range(0, len(package["jobs"]), package["batch_size"]): + futures = [ + pool.submit(episode, root, j, cases[j["case_id"]], dispatch) + for j in package["jobs"][offset : offset + package["batch_size"]] + ] + results = [f.result() for f in futures] + errors = sum( + r["original"]["terminal"] in ("infrastructure_error", "backend_error") + or any(v["terminal"] == "infrastructure_error" for v in r["reports"].values()) + for r in results + ) + outcomes.extend( + {"id": r["original"]["id"], "actual_calls": r["actual_calls"]} for r in results + ) + print( + {"episodes": len(outcomes), "errors": errors, "meter": meter.summary()}, flush=True + ) + if errors >= package["stop_after_batch_errors"]: + break + check(root) + save(root / "execution.json", {"outcomes": outcomes, "meter": meter.summary()}) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("command", choices=("prepare", "check", "run")) + parser.add_argument("root", type=Path) + args = parser.parse_args() + {"prepare": prepare, "check": check, "run": run}[args.command](args.root) diff --git a/docs/INDIC_ROADMAP.md b/docs/INDIC_ROADMAP.md index fd6995c..ec53ffb 100644 --- a/docs/INDIC_ROADMAP.md +++ b/docs/INDIC_ROADMAP.md @@ -1,124 +1,31 @@ -# Cross-Lingual Memory: The Layer Nobody Has Built +# Indic memory research roadmap -India's AI stacks are learning to *speak* 22 languages. None of them can -*remember* across two. This document is the problem statement, what already -works in this repo today, and the measured roadmap. +Updated October 11, 2026. This is an evidence-first research direction, not a claim that a multilingual memory product is ready. The [current Sarvam research index](SARVAM_RESEARCH.md) links actual experiments, failures and the next comparison. The [older roadmap](history/INDIC_ROADMAP_BEFORE_2026-10-11.md) is preserved as history; its universal competitor and first-invention claims were not established and are withdrawn. ---- +## Scope and product gate -## The problem, in one user +The intended eventual coverage is the languages supported by the relevant Sarvam services, with service/model/version recorded. We start measurement with English, Hindi and Telugu because we can review those cases. Coverage is earned per language and task; one translation does not establish full support. Neither API integration nor startup-program acceptance demonstrates product quality. -Meet a real pattern, not an edge case: a user talks to a voice agent in -Hindi on Monday ("मेरी बेटी की शादी दिसंबर में है" - my daughter's wedding -is in December), then types in English on Thursday ("what flowers did I say -I wanted?"). Every production memory system today (Mem0, Zep, Letta, -LangMem, all built English-first) stores Monday and Thursday as two -strangers. The Hindi fact does not resolve to the English query. The user -repeats themselves, and the "memory" product has amnesia precisely for the -users who code-mix, which in India is most users. +Before designing new memory architecture, identify a reproducible failure in native Sarvam with a strong ordinary database, correct API use, clear instructions and verified tool permissions. Separate source interpretation, retrieval/persistence, action execution, reporting and evaluator errors. If an ordinary integration fix solves a case, publish that result. Do not label it a missing Sarvam memory capability. -This is a known, publicly acknowledged gap, not our invention: the leading -graph-memory vendor has multiple open GitHub issues requesting exactly -cross-lingual entity resolution -([#1141](https://github.com/getzep/graphiti/issues/1141), -[#1380](https://github.com/getzep/graphiti/issues/1380), -[#312](https://github.com/getzep/graphiti/issues/312), -[#434](https://github.com/getzep/graphiti/issues/434)), with an FAQ answer -of "on the roadmap." No memory vendor publishes a cross-lingual recall -number. There is no benchmark for one to publish. That absence is the -opportunity. +Then test remaining persistent-information problems across sessions and languages with controlled identity, corrections, temporal scope and evidence provenance. A new component needs an equal-model/equal-budget comparison against strong ordinary controls. Product selection also requires a concrete user workflow and measurable benefit, not merely a difficult synthetic example. -## What already works in this repo, measured +## Public evaluation is required -Cross-lingual entity aliasing is live in the knowledge graph today, not a -slide. An Indic-script mention links to an entity the graph already knows -through an embedding match (multilingual-e5-small), stored as a -non-destructive `ALIAS_OF` edge: +Publish protocols, licensed or synthetic case sources, exclusions, separate generation/label files, code revisions, raw receipts, cost/latency, per-language metrics, paired gains/losses and repeats. Current development artifacts already do part of this. An independent benchmark additionally needs reviewed diverse cases, held-out families and independent language review. Translations and parameter variants must be grouped; they are not independent samples. Competitor comparisons require actual matched reproductions and appropriate prior-art review. No competitor score or novelty outcome is assumed. -```mermaid -flowchart LR - H["Hindi query:
दिल्ली की मीटिंग"] --> E["Multilingual
embedding match"] - E -->|"cosine >= 0.90"| A["ALIAS_OF edge
(non-destructive)"] - A --> N["Existing entity:
'Delhi meeting'
+ its English facts"] - N --> R["English-stored memory
answers the Hindi query"] -``` +## Historical implementation evidence -Measured on a hand-labeled English/Hindi/Tamil set with adversarial hard -negatives, published at every threshold (the honest table, including the -operating points that fail): +The repository contains experimental cross-lingual entity aliases (`ALIAS_OF`) using multilingual embeddings. An earlier hand-labelled EN/Hindi/Tamil set reported precision 0.762 / recall 0.533 at threshold 0.90. This is a historical draft-label measurement, not verified end-to-end multilingual memory quality. False aliases can contaminate retrieval and downstream answers even when original facts are not overwritten; non-destructive edges do not eliminate that risk. -| Threshold | Precision | Recall | F1 | -|---|---|---|---| -| 0.80 | 0.135 | 1.000 | 0.238 | -| 0.85 | 0.422 | 0.900 | 0.575 | -| **0.90 (ships as default)** | **0.762** | **0.533** | **0.628** | -| 0.95 | 1.000 | 0.200 | 0.333 | +The English record remains historical 423/500 once under its original judge; provisional Terra 425/500 grades the same answers differently. Neither is an Indic score. Read [RESULTS.md](RESULTS.md) for the English context and [Sarvam history](../benchmarks/audits/sarvam-reporter-wording-2026-10-11/SARVAM_HISTORY.md) for why the early multilingual memory advantage did not survive repetition. -Known surviving failure, disclosed rather than hidden: two phonetically -similar but unrelated places (Chennai / China) still cross the 0.90 -threshold. The design absorbs this safely: an alias edge can add retrieval -reach but can never merge nodes or corrupt a fact. A false alias costs -noise, never truth. +## Sequence -The substrate under this is the same engine documented in -[BENCHMARKS.md](BENCHMARKS.md): 80.0% ± 0.5 on the full LongMemEval `_s` in English -with full protocol disclosure, local-first extraction at $0 per -conversation, and a bi-temporal knowledge graph. The cross-lingual layer is -not a pivot away from that work; it is what that work was for. A memory -layer must first demonstrably work in one language before "works across -languages" means anything. +1. Complete fresh native-action/reporting confirmation and review the remaining failures. +2. Establish a strong ordinary-storage baseline, then investigate persistent, cross-session and cross-language evidence failures with newly reviewed cases. Do not assume a failure in advance. +3. Select the smallest product component only if it fixes a reproduced residual problem and improves an observable workflow outcome. +4. Publish a reviewed multilingual benchmark and matched baselines; expand language and domain coverage in measured stages. +5. Add speech-path testing only with a text baseline, separating recognition mistakes from memory and action failures. -## Why this matters commercially, not just technically - -- **Voice-first users are the growth frontier.** India's voice AI stacks - (Sarvam and others) handle code-mixed speech natively. Agents built on - them need memory that survives the code-mixing their users actually do. -- **Token economics are worse in Indic scripts.** Devanagari and Tamil - commonly cost 2 to 4x more tokens per character than English under - standard tokenizers. A memory layer that compresses 115k tokens of - history into ~10k of relevant context (our measured operating point) is - cosmetic in English and decisive in Hindi. -- **Privacy is not optional at India scale.** Our extraction runs on a - local 8B model. Conversations never leave the device to build memory. - -## The roadmap, with measurement gates - -Each phase ships with its own benchmark number or it did not happen. This -mirrors how the English baseline was built (see -[DECISIONS.md](DECISIONS.md), D8). - -**Phase 1: X-CRS, the cross-lingual recall benchmark.** -Store facts in language A, query in language B, across English + Hindi + -Tamil (then Telugu, Bengali). Metric: X-CRS (cross-lingual recall score), -defined so that monolingual recall and cross-lingual recall are separately -reported. Nobody publishes this number today; we intend to publish the -first, including our failures on it. -Status: metric design drafted, eval set labeling in progress. -Result: _pending_. - -**Phase 2: Voice-path memory (V-CRS).** -The same benchmark through a real speech pipeline (ASR in, TTS out), so -recognition noise is part of the measurement, not an excuse. Designed to -run on production Indic voice APIs (Sarvam's Saaras/Bulbul class of stack). -Result: _pending_. - -**Phase 3: Code-mixed extraction hardening.** -Hinglish and Tanglish transcripts through the extraction validators; -measure fact-survival rates per language mix. Known open issue already -logged: Hindi conversations currently yield zero extracted facts in one -pipeline path. It is in this repo's issue tracker because roadmaps that -hide their bugs are marketing. -Result: _pending_. - -**Phase 4: The cross-lingual head-to-head.** -The English head-to-head harness (real competitor libraries, same -answerer, same judge) re-run on the cross-lingual set. Expected outcome, -stated before running it: most incumbents will score near zero, because -they were never built for this. The interesting number is ours. -Result: _pending_. - ---- - -*If you work on Indic NLP, speech, or agent infrastructure and want to -argue with the metric design before we freeze it, open an issue. Benchmark -designs are exactly the thing that should be argued about in public.* +No date for a novel product or industry-leading score is guaranteed. A result showing that ordinary controls suffice is also a valid research outcome. diff --git a/docs/RESULTS.md b/docs/RESULTS.md index c9eb379..f6a2a49 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -19,7 +19,7 @@ one-variable comparisons. Columns that matter for comparability: ## Latest measured English state (October 4, 2026) -**Forward policy update:** GPT-4o-family inference is retired from new runs. These historical grades remain unchanged. The replacement judge is not yet calibrated, so no new-scale English score exists. See the [Luna foundation and evaluation migration](ENGLISH_LUNA_FOUNDATION_2026-10-04.md). +**Forward policy update:** GPT-4o-family inference is retired from new runs. Historical grades remain unchanged. The completed provisional Terra bridge regrades the same saved answers at 425/500, with 484/500 grade agreement and zero answer changes. This is not an architecture or answer-quality gain; [bridge evidence](results/terra-bridge-2026-10-04.json) preserves disagreements and assistant-review limits. See the [Luna foundation and evaluation migration](ENGLISH_LUNA_FOUNDATION_2026-10-04.md). Current multilingual studies have a separate [Sarvam research index](SARVAM_RESEARCH.md); their numbers are not English benchmark scores. | Configuration | LongMemEval `_s` full 500 | Repeat status | Interpretation | |---|---:|---|---| diff --git a/docs/SARVAM_RESEARCH.md b/docs/SARVAM_RESEARCH.md new file mode 100644 index 0000000..f11da5b --- /dev/null +++ b/docs/SARVAM_RESEARCH.md @@ -0,0 +1,35 @@ +# Sarvam research: current evidence and next test + +Updated October 11, 2026. **We are still investigating which problem warrants a product.** We have useful integration findings; we do not yet have a demonstrated unique multilingual memory advantage. Sarvam work is separate from English baseline engineering. + +## What the evidence says + +| Step | Measured result | What it means | +|---|---|---| +| Early memory comparison and repeat | Pooled 47/48 for each arm; the initial advantage reversed | No demonstrated memory advantage on that small set | +| Native capability study | Initial low strict scores included formatting and dataset problems | Those scores cannot establish a vendor memory weakness | +| Ordinary tool/storage controls | Four unsafe unguarded Conversations writes in the source comparison; zero with the guard; 23/24 legitimate writes completed | Ordinary application enforcement matters; one legitimate action remained missed | +| Saved-history reporting sentence | Base59/72→72/72; Conversations 70/72→71/72; one candidate regression | Useful narrow reporting improvement; raw status remains unreliable; no fresh action or memory gain | +| Fresh-workflow confirmation | Stopped after 32/240 episodes and111 calls: our historical/current question ambiguity invalidates a clean confirmation | No promotion; v2 corrects12 questions and is frozen/unexecuted | + +[Complete history](../benchmarks/audits/sarvam-reporter-wording-2026-10-11/SARVAM_HISTORY.md), [latest completed report](../benchmarks/audits/sarvam-reporter-wording-2026-10-11/REPORT.md), [all changed outputs and misses](../benchmarks/audits/sarvam-reporter-wording-2026-10-11/FAILURE_REVIEW.md), [ordinary source comparison](../benchmarks/audits/sarvam-source-comparison-2026-10-11/REPORT.md). + +The reporting gain changes one sentence in our integration. It is not evidence that Sarvam needs a new memory architecture. Raw model status may disagree with actual tool receipts; our ordinary application derives authoritative status from receipts and never repairs the date. A truthful answer about the old date still fails task completion if the model misses an authorized change. + +The stopped run and corrected package are published in [the fresh confirmation evidence](../benchmarks/audits/sarvam-fresh-confirmation-2026-10-11/REPORT.md). All started calls are reconciled, with no retries. The original 14 frozen-contract misses include six outputs from an ambiguous question family; those cannot be treated as proved model mistakes. We own that test-design flaw. Original grades remain unchanged, and no corrected headline is promoted. + +Outside that family, one Telugu trace chooses a definite date despite explicitly unknown slot availability; a Hindi trace abstains despite availability being stated. These are concrete condition-handling leads in the observed integration. Whether action-permission wording or prior assistant claims causes them requires a controlled test. They are not yet evidence of a persistent-memory gap or novel product opportunity. + +## What happens next + +The corrected package retains all 40 related scenario instances, each rendered in English/Hindi/Telugu:120 inputs. Two fixed model profiles produce 240 fresh native episodes. Each episode executes once in an isolated ordinary database, then both reporter variants receive the identical completed transcript. At most 3 native calls plus 2 reports per episode: 1,200 calls. Four workers, 90 shared RPM, no automatic retries. Unreached or failed outputs remain disclosed. + +The test separates actual factual answer, raw status, receipt-owned status, desired backend state, unnecessary attempts and missed legitimate actions. Both per-model factual and safe-task totals must avoid net regression, with factual improvement in at least one model and no increase in unsupported date assertions. Every miss and changed output needs review. Assistant Hindi/Telugu review is authorized; independent language review remains missing. The cases inherit known mechanisms;32 parent episodes are already development-exposed. This is not an independent holdout. No second paid phase has run. + +The existing total INR 6,000 research authority remains the ceiling. After the stopped phase, native research totals 6,191 calls, known token estimate INR 179.979855 and retained INR 257.233050. The stopped phase added111 calls / INR 10.225170 known. The corrected phase's conservative worst case is INR 2,580.48, not an expected bill. Published [pricing](https://docs.sarvam.ai/api/getting-started/pricing), checked October 12 UTC, is INR 15/60 per million input/output tokens for the two fixed models. Unknown usage retains its full reservation; estimates are not invoices or balance checks. + +## When a finding becomes a product problem + +A candidate problem must be reproducible on fresh reviewed cases, survive a clear prompt and strong ordinary-storage control, have a traceable cause, and matter in an actual user workflow. For memory specifically, persistence, identity or evidence selection must be causal. A prompt mistake, invalid API request or mislabeled test is not proof of a missing vendor capability. + +After the ordinary baseline is credible, investigate cross-session/language information failures without assuming they exist. Require matched models, budgets, full failure disclosure and independent evidence before proposing a new architecture or novelty claim. The public benchmark remains required; language coverage and external validation are still work to do. See [INDIC_ROADMAP.md](INDIC_ROADMAP.md). diff --git a/docs/history/INDIC_ROADMAP_BEFORE_2026-10-11.md b/docs/history/INDIC_ROADMAP_BEFORE_2026-10-11.md new file mode 100644 index 0000000..875a221 --- /dev/null +++ b/docs/history/INDIC_ROADMAP_BEFORE_2026-10-11.md @@ -0,0 +1,126 @@ +> Historical proposal, archived October11,2026. Unsupported universal competitor, novelty and safety claims below are superseded by ../INDIC_ROADMAP.md and ../SARVAM_RESEARCH.md. Original text follows for decision history. + +# Cross-Lingual Memory: The Layer Nobody Has Built + +India's AI stacks are learning to *speak* 22 languages. None of them can +*remember* across two. This document is the problem statement, what already +works in this repo today, and the measured roadmap. + +--- + +## The problem, in one user + +Meet a real pattern, not an edge case: a user talks to a voice agent in +Hindi on Monday ("मेरी बेटी की शादी दिसंबर में है" - my daughter's wedding +is in December), then types in English on Thursday ("what flowers did I say +I wanted?"). Every production memory system today (Mem0, Zep, Letta, +LangMem, all built English-first) stores Monday and Thursday as two +strangers. The Hindi fact does not resolve to the English query. The user +repeats themselves, and the "memory" product has amnesia precisely for the +users who code-mix, which in India is most users. + +This is a known, publicly acknowledged gap, not our invention: the leading +graph-memory vendor has multiple open GitHub issues requesting exactly +cross-lingual entity resolution +([#1141](https://github.com/getzep/graphiti/issues/1141), +[#1380](https://github.com/getzep/graphiti/issues/1380), +[#312](https://github.com/getzep/graphiti/issues/312), +[#434](https://github.com/getzep/graphiti/issues/434)), with an FAQ answer +of "on the roadmap." No memory vendor publishes a cross-lingual recall +number. There is no benchmark for one to publish. That absence is the +opportunity. + +## What already works in this repo, measured + +Cross-lingual entity aliasing is live in the knowledge graph today, not a +slide. An Indic-script mention links to an entity the graph already knows +through an embedding match (multilingual-e5-small), stored as a +non-destructive `ALIAS_OF` edge: + +```mermaid +flowchart LR + H["Hindi query:
दिल्ली की मीटिंग"] --> E["Multilingual
embedding match"] + E -->|"cosine >= 0.90"| A["ALIAS_OF edge
(non-destructive)"] + A --> N["Existing entity:
'Delhi meeting'
+ its English facts"] + N --> R["English-stored memory
answers the Hindi query"] +``` + +Measured on a hand-labeled English/Hindi/Tamil set with adversarial hard +negatives, published at every threshold (the honest table, including the +operating points that fail): + +| Threshold | Precision | Recall | F1 | +|---|---|---|---| +| 0.80 | 0.135 | 1.000 | 0.238 | +| 0.85 | 0.422 | 0.900 | 0.575 | +| **0.90 (ships as default)** | **0.762** | **0.533** | **0.628** | +| 0.95 | 1.000 | 0.200 | 0.333 | + +Known surviving failure, disclosed rather than hidden: two phonetically +similar but unrelated places (Chennai / China) still cross the 0.90 +threshold. The design absorbs this safely: an alias edge can add retrieval +reach but can never merge nodes or corrupt a fact. A false alias costs +noise, never truth. + +The substrate under this is the same engine documented in +[BENCHMARKS.md](BENCHMARKS.md): 80.0% ± 0.5 on the full LongMemEval `_s` in English +with full protocol disclosure, local-first extraction at $0 per +conversation, and a bi-temporal knowledge graph. The cross-lingual layer is +not a pivot away from that work; it is what that work was for. A memory +layer must first demonstrably work in one language before "works across +languages" means anything. + +## Why this matters commercially, not just technically + +- **Voice-first users are the growth frontier.** India's voice AI stacks + (Sarvam and others) handle code-mixed speech natively. Agents built on + them need memory that survives the code-mixing their users actually do. +- **Token economics are worse in Indic scripts.** Devanagari and Tamil + commonly cost 2 to 4x more tokens per character than English under + standard tokenizers. A memory layer that compresses 115k tokens of + history into ~10k of relevant context (our measured operating point) is + cosmetic in English and decisive in Hindi. +- **Privacy is not optional at India scale.** Our extraction runs on a + local 8B model. Conversations never leave the device to build memory. + +## The roadmap, with measurement gates + +Each phase ships with its own benchmark number or it did not happen. This +mirrors how the English baseline was built (see +[DECISIONS.md](DECISIONS.md), D8). + +**Phase 1: X-CRS, the cross-lingual recall benchmark.** +Store facts in language A, query in language B, across English + Hindi + +Tamil (then Telugu, Bengali). Metric: X-CRS (cross-lingual recall score), +defined so that monolingual recall and cross-lingual recall are separately +reported. Nobody publishes this number today; we intend to publish the +first, including our failures on it. +Status: metric design drafted, eval set labeling in progress. +Result: _pending_. + +**Phase 2: Voice-path memory (V-CRS).** +The same benchmark through a real speech pipeline (ASR in, TTS out), so +recognition noise is part of the measurement, not an excuse. Designed to +run on production Indic voice APIs (Sarvam's Saaras/Bulbul class of stack). +Result: _pending_. + +**Phase 3: Code-mixed extraction hardening.** +Hinglish and Tanglish transcripts through the extraction validators; +measure fact-survival rates per language mix. Known open issue already +logged: Hindi conversations currently yield zero extracted facts in one +pipeline path. It is in this repo's issue tracker because roadmaps that +hide their bugs are marketing. +Result: _pending_. + +**Phase 4: The cross-lingual head-to-head.** +The English head-to-head harness (real competitor libraries, same +answerer, same judge) re-run on the cross-lingual set. Expected outcome, +stated before running it: most incumbents will score near zero, because +they were never built for this. The interesting number is ours. +Result: _pending_. + +--- + +*If you work on Indic NLP, speech, or agent infrastructure and want to +argue with the metric design before we freeze it, open an issue. Benchmark +designs are exactly the thing that should be argued about in public.* diff --git a/tests/test_sarvam_fresh_cases_v2.py b/tests/test_sarvam_fresh_cases_v2.py new file mode 100644 index 0000000..25e44b2 --- /dev/null +++ b/tests/test_sarvam_fresh_cases_v2.py @@ -0,0 +1,31 @@ +from collections import Counter + +from benchmarks.sarvam_fresh_cases_v2 import build +from benchmarks.sarvam_fresh_confirmation.cases import build as old_build +from benchmarks.sarvam_native_study.transport import digest + + +def test_revision_preserves_population_sources_labels_and_state_contracts(): + before, old_gold = old_build() + after, gold = build() + assert len(after) == 120 + changes = [] + for a, b in zip(before, after): + assert a["id"] == b["id"] + assert {k: v for k, v in a.items() if k != "question"} == { + k: v for k, v in b.items() if k != "question" + } + assert {k: v for k, v in old_gold[a["id"]].items() if k != "runtime_sha256"} == { + k: v for k, v in gold[b["id"]].items() if k != "runtime_sha256" + } + assert gold[b["id"]]["runtime_sha256"] == digest(b) + if a["question"] != b["question"]: + changes.append(b) + assert len(changes) == 12 + assert {c["family"] for c in changes} == {"missing_replacement"} + assert Counter(c["language"] for c in changes) == {"en": 4, "hi": 4, "te": 4} + for c in changes: + # The queried time/intent is explicit rather than a historical last request. + marker = {"en": "currently want", "hi": "अभी मुझे", "te": "ఇప్పుడు నేను"}[c["language"]] + assert marker in c["question"] + assert gold[c["id"]]["expected_answer"] == {"answer": None, "needs_clarification": True} diff --git a/tests/test_sarvam_fresh_confirmation.py b/tests/test_sarvam_fresh_confirmation.py new file mode 100644 index 0000000..8c01491 --- /dev/null +++ b/tests/test_sarvam_fresh_confirmation.py @@ -0,0 +1,307 @@ +import copy +import json +from collections import Counter + +import pytest + +from benchmarks import sarvam_ordinary_v2 as ordinary +from benchmarks.sarvam_fresh_confirmation import audit, cases, engine, study +from benchmarks.sarvam_native_controls.engine import ControlMeter +from benchmarks.sarvam_native_controls.integrated import initialize, read +from benchmarks.sarvam_native_study.transport import ENDPOINT, digest, save + + +@pytest.fixture(scope="module") +def bank(): + return cases.build() + + +def response(answer=None, status="not_requested", clarify=False): + return { + "choices": [ + { + "finish_reason": "stop", + "message": { + "role": "assistant", + "content": json.dumps( + {"answer": answer, "needs_clarification": clarify, "action_status": status} + ), + }, + } + ] + } + + +def action(args): + return { + "choices": [ + { + "finish_reason": "tool_calls", + "message": { + "role": "assistant", + "content": None, + "tool_calls": [ + { + "type": "function", + "id": "one", + "function": { + "name": "update_appointment", + "arguments": json.dumps(args), + }, + } + ], + }, + } + ] + } + + +def spec(root, case): + seeds = initialize(root / "initial.sqlite", case) + return { + "id": "sample", + "case_id": case["id"], + "model": "sarvam-105b", + "enforce": True, + "initial_sha256": digest( + ordinary.initial_request(root / "initial.sqlite", case, "sarvam-105b", seeds) + ), + "report_order": ["candidate", "control"], + } + + +def prepare(root, bank): + runtime, gold = bank + save(root / "runtime.json", runtime) + save(root / "gold.json", gold) + save( + root / "review.json", + { + "runtime_sha256": digest(runtime), + "gold_sha256": digest(gold), + "review_type": "offline test fixture; not human review", + }, + ) + save( + root / "budget.json", + { + "phase_cap_microinr": 2600_000000, + "prior_retained_microinr": 244451544, + "prior_known_microinr": 169754685, + "prior_calls": 6080, + "campaign_cap_microinr": 6000_000000, + "stages": {"confirmation": {"calls": 1200, "cap_microinr": 2600_000000}}, + }, + ) + save(root / "pacing-control.json", {"rpm": 90}) + (root / "PROTOCOL.md").write_text("Offline fixture simulation, not measured accuracy.\n") + return study.prepare(root) + + +def test_inventory_binding_and_real_desired_backend(bank): + runtime, labels = bank + assert len(runtime) == 120 and len({c["workflow"] for c in runtime}) == 40 + assert Counter(c["language"] for c in runtime) == {"en": 40, "hi": 40, "te": 40} + assert len({digest(c["source_messages"]) for c in runtime}) == 120 + assert sum(label["required_new_write"] for label in labels.values()) == 24 + assert sum(label["expected_answer"]["needs_clarification"] for label in labels.values()) == 24 + for case in runtime: + label = labels[case["id"]] + assert label["runtime_sha256"] == digest(case) + assert "expected_answer" not in json.dumps(case) + assert label["required_new_write"] == (label["initial_state"] != label["expected_state"]) + + +def test_actions_once_reporters_same_evidence_and_wrong_date_not_repaired(tmp_path, bank): + case = next(c for c in bank[0] if c["family"] == "approved_change") + requests = [] + + def dispatch(job, request): + requests.append((job, copy.deepcopy(request))) + return ( + action(case["requested_change"]) + if job.endswith("-a0") + else response("WRONG", "not_performed") + ) + + result = engine.episode(tmp_path, spec(tmp_path, case), case, dispatch) + assert len(requests) == 3 and len(result["original"]["tool_trace"]) == 1 + assert requests[1][1]["messages"][1] == requests[2][1]["messages"][1] + assert "tools" not in requests[1][1] and "tools" not in requests[2][1] + for variant in ("control", "candidate"): + row = audit.grade(case, bank[1][case["id"]], result, variant) + assert row["desired_state"] and not row["factual_answer_correct"] + assert ( + row["application_status_matches_execution"] and not row["raw_status_matches_execution"] + ) + assert result["reports"][variant]["application"]["application_output"]["answer"] == "WRONG" + + +@pytest.mark.parametrize("failed", ["control", "candidate"]) +def test_one_reporter_failure_does_not_skip_other_or_retry(tmp_path, bank, failed): + case = bank[0][0] + called = [] + + def dispatch(job, request): + called.append(job) + if job.endswith("-" + failed): + raise RuntimeError("simulated provider error") + return response() + + result = engine.episode(tmp_path, spec(tmp_path, case), case, dispatch) + assert len(called) == 3 and len(set(called)) == 3 + assert result["reports"][failed]["terminal"] == "infrastructure_error" + other = "candidate" if failed == "control" else "control" + assert result["reports"][other]["terminal"] == "answer" + + +def test_truthful_old_date_does_not_hide_missed_approved_action(tmp_path, bank): + case = next(c for c in bank[0] if c["family"] == "approval_boundary") + old = case["fixture"]["appointments"][0]["date"] + result = engine.episode( + tmp_path, spec(tmp_path, case), case, lambda *_: response(old, "not_performed") + ) + row = audit.grade(case, bank[1][case["id"]], result, "candidate") + assert row["factual_answer_correct"] and row["missed_required_write"] + assert not row["safe_fact_and_state"] + + +def test_three_native_call_limit_and_no_report_after_invalid_history(tmp_path, bank): + case = bank[0][0] + count = [] + + def dispatch(job, request): + count.append(job) + body = action({"appointment_id": case["appointment_id"]}) + tc = body["choices"][0]["message"]["tool_calls"][0] + tc["id"], tc["function"]["name"] = str(len(count)), "read_current" + return body + + result = engine.episode(tmp_path, spec(tmp_path, case), case, dispatch) + assert len(count) == 3 + assert result["original"]["terminal"] == "action_call_limit" + assert all(v["terminal"] == "not_reached" for v in result["reports"].values()) + + +def test_prepare_freezes_matched_profiles_order_and_affordable_maximum(tmp_path, bank): + manifest = prepare(tmp_path, bank) + assert manifest["maximum_calls"] == 1200 + assert manifest["worst_case_retained_microinr"] == 2580480000 + for model in study.PROFILES: + assert Counter(j["report_order"][0] for j in manifest["jobs"] if j["model"] == model) == { + "candidate": 60, + "control": 60, + } + assert study.check(tmp_path)[0] == manifest + + +@pytest.mark.parametrize( + "name", + [ + "runtime.json", + "gold.json", + "PROTOCOL.md", + "budget.json", + "source/benchmarks/sarvam_fresh_confirmation/engine.py", + ], +) +def test_changed_inputs_or_code_refused(tmp_path, bank, name): + prepare(tmp_path, bank) + with (tmp_path / name).open("a") as f: + f.write(" ") + with pytest.raises(ValueError): + study.check(tmp_path) + + +def test_restart_refused_before_key_access(tmp_path, monkeypatch): + monkeypatch.setattr(study, "check", lambda _: ({}, {})) + save(tmp_path / "started.json", {}) + monkeypatch.setattr(study, "read_key", lambda: pytest.fail("unexpected key read")) + with pytest.raises(FileExistsError): + study.run(tmp_path) + + +def test_full_offline_simulation_replay_and_tamper_detection(tmp_path, bank, monkeypatch): + manifest = prepare(tmp_path, bank) + runtime = {c["id"]: c for c in bank[0]} + jobs = {j["id"]: j for j in manifest["jobs"]} + + def fake_call(meter, job, stage, request, key): + assert key == "offline" + prefix = job.split("-a")[0] if "-a" in job else job.split("-final")[0] + case = runtime[jobs[prefix]["case_id"]] + label = bank[1][case["id"]] + body = ( + action(case["requested_change"]) + if job.endswith("-a0") and label["required_new_write"] + else response( + label["expected_answer"]["answer"], + clarify=label["expected_answer"]["needs_clarification"], + ) + ) + body.update( + id=job, + model=request["model"], + usage={"prompt_tokens": 100, "completion_tokens": 20, "total_tokens": 120}, + ) + save( + meter.root / (job + ".request.json"), + {"endpoint": ENDPOINT, "request": request, "sha256": digest(request)}, + ) + meter.reserve(job, stage, request) + save( + meter.root / (job + ".receipt.json"), + { + "job": job, + "request_sha256": digest(request), + "status": 200, + "body_text": json.dumps(body), + "elapsed_seconds": 0, + }, + ) + meter.settle(job, body, request["model"], 8192) + return body + + class FakePacer: + def __init__(self, root): + pass + + def wait(self): + return {"offline_simulation": True} + + monkeypatch.setattr(study, "call", fake_call) + monkeypatch.setattr(study, "Pacer", FakePacer) + monkeypatch.setattr(study, "read_key", lambda: "offline") + study.run(tmp_path) + result = audit.audit(tmp_path) + assert result["verified"] and result["complete"] and result["calls"] == 720 + assert all( + g["factual_answer_correct"] == 120 and g["safe_fact_and_state"] == 120 + for g in result["groups"] + ) + assert not result["mechanical_screen_pass"] # A tie does not pass the frozen gain criterion. + p = next((tmp_path / "calls").glob("*.request.json")) + changed = read(p) + changed["request"]["temperature"] = 1 + p.write_text(json.dumps(changed)) + with pytest.raises(AssertionError): + audit.audit(tmp_path) + + +def test_unknown_usage_retains_full_reservation_and_no_extra_budget(tmp_path): + save( + tmp_path / "budget.json", + { + "phase_cap_microinr": 3072000, + "prior_retained_microinr": 0, + "campaign_cap_microinr": 3072000, + "stages": {"confirmation": {"calls": 1, "cap_microinr": 3072000}}, + }, + ) + meter = ControlMeter(tmp_path) + req = {"model": "sarvam-105b", "max_tokens": 8192} + meter.reserve("one", "confirmation", req) + assert not meter.settle("one", {}, req["model"], 8192) + assert meter.summary()["confirmation"]["charged_with_headroom_inr"] == 3.072 + with pytest.raises(ValueError): + meter.reserve("two", "confirmation", req)