From b32f1140a91a5018fe6f3b28ca7aa40f9e423343 Mon Sep 17 00:00:00 2001 From: tzhouam Date: Thu, 13 Aug 2026 16:39:59 +0800 Subject: [PATCH 01/55] Rework Strict review for the official model: deep passes, per-comment verification, self-proving evidence Rebuilds the Strict PR-review pipeline around the official deepseek-v4-pro (RFC: doc/RFC-strict-review-deep-engine.md; campaign report: doc/EVAL-goal-strict-vs-opus5.md). Measured on the 20-PR campaign splits vs the recorded Claude Code + Opus 5 baseline (blind gpt-5.6 judge, 3 replicates): val 8-7 (first head-to-head win; was 1-4 pre-campaign), one-shot frozen test 7-8 - combined parity at ~1/3 the cost. Engine (all behind default-on settings with kill switches): - Hybrid pass set: deep investigator + adversary (32-iter budgets, budget discipline) alongside the behavior/verification breadth lenses. - Per-comment agentic verify pass: refuted drops, unverifiable demotes, confirmed may tighten; fail-open; traced skip reasons. - Self-proving evidence: every comment quotes the decisive code verbatim with file:line (drafting contract + verifier + renderer) - the single biggest measured lever (4-11 -> 8-7). - Coverage promotion mines lens findings/blockers for uncovered protected-class concerns; corroboration-ranked 8-comment budget with rich-only overflow. Recall mechanics: value_flips planner signal (default flips never rule light), verification lens in the planner fallback, evidence caps 120k->260k / reducer 60k->280k, commit timeline + changed-symbol consumer sweep + hunk-location paging targets in the evidence pack. Knowledge: strict-review-checklist page injected via new manifest key knowledge.review_checklist (linter extended); vllm-omni module map 8->17 with platform/diffusion/distributed risk-tiered. Also includes the previously-untracked review-comment anchoring module (anchor.py + tests) this pipeline depends on, ARM_JOBS parallelism for eval sweeps, and the v13 arm artifacts + judgments backing the report. Note: config.py, manifest.yaml and run_copilot_arm.py carried small pre-existing in-flight local edits that ship here as-is. Co-Authored-By: Claude Fable 5 --- adapters/vllm_omni/manifest.yaml | 50 ++- doc/EVAL-goal-strict-vs-opus5.md | 97 +++++ doc/RFC-strict-review-deep-engine.md | 122 ++++++ .../copilot_v13_proof_test/pr4762.cost.json | 107 ++++++ .../arms/copilot_v13_proof_test/pr4762.md | 57 +++ .../copilot_v13_proof_test/pr4777.cost.json | 113 ++++++ .../arms/copilot_v13_proof_test/pr4777.md | 40 ++ .../copilot_v13_proof_test/pr4834.cost.json | 107 ++++++ .../arms/copilot_v13_proof_test/pr4834.md | 59 +++ .../copilot_v13_proof_test/pr4849.cost.json | 107 ++++++ .../arms/copilot_v13_proof_test/pr4849.md | 41 ++ .../copilot_v13_proof_test/pr4954.cost.json | 107 ++++++ .../arms/copilot_v13_proof_test/pr4954.md | 49 +++ .../copilot_v13_proof_val/pr4810.cost.json | 107 ++++++ .../arms/copilot_v13_proof_val/pr4810.md | 44 +++ .../copilot_v13_proof_val/pr4816.cost.json | 113 ++++++ .../arms/copilot_v13_proof_val/pr4816.md | 36 ++ .../copilot_v13_proof_val/pr4825.cost.json | 107 ++++++ .../arms/copilot_v13_proof_val/pr4825.md | 40 ++ .../copilot_v13_proof_val/pr4837.cost.json | 107 ++++++ .../arms/copilot_v13_proof_val/pr4837.md | 45 +++ .../copilot_v13_proof_val/pr4893.cost.json | 107 ++++++ .../arms/copilot_v13_proof_val/pr4893.md | 56 +++ .../judgments/goal_v13_test/JUDGE_REPORT.md | 36 ++ .../judgments/goal_v13_test/pr4762.r1.json | 40 ++ .../judgments/goal_v13_test/pr4762.r2.json | 40 ++ .../judgments/goal_v13_test/pr4762.r3.json | 40 ++ .../judgments/goal_v13_test/pr4777.r1.json | 40 ++ .../judgments/goal_v13_test/pr4777.r2.json | 40 ++ .../judgments/goal_v13_test/pr4777.r3.json | 40 ++ .../judgments/goal_v13_test/pr4834.r1.json | 40 ++ .../judgments/goal_v13_test/pr4834.r2.json | 40 ++ .../judgments/goal_v13_test/pr4834.r3.json | 40 ++ .../judgments/goal_v13_test/pr4849.r1.json | 40 ++ .../judgments/goal_v13_test/pr4849.r2.json | 40 ++ .../judgments/goal_v13_test/pr4849.r3.json | 40 ++ .../judgments/goal_v13_test/pr4954.r1.json | 40 ++ .../judgments/goal_v13_test/pr4954.r2.json | 40 ++ .../judgments/goal_v13_test/pr4954.r3.json | 40 ++ .../judgments/goal_v13_val/JUDGE_REPORT.md | 36 ++ .../judgments/goal_v13_val/pr4810.r1.json | 40 ++ .../judgments/goal_v13_val/pr4810.r2.json | 40 ++ .../judgments/goal_v13_val/pr4810.r3.json | 40 ++ .../judgments/goal_v13_val/pr4816.r1.json | 40 ++ .../judgments/goal_v13_val/pr4816.r2.json | 40 ++ .../judgments/goal_v13_val/pr4816.r3.json | 40 ++ .../judgments/goal_v13_val/pr4825.r1.json | 40 ++ .../judgments/goal_v13_val/pr4825.r2.json | 40 ++ .../judgments/goal_v13_val/pr4825.r3.json | 40 ++ .../judgments/goal_v13_val/pr4837.r1.json | 40 ++ .../judgments/goal_v13_val/pr4837.r2.json | 40 ++ .../judgments/goal_v13_val/pr4837.r3.json | 40 ++ .../judgments/goal_v13_val/pr4893.r1.json | 40 ++ .../judgments/goal_v13_val/pr4893.r2.json | 40 ++ .../judgments/goal_v13_val/pr4893.r3.json | 40 ++ eval/dataset/run_copilot_arm.py | 7 +- knowledge/repos/vllm-omni/review/_index.md | 1 + .../repos/vllm-omni/review/guides/_index.md | 1 + .../review/guides/strict-review-checklist.md | 86 +++++ knowledge/tools/check_wiki_lint.py | 16 + src/infermatrix_copilot/config.py | 45 ++- .../engine/agent_runtime/ensemble.py | 22 ++ .../engine/steps/pr/fetch.py | 15 +- .../engine/steps/review/prompts.py | 217 +++++++++-- .../engine/steps/review/steps.py | 358 +++++++++++++++++- .../engine/steps/review/utils.py | 57 ++- src/infermatrix_copilot/review/planner.py | 40 +- test/conftest.py | 3 + test/test_agent_ensemble.py | 240 +++++++++++- test/test_review_planner.py | 37 ++ test/test_review_step.py | 40 ++ test/test_v2_p0.py | 9 +- 72 files changed, 4193 insertions(+), 91 deletions(-) create mode 100644 doc/EVAL-goal-strict-vs-opus5.md create mode 100644 doc/RFC-strict-review-deep-engine.md create mode 100644 eval/dataset/arms/copilot_v13_proof_test/pr4762.cost.json create mode 100644 eval/dataset/arms/copilot_v13_proof_test/pr4762.md create mode 100644 eval/dataset/arms/copilot_v13_proof_test/pr4777.cost.json create mode 100644 eval/dataset/arms/copilot_v13_proof_test/pr4777.md create mode 100644 eval/dataset/arms/copilot_v13_proof_test/pr4834.cost.json create mode 100644 eval/dataset/arms/copilot_v13_proof_test/pr4834.md create mode 100644 eval/dataset/arms/copilot_v13_proof_test/pr4849.cost.json create mode 100644 eval/dataset/arms/copilot_v13_proof_test/pr4849.md create mode 100644 eval/dataset/arms/copilot_v13_proof_test/pr4954.cost.json create mode 100644 eval/dataset/arms/copilot_v13_proof_test/pr4954.md create mode 100644 eval/dataset/arms/copilot_v13_proof_val/pr4810.cost.json create mode 100644 eval/dataset/arms/copilot_v13_proof_val/pr4810.md create mode 100644 eval/dataset/arms/copilot_v13_proof_val/pr4816.cost.json create mode 100644 eval/dataset/arms/copilot_v13_proof_val/pr4816.md create mode 100644 eval/dataset/arms/copilot_v13_proof_val/pr4825.cost.json create mode 100644 eval/dataset/arms/copilot_v13_proof_val/pr4825.md create mode 100644 eval/dataset/arms/copilot_v13_proof_val/pr4837.cost.json create mode 100644 eval/dataset/arms/copilot_v13_proof_val/pr4837.md create mode 100644 eval/dataset/arms/copilot_v13_proof_val/pr4893.cost.json create mode 100644 eval/dataset/arms/copilot_v13_proof_val/pr4893.md create mode 100644 eval/dataset/judgments/goal_v13_test/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_v13_test/pr4762.r1.json create mode 100644 eval/dataset/judgments/goal_v13_test/pr4762.r2.json create mode 100644 eval/dataset/judgments/goal_v13_test/pr4762.r3.json create mode 100644 eval/dataset/judgments/goal_v13_test/pr4777.r1.json create mode 100644 eval/dataset/judgments/goal_v13_test/pr4777.r2.json create mode 100644 eval/dataset/judgments/goal_v13_test/pr4777.r3.json create mode 100644 eval/dataset/judgments/goal_v13_test/pr4834.r1.json create mode 100644 eval/dataset/judgments/goal_v13_test/pr4834.r2.json create mode 100644 eval/dataset/judgments/goal_v13_test/pr4834.r3.json create mode 100644 eval/dataset/judgments/goal_v13_test/pr4849.r1.json create mode 100644 eval/dataset/judgments/goal_v13_test/pr4849.r2.json create mode 100644 eval/dataset/judgments/goal_v13_test/pr4849.r3.json create mode 100644 eval/dataset/judgments/goal_v13_test/pr4954.r1.json create mode 100644 eval/dataset/judgments/goal_v13_test/pr4954.r2.json create mode 100644 eval/dataset/judgments/goal_v13_test/pr4954.r3.json create mode 100644 eval/dataset/judgments/goal_v13_val/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_v13_val/pr4810.r1.json create mode 100644 eval/dataset/judgments/goal_v13_val/pr4810.r2.json create mode 100644 eval/dataset/judgments/goal_v13_val/pr4810.r3.json create mode 100644 eval/dataset/judgments/goal_v13_val/pr4816.r1.json create mode 100644 eval/dataset/judgments/goal_v13_val/pr4816.r2.json create mode 100644 eval/dataset/judgments/goal_v13_val/pr4816.r3.json create mode 100644 eval/dataset/judgments/goal_v13_val/pr4825.r1.json create mode 100644 eval/dataset/judgments/goal_v13_val/pr4825.r2.json create mode 100644 eval/dataset/judgments/goal_v13_val/pr4825.r3.json create mode 100644 eval/dataset/judgments/goal_v13_val/pr4837.r1.json create mode 100644 eval/dataset/judgments/goal_v13_val/pr4837.r2.json create mode 100644 eval/dataset/judgments/goal_v13_val/pr4837.r3.json create mode 100644 eval/dataset/judgments/goal_v13_val/pr4893.r1.json create mode 100644 eval/dataset/judgments/goal_v13_val/pr4893.r2.json create mode 100644 eval/dataset/judgments/goal_v13_val/pr4893.r3.json create mode 100644 knowledge/repos/vllm-omni/review/guides/strict-review-checklist.md diff --git a/adapters/vllm_omni/manifest.yaml b/adapters/vllm_omni/manifest.yaml index ee417d5c9..02eee0329 100644 --- a/adapters/vllm_omni/manifest.yaml +++ b/adapters/vllm_omni/manifest.yaml @@ -38,6 +38,11 @@ knowledge: # on-demand through doc_read/doc_search. briefing_docs_extra: - repos/vllm-omni/review/guides/maintainer-pattern-routing.md + # repo-specific review checklist injected (first 4k chars) into the Strict + # reviewer's system prompt — triggered checks distilled from maintainer + # review patterns; the page lives in the shared knowledge tree so Direct + # and knowledge maintenance see the same source + review_checklist: repos/vllm-omni/review/guides/strict-review-checklist.md modules: worker_runner: @@ -61,11 +66,54 @@ modules: model_config: local_paths: [vllm_omni/config/] wave: 1 + # risk high: platform code sets process-wide defaults (op priorities, + # capability gates) inherited by EVERY model on that platform — a 3-file + # 61-line default flip here is a repo-wide behavior change (PR 4964/5009 + # class), so small diffs must still get the full review ensemble platform: local_paths: [vllm_omni/platforms/] wave: 1 + risk: high + # shared diffusion pipeline: denoise loop, attention backends, DiT/VAE — + # multi-model shared code where one-model-validated changes have cross-model + # blast radius (kernel-provider and hub-loading changes live here) + diffusion: + local_paths: [vllm_omni/diffusion/] + wave: 1 + risk: high + # deploy YAMLs are the serving registry: defaults and per-platform blocks + # (cudagraph modes, capacity) that decide runtime behavior for CLI and yaml + # launches alike. NOT risk-tagged: a 2-line deploy deletion measured as a + # precision WIN at light depth (pr4970), so tiny deploy diffs keep the + # cheap tier; deploy changes bundled with code still escalate via the + # code-side signals + deploy: + local_paths: [vllm_omni/deploy/] + wave: 1 + distributed: + local_paths: [vllm_omni/distributed/] + wave: 1 + risk: high + engine: + local_paths: [vllm_omni/engine/] + wave: 1 + attention: + local_paths: [vllm_omni/attention/] + wave: 1 + quantization: + local_paths: [vllm_omni/quantization/] + wave: 1 + sampling: + local_paths: [vllm_omni/sample/] + wave: 1 + tokenizers: + local_paths: [vllm_omni/tokenizers/] + wave: 1 + metrics: + local_paths: [vllm_omni/metrics/] + wave: 2 benchmarks: - local_paths: [benchmarks/] + local_paths: [benchmarks/, vllm_omni/benchmarks/] wave: 2 review_routes: diff --git a/doc/EVAL-goal-strict-vs-opus5.md b/doc/EVAL-goal-strict-vs-opus5.md new file mode 100644 index 000000000..4e6aaedcd --- /dev/null +++ b/doc/EVAL-goal-strict-vs-opus5.md @@ -0,0 +1,97 @@ +# Goal campaign — DeepSeek-v4-pro Strict vs Claude Code + Opus 5 (2026-08-12/13) + +> **Final update (2026-08-13, official-model rework):** after the official +> `deepseek-v4-pro` release the pipeline was rebuilt for a strong generator +> (deep investigator+adversary passes alongside the behavior/verification +> breadth lenses; a per-comment agentic verify pass; self-proving verbatim +> evidence quotes end-to-end). Result — **val: ARM 8—7 (first head-to-head +> win); frozen test, one-shot confirmatory: ARM 7—8. Combined 15—15** vs +> the Opus 5 baseline at roughly 1/3 the cost, from 1—4 (val) pre-campaign. +> Test means: r .689 vs .762, p .687 vs .719 (arm's best precision on any +> split); the GOLD gap item pr4834 was swept 3/3 clear. Rubric-mean gaps +> (.03-.07) sit at the magnitude of measured judge drift on identical +> baseline reviews (±.05-.10 across gates). Honest verdict: the copilot +> moved from clearly-worse to statistically indistinguishable from the +> agentic Opus 5 baseline; "strictly better on both means" is NOT +> demonstrated. Both tuning splits are now spent (8 val submissions, 1 +> test shot) — any further iteration requires fresh items (wave2 snapshots +> exist under `eval/dataset/gt/` for exactly this). Key structural lesson +> for successors: the judge is tool-less — repo-side findings score as +> speculation unless the comment carries verbatim-quoted proof; making +> evidence self-proving was the single biggest win (4—11 → 8—7). + +Goal: make the Strict pipeline (DeepSeek-v4-pro generation) beat the recorded +`baselines/claudecode_opus5` agentic review on the 5-PR val split. Protocol: +adaptation on train only; judge `gpt-5.6-sol-high` via cursor (blind pairwise, +`judge_val.py`); PR-time trees pinned to `expected_pr_heads.json`; +`PR_CONTEXT_MODE=no_discussion`. All arms/judgments under +`eval/dataset/arms/copilot_v{7,8,9}_*` and `judgments/goal_*`. + +## Verdict — the gate did NOT pass + +| gate | arm | model | wins (arm—opus) | arm r/p | opus r/p | +|---|---|---|---|---|---| +| pre-campaign (g56) | copilot_v4_pr20_r1 | v4-pro[1m] preview | 1—4 | 0.42/0.51 | 0.82/0.54 | +| val gate 1 | copilot_v8_official_r1 | v4-pro official | 3—12 (×3 reps) | **0.86**/0.56 | 0.79/0.66 | +| val gate 2 (precision trim) | copilot_v9_official_r1 | v4-pro official | 2—13 (×3 reps) | 0.80/0.54 | 0.82/0.65 | + +What the campaign did achieve, measured: + +- **Val recall 0.42 → 0.80–0.86** — at or above the Opus 5 baseline (0.79–0.82). + The pre-campaign recall deficit (−0.40) is gone. +- Gate 1 **swept the GOLD latent-gap item pr4810 3/3** (the arm engaged the + gap both gates; verdict wins depend on the noisy precision margin). +- Train hardest-6: recall 0.42 → 0.50 monotonically over calibration rounds; + pr4870 flipped to a consistent 2/2 win (r≈0.83, p≈0.84). +- Cost ~$0.26–0.53/item ≈ 5× cheaper than the baseline. +- The loss driver flipped: pre-campaign the arm lost on recall; now it loses + on **precision** (0.54–0.56 vs 0.65) — the judge discounts individually + under-grounded or speculative comments, an agentic-verification property of + the generator that two calibration rounds of budget/reducer tuning did not + buy. Head-to-head margins are mostly "slight". + +## What changed (all in-tree, tests + knowledge validators green) + +Workflow: planner `value_flips` signal (default flips can't rule light); +verification lens in the fallback set; evidence caps 120k→260k / reducer +60k→280k (a 170k diff was losing 30% of its hunks); commit timeline + +deterministic changed-symbol consumer sweep (`git grep -nw`) in the evidence +pack; hunk-location paging targets; cross-lens corroboration tags + reducer +protected-class drop rules; comment budget 6 (+≤3 coverage promotions, +overflow only on evidence-rich reviews); coverage-promotion pass converting +lens findings/blockers into grounded comments; severity + red-CI calibration. + +Knowledge: `repos/vllm-omni/review/guides/strict-review-checklist.md` +(train-distilled triggered checks) injected via new manifest key +`knowledge.review_checklist` (linter extended); module map 8→17 entries, +risk tiers +platform/diffusion/distributed (small diffs there now get the +full ensemble — this alone fixed the pr5009/4977 class). + +## Forensic lessons (train, 3-agent trace analysis + 5 judged rounds) + +1. Depth misfires and evidence truncation were real and fixed (verified live). +2. The 5-comment severity-only cap deleted reducer-kept GT concerns; but + widening budgets converts to noise — the judge is recall-dominant between + arms yet punishes volume within an arm. The stable operating point is a + small budget plus class-protected promotion. +3. Generation variance and judge variance are both large: single-replicate + verdicts flipped items across identical configs. Never calibrate on one + replicate (this campaign's r2 was wasted that way). +4. The residual deficit is per-comment grounding depth (the baseline + verifies each claim agentically before asserting it). Workflow scaffolding + moved recall dramatically; it did not move groundedness. + +## Next levers, ranked + +1. Per-comment agentic verify pass (tools, one comment at a time) before the + budget — directly attacks the precision deficit; ~+30% cost. +2. `ensemble_lens_max_iters` ↑ for the drafting lenses on the official model + (the baseline reads 3–10× more code per finding). +3. MoA re-test with the official pro model as reducer over stronger members + (the old regression used weak mimo/qwen members). +4. Re-baseline: the official v4-pro replaced the [1m] preview mid-campaign; + train-side numbers predate it. + +Judgments for the two gates: `judgments/goal_v8_val`, `judgments/goal_v9_val` +(the 402-outage set is quarantined as `INVALID_apierror_goal_v7_val`). +Test split remains frozen and untouched. diff --git a/doc/RFC-strict-review-deep-engine.md b/doc/RFC-strict-review-deep-engine.md new file mode 100644 index 000000000..893f2afc8 --- /dev/null +++ b/doc/RFC-strict-review-deep-engine.md @@ -0,0 +1,122 @@ +# RFC — Strict review for a strong generator: deep passes, per-comment verification, self-proving evidence + +- Status: implemented behind flags (all default ON); measured on the 20-PR + campaign splits (val + one-shot frozen test) +- Owner: review pipeline (`engine/steps/review/`, `review/planner.py`) +- Evidence: `doc/EVAL-goal-strict-vs-opus5.md`, + `eval/dataset/judgments/goal_v13_val/`, `eval/dataset/judgments/goal_v13_test/` + +## Motivation + +The Strict review pipeline was engineered around a weak generator: four +narrow lenses forcing enumeration inside templates, tight tool budgets, a +tool-less reducer, and a 5-comment severity cap. With the official +`deepseek-v4-pro` (a large capability jump over the `[1m]` preview), that +scaffolding became the bottleneck — measured symptoms across judged sweeps: + +1. **Recall losses were mechanical, not analytical**: the budget cap deleted + reducer-kept maintainer concerns; depth planning misrouted small + high-blast-radius diffs to the no-ensemble light tier; a 170k-char diff + lost 30% of its hunks to the 120k evidence cap; whole question classes + (blast radius, dependency floors, test integrity, lifecycle-on-abort) had + no owning lens. +2. **Precision was stuck at 0.54–0.56 across five different configurations** + while recall swung 0.55–0.86 — because the blind judge is TOOL-LESS: it + sees only the diff and the human threads, so any repo-side finding + (consumers elsewhere, CI lane rules, version floors) scores as + speculation unless the comment itself carries checkable proof. + +## Design + +Three mechanisms, composed with the existing ensemble/reducer machinery: + +### 1. Hybrid pass set (`review_deep_engine`, default on) + +Full depth runs two **deep passes** — `investigator` (central-change-first +free investigation; verify-before-assert; budget discipline: reserve the +final rounds for filing) and `adversary` (independent hunt over the +systematically-missed classes) — **plus** the two highest-value breadth +lenses (`behavior`, `verification`). Standard depth runs +investigator+behavior. Light is unchanged. Deep passes get +`review_deep_max_iters` (32) tool rounds; measured alone they ground +claims well (precision .82 on the hardest train items) but under-cover +(val recall .55); measured with the breadth lenses they hold recall .71–.86. + +### 2. Per-comment agentic verification (`review_verify_comments`, default on) + +Every merged draft comment gets a small tool-loop (4 rounds, concurrency 6, +shared diff cache prefix) that must re-derive the claim on the PR-time +tree: `refuted` drops, `unverifiable` demotes one severity step, +`confirmed` may tighten wording/position — and a verification failure +keeps the comment (the pass can only raise precision, never silently +delete recall). Skip/failure reasons are traced +(`review_comments_verified` / `review_coverage_skipped`). + +### 3. Self-proving evidence (drafting + verify + render) + +The single biggest measured lever (4—11 → 8—7 on the val gate): every +comment's `evidence` field must QUOTE the decisive code line(s) verbatim +with file:line so a reader holding only the review and the diff can check +the claim. Enforced in the drafting contract, re-emitted by the verifier +from the code it actually read, rendered as-is. + +Supporting changes shipped with this RFC: `value_flips` diff signal (a +default flip can never rule light), verification lens in the planner +fallback set, evidence caps 120k→260k / reducer 60k→280k, commit timeline ++ deterministic changed-symbol consumer sweep (`git grep -nw`) + per-file +hunk-location paging targets in the evidence pack, cross-lens +corroboration tags with protected-class reducer rules, comment budget +8 + rich-only overflow, coverage-promotion pass (mines the run's own +findings/blockers for uncovered protected-class concerns), knowledge page +`repos/vllm-omni/review/guides/strict-review-checklist.md` injected via +the new adapter manifest key `knowledge.review_checklist`, and the +vllm-omni module map extended to 17 modules with 7 risk tiers. + +## Measured results (blind gpt-5.6 judge, 3 replicates, vs recorded CC+Opus 5) + +| gate | config | head-to-head (arm—opus) | arm r/p | opus r/p | +|---|---|---|---|---| +| pre-campaign | preview model, 4 lenses | 1—4 | .42/.51 | .82/.54 | +| v8 | official model, lenses | 3—12 | .86/.56 | .79/.66 | +| v12 | hybrid, narrative evidence | 4—11 | .71/.55 | .81/.66 | +| **v13 val** | hybrid + self-proving | **8—7** | .857/.555 | .889/.589 | +| **v13 frozen test** (one shot) | same | 7—8 | .689/.687 | .762/.719 | + +Combined fresh verdicts: **15—15** at ~1/3 the baseline's cost +(~$0.75–1.2/item vs ~$1.5–3). GOLD latent-gap items: pr4810 2/3, pr4834 +swept 3/3 clear. Rubric-mean gaps (.03–.07) are at the magnitude of +measured judge drift on identical baseline reviews (±.05–.10). + +## Alternatives considered + +- **MoA re-test with stronger members** — excluded by owner direction. +- **Deep passes only** (no breadth lenses): precision transferred but val + recall collapsed to .55; rejected (gate v11, 2—13). +- **Budget/reducer tuning alone**: five configurations moved precision by + ≤.02; the deficit was structural (tool-less judge), not calibrational. + +## Rollout / compatibility + +All three mechanisms sit behind settings with kill switches +(`review_deep_engine`, `review_verify_comments`; legacy lens set retained +verbatim and used when the deep engine is off). Offline tests cover pass +selection, verify verdict handling (drop/demote/tighten/fail-open), +budget/overflow, corroboration ordering, promotion, and the planner +signal; the full suite and both knowledge validators are green. Cost is +~2–3× the old lens pipeline per item and remains ~3× cheaper than the +baseline. `ARM_JOBS` in `run_copilot_arm.py` enables item-level parallel +sweeps (endpoint allows 500 concurrent requests on v4-pro). + +## Open questions + +1. Both tuning splits are spent (8 val submissions, 1 frozen-test shot). + Further iteration needs wave-2 items (`gt/pr5957*`, `gt/pr5976*`, + `build_wave2.py`) judged under the same protocol. +2. The judge protocol's tool-less-ness now shapes the reviews (verbose + quoted evidence). If the product's real audience is maintainers rather + than a repo-less judge, the evidence verbosity should become a render + option rather than a fixed behavior. +3. `pr4762`/`pr4954`-class losses (broad multi-concern PRs) remain: the + baseline still out-reads the arm per finding. The next structural lever + is a coverage-driven second investigation round seeded by the reducer's + uncovered-GT-class checklist rather than a fixed pass count. diff --git a/eval/dataset/arms/copilot_v13_proof_test/pr4762.cost.json b/eval/dataset/arms/copilot_v13_proof_test/pr4762.cost.json new file mode 100644 index 000000000..fabba3359 --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_test/pr4762.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 1393.7, + "split": "test", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test/runs/pr4762/run-20260813-154730-435fc7", + "recorded_at": "2026-08-13T16:10:40", + "attempts": [ + { + "invocation_id": "inv-fc3b7db5", + "outcome": "run:run-20260813-154730-435fc7", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-154730-435fc7", + "llm_calls": 172, + "input_tokens": 1566512, + "output_tokens": 245096, + "usd": 1.161101, + "source": "spans" + } + ], + "attempt_usd_total": 1.161101, + "llm_calls": 172, + "input_tokens": 1566512, + "output_tokens": 245096, + "usd": 1.161101, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-154730-435fc7", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 1.1611, + "minutes": 23.16, + "ci_minutes": 0, + "input_tokens": 1566512, + "output_tokens": 245096, + "tool_calls": 276, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 2.0296, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 172, + "by_role": { + "agent": { + "usd": 1.132141, + "input_tokens": 1543707, + "output_tokens": 224398, + "calls": 170 + }, + "reducer": { + "usd": 0.02896, + "input_tokens": 22805, + "output_tokens": 20698, + "calls": 2 + } + }, + "cache_read_tokens": 17353216, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 3.61, + "gate": 1.68, + "review": 1384.3, + "report": 0.07 + }, + "time_to_first_result_s": 1389.65 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4762.trace.json.gz", + "trace_bytes": 44866 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v13_proof_test/pr4762.md b/eval/dataset/arms/copilot_v13_proof_test/pr4762.md new file mode 100644 index 000000000..42008a9ac --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_test/pr4762.md @@ -0,0 +1,57 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4762, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head cdc825f03b7d) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 7 finding(s) below | +| Security | no finding reported | +| Docs / comments | 1 finding(s) below | +| Behavior / compatibility | 2 finding(s) below | +| Correctness | 1 finding(s) below | + +**Validated:** +- [sweep] tests/e2e/online_serving/test_qwen3_omni.py:252 `assert not responses[0].success` — read HttpResponse/_http_response_from_requests (success = 200<=status<300) and assert_http_error (asserts status==400); the assertion is valid and the test drives the real server path. +- [sweep] config_factory.py new branches (lines 131,138,139,144,155,161,180,183,203,213,252,256,258,279,287) read in full; fallback chains mirror the pre-refactor logic except the trust_remote_code default flip (review comment 1). +- [sweep] api_server.py:514/516 hasattr/else branches — engine_client is always AsyncOmni in omni_run_server_worker, so the else is defensive-only. +- [sweep] non-test touched files mapped: endpoint_policy -> tests/config/test_endpoint_policy.py; config_factory -> tests/test_config_factory.py + tests/entrypoints/test_resolve_dreamzero_config.py; async_omni_engine/async_omni/api_server/utils/qwen3 pipeline -> tests/e2e/online_serving/test_qwen3_omni.py + updated entrypoint tests. +- [validated] tests/config/test_endpoint_policy.py pytestmark=[core_model,cpu] is selected by .buildkite/test-merge.yml:29 (`pytest -sv -m 'core_model and cpu' --ignore=tests/diffusion --ignore=tests/model_executor`). +- [validated] shutdown_unsupported_routes unit tests cover rejection, unrestricted passthrough, idempotency, and missing-route safety; the e2e test verifies real /v1/completions rejection for qwen3_omni_moe. +- [validated] merge_pipeline_deploy only reads PipelineConfig and builds new StageConfig objects, so functools.cache on get_pipeline_config does not share mutable stage state. +- [sweep] tests/test_config_factory.py autouse clear_config_factory_caches clears the three caches after each test; test_resolve_dreamzero_config.py monkeypatches _try_infer_model_type without cache_clear, but no other test uses the same model key, so no current cross-test contamination found. + +`tests/e2e/online_serving/test_qwen3_omni.py:252` [major] — Buildkite `vllm-omni-amd-ci` and `vllm-omni-npu-ci` are red on PR 4762 head (cdc825f03b7d). The new e2e test `test_completions_rejected_for_thinker_talker` is marked `rocm: MI325` (and `cuda: H100`) at tests/e2e/online_serving/test_qwen3_omni.py:235. Please confirm whether these failures are caused by this PR or are pre-existing on main, and do not merge until the gate is green or explicitly waived. (evidence: tests/e2e/online_serving/test_qwen3_omni.py:235 `@hardware_test(res={"cuda": "H100", "rocm": "MI325"}, num_cards=2)` — new test is ROCm/MI325-marked. gh_ci_read(4762) returned `{"name":"buildkite/vllm-omni-amd-ci","state":"FAILURE"}` and `{"name":"buildkite/vllm-omni-npu-ci","state":"FAILURE"}`.) + +`vllm_omni/entrypoints/utils.py:348` [major] — The diff removes the old detection default `trust_remote_code = cli_overrides.get("trust_remote_code", True)` and `entrypoints/utils.py` now passes `cli_overrides.get("trust_remote_code", False)` into `StageConfigFactory.create_from_model`, whose new signature is `trust_remote_code: bool = False`. Because `get_hf_config` returns `None` when `get_config` raises, pipelines that are matched only via `hf_architectures` + `hf_config_predicate` (e.g. MiniCPM-o 4.5, Ming TTS) can no longer reach that branch if their custom `model_type` is not in OMNI_PIPELINES: the raw config.json fallback returns the unregistered model_type, `get_pipeline_config` returns `None`, and `load_stage_configs_from_model` falls back to legacy/default-diffusion. Keep the detection-only trust default at `True` (or read the deploy YAML's trust value for detection) and add a regression test for a predicate/arch-routed model served without `--trust-remote-code`. (evidence: diff removed: `- trust_remote_code = cli_overrides.get("trust_remote_code", True)`; `vllm_omni/config/config_factory.py:267`: `trust_remote_code: bool = False,`; `vllm_omni/model_executor/models/minicpmo_4_5/pipeline.py:31-36`: `MiniCPM-o 4.5's HF config.json reports `model_type="minicpmo"` and `architectures=["MiniCPMO"]` ... `hf_config_predicate=lambda c: str(getattr(c, "version", "")) == "4.5",`; `vllm_omni/model_executor/models/ming_tts/pipeline.py:15,18`: `hf_architectures=("MingTTSForConditionalGeneration", "BailingMMNativeForConditionalGeneration"),` and `hf_config_predicate=lambda c: str(getattr(c, "model_type", "")) == "dense",`.) + +`vllm_omni/config/endpoint_policy.py:44` [minor] — vllm_omni/config/endpoint_policy.py:68 `restricted_endpoints = (*endpoint_restrictions, *UNSUPPORTED_ROUTES)` always applies the `/v1/chat/completions/batch` ban, but no test covers it: tests/config/test_endpoint_policy.py only targets `/v1/completions`. Add a unit test calling `shutdown_unsupported_routes(app, ())` and asserting POST `/v1/chat/completions/batch` returns 400 with `"Batched chat completions are not yet supported by vLLM Omni."` so a route mismatch cannot silently no-op. (evidence: vllm_omni/config/endpoint_policy.py:46 `EndpointRestriction( + OmniServingCapability.CHAT_COMPLETIONS_BATCH, + "Batched chat completions are not yet supported by vLLM Omni.", + ),`; vllm_omni/config/endpoint_policy.py:68 `restricted_endpoints = (*endpoint_restrictions, *UNSUPPORTED_ROUTES)`; tests/config/test_endpoint_policy.py:17 `SERVER_ROUTE = "/v1/completions"`) + +`vllm_omni/entrypoints/openai/api_server.py:514` [minor] — The call is gated behind `hasattr(engine_client, "endpoint_restrictions")` and the `else` branch only logs. Because `shutdown_unsupported_routes()` always appends the global `UNSUPPORTED_ROUTES` (the `/v1/chat/completions/batch` 400), any engine client without that attribute also skips the global batch-route shutdown. Call it unconditionally with `shutdown_unsupported_routes(app, getattr(engine_client, "endpoint_restrictions", ()))` so the batch route is always rejected. (evidence: vllm_omni/entrypoints/openai/api_server.py:514 `if hasattr(engine_client, "endpoint_restrictions"):`; :515 `shutdown_unsupported_routes(app, engine_client.endpoint_restrictions)`; :516 `else:`; :517 `logger.warning("engine client has no endpoint restrictions attribute")`. vllm_omni/config/endpoint_policy.py:44 `UNSUPPORTED_ROUTES: tuple[EndpointRestriction, ...] = (`; :46 `OmniServingCapability.CHAT_COMPLETIONS_BATCH,`; :76 `restricted_endpoints = (*endpoint_restrictions, *UNSUPPORTED_ROUTES)`.) + +`vllm_omni/config/config_factory.py:75` [minor] — `get_hf_config` is decorated with `@functools.cache` (config_factory.py:75) and, on any `get_config` exception, logs and returns the pre-initialized `hf_config = None` (config_factory.py:86-91). `functools.cache` memoizes `None` too, so one transient failure caches an empty config for that `(model, trust_remote_code)` key for the life of the process; every later `get_hf_config`/`_try_infer_model_type` call skips retry and permanently loses the hf_config-based routing path (fallbacks may still recover some models, but any model relying on the loaded HF config is stuck). Cache only successful reads, or evict/retry on failure. (evidence: vllm_omni/config/config_factory.py:75 ` @functools.cache` +vllm_omni/config/config_factory.py:76 ` def get_hf_config(cls, model: str, trust_remote_code: bool) -> PretrainedConfig | None:` +vllm_omni/config/config_factory.py:86 ` hf_config = None` +vllm_omni/config/config_factory.py:88 ` return get_config(model, trust_remote_code=trust_remote_code)` +vllm_omni/config/config_factory.py:89 ` except Exception as e:` +vllm_omni/config/config_factory.py:91 ` return hf_config`) + +`vllm_omni/config/endpoint_policy.py:42` [minor] — `UNSUPPORTED_ROUTES` in `vllm_omni/config/endpoint_policy.py` makes `/v1/chat/completions/batch` return 400 for every Omni model, and no user-facing doc or changelog mentions this route (repo-wide grep finds only the code). Please add a user-facing note or changelog entry, or confirm the silent all-models batch ban is intended. (evidence: vllm_omni/config/endpoint_policy.py:42-49 `# Routes that are not supported for any model, but are supported in vLLM.` / `# This is only temporary to avoid 500s for batched chat completions.` / `UNSUPPORTED_ROUTES: tuple[EndpointRestriction, ...] = (` / ` EndpointRestriction(` / ` OmniServingCapability.CHAT_COMPLETIONS_BATCH,` / ` "Batched chat completions are not yet supported by vLLM Omni.",` / ` ),` / `)`; vllm_omni/config/endpoint_policy.py:76 `restricted_endpoints = (*endpoint_restrictions, *UNSUPPORTED_ROUTES)`) + +`vllm_omni/config/config_factory.py:75` [minor] — vllm_omni/config/config_factory.py adds @functools.cache to get_hf_config (line 75), try_infer_model_type (line 94), and get_pipeline_config (line 189). Production has no cache invalidation: no cache_clear call exists anywhere under vllm_omni, while tests/test_config_factory.py:47-49 clears these caches only in a test fixture. Please confirm that runtime model-file changes/reloads are out of scope, or add invalidation/versioning for these caches. (evidence: vllm_omni/config/config_factory.py:75 ` @functools.cache` (immediately preceding `get_hf_config`); vllm_omni/config/config_factory.py:94 ` @functools.cache` (preceding `try_infer_model_type`); vllm_omni/config/config_factory.py:189 ` @functools.cache` (preceding `get_pipeline_config`); tests/test_config_factory.py:47 ` StageConfigFactory.get_hf_config.cache_clear()` and tests/test_config_factory.py:48-49 clear try_infer_model_type and get_pipeline_config — while grep for `cache_clear` under `vllm_omni` returns no matches, proving no production invalidation.) + +`vllm_omni/model_executor/models/qwen3_omni/pipeline.py:25` [minor] — The PR description states the completions crash is not Qwen3Omni-specific and that other models (e.g. TTS models) likely have vulnerable endpoints, yet the diff adds a COMPLETIONS `EndpointRestriction` only to the qwen3_omni pipeline (vllm_omni/model_executor/models/qwen3_omni/pipeline.py:25). Sibling thinker->talker pipelines such as qwen2_5_omni (vllm_omni/model_executor/models/qwen2_5_omni/pipeline.py:18) define no endpoint restrictions and inherit the default empty tuple. Either add COMPLETIONS restrictions to the vulnerable siblings or track an explicit audit so the new mechanism does not ship while the acknowledged siblings remain reachable. (evidence: ['vllm_omni/model_executor/models/qwen3_omni/pipeline.py:25 ` endpoint_restrictions=(` — the only model-level endpoint_restrictions assignment in vllm_omni/', 'vllm_omni/model_executor/models/qwen2_5_omni/pipeline.py:18 `QWEN2_5_OMNI_PIPELINE = PipelineConfig(` — sibling thinker->talker pipeline with no endpoint_restrictions (defaults to ())', 'PR body: "Importantly, this issue is not Qwen3Omni specific; other models, e.g., TTS models, will probably have their own issues with things like the completion endpoint."']) + +**Additional observations (beyond the comment budget):** +- `vllm_omni/config/config_factory.py:257` [minor] `_get_deploy_override_pipe_config` (config_factory.py:257) calls `load_deploy_config(deploy_path)` with no try/except, and its callers `get_pipeline_config` -> `get_pipeline_endpoint_restrictions`/`create_from_model` do +- `tests/config/test_endpoint_policy.py:73` [minor] `test_invalid_route_rejection` at tests/config/test_endpoint_policy.py:73 creates `app = FastAPI()` with no matching route and asserts a 400 after `shutdown_unsupported_routes`, but `shutdown_unsupported_routes` at vllm_ +- `tests/e2e/online_serving/test_qwen3_omni.py:252` [minor] The e2e test passes `err_code=400` but no `err_message`, so `send_completions_http_request` already verifies the 400 via its internal `assert_http_error` call, making the trailing `assert not responses[0].success` redund + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v13_proof_test/pr4777.cost.json b/eval/dataset/arms/copilot_v13_proof_test/pr4777.cost.json new file mode 100644 index 000000000..3d8d30a7e --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_test/pr4777.cost.json @@ -0,0 +1,113 @@ +{ + "wall_s": 900.7, + "split": "test", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test/runs/pr4777/run-20260813-154730-4e37a7", + "recorded_at": "2026-08-13T16:02:27", + "attempts": [ + { + "invocation_id": "inv-db32c28a", + "outcome": "run:run-20260813-154730-4e37a7", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-154730-4e37a7", + "llm_calls": 86, + "input_tokens": 683668, + "output_tokens": 109108, + "usd": 0.504718, + "source": "spans" + } + ], + "attempt_usd_total": 0.504718, + "llm_calls": 86, + "input_tokens": 683668, + "output_tokens": 109108, + "usd": 0.504718, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-154730-4e37a7", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.5047, + "minutes": 14.94, + "ci_minutes": 0, + "input_tokens": 683668, + "output_tokens": 109108, + "tool_calls": 148, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.6449, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 86, + "by_role": { + "planner": { + "usd": 0.000923, + "input_tokens": 1788, + "output_tokens": 400, + "calls": 1 + }, + "agent": { + "usd": 0.480251, + "input_tokens": 672509, + "output_tokens": 89635, + "calls": 83 + }, + "reducer": { + "usd": 0.023545, + "input_tokens": 9371, + "output_tokens": 19073, + "calls": 2 + } + }, + "cache_read_tokens": 7411456, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 4.21, + "gate": 2.55, + "review": 889.8, + "report": 0.07 + }, + "time_to_first_result_s": 896.61 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4777.trace.json.gz", + "trace_bytes": 23079 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v13_proof_test/pr4777.md b/eval/dataset/arms/copilot_v13_proof_test/pr4777.md new file mode 100644 index 000000000..e873f9f81 --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_test/pr4777.md @@ -0,0 +1,40 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4777, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head 51ce34a55a74) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 4 finding(s) below | +| Security | no finding reported | +| Docs / comments | 1 finding(s) below | +| Behavior / compatibility | 1 finding(s) below | +| Correctness | no finding reported | + +**Validated:** +- [sweep] Read vllm_omni/entrypoints/openai/image_api_utils.py in tree: line 17 `SUPPORTED_LAYERED_LAYERS_RANGE = range(2, 11)`; upper bound 10 is preserved because `stop=11` is excluded. +- [validated] image_api_utils.py:115-125 `validate_layered_layers` passes None through unchanged; `2 in range(2,11)` is accepted, `1` and `11` raise; error string uses `range.start` / `range.stop-1`, so the message is automatically 'between 2 and 10 inclusive' — no off-by-one or stale literal. +- [validated] vllm_omni/entrypoints/openai/protocol/images.py:60-63 field description updated to 'Supported range: 2-10.', matching the new constant. +- [sweep] All `validate_layered_layers` consumers share the helper and have no separate hardcoded 3: protocol/images.py:93, api_server.py:2070, serving_chat.py:3287. +- [sweep] Pipeline vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image_layered.py:667 reads `req.sampling_params.layers`; prepare_latents/_unpack_latents use `layers+1` and forward() drops the first frame via `latents = latents[:, :, 1:]`, so layers=2 is semantically valid (2 output images) with no `layers >= 3` assertion. +- [sweep] Repo-wide grep for 'between 3 and 10' and '3-10' found no other Qwen-Image-Layered range references; the only '3-10' hits are unrelated TTS reference-audio docs. +- [sweep] Diff-touched tests verified in tree: test_image_server.py:1247/1666/1680 now use layers=1 lower-bound and 'between 2 and 10 inclusive'; test_qwen_image_layered_expansion.py:10/116/120 restored `layers=2`. +- [sweep] changed_symbol_consumers checked: the two dfx reliability tests are NOT in the diff and still assert the old '3 to 10' contract — see review comments. + +`tests/dfx/reliability/invalid_param_test/test_invalid_image_generation.py:?` [major] — The diff changes SUPPORTED_LAYERED_LAYERS_RANGE to range(2, 11) in vllm_omni/entrypoints/openai/image_api_utils.py:17, so layers=2 now passes validation. But tests/dfx/reliability/invalid_param_test/test_invalid_image_generation.py:90 still submits {"layers": 2} and expects 400 with ('layers', 'value_error', '3', '10'), so the case will now fail. Update the below-min param to {"layers": 1} and the expected error pieces to ('layers', 'value_error', '2', '10'). (evidence: vllm_omni/entrypoints/openai/image_api_utils.py:17 `SUPPORTED_LAYERED_LAYERS_RANGE = range(2, 11)` (2 is inside the valid range, so layers=2 no longer raises) versus tests/dfx/reliability/invalid_param_test/test_invalid_image_generation.py:90 `pytest.param({"layers": 2}, ("layers", "value_error", "3", "10"), id="layers_below_min"),` (still expects a 400 with the old bounds 3/10).) + +`tests/dfx/reliability/invalid_param_test/test_invalid_image_editing.py:?` [major] — The diff widens SUPPORTED_LAYERED_LAYERS_RANGE to range(2, 11), so /v1/images/edits now accepts layers=2 (Form value "2" coerced to int 2 passes validate_layered_layers). tests/dfx/reliability/invalid_param_test/test_invalid_image_editing.py:65 still sends layers="2" and line 66 expects 400 with ("Invalid layers", "3", "1"); the server will now return 200 and the test fails. Change the below-min input to "1" (or 1) and update the expected substrings to ("Invalid layers", "2", "10"). (evidence: tests/dfx/reliability/invalid_param_test/test_invalid_image_editing.py:65 `{"prompt": "edit", "model": _IMAGE_EDITS_SERVER_MODEL, "layers": "2"},` ; tests/dfx/reliability/invalid_param_test/test_invalid_image_editing.py:66 `("Invalid layers", "3", "1"),` ; vllm_omni/entrypoints/openai/image_api_utils.py:17 `SUPPORTED_LAYERED_LAYERS_RANGE = range(2, 11)` ; vllm_omni/entrypoints/openai/image_api_utils.py:119 ` if layers not in SUPPORTED_LAYERED_LAYERS_RANGE:` ; vllm_omni/entrypoints/openai/api_server.py:2070 ` layers = validate_layered_layers(layers)`) + +`vllm_omni/entrypoints/openai/image_api_utils.py:17` [minor] — `SUPPORTED_LAYERED_LAYERS_RANGE = range(2, 11)` (image_api_utils.py:17) feeds all three generic validation call sites — protocol/images.py:93, serving_chat.py:3287, and api_server.py:2070 — and each path copies the value onto `gen_params.layers` (serving_chat.py:3334, api_server.py:1853, api_server.py:2124). The only live read of `sampling_params.layers` is pipeline_qwen_image_layered.py:667 (`layers = req.sampling_params.layers`), so other diffusion models served through these endpoints will now accept `layers=2` without consuming it. Is accepting-and-ignoring intended, or should the bound be scoped to Qwen-Image-Layered? (evidence: vllm_omni/entrypoints/openai/image_api_utils.py:17 `SUPPORTED_LAYERED_LAYERS_RANGE = range(2, 11)`; vllm_omni/entrypoints/openai/protocol/images.py:93 `return validate_layered_layers(v)`; vllm_omni/entrypoints/openai/serving_chat.py:3287 `layers = validate_layered_layers(layers)`; vllm_omni/entrypoints/openai/api_server.py:2070 `layers = validate_layered_layers(layers)`; vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image_layered.py:667 `layers = req.sampling_params.layers` — repo-wide grep for `sampling_params.layers` returns no other consumer.) + +`tests/entrypoints/openai_api/test_image_server.py:1247` [minor] — The diff changes the lower-bound rejection case at tests/entrypoints/openai_api/test_image_server.py:1247 from `layers=2` to `layers=1`, so the newly valid `layers=2` is never positively asserted in this file. Add `ImageGenerationRequest(prompt="test", layers=2)` (and ideally an HTTP 200 path) to lock in the range(2, 11) boundary. (evidence: tests/entrypoints/openai_api/test_image_server.py:1247 ` ImageGenerationRequest(prompt="test", layers=1)` — inside `with pytest.raises(ValueError):`, with no positive `layers=2` assertion in the file; vllm_omni/entrypoints/openai/image_api_utils.py:17 `SUPPORTED_LAYERED_LAYERS_RANGE = range(2, 11)` — makes 2 the newly valid lower bound.) + +`tests/e2e/online_serving/test_qwen_image_layered_expansion.py:?` [minor] — Verified against the PR-time tree: LAYERS_GUARD_CASES now passes expected_layers=2 with id="layers_guard_001_layers2" (lines 119-120), and test_layered_output_image_count asserts len(image_items) == expected_layers at line 200. PR body confirms the H100 L2 regression was not run. Note PR 4777 is already MERGED, so the ask should be post-merge/CI execution of pytest -m "diffusion and full_model" tests/e2e/online_serving/test_qwen_image_layered_expansion.py::test_layered_output_image_count -k layers_guard_001_layers2 rather than 'before merge'. (evidence: tests/e2e/online_serving/test_qwen_image_layered_expansion.py:119 `2,`; tests/e2e/online_serving/test_qwen_image_layered_expansion.py:120 `id="layers_guard_001_layers2",`; tests/e2e/online_serving/test_qwen_image_layered_expansion.py:200 `assert len(image_items) == expected_layers, (`) + +`vllm_omni/entrypoints/openai/image_api_utils.py:17` [nit] — CI shows only `buildkite/vllm-omni-amd-ci` FAILING (all other reported lanes SUCCESS). Since this diff changes shared OpenAI image entrypoint validation (`SUPPORTED_LAYERED_LAYERS_RANGE = range(2, 11)` at image_api_utils.py:17), please confirm in the PR thread that the AMD failure is pre-existing on `main` and not introduced by the range change; if introduced, fix before merge. (evidence: vllm_omni/entrypoints/openai/image_api_utils.py:17 `SUPPORTED_LAYERED_LAYERS_RANGE = range(2, 11)` — the exact changed shared-entrypoint validation line the comment anchors on. gh_ci_read(4777) returns `buildkite/vllm-omni-amd-ci` with state FAILURE as the only failing check, while build (3.11), build (3.12), pre-commit, DCO, buildkite/vllm-omni, buildkite/vllm-omni-intel-ci, buildkite/vllm-omni-npu-ci, and docs/readthedocs.org:vllm-omni are all SUCCESS.) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v13_proof_test/pr4834.cost.json b/eval/dataset/arms/copilot_v13_proof_test/pr4834.cost.json new file mode 100644 index 000000000..3279c0127 --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_test/pr4834.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 1924.4, + "split": "test", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test/runs/pr4834/run-20260813-154730-99b324", + "recorded_at": "2026-08-13T16:19:30", + "attempts": [ + { + "invocation_id": "inv-7a4522ed", + "outcome": "run:run-20260813-154730-99b324", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-154730-99b324", + "llm_calls": 139, + "input_tokens": 1042484, + "output_tokens": 264226, + "usd": 0.838421, + "source": "spans" + } + ], + "attempt_usd_total": 0.838421, + "llm_calls": 139, + "input_tokens": 1042484, + "output_tokens": 264226, + "usd": 0.838421, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-154730-99b324", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.8384, + "minutes": 32.01, + "ci_minutes": 0, + "input_tokens": 1042484, + "output_tokens": 264226, + "tool_calls": 290, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 2.0526, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 139, + "by_role": { + "agent": { + "usd": 0.809709, + "input_tokens": 1022906, + "output_tokens": 242961, + "calls": 137 + }, + "reducer": { + "usd": 0.028712, + "input_tokens": 19578, + "output_tokens": 21265, + "calls": 2 + } + }, + "cache_read_tokens": 9863040, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 4.1, + "gate": 2.12, + "review": 1914.01, + "report": 0.07 + }, + "time_to_first_result_s": 1920.27 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4834.trace.json.gz", + "trace_bytes": 38961 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v13_proof_test/pr4834.md b/eval/dataset/arms/copilot_v13_proof_test/pr4834.md new file mode 100644 index 000000000..f5c72df68 --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_test/pr4834.md @@ -0,0 +1,59 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4834, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head 5793cd6f4bba) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 6 finding(s) below | +| Security | no finding reported | +| Docs / comments | 2 finding(s) below | +| Behavior / compatibility | no finding reported | +| Correctness | 3 finding(s) below | + +**Validated:** +- [validated] async_omni.py:311 `if _sleeping_tags:` — guard is reachable on every generate() and rejects while any tag is asleep; no off-by-one/None issue in normal construction. +- [validated] async_omni.py:934 `if not hasattr(self, '_sleeping_tags')` and 986 `if hasattr(...)` — redundant; __init__ always sets _sleeping_tags (line 144) and _level2_sleeping. +- [validated] async_omni.py:937 `if level == 2:` — sets _level2_sleeping=True after sleep RPC; flag is sticky because wake_up raises before it can be cleared. +- [sweep] async_omni.py:944 `if getattr(self, '_level2_sleeping', False):` — read in situ; blanket raise blocks diffusion level-2 wake (see review comment 1). +- [validated] async_omni.py:952/954/956 tag filtering — None -> all current tags, provided tags filtered to sleeping ones, empty -> [] idempotent no-op; sequential duplicate wake works as the new test asserts. +- [validated] async_omni.py:990 `if not getattr(self, '_sleeping_tags', None)` — unreachable reset branch: level-2 flag True makes wake_up raise before the discard loop, so this can never execute. +- [validated] vllm_omni/diffusion/data.py CuMemTag added; only consumer is async_omni.py:936 — no other callers need updating. +- [sweep] vllm_omni/diffusion/data.py:1489 hunk — read OmniACK/OmniSleepTask/OmniWakeTask definitions; OmniACK.status is a required str ('SUCCESS'/'ERROR') used by the finding about failed wake. + +`vllm_omni/entrypoints/async_omni.py:944` [blocker] — wake_up() raises NotImplementedError as soon as _level2_sleeping is set (async_omni.py:944), and sleep() sets it for every sleep(level=2) call (async_omni.py:938). DiffusionWorker.sleep(level=2) does save named buffers to CPU (diffusion_worker.py:539) and wake_up restores them (diffusion_worker.py:589), so the blanket guard breaks supported diffusion sleep/wake and makes existing sleep(level=2)->wake_up() tests fail (test_omni_sleep_mode.py:212/221, 269/272, 357/376, 488/492, 544/547). Scope the guard to stages whose worker truly discards weights without a restore path, or update the tests/CI if a global level-2 contract is intentional. (evidence: vllm_omni/entrypoints/async_omni.py:938 `if level == 2:` / `self._level2_sleeping = True`; vllm_omni/entrypoints/async_omni.py:944 `if getattr(self, "_level2_sleeping", False):` / `raise NotImplementedError(`; vllm_omni/diffusion/worker/diffusion_worker.py:539 `self._sleep_saved_buffers = {name: buffer.cpu().clone() for name, buffer in model.named_buffers()}`; vllm_omni/diffusion/worker/diffusion_worker.py:589 `buffer.data.copy_(self._sleep_saved_buffers[name].data)`; tests/entrypoints/test_omni_sleep_mode.py:212 `acks = await diffusion_engine.sleep(stage_ids=[0], level=2)` and :221 `await diffusion_engine.wake_up(stage_ids=[0])`; tests/entrypoints/test_omni_sleep_mode.py:544 `acks = await engine.sleep(stage_ids=[0, 1], level=2)` and :547 `await engine.wake_up(stage_ids=[0, 1])`) + +`vllm_omni/entrypoints/async_omni.py:985` [major] — wake_up() collects worker ACKs with `final_acks.append(ack)` (async_omni.py:981) and then unconditionally discards every requested tag from `_sleeping_tags` (async_omni.py:985-987) without checking `ack.status`. Workers return `OmniACK(status="ERROR")` on failure (worker/base.py:317, diffusion_worker.py:695), so a failed wake clears the guard and lets generate() (async_omni.py:310-317) admit requests against a stage that is still asleep. Only discard tags whose ACK status is "SUCCESS", or raise before mutating state when any ERROR ACK is received. (evidence: vllm_omni/entrypoints/async_omni.py:981 `final_acks.append(ack)`; vllm_omni/entrypoints/async_omni.py:985-987 `for t in requested_tags:` / `if hasattr(self, "_sleeping_tags"):` / `self._sleeping_tags.discard(t)`; vllm_omni/entrypoints/async_omni.py:310-317 `_sleeping_tags = getattr(self, "_sleeping_tags", None)` / `if _sleeping_tags:` / `raise RuntimeError(...)`; vllm_omni/worker/base.py:317 `return OmniACK(task_id=tid, status="ERROR", error_msg=str(e))`; vllm_omni/diffusion/worker/diffusion_worker.py:695 `return OmniACK(task_id=task.task_id, status="ERROR", error_msg=str(e))`) + +`vllm_omni/entrypoints/async_omni.py:936` [major] — sleep() marks a single global `_sleeping_tags` set (async_omni.py:936) without recording which `stage_ids` it slept, and wake_up() computes `requested_tags = list(_current_tags)` (async_omni.py:953) then discards those tags after RPC-ing only the requested stages (async_omni.py:985/987). So sleep(stage_ids=[0,1], level=1) then wake_up(stage_ids=[0]) empties `_sleeping_tags` while stage 1 remains offloaded, and generate()'s guard (async_omni.py:310-311) passes. Track tags per stage (dict[stage_id, set[str]]) or reject partial sleep/wake explicitly; the current tests only wake the same subset they slept, so this asymmetric case is not covered. (evidence: vllm_omni/entrypoints/async_omni.py:936 `self._sleeping_tags.update([CuMemTag.WEIGHTS.value, CuMemTag.KV_CACHE.value])` (global set updated regardless of `stage_ids`); :953 `requested_tags = list(_current_tags)` (full wake defaults to all tracked tags); :985 `for t in requested_tags:`; :987 `self._sleeping_tags.discard(t)` (discard after waking only requested stages); :310-311 `_sleeping_tags = getattr(self, "_sleeping_tags", None)` / `if _sleeping_tags:` (generate guard checks only the global set).) + +`vllm_omni/entrypoints/async_omni.py:936` [major] — sleep() unconditionally records both CuMemTag.WEIGHTS and CuMemTag.KV_CACHE in the engine-global _sleeping_tags (async_omni.py:936), but diffusion stages only allocate a 'weights' CuMem pool (diffusion_model_runner.py:191). On a diffusion-only engine, sleep(level=1) leaves a phantom kv_cache tag: generate() remains rejected after wake_up(tags=["weights"]) because _sleeping_tags is still non-empty, and wake_up(tags=None) forwards ["weights","kv_cache"] to DiffusionWorker.wake_up -> allocator.wake_up(tags) for a pool that was never created. Derive the tag set from each selected stage backend, or add kv_cache only for stages that actually allocate it. (evidence: vllm_omni/entrypoints/async_omni.py:936 `self._sleeping_tags.update([CuMemTag.WEIGHTS.value, CuMemTag.KV_CACHE.value])`; vllm_omni/diffusion/worker/diffusion_model_runner.py:191 `return memory_pool_context_fn(tag="weights")`; vllm_omni/diffusion/worker/diffusion_worker.py:572-573 `allocator.wake_up(tags)` after `def wake_up(self, tags: list[str] | None = None) -> bool` at line 570; vllm_omni/entrypoints/async_omni.py:310-317 `_sleeping_tags = getattr(self, "_sleeping_tags", None)` / `if _sleeping_tags: raise RuntimeError(...)`.) + +`tests/entrypoints/test_omni_sleep_mode.py:?` [major] — The three new regression tests in TestOmniSleepMode (test_omni_sleep_mode.py:412 ff) are marked only @pytest.mark.asyncio + @hardware_test(...). The module pytestmark (lines 21-24) is only advanced_model + usefixtures, hardware_test adds only platform/resource/distributed marks (never omni), and .buildkite/test-amd.yaml:118 selects -m "advanced_model and omni and MI325". The new tests are therefore silently deselected on the MI325 lane and the claimed regression coverage never executes. Add @pytest.mark.omni to each new test (the module-level e2e tests already do at lines 457/501/554) or adjust the lane marker expression. (evidence: tests/entrypoints/test_omni_sleep_mode.py:21-24 `pytestmark = [ + pytest.mark.advanced_model, + pytest.mark.usefixtures("clean_gpu_memory_between_tests"), +]`; tests/entrypoints/test_omni_sleep_mode.py:410-412 ` @pytest.mark.asyncio + @hardware_test(res={"cuda": "H100", "rocm": "MI325"}, num_cards=2) + async def test_level2_sleep_wake_raises(self, llm_engine: AsyncOmni):`; .buildkite/test-amd.yaml:118 ` - pytest -s -v tests/entrypoints/test_omni_sleep_mode.py -m "advanced_model and omni and MI325" --run-level "advanced_model"`) + +`vllm_omni/entrypoints/openai/api_server.py:3388` [major] — PR makes AsyncOmni.wake_up raise NotImplementedError after sleep(level=2), but the /v1/omni/wakeup endpoint (api_server.py:3388) neither catches it nor maps it to a client error, so a level-2 sleep (the OmniSleepRequest default) followed by wake returns an unhandled 500 and leaves app.state.sleeping_stages stale. The mock tests (e.g. test_sleep_then_wakeup_roundtrip) don't exercise the real raise and still assert the old 200-and-cleared contract; update the endpoint and tests to the new level-2 contract, or explicitly reject level-2 sleep via HTTP. (evidence: vllm_omni/entrypoints/async_omni.py:944 ` if getattr(self, "_level2_sleeping", False):` +vllm_omni/entrypoints/async_omni.py:945 ` raise NotImplementedError(` +vllm_omni/entrypoints/openai/api_server.py:3388 ` acks = await engine_client.wake_up(stage_ids=request.stage_ids)` +vllm_omni/entrypoints/openai/api_server.py:3389-3391 ` for sid in request.stage_ids:` / ` if sid in sleeping_set:` / ` sleeping_set.remove(sid)` +tests/entrypoints/openai_api/test_omni_sleep_wakeup.py:194 ` assert app.state.sleeping_stages == {0, 1}` +tests/entrypoints/openai_api/test_omni_sleep_wakeup.py:198 ` assert app.state.sleeping_stages == set()`) + +`vllm_omni/entrypoints/async_omni.py:990` [minor] — Confirmed. `wake_up()`'s level-2 clear at async_omni.py:990-991 is dead code: the guard at async_omni.py:944 raises `NotImplementedError` before line 991 whenever `_level2_sleeping` is True, and when the flag is False the assignment is a no-op. Remove lines 990-991 and the future-support comment, or move a real full-wake clear ahead of the level-2 guard. (evidence: async_omni.py:938 ` self._level2_sleeping = True`; async_omni.py:944 ` if getattr(self, "_level2_sleeping", False):`; async_omni.py:945 ` raise NotImplementedError(`; async_omni.py:990 ` if not getattr(self, "_sleeping_tags", None):`; async_omni.py:991 ` self._level2_sleeping = False`. The only assignments to `_level2_sleeping` are line 938 (set True inside sleep(level=2)) and line 991 (set False), and line 991 is only reachable after the line 944 guard has already returned/raised whenever the flag is True.) + +`tests/diffusion/test_diffusion_streaming_output.py:?` [minor] — AsyncOmni.__init__ now replaces `_is_sleeping` with `_sleeping_tags`/`_level2_sleeping` (async_omni.py:144-145), but tests/diffusion/test_diffusion_streaming_output.py:431 still sets `omni._is_sleeping = False` on a manually-constructed AsyncOmni; generate() and is_sleeping() now read `_sleeping_tags` (async_omni.py:310-311, 1001), so that assignment is dead and the test's awake-state setup is silently ineffective. Replace it with `omni._sleeping_tags = set()` and `omni._level2_sleeping = False`, or delete it. (evidence: tests/diffusion/test_diffusion_streaming_output.py:431 ` omni._is_sleeping = False`; vllm_omni/entrypoints/async_omni.py:144 ` self._sleeping_tags: set[str] = set()`; vllm_omni/entrypoints/async_omni.py:310 ` _sleeping_tags = getattr(self, "_sleeping_tags", None)`; vllm_omni/entrypoints/async_omni.py:311 ` if _sleeping_tags:`; vllm_omni/entrypoints/async_omni.py:1001 ` return bool(getattr(self, "_sleeping_tags", None))`) + +**Additional observations (beyond the comment budget):** +- `tests/entrypoints/test_omni_sleep_mode.py:412` [minor] The new tests declare res={"cuda":"H100","rocm":"MI325"} but the PR Test Plan says the standalone probe ran on 4 x A100 80GB, which would not select these tests under the H100 marker. Please align the hardware markers wi +- `docs/features/sleep_mode.md:60` [minor] wake_up() now raises NotImplementedError after `sleep(level=2)` (vllm_omni/entrypoints/async_omni.py:944-950, gated on `_level2_sleeping` set at line 938), but docs/features/sleep_mode.md still shows `acks = await engine +- `vllm_omni/entrypoints/async_omni.py:953` [minor] wake_up() snapshots `_sleeping_tags` at async_omni.py:952-953 and only discards tags at async_omni.py:985-987, after `await self.collective_rpc(...)` at async_omni.py:974; there is no lock or in-flight marking. Since the + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v13_proof_test/pr4849.cost.json b/eval/dataset/arms/copilot_v13_proof_test/pr4849.cost.json new file mode 100644 index 000000000..f167dbd0d --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_test/pr4849.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 824.7, + "split": "test", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test/runs/pr4849/run-20260813-154730-b0930c", + "recorded_at": "2026-08-13T16:01:11", + "attempts": [ + { + "invocation_id": "inv-6496f756", + "outcome": "run:run-20260813-154730-b0930c", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-154730-b0930c", + "llm_calls": 94, + "input_tokens": 634811, + "output_tokens": 145744, + "usd": 0.507182, + "source": "spans" + } + ], + "attempt_usd_total": 0.507182, + "llm_calls": 94, + "input_tokens": 634811, + "output_tokens": 145744, + "usd": 0.507182, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-154730-b0930c", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.5072, + "minutes": 13.68, + "ci_minutes": 0, + "input_tokens": 634811, + "output_tokens": 145744, + "tool_calls": 161, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.6192, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 94, + "by_role": { + "agent": { + "usd": 0.484433, + "input_tokens": 619102, + "output_tokens": 128950, + "calls": 92 + }, + "reducer": { + "usd": 0.022749, + "input_tokens": 15709, + "output_tokens": 16794, + "calls": 2 + } + }, + "cache_read_tokens": 6498688, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 4.03, + "gate": 1.96, + "review": 814.51, + "report": 0.07 + }, + "time_to_first_result_s": 820.54 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4849.trace.json.gz", + "trace_bytes": 30108 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v13_proof_test/pr4849.md b/eval/dataset/arms/copilot_v13_proof_test/pr4849.md new file mode 100644 index 000000000..fbb554c68 --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_test/pr4849.md @@ -0,0 +1,41 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4849, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head f7bf64f2b275) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 3 finding(s) below | +| Security | no finding reported | +| Docs / comments | 1 finding(s) below | +| Behavior / compatibility | no finding reported | +| Correctness | 1 finding(s) below | + +**Validated:** +- [sweep] Grep 'ar2diffusion' repo-wide: only production caller is the diffusion stage via pipeline config (vllm_omni/model_executor/models/hunyuan_image3/pipeline.py:45); all other hits are the two updated tests or unrelated GLM code. No stale list-expecting consumer remains. +- [sweep] Read vllm_omni/engine/orchestrator.py:1204-1290 _forward_to_next_stage: diffusion branch builds diffusion_source_outputs = [output, *companion_outputs] (line 1211), calls _fn(diffusion_source_outputs, req_state.prompt, requires_multimodal_data), then handles None/empty-list and leaves a single dict intact; new return type is consumed correctly. +- [sweep] Read vllm_omni/model_executor/stage_input_processors/glm_image.py:215-410: GLM ar2diffusion already returns dict|None with source_outputs[0] and identical prompt normalization; Hunyuan now matches that sibling contract. +- [validated] hunyuan_image3.py:114-115 'if not source_outputs: return None' makes line 118 'ar_output = source_outputs[0]' safe against IndexError on empty input. +- [validated] hunyuan_image3.py:118-119 'output = ar_output.outputs[0]' is unchanged from the old loop body; orchestrator parent outputs always carry at least one completion in this path. +- [validated] hunyuan_image3.py:125 'original_prompt = prompt[0] if prompt else {}' is guarded by the ternary; branches at 124/126/128 are equivalent to the old normalization for the single parent case. +- [validated] Branches at hunyuan_image3.py:155, 197, 199, 206, 208, 210, 214, 216, 221 are all reachable and carry over the old loop-body semantics verbatim; no dead/redundant branch introduced. +- [sweep] Read tests/model_executor/stage_input_processors/test_hunyuan_image3_bridge.py fully: new tests cover empty source_outputs -> None, companion presence -> parent-first, single-dict return, ratio override and custom system prompt forwarding; file has pytestmark core_model+cpu. + +`tests/model_executor/stage_input_processors/test_hunyuan_image3_bridge.py:48` [minor] — The diff rewrites `ar2diffusion` to return `dict[str, Any] | None` and adds `test_ar2diffusion_returns_one_request_payload_for_request_level_batching` at tests/model_executor/stage_input_processors/test_hunyuan_image3_bridge.py:48; the file declares `pytestmark = [pytest.mark.core_model, pytest.mark.cpu]` (line 19). .buildkite/test-amd-merge.yml:32 runs `tests/model_executor -m 'core_model and cpu'`, so the AMD lane selects this test. CI shows `buildkite/vllm-omni-amd-ci` and `buildkite/vllm-omni-npu-ci` FAILURE while `buildkite/vllm-omni` and `buildkite/vllm-omni-intel-ci` pass. Confirm whether the two failing lanes are red on main or broke on this PR, and attach the failing logs before treating the PR as clean. (evidence: tests/model_executor/stage_input_processors/test_hunyuan_image3_bridge.py:48 `def test_ar2diffusion_returns_one_request_payload_for_request_level_batching():`; tests/model_executor/stage_input_processors/test_hunyuan_image3_bridge.py:19 `pytestmark = [pytest.mark.core_model, pytest.mark.cpu]`; .buildkite/test-amd-merge.yml:32 `- "timeout 60m pytest -sv tests/model_executor -m 'core_model and cpu' --cov=vllm_omni --cov-report=term-missing:skip-covered --cov-report=xml"`) + +`tests/dfx/perf/tests/test_hunyuan_image3_it2i.json:?` [minor] — The diff changes the shared HunyuanImage3 bridge `ar2diffusion` to return a single `dict[str, Any] | None` (vllm_omni/model_executor/stage_input_processors/hunyuan_image3.py:99). PR #4849's Validation section lists only `test_image_to_image_alignment_online`; issue #4832 also lists `tests/dfx/perf/scripts/run_diffusion_benchmark.py::test_diffusion_performance_benchmark[0-test_hunyuan_image3_ti2i_inline]`, and tests/dfx/perf/tests/test_hunyuan_image3_it2i.json:3 names that node. Please run that benchmark node (or explicitly track it separately) before claiming `Fixes #4832`. (evidence: vllm_omni/model_executor/stage_input_processors/hunyuan_image3.py:99 `) -> dict[str, Any] | None:` +tests/dfx/perf/tests/test_hunyuan_image3_it2i.json:3 ` "test_name": "test_hunyuan_image3_ti2i_inline",` +PR #4849 body: `- \`test_image_to_image_alignment_online\` passed without the list-prompt error.` (Validation lists no benchmark node; only compileall/git diff --check accompany it) +Issue #4832 body: `**Test nodes:** ... \`tests/dfx/perf/scripts/run_diffusion_benchmark.py::test_diffusion_performance_benchmark[0-test_hunyuan_image3_ti2i_inline]\`` and `The diffusion performance benchmark also completed \`0/96\` requests ... then failed the completion assertion.`) + +`vllm_omni/model_executor/stage_input_processors/hunyuan_image3.py:?` [minor] — The new `if isinstance(prompt, list): original_prompt = prompt[0] if prompt else {}` path (vllm_omni/model_executor/stage_input_processors/hunyuan_image3.py:123-124) silently discards prompt entries 1..N for a multi-entry list, whereas the old loop mapped each prompt entry to its output. Reject `len(prompt) > 1` at the bridge boundary or remove `list` from the signature so the stated contract ('genuine multi-request lists remain rejected') is enforced instead of truncating. (evidence: vllm_omni/model_executor/stage_input_processors/hunyuan_image3.py:123-124 `if isinstance(prompt, list):\n original_prompt = prompt[0] if prompt else {}`) + +`tests/engine/test_orchestrator_stage_input_bridge.py:?` [minor] — The diff changes `ar2diffusion` to return `dict | None` (vllm_omni/model_executor/stage_input_processors/hunyuan_image3.py:99), and the orchestrator's diffusion branch still contains list-specific handling at vllm_omni/engine/orchestrator.py:1264 after building `diffusion_source_outputs = [output, *companion_outputs]` at line 1211. tests/engine/test_orchestrator_stage_input_bridge.py:30 only drives `stage_type = "llm"` and never exercises this diffusion path, so the new dict-return contract lacks an orchestrator-level integration test. Add a `stage_type="diffusion"` fake whose `custom_process_input_func` returns the dict for `[parent, *companions]`, and assert `submit_initial` receives the dict unchanged without list iteration. (evidence: vllm_omni/model_executor/stage_input_processors/hunyuan_image3.py:99 `) -> dict[str, Any] | None:`; vllm_omni/engine/orchestrator.py:1211 `diffusion_source_outputs = [output, *companion_outputs]`; vllm_omni/engine/orchestrator.py:1264 `if isinstance(diffusion_prompt, list):`; tests/engine/test_orchestrator_stage_input_bridge.py:30 `self.stage_type = "llm"`) + +`vllm_omni/model_executor/stage_input_processors/hunyuan_image3.py:97` [nit] — The rewritten `ar2diffusion` still declares `requires_multimodal_data: bool = False` (vllm_omni/model_executor/stage_input_processors/hunyuan_image3.py:97) and its docstring says it controls whether multimodal data is forwarded (line 108), but the body reads and forwards `multi_modal_data` unconditionally when present (lines 205-211) and never reads the parameter. Guard the forwarding on the flag, or remove the parameter and update the docstring. (evidence: hunyuan_image3.py:97 ` requires_multimodal_data: bool = False,`; hunyuan_image3.py:108 ` requires_multimodal_data: Whether to forward multimodal data.`; hunyuan_image3.py:205 ` mm_data = original_prompt.get("multi_modal_data")`; hunyuan_image3.py:211 ` diffusion_input["multi_modal_data"] = {"image": prompt_images}` — the parameter is never read anywhere in the function body.) + +**Verdict:** COMMENT diff --git a/eval/dataset/arms/copilot_v13_proof_test/pr4954.cost.json b/eval/dataset/arms/copilot_v13_proof_test/pr4954.cost.json new file mode 100644 index 000000000..6ae2f46a5 --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_test/pr4954.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 1517.0, + "split": "test", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test/runs/pr4954/run-20260813-154730-d2dafe", + "recorded_at": "2026-08-13T16:12:43", + "attempts": [ + { + "invocation_id": "inv-ff963c10", + "outcome": "run:run-20260813-154730-d2dafe", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-154730-d2dafe", + "llm_calls": 153, + "input_tokens": 1070904, + "output_tokens": 262912, + "usd": 0.863861, + "source": "spans" + } + ], + "attempt_usd_total": 0.863861, + "llm_calls": 153, + "input_tokens": 1070904, + "output_tokens": 262912, + "usd": 0.863861, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-154730-d2dafe", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.8639, + "minutes": 25.22, + "ci_minutes": 0, + "input_tokens": 1070904, + "output_tokens": 262912, + "tool_calls": 264, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.965, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 153, + "by_role": { + "agent": { + "usd": 0.818728, + "input_tokens": 1049315, + "output_tokens": 227237, + "calls": 150 + }, + "reducer": { + "usd": 0.045134, + "input_tokens": 21589, + "output_tokens": 35675, + "calls": 3 + } + }, + "cache_read_tokens": 10574592, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 3.62, + "gate": 2.16, + "review": 1507.05, + "report": 0.07 + }, + "time_to_first_result_s": 1512.88 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4954.trace.json.gz", + "trace_bytes": 32463 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v13_proof_test/pr4954.md b/eval/dataset/arms/copilot_v13_proof_test/pr4954.md new file mode 100644 index 000000000..8021cf118 --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_test/pr4954.md @@ -0,0 +1,49 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4954, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head 0b2ef4aea1c2) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 11 finding(s) below | +| Security | no finding reported | +| Docs / comments | no finding reported | +| Behavior / compatibility | no finding reported | +| Correctness | no finding reported | + +**Validated:** +- [sweep] Read all diff hunks in situ: tests/helpers/assertions.py:19-64 and 628-664, tests/helpers/test_assertions.py (whole), tests/model_executor/models/voxtral_tts/test_tts_preprocess.py (whole), vllm_omni/model_executor/models/voxtral_tts/voxtral_tts.py:209-221. +- [sweep] Indexed accesses test_tts_preprocess.py:60 and :77 (`embed_multimodal_calls[0]`) are each preceded by `assert len(model.model.embed_multimodal_calls) == 1` (lines 59 and 76), so the index is safe. +- [sweep] New branches assertions.py:51/56 guard empty preprocessed text/word lists (reachable and correct); assertions.py:658 is the containment gate; voxtral_tts.py:214 legacy fallback is reachable when `codes` is absent or lacks `audio`. +- [sweep] Confirmed `codes` in `model_intermediate_buffer` is always a plain dict (gpu_model_runner.py `_update_intermediate_buffer` builds `existing.setdefault(k, {})`), so `isinstance(codes, Mapping)` holds for the feedback path. +- [validated] Central fix: producer writes nested codes at voxtral_tts_audio_generation.py:1062-1063 (`audio_list = list(torch.split(audio_codes.unsqueeze(1), 1, dim=0))`; `mm_tokens = {"codes": {"audio": audio_list}}`), and the new voxtral_tts.py:212-213 reads the same nested key. +- [validated] Legacy path preserved: voxtral_tts.py:217 still falls back to `info_dict.pop("audio", None)`; test_tts_preprocess_keeps_legacy_top_level_audio_feedback exercises it. +- [validated] Deploy config voxtral_tts.yaml sets `enable_prefix_caching: false` for both stages, so the postprocess/preprocess feedback loop is not affected by the flattened prefix-cache multimodal payload path. +- [sweep] Shared-helper blast radius: `assert_audio_speech_response` is called from tests/helpers/runtime.py:1888/1933/1970 and tests/e2e/offline_inference/test_glm_tts_expansion.py:131, so the new fallback applies to all speech models; only Voxtral e2e was validated in the PR. + +`tests/helpers/test_assertions.py:9` [major] — tests/helpers/test_assertions.py has no pytestmark, so both new tests are deselected by the `-m 'core_model and cpu'` expression in the 'Simple · Other Test' lane (.buildkite/test-merge.yml:29). Repo-wide grep found no pytest_collection_modifyitems/add_marker hook that auto-marks unmarked tests, so nothing compensates for the omission. Add `pytestmark = [pytest.mark.core_model, pytest.mark.cpu]` (as tests/model_executor/models/voxtral_tts/test_tts_preprocess.py:10 does) so the new assertion tests actually run in CI. (evidence: ['.buildkite/test-merge.yml:29 "timeout 20m pytest -sv -m \'core_model and cpu\' --ignore=tests/diffusion --ignore=tests/model_executor --cov=vllm_omni --cov-report=term-missing:skip-covered --cov-report=xml"', 'tests/helpers/test_assertions.py:9 `def test_short_transcript_repeat_passes_containment_fallback():` — the new 24-line file has only `import pytest` plus two unmarked tests; there is no `pytestmark` line.', 'tests/model_executor/models/voxtral_tts/test_tts_preprocess.py:10 `pytestmark = [pytest.mark.core_model, pytest.mark.cpu]`']) + +`tests/helpers/test_assertions.py:9` [major] — tests/helpers/test_assertions.py has no `pytestmark`, and the only merge-pipeline lane that could collect it, 'Simple · Other Test' (.buildkite/test-merge.yml:29), selects with `-m 'core_model and cpu'`, so both new tests are deselected and CI never runs them. Add `pytestmark = [pytest.mark.core_model, pytest.mark.cpu]` (both registered in pyproject.toml) or wire the file into a path-based lane; the PR body's `4 passed` only proves a local run. (evidence: tests/helpers/test_assertions.py:1-25 (whole file) contains only `# SPDX...`, `import pytest`, `from tests.helpers.assertions import _assert_transcript_matches`, `def test_short_transcript_repeat_passes_containment_fallback():` and `def test_short_transcript_unrelated_text_still_fails():` — no `pytestmark` assignment anywhere. .buildkite/test-merge.yml:29 `- "timeout 20m pytest -sv -m 'core_model and cpu' --ignore=tests/diffusion --ignore=tests/model_executor --cov=vllm_omni --cov-report=term-missing:skip-covered --cov-report=xml"`. .buildkite/test-merge.yml:13 `- "timeout 20m pytest -sv tests/diffusion tests/model_executor -m 'core_model and cpu' --cov=vllm_omni --cov-report=term-missing:skip-covered --cov-report=xml"` (the other simple lane only targets those two dirs). pyproject.toml:236 `"core_model: L1&L2 tests (run in each PR)",` and the markers block also registers `"cpu: Tests that run on CPU",`.) + +`tests/helpers/test_assertions.py:9` [major] — `tests/helpers/test_assertions.py` (tests at lines 9 and 16) is not collected by the merge gate: `.buildkite/test-merge.yml:29` selects `pytest -sv -m 'core_model and cpu' --ignore=tests/diffusion --ignore=tests/model_executor`, and this file has no `pytestmark = [pytest.mark.core_model, pytest.mark.cpu]`. Add that marker (as `tests/model_executor/models/voxtral_tts/test_tts_preprocess.py:11` does) so the containment-fallback branch is actually exercised in CI. (evidence: ['.buildkite/test-merge.yml:29 ` - "timeout 20m pytest -sv -m \'core_model and cpu\' --ignore=tests/diffusion --ignore=tests/model_executor --cov=vllm_omni --cov-report=term-missing:skip-covered --cov-report=xml"`', 'tests/helpers/test_assertions.py:4 `import pytest` (file read shows no `pytestmark` line anywhere in the file)', 'tests/model_executor/models/voxtral_tts/test_tts_preprocess.py:11 `pytestmark = [pytest.mark.core_model, pytest.mark.cpu]`']) + +`tests/helpers/assertions.py:?` [major] — tests/helpers/assertions.py:658 — after a cosine miss, `_short_transcript_contains_expected` can accept a negated transcript. With transcript='not how are you' and expected='how are you', `preprocess_text` leaves both texts intact, the short-text (12<=15) and word-count (4<=3+2) guards pass, and 'how are you' in 'not how are you' is True, so the fallback passes a transcript that contradicts the expected text. The substring check has no word-boundary or negation handling. (evidence: tests/helpers/assertions.py:59 ` short_text = min(len(transcript_clean), len(expected_clean)) <= 15`; tests/helpers/assertions.py:60 ` small_word_delta = len(transcript_words) <= len(expected_words) + 2`; tests/helpers/assertions.py:61 ` return short_text and small_word_delta and expected_clean in transcript_clean`; tests/helpers/assertions.py:658-660 ` if _short_transcript_contains_expected(transcript, expected):` / ` print("short speech containment check passed")` / ` return`) + +`tests/model_executor/models/voxtral_tts/test_tts_preprocess.py:51` [minor] — Confirmed. The test feeds a 2D `[1, 4]` tensor as `codes.audio` (test_tts_preprocess.py:51), while the real producer emits 3D `[1, 1, C]` frames (`compute_mm_logits` → `torch.split(audio_codes.unsqueeze(1), 1, dim=0)`), and the real `embed_multimodal` asserts `audio_tokens.dim() == 3` (voxtral_tts_audio_generation.py:1023). `FakeAudioGeneration.embed_multimodal` only records kwargs, so the test never exercises the shape contract and would pass with a shape the real model rejects. Use the production `[1, 1, C]` shape or drive `_parse_and_validate_audio_arrays`. Minor. (evidence: tests/model_executor/models/voxtral_tts/test_tts_preprocess.py:51 ` audio_tokens = torch.tensor([[1, 2, 3, 4]])`; tests/model_executor/models/voxtral_tts/test_tts_preprocess.py:32-34 ` def embed_multimodal(self, **kwargs):` / ` self.embed_multimodal_calls.append(kwargs)` / ` return [torch.full((1, 4), 7.0)]`; vllm_omni/model_executor/models/voxtral_tts/voxtral_tts_audio_generation.py:1023 ` assert audio_tokens.dim() == 3, f"{audio_tokens.ndim=} {audio_tokens.shape=}"`; vllm_omni/model_executor/models/voxtral_tts/voxtral_tts_audio_generation.py:1062 ` audio_list = list(torch.split(audio_codes.unsqueeze(1), 1, dim=0))`) + +`tests/helpers/assertions.py:658` [minor] — This containment fallback is a repo-wide relaxation of the speech transcript gate: _assert_transcript_matches is reached from assert_audio_speech_response (tests/helpers/assertions.py:738) for every TTS model that uses the shared speech helper, not just Voxtral. The PR verification only runs Voxtral's test_speech_english_basic plus the new direct unit tests, so no other model's short-clip gate has been re-validated under the new pass condition. Either scope the fallback to the flaky case or run one short-utterance speech test for another model to show the loosened gate does not mask a regression. (evidence: tests/helpers/assertions.py:658 ` if _short_transcript_contains_expected(transcript, expected):` followed by `print("short speech containment check passed")` and `return` — the fallback lives inside the shared `_assert_transcript_matches`. tests/helpers/assertions.py:738 ` _assert_transcript_matches(` with `threshold=0.9` and `escalation_model=request_config.get("transcript_escalation_model")` inside `assert_audio_speech_response`. tests/helpers/runtime.py:1888 ` assert_audio_speech_response(omni_resp, request_config, run_level=self.run_level)`. tests/e2e/offline_inference/test_glm_tts_expansion.py:131 ` assert_audio_speech_response(` passing `run_level="full_model"` with `{"input": synth_text, "response_format": "wav"}`.) + +`tests/model_executor/models/voxtral_tts/test_tts_preprocess.py:10` [minor] — CI shows `buildkite/vllm-omni-amd-ci` and `buildkite/vllm-omni-npu-ci` FAILURE while `buildkite/vllm-omni` is SUCCESS. This PR adds `tests/model_executor/models/voxtral_tts/test_tts_preprocess.py` (marked `core_model`, `cpu`) and changes shared helper `tests/helpers/assertions.py`, so the AMD/NPU lanes may plausibly run this code. Please confirm the red lanes are pre-existing on main or link the failing logs, since the logs are not visible here. (evidence: tests/model_executor/models/voxtral_tts/test_tts_preprocess.py:10 `pytestmark = [pytest.mark.core_model, pytest.mark.cpu]`; gh_ci_read(4954) returned `{"name": "buildkite/vllm-omni-amd-ci", "state": "FAILURE"}`, `{"name": "buildkite/vllm-omni-npu-ci", "state": "FAILURE"}`, and `{"name": "buildkite/vllm-omni", "state": "SUCCESS"}`.) + +`tests/model_executor/models/voxtral_tts/test_tts_preprocess.py:51` [minor] — This test passes `codes.audio` as a 2-D tensor (`(1, 4)`), but the real producer emits a list of 3-D frames (`(1, 1, C)`) via `compute_mm_logits`, and the real consumer `_parse_and_validate_audio_arrays` asserts `audio_tokens.dim() == 3` for tensor input. FakeAudioGeneration bypasses that assertion, so the test stays green on an invalid shape. Pin the real shape (e.g. `(1, 1, 4)`) or have the fake mirror the dim contract. (evidence: tests/model_executor/models/voxtral_tts/test_tts_preprocess.py:51 `audio_tokens = torch.tensor([[1, 2, 3, 4]])`; vllm_omni/model_executor/models/voxtral_tts/voxtral_tts_audio_generation.py:1062 `audio_list = list(torch.split(audio_codes.unsqueeze(1), 1, dim=0))`; vllm_omni/model_executor/models/voxtral_tts/voxtral_tts_audio_generation.py:1063 `mm_tokens = {"codes": {"audio": audio_list}}`; vllm_omni/model_executor/models/voxtral_tts/voxtral_tts_audio_generation.py:1023 `assert audio_tokens.dim() == 3, f"{audio_tokens.ndim=} {audio_tokens.shape=}"`; tests/model_executor/models/voxtral_tts/test_tts_preprocess.py:32-34 `def embed_multimodal(self, **kwargs):` / `self.embed_multimodal_calls.append(kwargs)` / `return [torch.full((1, 4), 7.0)]`) + +**Additional observations (beyond the comment budget):** +- `tests/model_executor/models/voxtral_tts/test_tts_preprocess.py:10` [minor] CI for this PR: `buildkite/vllm-omni-amd-ci` and `buildkite/vllm-omni-npu-ci` are FAILURE while `buildkite/vllm-omni` is SUCCESS. The new file is CPU/core_model-marked (tests/model_executor/models/voxtral_tts/test_tts_pr +- `tests/helpers/assertions.py:59` [minor] Confirmed: `_short_transcript_contains_expected` (tests/helpers/assertions.py:59-61) is not language-aware. `preprocess_text` strips whitespace between CJK characters (tests/helpers/media.py:556), so a CJK transcript `.s +- `vllm_omni/model_executor/models/voxtral_tts/voxtral_tts.py:213` [minor] This hunk switches tts_preprocess to consume nested `codes.audio` (voxtral_tts.py:213). PR #4954's e2e verification runs only `test_speech_english_basic`, while issue #4949 reports five failing expansion tests (`test_spe + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v13_proof_val/pr4810.cost.json b/eval/dataset/arms/copilot_v13_proof_val/pr4810.cost.json new file mode 100644 index 000000000..148be6d9a --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_val/pr4810.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 1540.3, + "split": "val", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val/runs/pr4810/run-20260813-150851-b382ad", + "recorded_at": "2026-08-13T15:34:28", + "attempts": [ + { + "invocation_id": "inv-5a3df287", + "outcome": "run:run-20260813-150851-b382ad", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-150851-b382ad", + "llm_calls": 148, + "input_tokens": 1325930, + "output_tokens": 266573, + "usd": 1.02102, + "source": "spans" + } + ], + "attempt_usd_total": 1.02102, + "llm_calls": 148, + "input_tokens": 1325930, + "output_tokens": 266573, + "usd": 1.02102, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-150851-b382ad", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 1.021, + "minutes": 25.61, + "ci_minutes": 0, + "input_tokens": 1325930, + "output_tokens": 266573, + "tool_calls": 255, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 2.0256, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 148, + "by_role": { + "agent": { + "usd": 0.996112, + "input_tokens": 1310850, + "output_tokens": 247662, + "calls": 146 + }, + "reducer": { + "usd": 0.024908, + "input_tokens": 15080, + "output_tokens": 18911, + "calls": 2 + } + }, + "cache_read_tokens": 13695872, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 3.9, + "gate": 1.64, + "review": 1530.68, + "report": 0.07 + }, + "time_to_first_result_s": 1536.26 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4810.trace.json.gz", + "trace_bytes": 32497 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v13_proof_val/pr4810.md b/eval/dataset/arms/copilot_v13_proof_val/pr4810.md new file mode 100644 index 000000000..bf69eebce --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_val/pr4810.md @@ -0,0 +1,44 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4810, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head f9665cc223c8) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 6 finding(s) below | +| Security | no finding reported | +| Docs / comments | 2 finding(s) below | +| Behavior / compatibility | no finding reported | +| Correctness | no finding reported | + +**Validated:** +- [sweep] grep 'get_cache_scale' in vllm_omni -> hits: diffusion/models/hunyuan_image3/hunyuan_image3_transformer.py:2238 (stale removed-API call), mimo_audio_llm.py:1157 and mammoth_moda2.py:487 (new get_cache_scale_mapper calls only). +- [sweep] grep 'get_cache_scale_mapper' in vllm_omni -> only the two new direct-loader call sites (mammoth_moda2.py:487, mimo_audio_llm.py:1157); no other direct loader applies the mapper. +- [validated] hunyuan_image3.py:222-226 removed get_cache_scale branch; outer HunyuanImage3ForConditionalGeneration.load_weights at 2178 uses AutoWeightsLoader, so HunyuanModel receives mapped names. +- [validated] mammoth_moda2.py:483-489 applies get_cache_scale_mapper before the loop; test_direct_custom_loaders_apply_cache_scale_mapper asserts mapper.called and mapped load. +- [validated] mimo_audio_llm.py:1153-1160 applies mapper before the loop; outer MiMoAudioForConditionalGeneration.load_weights (mimo_audio.py:701-714) routes model-prefixed weights to this loader. +- [validated] qwen2_old.py:335-340 removed the getattr(get_cache_scale) branch; parent Qwen2ForCausalLM.load_weights at 443-448 uses AutoWeightsLoader. +- [sweep] read all four changed hunks in situ plus the new test file; test has pytestmark=[core_model,cpu] so it is marker-selected. +- [sweep] registry.py:175 maps MiMoAudioLLMModel -> mimo_audio_llm.MiMoAudioLLMForConditionalGeneration, confirming the changed MiMo loader is a registered direct-load path. + +`vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_transformer.py:?` [major] — The PR's purpose is to remove every `get_cache_scale` call, but `vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_transformer.py:2238` still calls `self.quant_config.get_cache_scale(name)` inside `HunyuanImage3Model.load_weights`. `pipeline_hunyuan_image3.py:386` instantiates this model with `quant_config=quant_config` and the pipeline's `load_weights` delegates to it via `AutoWeightsLoader`, so a quantized HunyuanImage3 diffusion checkpoint still hits the removed API and crashes. The new `_STALE_API_FILES` omits this path, so the added regression test passes without covering it. Fix the diffusion loader (use `get_cache_scale_mapper()` or the outer loader) and include it in `_STALE_API_FILES`. (evidence: vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_transformer.py:2238 ` if self.quant_config is not None and (scale_name := self.quant_config.get_cache_scale(name)):`; vllm_omni/diffusion/models/hunyuan_image3/pipeline_hunyuan_image3.py:386 ` self.model = HunyuanImage3Model(self.hf_config, quant_config=quant_config)`; tests/model_executor/models/test_kv_cache_scale_mapper.py:15-20 `_STALE_API_FILES = [` ` "vllm_omni/model_executor/models/hunyuan_image3/hunyuan_image3.py",` ` "vllm_omni/model_executor/models/mammoth_moda2/mammoth_moda2.py",` ` "vllm_omni/model_executor/models/mimo_audio/mimo_audio_llm.py",` ` "vllm_omni/model_executor/models/qwen2_5_omni/qwen2_old.py",` `]` — the diffusion transformer path is absent.) + +`tests/model_executor/models/test_kv_cache_scale_mapper.py:?` [minor] — tests/model_executor/models/test_kv_cache_scale_mapper.py:81 `assert ".get_cache_scale(" not in source` only matches method-call syntax. The removed qwen2_old.py branch was `get_cache_scale = getattr(self.quant_config, "get_cache_scale", None)` followed by `if get_cache_scale is not None and (scale_name := get_cache_scale(name)):`, which contains no `.get_cache_scale(` substring, so the static scan passes for qwen2_old.py both before and after this PR. The Qwen2 behavioral case uses `_QuantConfigWithoutOldCacheScale`, which defines only `get_cache_scale_mapper`; pre-PR, `getattr(self.quant_config, "get_cache_scale", None)` would be `None` and the mapped weight would load via the fall-through path, so this test would not catch a regression. Strengthen the scan (e.g. also reject `getattr(self.quant_config, "get_cache_scale"`) or give the Qwen2 fake quant_config a `get_cache_scale` so the removed branch is actually exercised. (evidence: tests/model_executor/models/test_kv_cache_scale_mapper.py:81 `assert ".get_cache_scale(" not in source`; tests/model_executor/models/test_kv_cache_scale_mapper.py:39-43 `_QuantConfigWithoutOldCacheScale` defines only `get_cache_scale_mapper`; diff-removed qwen2_old.py lines: `get_cache_scale = getattr(self.quant_config, "get_cache_scale", None)` and `if get_cache_scale is not None and (scale_name := get_cache_scale(name)):` — no `.get_cache_scale(` substring, so the static scan and the old-code fall-through both pass.) + +`vllm_omni/model_executor/models/mimo_audio_llm.py:1157` [minor] — This load_weights path now relies on `quant_config.get_cache_scale_mapper()` (external vLLM API) at vllm_omni/model_executor/models/mimo_audio/mimo_audio_llm.py:1157, but requirements/*.txt and setup.py declare no vLLM version floor (vLLM is user-provided). Should the PR document or enforce a minimum vLLM version that provides `get_cache_scale_mapper`? (evidence: vllm_omni/model_executor/models/mimo_audio/mimo_audio_llm.py:1157 `cache_scale_mapper = self.quant_config.get_cache_scale_mapper()` — the changed load_weights calls the external API. setup.py:244 `install_requires = get_install_requires()` and setup.py:249 `install_requires=install_requires,` — dependencies come only from `requirements/{device}.txt`; `requirements/common.txt` contains no `vllm` requirement, so no floor for the API is declared.) + +`tests/model_executor/models/test_kv_cache_scale_mapper.py:144` [minor] — The direct-loader regression only feeds a scalar `torch.ones(())` (line 144), and `_RecordingCacheScaleMapper.apply` passes the tensor through unchanged, so it never exercises the dim>0 shape that the removed `loaded_weight.dim() == 0` slicing used to normalize. Add a non-scalar scale input for mammoth_moda2/mimo_audio and a `model.`-prefixed variant for MiMo (the mapper now runs before the `name.startswith("model.")` rewrite), or verify the mapped `.attn.k_scale` weight_loader accepts the full tensor. (evidence: tests/model_executor/models/test_kv_cache_scale_mapper.py:144 ` loaded = load_weights(fake_model, [(_SOURCE_SCALE_NAME, torch.ones(()))])`; tests/model_executor/models/test_kv_cache_scale_mapper.py:36 ` yield name.replace(".k_proj.output_scale", ".attn.k_scale"), weight`; vllm_omni/model_executor/models/mammoth_moda2/mammoth_moda2.py:489 ` weights = cache_scale_mapper.apply) + +`tests/model_executor/models/test_kv_cache_scale_mapper.py:?` [minor] — The test `test_auto_weights_loader_delegated_loaders_accept_mapped_cache_scale` binds `load_weights = getattr(module, class_name).load_weights` (line 110) and calls it directly at line 112 with the already-mapped `_MAPPED_SCALE_NAME`; no `AutoWeightsLoader` is ever constructed in this file (grep for `AutoWeightsLoader`: no matches). It therefore proves only that the delegated sub-loaders accept a pre-mapped name, not that the outer `AutoWeightsLoader` applies `get_cache_scale_mapper` — the premise making removal of the Hunyuan/Qwen branches safe. Drive the top-level `AutoWeightsLoader` path with a source `...k_proj.output_scale` name, or rename the test to state what it actually asserts. (evidence: tests/model_executor/models/test_kv_cache_scale_mapper.py:23 `_MAPPED_SCALE_NAME = "layers.0.self_attn.attn.k_scale"`; tests/model_executor/models/test_kv_cache_scale_mapper.py:110 `load_weights = getattr(module, class_name).load_weights`; tests/model_executor/models/test_kv_cache_scale_mapper.py:112 `loaded = load_weights(fake_model, [(_MAPPED_SCALE_NAME, torch.ones(()))])`; grep `AutoWeightsLoader` in tests/model_executor/models/test_kv_cache_scale_mapper.py returns no matches.) + +`tests/model_executor/models/test_kv_cache_scale_mapper.py:12` [nit] — The new test file declares `pytestmark = [pytest.mark.core_model, pytest.mark.cpu]` (line 12), and `.buildkite/test-amd-merge.yml:32` runs `timeout 60m pytest -sv tests/model_executor -m 'core_model and cpu' --cov=vllm_omni --cov-report=term-missing:skip-covered --cov-report=xml`, which collects this new file in the AMD merge lane while `buildkite/vllm-omni-amd-ci` and `buildkite/vllm-omni-npu-ci` are FAILURE and `buildkite/vllm-omni` is SUCCESS. Before merging, confirm from logs or main history that these lanes are pre-existing red; if this new test file is the trigger, adjust test collection or lane selection rather than leaving unexplained red gates. (evidence: tests/model_executor/models/test_kv_cache_scale_mapper.py:12 `pytestmark = [pytest.mark.core_model, pytest.mark.cpu]`; .buildkite/test-amd-merge.yml:32 `"timeout 60m pytest -sv tests/model_executor -m 'core_model and cpu' --cov=vllm_omni --cov-report=term-missing:skip-covered --cov-report=xml"`; gh_ci_read PR 4810: `buildkite/vllm-omni-amd-ci` state FAILURE, `buildkite/vllm-omni-npu-ci` state FAILURE, `buildkite/vllm-omni` state SUCCESS.) + +`tests/model_executor/models/test_kv_cache_scale_mapper.py:15` [nit] — Keep this CPU regression test as the guard against reintroducing the removed `get_cache_scale` API; the PR body says it will be removed after validation. Once the diffusion transformer path is fixed, add `vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_transformer.py` to `_STALE_API_FILES` (it still calls `get_cache_scale` at line 2238) rather than deleting the coverage. (evidence: tests/model_executor/models/test_kv_cache_scale_mapper.py:15 `_STALE_API_FILES = [`; tests/model_executor/models/test_kv_cache_scale_mapper.py:81 ` assert ".get_cache_scale(" not in source`; vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_transformer.py:2238 ` if self.quant_config is not None and (scale_name := self.quant_config.get_cache_scale(name)):`) + +`vllm_omni/model_executor/models/mammoth_moda2.py:487` [nit] — Could you confirm whether an outer MammothModa2ForConditionalGeneration / AutoWeightsLoader already applies `get_cache_scale_mapper` before reaching MammothModa2Qwen2ForCausalLM.load_weights? If it does, please clarify that the new per-loader mapper call here is not applying the mapping twice. This could not be verified from repo sources; if it remains unverified, please note it as a follow-up. (evidence: [unverified] MammothModa2ARForConditionalGeneration defines no load_weights and inherits from vLLM Qwen2_5_VLForConditionalGeneration, while top-level MammothModa2ForConditionalGeneration.load_weights delegates to `self.ar.load_weights`; whether an outer AutoWeightsLoader already applies the mapper before reaching MammothModa2Qwen2ForCausalLM.load_weights could not be verified from repo sources, so no double-mapper comment was filed.) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v13_proof_val/pr4816.cost.json b/eval/dataset/arms/copilot_v13_proof_val/pr4816.cost.json new file mode 100644 index 000000000..226ef170e --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_val/pr4816.cost.json @@ -0,0 +1,113 @@ +{ + "wall_s": 1979.7, + "split": "val", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val/runs/pr4816/run-20260813-150851-c82d91", + "recorded_at": "2026-08-13T15:41:47", + "attempts": [ + { + "invocation_id": "inv-78e49ca6", + "outcome": "run:run-20260813-150851-c82d91", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-150851-c82d91", + "llm_calls": 109, + "input_tokens": 1012000, + "output_tokens": 180967, + "usd": 0.763727, + "source": "spans" + } + ], + "attempt_usd_total": 0.763727, + "llm_calls": 109, + "input_tokens": 1012000, + "output_tokens": 180967, + "usd": 0.763727, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-150851-c82d91", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.7637, + "minutes": 32.94, + "ci_minutes": 0, + "input_tokens": 1012000, + "output_tokens": 180967, + "tool_calls": 183, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 2.0352, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 109, + "by_role": { + "planner": { + "usd": 0.000515, + "input_tokens": 87, + "output_tokens": 400, + "calls": 1 + }, + "agent": { + "usd": 0.730978, + "input_tokens": 994731, + "output_tokens": 155537, + "calls": 105 + }, + "reducer": { + "usd": 0.032234, + "input_tokens": 17182, + "output_tokens": 25030, + "calls": 3 + } + }, + "cache_read_tokens": 10793472, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 3.28, + "gate": 1.65, + "review": 1971.25, + "report": 0.07 + }, + "time_to_first_result_s": 1976.22 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4816.trace.json.gz", + "trace_bytes": 24035 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v13_proof_val/pr4816.md b/eval/dataset/arms/copilot_v13_proof_val/pr4816.md new file mode 100644 index 000000000..da56acd85 --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_val/pr4816.md @@ -0,0 +1,36 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4816, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head e1dac23ac02c) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 3 finding(s) below | +| Security | no finding reported | +| Docs / comments | 1 finding(s) below | +| Behavior / compatibility | no finding reported | +| Correctness | no finding reported | + +**Validated:** +- [validated] api_server.py:754,1017,1157,1248,1286,1329 read in situ — pure-diffusion branch sets `serving_tokenization = None`, LLM branch sets `ServingTokenization(...)`, and all four getattr sites use the new name consistently. +- [sweep] grep `openai_serving_tokenization` repo-wide -> no matches; no stale reader/consumer remains. +- [sweep] grep `serving_tokenization` repo-wide -> only api_server.py (6 sites) and tests/entrypoints/openai_api/test_serving_speech.py (3 sites); all updated. +- [sweep] grep `base(raw_request)` -> single caller `_create_speech_error_json_response` at api_server.py:376; grep `instrumentator` -> only the import at api_server.py:67. +- [sweep] test_serving_speech.py:2306-2636 read in situ — `_patch_api_server_base` (2339-2352) patches `api_server_module.base`, so list_voices/upload_voice/delete_voice tests bypass the real base()/state path; only create_speech/batch/audio_generate tests drive the new getattr. +- [validated] test_serving_speech.py:2533,2546,2615 updated tests set `serving_tokenization` and would fail if the create_* getattr rename were reverted. +- [sweep] grep `init_app_state` repo-wide -> only `omni_init_app_state` is used; no upstream `init_app_state` call to reconcile. +- [sweep] pyproject.toml + requirements/common.txt read -> vllm is not pinned in requirements; docs/CI reference vllm==0.24.0 (cuda.inc.md:23), VLLM_VERSION v0.23.0 (.buildkite/pipeline-intel.yaml:13), and vllm==0.22.0 (apps/lvsa-omni-showcase/README.md:35). + +`tests/entrypoints/openai_api/test_serving_speech.py:?` [major] — The three updated tests (2533/2546/2615) only hit the direct `getattr(raw_request.app.state, "serving_tokenization", None)` branches of create_speech/create_speech_batch/create_audio_generate and return `base_server.create_error_response(...)`; they never reach `_create_speech_error_json_response` (call sites begin at api_server.py:1374). The real path — `_create_speech_error_json_response` -> `base(raw_request).create_error_response(...)` (api_server.py:376) — reads app.state via upstream base(), but test_serving_speech.py:2352 mocks `api_server_module.base` for the voice routes, so the rename at api_server.py:754/1017 has no consumer test that would fail if it regressed. Add a regression test that sets `app.state.serving_tokenization` and calls list_voices/upload_voice WITHOUT patching `api_server_module.base`. (evidence: vllm_omni/entrypoints/openai/api_server.py:376 ` err = base(raw_request).create_error_response(` — real upstream base() is the path the fix must feed. vllm_omni/entrypoints/openai/api_server.py:1017 ` state.serving_tokenization = ServingTokenization(` — the renamed assignment with no un-mocked consumer test. tests/entrypoints/openai_api/test_serving_speech.py:2352 ` mocker.patch.object(api_server_module, "base", return_value=fake_base)` — voice-route tests bypass the real base(). tests/entrypoints/openai_api/test_serving_speech.py:2533 ` raw_request.app.state.serving_tokenization = fake_base` — updated test sets state directly and never reaches `_create_speech_error_json_response`.) + +`vllm_omni/entrypoints/openai/api_server.py:?` [minor] — This hunk renames the pure-diffusion slot to `serving_tokenization` but keeps its value `None`. Upstream `base()` (imported at api_server.py:67) now returns `request.app.state.serving_tokenization`, and `_create_speech_error_json_response` calls `base(raw_request).create_error_response(...)` with no None guard, so on a pure-diffusion server every speech error route (`upload_voice`/`delete_voice` ValueError, or `list_voices` with no handler) still raises `AttributeError: 'NoneType' object has no attribute 'create_error_response'` and returns 500 — the exact failure mode this PR is meant to eliminate, only shifted from a missing `State` attribute to `NoneType`. Guard the `base()` result in `_create_speech_error_json_response` (fall back to a tokenization-independent error builder when None), or give diffusion mode a real error-response handler; then add a pure-diffusion test that POSTs an undecodable audio_sample and asserts a 400. (evidence: vllm_omni/entrypoints/openai/api_server.py:754 'state.serving_tokenization = None'; :376 'err = base(raw_request).create_error_response('; :67 'from vllm.entrypoints.serve.instrumentator.basic import base'; :1483 'return _create_speech_error_json_response(raw_request, str(e))' inside upload_voice's `except ValueError`. PR description quotes upstream tokenization() as 'return request.app.state.serving_tokenization'. vllm_omni/entrypoints/openai/serving_speech.py `for_diffusion` (line 416) creates a speech handler that calls `_init_speaker_storage` and its `upload_voice` (line 1214) raises ValueError on undecodable audio, so the None path is reachable in pure diffusion mode.) + +`vllm_omni/entrypoints/openai/api_server.py:1017` [minor] — The rename is internally complete (no `openai_serving_tokenization` references remain; all in-repo readers and the serving_speech tests are updated), but buildkite/vllm-omni-amd-ci, buildkite/vllm-omni-intel-ci, and buildkite/vllm-omni-npu-ci are FAILURE while buildkite/vllm-omni is SUCCESS. Before merging, confirm those three reds are pre-existing on main and not caused by this shared-entrypoint rename. (evidence: vllm_omni/entrypoints/openai/api_server.py:1017 ` state.serving_tokenization = ServingTokenization(` — writer renamed; vllm_omni/entrypoints/openai/api_server.py:1157 ` base_server = getattr(raw_request.app.state, "serving_tokenization", None)` — reader updated; tests/entrypoints/openai_api/test_serving_speech.py:2533 ` raw_request.app.state.serving_tokenization = fake_base` — test updated; repo-wide grep for `openai_serving_tokenization` returns no matches.) + +`vllm_omni/entrypoints/openai/api_server.py:?` [minor] — This rename only works when the installed vLLM's `vllm/entrypoints/serve/instrumentator/basic.py` reads `request.app.state.serving_tokenization` (upstream PR #46022). vLLM is not pinned in requirements, while docs/CI reference several versions (vllm==0.24.0, VLLM_VERSION v0.23.0, vllm==0.22.0). On any of those that predate #46022 the old `base()` reader (`request.app.state.openai_serving_tokenization`) now misses and `_create_speech_error_json_response` regresses to the same AttributeError. Verify the minimum supported vLLM revision and pin/document it. (evidence: requirements/common.txt contains no vllm dependency (grep over requirements/ only hits a comment); docs/getting_started/installation/gpu/cuda.inc.md:23 `uv pip install vllm==0.24.0 --torch-backend=auto`; .buildkite/pipeline-intel.yaml:13 `VLLM_VERSION: "v0.23.0"`; apps/lvsa-omni-showcase/README.md:35 `pip install "vllm==0.22.0"`.) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v13_proof_val/pr4825.cost.json b/eval/dataset/arms/copilot_v13_proof_val/pr4825.cost.json new file mode 100644 index 000000000..2fd3c0def --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_val/pr4825.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 1052.5, + "split": "val", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val/runs/pr4825/run-20260813-150849-f93cce", + "recorded_at": "2026-08-13T15:26:20", + "attempts": [ + { + "invocation_id": "inv-858057cc", + "outcome": "run:run-20260813-150849-f93cce", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-150849-f93cce", + "llm_calls": 121, + "input_tokens": 707195, + "output_tokens": 195578, + "usd": 0.600237, + "source": "spans" + } + ], + "attempt_usd_total": 0.600237, + "llm_calls": 121, + "input_tokens": 707195, + "output_tokens": 195578, + "usd": 0.600237, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-150849-f93cce", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.6002, + "minutes": 17.51, + "ci_minutes": 0, + "input_tokens": 707195, + "output_tokens": 195578, + "tool_calls": 231, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.7334, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 121, + "by_role": { + "agent": { + "usd": 0.57778, + "input_tokens": 695266, + "output_tokens": 178122, + "calls": 119 + }, + "reducer": { + "usd": 0.022457, + "input_tokens": 11929, + "output_tokens": 17456, + "calls": 2 + } + }, + "cache_read_tokens": 7191040, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 3.61, + "gate": 1.61, + "review": 1045.14, + "report": 0.07 + }, + "time_to_first_result_s": 1050.41 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4825.trace.json.gz", + "trace_bytes": 30778 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v13_proof_val/pr4825.md b/eval/dataset/arms/copilot_v13_proof_val/pr4825.md new file mode 100644 index 000000000..48776047f --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_val/pr4825.md @@ -0,0 +1,40 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4825, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head ac3a2ce5a9b9) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 4 finding(s) below | +| Security | no finding reported | +| Docs / comments | 2 finding(s) below | +| Behavior / compatibility | no finding reported | +| Correctness | no finding reported | + +**Validated:** +- [sweep] Read the production hunk in situ (vllm_omni/diffusion/lora/manager.py:368-383); the only production change is adding `"unet"` to `default_components`, no other logic is modified. +- [sweep] Read the added test in situ (tests/diffusion/lora/test_lora_manager.py:580-619); the `if isinstance(layer, _FakeLinearBase)` branch at line 587 is reachable for the `proj` leaf and intentionally not for plain nn.Module containers. +- [validated] The new test is a real regression guard: without `"unet"` in `default_components`, the loop would never scan `pipeline.unet`, leaving `replace_calls`/`_lora_modules` empty and failing the asserts at lines 616-617. +- [validated] Blast radius in-tree: `self.unet` at top level exists only in `pipeline_sdxl.py:110`; the `ovis_image`/`flux2_klein` `unet` hits are docstring text only, and `soulx_singer/rmvpe.py:236` is a nested submodule, not a top-level pipeline attribute. +- [validated] The `_lora_components` opt-in path (sensenova_u1 `pipeline_sensenova_u1.py:543`) is unaffected; a duplicate component in defaults+opt-ins would be skipped by the `full_module_name in self._lora_modules` guard. +- [validated] SDXL packed QKV is wired: `SDXLUNet2DConditionModel.load_weights` sets `stacked_params_mapping` (sdxl_unet.py:823-829) during model load, before `DiffusionLoRAManager` init (diffusion_worker.py `load_model` then `init_lora_manager`), so `_compute_packed_modules_mapping` derives `to_qkv -> to_q/to_k/to_v`. +- [sweep] All `changed_symbol_consumers` references (`_fake_from_layer_diffusion`, `test_lora_manager_discovers_unet_component`, `_matches_target`) resolve inside the touched files; no stale in-repo caller of the changed behavior found. +- [sweep] CI attribution: `gh_ci_read` shows `buildkite/vllm-omni-amd-ci` and `buildkite/vllm-omni-npu-ci` FAILURE; build(3.11/3.12), pre-commit, DCO, buildkite/vllm-omni, intel-ci and docs pass. + +`vllm_omni/diffusion/lora/manager.py:380` [major] — Confirmed at manager.py:380. Adding `"unet"` to default_components routes SDXL `unet.*` names through `_match_target_modules`, but utils.py:25 suffix-matches `.*\.to_out$`; a PEFT `target_modules` list entry `to_out` (without `.0`) therefore matches the `nn.ModuleList` container `...attn1.to_out` while the actual `RowParallelLinear` at `...attn1.to_out.0` fails the match. `from_layer_diffusion` returns the container unchanged, so `to_out` is silently skipped while `to_q/to_k/to_v` are wrapped — a partially-applied adapter with no error. Normalize `to_out.0`/`to_out` before matching, or explicitly reject/document the `to_out.0` spelling, and add a regression test using real SDXL `unet.*.attn1.to_out.0` names. (evidence: ['vllm_omni/diffusion/lora/manager.py:380 ` "unet",` — newly added to default_components, enabling SDXL unet.* matching.', 'vllm_omni/diffusion/lora/utils.py:25 ` re.match(rf".*\\.{target_module}$", module_name) or target_module == module_name` — suffix matcher that matches `...to_out` but not `...to_out.0`.', 'vllm_omni/diffusion/models/sdxl/sdxl_unet.py:151-153 ` self.to_out = nn.ModuleList(` / ` [` / ` RowParallelLinear(inner_dim, dim, bias=True),` — the real linear layer lives at `.to_out.0`, while the container occupies `.to_out`.']) + +`vllm_omni/diffusion/lora/manager.py:380` [minor] — Concrete change: the diff adds "unet" to the hardcoded default_components tuple in vllm_omni/diffusion/lora/manager.py (lines 375-381). This tuple is now another in-repo enumeration of denoiser components alongside vllm_omni/diffusion/registry.py:427 (transformer_attrs = ["transformer", "transformer_2", "dit", "unet"], SP-scoped) and vllm_omni/diffusion/models/sdxl/pipeline_sdxl.py:46 (_dit_modules = ["unet"], per-pipeline). The bug being fixed was exactly a component missing from this parallel hardcoded list, so a separate manual enumeration invites the same drift. Where the mapping allows, derive scanned components from a shared source; otherwise pin the lists together with a comment/test (non-blocking). (evidence: vllm_omni/diffusion/lora/manager.py:375-381 `default_components = (` / `"transformer",` / `"transformer_2",` / `"dit",` / `"bagel",` / `"unet",` / `)`. vllm_omni/diffusion/registry.py:427 `transformer_attrs = ["transformer", "transformer_2", "dit", "unet"]`. vllm_omni/diffusion/models/sdxl/pipeline_sdxl.py:46 `_dit_modules: list[str] = ["unet"]`.) + +`vllm_omni/diffusion/lora/manager.py:380` [minor] — Concrete change confirmed: `vllm_omni/diffusion/lora/manager.py:380` adds `"unet"` to `default_components`. CI read shows `buildkite/vllm-omni-amd-ci` and `buildkite/vllm-omni-npu-ci` as FAILURE while `buildkite/vllm-omni` is SUCCESS. The AMD lane runs `pytest -s -v tests/diffusion/lora/test_images_generations_lora.py` (.buildkite/test-amd.yaml:23), the NPU-ready lane runs `pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py -m 'core_model' --run-level 'core_model'` (.buildkite/test-npu-ready.yaml:21), and the merge lane runs the touched CPU unit test via `timeout 60m pytest -sv tests/diffusion -m 'core_model and cpu' ...` (.buildkite/test-amd-merge.yml:21). These failures may be pre-existing and should be attributed against main before treating them as blockers. (evidence: vllm_omni/diffusion/lora/manager.py:380 ` "unet",`; .buildkite/test-amd.yaml:23 ` - pytest -s -v tests/diffusion/lora/test_images_generations_lora.py`; .buildkite/test-npu-ready.yaml:21 ` - pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py -m 'core_model' --run-level 'core_model'`; .buildkite/test-amd-merge.yml:21 ` - "timeout 60m pytest -sv tests/diffusion -m 'core_model and cpu' --ignore=tests/diffusion/cache/test_teacache_extractors.py --cov=vllm_omni --cov-report=term-missing:skip-covered --cov-report=xml"`; gh_ci_read(4825): `{"bucket":"fail","name":"buildkite/vllm-omni-amd-ci","state":"FAILURE"}`, `{"bucket":"fail","name":"buildkite/vllm-omni-npu-ci","state":"FAILURE"}`, `{"bucket":"pass","name":"buildkite/vllm-omni","state":"SUCCESS"}`.) + +`tests/diffusion/lora/test_lora_manager.py:580` [minor] — New test `test_lora_manager_discovers_unet_component` (`tests/diffusion/lora/test_lora_manager.py:580`) uses `peft_helper = type("_PH", (), {"r": 1})()` (`:613`) with no `target_modules`, so `_matches_target` takes the `target_modules_list is None -> return True` branch and replaces every `_FakeLinearBase` under `unet`; it exercises only the component-scan plumbing, not the `target_modules`-filtered path (`_match_target_modules`) that real SDXL adapters (e.g. `to_q`/`to_k`/`to_v`/`to_out.0`) rely on. The only LoRA e2e (`tests/diffusion/lora/test_images_generations_lora.py:30,57`) targets Z-Image `transformer.layers.0.attention.to_qkv` and never touches `unet`. Minor: consider adding an SDXL/unet test that sets `target_modules` (including a `to_out` variant) or an SDXL LoRA e2e to prove the claimed fix. (evidence: ['tests/diffusion/lora/test_lora_manager.py:613 `peft_helper = type("_PH", (), {"r": 1})()` — helper has no `target_modules`, so the manager\'s `target_modules_list` remains `None`.', 'tests/diffusion/lora/test_lora_manager.py:580 `def test_lora_manager_discovers_unet_component(monkeypatch):`', 'vllm_omni/diffusion/lora/manager.py:380 ` "unet",`', 'vllm_omni/diffusion/lora/manager.py (inside `_replace_layers_with_lora`): `if target_modules_list is None:\n return True\n return _match_target_modules(module_name, target_modules_list)` — proves all modules match when no `target_modules` is present.', 'tests/diffusion/lora/test_images_generations_lora.py:30 `MODEL = "Tongyi-MAI/Z-Image-Turbo"`', 'tests/diffusion/lora/test_images_generations_lora.py:57 ` module_name = "transformer.layers.0.attention.to_qkv"`']) + +`vllm_omni/diffusion/lora/manager.py:376` [minor] — manager.py:376-382: `unet` is now in the global `default_components` tuple, and `_replace_layers_with_lora` iterates it for every pipeline via `hasattr(self.pipeline, component_name)` with no opt-out for out-of-tree pipelines. Please confirm custom pipelines with a top-level `unet` are expected to be scanned, or document how they can opt out when `unet` is not a denoiser. (evidence: vllm_omni/diffusion/lora/manager.py:374 `default_components = (` ... vllm_omni/diffusion/lora/manager.py:380 ` "unet",` vllm_omni/diffusion/lora/manager.py:382 ` extra_components = tuple(getattr(self.pipeline, "_lora_components", ()) or ())` vllm_omni/diffusion/lora/manager.py:383 ` for component_name in (*default_components, *extra_components):` vllm_omni/diffusion/lora/manager.py:384 ` if not hasattr(self.pipeline, component_name):`) + +`tests/diffusion/lora/test_lora_manager.py:580` [nit] — The proposed/actual SDXL LoRA e2e uses `cosmic-tf/sdxl_finetuned_8styles`, but this adapter's config could not be fetched. In this repo SDXL FFN modules are `ff.geglu.proj`/`ff.out_proj`, while diffusers adapters commonly target `ff.net.0.proj`/`ff.net.2`; if the linked adapter uses those names, only attention layers may actually match, weakening the e2e. Please confirm the adapter's `target_modules` or select an adapter that matches SDXL FFN naming so the added SDXL path is meaningfully covered. (evidence: Could not fetch the linked adapter `cosmic-tf/sdxl_finetuned_8styles` config; SDXL FFN naming here is `ff.geglu.proj`/`ff.out_proj` while diffusers adapters commonly target `ff.net.0.proj`/`ff.net.2`, so only attention layers may actually match. Recorded as residual risk, not a review comment.) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v13_proof_val/pr4837.cost.json b/eval/dataset/arms/copilot_v13_proof_val/pr4837.cost.json new file mode 100644 index 000000000..27d4cafa8 --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_val/pr4837.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 1179.2, + "split": "val", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val/runs/pr4837/run-20260813-150850-f17b75", + "recorded_at": "2026-08-13T15:28:27", + "attempts": [ + { + "invocation_id": "inv-c06c2a1e", + "outcome": "run:run-20260813-150850-f17b75", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-150850-f17b75", + "llm_calls": 151, + "input_tokens": 1081831, + "output_tokens": 206241, + "usd": 0.812744, + "source": "spans" + } + ], + "attempt_usd_total": 0.812744, + "llm_calls": 151, + "input_tokens": 1081831, + "output_tokens": 206241, + "usd": 0.812744, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-150850-f17b75", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.8127, + "minutes": 19.61, + "ci_minutes": 0, + "input_tokens": 1081831, + "output_tokens": 206241, + "tool_calls": 238, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.8516, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 151, + "by_role": { + "agent": { + "usd": 0.792538, + "input_tokens": 1072116, + "output_tokens": 190291, + "calls": 149 + }, + "reducer": { + "usd": 0.020206, + "input_tokens": 9715, + "output_tokens": 15950, + "calls": 2 + } + }, + "cache_read_tokens": 10880896, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 3.91, + "gate": 2.36, + "review": 1170.31, + "report": 0.07 + }, + "time_to_first_result_s": 1176.63 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4837.trace.json.gz", + "trace_bytes": 28443 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v13_proof_val/pr4837.md b/eval/dataset/arms/copilot_v13_proof_val/pr4837.md new file mode 100644 index 000000000..418df85bf --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_val/pr4837.md @@ -0,0 +1,45 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4837, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head 78cc4dcbb161) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 1 finding(s) below | +| Security | no finding reported | +| Docs / comments | 3 finding(s) below | +| Behavior / compatibility | no finding reported | +| Correctness | 2 finding(s) below | + +**Validated:** +- [sweep] Read orchestrator.py:1178-1310 in situ; confirmed the new branch `if len(diffusion_prompt) == 1:` (line 1290) unwraps single-element lists for BOTH `submit_update` and `submit_initial` paths. +- [sweep] Read StagePool.submit_initial (stage_pool.py:932) and submit_update (stage_pool.py:1001); both raise ValueError on `isinstance(request, list)`, so the unwrap is required on the initial path and now happens there. +- [sweep] Read thinker2imagegen (ming_flash_omni.py:346); it returns exactly `[{"prompt": "", "extra": extra}]`, so `len(diffusion_prompt) == 1` covers the reported Ming-flash-omni case. +- [sweep] Grepped all `custom_process_input_func` assignments repo-wide; the len>1 list case for diffusion stages remains rejected by StagePool, which is the intended 'no diffusion list batching' contract. +- [sweep] Read deploy/ming_flash_omni_image.yaml; async_chunk=false and a single diffusion stage, so the initial-submit path is the one the issue exercised. +- [validated] pipeline_z_image.py:720 returns `DiffusionOutput(output=image, stage_durations=stage_durations)`, so the new `return outputs` preserves `stage_durations` plus `error/finished/peak_memory_mb/chunk_index/total_chunks` that the old `DiffusionOutput(output=raw)` dropped. +- [sweep] Read tests/engine/test_orchestrator_stage_input_bridge.py and tests/engine/test_orchestrator_kv_sender_info.py; neither drives a non-empty single-element list through the initial-submit diffusion path (kv_sender_info covers empty-list error and plain dict prompt only). +- [sweep] gh_ci_read for PR 4837: pre-commit/build/DCO/buildkite/vllm-omni/intel/docs pass; buildkite/vllm-omni-amd-ci and buildkite/vllm-omni-npu-ci FAILURE — recorded as a minor pre-existing-vs-introduced question, not a blocker. + +`vllm_omni/diffusion/models/ming_flash_omni/pipeline_ming_imagegen.py:449` [blocker] — MingImagePipeline.forward calls super().forward(z_req, prompt=None, height=..., width=..., num_inference_steps=..., guidance_scale=..., prompt_embeds=..., negative_prompt_embeds=..., generator=..., output_type="pt", return_dict=True) at pipeline_ming_imagegen.py:449-462, but the inherited ZImagePipeline.forward signature is def forward(self, req: DiffusionRequestBatch) -> DiffusionOutput: (pipeline_z_image.py:413). That call raises TypeError: ZImagePipeline.forward() got an unexpected keyword argument 'prompt' on the first image-gen request. Additionally, _ZPipelineSamplingParams (pipeline_ming_imagegen.py:63-76) lacks latents, cfg_normalize, extra_args, and output_type, which ZImagePipeline.forward reads at pipeline_z_image.py:464-471, so fixing the signature alone still raises AttributeError. Pass only z_req and add those fields; then rerun the serve+curl smoke. (evidence: vllm_omni/diffusion/models/z_image/pipeline_z_image.py:413 `def forward(self, req: DiffusionRequestBatch) -> DiffusionOutput:`; vllm_omni/diffusion/models/ming_flash_omni/pipeline_ming_imagegen.py:449 `outputs: DiffusionOutput = super().forward(` with keyword args `prompt=None, height=height, width=width, num_inference_steps=num_inference_steps, guidance_scale=guidance_scale, prompt_embeds=prompt_embeds, negative_prompt_embeds=negative_prompt_embeds, generator=generator, output_type="pt", return_dict=True`; vllm_omni/diffusion/models/z_image/pipeline_z_image.py:464 `latents = req.sampling_params.latents`, :466 `cfg_normalization = req.sampling_params.cfg_normalize`, :467 `cfg_truncation = req.sampling_params.extra_args.get("cfg_truncation", 1.0)`, :471 `output_type = req.sampling_params.output_type or "pil"`.) + +`vllm_omni/engine/orchestrator.py:?` [minor] — The removal of `already_submitted and` is the intended fix: a single-element diffusion bridge list is now unwrapped on the initial submit path too, matching the update path and avoiding stage_pool's list-prompt ValueError. The change is correct but untested. Add a regression test in tests/engine/test_orchestrator_kv_sender_info.py whose custom_process_input_func returns [{"prompt": ""}] and assert diffusion_stage.calls[0]["prompt"] is the unwrapped dict, not a list. (evidence: vllm_omni/engine/orchestrator.py:1290 `if len(diffusion_prompt) == 1:` / vllm_omni/engine/orchestrator.py:1291 `diffusion_prompt = diffusion_prompt[0]` — runs before vllm_omni/engine/orchestrator.py:1295 `if already_submitted:`, so it applies to both initial submit and update paths. tests/engine/test_orchestrator_kv_sender_info.py:196 `diffusion_stage.custom_process_input_func = lambda *_args, **_kwargs: []` — the only non-None custom_process_input_func test case in tests/engine; all other files set it to None.) + +`vllm_omni/engine/orchestrator.py:1290` [minor] — Confirmed: with `already_submitted` False, the unwrapped single element is now passed to `submit_initial` (orchestrator.py:1290-1291, 1296-1302), so the len==1 unwrap runs for every diffusion bridge with a custom_process_input_func, not just the Ming path. Previously single-element lists reached initial submit as lists; multi-element lists still pass through unchanged. Please confirm the broader blast radius is intended. (evidence: ['vllm_omni/engine/orchestrator.py:1290 `if len(diffusion_prompt) == 1:`', 'vllm_omni/engine/orchestrator.py:1291 `diffusion_prompt = diffusion_prompt[0]`', 'vllm_omni/engine/orchestrator.py:1296 `await next_pool.submit_update(req_id, req_state, diffusion_prompt)`', 'vllm_omni/engine/orchestrator.py:1301 `diffusion_prompt,` (argument to `submit_initial` when `already_submitted` is False)']) + +`vllm_omni/engine/orchestrator.py:1290` [minor] — Confirmed: PR 4837 CI shows buildkite/vllm-omni-amd-ci and buildkite/vllm-omni-npu-ci as FAILURE while buildkite/vllm-omni and buildkite/vllm-omni-intel-ci pass. This is a triage question only (pre-existing vs introduced), correctly rated minor and non-blocking. (evidence: orchestrator.py:1290 `if len(diffusion_prompt) == 1:` (the diff's changed line; line 1291 is `diffusion_prompt = diffusion_prompt[0]`). gh_ci_read PR 4837 returned: pre-commit SUCCESS, build (3.12) SUCCESS, build (3.11) SUCCESS, DCO SUCCESS, buildkite/vllm-omni-amd-ci FAILURE, buildkite/vllm-omni-npu-ci FAILURE, buildkite/vllm-omni SUCCESS, buildkite/vllm-omni-intel-ci SUCCESS, docs/readthedocs.org:vllm-omni SUCCESS.) + +`vllm_omni/diffusion/models/ming_flash_omni/pipeline_ming_imagegen.py:411` [minor] — pipeline_ming_imagegen.py:411-413 — the comment 'matches ZImagePipeline's contract when prompt_embeds are pre-computed' is stale: ZImagePipeline.forward (pipeline_z_image.py:413) now only accepts `req` and has no `prompt_embeds` parameter on the forward path. Update or remove the comment. (evidence: vllm_omni/diffusion/models/ming_flash_omni/pipeline_ming_imagegen.py:411-413 + # Format prompt_embeds / negative_prompt_embeds as list[Tensor] + # (one entry per request) — matches ZImagePipeline's contract when + # prompt_embeds are pre-computed. +vllm_omni/diffusion/models/z_image/pipeline_z_image.py:413 + def forward(self, req: DiffusionRequestBatch) -> DiffusionOutput:) + +`vllm_omni/diffusion/models/ming_flash_omni/pipeline_ming_imagegen.py:?` [nit] — The diff changes `return DiffusionOutput(output=raw)` to `return outputs` (pipeline_ming_imagegen.py:475), so the returned object now carries `stage_durations` from `ZImagePipeline.forward` instead of discarding them. `MingImagePipeline.__init__` enables the diffusion pipeline profiler, and request_batch.py:47 / output_formatter.py:200 read `diffusion_output.stage_durations`; if propagating those profiling timings is intended, add a one-line comment at `return outputs` to make that explicit — no code change required. (evidence: vllm_omni/diffusion/models/ming_flash_omni/pipeline_ming_imagegen.py:475 `return outputs`; vllm_omni/diffusion/models/z_image/pipeline_z_image.py:719 `stage_durations = self.stage_durations if hasattr(self, "stage_durations") else None`; vllm_omni/diffusion/models/z_image/pipeline_z_image.py:720 `return DiffusionOutput(output=image, stage_durations=stage_durations)`; vllm_omni/diffusion/worker/request_batch.py:47 `stage_durations=result.stage_durations,`; vllm_omni/diffusion/output_formatter.py:200 `stage_durations=diffusion_output.stage_durations,`) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v13_proof_val/pr4893.cost.json b/eval/dataset/arms/copilot_v13_proof_val/pr4893.cost.json new file mode 100644 index 000000000..c2128a545 --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_val/pr4893.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 1117.9, + "split": "val", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val/runs/pr4893/run-20260813-150850-29bbab", + "recorded_at": "2026-08-13T15:27:26", + "attempts": [ + { + "invocation_id": "inv-7f701d67", + "outcome": "run:run-20260813-150850-29bbab", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-150850-29bbab", + "llm_calls": 133, + "input_tokens": 1001920, + "output_tokens": 208039, + "usd": 0.703189, + "source": "spans" + } + ], + "attempt_usd_total": 0.703189, + "llm_calls": 133, + "input_tokens": 1001920, + "output_tokens": 208039, + "usd": 0.703189, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-150850-29bbab", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.7032, + "minutes": 18.58, + "ci_minutes": 0, + "input_tokens": 1001920, + "output_tokens": 208039, + "tool_calls": 238, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.7928, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 133, + "by_role": { + "agent": { + "usd": 0.677661, + "input_tokens": 985311, + "output_tokens": 188943, + "calls": 131 + }, + "reducer": { + "usd": 0.025528, + "input_tokens": 16609, + "output_tokens": 19096, + "calls": 2 + } + }, + "cache_read_tokens": 7549184, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 4.12, + "gate": 1.97, + "review": 1108.73, + "report": 0.07 + }, + "time_to_first_result_s": 1114.87 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4893.trace.json.gz", + "trace_bytes": 35345 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v13_proof_val/pr4893.md b/eval/dataset/arms/copilot_v13_proof_val/pr4893.md new file mode 100644 index 000000000..48dffb179 --- /dev/null +++ b/eval/dataset/arms/copilot_v13_proof_val/pr4893.md @@ -0,0 +1,56 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4893, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head 0128660809e1) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 6 finding(s) below | +| Security | no finding reported | +| Docs / comments | no finding reported | +| Behavior / compatibility | no finding reported | +| Correctness | 2 finding(s) below | + +**Validated:** +- [sweep] Indexed access hunyuan_fused_moe.py:91 hidden_states.shape[0] — read in situ: hidden_states comes from kwargs.get('hidden_states') or args[0], both guarded for None; shape[0] is the MoE batch dim (contract: input rank>=2), consistent with the existing _set_forward_context_num_tokens call. +- [sweep] Branch tests/diffusion/distributed/test_expert_parallel_layout.py:24 (if reduce_scatter is not None) — mirrors vLLM GroupCoordinator capability; reachable only for vllm_* fakes (fake_init_vllm_model_parallel_group passes reduce_scatter), diffusion fakes leave it unset; no dead arm. +- [sweep] Branch parallel_state.py:868/875 (TP vllm-vs-diffusion) — both arms reachable: MoE mapping -> vLLM coordinator, otherwise diffusion tensor group; covered by test_moe_ep_maps... and test_cfg_parallel_keeps_diffusion_dp_without_ep. +- [sweep] Branch parallel_state.py:882 (DP remap) — condition dropped cfg_parallel_size>1 so it now runs for all MoE mappings (incl. cfgp1), matching the tp2_sp2 fix intent; verified test asserts vllm _DP distinct from diffusion _DP with world_size 4 for cfgp2. +- [sweep] Branch hunyuan_fused_moe.py:31/34/38/47 — all reachable: context unavailable -> return; existing dp_metadata -> return; dp_group None or world_size<=1 -> return; any(count is None) -> RuntimeError; error path verified by code read. +- [validated] parallel_state.py:816 sets vllm_parallel_state._DP = diffusion _DP; only the use_moe_parallel_mapping block at :882-885 replaces it with a vLLM coordinator; destroy_model_parallel :919-921 destroys it when distinct — lifecycle is symmetric on the happy path. +- [validated] destroy_model_parallel handles the new vLLM _PCP/_TP/_DP/_EP (parallel_state.py:919-945); _PCP is no longer aliased to _SP so the pre-existing 'is not _SP' guard now destroys it correctly. +- [validated] grep 'vllm_parallel_state._' across vllm_omni finds no consumer of _PCP/_TP/_DP/_EP outside parallel_state.py, so no in-repo caller is left assuming the old diffusion-coordinator API on those slots. + +`tests/diffusion/distributed/test_expert_parallel_layout.py:73` [major] — tests/diffusion/distributed/test_expert_parallel_layout.py:92 monkeypatches `parallel_state.init_vllm_model_parallel_group` with `fake_init_vllm_model_parallel_group`, so the real wrapper (vllm_omni/diffusion/distributed/parallel_state.py:515-527) never runs in CI. The fake injects `device_communicator=object()` and `reduce_scatter=...` (lines 73-74), which are exactly what the new assertions check; deleting `use_device_communicator=True` from the real wrapper would keep every changed test green. Make the fake record/assert the forwarded kwargs, or add a focused test that calls the real wrapper with a mocked `vllm_parallel_state.init_model_parallel_group` and asserts `use_device_communicator=True`. (evidence: tests/diffusion/distributed/test_expert_parallel_layout.py:92 `monkeypatch.setattr(parallel_state, "init_vllm_model_parallel_group", fake_init_vllm_model_parallel_group)` — replaces the real wrapper, so initialize_model_parallel() calls the fake. tests/diffusion/distributed/test_expert_parallel_layout.py:73-74 `device_communicator=object(),` / `reduce_scatter=lambda tensor, **kwargs: tensor,` — the fake injects exactly the two attributes the new assertions check. vllm_omni/diffusion/distributed/parallel_state.py:526 `use_device_communicator=True,` — the real behavior that no test exercises.) + +`vllm_omni/diffusion/models/hunyuan_image3/hunyuan_fused_moe.py:91` [minor] — The diff adds `_set_forward_context_dp_metadata(hidden_states.shape[0])` unconditionally inside `_num_tokens_pre_hook` (hunyuan_fused_moe.py:91). The helper only early-returns when `_vllm_ps._DP` is None or `world_size <= 1` (hunyuan_fused_moe.py:38), so on multi-DP runs where MoE expert-parallel mapping is off `_vllm_ps._DP` is still the diffusion DP coordinator (parallel_state.py:816; only the `use_moe_parallel_mapping` block at :882-885 swaps in a vLLM coordinator) and every forward performs the new `all_gather_object` and sets vLLM `dp_metadata` from a diffusion group. Gate the helper/call to the MoE-mapping path (e.g. require `device_communicator` on `_vllm_ps._DP`) so non-EP multi-DP runs do not take the new collective and set unvalidated vLLM dp_metadata. (evidence: vllm_omni/diffusion/models/hunyuan_image3/hunyuan_fused_moe.py:91 ` _set_forward_context_dp_metadata(hidden_states.shape[0])` — unconditional in the pre-hook. vllm_omni/diffusion/models/hunyuan_image3/hunyuan_fused_moe.py:37 ` dp_group = getattr(_vllm_ps, "_DP", None)`; :38 ` if dp_group is None or getattr(dp_group, "world_size", 1) <= 1:`; :39 ` return`; :42 ` torch.distributed.all_gather_object(`; :45 ` group=dp_group.cpu_group,`; :50 ` forward_context.dp_metadata = _vllm_fc.DPMetadata(torch.tensor(gathered_num_tokens, dtype=torch.int64))` — the helper runs the collective whenever _DP.world_size > 1, regardless of MoE-mapping. vllm_omni/diffusion/distributed/parallel_state.py:816 ` vllm_parallel_state._DP = _DP` — non-mapping path keeps the diffusion coordinator. vllm_omni/diffusion/distributed/parallel_state.py:882 ` if use_moe_parallel_mapping:`; :885 ` vllm_parallel_state._DP = init_vllm_model_parallel_group(` — only this block swaps in the vLLM coordinator.) + +`vllm_omni/diffusion/models/hunyuan_image3/hunyuan_fused_moe.py:91` [minor] — `_num_tokens_pre_hook` (hunyuan_fused_moe.py:85-91) is the only caller of `_set_forward_context_dp_metadata`, but no test invokes it. The new `TestSetForwardContextDPMetadata` tests only the helper directly, so dropping the line 91 call or passing `shape[1]` would pass the suite. Add a test that mocks the FusedMoE factory and calls the registered hook with 1024-row `hidden_states`, asserting both `num_tokens` and `dp_metadata` on the forward context. (evidence: vllm_omni/diffusion/models/hunyuan_image3/hunyuan_fused_moe.py:91 ` _set_forward_context_dp_metadata(hidden_states.shape[0])` inside `_num_tokens_pre_hook`, registered at line 93 ` moe_runner.register_forward_pre_hook(_num_tokens_pre_hook, with_kwargs=True)`; tests/diffusion/models/hunyuan_image3/test_hunyuan_fused_moe.py:74 ` hunyuan_moe._set_forward_context_dp_metadata(1024)` calls the helper directly; grep for `_num_tokens_pre_hook` under `tests/` returns no matches.) + +`vllm_omni/diffusion/models/hunyuan_image3/hunyuan_fused_moe.py:47` [minor] — New TestSetForwardContextDPMetadata covers only the success and keep-existing branches. The `None`-entry failure path at hunyuan_fused_moe.py:47-48 (`if any(count is None ...)` / `raise RuntimeError(...)`) is untested. Add a test where mocked `all_gather_object` leaves a `None` entry and assert `RuntimeError('Failed to gather MoE DP token counts: ...')` fires, so a later silent-default change is caught. (evidence: vllm_omni/diffusion/models/hunyuan_image3/hunyuan_fused_moe.py:47 ` if any(count is None for count in gathered_num_tokens):`; vllm_omni/diffusion/models/hunyuan_image3/hunyuan_fused_moe.py:48 ` raise RuntimeError(f"Failed to gather MoE DP token counts: {gathered_num_tokens}")`; tests/diffusion/models/hunyuan_image3/test_hunyuan_fused_moe.py:56 ` def test_sets_dp_metadata_from_vllm_dp_group(self, mocker):` and :79 ` def test_keeps_existing_dp_metadata(self, mocker):` are the only tests calling `_set_forward_context_dp_metadata`, and no test references the RuntimeError string.) + +`vllm_omni/diffusion/distributed/parallel_state.py:515` [minor] — The new `init_vllm_model_parallel_group` (parallel_state.py:515) calls `vllm_parallel_state.init_model_parallel_group(..., use_device_communicator=True)` and hunyuan_fused_moe.py:50 constructs `_vllm_fc.DPMetadata`, but pyproject.toml/requirements do not declare a vllm lower bound; only docker/Dockerfile.ci pins `VLLM_BASE_TAG=v0.24.0`. The monkeypatched distributed test exercises the fake wrapper, not the real vLLM 0.24 GroupCoordinator path, so CI does not prove the installed vLLM provides these APIs. Consider adding an explicit version constraint/pin or a compatibility test that exercises the real `init_model_parallel_group(use_device_communicator=True)`. (evidence: vllm_omni/diffusion/distributed/parallel_state.py:515 `def init_vllm_model_parallel_group(` ... vllm_omni/diffusion/distributed/parallel_state.py:526 `use_device_communicator=True,` — wrapper passes the new vLLM kwarg. vllm_omni/diffusion/models/hunyuan_image3/hunyuan_fused_moe.py:50 `forward_context.dp_metadata = _vllm_fc.DPMetadata(torch.tensor(gathered_num_tokens, dtype=torch.int64))` — new upstream API used with no version guard. pyproject.toml:10 `dynamic = ["version", "dependencies"]` and pyproject.toml:17 `# Dependencies are now managed dynamically via setup.py...`; requirements/common.txt has only a comment mentioning vllm, no pin. docker/Dockerfile.ci:5 `ARG VLLM_BASE_TAG=v0.24.0` — only CI image pin. tests/diffusion/distributed/test_expert_parallel_layout.py:92 `monkeypatch.setattr(parallel_state, "init_vllm_model_parallel_group", fake_init_vllm_model_parallel_group)` — the real wrapper is bypassed in the test.) + +`vllm_omni/diffusion/distributed/parallel_state.py:882` [minor] — parallel_state.py:882 — the DP remap now runs for every MoE mapping (the `cfg_parallel_size > 1` gate is gone), including cfg_parallel_size=1, but test_expert_parallel_layout.py only exercises cfg_parallel_size=2 (vllm `_DP.world_size == 4`, local_group [0,8,16,24]). Please confirm the broadened cfgp1 behavior is intended and add a cfg_parallel_size=1 MoE-mapping case. (evidence: parallel_state.py:882 ` if use_moe_parallel_mapping:`; parallel_state.py:885-886 ` vllm_parallel_state._DP = init_vllm_model_parallel_group(` / ` group_ranks=rank_generator.get_ranks("cfg-dp"),` — no `cfg_parallel_size > 1` gate remains. PR diff hunk: `- if use_moe_parallel_mapping and cfg_parallel_size > 1:` -> `+ if use_moe_parallel_mapping:`. tests/diffusion/distributed/test_expert_parallel_layout.py:106 ` cfg_parallel_size=2,`; :116 ` assert parallel_state.vllm_parallel_state._DP.world_size == 4` — the only MoE-mapping case asserts the cfgp2 configuration, and a grep for `cfg_parallel_size=1` in that file returns no matches.) + +`vllm_omni/diffusion/distributed/parallel_state.py:885` [minor] — The diff switches `_PCP`/`_TP`/`_DP`/`_EP` to real vLLM `GroupCoordinator`s (`use_device_communicator=True`) whenever `enable_expert_parallel` is selected (parallel_state.py:853-907), and `initialize_model_parallel` runs for every diffusion worker (diffusion_worker.py:306). This is not CUDA-only: the XPU stage 1 config sets `enable_expert_parallel: true` (hunyuan_image_3_moe.yaml:131), and the AMD/NPU CI lanes are currently failing. Confirm `init_vllm_model_parallel_group(use_device_communicator=True)` is validated on XPU/ROCm/NPU, or gate the coordinator switch to CUDA, before merge. (evidence: vllm_omni/diffusion/distributed/parallel_state.py:526 ` use_device_communicator=True,` +vllm_omni/diffusion/distributed/parallel_state.py:885 ` vllm_parallel_state._DP = init_vllm_model_parallel_group(` +vllm_omni/diffusion/worker/diffusion_worker.py:306 ` initialize_model_parallel(` +vllm_omni/deploy/hunyuan_image_3_moe.yaml:131 ` enable_expert_parallel: true`) + +`vllm_omni/diffusion/models/hunyuan_image3/hunyuan_fused_moe.py:50` [nit] — At vllm_omni/diffusion/models/hunyuan_image3/hunyuan_fused_moe.py:50, `forward_context.dp_metadata = _vllm_fc.DPMetadata(torch.tensor(gathered_num_tokens, dtype=torch.int64))` is set whenever metadata is absent. Add a brief comment (or assert on `len(gathered_num_tokens) == dp_group.world_size`) noting this is the diffusion-scoped forward context and that `all_gather_object` results are ordered by DP rank. (evidence: vllm_omni/diffusion/models/hunyuan_image3/hunyuan_fused_moe.py:34 ` if getattr(forward_context, "dp_metadata", None) is not None:` ; vllm_omni/diffusion/models/hunyuan_image3/hunyuan_fused_moe.py:42-50 ` gathered_num_tokens: list[int | None] = [None] * dp_group.world_size + torch.distributed.all_gather_object( + gathered_num_tokens, + int(num_tokens), + group=dp_group.cpu_group, + ) + if any(count is None for count in gathered_num_tokens): + raise RuntimeError(f"Failed to gather MoE DP token counts: {gathered_num_tokens}") + + forward_context.dp_metadata = _vllm_fc.DPMetadata(torch.tensor(gathered_num_tokens, dtype=torch.int64))`) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/judgments/goal_v13_test/JUDGE_REPORT.md b/eval/dataset/judgments/goal_v13_test/JUDGE_REPORT.md new file mode 100644 index 000000000..e29bdd70b --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/JUDGE_REPORT.md @@ -0,0 +1,36 @@ +# Judgment: copilot_v13_proof_test vs claudecode_opus5 + +Judge: gpt-5.6-sol-high (blind, randomized order, 3 replicate(s) x 5 item(s) = 15 verdicts) + +## Wins + +- copilot_v13_proof_test: 7 +- claudecode_opus5: 8 +- tie: 0 + +## Mean rubric scores + +| arm | actionability | gap_hit | precision | recall | +|---|---|---|---|---| +| claudecode_opus5 | 0.95 | 0.20 | 0.72 | 0.76 | +| copilot_v13_proof_test | 0.90 | 0.20 | 0.69 | 0.69 | + +## Per-verdict detail + +| item.rep | winner | margin | rationale (head) | +|---|---|---|---| +| pr4762.r1 | claudecode_opus5 | clear | X covers more ground-truth concerns, notably trust_remote_code regressions, deploy-pipeline policy resolution, cache mutability, positional compatibility, CI, and endpoint auditing. Y is actionable bu | +| pr4762.r2 | claudecode_opus5 | clear | Y covers more ground-truth concerns, including trust-remote-code propagation, mutable caching, positional compatibility, final-pipeline resolution, CI, and unsupported endpoint paths. Both include low | +| pr4762.r3 | claudecode_opus5 | clear | Y covers more ground-truth concerns, including final-pipeline resolution, mutable cached configs, positional compatibility, CI failures, and unsupported endpoint exposure. It also identifies concrete, | +| pr4777.r1 | copilot_v13_proof_test | slight | Y catches both concrete stale DFX tests, while X misses the image-generation case. X is less noisy, but both make weak or contradicted claims about missing acceptance coverage and unrun E2E validation | +| pr4777.r2 | claudecode_opus5 | slight | The ground truth contains no actionable concerns, so recall is vacuously complete for both. Y is more focused and concrete; X includes more speculative or contradicted findings, particularly the unrun | +| pr4777.r3 | claudecode_opus5 | slight | There are no ground-truth concerns, so recall is vacuously complete for both. Both add plausible repo-wide findings, but also overstate testing gaps and repeat an outdated claim that the H100 regressi | +| pr4834.r1 | copilot_v13_proof_test | clear | Both cover the testing, enum, and CI concerns and explicitly identify the latent over-strict level-2 guard. Y is more focused and consistently ties findings to concrete evidence and fixes, while X add | +| pr4834.r2 | copilot_v13_proof_test | clear | Both cover the regression-test and enum concerns and correctly identify the latent over-strict level-2 guard. Y is more focused and consistently evidence-backed; X adds more speculative or low-value f | +| pr4834.r3 | copilot_v13_proof_test | clear | Both candidates catch the latent over-strict level-2 guard, missing CI coverage, global state, and unchecked error ACKs. X stays more tightly grounded in concrete paths and behavior, while Y adds seve | +| pr4849.r1 | copilot_v13_proof_test | clear | Y covers the parent-first ordering concern, explicitly requests the missing benchmark, and flags failing CI that may explain the precommit concern. X is technically detailed but misses the requested b | +| pr4849.r2 | copilot_v13_proof_test | clear | X directly covers the parent-output ordering concern and the requested benchmark, though it misses the explicit precommit issue and adds speculative or pre-existing findings. Y strongly validates pare | +| pr4849.r3 | copilot_v13_proof_test | slight | X covers the parent-ordering concern and explicitly requests the missing benchmark validation, but several findings are speculative or pre-existing. Y is highly concrete and validates parent ordering | +| pr4954.r1 | claudecode_opus5 | clear | Y captures both underlying human concerns: the shared helper’s changed strictness/documentation contract and the questionable live need for the legacy payload branch. X finds several real issues, but | +| pr4954.r2 | claudecode_opus5 | clear | X covers the shared-helper behavioral concern and directly questions whether the legacy fallback has a live producer, though it overstates several speculative risks and misses the exact documentation | +| pr4954.r3 | claudecode_opus5 | clear | X covers the shared-helper contract change and directly questions whether the legacy fallback has a live producer, closely matching both substantive human concerns. Its comments are highly concrete, t | diff --git a/eval/dataset/judgments/goal_v13_test/pr4762.r1.json b/eval/dataset/judgments/goal_v13_test/pr4762.r1.json new file mode 100644 index 000000000..296784a01 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/pr4762.r1.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.8, + "precision": 0.82, + "actionability": 0.95, + "gap_hit": false + }, + "y": { + "recall": 0.55, + "precision": 0.74, + "actionability": 0.9, + "gap_hit": false + }, + "winner": "X", + "margin": "clear", + "rationale": "X covers more ground-truth concerns, notably trust_remote_code regressions, deploy-pipeline policy resolution, cache mutability, positional compatibility, CI, and endpoint auditing. Y is actionable but misses several central review concerns and includes more speculative cache/runtime and sibling-model findings.", + "_usage": { + "input": 3, + "output": 1138 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_test" + }, + "_roles": { + "arm": "copilot_v13_proof_test", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test", + "arm_a_sha256": "6b0172f07026db6023636a8b271947c48c883bb6a04a698a4c7cc81627c36489", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "c615f14501463b018880cd05f5661edb7f27b80dd2a94250edda06d342b8df34", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 1 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_test/pr4762.r2.json b/eval/dataset/judgments/goal_v13_test/pr4762.r2.json new file mode 100644 index 000000000..c405faf48 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/pr4762.r2.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.63, + "precision": 0.7, + "actionability": 0.9, + "gap_hit": false + }, + "y": { + "recall": 0.75, + "precision": 0.78, + "actionability": 0.96, + "gap_hit": false + }, + "winner": "Y", + "margin": "clear", + "rationale": "Y covers more ground-truth concerns, including trust-remote-code propagation, mutable caching, positional compatibility, final-pipeline resolution, CI, and unsupported endpoint paths. Both include lower-value speculative findings, but Y's primary findings are more concrete and technically grounded.", + "_usage": { + "input": 3, + "output": 1979 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "copilot_v13_proof_test", + "Y": "claudecode_opus5" + }, + "_roles": { + "arm": "copilot_v13_proof_test", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test", + "arm_a_sha256": "6b0172f07026db6023636a8b271947c48c883bb6a04a698a4c7cc81627c36489", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "c615f14501463b018880cd05f5661edb7f27b80dd2a94250edda06d342b8df34", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 2 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_test/pr4762.r3.json b/eval/dataset/judgments/goal_v13_test/pr4762.r3.json new file mode 100644 index 000000000..0e3eec2ad --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/pr4762.r3.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.56, + "precision": 0.64, + "actionability": 0.9, + "gap_hit": false + }, + "y": { + "recall": 0.78, + "precision": 0.76, + "actionability": 0.95, + "gap_hit": false + }, + "winner": "Y", + "margin": "clear", + "rationale": "Y covers more ground-truth concerns, including final-pipeline resolution, mutable cached configs, positional compatibility, CI failures, and unsupported endpoint exposure. It also identifies concrete, well-grounded regressions and bypasses, though several polish items are speculative. X is actionable but spends more findings on weak testing, documentation, and cache-invalidation concerns while missing multiple human-review issues.", + "_usage": { + "input": 3, + "output": 1300 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "copilot_v13_proof_test", + "Y": "claudecode_opus5" + }, + "_roles": { + "arm": "copilot_v13_proof_test", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test", + "arm_a_sha256": "6b0172f07026db6023636a8b271947c48c883bb6a04a698a4c7cc81627c36489", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "c615f14501463b018880cd05f5661edb7f27b80dd2a94250edda06d342b8df34", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 3 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_test/pr4777.r1.json b/eval/dataset/judgments/goal_v13_test/pr4777.r1.json new file mode 100644 index 000000000..385d07e98 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/pr4777.r1.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.65, + "precision": 0.62, + "actionability": 0.94, + "gap_hit": false + }, + "y": { + "recall": 0.82, + "precision": 0.58, + "actionability": 0.93, + "gap_hit": false + }, + "winner": "Y", + "margin": "slight", + "rationale": "Y catches both concrete stale DFX tests, while X misses the image-generation case. X is less noisy, but both make weak or contradicted claims about missing acceptance coverage and unrun E2E validation despite the recorded successful checks.", + "_usage": { + "input": 3, + "output": 1356 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_test" + }, + "_roles": { + "arm": "copilot_v13_proof_test", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test", + "arm_a_sha256": "127fd3110d7c2da2857074253d2c427bf6506b3e51c4f10a7cb5702f6d634013", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "2ca1b51d7d415cbacfbfc4d8071f275707165b25374e82be3eaeccc99c7b0891", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 1 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_test/pr4777.r2.json b/eval/dataset/judgments/goal_v13_test/pr4777.r2.json new file mode 100644 index 000000000..952eca69b --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/pr4777.r2.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 1.0, + "precision": 0.55, + "actionability": 0.95, + "gap_hit": false + }, + "y": { + "recall": 1.0, + "precision": 0.68, + "actionability": 0.97, + "gap_hit": false + }, + "winner": "Y", + "margin": "slight", + "rationale": "The ground truth contains no actionable concerns, so recall is vacuously complete for both. Y is more focused and concrete; X includes more speculative or contradicted findings, particularly the unrun-e2e claim and AMD follow-up, although both identify plausible stale tests outside the truncated diff.", + "_usage": { + "input": 3, + "output": 1835 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "copilot_v13_proof_test", + "Y": "claudecode_opus5" + }, + "_roles": { + "arm": "copilot_v13_proof_test", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test", + "arm_a_sha256": "127fd3110d7c2da2857074253d2c427bf6506b3e51c4f10a7cb5702f6d634013", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "2ca1b51d7d415cbacfbfc4d8071f275707165b25374e82be3eaeccc99c7b0891", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 2 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_test/pr4777.r3.json b/eval/dataset/judgments/goal_v13_test/pr4777.r3.json new file mode 100644 index 000000000..a25a922c1 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/pr4777.r3.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 1.0, + "precision": 0.5, + "actionability": 0.96, + "gap_hit": false + }, + "y": { + "recall": 1.0, + "precision": 0.62, + "actionability": 0.97, + "gap_hit": false + }, + "winner": "Y", + "margin": "slight", + "rationale": "There are no ground-truth concerns, so recall is vacuously complete for both. Both add plausible repo-wide findings, but also overstate testing gaps and repeat an outdated claim that the H100 regression was not run, despite the recorded successful validation. Y is slightly more focused and consistently actionable, while X includes more speculative or irrelevant findings.", + "_usage": { + "input": 3, + "output": 1067 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "copilot_v13_proof_test", + "Y": "claudecode_opus5" + }, + "_roles": { + "arm": "copilot_v13_proof_test", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test", + "arm_a_sha256": "127fd3110d7c2da2857074253d2c427bf6506b3e51c4f10a7cb5702f6d634013", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "2ca1b51d7d415cbacfbfc4d8071f275707165b25374e82be3eaeccc99c7b0891", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 3 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_test/pr4834.r1.json b/eval/dataset/judgments/goal_v13_test/pr4834.r1.json new file mode 100644 index 000000000..30988958c --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/pr4834.r1.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.95, + "precision": 0.82, + "actionability": 0.97, + "gap_hit": true + }, + "y": { + "recall": 0.95, + "precision": 0.93, + "actionability": 0.97, + "gap_hit": true + }, + "winner": "Y", + "margin": "clear", + "rationale": "Both cover the testing, enum, and CI concerns and explicitly identify the latent over-strict level-2 guard. Y is more focused and consistently ties findings to concrete evidence and fixes, while X adds several speculative or lower-value claims that reduce precision.", + "_usage": { + "input": 3, + "output": 1231 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_test" + }, + "_roles": { + "arm": "copilot_v13_proof_test", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test", + "arm_a_sha256": "b2e9e7f194924e5a699b414eb116e6078eb4a4f20ab5696c304cf093a4259cc0", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "05ba88edbe75596d53ffc0adcf76df898385eb4f6f271110f37f19da2cac4adf", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 1 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_test/pr4834.r2.json b/eval/dataset/judgments/goal_v13_test/pr4834.r2.json new file mode 100644 index 000000000..8d783213e --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/pr4834.r2.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.67, + "precision": 0.84, + "actionability": 0.95, + "gap_hit": true + }, + "y": { + "recall": 0.67, + "precision": 0.92, + "actionability": 0.96, + "gap_hit": true + }, + "winner": "Y", + "margin": "clear", + "rationale": "Both cover the regression-test and enum concerns and correctly identify the latent over-strict level-2 guard. Y is more focused and consistently evidence-backed; X adds more speculative or low-value findings and incorrectly ties the known unrelated CI failure to this change.", + "_usage": { + "input": 3, + "output": 1368 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_test" + }, + "_roles": { + "arm": "copilot_v13_proof_test", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test", + "arm_a_sha256": "b2e9e7f194924e5a699b414eb116e6078eb4a4f20ab5696c304cf093a4259cc0", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "05ba88edbe75596d53ffc0adcf76df898385eb4f6f271110f37f19da2cac4adf", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 2 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_test/pr4834.r3.json b/eval/dataset/judgments/goal_v13_test/pr4834.r3.json new file mode 100644 index 000000000..3f1704c13 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/pr4834.r3.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 1.0, + "precision": 0.95, + "actionability": 0.98, + "gap_hit": true + }, + "y": { + "recall": 1.0, + "precision": 0.86, + "actionability": 0.96, + "gap_hit": true + }, + "winner": "X", + "margin": "clear", + "rationale": "Both candidates catch the latent over-strict level-2 guard, missing CI coverage, global state, and unchecked error ACKs. X stays more tightly grounded in concrete paths and behavior, while Y adds several speculative or unsafe recommendations, including clearing sleep state despite discarded weights and tangential upstream-layer critiques.", + "_usage": { + "input": 3, + "output": 649 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "copilot_v13_proof_test", + "Y": "claudecode_opus5" + }, + "_roles": { + "arm": "copilot_v13_proof_test", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test", + "arm_a_sha256": "b2e9e7f194924e5a699b414eb116e6078eb4a4f20ab5696c304cf093a4259cc0", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "05ba88edbe75596d53ffc0adcf76df898385eb4f6f271110f37f19da2cac4adf", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 3 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_test/pr4849.r1.json b/eval/dataset/judgments/goal_v13_test/pr4849.r1.json new file mode 100644 index 000000000..01dcbffe3 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/pr4849.r1.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.4, + "precision": 0.68, + "actionability": 0.94, + "gap_hit": false + }, + "y": { + "recall": 0.85, + "precision": 0.62, + "actionability": 0.9, + "gap_hit": false + }, + "winner": "Y", + "margin": "clear", + "rationale": "Y covers the parent-first ordering concern, explicitly requests the missing benchmark, and flags failing CI that may explain the precommit concern. X is technically detailed but misses the requested benchmark and adds several speculative, pre-existing, or out-of-scope findings.", + "_usage": { + "input": 3, + "output": 1157 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_test" + }, + "_roles": { + "arm": "copilot_v13_proof_test", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test", + "arm_a_sha256": "ce0266983c595171c18dd5c48c208bbb2687522be8a7c7e913135ac8252e59df", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "5eabb9c2ecaee56109ca8916c98f608c63cff4946702c512694924d4704baea8", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 1 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_test/pr4849.r2.json b/eval/dataset/judgments/goal_v13_test/pr4849.r2.json new file mode 100644 index 000000000..be5b6d017 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/pr4849.r2.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.67, + "precision": 0.6, + "actionability": 0.92, + "gap_hit": false + }, + "y": { + "recall": 0.5, + "precision": 0.55, + "actionability": 0.94, + "gap_hit": false + }, + "winner": "X", + "margin": "clear", + "rationale": "X directly covers the parent-output ordering concern and the requested benchmark, though it misses the explicit precommit issue and adds speculative or pre-existing findings. Y strongly validates parent ordering and flags failing CI, but does not request the specified benchmark and spends substantial attention on adjacent or pre-existing issues.", + "_usage": { + "input": 3, + "output": 1093 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "copilot_v13_proof_test", + "Y": "claudecode_opus5" + }, + "_roles": { + "arm": "copilot_v13_proof_test", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test", + "arm_a_sha256": "ce0266983c595171c18dd5c48c208bbb2687522be8a7c7e913135ac8252e59df", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "5eabb9c2ecaee56109ca8916c98f608c63cff4946702c512694924d4704baea8", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 2 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_test/pr4849.r3.json b/eval/dataset/judgments/goal_v13_test/pr4849.r3.json new file mode 100644 index 000000000..0847ef10d --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/pr4849.r3.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.63, + "precision": 0.55, + "actionability": 0.92, + "gap_hit": false + }, + "y": { + "recall": 0.48, + "precision": 0.58, + "actionability": 0.94, + "gap_hit": false + }, + "winner": "X", + "margin": "slight", + "rationale": "X covers the parent-ordering concern and explicitly requests the missing benchmark validation, but several findings are speculative or pre-existing. Y is highly concrete and validates parent ordering well, yet misses the requested benchmark and precommit concerns while adding substantial out-of-scope follow-up commentary.", + "_usage": { + "input": 3, + "output": 1060 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "copilot_v13_proof_test", + "Y": "claudecode_opus5" + }, + "_roles": { + "arm": "copilot_v13_proof_test", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test", + "arm_a_sha256": "ce0266983c595171c18dd5c48c208bbb2687522be8a7c7e913135ac8252e59df", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "5eabb9c2ecaee56109ca8916c98f608c63cff4946702c512694924d4704baea8", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 3 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_test/pr4954.r1.json b/eval/dataset/judgments/goal_v13_test/pr4954.r1.json new file mode 100644 index 000000000..8514e2b85 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/pr4954.r1.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.35, + "precision": 0.64, + "actionability": 0.9, + "gap_hit": false + }, + "y": { + "recall": 0.85, + "precision": 0.72, + "actionability": 0.96, + "gap_hit": false + }, + "winner": "Y", + "margin": "clear", + "rationale": "Y captures both underlying human concerns: the shared helper’s changed strictness/documentation contract and the questionable live need for the legacy payload branch. X finds several real issues, but misses the documentation concern and heavily duplicates findings; Y is more actionable, though its final robustness claims are speculative.", + "_usage": { + "input": 3, + "output": 1070 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "copilot_v13_proof_test", + "Y": "claudecode_opus5" + }, + "_roles": { + "arm": "copilot_v13_proof_test", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test", + "arm_a_sha256": "9c0bc66cc0ec8c5b6746a9b3f7c1f0737dcd37bc9280b775127e2be759f2a9f4", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "876219c8b0b10edabcc0c302eb8ce6951e35bdecf6437bbb6247e39bc44276e3", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 1 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_test/pr4954.r2.json b/eval/dataset/judgments/goal_v13_test/pr4954.r2.json new file mode 100644 index 000000000..0d8629ab2 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/pr4954.r2.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.75, + "precision": 0.72, + "actionability": 0.91, + "gap_hit": false + }, + "y": { + "recall": 0.35, + "precision": 0.7, + "actionability": 0.78, + "gap_hit": false + }, + "winner": "X", + "margin": "clear", + "rationale": "X covers the shared-helper behavioral concern and directly questions whether the legacy fallback has a live producer, though it overstates several speculative risks and misses the exact documentation request. Y identifies valid CI-marker, containment, and tensor-shape issues, but largely misses the ground-truth documentation and legacy-path concerns while repeating findings multiple times.", + "_usage": { + "input": 3, + "output": 1339 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_test" + }, + "_roles": { + "arm": "copilot_v13_proof_test", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test", + "arm_a_sha256": "9c0bc66cc0ec8c5b6746a9b3f7c1f0737dcd37bc9280b775127e2be759f2a9f4", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "876219c8b0b10edabcc0c302eb8ce6951e35bdecf6437bbb6247e39bc44276e3", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 2 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_test/pr4954.r3.json b/eval/dataset/judgments/goal_v13_test/pr4954.r3.json new file mode 100644 index 000000000..015240232 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_test/pr4954.r3.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.85, + "precision": 0.74, + "actionability": 0.94, + "gap_hit": false + }, + "y": { + "recall": 0.3, + "precision": 0.68, + "actionability": 0.7, + "gap_hit": false + }, + "winner": "X", + "margin": "clear", + "rationale": "X covers the shared-helper contract change and directly questions whether the legacy fallback has a live producer, closely matching both substantive human concerns. Its comments are highly concrete, though several production/test assertions are speculative or overstated. Y finds valid ancillary issues but misses the documentation concern, does not meaningfully challenge the legacy path, and repeats findings.", + "_usage": { + "input": 3, + "output": 1174 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_test" + }, + "_roles": { + "arm": "copilot_v13_proof_test", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_test", + "arm_a_sha256": "9c0bc66cc0ec8c5b6746a9b3f7c1f0737dcd37bc9280b775127e2be759f2a9f4", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "876219c8b0b10edabcc0c302eb8ce6951e35bdecf6437bbb6247e39bc44276e3", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 3 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_val/JUDGE_REPORT.md b/eval/dataset/judgments/goal_v13_val/JUDGE_REPORT.md new file mode 100644 index 000000000..20080a2d4 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/JUDGE_REPORT.md @@ -0,0 +1,36 @@ +# Judgment: copilot_v13_proof_val vs claudecode_opus5 + +Judge: gpt-5.6-sol-high (blind, randomized order, 3 replicate(s) x 5 item(s) = 15 verdicts) + +## Wins + +- copilot_v13_proof_val: 8 +- claudecode_opus5: 7 +- tie: 0 + +## Mean rubric scores + +| arm | actionability | gap_hit | precision | recall | +|---|---|---|---|---| +| claudecode_opus5 | 0.93 | 0.20 | 0.59 | 0.89 | +| copilot_v13_proof_val | 0.90 | 0.20 | 0.56 | 0.86 | + +## Per-verdict detail + +| item.rep | winner | margin | rationale (head) | +|---|---|---|---| +| pr4810.r1 | claudecode_opus5 | slight | Both catch the missed diffusion-loader caller and identify weaknesses in fake-only testing. X provides the clearest causal analysis and exact fix, but its claim that the test is absent from CI is cont | +| pr4810.r2 | copilot_v13_proof_val | slight | Both correctly identify the missed diffusion-loader caller and provide concrete fixes. X more directly covers the human concern about fake-parameter coverage, while Y makes false CI claims, including | +| pr4810.r3 | copilot_v13_proof_val | slight | Both correctly identify the missed HunyuanImage3 diffusion caller and provide concrete remediation. X more directly covers the human concern about fake-only loader validation, while Y adds unsupported | +| pr4816.r1 | copilot_v13_proof_val | slight | The ground truth contains no substantive reviewer concerns, so recall is vacuously complete for both. Both offer concrete, plausible comments, but several rely on unverifiable repository/CI details an | +| pr4816.r2 | copilot_v13_proof_val | slight | There are no substantive ground-truth concerns, so recall is vacuously equal. Both reviews add speculative findings beyond the approved mechanical rename; Y is slightly more precise because it avoids | +| pr4816.r3 | copilot_v13_proof_val | slight | There are no substantive ground-truth concerns, so recall is vacuously complete. Both candidates over-review an approved mechanical rename with speculative scope expansion and unsupported compatibilit | +| pr4825.r1 | claudecode_opus5 | clear | X more directly covers the naming-mapping concern and provides concrete, grounded fixes and tests, though several findings extend beyond the PR’s scope. Y identifies useful test and metadata gaps, but | +| pr4825.r2 | claudecode_opus5 | clear | X substantially covers the naming-mapping concern and gives concrete locations and remedies, though several CI, test-structure, and pre-existing compatibility findings exceed the reviewed diff’s scope | +| pr4825.r3 | claudecode_opus5 | clear | X substantially covers the naming/mapping concern and validation expectations with concrete locations and remedies, though several findings are pre-existing or beyond this narrow diff. Y offers action | +| pr4837.r1 | claudecode_opus5 | clear | Both candidates fully cover the ground-truth rationale for unconditional singleton-list normalization. X provides more complete, concrete analysis of the Ming incompatibility, while Y adds duplicative | +| pr4837.r2 | claudecode_opus5 | clear | Both candidates accurately cover the orchestrator rationale in the ground truth. Y more completely diagnoses the valid Ming pipeline incompatibility, especially the missing prompt-embedding route, whi | +| pr4837.r3 | claudecode_opus5 | clear | Both candidates fully cover the sole ground-truth concern and explain why singleton normalization must apply to both submission paths. X is more technically complete and actionable, especially by trac | +| pr4893.r1 | copilot_v13_proof_val | clear | Both candidates catch the sole ground-truth concern: the reduce_scatter assertions validate attributes injected by the fake rather than the real wrapper. X is more precise overall, while Y adds more s | +| pr4893.r2 | copilot_v13_proof_val | clear | Both candidates directly identify the ground-truth gap: the mocked coordinator supplies reduce_scatter itself, so the real wrapper behavior is not verified. X has fewer speculative or irrelevant findi | +| pr4893.r3 | copilot_v13_proof_val | slight | Both identify the sole substantive ground-truth concern: the reduce_scatter assertions validate mock-injected behavior rather than the real wrapper. Y is slightly more precise and equally concrete, wh | diff --git a/eval/dataset/judgments/goal_v13_val/pr4810.r1.json b/eval/dataset/judgments/goal_v13_val/pr4810.r1.json new file mode 100644 index 000000000..01af4e440 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/pr4810.r1.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.8, + "precision": 0.72, + "actionability": 0.94, + "gap_hit": true + }, + "y": { + "recall": 0.85, + "precision": 0.68, + "actionability": 0.9, + "gap_hit": true + }, + "winner": "X", + "margin": "slight", + "rationale": "Both catch the missed diffusion-loader caller and identify weaknesses in fake-only testing. X provides the clearest causal analysis and exact fix, but its claim that the test is absent from CI is contradicted by Y's concrete pipeline evidence. Y covers the runtime-coverage concern slightly better, though speculative versioning, CI, and double-mapping comments reduce precision.", + "_usage": { + "input": 3, + "output": 1403 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_val" + }, + "_roles": { + "arm": "copilot_v13_proof_val", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val", + "arm_a_sha256": "d49e3c49e8f0c08a156355ba589cec17da074e0868d9cb981d607052b1e75a02", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "62ede4acd697d051f739a163828672d894ec4150b7b17c941bb4b33f4b81409e", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 1 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_val/pr4810.r2.json b/eval/dataset/judgments/goal_v13_val/pr4810.r2.json new file mode 100644 index 000000000..5117c3c21 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/pr4810.r2.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.8, + "precision": 0.6, + "actionability": 0.9, + "gap_hit": true + }, + "y": { + "recall": 0.65, + "precision": 0.55, + "actionability": 0.92, + "gap_hit": true + }, + "winner": "X", + "margin": "slight", + "rationale": "Both correctly identify the missed diffusion-loader caller and provide concrete fixes. X more directly covers the human concern about fake-parameter coverage, while Y makes false CI claims, including that dynamic imports inside test functions occur at collection time; X is also noisy with several speculative findings, keeping the margin slight.", + "_usage": { + "input": 3, + "output": 1685 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "copilot_v13_proof_val", + "Y": "claudecode_opus5" + }, + "_roles": { + "arm": "copilot_v13_proof_val", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val", + "arm_a_sha256": "d49e3c49e8f0c08a156355ba589cec17da074e0868d9cb981d607052b1e75a02", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "62ede4acd697d051f739a163828672d894ec4150b7b17c941bb4b33f4b81409e", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 2 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_val/pr4810.r3.json b/eval/dataset/judgments/goal_v13_val/pr4810.r3.json new file mode 100644 index 000000000..b6223d378 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/pr4810.r3.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.88, + "precision": 0.62, + "actionability": 0.9, + "gap_hit": true + }, + "y": { + "recall": 0.62, + "precision": 0.58, + "actionability": 0.94, + "gap_hit": true + }, + "winner": "X", + "margin": "slight", + "rationale": "Both correctly identify the missed HunyuanImage3 diffusion caller and provide concrete remediation. X more directly covers the human concern about fake-only loader validation, while Y adds unsupported or incorrect claims about CI wiring and collection-time imports. X is also noisy with speculative compatibility, CI, and double-mapping concerns, limiting its precision.", + "_usage": { + "input": 3, + "output": 1965 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "copilot_v13_proof_val", + "Y": "claudecode_opus5" + }, + "_roles": { + "arm": "copilot_v13_proof_val", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val", + "arm_a_sha256": "d49e3c49e8f0c08a156355ba589cec17da074e0868d9cb981d607052b1e75a02", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "62ede4acd697d051f739a163828672d894ec4150b7b17c941bb4b33f4b81409e", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 3 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_val/pr4816.r1.json b/eval/dataset/judgments/goal_v13_val/pr4816.r1.json new file mode 100644 index 000000000..0eb8080f3 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/pr4816.r1.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 1.0, + "precision": 0.5, + "actionability": 0.94, + "gap_hit": false + }, + "y": { + "recall": 1.0, + "precision": 0.58, + "actionability": 0.93, + "gap_hit": false + }, + "winner": "Y", + "margin": "slight", + "rationale": "The ground truth contains no substantive reviewer concerns, so recall is vacuously complete for both. Both offer concrete, plausible comments, but several rely on unverifiable repository/CI details and speculative compatibility risks; Y is slightly more focused and avoids X's additional sibling-state speculation.", + "_usage": { + "input": 3, + "output": 1163 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_val" + }, + "_roles": { + "arm": "copilot_v13_proof_val", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val", + "arm_a_sha256": "db77e7acdd47554d47708715e1e68a9a2beb888bfe1661fb483733b01cf2f0cb", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "b9fe4c00140a73cf5cc08ceded6d03eb8b8faea45f6c3b50e6f9753b9a1a1a01", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 1 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_val/pr4816.r2.json b/eval/dataset/judgments/goal_v13_val/pr4816.r2.json new file mode 100644 index 000000000..361f25737 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/pr4816.r2.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 1.0, + "precision": 0.45, + "actionability": 0.95, + "gap_hit": false + }, + "y": { + "recall": 1.0, + "precision": 0.52, + "actionability": 0.95, + "gap_hit": false + }, + "winner": "Y", + "margin": "slight", + "rationale": "There are no substantive ground-truth concerns, so recall is vacuously equal. Both reviews add speculative findings beyond the approved mechanical rename; Y is slightly more precise because it avoids X's especially tenuous sibling-state audit, while both provide concrete locations and proposed changes.", + "_usage": { + "input": 3, + "output": 1162 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_val" + }, + "_roles": { + "arm": "copilot_v13_proof_val", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val", + "arm_a_sha256": "db77e7acdd47554d47708715e1e68a9a2beb888bfe1661fb483733b01cf2f0cb", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "b9fe4c00140a73cf5cc08ceded6d03eb8b8faea45f6c3b50e6f9753b9a1a1a01", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 2 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_val/pr4816.r3.json b/eval/dataset/judgments/goal_v13_val/pr4816.r3.json new file mode 100644 index 000000000..303f24ef7 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/pr4816.r3.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 1.0, + "precision": 0.2, + "actionability": 0.9, + "gap_hit": false + }, + "y": { + "recall": 1.0, + "precision": 0.25, + "actionability": 0.9, + "gap_hit": false + }, + "winner": "Y", + "margin": "slight", + "rationale": "There are no substantive ground-truth concerns, so recall is vacuously complete. Both candidates over-review an approved mechanical rename with speculative scope expansion and unsupported compatibility/CI concerns, though the test-coverage observation is grounded and actionable. Y is slightly more focused and avoids X's additional sibling-state audit.", + "_usage": { + "input": 3, + "output": 769 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_val" + }, + "_roles": { + "arm": "copilot_v13_proof_val", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val", + "arm_a_sha256": "db77e7acdd47554d47708715e1e68a9a2beb888bfe1661fb483733b01cf2f0cb", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "b9fe4c00140a73cf5cc08ceded6d03eb8b8faea45f6c3b50e6f9753b9a1a1a01", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 3 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_val/pr4825.r1.json b/eval/dataset/judgments/goal_v13_val/pr4825.r1.json new file mode 100644 index 000000000..bc10149c0 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/pr4825.r1.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.65, + "precision": 0.82, + "actionability": 0.95, + "gap_hit": false + }, + "y": { + "recall": 0.5, + "precision": 0.58, + "actionability": 0.88, + "gap_hit": false + }, + "winner": "X", + "margin": "clear", + "rationale": "X more directly covers the naming-mapping concern and provides concrete, grounded fixes and tests, though several findings extend beyond the PR’s scope. Y identifies useful test and metadata gaps, but its central to_out claim is questionable and several findings are speculative; neither clearly requests posted before/after validation.", + "_usage": { + "input": 3, + "output": 929 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_val" + }, + "_roles": { + "arm": "copilot_v13_proof_val", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val", + "arm_a_sha256": "1834eeeeb4d689654568d9d5f648bfe1d1a45e7dfc8d932dfd784d5ee5269791", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "9201be5ea408a41833fb8e6040e5eb97d26044159a083e4d1896451ac3585494", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 1 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_val/pr4825.r2.json b/eval/dataset/judgments/goal_v13_val/pr4825.r2.json new file mode 100644 index 000000000..90591d5b0 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/pr4825.r2.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.72, + "precision": 0.7, + "actionability": 0.94, + "gap_hit": false + }, + "y": { + "recall": 0.38, + "precision": 0.48, + "actionability": 0.83, + "gap_hit": false + }, + "winner": "X", + "margin": "clear", + "rationale": "X substantially covers the naming-mapping concern and gives concrete locations and remedies, though several CI, test-structure, and pre-existing compatibility findings exceed the reviewed diff’s scope. Y offers useful test and hardcoded-component observations, but misses more ground-truth intent and relies on a questionable to_out mismatch plus speculative peripheral findings.", + "_usage": { + "input": 3, + "output": 990 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_val" + }, + "_roles": { + "arm": "copilot_v13_proof_val", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val", + "arm_a_sha256": "1834eeeeb4d689654568d9d5f648bfe1d1a45e7dfc8d932dfd784d5ee5269791", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "9201be5ea408a41833fb8e6040e5eb97d26044159a083e4d1896451ac3585494", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 2 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_val/pr4825.r3.json b/eval/dataset/judgments/goal_v13_val/pr4825.r3.json new file mode 100644 index 000000000..38f83280b --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/pr4825.r3.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 0.9, + "precision": 0.55, + "actionability": 0.9, + "gap_hit": false + }, + "y": { + "recall": 0.45, + "precision": 0.35, + "actionability": 0.82, + "gap_hit": false + }, + "winner": "X", + "margin": "clear", + "rationale": "X substantially covers the naming/mapping concern and validation expectations with concrete locations and remedies, though several findings are pre-existing or beyond this narrow diff. Y offers actionable references but misses the general packed-module mapping concern and relies on weaker or speculative findings, especially around `to_out`, unrelated CI failures, and custom-pipeline opt-out behavior.", + "_usage": { + "input": 3, + "output": 1030 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_val" + }, + "_roles": { + "arm": "copilot_v13_proof_val", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val", + "arm_a_sha256": "1834eeeeb4d689654568d9d5f648bfe1d1a45e7dfc8d932dfd784d5ee5269791", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "9201be5ea408a41833fb8e6040e5eb97d26044159a083e4d1896451ac3585494", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 3 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_val/pr4837.r1.json b/eval/dataset/judgments/goal_v13_val/pr4837.r1.json new file mode 100644 index 000000000..60f29c3c9 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/pr4837.r1.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 1.0, + "precision": 0.82, + "actionability": 0.96, + "gap_hit": false + }, + "y": { + "recall": 1.0, + "precision": 0.68, + "actionability": 0.87, + "gap_hit": false + }, + "winner": "X", + "margin": "clear", + "rationale": "Both candidates fully cover the ground-truth rationale for unconditional singleton-list normalization. X provides more complete, concrete analysis of the Ming incompatibility, while Y adds duplicative or low-value findings, including an unnecessary blast-radius confirmation and weakly located CI commentary.", + "_usage": { + "input": 3, + "output": 640 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_val" + }, + "_roles": { + "arm": "copilot_v13_proof_val", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val", + "arm_a_sha256": "b41ea5e119b68a1bd8f36307d0ff894c335404476ef7c9825c3c8095a39c8379", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "bc98460632105aef812a50d392afd9f1bea66b3f783e8a9cd888456ce013cba3", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 1 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_val/pr4837.r2.json b/eval/dataset/judgments/goal_v13_val/pr4837.r2.json new file mode 100644 index 000000000..3d8039bee --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/pr4837.r2.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 1.0, + "precision": 0.58, + "actionability": 0.9, + "gap_hit": false + }, + "y": { + "recall": 1.0, + "precision": 0.72, + "actionability": 0.95, + "gap_hit": false + }, + "winner": "Y", + "margin": "clear", + "rationale": "Both candidates accurately cover the orchestrator rationale in the ground truth. Y more completely diagnoses the valid Ming pipeline incompatibility, especially the missing prompt-embedding route, while X proposes an incomplete fix and adds several weak comments about blast radius, CI, and documentation. Y also includes some out-of-scope follow-ups, but its core findings are better grounded and more actionable.", + "_usage": { + "input": 3, + "output": 1232 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "copilot_v13_proof_val", + "Y": "claudecode_opus5" + }, + "_roles": { + "arm": "copilot_v13_proof_val", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val", + "arm_a_sha256": "b41ea5e119b68a1bd8f36307d0ff894c335404476ef7c9825c3c8095a39c8379", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "bc98460632105aef812a50d392afd9f1bea66b3f783e8a9cd888456ce013cba3", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 2 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_val/pr4837.r3.json b/eval/dataset/judgments/goal_v13_val/pr4837.r3.json new file mode 100644 index 000000000..ae4aa24e7 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/pr4837.r3.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 1.0, + "precision": 0.78, + "actionability": 0.94, + "gap_hit": false + }, + "y": { + "recall": 1.0, + "precision": 0.61, + "actionability": 0.86, + "gap_hit": false + }, + "winner": "X", + "margin": "clear", + "rationale": "Both candidates fully cover the sole ground-truth concern and explain why singleton normalization must apply to both submission paths. X is more technically complete and actionable, especially by tracing the inherited pipeline incompatibility through prompt-embedding routing, while Y proposes an incomplete fix and adds several redundant or low-value findings.", + "_usage": { + "input": 3, + "output": 613 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_val" + }, + "_roles": { + "arm": "copilot_v13_proof_val", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val", + "arm_a_sha256": "b41ea5e119b68a1bd8f36307d0ff894c335404476ef7c9825c3c8095a39c8379", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "bc98460632105aef812a50d392afd9f1bea66b3f783e8a9cd888456ce013cba3", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 3 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_val/pr4893.r1.json b/eval/dataset/judgments/goal_v13_val/pr4893.r1.json new file mode 100644 index 000000000..23eaaf163 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/pr4893.r1.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 1.0, + "precision": 0.58, + "actionability": 0.95, + "gap_hit": false + }, + "y": { + "recall": 1.0, + "precision": 0.42, + "actionability": 0.9, + "gap_hit": false + }, + "winner": "X", + "margin": "clear", + "rationale": "Both candidates catch the sole ground-truth concern: the reduce_scatter assertions validate attributes injected by the fake rather than the real wrapper. X is more precise overall, while Y adds more speculative, pre-existing, stylistic, or follow-up concerns; both provide concrete locations and fixes.", + "_usage": { + "input": 3, + "output": 574 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "copilot_v13_proof_val", + "Y": "claudecode_opus5" + }, + "_roles": { + "arm": "copilot_v13_proof_val", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val", + "arm_a_sha256": "3a910d274a0f99984473aba477ab54e27e47789d293bda139eefb5246199da7f", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "0d4ddd4027e23f1f32216e748289588ea9a0e74ee8e599fa34b9d50f503d2f0d", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 1 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_val/pr4893.r2.json b/eval/dataset/judgments/goal_v13_val/pr4893.r2.json new file mode 100644 index 000000000..acc3fcb18 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/pr4893.r2.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 1.0, + "precision": 0.58, + "actionability": 0.95, + "gap_hit": false + }, + "y": { + "recall": 1.0, + "precision": 0.44, + "actionability": 0.94, + "gap_hit": false + }, + "winner": "X", + "margin": "clear", + "rationale": "Both candidates directly identify the ground-truth gap: the mocked coordinator supplies reduce_scatter itself, so the real wrapper behavior is not verified. X has fewer speculative or irrelevant findings, while Y adds several pre-existing, non-defect, or questionable concerns despite strong technical detail. Both reviews provide precise locations and concrete test or code changes.", + "_usage": { + "input": 3, + "output": 1633 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "copilot_v13_proof_val", + "Y": "claudecode_opus5" + }, + "_roles": { + "arm": "copilot_v13_proof_val", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val", + "arm_a_sha256": "3a910d274a0f99984473aba477ab54e27e47789d293bda139eefb5246199da7f", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "0d4ddd4027e23f1f32216e748289588ea9a0e74ee8e599fa34b9d50f503d2f0d", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 2 + } +} \ No newline at end of file diff --git a/eval/dataset/judgments/goal_v13_val/pr4893.r3.json b/eval/dataset/judgments/goal_v13_val/pr4893.r3.json new file mode 100644 index 000000000..816eb6b46 --- /dev/null +++ b/eval/dataset/judgments/goal_v13_val/pr4893.r3.json @@ -0,0 +1,40 @@ +{ + "x": { + "recall": 1.0, + "precision": 0.58, + "actionability": 0.94, + "gap_hit": false + }, + "y": { + "recall": 1.0, + "precision": 0.64, + "actionability": 0.96, + "gap_hit": false + }, + "winner": "Y", + "margin": "slight", + "rationale": "Both identify the sole substantive ground-truth concern: the reduce_scatter assertions validate mock-injected behavior rather than the real wrapper. Y is slightly more precise and equally concrete, while both dilute strong findings with speculative compatibility risks and low-value coverage or style comments.", + "_usage": { + "input": 3, + "output": 976 + }, + "_judge_resolved_model": "GPT-5.6 Sol 272K High", + "_blinding": { + "X": "claudecode_opus5", + "Y": "copilot_v13_proof_val" + }, + "_roles": { + "arm": "copilot_v13_proof_val", + "baseline": "claudecode_opus5" + }, + "_arm_meta": { + "arm_a_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v13_proof_val", + "arm_a_sha256": "3a910d274a0f99984473aba477ab54e27e47789d293bda139eefb5246199da7f", + "arm_b_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/baselines/claudecode_opus5", + "arm_b_sha256": "0d4ddd4027e23f1f32216e748289588ea9a0e74ee8e599fa34b9d50f503d2f0d", + "judge_backend": "cursor", + "judge_model": "gpt-5.6-sol-high", + "judge_resolved_model": "GPT-5.6 Sol 272K High", + "judge_rep": 3 + } +} \ No newline at end of file diff --git a/eval/dataset/run_copilot_arm.py b/eval/dataset/run_copilot_arm.py index a888418e4..e4ba815d8 100644 --- a/eval/dataset/run_copilot_arm.py +++ b/eval/dataset/run_copilot_arm.py @@ -231,7 +231,12 @@ def main() -> None: "pr_context_mode": _os.environ.get("PR_CONTEXT_MODE", "(default)"), }, indent=2)) print(f"[copilot-arm] {len(items)} items -> {OUT}", flush=True) - with ThreadPoolExecutor(max_workers=2) as ex: + # ARM_JOBS: item-level concurrency. Historical default 2 predates knowing + # the endpoint's real limits (deepseek-v4-pro allows 500 concurrent + # requests; one item peaks at ~15 in-flight calls) — a full split can run + # wide. Item starts are already staggered; RUN_ROOT is private per item. + with ThreadPoolExecutor( + max_workers=int(_os.environ.get("ARM_JOBS", "2"))) as ex: futs = {ex.submit(one, *t): t for t in items} for f in as_completed(futs): try: diff --git a/knowledge/repos/vllm-omni/review/_index.md b/knowledge/repos/vllm-omni/review/_index.md index 69dec1fe8..60214d8e3 100644 --- a/knowledge/repos/vllm-omni/review/_index.md +++ b/knowledge/repos/vllm-omni/review/_index.md @@ -17,4 +17,5 @@ sources: [] | PR 描述如何路由精确 owner/model 代码地图 | [maintainer pattern routing](guides/maintainer-pattern-routing.md) | | 模型适配是否漏掉必要链路 | [model adaptation guardrails](guides/model-adaptation-guardrails.md) | | 模型验证是否证明语义正确 | [model validation](guides/model-validation.md) | +| Strict 审查的仓库专属检查单（按触发条件执行） | [strict review checklist](guides/strict-review-checklist.md) | | 维护或浏览本目录 | [guides index](guides/_index.md) | diff --git a/knowledge/repos/vllm-omni/review/guides/_index.md b/knowledge/repos/vllm-omni/review/guides/_index.md index b3e1ee52e..30d78f721 100644 --- a/knowledge/repos/vllm-omni/review/guides/_index.md +++ b/knowledge/repos/vllm-omni/review/guides/_index.md @@ -13,4 +13,5 @@ sources: [] |---|---| | 按 PR 描述先路由到精确 owner/model 代码地图，再用 changed files 校验范围 | [PR intent maintainer routing](maintainer-pattern-routing.md) | | 审查模型适配 PR 的完整性 | [model adaptation guardrails](model-adaptation-guardrails.md) | +| Strict 审查注入的触发式检查单（streaming 生命周期、平台默认值影响面、依赖下界、CI marker 选择、测试有效性） | [strict review checklist](strict-review-checklist.md) | | 判断新模型验证是否证明语义正确 | [model validation](model-validation.md) | diff --git a/knowledge/repos/vllm-omni/review/guides/strict-review-checklist.md b/knowledge/repos/vllm-omni/review/guides/strict-review-checklist.md new file mode 100644 index 000000000..a685c7446 --- /dev/null +++ b/knowledge/repos/vllm-omni/review/guides/strict-review-checklist.md @@ -0,0 +1,86 @@ +--- +title: "Strict 审查触发式检查单" +created: 2026-08-12 +updated: 2026-08-12 +type: guide +tags: [vllm-omni, review] +sources: [] +--- + +# Strict 审查触发式检查单 + +Distilled from maintainer review patterns on merged PRs (train split of the +20-PR review campaign). Each line is a check to RUN when its trigger matches, +not background reading. The Strict reviewer injects the first 4,000 chars of +this page as its repo-specific checklist (`knowledge.review_checklist` in the +adapter manifest); keep the actionable core inside that budget. + +## Streaming / TTS serving (serving_speech, stream sessions) + +- `async_chunk` defaults to **False** (`OmniModelConfig`). Any gate or + coercion keyed on it applies to EVERY async_chunk=false model (moss_tts, + voxcpm2, higgs_audio, indextts2, qwen2_5_omni, ...), not just the model + named in the PR title — demand scoping or cross-model evidence. +- Stream slots/sessions: acquisition must have a release on client + **abort/disconnect**, not only on `finished=True`. Check the unscheduled/ + abort reap path releases too; leaked slots starve the pool silently. +- Cumulative vs delta chunk semantics: flipping re-emits the accumulated + waveform (audible rewind). Removing a guard/warning around this without a + replacement is itself a finding. +- Per-request seeding (`tts_local_seed` etc.): verify determinism survives + CUDA-graph replay — replayed graphs discard per-request state; "seeded" in + eager mode proves nothing about graph mode. + +## Model executor / checkpoints + +- Legacy vs v2 codec/tokenizer selection: legacy checkpoints can share + `model_type` with v2 — selection must be gated by an explicit config + marker, never try-instantiate-and-fallback (misload = wrong decode layout). +- Never mutate shared config objects (`hf_config`/`model_config` fields) at + load/init: later readers (weight loading, cache sizing) see the mutated + value. Copy, don't overwrite. +- Vendored model files carry a "Vendored from @ " header + (see `audio_tokenizer.py`); new vendored files without one are a finding. + +## Platforms / kernels / global defaults + +- Anything under `vllm_omni/platforms/` or in a `get_default_*` path is + inherited by EVERY model on that platform. Validated-on-one-model changes + need per-model scoping (registry override, e.g. `ir_op_priority_func`) or + A/B evidence on the other affected families. +- Kernel-provider/priority switches: require before/after latency AND + same-seed accuracy vs main, measured PER bundled change — a combined + number cannot attribute the win. +- Hardware gates must match the claimed support matrix: `major >= 9` admits + future arches the claim never covered; and check what CI actually runs — + an L4 lane (sm_89) never executes an SM90-gated path, so its green is not + coverage. + +## Dependencies + +- New kwarg/API into an optional third-party package (`kernels`, HF hub): + read the declared range in `pyproject.toml` — does the OLDEST allowed + version support it? A docs pin (`pkg==X`) is not a lower bound. +- When the commit timeline shows a described kwarg/API was later DROPPED, + state the compatibility reason explicitly (old versions inside the + declared range reject unknown kwargs, so every call would fail) — the + description-vs-diff mismatch is the fix working, not an omission. + +## CI / tests + +- Tests are selected by pytest **markers** (`-m "core_model and cpu"` style + expressions in `.buildkite/`), not by path: a new test file without + `pytestmark` is silently deselected — CI green while never running it. +- A test that cannot fail is worse than none: try/except-and-continue around + the assert, degenerate inputs (`k = q.clone()`), or asserting the value + the fake injected. Flag these as test-integrity findings. +- L2 covers CPU/mock plumbing only; real weights, precision, and perf are + L4. `0 missing / 0 unexpected` + shape smoke ≠ semantic correctness. + +## Process norms maintainers enforce + +- A PR fixing one of N problems reported in a linked issue: one comment must + note what remains unfixed and ask for tracking or a split PR. +- An unexplained capacity/threshold/config retune riding along in a PR gets + a benchmark-rationale ask — that is a review finding, not a process nit. +- Red CI on the PR head: name the failing check and ask for resolution. diff --git a/knowledge/tools/check_wiki_lint.py b/knowledge/tools/check_wiki_lint.py index d2741f2b7..3d75bc841 100644 --- a/knowledge/tools/check_wiki_lint.py +++ b/knowledge/tools/check_wiki_lint.py @@ -30,6 +30,9 @@ "briefing_docs", "briefing_docs_extra", "performance_briefing_docs", + # single knowledge page injected as the Strict reviewer's repo-specific + # checklist (first 4k chars); resolved relative to the knowledge root + "review_checklist", } DATE = re.compile(r"^\d{4}-\d{2}-\d{2}$") MARKDOWN_LINK = re.compile(r"!?\[[^\]]*\]\(([^)]+)\)") @@ -96,6 +99,19 @@ def check_adapter_briefings() -> None: if set(Path(doc).parts) & RAW_PARTS: errors.append( f"adapter {field} 禁止加载原始证据层页面（{doc}）：{manifest}") + checklist = knowledge.get("review_checklist") + if checklist is not None: + if not isinstance(checklist, str): + errors.append( + f"adapter review_checklist 必须是字符串：{manifest}") + elif set(Path(checklist).parts) & RAW_PARTS: + errors.append( + f"adapter review_checklist 禁止加载原始证据层页面" + f"（{checklist}）：{manifest}") + elif not (ROOT / checklist).is_file(): + errors.append( + f"adapter review_checklist 指向不存在的页面" + f"（{checklist}）：{manifest}") def frontmatter(path: Path) -> dict | None: diff --git a/src/infermatrix_copilot/config.py b/src/infermatrix_copilot/config.py index df2cc2425..6f8d1c10b 100644 --- a/src/infermatrix_copilot/config.py +++ b/src/infermatrix_copilot/config.py @@ -209,8 +209,14 @@ class Settings(BaseSettings): # results — uncached tokens x n_lenses; # evidence lives ONCE in the shared # cached prefix instead - evidence_caps: dict[str, int] = {"pr_diff": 120_000, "issue_text": 30_000, + evidence_caps: dict[str, int] = {"pr_diff": 260_000, "issue_text": 30_000, "pr_context": 15_000} # per-item cap; full text archived to run dir + # pr_diff 120k->260k: a 170k-char diff + # (pr4804) lost ~30% of its hunks to the + # cap and review recall collapsed to 0.16 + # — the cap must clear real large PRs; + # the diff sits in the shared cached + # prefix so the cost is one lens's worth # PR context bundle (W1): description/discussion/linked issues fed to the # reviewer. "no_discussion" excludes comments/review threads — REQUIRED for # eval arms (the frozen dataset's ground truth IS the review discussion; @@ -229,14 +235,39 @@ class Settings(BaseSettings): ensemble_stagger_seconds: float = 8.0 # head start for lens 0 so the # shared prompt prefix is cached # before sibling lenses send it - ensemble_lens_max_iters: int = 10 # rounds are ~3x cheaper with windowed + ensemble_lens_max_iters: int = 14 # rounds are ~3x cheaper with windowed # reads; 6 starved lenses into paging - # death (and 38/40 truncated at T0) # per-lens tool budget — replicate means - # dropped when this was cut to 4 (recall - # starvation); 6 is the measured setting - ensemble_merge_evidence_chars: int = 60_000 # must fit the pr_diff — a + # death (and 38/40 truncated at T0). + # 10→14 with the official v4-pro: the + # judged deficit vs the agentic + # baseline is per-finding reading + # depth, and the baseline reads + # 3-10x more code per finding + ensemble_merge_evidence_chars: int = 280_000 # must fit the pr_diff — a # reducer that can't see the diff - # can't verify (T3 forensics #5) + # can't verify (T3 forensics #5); + # raised with evidence_caps.pr_diff + # (60k saw ~35% of a 170k diff) + + # Per-comment agentic verification (val-gate lesson: with recall at + # parity the arm lost on per-comment grounding — precision .54-.56 vs + # .65). Every merged draft comment gets one small tool-loop that must + # anchor and re-derive its claim on the PR-time tree before the budget. + review_verify_comments: bool = True + review_verify_max_iters: int = 4 # tiny loop: read anchor, one grep, + # one consumer read, conclude + review_verify_concurrency: int = 6 # verify calls share the pr_diff + # cache prefix; modest fan-out + + # Deep-investigation engine (official-model rethink): replace the four + # narrow lenses with one long free investigation pass (+ an independent + # adversary pass at full depth). The lens templates were variance + # scaffolding for a weak generator; a strong generator produces its best + # work in the agentic-investigation shape the winning baseline uses. + review_deep_engine: bool = True + review_deep_max_iters: int = 32 # investigation budget per pass — + # the baseline reads 3-10x more code + # per finding than a 14-iter lens # Adaptive review depth (hybrid planner, review/planner.py): deterministic # rules decide the clear cases in pure code; only the gray middle zone diff --git a/src/infermatrix_copilot/engine/agent_runtime/ensemble.py b/src/infermatrix_copilot/engine/agent_runtime/ensemble.py index b27d10937..1c945ac17 100644 --- a/src/infermatrix_copilot/engine/agent_runtime/ensemble.py +++ b/src/infermatrix_copilot/engine/agent_runtime/ensemble.py @@ -187,6 +187,26 @@ async def _one_lens(lens: dict, j: int, idx: int = 0, by_sig[sig] = tagged candidates.append(tagged) + # near-duplicate corroboration across lenses: the same concern surfaces at + # slightly different lines/wording, which exact-signature dedup cannot see + # — consensus stays 1 and the reducer treats independent agreement as + # noise (a 2-lens candidate was dropped as "unlikely" in a live run whose + # ground truth confirmed it). Tag file/line-proximate agreement so the + # reducer's drop rule can weigh it. + for i, a in enumerate(candidates): + fa, la = str(a.get("file", "")), a.get("line") + if not fa or not isinstance(la, int): + continue + others = {name + for j, b in enumerate(candidates) if j != i + for name in b.get("lenses") or [] + if str(b.get("file", "")) == fa + and isinstance(b.get("line"), int) + and abs(b["line"] - la) <= 8 + and name not in (a.get("lenses") or [])} + if others: + a["corroborated_by"] = sorted(others) + def _dedup_union(key: str) -> list: """Union the list-valued `key` across all samples, dropping exact duplicates (dict/list items compared by canonical JSON, scalars by str), @@ -221,6 +241,8 @@ def _dedup_union(key: str) -> list: # reducer's latency. A single-lens singleton does NOT qualify: an # unreplicated claim must still face verification (a hallucinated # blocker once sailed through here and became the entire review) + # `corroborated_by` survives into the merged items: downstream comment + # budgeting ranks independently-corroborated findings above singletons merged[merge_key] = [ {k: v for k, v in c.items() if k not in ("lenses", "consensus")} for c in candidates] diff --git a/src/infermatrix_copilot/engine/steps/pr/fetch.py b/src/infermatrix_copilot/engine/steps/pr/fetch.py index 282bb734c..1014a82e1 100644 --- a/src/infermatrix_copilot/engine/steps/pr/fetch.py +++ b/src/infermatrix_copilot/engine/steps/pr/fetch.py @@ -203,7 +203,8 @@ def _pr_context_bundle(ctx: StepContext, repo: str, pr: int) -> str: mode = str(getattr(ctx.settings, "pr_context_mode", "full") or "full") parts: list[str] = [] code, out = _gh(["pr", "view", str(pr), "--json", - "title,body,labels,headRefName,comments,reviews"], cwd=repo) + "title,body,labels,headRefName,comments,reviews,commits"], + cwd=repo) data: dict = {} if code == 0: try: @@ -217,6 +218,18 @@ def _pr_context_bundle(ctx: StepContext, repo: str, pr: int) -> str: parts.append(f"## PR description\n### {data.get('title', '')}" + (f" [labels: {labels}]" if labels else "") + f"\n{_clip(data.get('body'), 4000)}") + # commit timeline (both modes — the author's own commits, not review + # discussion): a squashed head diff hides add-then-revert churn, and a + # reviewer who cannot see that a described change was later removed + # misreads the description-vs-diff mismatch (it invited re-adding a + # reverted regression in a live run) + subjects = [ + f"- {str(c.get('oid') or '')[:8]} " + f"{_clip((c.get('messageHeadline') or ''), 100)}" + for c in (data.get("commits") or [])[-20:]] + if subjects: + parts.append("## Commit timeline (subjects only — the diff below " + "is the squashed net change)\n" + "\n".join(subjects)) if mode == "full" and data: comments = [f"@{c.get('author', {}).get('login', '?')}: {_clip(c.get('body'))}" for c in _last_n(data.get("comments") or [])] diff --git a/src/infermatrix_copilot/engine/steps/review/prompts.py b/src/infermatrix_copilot/engine/steps/review/prompts.py index 68de304a3..535c2ca16 100644 --- a/src/infermatrix_copilot/engine/steps/review/prompts.py +++ b/src/infermatrix_copilot/engine/steps/review/prompts.py @@ -18,7 +18,11 @@ linked issues: treat the description + linked issues as the acceptance criteria the \ diff must satisfy (scope dropped from the description is a finding). DO NOT repeat \ concerns maintainers already raised in the discussion — build on or extend them, or \ -stay silent on that point. For EVERY public symbol whose signature/default the diff \ +stay silent on that point. Verifying that a hunk does what it says is the FLOOR, not \ +the review: a hunk that narrows, gates, or fixes an earlier problem usually leaves \ +residual risk — interrogate the residual (other batch sizes, other platforms, other \ +launch paths, other models sharing the code) instead of recording the fix as \ +validated and moving on. For EVERY public symbol whose signature/default the diff \ changes, grep the repo for its callers and name each one left stale. When the diff \ ADDS a test, verify the test actually exercises the behavior it names (drive the \ unmapped input through the real entry point) — a test asserting a pre-transformed \ @@ -53,12 +57,38 @@ under the NEW behavior. 7. Undocumented assumptions or invariants the change introduces or relies on (ordering, \ "first element is X", implicit units/thresholds) — these deserve a comment or an assert. -8. Scope: files touched beyond the PR's stated purpose. +8. Scope, both directions: files touched beyond the PR's stated purpose, AND stated \ +purpose the diff does not deliver — if the description or a linked issue reports N \ +related problems and this diff fixes fewer, one comment must name what remains and ask \ +for tracking or a split. +9. Blast radius of changed values: for EVERY changed default, flag, priority list, or \ +dispatch table, enumerate WHO INHERITS the new value (which models/configs/platforms \ +reach this code path) — if the PR validates only a subset, ask for scoping to that \ +subset or evidence on the rest. A one-line value flip in shared code is a repo-wide \ +behavior change until proven otherwise. +10. Dependency compatibility: for EVERY new/changed call into an external library \ +(new kwarg, new API), check the declared version range (pyproject/requirements) — does \ +the OLDEST allowed version support it? A missing lower bound that admits versions \ +which reject the call is a breaking finding. +11. Failure paths that fabricate: on except/fallback branches, is the produced value \ +semantically valid or merely type-valid (e.g. raw bytes standing in for decoded audio, \ +zeros standing in for real stats)? Silent plausible-but-wrong output is worse than an \ +exception. +12. Resource lifecycle symmetry: for every acquire/register/allocate the diff adds or \ +moves, name the release path and WHO calls it on the abort/disconnect/early-exit routes \ +— not just the happy path. + +If the `gate_report` evidence lists failing CI checks AND the diff plausibly touches \ +what a failing lane tests, add ONE minor comment naming the check and asking whether \ +the failure is pre-existing on main or introduced here. Never speculate a red check \ +into a blocker — you cannot see whether main is also red, so attribution claims are \ +guesses and read as fabrication. Severity semantics (they drive the verdict, so assign them honestly): - blocker: merging as-is causes breakage or data loss. -- major: a real defect, or a consumer/doc/test update this change requires but the diff \ -does not contain. +- major: a real defect IN THE CHANGED CODE, or a consumer/doc/test update this change \ +requires but the diff does not contain. "Consider adding X" improvements are never \ +major — a maintainer would not block on them. - minor: a concrete improvement that belongs in THIS PR (a simplification, a stale \ docstring fix, a missing assert, a missing verification run). - nit: optional polish; does NOT block approval. @@ -73,8 +103,10 @@ comment its inline position — worse than giving no snippet at all. One or two lines is \ ideal; enough to be unique within the file, no more. - EVIDENCE-GROUNDING: every comment must be verifiable from the diff or from repo \ -evidence you actually gathered and NAME in the comment (the file you read or grep you \ -ran, and what it showed). The comment's FIRST sentence must state the concrete change \ +evidence you actually gathered — and the `evidence` field must be SELF-CONTAINED \ +PROOF: QUOTE the decisive line(s) VERBATIM with their file:line, so a reader holding \ +only your review and the diff can check the claim without the repo. "I read X and it \ +shows Y" is narrative, not proof, and scores as speculation. The comment's FIRST sentence must state the concrete change \ THIS DIFF makes (quote or paraphrase the hunk) — only then the repo-side consequence \ and the directive; a reader holding only the diff must see immediately which change the \ comment hangs on. For comments about diff code, `line` is a line the diff touches; for \ @@ -123,53 +155,135 @@ one to leave silent. 4. Stop when every path is resolved. Do not spend remaining budget on searches that only raise your confidence in a conclusion you already support. -5. Deliver exactly one consolidated review, with every finding anchored to a +5. Your job is to CHALLENGE the change, not to certify it: for each changed + value or gated behavior, ask who else inherits it (checklist item 9) and + what the failure paths fabricate (item 11) BEFORE concluding no-issue. A + pass that only validates the author's claims has not reviewed the PR. +6. Deliver exactly one consolidated review, with every finding anchored to a real file and line you actually read. No finding without evidence you can point at; if you found nothing, say so plainly and name any path you could - not verify. + not verify — and even then, scope/follow-up observations (item 8) still + belong in review_comments, not in your private notes. """ _REVIEW_LENSES = [ {"name": "logic", - "focus": "Checklist items 1, 2 and 4, as a MECHANICAL SWEEP OF THE DIFF: " - "for EVERY hunk, in order, ask (a) can the new branches/" + "focus": "Checklist items 1, 2, 4 and 11, as a MECHANICAL SWEEP OF THE " + "DIFF: for EVERY hunk, in order, ask (a) can the new branches/" "conditions actually all occur — a branch for a case that " "cannot co-occur is a finding whose fix is REMOVE/simplify, " "never document; (b) does the new code re-derive by hand a " "value an existing helper provides (grep the repo for the " "computation before flagging); (c) None/empty handling, " - "off-by-one, error paths; (d) rebase/merge damage (duplicated " - "code, moved/renamed symbols). Work hunk by hunk; do not skip " - "any. Use repo tools only to CONFIRM a suspicion from the " - "diff."}, + "off-by-one, error paths — and for every arithmetic hunk, does " + "the math match its own comment (floor where the comment says " + "ceil is a finding, and check downstream lengths/offsets are " + "updated consistently); (d) rebase/merge damage (duplicated " + "code, moved/renamed symbols); (e) except/fallback branches " + "that fabricate — is the fallback value semantically valid or " + "merely type-valid? Work hunk by hunk; do not skip any. Use " + "repo tools only to CONFIRM a suspicion from the diff."}, {"name": "behavior", - "focus": "Checklist item 3, diff-first: for EVERY hunk that changes a " - "default, API, protocol or output format, list who depends on " - "the OLD behavior — grep the repo for in-repo consumers " - "(examples/, docs/, clients, tests, READMEs) — then check " - "whether THIS diff updates each one; name every consumer left " - "stale, quoting it. If the diff changes no default/API, say so " + "focus": "Checklist items 3 and 9, diff-first: for EVERY hunk that " + "changes a default, API, protocol or output format, list who " + "depends on the OLD behavior — grep the repo for in-repo " + "consumers (examples/, docs/, clients, tests, READMEs) — then " + "check whether THIS diff updates each one; name every consumer " + "left stale, quoting it. Then the blast radius: enumerate which " + "models/configs/platforms INHERIT each changed value (who else " + "reaches this code path) and compare against what the PR " + "actually validated — validated-on-one-model changes to shared " + "code get a scoping ask. Also ask what user-visible guarantee " + "(determinism, seeding, precision, streaming latency) is " + "silently weakened. If the diff changes no default/API, say so " "and report nothing for this item."}, {"name": "contracts", - "focus": "Checklist items 6 and 7, as a MECHANICAL SWEEP OF THE TOUCHED " - "FILES: (a) enumerate EVERY docstring, inline comment, and " - "field description in each touched file that the change makes " - "stale or misleading — verify each still tells the truth under " - "the NEW behavior, quoting any that don't (stopping after the " - "first is the most common failure); (b) for EVERY indexed or " - "first-element access the diff adds (xs[0], 'first element is " - "X', ordering, implicit units/thresholds), state the " - "assumption it encodes and what guarantees it — if nothing " - "does, ask for an assert or comment."}, + "focus": "Checklist items 6, 7 and 12, as a MECHANICAL SWEEP OF THE " + "TOUCHED FILES: (a) enumerate EVERY docstring, inline comment, " + "and field description in each touched file that the change " + "makes stale or misleading — verify each still tells the truth " + "under the NEW behavior, quoting any that don't (stopping " + "after the first is the most common failure); check every new " + "config/class docstring's stated defaults against the actual " + "signature defaults; (b) for EVERY indexed or first-element " + "access the diff adds (xs[0], 'first element is X', ordering, " + "implicit units/thresholds), state the assumption it encodes " + "and what guarantees it — if nothing does, ask for an assert " + "or comment; (c) for every acquire/register/allocate the diff " + "adds or moves, name the release path and who invokes it on " + "abort/disconnect/early-exit — an unreleased resource on a " + "non-happy path is a finding, not a nit."}, {"name": "verification", - "focus": "Checklist item 5, diff-first: for EVERY behavior-changing " - "hunk, name the specific existing test or benchmark that " - "exercises the changed path (grep tests/, benchmarks/ for the " - "touched symbols). Behavior changed with no test change, a " + "focus": "Checklist items 5 and 10, diff-first: for EVERY behavior-" + "changing hunk, name the specific existing test or benchmark " + "that exercises the changed path (grep tests/, benchmarks/ for " + "the touched symbols). Behavior changed with no test change, a " "changed path no test exercises, new skips, or loosened " - "thresholds are findings. If the PR gives no sign the relevant " - "test/benchmark was run, ask for exactly that run/extension, " - "citing the concrete regression risk it guards."}, + "thresholds are findings. TEST INTEGRITY outranks test " + "existence: a test that cannot fail (try/except-and-continue, " + "degenerate inputs, asserting the value it injected) and a " + "test CI never selects (check the added file's markers against " + "how CI selects tests, and hardware gates against what CI " + "hardware provides) are stronger findings than a missing test. " + "For hardware-capability gates, compare the gate expression " + "against the claimed support matrix (an open-ended >= admits " + "future arches the claim never covered). For perf/capacity/" + "kernel-provider changes, demand before/after evidence that " + "isolates EACH bundled change; for new external-library calls, " + "check the declared dependency range supports them. Do NOT " + "emit generic 'add a unit test' asks: every test finding names " + "the specific behavior at risk and the exact test to run or " + "extend."}, +] + +# Deep-investigation passes (review_deep_engine): the narrow-lens ensemble +# was scaffolding for a weak generator — it compensated for variance by +# forcing enumeration inside four templates, and with a strong generator the +# same templates produce template-shaped, under-grounded output (the measured +# residual loss driver after the recall fixes). A strong model wins in the +# baseline's own shape: long free agentic investigation — so these passes +# hand it that shape, plus everything the baseline does not have (PR-time +# tree, repo checklist, consumer sweep, commit timeline, hunk locations). +_REVIEW_DEEP_PASSES = [ + {"name": "investigator", + "focus": "You are the PRIMARY reviewer. Work like the strongest human " + "maintainer: start from the PR's CENTRAL change (what the " + "title/description is about), read the changed code IN THE " + "TREE — not just the diff — then its consumers (the " + "changed_symbol_consumers evidence lists them), its tests, " + "and the siblings that share the code path, until you can " + "state with evidence what this PR breaks, weakens, or leaves " + "unfinished. The checklist is your PRIORITY LIST, not a form: " + "spend budget where the risk is, cover the central change " + "before anything peripheral. Every comment must be a claim " + "you VERIFIED by reading code — if you did not read it, do " + "not assert it. Depth over breadth: one verified major " + "outweighs five plausible minors. BUDGET DISCIPLINE: your " + "tool budget is fixed — track it, stop investigating while " + "you can still WRITE, and reserve your last two rounds for " + "emitting the full output contract; an investigation that " + "never files its review scores zero. Your comment allowance " + "is 10 (it overrides the general cap — the merge stage " + "unions only you and at most one peer)."}, + {"name": "adversary", + "focus": "You are the SECOND reviewer and you assume the first missed " + "something important. Hunt specifically where reviews " + "systematically fail: blast radius of changed defaults/" + "shared values (enumerate who else inherits the code path), " + "resource lifecycle on abort/disconnect routes, test " + "integrity (a test that cannot fail, or that CI never " + "selects — check markers against the CI lane rules), " + "dependency version floors for new external calls, silently " + "weakened user-visible guarantees (determinism, seeding, " + "precision, streaming latency), and scope the description " + "promises but the diff does not deliver. Read the code that " + "decides each question before asserting. Emit only verified " + "claims — your job is the true misses, not volume. BUDGET " + "DISCIPLINE: your tool budget is fixed — pick the 3-4 most " + "dangerous questions FIRST, close each one, and reserve your " + "last two rounds for emitting the full output contract; an " + "investigation that never files its review scores zero. Your " + "comment allowance is 10 (it overrides the general cap)."}, ] _REVIEW_MERGE = ( @@ -178,11 +292,36 @@ "THIS PR; nit = optional polish. Severities above nit request changes — " "demote to nit anything genuinely optional; a VERIFIED but optional " "comment is demoted, not dropped. Drop comments whose evidence is " - "UNVERIFIED unless a second lens corroborates them. A candidate whose " + "UNVERIFIED unless a second lens corroborates them. Dropping a candidate " + "that two or more lenses produced independently (see its consensus/" + "corroboration tag) requires evidence that it is IMPOSSIBLE, not merely " + "that current callers happen to avoid it — when in doubt, demote to " + "minor instead of dropping. A candidate whose " "own text or evidence declares uncertainty (\'uncertain\', \'could not " "verify\', \'budget exhausted\') must be demoted to nit or rewritten as " "a question — NEVER kept at blocker/major (an uncertain claim cannot " - "block a merge). When you rewrite a " + "block a merge). CENTRALITY comes first: identify the PR's PRIMARY " + "change — the behavior the title/description is about — and make sure " + "the top comments engage IT or its direct consequences (correctness of " + "the changed logic, who else inherits the changed value, what evidence " + "supports it, what residual risk the fix leaves). A review whose kept " + "comments are all about secondary files' tests and docstrings has " + "missed the PR, whatever their individual merit. NEVER drop a verified " + "instance of these classes (they are what human maintainers actually " + "raise): blast-radius scoping of a changed default/shared value, " + "dependency-version compatibility, benchmark evidence for a perf/" + "capacity change, test-INTEGRITY (a test that cannot fail or is never " + "selected by CI), resource lifecycle on abort paths, and scope-split of " + "partially-fixed linked issues — demote to minor at most. Ranking " + "within a severity: behavioral and architecture/ownership findings " + "outrank test-gap asks; test-integrity outranks doc/duplication nits. " + "DO drop (not demote): 'consider adding/documenting X' polish on " + "secondary files, cross-platform observations about code this diff " + "does not change, and CI-lane attribution guesses — unless corroborated " + "or matching a protected class above. VERIFY the internal logic of " + "every kept claim (a degenerate-input or mathematical assertion that " + "is simply wrong — e.g. what an identity input implies — is an instant " + "drop, whatever lens agreement it has). When you rewrite a " "comment, its " "FIRST sentence must state the concrete change the diff makes (quote or " "paraphrase the hunk) — for repo-impact comments too, where the " diff --git a/src/infermatrix_copilot/engine/steps/review/steps.py b/src/infermatrix_copilot/engine/steps/review/steps.py index c666c220f..31fbb4812 100644 --- a/src/infermatrix_copilot/engine/steps/review/steps.py +++ b/src/infermatrix_copilot/engine/steps/review/steps.py @@ -13,8 +13,12 @@ from __future__ import annotations import asyncio +import json import subprocess from dataclasses import replace +from pathlib import Path + +from ....llm import parse_json_reply from ....review.diff_summary import build_diff_summary from ....review.planner import DEFAULT_STANDARD_LENSES, DEPTHS, plan_review @@ -26,6 +30,7 @@ from .._common import step from .anchor import resolve_review_comments from .prompts import ( + _REVIEW_DEEP_PASSES, _REVIEW_LENSES, _REVIEW_LIGHT_PROTOCOL, _REVIEW_MERGE, @@ -94,6 +99,250 @@ def _risk_paths(adapter, settings) -> tuple[str, ...]: return tuple(settings.high_risk_modules) +def _consumer_sweep(repo: str | None, diff: str) -> str: + """Deterministic consumer listing: for each symbol whose definition the + diff touches, every in-repo reference (`git grep -nw`), capped. The + highest-variance recall failure was consumers of changed behavior living + in files (or file regions) no lens ever paged to; listing the exact + locations makes that coverage mechanical instead of stochastic. Runs + against the PR-time worktree, so hits reflect the reviewed tree.""" + if not repo: + return "" + from .utils import _changed_symbols + + out: list[str] = [] + for sym in _changed_symbols(diff)[:8]: + try: + r = subprocess.run(["git", "grep", "-n", "-w", sym], + cwd=repo, capture_output=True, text=True, + timeout=20) + except (OSError, subprocess.SubprocessError): + continue + hits = [h for h in (r.stdout or "").splitlines() + if not h.split(":", 1)[0].endswith((".md", ".rst"))][:10] + if hits: + out.append(f"`{sym}` referenced at:") + out += [f"- {h[:160]}" for h in hits] + if not out: + return "" + return ("IN-REPO REFERENCES OF CHANGED SYMBOLS — for each, check the " + "caller still holds under the NEW behavior (page to it if the " + "file is large); a consumer in a file the diff does not touch " + "is where breakage hides:\n" + "\n".join(out)) + + +async def _promote_uncovered(ctx: StepContext, output: dict, + spec: dict) -> dict: + """Coverage-promotion pass: one tool-less LLM call that promotes + maintainer-relevant concerns ALREADY PRESENT in the run's own findings/ + validated/blockers lines into review comments. + + Train forensics showed the recall loss is often not investigation but + conversion: the ground-truth concern sat verbatim in a lens's findings or + the run's blockers ("the VoxCPM2 regression is a separate mechanism", + "NPU lane is red", "diff evidence was truncated") and never became a + comment. Promotion is grounded by construction — the call may only cite + material from the lines it is shown, never new claims — so it raises + recall without the speculation cost of widening the lens budget.""" + findings = [str(x) for x in (output.get("findings") or [])][:60] + blockers = [str(x) for x in (output.get("blockers") or [])][:10] + assumptions = [str(x) for x in (output.get("assumptions") or [])][:10] + comments = output.get("review_comments") or [] + if not (findings or blockers) or ctx.llm is None \ + or not getattr(ctx.llm, "available", False): + ctx.trace.record("review_coverage_skipped", + step="agent.review_diff", + reason="no findings" if not (findings or blockers) + else "no llm") + return output + system = ( + "You are the coverage editor for a PR review. You receive the " + "review's KEPT COMMENTS plus the reviewers' raw FINDINGS/BLOCKERS/" + "ASSUMPTIONS lines. Identify up to 3 maintainer-relevant concerns " + "that appear in the raw lines but are NOT covered by any kept " + "comment, and promote each into a comment object. Prioritize these " + "classes: blast-radius of a changed default/shared value, benchmark " + "evidence for perf/capacity changes, test integrity (a test that " + "cannot fail or is never selected), dependency-version " + "compatibility, resource lifecycle on abort paths, scope explicitly " + "left unfixed (linked-issue remainder), red CI on the head. Rules: " + "(1) ONLY promote what the raw lines already state — no new claims, " + "no re-investigation; quote the source line in `evidence`. (2) A " + "promoted validation line becomes a question or scoping ask, not an " + "invented defect. (3) severity: minor unless the raw line " + "demonstrates breakage (then major). (4) If everything relevant is " + "already covered, return no additions. Reply with exactly one JSON " + 'object: {"additions": [{"file": str, "line": int, "severity": ' + '"major"|"minor", "comment": str, "evidence": str}]}') + prompt = ( + "## KEPT COMMENTS\n" + + json.dumps([{k: c.get(k) for k in ("file", "line", "severity", + "comment")} + for c in comments], ensure_ascii=False, indent=1) + + "\n\n## RAW FINDINGS\n" + "\n".join(f"- {x}" for x in findings) + + ("\n\n## BLOCKERS\n" + "\n".join(f"- {x}" for x in blockers) + if blockers else "") + + ("\n\n## ASSUMPTIONS\n" + "\n".join(f"- {x}" for x in assumptions) + if assumptions else "")) + try: + _tt = ctx.settings.tier_target(spec.get("mode", "eco")) + llm = (ctx.llm.for_target(_tt) + if hasattr(ctx.llm, "for_target") else ctx.llm) + reply = await asyncio.to_thread( + llm.create, system=system, + messages=[{"role": "user", "content": prompt}], + model=_tt.model, role="reducer", + max_tokens=max(4096, ctx.settings.llm_max_tokens)) + obj = parse_json_reply(reply.text or "") + except Exception as exc: # never fail the review over the extra pass — + ctx.trace.record("review_coverage_skipped", # but say WHY, loudly + step="agent.review_diff", + reason=f"{type(exc).__name__}: {exc}"[:200]) + return output + additions = (obj or {}).get("additions") if isinstance(obj, dict) else None + kept: list[dict] = [] + for a in (additions or [])[:3]: + if not isinstance(a, dict) or not a.get("comment") \ + or not a.get("evidence"): + continue + kept.append({"file": str(a.get("file") or "?"), + "line": a.get("line"), + "severity": str(a.get("severity") or "minor").lower(), + "comment": str(a["comment"]), + "evidence": str(a["evidence"]), + # ordering tag: promoted items carry a protected-class + # concern; under the comment budget they rank ahead of + # uncorroborated same-severity items (stripped at cap) + "corroborated_by": ["coverage"]}) + if kept: + output = dict(output) + output["review_comments"] = list(comments) + kept + ctx.trace.record("review_coverage_promoted", + step="agent.review_diff", added=len(kept)) + else: + ctx.trace.record("review_coverage_skipped", + step="agent.review_diff", + reason="model returned no valid additions", + parsed=obj is not None, + reply_head=str(reply.text or "")[:200]) + return output + + +_VERIFY_GUIDANCE = """You verify ONE draft review comment against the PR-time tree. + +In order, with the minimum tool calls (your budget is small): +1. Anchor: does the cited file:line (or quoted snippet) exist as the comment + claims? Read that region of the file. +2. Claim: is the asserted problem true of the code you just read? If the + comment asserts consumers/callers/tests elsewhere, grep for them and read + the one that decides the claim. +3. Severity: major requires a real defect in the changed code or a required + update the diff lacks; "consider adding X" polish is minor at most. + +Verdicts: +- confirmed: you READ the code that makes the claim true. You MUST return + `evidence` as SELF-CONTAINED PROOF a reader with no repo access can check: + the decisive code line(s) QUOTED VERBATIM with their file:line, e.g. + 'serving_speech.py:3711 `extra_args["tts_local_seed"] = seed` — set for + every model, no qwen3_tts gate'. A narrative like "read the file, claim + holds" is NOT proof and scores as speculation downstream. Optionally + return a tightened `comment` (sharper wording, exact file/line) — keep + the substance, never soften a confirmed defect. +- refuted: the code CONTRADICTS the claim (misread, already handled, wrong + file). Refuted is an evidence conclusion, never a budget one. +- unverifiable: you could not decide within budget. + +Confirming from plausibility alone is the one failure this pass exists to +prevent — when in doubt between confirmed and unverifiable, unverifiable.""" + + +async def _verify_comments(ctx: StepContext, output: dict, + common: dict) -> dict: + """Per-comment agentic verification (val-gate lesson: with recall at + parity the arm lost on per-comment grounding). Each draft comment gets + one small tool-loop that must anchor and re-derive the claim on the + PR-time tree: refuted comments drop, unverifiable ones demote one + severity step, confirmed ones may be tightened in wording/position. A + verification-step failure keeps the comment unchanged — this pass may + only improve precision, never silently delete recall.""" + from ...agent_runtime import run_agent_step + + comments = list(output.get("review_comments") or []) + if not comments or not ctx.settings.review_verify_comments: + return output + sem = asyncio.Semaphore(ctx.settings.review_verify_concurrency) + diff_ev = str((common.get("evidence") or {}).get("pr_diff") or "") + + async def _one(i: int, c: dict): + async with sem: + # pr_diff leads the evidence pack so the big block is byte- + # identical across the fan-out and rides the provider cache + probe = {k: v for k, v in c.items() if k != "corroborated_by"} + result, out = await run_agent_step( + ctx, step_name=f"agent.verify_comment#{i}", + purpose="Verify one draft review comment against the " + "PR-time tree.", + guidance=_VERIFY_GUIDANCE, + expected="verdict confirmed|refuted|unverifiable, with " + "optional comment/line/severity corrections", + evidence={"pr_diff": diff_ev, + "draft_comment": json.dumps( + probe, ensure_ascii=False)}, + output_extension={ + "verdict": "confirmed|refuted|unverifiable", + "comment": "optional tightened rewrite", + "line": "optional corrected line number", + "severity": "optional corrected severity", + "evidence": "confirmed only: verbatim-quoted decisive " + "code line(s) with file:line — proof a " + "repo-less reader can check"}, + extra_tools=common.get("extra_tools"), + max_iters=ctx.settings.review_verify_max_iters) + return i, result, (out or {}) + + results = await asyncio.gather(*(_one(i, c) + for i, c in enumerate(comments))) + demote = {"blocker": "major", "major": "minor", "minor": "nit"} + kept: list[dict] = [] + n_drop = n_demote = 0 + for i, result, out in sorted(results, key=lambda r: r[0]): + c = dict(comments[i]) + verdict = str(out.get("verdict") or "").lower() + if not result.ok or verdict not in ("confirmed", "refuted", + "unverifiable"): + kept.append(c) # fail-open: never delete on pass failure + continue + if verdict == "refuted": + n_drop += 1 + continue + if verdict == "confirmed": + if out.get("comment"): + c["comment"] = str(out["comment"]) + if isinstance(out.get("line"), int): + c["line"] = out["line"] + sev = str(out.get("severity") or "").lower() + if sev in _SEVERITY_ORDER: + c["severity"] = sev + if out.get("evidence"): + # replace the drafting-stage narrative with the verifier's + # quoted proof: the review is judged by readers with no repo + # access, and a claim whose evidence they can check from the + # quote alone is the difference between "grounded" and + # "speculative" (precision sat at .55 across five configs + # until the rendered evidence became self-proving) + c["evidence"] = str(out["evidence"]) + else: # unverifiable: keep, one step down + s = str(c.get("severity", "minor")).lower() + c["severity"] = demote.get(s, "nit") + n_demote += 1 + kept.append(c) + ctx.trace.record("review_comments_verified", step="agent.review_diff", + total=len(comments), dropped=n_drop, demoted=n_demote) + out2 = dict(output) + out2["review_comments"] = kept + return out2 + + @step("agent.review_diff", "agent", "read", "Evidence-grounded two-stage review: tool-loop investigation draft, " "then verify-and-rewrite editor pass.") @@ -120,14 +369,27 @@ async def _review_diff(ctx: StepContext) -> StepResult: if adapter is not None: language = str(adapter.manifest.get("repo", {}).get("language") or "python") - review_md = adapter.profile_dir / "review.md" - try: - if review_md.exists() and ctx.settings.profile_briefing_enabled: - guidance += ("\n\n## Repo-specific review checklist " - "(from the repo profile)\n" - + review_md.read_text(encoding="utf-8")[:4_000]) - except OSError: - pass + # checklist resolution: the adapter's knowledge slice first (shared + # data plane — `knowledge.review_checklist`, relative to the + # knowledge root, escape-guarded), then the legacy profile/review.md + # for adapters that still carry one + candidates: list[Path] = [] + rel = str((adapter.manifest.get("knowledge") or {}) + .get("review_checklist") or "") + if rel: + kroot = Path(ctx.settings.knowledge_dir).resolve() + kpath = (kroot / rel).resolve() + if kpath.is_relative_to(kroot): + candidates.append(kpath) + candidates.append(adapter.profile_dir / "review.md") + for review_md in candidates: + try: + if review_md.exists() and ctx.settings.profile_briefing_enabled: + guidance += ("\n\n## Repo-specific review checklist\n" + + review_md.read_text(encoding="utf-8")[:4_000]) + break + except OSError: + continue common = dict( step_name="agent.review_diff", @@ -139,7 +401,9 @@ async def _review_diff(ctx: StepContext) -> StepResult: evidence={"pr_diff": str(diff), "pr_context": ctx.state.get("pr_context", ""), "gate_report": ctx.state.get("gate_report", ""), - "sweep_targets": _sweep_targets(str(diff), language)}, + "sweep_targets": _sweep_targets(str(diff), language), + "changed_symbol_consumers": await asyncio.to_thread( + _consumer_sweep, _repo_path(ctx), str(diff))}, output_extension={"review_comments": "list of {file, line, anchor_snippet, severity: " "blocker|major|minor|nit, comment, evidence}"}, @@ -193,18 +457,30 @@ async def _review_diff(ctx: StepContext) -> StepResult: reason=plan.reason + "; escalated: light found nothing") ctx.state["_review_depth"] = plan.depth if plan.depth != "light": - lenses = [l for l in _REVIEW_LENSES - if l["name"] in plan.lens_names] or list(_REVIEW_LENSES) - result, output = await run_agent_step_ensemble( - ctx, lenses=lenses, merge_key="review_comments", - merge_guidance=_REVIEW_MERGE, **common) - # deterministic comment budget: severity-ordered, capped at 5 — the - # low-signal tail goes first (reducers ignored a prompted cap; the cap is - # a product budget, so it applies to every depth) - comments = sorted(output.get("review_comments") or [], - key=lambda c: _SEVERITY_ORDER.get( - str(c.get("severity", "minor")).lower(), 2)) - output["review_comments"] = comments[:5] + if ctx.settings.review_deep_engine: + # hybrid mode: the planner still owns DEPTH. Deep passes buy + # per-claim grounding (probe: precision .82 vs baseline .75) + # but their 4-5 focused findings collapsed val recall to .55; + # the breadth lenses held recall at .80-.86 but under-ground. + # Full depth runs both shapes and lets the reducer/verify + # machinery arbitrate: investigator + adversary for depth, + # behavior + verification for coverage. + breadth = [l for l in _REVIEW_LENSES + if l["name"] in ("behavior", "verification")] + passes = (list(_REVIEW_DEEP_PASSES) + breadth + if plan.depth == "full" + else list(_REVIEW_DEEP_PASSES[:1]) + breadth[:1]) + result, output = await run_agent_step_ensemble( + ctx, lenses=passes, merge_key="review_comments", + merge_guidance=_REVIEW_MERGE, + max_iters=ctx.settings.review_deep_max_iters, **common) + else: + lenses = [l for l in _REVIEW_LENSES + if l["name"] in plan.lens_names] \ + or list(_REVIEW_LENSES) + result, output = await run_agent_step_ensemble( + ctx, lenses=lenses, merge_key="review_comments", + merge_guidance=_REVIEW_MERGE, **common) if not result.ok and output.get("review_comments"): # A review that FOUND defects is a successful review whose verdict is # REQUEST CHANGES — not a failed step. Agents conflate the PR's @@ -217,6 +493,46 @@ async def _review_diff(ctx: StepContext) -> StepResult: f"{result.summary}", outputs=result.outputs, changed_files=result.changed_files) + if plan is not None and plan.depth != "light" and result.ok: + # coverage promotion mines the run's own findings/blockers for + # uncovered protected-class concerns; runs AFTER the salvage (an + # ensemble that self-reported needs_review still carries findings + # worth mining — gating on the pre-salvage status silently disabled + # the pass on exactly the GT-rich items it exists for) and BEFORE + # the verify pass, so promoted items face the same scrutiny + output = await _promote_uncovered(ctx, output, spec) + # per-comment agentic verification: every surviving draft comment is + # checked against the PR-time tree by a small tool-loop before the + # budget — the measured loss driver after the recall fixes was + # per-comment grounding (val: arm precision .54-.56 vs baseline + # .65 with recall at parity), which packaging cannot buy + output = await _verify_comments(ctx, output, common) + # deterministic comment budget: severity-ordered, corroboration-aware, + # capped at 6 — the low-signal tail goes first (reducers ignored a + # prompted cap; the cap is a product budget, so it applies to every + # depth). The old 5-comment, severity-only cut deleted reducer-KEPT + # findings that were literal maintainer concerns on large PRs, while + # 8 + nit-heavy overflow read as noise to the blind judge; 6 kept + # comments + the promotion pass's protected-class additions is the + # measured operating point. Within a severity, findings corroborated by + # more independent lenses go first; the minor-plus overflow tail + # renders ONLY on evidence-rich reviews (≥2 major/blocker findings) — + # a quiet PR stays terse (a thin-GT val item rendering 12 findings + # lost 3/3 on noise). + comments = sorted(output.get("review_comments") or [], + key=lambda c: (_SEVERITY_ORDER.get( + str(c.get("severity", "minor")).lower(), 2), + -len(c.get("corroborated_by") or []))) + for c in comments: + c.pop("corroborated_by", None) + output["review_comments"] = comments[:8] + rich = sum(1 for c in comments[:8] + if _SEVERITY_ORDER.get(str(c.get("severity", "minor")).lower(), + 2) <= _SEVERITY_ORDER["major"]) >= 2 + output["_review_overflow"] = [ + c for c in comments[8:] + if _SEVERITY_ORDER.get(str(c.get("severity", "minor")).lower(), 2) + <= _SEVERITY_ORDER["minor"]][:4] if rich else [] if plan is not None: result.outputs["review_plan"] = {"depth": plan.depth, "planner": plan.planner, diff --git a/src/infermatrix_copilot/engine/steps/review/utils.py b/src/infermatrix_copilot/engine/steps/review/utils.py index ac0142e0f..151910edb 100644 --- a/src/infermatrix_copilot/engine/steps/review/utils.py +++ b/src/infermatrix_copilot/engine/steps/review/utils.py @@ -34,6 +34,7 @@ def _sweep_targets(diff: str, language: str = "python") -> str: branches: list[str] = [] files: set[str] = set() test_files: set[str] = set() + regions: dict[str, list[int]] = {} for line in diff.splitlines(): if line.startswith("+++ b/"): current = line[6:] @@ -43,6 +44,8 @@ def _sweep_targets(diff: str, language: str = "python") -> str: elif line.startswith("@@"): m = re.search(r"\+(\d+)", line) new_line = int(m.group(1)) if m else 0 + if current and m: + regions.setdefault(current, []).append(new_line) elif current and line.startswith("+") and not line.startswith("+++"): code = line[1:] stripped = code.strip() @@ -68,11 +71,50 @@ def _sweep_targets(diff: str, language: str = "python") -> str: out.append("NON-TEST FILES TOUCHED — verification lens must name the " "test/benchmark covering each changed path:") out += [f"- {f}" for f in non_test[:20]] + if regions: + # a windowed read starts at the file top; a hunk at line 3500 of a + # large file is otherwise never in view (measured: a ground-truth + # consumer bug sat exactly there). Hunk-start line numbers tell the + # lens where to page to. + out.append("DIFF HUNK LOCATIONS (file: new-file line numbers) — when " + "you read one of these files, PAGE with `offset` until " + "the listed lines are inside your window, and also read " + "the surrounding in-file consumers of what changed; " + "verifying a hunk you have not seen in situ is guesswork:") + out += [f"- {f}: lines {', '.join(str(n) for n in ns[:12])}" + for f, ns in sorted(regions.items())[:20]] out.append("TEST FILES TOUCHED IN THIS DIFF: " + (", ".join(sorted(test_files)) or "NONE")) return "\n".join(out) +def _changed_symbols(diff: str) -> list[str]: + """Symbol names whose definition or value the diff touches — the seeds + for the consumer sweep. Deterministic and diff-only: function/class names + on ± def/class lines, plus enclosing-function names from hunk headers + (`@@ ... def f(...)`), deduped in first-seen order.""" + import re + + names: list[str] = [] + seen: set[str] = set() + + def add(name: str) -> None: + if name and name not in seen and not name.startswith("_" * 2): + seen.add(name) + names.append(name) + + for line in diff.splitlines(): + m = re.match(r"^[+-]\s*(?:async\s+)?def\s+(\w+)|^[+-]\s*class\s+(\w+)", + line) + if m: + add(m.group(1) or m.group(2)) + continue + m = re.match(r"^@@[^@]*@@.*?\bdef\s+(\w+)", line) + if m: + add(m.group(1)) + return names + + _CATEGORY_RULES = ( # deterministic finding→category keyword map (W2: no PASS inference — a # row without findings renders "no finding reported", an honest coverage @@ -134,7 +176,10 @@ def _review_summary_parts(output: dict) -> list[str]: ) ][:8] counts: dict[str, int] = {} - for comment in comments: + # budget-cut overflow counts in the scan too — a capped finding was still + # found, and "no finding reported" over a category the lenses DID flag + # misrepresents the review's coverage + for comment in list(comments) + list(output.get("_review_overflow") or []): category = _category_of(comment) counts[category] = counts.get(category, 0) + 1 scan = ["| Category | Result |", "|---|---|"] @@ -189,5 +234,15 @@ def _render_review_md(output: dict, pr_state: str = "") -> str: f"{c.get('comment', '')}{ev}") parts = _review_summary_parts(output) parts.append("\n\n".join(lines) if lines else output.get("summary", "No findings.")) + overflow = output.get("_review_overflow") or [] + if overflow: + # findings the comment budget cut — one line each, so a real (often + # minor) concern the reducer kept is visible instead of vanishing + parts.append("**Additional observations (beyond the comment budget):**\n" + + "\n".join( + f"- `{c.get('file', '?')}:{c.get('line', '?')}` " + f"[{c.get('severity', 'minor')}] " + f"{str(c.get('comment', '')).strip()[:220]}" + for c in overflow)) body = "\n\n".join(parts) return f"{body}\n\n**Verdict:** {_review_verdict(comments, pr_state)}" diff --git a/src/infermatrix_copilot/review/planner.py b/src/infermatrix_copilot/review/planner.py index 2cce11dae..173f04c8c 100644 --- a/src/infermatrix_copilot/review/planner.py +++ b/src/infermatrix_copilot/review/planner.py @@ -26,7 +26,13 @@ from ..llm import parse_json_reply DEPTHS = ("light", "standard", "full") -DEFAULT_STANDARD_LENSES = ("logic", "behavior") # fail-safe pair, priority order +# fail-safe set, priority order. Verification is included: the planner +# fallback runs when the gray-zone call fails, i.e. with no information — and +# a review with no verification lens misses the test-integrity/CI-selection +# class entirely (observed on a live gray-zone item: standard=[logic, +# behavior] never read the CI lane rules, which held the baseline's headline +# finding). +DEFAULT_STANDARD_LENSES = ("logic", "behavior", "verification") _DOC_SUFFIXES = (".md", ".rst", ".txt", ".adoc") _CONFIG_SUFFIXES = (".yaml", ".yml", ".toml", ".ini", ".cfg", ".json") @@ -48,6 +54,13 @@ # must not disqualify a tiny PR from the light tier. _API_LINE = re.compile( r"^[+-]\s*(?:async\s+def\s+\w+|def\s+\w+\s*\(|class\s+\w+|[A-Z][A-Z0-9_]{2,}\s*=)") +# assignment target on a ± code line: a `-`/`+` pair sharing the LHS with a +# DIFFERENT RHS is a value flip — an existing behavior knob changed (default +# lists, feature flags, thresholds). `_API_LINE` misses these (a default flip +# inside a function body has no def/class/CONST shape), and a tiny diff that +# flips a value can have repo-wide blast radius, so it must never rule light. +_ASSIGN_LINE = re.compile( + r"^[+-]\s*(?P[A-Za-z_][\w.]*(?:\[[^\]=]+\])?)\s*=(?![=])\s*(?P.+?)\s*$") def _unquote(path: str) -> str: @@ -84,6 +97,7 @@ class DiffSignals: high_risk_files: tuple[str, ...] = () api_change_hints: tuple[str, ...] = () # `-` lines: changed/removed surface api_added: int = 0 # `+` def/class/const: new surface + value_flips: tuple[str, ...] = () # -/+ pairs: same LHS, new RHS code_insertions: int = 0 code_deletions: int = 0 @@ -109,7 +123,8 @@ def as_dict(self) -> dict: "asset_files": len(self.asset_files), "high_risk_files": list(self.high_risk_files), "api_change_hints": len(self.api_change_hints), - "api_added": self.api_added} + "api_added": self.api_added, + "value_flips": list(self.value_flips)} @dataclass(frozen=True) @@ -147,6 +162,11 @@ def diff_signals(diff_text: str, code_insertions = code_deletions = 0 api_hints: list[str] = [] api_added = 0 + # (path, lhs) -> rhs of removed assignments; a later `+` line with the + # same LHS but a different RHS is a value flip. Same-RHS pairs are code + # motion and stay silent. + removed_assigns: dict[tuple[str, str], str] = {} + value_flips: list[str] = [] current_path = "" current_kind = "code" @@ -191,6 +211,17 @@ def add(path: str) -> None: api_hints.append(f"{current_path}: `{raw[:120].strip()}`") else: api_added += 1 + if current_kind == "code": + m = _ASSIGN_LINE.match(raw) + if m: + key = (current_path, m.group("lhs")) + if raw[0] == "-": + removed_assigns.setdefault(key, m.group("rhs")) + elif key in removed_assigns \ + and removed_assigns[key] != m.group("rhs"): + if len(value_flips) < 10: + value_flips.append( + f"{current_path}: `{m.group('lhs')}`") by_kind: dict[str, list[str]] = {"doc": [], "test": [], "config": [], "asset": [], "code": []} @@ -205,7 +236,8 @@ def add(path: str) -> None: asset_files=tuple(by_kind["asset"]), code_files=tuple(by_kind["code"]), high_risk_files=risky, api_change_hints=tuple(api_hints), - api_added=api_added, code_insertions=code_insertions, + api_added=api_added, value_flips=tuple(value_flips), + code_insertions=code_insertions, code_deletions=code_deletions) @@ -224,7 +256,7 @@ def classify(sig: DiffSignals, settings: Any) -> tuple[str, str] | None: f"{list(sig.high_risk_files)}") if sig.files and len(sig.files) <= settings.review_light_max_files \ and sig.lines_changed <= settings.review_light_max_lines \ - and not sig.api_change_hints: + and not sig.api_change_hints and not sig.value_flips: return "light", (f"small low-risk diff: {sig.lines_changed} lines / " f"{len(sig.files)} files, no API/default changes") return None diff --git a/test/conftest.py b/test/conftest.py index 0d0894876..c056b603e 100644 --- a/test/conftest.py +++ b/test/conftest.py @@ -26,6 +26,9 @@ def settings(tmp_path: Path) -> Settings: ensemble_samples_per_lens=1, # sampling tests opt in explicitly ensemble_stagger_seconds=0, # tests never sleep for cache warm-up ensemble_zero_yield_retry=False, # scripted fakes opt in explicitly + review_verify_comments=False, # verify-pass tests opt in explicitly + review_deep_engine=False, # deep-pass tests opt in explicitly; the + # legacy scripted fakes assume 4 lenses ) diff --git a/test/test_agent_ensemble.py b/test/test_agent_ensemble.py index 2de1969ef..11ad3fc28 100644 --- a/test/test_agent_ensemble.py +++ b/test/test_agent_ensemble.py @@ -301,21 +301,24 @@ def test_ensemble_samples_per_lens_union(settings, trace, tmp_path): def test_review_step_caps_comments_deterministically(settings, trace, tmp_path, git_repo): - """The comment budget is enforced in code (severity-ordered, cap 5) — the - low-signal nit tail goes first. Reducers ignored a prompted cap.""" + """The comment budget is enforced in code (severity-ordered, cap 8) — the + low-signal nit tail goes first; on a review that is not evidence-rich + (<2 major findings) the overflow tail does not render at all (val gate: + a thin-GT item rendering 12 findings lost 3/3 on noise). Reducers + ignored a prompted cap.""" settings.review_ensemble = True settings.review_depth = "full" # pin: this test exercises ensemble mechanics many = ([{"file": "m.py", "line": i, "severity": "nit", - "comment": f"n{i}", "evidence": "hunk"} for i in range(4)] + "comment": f"n{i}", "evidence": "hunk"} for i in range(6)] + [{"file": "m.py", "line": 9, "severity": "major", "comment": "big", "evidence": "hunk"}] + [{"file": "m.py", "line": 20 + i, "severity": "minor", - "comment": f"m{i}", "evidence": "hunk"} for i in range(3)]) + "comment": f"m{i}", "evidence": "hunk"} for i in range(5)]) from infermatrix_copilot.engine.steps import register_builtin_steps llm = ScriptedLLM( [contract(review_comments=many)] + [contract(review_comments=[])] * (len(_REVIEW_LENSES) - 1) - + [verdicts_reply()]) # reducer silent -> all 8 kept, then capped + + [verdicts_reply()]) # reducer silent -> all 12 kept, then capped state = {"diff_text": "diff --git a/m.py b/m.py\n+A = 1", "task_spec": {"pr": 9}, "repo_path": str(git_repo)} registry = register_builtin_steps(StepRegistry()) @@ -323,12 +326,214 @@ def test_review_step_caps_comments_deterministically(settings, trace, tmp_path, _ctx(settings, trace, tmp_path, state, llm=llm))) assert result.ok, result.summary kept = result.outputs["review_comments"] - assert len(kept) == 5 + assert len(kept) == 8 sevs = [c["severity"] for c in kept] - assert sevs == ["major", "minor", "minor", "minor", "nit"] + assert sevs == ["major", "minor", "minor", "minor", "minor", "minor", + "nit", "nit"] + # 1 major < the rich threshold: the overflow tail does not render + assert "Additional observations" not in state["review_text"] assert state["review_text"].endswith("**Verdict:** REQUEST CHANGES") +def test_review_overflow_renders_reducer_kept_minors(settings, trace, tmp_path, + git_repo): + """On an evidence-rich review (≥2 major findings), minor-and-above + findings past the cap render as one-line observations (the pr4859 class: + reducer-kept ground-truth minors must stay visible); nits past the cap + are dropped silently.""" + settings.review_ensemble = True + settings.review_depth = "full" + many = ([{"file": "m.py", "line": i, "severity": "major", + "comment": f"maj{i}", "evidence": "hunk"} for i in range(8)] + + [{"file": "m.py", "line": 30, "severity": "minor", + "comment": "gt-minor-survives", "evidence": "hunk"}] + + [{"file": "m.py", "line": 40, "severity": "nit", + "comment": "nit-vanishes", "evidence": "hunk"}]) + from infermatrix_copilot.engine.steps import register_builtin_steps + llm = ScriptedLLM( + [contract(review_comments=many)] + + [contract(review_comments=[])] * (len(_REVIEW_LENSES) - 1) + + [verdicts_reply()]) + state = {"diff_text": "diff --git a/m.py b/m.py\n+A = 1", + "task_spec": {"pr": 9}, "repo_path": str(git_repo)} + registry = register_builtin_steps(StepRegistry()) + result = asyncio.run(registry.get("agent.review_diff").handler( + _ctx(settings, trace, tmp_path, state, llm=llm))) + assert result.ok, result.summary + assert len(result.outputs["review_comments"]) == 8 + assert "Additional observations" in state["review_text"] + assert "gt-minor-survives" in state["review_text"] + assert "nit-vanishes" not in state["review_text"] + + +def test_cross_lens_corroboration_ranks_first_at_the_cap(settings, trace, + tmp_path, git_repo): + """Two lenses independently flagging the same file within a few lines is + the strongest signal in the ensemble; under the comment budget those + findings outrank same-severity singletons (which previously survived on + list order while corroborated ground-truth concerns were cut).""" + settings.review_ensemble = True + settings.review_depth = "full" + settings.ensemble_zero_yield_retry = False + corro_a = {"file": "hot.py", "line": 10, "severity": "minor", + "comment": "shared concern", "evidence": "hunk"} + corro_b = {"file": "hot.py", "line": 12, "severity": "minor", + "comment": "shared concern, other wording", "evidence": "hunk"} + singles = [{"file": f"s{i}.py", "line": 1, "severity": "minor", + "comment": f"single {i}", "evidence": "hunk"} + for i in range(8)] + llm = ScriptedLLM([ + contract(review_comments=[corro_a] + singles), # lens 1 + contract(review_comments=[corro_b]), # lens 2 + contract(review_comments=[]), # lens 3 + contract(review_comments=[]), # lens 4 + verdicts_reply(), # reducer silent: keep all + ]) + state = {"diff_text": "diff --git a/hot.py b/hot.py\n+A = 1", + "task_spec": {"pr": 9}, "repo_path": str(git_repo)} + registry = register_builtin_steps(StepRegistry()) + result = asyncio.run(registry.get("agent.review_diff").handler( + _ctx(settings, trace, tmp_path, state, llm=llm))) + assert result.ok, result.summary + kept = result.outputs["review_comments"] + assert len(kept) == 8 + assert {kept[0]["file"], kept[1]["file"]} == {"hot.py"} + assert all("corroborated_by" not in c for c in kept) + + +def test_coverage_promotion_converts_findings_to_comments(settings, trace, + tmp_path, git_repo): + """A maintainer-relevant concern that a lens recorded in `findings` but + never emitted as a comment is promoted (grounded in the recorded line) by + the post-budget coverage pass; a scripted no-addition reply leaves the + review untouched.""" + settings.review_ensemble = True + settings.review_depth = "full" + settings.ensemble_zero_yield_retry = False + base = {"file": "m.py", "line": 1, "severity": "major", + "comment": "real defect", "evidence": "hunk"} + lens1 = contract(review_comments=[base], + findings=["[sweep] linked issue #99 reports two " + "regressions; this PR fixes only one"]) + promotion = Reply(blocks=[Block(type="text", text=json.dumps({ + "additions": [{"file": "m.py", "line": 2, "severity": "minor", + "comment": "issue #99's second regression remains " + "unfixed — track or split it", + "evidence": "[sweep] linked issue #99 reports two " + "regressions; this PR fixes only one"}]}))]) + llm = ScriptedLLM([ + lens1, + contract(review_comments=[]), # lens 2 + contract(review_comments=[]), # lens 3 + contract(review_comments=[]), # lens 4 + verdicts_reply({"i": 0, "action": "keep"}), + promotion, + ]) + state = {"diff_text": "diff --git a/m.py b/m.py\n+A = 1", + "task_spec": {"pr": 9}, "repo_path": str(git_repo)} + registry = register_builtin_steps(StepRegistry()) + result = asyncio.run(registry.get("agent.review_diff").handler( + _ctx(settings, trace, tmp_path, state, llm=llm))) + assert result.ok, result.summary + kept = result.outputs["review_comments"] + assert len(kept) == 2 + assert "remains unfixed" in kept[1]["comment"] + assert next(trace.events("review_coverage_promoted"))["added"] == 1 + assert not llm._replies + + +def test_deep_engine_runs_investigator_plus_adversary(settings, trace, + tmp_path, git_repo): + """review_deep_engine runs the hybrid pass set: full depth = deep + investigator + adversary plus the behavior + verification breadth + lenses (4 dispatches); standard = investigator + behavior. Depth still + comes from the planner.""" + settings.review_ensemble = True + settings.review_deep_engine = True + settings.review_depth = "full" + settings.ensemble_zero_yield_retry = False + c1 = [{"file": "a.py", "line": 1, "severity": "major", + "comment": "central-change defect", "evidence": "read a.py"}] + c2 = [{"file": "b.py", "line": 2, "severity": "minor", + "comment": "blast radius question", "evidence": "read b.py"}] + llm = ScriptedLLM([ + contract(review_comments=c1), # investigator + contract(review_comments=c2), # adversary + contract(review_comments=[]), # behavior + contract(review_comments=[]), # verification + verdicts_reply({"i": 0, "action": "keep"}, + {"i": 1, "action": "keep"}), + ]) + state = {"diff_text": "diff --git a/a.py b/a.py\n+A = 1", + "task_spec": {"pr": 9}, "repo_path": str(git_repo)} + registry = register_builtin_steps(StepRegistry()) + result = asyncio.run(registry.get("agent.review_diff").handler( + _ctx(settings, trace, tmp_path, state, llm=llm))) + assert result.ok, result.summary + steps_run = [e["step"] for e in trace.events("agent_dispatch")] + assert steps_run == ["agent.review_diff#investigator", + "agent.review_diff#adversary", + "agent.review_diff#behavior", + "agent.review_diff#verification"] + ens = next(trace.events("agent_ensemble")) + assert ens["lenses"] == ["investigator", "adversary", + "behavior", "verification"] + assert len(result.outputs["review_comments"]) == 2 + assert not llm._replies + + +def test_verify_pass_drops_refuted_demotes_unverifiable(settings, trace, + tmp_path, git_repo): + """Per-comment verification: refuted comments drop, unverifiable ones + keep but demote one severity step, confirmed ones may be tightened; a + garbage verdict fails open (comment kept unchanged).""" + settings.review_ensemble = True + settings.review_depth = "full" + settings.ensemble_zero_yield_retry = False + settings.review_verify_comments = True + settings.review_verify_concurrency = 1 # ordered ScriptedLLM needs determinism + cs = [ + {"file": "a.py", "line": 1, "severity": "major", + "comment": "refute me", "evidence": "hunk"}, + {"file": "b.py", "line": 2, "severity": "major", + "comment": "confirm me", "evidence": "hunk"}, + {"file": "c.py", "line": 3, "severity": "major", + "comment": "unverifiable claim", "evidence": "hunk"}, + {"file": "d.py", "line": 4, "severity": "minor", + "comment": "garbage verdict for me", "evidence": "hunk"}, + ] + llm = ScriptedLLM([ + contract(review_comments=cs), # lens 1 + contract(review_comments=[]), # lenses 2-4 + contract(review_comments=[]), + contract(review_comments=[]), + verdicts_reply({"i": 0, "action": "keep"}, # reducer keeps all + {"i": 1, "action": "keep"}, + {"i": 2, "action": "keep"}, + {"i": 3, "action": "keep"}), + contract(verdict="refuted"), # verify c0 + contract(verdict="confirmed", review_comments=[], + comment="tightened: verified against b.py"), # verify c1 + contract(verdict="unverifiable"), # verify c2 + contract(verdict="banana"), # verify c3 + ]) + state = {"diff_text": "diff --git a/a.py b/a.py\n+A = 1", + "task_spec": {"pr": 9}, "repo_path": str(git_repo)} + registry = register_builtin_steps(StepRegistry()) + result = asyncio.run(registry.get("agent.review_diff").handler( + _ctx(settings, trace, tmp_path, state, llm=llm))) + assert result.ok, result.summary + kept = result.outputs["review_comments"] + by_file = {c["file"]: c for c in kept} + assert "a.py" not in by_file # refuted → dropped + assert by_file["b.py"]["comment"].startswith("tightened:") + assert by_file["c.py"]["severity"] == "minor" # major demoted + assert by_file["d.py"]["comment"] == "garbage verdict for me" # fail-open + ev = next(trace.events("review_comments_verified")) + assert (ev["total"], ev["dropped"], ev["demoted"]) == (4, 1, 1) + assert not llm._replies + + def test_render_verdict_calibration(): """T4 calibration: only verified blocker/major block; other comments are COMMENT (mergeable with asks); none -> APPROVE. Self-declared-uncertain @@ -425,7 +630,9 @@ def test_review_step_invalid_override_blocks_before_any_llm(settings, trace, def test_review_step_gray_zone_falls_back_to_standard(settings, trace, tmp_path, git_repo): """Mid-size diff + unparseable planner reply → deterministic standard - (logic+behavior): 1 garbage planner call, 2 lens passes, 1 reducer.""" + (logic+behavior+verification): 1 garbage planner call, 3 lens passes, + 1 reducer. Verification rides the fallback so the test-integrity class + is never silently skipped on planner failure.""" settings.review_ensemble = True body = "\n".join(f"+line {i}" for i in range(60)) state = {"diff_text": "\n".join( @@ -436,12 +643,16 @@ def test_review_step_gray_zone_falls_back_to_standard(settings, trace, "comment": "breaks the consumer contract", "evidence": "hunk"}] c2 = [{"file": "src/f1.py", "line": 2, "severity": "minor", "comment": "stale docstring", "evidence": "hunk"}] + c3 = [{"file": "src/f2.py", "line": 3, "severity": "minor", + "comment": "changed path has no test", "evidence": "hunk"}] llm = ScriptedLLM([ Reply(blocks=[Block(type="text", text="prose, not json")]), # planner contract(review_comments=c1), # lens 1 contract(review_comments=c2), # lens 2 + contract(review_comments=c3), # lens 3 verdicts_reply({"i": 0, "action": "keep"}, # reducer - {"i": 1, "action": "keep"}), + {"i": 1, "action": "keep"}, + {"i": 2, "action": "keep"}), ]) registry = register_builtin_steps(StepRegistry()) result = asyncio.run(registry.get("agent.review_diff").handler( @@ -449,18 +660,18 @@ def test_review_step_gray_zone_falls_back_to_standard(settings, trace, assert result.ok, result.summary plan_ev = next(trace.events("review_plan")) assert plan_ev["planner"] == "llm-fallback" - assert plan_ev["lenses"] == ["logic", "behavior"] - assert len(list(trace.events("agent_dispatch"))) == 2 + assert plan_ev["lenses"] == ["logic", "behavior", "verification"] + assert len(list(trace.events("agent_dispatch"))) == 3 assert not llm._replies # the whole script was consumed def test_review_step_cap_applies_to_light_path(settings, trace, tmp_path, git_repo): - """The 5-comment severity-ordered budget is a product cap, not ensemble + """The 8-comment severity-ordered budget is a product cap, not ensemble mechanics — it applies to the light single pass too.""" settings.review_ensemble = True many = ([{"file": "m.py", "line": i, "severity": "nit", "comment": f"n{i}", - "evidence": "hunk"} for i in range(6)] + "evidence": "hunk"} for i in range(9)] + [{"file": "m.py", "line": 9, "severity": "major", "comment": "big", "evidence": "hunk"}]) llm = ScriptedLLM([contract(review_comments=many)]) @@ -472,7 +683,7 @@ def test_review_step_cap_applies_to_light_path(settings, trace, tmp_path, _ctx(settings, trace, tmp_path, state, llm=llm))) assert result.ok, result.summary kept = result.outputs["review_comments"] - assert len(kept) == 5 and kept[0]["severity"] == "major" + assert len(kept) == 8 and kept[0]["severity"] == "major" def test_zero_yield_lens_gets_one_retry(settings, trace, tmp_path): @@ -662,6 +873,7 @@ def test_zero_yield_light_pass_escalates_once_to_standard(settings, trace, contract(review_comments=[]), # light pass: nothing contract(review_comments=found), # standard lens 1 contract(review_comments=found), # standard lens 2 + contract(review_comments=found), # standard lens 3 verdicts_reply({"i": 0, "action": "keep"}), ]) state = {"diff_text": "diff --git a/mod_a.py b/mod_a.py\n" diff --git a/test/test_review_planner.py b/test/test_review_planner.py index dd90be41b..3fe3c830b 100644 --- a/test/test_review_planner.py +++ b/test/test_review_planner.py @@ -198,6 +198,43 @@ def test_rules_api_change_disqualifies_light(settings): assert classify(diff_signals(diff), settings) is None # gray, not light +def test_rules_value_flip_disqualifies_light(settings): + # a tiny diff that flips an existing assignment's value (default lists, + # flags, thresholds) can have repo-wide blast radius — it must go gray, + # never light, even though no def/class/CONST shape changed + diff = ("diff --git a/src/platform.py b/src/platform.py\n" + "--- a/src/platform.py\n+++ b/src/platform.py\n" + '- default = ["native"] if using_inductor else ["c", "native"]\n' + '+ default = ["c", "native"]\n') + sig = diff_signals(diff) + assert sig.value_flips == ('src/platform.py: `default`',) + assert classify(sig, settings) is None # gray, not light + + +def test_signals_code_motion_is_not_a_value_flip(settings): + # same LHS, same RHS: the assignment merely moved — still light + diff = ("diff --git a/src/mod.py b/src/mod.py\n" + "--- a/src/mod.py\n+++ b/src/mod.py\n" + "- limit = 10\n" + "+ limit = 10\n") + sig = diff_signals(diff) + assert sig.value_flips == () + assert classify(sig, settings)[0] == "light" + + +def test_signals_config_file_flip_stays_light(settings): + # the flip signal is code-file-only: a 2-line yaml value change keeps the + # light tier (config semantics are the config owner's checklist, and tiny + # config deletions measured as safe light items) + diff = ("diff --git a/deploy/a.yaml b/deploy/a.yaml\n" + "--- a/deploy/a.yaml\n+++ b/deploy/a.yaml\n" + "-timeout = 30\n" + "+timeout = 60\n") + sig = diff_signals(diff) + assert sig.value_flips == () + assert classify(sig, settings)[0] == "light" + + # ---- plan_review: overrides and invariants ---------------------------------- def test_override_wins_with_zero_llm_calls(settings): diff --git a/test/test_review_step.py b/test/test_review_step.py index def167c2f..ee79f2dbe 100644 --- a/test/test_review_step.py +++ b/test/test_review_step.py @@ -100,6 +100,46 @@ def test_review_runtime_flow(settings, trace, tmp_path, git_repo): assert out_ev["status"] == "success" and out_ev["tool_calls"] == 1 +def test_review_checklist_resolves_from_adapter_knowledge(settings, trace, + tmp_path, git_repo): + """`knowledge.review_checklist` in the adapter manifest injects the named + knowledge page (first 4k chars) into the reviewer guidance; a path that + escapes the knowledge root is ignored.""" + kroot = tmp_path / "knowledge" + (kroot / "repos" / "repo_x").mkdir(parents=True) + (kroot / "repos" / "repo_x" / "checklist.md").write_text( + "REPO-X-CHECKLIST-MARKER: check the frobnicator") + settings.knowledge_dir = kroot + adir = settings.adapters_dir / "repo_x" + adir.mkdir(parents=True) + (adir / "manifest.yaml").write_text( + "name: repo_x\nstatus: active\n" + f"repo: {{path: {git_repo}}}\n" + "knowledge: {review_checklist: repos/repo_x/checklist.md}\n") + llm = ScriptedLLM([_contract_reply([])]) + state = {"diff_text": "diff --git a/mod_a.py b/mod_a.py\n" + "--- a/mod_a.py\n+++ b/mod_a.py\n+x = 1", + "task_spec": {"pr": 9, "repo": "repo-x"}, + "repo_path": str(git_repo)} + result = asyncio.run(_registry().get("agent.review_diff").handler( + _ctx(settings, trace, tmp_path, state, llm=llm))) + assert result.ok, result.summary + prompt = llm.calls[0]["messages"][0]["content"] + assert "REPO-X-CHECKLIST-MARKER" in prompt + + # escape guard: a traversal path is ignored, not read + (adir / "manifest.yaml").write_text( + "name: repo_x\nstatus: active\n" + f"repo: {{path: {git_repo}}}\n" + "knowledge: {review_checklist: ../../../etc/passwd}\n") + llm2 = ScriptedLLM([_contract_reply([])]) + state2 = dict(state, diff_text=state["diff_text"]) + result2 = asyncio.run(_registry().get("agent.review_diff").handler( + _ctx(settings, trace, tmp_path, state2, llm=llm2))) + assert result2.ok + assert "root:" not in llm2.calls[0]["messages"][0]["content"] + + def test_review_contract_repair_round(settings, trace, tmp_path, git_repo): """Prose final output triggers exactly one repair call that must yield JSON.""" llm = ScriptedLLM([ diff --git a/test/test_v2_p0.py b/test/test_v2_p0.py index 8cf5bd7f9..ab2e09b3b 100644 --- a/test/test_v2_p0.py +++ b/test/test_v2_p0.py @@ -283,8 +283,13 @@ def test_high_risk_modules_from_adapter_override(): def test_adapter_zero_declares_risk_tiers(): from infermatrix_copilot.adapters.base import load_adapter adapter = load_adapter(REPO_ROOT / "adapters" / "vllm_omni") - assert set(adapter.high_risk_modules) == {"worker_runner", "model_executor", - "scheduler"} + assert set(adapter.high_risk_modules) == { + "worker_runner", "model_executor", "scheduler", + # extended from review-campaign forensics: platform code sets + # process-wide defaults, diffusion is multi-model shared code, and + # distributed owns cross-stage transport — small diffs there still + # need the full review ensemble + "platform", "diffusion", "distributed"} # -- repo-neutral core guard (§V2.2.1) ------------------------------------------- From 81e4b6f9cfc0f553692a8f6cdffaf8583784a589 Mon Sep 17 00:00:00 2001 From: tzhouam Date: Fri, 14 Aug 2026 18:02:56 +0800 Subject: [PATCH 02/55] Eval harness: pipelined gen+judge orchestrator, cursor-agent harness arm, wave-2 holdout wiring - run_campaign_pipelined.py: overlap generation and blind judging per item (ARM_JOBS / JUDGE_ITEM_JOBS parallelism, resumable, judge retries) - run_cursor_arm.py: headless cursor-agent arm on the pinned PR-time tree; detective leakage audit over the full stream-json tool log (no tool allowlist exists), gh auth stripped, items with violations disqualified - judge_val.py: holdout split (wave-2, 10 fresh PRs, human-only GT) - run_copilot_arm.py / run_baseline_pinned.py / build_pr_snapshots.py: wave-2 items, ARM_JOBS concurrency, commit timeline in frozen snapshots (information parity across arms) Co-Authored-By: Claude Fable 5 --- adapters/vllm_omni/skills/_candidates.json | 17 +- eval/dataset/build_pr_snapshots.py | 18 +- eval/dataset/judge_val.py | 5 + eval/dataset/run_baseline_pinned.py | 3 +- eval/dataset/run_campaign_pipelined.py | 137 ++++++++++++ eval/dataset/run_copilot_arm.py | 4 +- eval/dataset/run_cursor_arm.py | 243 +++++++++++++++++++++ 7 files changed, 422 insertions(+), 5 deletions(-) create mode 100644 eval/dataset/run_campaign_pipelined.py create mode 100644 eval/dataset/run_cursor_arm.py diff --git a/adapters/vllm_omni/skills/_candidates.json b/adapters/vllm_omni/skills/_candidates.json index 9e26dfeeb..4b3082bdd 100644 --- a/adapters/vllm_omni/skills/_candidates.json +++ b/adapters/vllm_omni/skills/_candidates.json @@ -1 +1,16 @@ -{} \ No newline at end of file +{ + "behavior-lens-device-rng-review": { + "name": "behavior-lens-device-rng-review", + "description": "Behavior lens for PRs that flip a device guard from ==cuda to !=cpu: enumerate platform inheritance and validate the claimed allowlist", + "body": "When a diff changes `device.type == \"cuda\"` to `device.type != \"cpu\"` (or similar) in shared model code, treat the PR's stated allowlist as a contract. Grep the platform plugin registry and recipes to enumerate every device type that reaches the path; if the PR validates only a subset, the finding is \"implement the explicit allowlist or validate the rest\". Also check whether a device module is captured once with no argument (`torch.get_device_module()`) instead of bound to the actual tensor device.", + "modules": [], + "proposed_at": 1786614914.8996305 + }, + "verification-lens-pytestmark-deselection": { + "name": "verification-lens-pytestmark-deselection", + "description": "PR review verification lens: new test files without pytestmark are silently deselected by marker-filtered CI; check CI selection command vs each added test file's markers.", + "body": "When reviewing a PR that adds test files, grep each added file for `pytestmark`. In vllm-omni, CI selects tests with `-m 'core_model and cpu'` (`.buildkite/cuda/test-ready.yml:13/23/28`, `.buildkite/amd/test-amd-ready.yml:19`) and the XPU lane with `-m 'core_model and xpu and B60'`. There is no `pytest_collection_modifyitems` hook that auto-adds `core_model`, so a new test file without `pytestmark = [pytest.mark.core_model, pytest.mark.cpu]` is silently deselected (green CI, zero execution). Test integrity outranks test existence: report it as major and ask for the module-level marker matching every sibling file in the directory.", + "modules": [], + "proposed_at": 1786692376.3324893 + } +} \ No newline at end of file diff --git a/eval/dataset/build_pr_snapshots.py b/eval/dataset/build_pr_snapshots.py index be00321f5..f1c89f713 100644 --- a/eval/dataset/build_pr_snapshots.py +++ b/eval/dataset/build_pr_snapshots.py @@ -53,7 +53,8 @@ # Fields we are allowed to request. Anything carrying reviewer opinion — comments, # reviews, or the pulls//comments endpoint — is absent by construction, not by # filtering after the fact. -_VIEW_FIELDS = "title,body,labels,headRefName,state,isDraft,mergeable,mergeStateStatus" +_VIEW_FIELDS = ("title,body,labels,headRefName,state,isDraft,mergeable," + "mergeStateStatus,commits") def _gh(args: list[str]) -> tuple[int, str]: @@ -78,6 +79,18 @@ def _context_bundle(pr: int, data: dict) -> str: + f"\n{_clip(data.get('body'), 4000)}") else: parts.append("(pr view unavailable — partial context)") + if data: + # commit timeline — reproduced from fetch.py `_pr_context_bundle` + # (added there for the squashed-diff add-then-revert class); the + # snapshot must carry it too or the copilot arm holds information + # the reference arms were never given + subjects = [ + f"- {str(c.get('oid') or '')[:8]} " + f"{_clip((c.get('messageHeadline') or ''), 100)}" + for c in (data.get("commits") or [])[-20:]] + if subjects: + parts.append("## Commit timeline (subjects only — the diff below " + "is the squashed net change)\n" + "\n".join(subjects)) hay = f"{(data.get('body') or '')} {(data.get('headRefName') or '')}" for num in list(dict.fromkeys(_LINKED_ISSUE.findall(hay)))[:2]: code, out = _gh(["issue", "view", num, "--json", "title,body"]) @@ -142,7 +155,8 @@ def build(pr: int) -> tuple[str, str]: def main() -> int: splits = set((sys.argv[1] if len(sys.argv) > 1 else "train,val,test").split(",")) ds = yaml.safe_load(DATASET.read_text(encoding="utf-8")) - items = [i for i in ds["pr_review"] if i.get("split") in splits] + items = [i for i in ds["pr_review"] + (ds.get("pr_review_wave2") or []) + if i.get("split") in splits] OUT.mkdir(parents=True, exist_ok=True) failures = [] diff --git a/eval/dataset/judge_val.py b/eval/dataset/judge_val.py index c452961ac..a5abd08b9 100644 --- a/eval/dataset/judge_val.py +++ b/eval/dataset/judge_val.py @@ -66,6 +66,11 @@ 4893, 4810, 4825, 4837, 4816, # val 4762, 4834, 4849, 4954, 4777], # test "issues": []}, + # Wave 2 (build_wave2.py): 10 recent PRs, pure frozen holdout — human-only + # GT, no GOLD gap items (recent PRs lack the history to prove one). + "holdout": {"prs": [5509, 5550, 5610, 5703, 5715, 5840, 5863, 5884, + 5957, 5976], + "issues": []}, } _SPLIT = os.environ.get("SPLIT", "val") PR_ITEMS = _SPLITS[_SPLIT]["prs"] diff --git a/eval/dataset/run_baseline_pinned.py b/eval/dataset/run_baseline_pinned.py index 8bb575523..c4c38edf1 100644 --- a/eval/dataset/run_baseline_pinned.py +++ b/eval/dataset/run_baseline_pinned.py @@ -75,7 +75,8 @@ def main() -> int: only = sys.argv[2] if len(sys.argv) > 2 else "" ds = yaml.safe_load(DATASET.read_text(encoding="utf-8")) heads = json.loads(EXPECTED_HEADS.read_text(encoding="utf-8")) - items = [i for i in ds["pr_review"] if i.get("split") in splits] + items = [i for i in ds["pr_review"] + (ds.get("pr_review_wave2") or []) + if i.get("split") in splits] if only: items = [i for i in items if f"pr{i['pr']}" == only] OUT.mkdir(parents=True, exist_ok=True) diff --git a/eval/dataset/run_campaign_pipelined.py b/eval/dataset/run_campaign_pipelined.py new file mode 100644 index 000000000..4bbdd95a9 --- /dev/null +++ b/eval/dataset/run_campaign_pipelined.py @@ -0,0 +1,137 @@ +#!/usr/bin/env python3 +"""Pipelined campaign: generation and judging overlap per item. + +The serial shape (generate every review, then judge every review) costs +wall-clock equal to the SUM of the two phases. Generation and judging use +disjoint capacity (the arm burns the copilot's LLM endpoint; the judge burns +the judge backend), so the phases can overlap: the moment one item's review +lands on disk, its verdicts can start while sibling items are still +generating. Wall-clock collapses to ~max(generation) + one item's judging. + +Usage: + run_campaign_pipelined.py + + splits comma list over BOTH dataset blocks (train,val,test,holdout) + arm_out arm directory name under arms/ (also the generation ARM_OUT) + +Env (forwarded): + generation — KINDS (forced pr_review here), MOA_WHEN, ARM_JOBS, OMNI_CLI + judging — ARM_B_DIR (default baselines/claudecode_opus5), JUDGE_OUT + (default judgments/_pipelined), REPLICATES, + JUDGE_BACKEND, JUDGE_MODEL + JUDGE_ITEM_JOBS concurrent per-item judge processes (default 3) + GEN_CMD override the generation command (testing) + JUDGE_CMD override the per-item judge command (testing; receives + the PR number appended as its last argument) + +Judging per item is one `judge_val.py` invocation with ONLY_ITEMS= +(resumable, so a crashed pipeline rerun skips finished verdicts). A judge +failure is retried up to 3 times, then reported — never silently averaged +over. The pipeline exits non-zero if any item failed to generate or judge. +""" +from __future__ import annotations + +import json +import os +import subprocess +import sys +import threading +import time +from pathlib import Path + +import yaml + +HERE = Path(__file__).parent +DATASET = HERE / "vllm_omni_dataset.yaml" + + +def expected_prs(splits: set[str]) -> list[int]: + ds = yaml.safe_load(DATASET.read_text(encoding="utf-8")) + items = ds["pr_review"] + (ds.get("pr_review_wave2") or []) + return [int(i["pr"]) for i in items if i.get("split") in splits] + + +def main() -> int: + if len(sys.argv) < 3: + print(__doc__.strip().splitlines()[0]) + print("usage: run_campaign_pipelined.py ") + return 2 + splits = set(sys.argv[1].split(",")) + arm_out = sys.argv[2] + prs = expected_prs(splits) + if not prs: + print(f"no pr_review items in splits {sorted(splits)}") + return 2 + arm_dir = HERE / "arms" / arm_out + judge_out = os.environ.get("JUDGE_OUT", f"judgments/{arm_out}_pipelined") + reps = int(os.environ.get("REPLICATES", "3")) + item_jobs = int(os.environ.get("JUDGE_ITEM_JOBS", "3")) + + gen_cmd = os.environ.get("GEN_CMD") + gen_argv = (gen_cmd.split() if gen_cmd else + [sys.executable, str(HERE / "run_copilot_arm.py"), + ",".join(sorted(splits))]) + gen_env = dict(os.environ, KINDS="pr_review", ARM_OUT=arm_out) + print(f"[pipeline] generating {len(prs)} item(s) -> {arm_dir.name}; " + f"judging {reps}x each -> {judge_out} as they land", flush=True) + gen = subprocess.Popen(gen_argv, env=gen_env, cwd=str(HERE)) + + sem = threading.Semaphore(item_jobs) + failures: list[str] = [] + lock = threading.Lock() + + def judge_item(pr: int) -> None: + with sem: + judge_cmd = os.environ.get("JUDGE_CMD") + argv = (judge_cmd.split() + [str(pr)] if judge_cmd else + [sys.executable, str(HERE / "judge_val.py")]) + env = dict(os.environ, SPLIT="all_pr" if "holdout" not in splits + else "holdout", ONLY_ITEMS=str(pr), + ARM_A_DIR=f"arms/{arm_out}", + JUDGE_OUT=judge_out, REPLICATES=str(reps)) + for attempt in range(1, 4): + rc = subprocess.run(argv, env=env, cwd=str(HERE)).returncode + if rc == 0: + print(f"[pipeline] judged pr{pr}", flush=True) + return + print(f"[pipeline] judge pr{pr} attempt {attempt} rc={rc}", flush=True) + time.sleep(5 * attempt) + with lock: + failures.append(f"pr{pr}: judging failed after 3 attempts") + + threads: list[threading.Thread] = [] + pending = set(prs) + while pending: + for pr in sorted(pending): + md = arm_dir / f"pr{pr}.md" + if md.is_file() and md.stat().st_size > 0: + pending.discard(pr) + t = threading.Thread(target=judge_item, args=(pr,), daemon=True) + t.start() + threads.append(t) + break + else: + if gen.poll() is not None and pending: + # generation exited with items missing: report, do not spin + with lock: + failures.extend(f"pr{p}: never generated" + for p in sorted(pending)) + pending.clear() + time.sleep(5) + try: + gen.wait(timeout=300) + except subprocess.TimeoutExpired: + # every expected item is on disk; a generator that lingers past its + # grace (or a never-exiting GEN_CMD stub) must not hang the campaign + gen.terminate() + for t in threads: + t.join() + print(f"[pipeline] complete: {len(prs) - len(failures)}/{len(prs)} ok", + flush=True) + for f in failures: + print(f" FAILED {f}") + return 1 if failures else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/eval/dataset/run_copilot_arm.py b/eval/dataset/run_copilot_arm.py index e4ba815d8..8ba812e90 100644 --- a/eval/dataset/run_copilot_arm.py +++ b/eval/dataset/run_copilot_arm.py @@ -52,7 +52,7 @@ RUN_ROOT = Path.home() / ".infermatrix-copilot" / "runs" CLI = _os.environ.get("OMNI_CLI") or _shutil.which("infermatrix-copilot") or "infermatrix-copilot" CWD = HERE.parent.parent # repo root, where .env lives -SPLIT_ORDER = {"val": 0, "train": 1, "test": 2} +SPLIT_ORDER = {"val": 0, "train": 1, "test": 2, "holdout": 3} def _find_run_dir(private_root: Path, kind: str, n: int) -> Path | None: @@ -214,6 +214,8 @@ def main() -> None: kinds = [k for k in (_os.environ.get("KINDS") or "pr_review,issue_answer").split(",") if k] items = ([("pr_review", i["pr"], i["split"]) for i in d["pr_review"]] + + [("pr_review", i["pr"], i["split"]) + for i in (d.get("pr_review_wave2") or [])] + [("issue_answer", i["issue"], i["split"]) for i in d["issue_answer"]]) items = [t for t in items if t[2] in want and t[0] in kinds] if only: diff --git a/eval/dataset/run_cursor_arm.py b/eval/dataset/run_cursor_arm.py new file mode 100644 index 000000000..f2160fcb8 --- /dev/null +++ b/eval/dataset/run_cursor_arm.py @@ -0,0 +1,243 @@ +#!/usr/bin/env python3 +"""Cursor Composer arm: headless `cursor-agent` on the pinned PR-time tree. + +Same measurement contract as `run_baseline_pinned.py` — pinned worktree, +frozen sanitized snapshot (byte-identical to what every other arm received), +no PR-discussion access — with one structural difference that is disclosed +rather than hidden: `cursor-agent` has no tool allowlist, so where the +Claude baseline PREVENTS discussion access (allowlist), this arm DETECTS it +(post-run audit over the full stream-json tool log) and fails the item on +any violation. Plan mode was probed and rejected: it disables the shell, +which would strip `git diff`/grep and cripple the arm relative to the +baseline it is compared against. + +gh auth is stripped for the subprocess (empty GH_CONFIG_DIR, tokens +removed); the repo is public, so unauthenticated fetches remain *possible* +— which is exactly why the audit, not the environment, is the gate. + +Usage: run_cursor_arm.py [splits] [only_stem] +Env: ARM_OUT (default cursor_composer25), ARM_MODEL (default composer-2.5), + ARM_JOBS (default 3), CURSOR_TIMEOUT_S (default 1800) +Outputs: arms//pr.{md,cost.json,events.jsonl} + manifest.json +""" +from __future__ import annotations + +import json +import os +import re +import shutil +import subprocess +import sys +import tempfile +import threading +import time +from concurrent.futures import ThreadPoolExecutor, as_completed +from pathlib import Path + +import yaml + +import cc_arm_common as cc +from run_ocr_arm import _prepare_worktree, _write_atomic, EXPECTED_HEADS, JUDGE_CAP + +HERE = Path(__file__).parent +DATASET = HERE / "vllm_omni_dataset.yaml" +OUT = HERE / "arms" / os.environ.get("ARM_OUT", "cursor_composer25") +MODEL = os.environ.get("ARM_MODEL", "composer-2.5") +TIMEOUT = int(os.environ.get("CURSOR_TIMEOUT_S", "1800")) +ARM = f"cursor_{MODEL.replace('-', '')}" + +# Discussion-access signatures. `git fetch pull/N/head` is CODE and allowed; +# what must never be reachable is the human review thread the ground truth is +# made of. Case-insensitive, matched against every shell command the agent ran. +_FORBIDDEN = re.compile( + r"(?i)\bgh\s+(pr|issue|api)\b" + r"|api\.github\.com/repos/[^ ]*/(pulls|issues)/\d+/(comments|reviews)" + r"|github\.com/[^ ]*/pull/\d+([^0-9]|$)(?![^ ]*\.diff)" + r"|(curl|wget)[^|;&]*github\.com[^|;&]*(comment|review|discussion)") + + +def prompt(pr: int, snap: str, base: str, head: str) -> str: + return ( + f"Review pull request #{pr} of {cc.REPO} as an expert reviewer.\n\n" + f"Your working directory IS a read-only checkout of the repository at this " + f"PR's exact head commit ({head[:12]}) — the merge-base with main is " + f"{base[:12]}, so `git diff {base} HEAD` is precisely this PR's change. The " + f"code around it is the code as it stood when the PR was opened.\n\n" + f"Frozen PR context (this is all of it — there is no PR discussion available, " + f"deliberately; do NOT try to fetch any):\n\n{snap}\n\n" + f"Investigate the code before concluding: read the changed files and the code " + f"that calls into them. Do not modify any file, and do not read ANY file " + f"outside this working directory (no home-directory configs, no other " + f"checkouts) — reviews that do are disqualified.\n\n" + f"Budget: spend at most two thirds of your effort investigating and then " + f"WRITE THE REVIEW — an unwritten review scores as silence no matter how " + f"good the investigation behind it was.\n\n" + f"IMPORTANT: do not post anything to GitHub. Output the complete review as " + f"your final message — an overall verdict, then specific comments with " + f"file:line references and what to change." + ) + + +def _run_cursor(text: str, cwd: str, gh_empty: Path) -> list[dict]: + env = {k: v for k, v in os.environ.items() + if k not in ("GH_TOKEN", "GITHUB_TOKEN")} + env["GH_CONFIG_DIR"] = str(gh_empty) + p = subprocess.run( + ["cursor-agent", "--print", "--trust", "--force", + "--output-format", "stream-json", "--model", MODEL, text], + cwd=cwd, env=env, capture_output=True, text=True, + encoding="utf-8", errors="replace", timeout=TIMEOUT) + events = [] + for line in (p.stdout or "").splitlines(): + line = line.strip() + if line.startswith("{"): + try: + events.append(json.loads(line)) + except json.JSONDecodeError: + continue + return events + + +def _audit(events: list[dict], worktree: str) -> tuple[list[str], int, int]: + """(violations, shell_commands, reads). Reads outside the worktree and any + discussion-access command are violations — same policy the baseline's + audit enforces, applied detectively instead of preventively.""" + violations: list[str] = [] + n_shell = n_read = 0 + wt_real = os.path.realpath(worktree) + for e in events: + tc = (e.get("tool_call") or {}) if e.get("type") == "tool_call" else {} + shell = tc.get("shellToolCall") + if shell and "result" not in shell: + cmd = str((shell.get("args") or {}).get("command") or "") + n_shell += 1 + if _FORBIDDEN.search(cmd): + violations.append(f"shell: forbidden discussion access: " + f"{cmd[:160]}") + read = tc.get("readToolCall") + if read and "result" not in read: + path = str((read.get("args") or {}).get("path") or "") + n_read += 1 + # realpath both sides: the machine reaches the same worktree via + # /home and /data prefixes, and a symlinked HOME must not read as + # an out-of-tree violation + if path and not (os.path.realpath(path) + "/").startswith( + wt_real + "/"): + violations.append(f"read outside worktree: {path[:160]}") + write = tc.get("writeToolCall") or tc.get("editToolCall") + if write: + violations.append("write attempted in read-only review") + return violations, n_shell, n_read + + +def _final_text(events: list[dict]) -> str: + for e in reversed(events): + if e.get("type") == "result": + return str(e.get("result") or "").strip() + return "" + + +def main() -> int: + splits = set((sys.argv[1] if len(sys.argv) > 1 else "holdout").split(",")) + only = sys.argv[2] if len(sys.argv) > 2 else "" + ds = yaml.safe_load(DATASET.read_text(encoding="utf-8")) + heads = json.loads(EXPECTED_HEADS.read_text(encoding="utf-8")) + items = [i for i in ds["pr_review"] + (ds.get("pr_review_wave2") or []) + if i.get("split") in splits] + if only: + items = [i for i in items if f"pr{i['pr']}" == only] + OUT.mkdir(parents=True, exist_ok=True) + gh_empty = Path(tempfile.mkdtemp(prefix="cursor-arm-gh-")) + + failures: list[str] = [] + done = 0 + lock = threading.Lock() + + pending = [] + for item in items: # serial: worktree lock + pr = int(item["pr"]) + stem = f"pr{pr}" + expected_head = heads.get(str(pr)) + if not expected_head: + failures.append(f"{stem}: no pinned head") + continue + if (OUT / f"{stem}.md").is_file() and (OUT / f"{stem}.md").stat().st_size: + done += 1 + print(f" {stem}: already done, skipping") + continue + try: + wt, base = _prepare_worktree(str(pr), expected_head, + int(item["size"]["files"])) + except Exception as exc: # noqa: BLE001 + failures.append(f"{stem}: {exc}") + continue + pending.append((item, str(wt), str(base), expected_head)) + + def _review(job) -> None: + nonlocal done + item, wt, base, expected_head = job + pr = int(item["pr"]) + stem = f"pr{pr}" + snap, snap_sha = cc.snapshot(pr) + t0 = time.time() + events = _run_cursor(prompt(pr, snap, base, expected_head), wt, gh_empty) + wall = round(time.time() - t0, 1) + body = _final_text(events) + violations, n_shell, n_read = _audit(events, wt) + cost = {"arm": ARM, "model_requested": MODEL, "head": expected_head, + "base": base, "snapshot_sha256": snap_sha, "wall_s": wall, + "shell_commands": n_shell, "file_reads": n_read, + "cost_usd": None, # cursor does not expose per-run billing + "audit_ok": not violations, "audit_violations": violations, + "artifact_chars": len(body), "split": item.get("split")} + _write_atomic(OUT / f"{stem}.cost.json", + json.dumps(cost, indent=2) + "\n") + _write_atomic(OUT / f"{stem}.events.jsonl", + "\n".join(json.dumps(e, ensure_ascii=False) + for e in events) + "\n") + _write_atomic(OUT / f"{stem}.md", (body or "(no output)") + "\n") + with lock: + if violations: + failures.append(f"{stem}: audit violations {violations[:2]}") + print(f" {stem}: AUDIT FAILED — {violations[:2]}") + return + if not body: + failures.append(f"{stem}: empty review") + print(f" {stem}: EMPTY REVIEW") + return + done += 1 + print(f" {stem}: wall={wall}s shell={n_shell} reads={n_read} " + f"chars={len(body)}", flush=True) + + jobs = max(1, int(os.environ.get("ARM_JOBS", "3"))) + try: + with ThreadPoolExecutor(max_workers=jobs) as ex: + futs = [ex.submit(_review, j) for j in pending] + for f in as_completed(futs): + if f.exception(): + with lock: + failures.append(f"worker crashed: {f.exception()}") + _write_atomic(OUT / "manifest.json", json.dumps({ + "arm": OUT.name, "model_requested": MODEL, + "harness": "cursor-agent --print --trust --force stream-json", + "leakage_control": "DETECTIVE, not preventive: gh auth stripped " + "(empty GH_CONFIG_DIR, tokens removed) and every " + "shell command/read audited post-run; items with " + "violations are failed. cursor-agent has no tool " + "allowlist, so prevention equivalent to the " + "claude baseline is not available.", + "pr_context": "frozen snapshot (no_discussion), byte-identical to " + "the other arms", + "dataset": DATASET.name, "splits": sorted(splits), + "n_items": done, "concurrency": jobs, + }, indent=2) + "\n") + finally: + shutil.rmtree(gh_empty, ignore_errors=True) + print(f"\n{done}/{len(items)} items written to {OUT}") + for f in failures: + print(f" FAILED {f}") + return 1 if failures else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From b3a0f07f95b3626d57e00df5005697ec1cd6b793 Mon Sep 17 00:00:00 2001 From: tzhouam Date: Fri, 14 Aug 2026 18:03:21 +0800 Subject: [PATCH 03/55] Eval data: wave-2 holdout arms, judgments, and consolidated model-comparison results Arms: DS v4-pro v13 (r1+r2), MiMo-v2.5 v13, cursor Composer-2.5 and Grok-4.5 harness arms (with tool-log audits; Grok 9/10 contaminated), plus the v7-v13 campaign iteration arms. Judgments: GPT-5.6 and claude-sonnet-5 sets, never pooled. results/: aggregator + model_comparison.{md,json,csv} computed from raw verdicts. Co-Authored-By: Claude Fable 5 --- .../arms/copilot_v11_deep_r1/manifest.json | 20 + .../arms/copilot_v11_deep_r1/pr4804.cost.json | 107 + .../arms/copilot_v11_deep_r1/pr4804.md | 38 + .../copilot_v11_deep_r1/pr4804.trace.json.gz | Bin 0 -> 55624 bytes .../arms/copilot_v11_deep_r1/pr4923.cost.json | 107 + .../arms/copilot_v11_deep_r1/pr4923.md | 40 + .../copilot_v11_deep_r1/pr4923.trace.json.gz | Bin 0 -> 24025 bytes .../arms/copilot_v11_deep_r1/pr4926.cost.json | 107 + .../arms/copilot_v11_deep_r1/pr4926.md | 44 + .../copilot_v11_deep_r1/pr4926.trace.json.gz | Bin 0 -> 29557 bytes .../arms/copilot_v11_deep_r1/pr4977.cost.json | 107 + .../arms/copilot_v11_deep_r1/pr4977.md | 36 + .../copilot_v11_deep_r1/pr4977.trace.json.gz | Bin 0 -> 19100 bytes .../arms/copilot_v11_deep_r2/manifest.json | 20 + .../arms/copilot_v11_deep_r2/pr4804.cost.json | 107 + .../arms/copilot_v11_deep_r2/pr4804.md | 40 + .../copilot_v11_deep_r2/pr4804.trace.json.gz | Bin 0 -> 58245 bytes .../arms/copilot_v11_deep_r2/pr4923.cost.json | 107 + .../arms/copilot_v11_deep_r2/pr4923.md | 40 + .../copilot_v11_deep_r2/pr4923.trace.json.gz | Bin 0 -> 24331 bytes .../arms/copilot_v11_deep_r2/pr4926.cost.json | 107 + .../arms/copilot_v11_deep_r2/pr4926.md | 46 + .../copilot_v11_deep_r2/pr4926.trace.json.gz | Bin 0 -> 35479 bytes .../arms/copilot_v11_deep_r2/pr4977.cost.json | 107 + .../arms/copilot_v11_deep_r2/pr4977.md | 34 + .../copilot_v11_deep_r2/pr4977.trace.json.gz | Bin 0 -> 15467 bytes .../arms/copilot_v11_deep_val/manifest.json | 20 + .../copilot_v11_deep_val/pr4810.cost.json | 107 + .../arms/copilot_v11_deep_val/pr4810.md | 38 + .../copilot_v11_deep_val/pr4810.trace.json.gz | Bin 0 -> 24449 bytes .../copilot_v11_deep_val/pr4816.cost.json | 113 + .../arms/copilot_v11_deep_val/pr4816.md | 38 + .../copilot_v11_deep_val/pr4816.trace.json.gz | Bin 0 -> 14821 bytes .../copilot_v11_deep_val/pr4825.cost.json | 115 ++ .../arms/copilot_v11_deep_val/pr4825.md | 38 + .../copilot_v11_deep_val/pr4825.trace.json.gz | Bin 0 -> 37222 bytes .../copilot_v11_deep_val/pr4837.cost.json | 107 + .../arms/copilot_v11_deep_val/pr4837.md | 36 + .../copilot_v11_deep_val/pr4837.trace.json.gz | Bin 0 -> 20549 bytes .../copilot_v11_deep_val/pr4893.cost.json | 107 + .../arms/copilot_v11_deep_val/pr4893.md | 38 + .../copilot_v11_deep_val/pr4893.trace.json.gz | Bin 0 -> 27194 bytes .../arms/copilot_v12_hybrid_val/manifest.json | 20 + .../copilot_v12_hybrid_val/pr4810.cost.json | 107 + .../arms/copilot_v12_hybrid_val/pr4810.md | 44 + .../pr4810.trace.json.gz | Bin 0 -> 36777 bytes .../copilot_v12_hybrid_val/pr4816.cost.json | 113 + .../arms/copilot_v12_hybrid_val/pr4816.md | 36 + .../pr4816.trace.json.gz | Bin 0 -> 17449 bytes .../copilot_v12_hybrid_val/pr4825.cost.json | 107 + .../arms/copilot_v12_hybrid_val/pr4825.md | 40 + .../pr4825.trace.json.gz | Bin 0 -> 28905 bytes .../copilot_v12_hybrid_val/pr4837.cost.json | 107 + .../arms/copilot_v12_hybrid_val/pr4837.md | 44 + .../pr4837.trace.json.gz | Bin 0 -> 28683 bytes .../copilot_v12_hybrid_val/pr4893.cost.json | 107 + .../arms/copilot_v12_hybrid_val/pr4893.md | 44 + .../pr4893.trace.json.gz | Bin 0 -> 34328 bytes .../copilot_v13_ds_wave2_r2/manifest.json | 20 + .../copilot_v13_ds_wave2_r2/pr5509.cost.json | 107 + .../arms/copilot_v13_ds_wave2_r2/pr5509.md | 53 + .../pr5509.trace.json.gz | Bin 0 -> 43253 bytes .../copilot_v13_ds_wave2_r2/pr5550.cost.json | 107 + .../arms/copilot_v13_ds_wave2_r2/pr5550.md | 64 + .../pr5550.trace.json.gz | Bin 0 -> 39745 bytes .../copilot_v13_ds_wave2_r2/pr5610.cost.json | 113 + .../arms/copilot_v13_ds_wave2_r2/pr5610.md | 40 + .../pr5610.trace.json.gz | Bin 0 -> 30660 bytes .../copilot_v13_ds_wave2_r2/pr5703.cost.json | 107 + .../arms/copilot_v13_ds_wave2_r2/pr5703.md | 52 + .../pr5703.trace.json.gz | Bin 0 -> 35196 bytes .../copilot_v13_ds_wave2_r2/pr5715.cost.json | 101 + .../arms/copilot_v13_ds_wave2_r2/pr5715.md | 38 + .../pr5715.trace.json.gz | Bin 0 -> 12478 bytes .../copilot_v13_ds_wave2_r2/pr5840.cost.json | 107 + .../arms/copilot_v13_ds_wave2_r2/pr5840.md | 50 + .../pr5840.trace.json.gz | Bin 0 -> 45567 bytes .../copilot_v13_ds_wave2_r2/pr5863.cost.json | 107 + .../arms/copilot_v13_ds_wave2_r2/pr5863.md | 46 + .../pr5863.trace.json.gz | Bin 0 -> 28294 bytes .../copilot_v13_ds_wave2_r2/pr5884.cost.json | 107 + .../arms/copilot_v13_ds_wave2_r2/pr5884.md | 52 + .../pr5884.trace.json.gz | Bin 0 -> 32883 bytes .../copilot_v13_ds_wave2_r2/pr5957.cost.json | 107 + .../arms/copilot_v13_ds_wave2_r2/pr5957.md | 60 + .../pr5957.trace.json.gz | Bin 0 -> 134530 bytes .../copilot_v13_ds_wave2_r2/pr5976.cost.json | 107 + .../arms/copilot_v13_ds_wave2_r2/pr5976.md | 54 + .../pr5976.trace.json.gz | Bin 0 -> 72705 bytes .../arms/copilot_v13_mimo_wave2/manifest.json | 20 + .../copilot_v13_mimo_wave2/pr5509.cost.json | 107 + .../arms/copilot_v13_mimo_wave2/pr5509.md | 54 + .../pr5509.trace.json.gz | Bin 0 -> 34876 bytes .../copilot_v13_mimo_wave2/pr5550.cost.json | 107 + .../arms/copilot_v13_mimo_wave2/pr5550.md | 52 + .../pr5550.trace.json.gz | Bin 0 -> 33421 bytes .../copilot_v13_mimo_wave2/pr5610.cost.json | 113 + .../arms/copilot_v13_mimo_wave2/pr5610.md | 36 + .../pr5610.trace.json.gz | Bin 0 -> 20781 bytes .../copilot_v13_mimo_wave2/pr5703.cost.json | 107 + .../arms/copilot_v13_mimo_wave2/pr5703.md | 46 + .../pr5703.trace.json.gz | Bin 0 -> 28785 bytes .../copilot_v13_mimo_wave2/pr5715.cost.json | 107 + .../arms/copilot_v13_mimo_wave2/pr5715.md | 45 + .../pr5715.trace.json.gz | Bin 0 -> 17813 bytes .../copilot_v13_mimo_wave2/pr5840.cost.json | 107 + .../arms/copilot_v13_mimo_wave2/pr5840.md | 50 + .../pr5840.trace.json.gz | Bin 0 -> 37387 bytes .../copilot_v13_mimo_wave2/pr5863.cost.json | 101 + .../arms/copilot_v13_mimo_wave2/pr5863.md | 36 + .../pr5863.trace.json.gz | Bin 0 -> 13048 bytes .../copilot_v13_mimo_wave2/pr5884.cost.json | 107 + .../arms/copilot_v13_mimo_wave2/pr5884.md | 54 + .../pr5884.trace.json.gz | Bin 0 -> 22667 bytes .../copilot_v13_mimo_wave2/pr5957.cost.json | 107 + .../arms/copilot_v13_mimo_wave2/pr5957.md | 63 + .../pr5957.trace.json.gz | Bin 0 -> 124711 bytes .../copilot_v13_mimo_wave2/pr5976.cost.json | 107 + .../arms/copilot_v13_mimo_wave2/pr5976.md | 58 + .../pr5976.trace.json.gz | Bin 0 -> 63927 bytes .../arms/copilot_v13_proof_test/manifest.json | 20 + .../pr4762.trace.json.gz | Bin 0 -> 44866 bytes .../pr4777.trace.json.gz | Bin 0 -> 23079 bytes .../pr4834.trace.json.gz | Bin 0 -> 38961 bytes .../pr4849.trace.json.gz | Bin 0 -> 30108 bytes .../pr4954.trace.json.gz | Bin 0 -> 32463 bytes .../arms/copilot_v13_proof_val/manifest.json | 20 + .../pr4810.trace.json.gz | Bin 0 -> 32497 bytes .../pr4816.trace.json.gz | Bin 0 -> 24035 bytes .../pr4825.trace.json.gz | Bin 0 -> 30778 bytes .../pr4837.trace.json.gz | Bin 0 -> 28443 bytes .../pr4893.trace.json.gz | Bin 0 -> 35345 bytes .../arms/copilot_v13_wave2/manifest.json | 20 + .../arms/copilot_v13_wave2/pr5509.cost.json | 107 + eval/dataset/arms/copilot_v13_wave2/pr5509.md | 70 + .../copilot_v13_wave2/pr5509.trace.json.gz | Bin 0 -> 44396 bytes .../arms/copilot_v13_wave2/pr5550.cost.json | 107 + eval/dataset/arms/copilot_v13_wave2/pr5550.md | 66 + .../copilot_v13_wave2/pr5550.trace.json.gz | Bin 0 -> 38779 bytes .../arms/copilot_v13_wave2/pr5610.cost.json | 113 + eval/dataset/arms/copilot_v13_wave2/pr5610.md | 42 + .../copilot_v13_wave2/pr5610.trace.json.gz | Bin 0 -> 27821 bytes .../arms/copilot_v13_wave2/pr5703.cost.json | 107 + eval/dataset/arms/copilot_v13_wave2/pr5703.md | 59 + .../copilot_v13_wave2/pr5703.trace.json.gz | Bin 0 -> 37901 bytes .../arms/copilot_v13_wave2/pr5715.cost.json | 101 + eval/dataset/arms/copilot_v13_wave2/pr5715.md | 32 + .../copilot_v13_wave2/pr5715.trace.json.gz | Bin 0 -> 12109 bytes .../arms/copilot_v13_wave2/pr5840.cost.json | 107 + eval/dataset/arms/copilot_v13_wave2/pr5840.md | 53 + .../copilot_v13_wave2/pr5840.trace.json.gz | Bin 0 -> 47296 bytes .../arms/copilot_v13_wave2/pr5863.cost.json | 107 + eval/dataset/arms/copilot_v13_wave2/pr5863.md | 50 + .../copilot_v13_wave2/pr5863.trace.json.gz | Bin 0 -> 29820 bytes .../arms/copilot_v13_wave2/pr5884.cost.json | 107 + eval/dataset/arms/copilot_v13_wave2/pr5884.md | 56 + .../copilot_v13_wave2/pr5884.trace.json.gz | Bin 0 -> 32833 bytes .../arms/copilot_v13_wave2/pr5957.cost.json | 107 + eval/dataset/arms/copilot_v13_wave2/pr5957.md | 53 + .../copilot_v13_wave2/pr5957.trace.json.gz | Bin 0 -> 134505 bytes .../arms/copilot_v13_wave2/pr5976.cost.json | 107 + eval/dataset/arms/copilot_v13_wave2/pr5976.md | 48 + .../copilot_v13_wave2/pr5976.trace.json.gz | Bin 0 -> 64250 bytes .../arms/copilot_v7_goal_r1/manifest.json | 20 + .../arms/copilot_v7_goal_r1/pr4804.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r1/pr4804.md | 54 + .../copilot_v7_goal_r1/pr4804.trace.json.gz | Bin 0 -> 67197 bytes .../arms/copilot_v7_goal_r1/pr4870.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r1/pr4870.md | 42 + .../copilot_v7_goal_r1/pr4870.trace.json.gz | Bin 0 -> 23674 bytes .../arms/copilot_v7_goal_r1/pr4923.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r1/pr4923.md | 42 + .../copilot_v7_goal_r1/pr4923.trace.json.gz | Bin 0 -> 24342 bytes .../arms/copilot_v7_goal_r1/pr4926.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r1/pr4926.md | 51 + .../copilot_v7_goal_r1/pr4926.trace.json.gz | Bin 0 -> 31916 bytes .../arms/copilot_v7_goal_r1/pr4977.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r1/pr4977.md | 44 + .../copilot_v7_goal_r1/pr4977.trace.json.gz | Bin 0 -> 20943 bytes .../arms/copilot_v7_goal_r1/pr5009.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r1/pr5009.md | 44 + .../copilot_v7_goal_r1/pr5009.trace.json.gz | Bin 0 -> 25234 bytes .../arms/copilot_v7_goal_r2/manifest.json | 20 + .../arms/copilot_v7_goal_r2/pr4804.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r2/pr4804.md | 46 + .../copilot_v7_goal_r2/pr4804.trace.json.gz | Bin 0 -> 63002 bytes .../arms/copilot_v7_goal_r2/pr4870.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r2/pr4870.md | 40 + .../copilot_v7_goal_r2/pr4870.trace.json.gz | Bin 0 -> 24482 bytes .../arms/copilot_v7_goal_r2/pr4923.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r2/pr4923.md | 40 + .../copilot_v7_goal_r2/pr4923.trace.json.gz | Bin 0 -> 23186 bytes .../arms/copilot_v7_goal_r2/pr4926.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r2/pr4926.md | 44 + .../copilot_v7_goal_r2/pr4926.trace.json.gz | Bin 0 -> 30983 bytes .../arms/copilot_v7_goal_r2/pr4977.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r2/pr4977.md | 34 + .../copilot_v7_goal_r2/pr4977.trace.json.gz | Bin 0 -> 19255 bytes .../arms/copilot_v7_goal_r2/pr5009.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r2/pr5009.md | 32 + .../copilot_v7_goal_r2/pr5009.trace.json.gz | Bin 0 -> 21930 bytes .../arms/copilot_v7_goal_r3/manifest.json | 20 + .../arms/copilot_v7_goal_r3/pr4804.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r3/pr4804.md | 46 + .../copilot_v7_goal_r3/pr4804.trace.json.gz | Bin 0 -> 64928 bytes .../arms/copilot_v7_goal_r3/pr4870.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r3/pr4870.md | 40 + .../copilot_v7_goal_r3/pr4870.trace.json.gz | Bin 0 -> 25263 bytes .../arms/copilot_v7_goal_r3/pr4923.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r3/pr4923.md | 40 + .../copilot_v7_goal_r3/pr4923.trace.json.gz | Bin 0 -> 21788 bytes .../arms/copilot_v7_goal_r3/pr4926.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r3/pr4926.md | 46 + .../copilot_v7_goal_r3/pr4926.trace.json.gz | Bin 0 -> 35380 bytes .../arms/copilot_v7_goal_r3/pr4977.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r3/pr4977.md | 44 + .../copilot_v7_goal_r3/pr4977.trace.json.gz | Bin 0 -> 22776 bytes .../arms/copilot_v7_goal_r3/pr5009.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r3/pr5009.md | 44 + .../copilot_v7_goal_r3/pr5009.trace.json.gz | Bin 0 -> 24417 bytes .../arms/copilot_v7_goal_r4/manifest.json | 20 + .../arms/copilot_v7_goal_r4/pr4804.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r4/pr4804.md | 47 + .../copilot_v7_goal_r4/pr4804.trace.json.gz | Bin 0 -> 63345 bytes .../arms/copilot_v7_goal_r4/pr4870.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r4/pr4870.md | 38 + .../copilot_v7_goal_r4/pr4870.trace.json.gz | Bin 0 -> 22854 bytes .../arms/copilot_v7_goal_r4/pr4923.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r4/pr4923.md | 44 + .../copilot_v7_goal_r4/pr4923.trace.json.gz | Bin 0 -> 23488 bytes .../arms/copilot_v7_goal_r4/pr4926.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r4/pr4926.md | 50 + .../copilot_v7_goal_r4/pr4926.trace.json.gz | Bin 0 -> 33322 bytes .../arms/copilot_v7_goal_r4/pr4977.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r4/pr4977.md | 40 + .../copilot_v7_goal_r4/pr4977.trace.json.gz | Bin 0 -> 19306 bytes .../arms/copilot_v7_goal_r4/pr5009.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r4/pr5009.md | 42 + .../copilot_v7_goal_r4/pr5009.trace.json.gz | Bin 0 -> 21906 bytes .../arms/copilot_v7_goal_r5/manifest.json | 20 + .../arms/copilot_v7_goal_r5/pr4804.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r5/pr4804.md | 47 + .../copilot_v7_goal_r5/pr4804.trace.json.gz | Bin 0 -> 64522 bytes .../arms/copilot_v7_goal_r5/pr4870.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r5/pr4870.md | 38 + .../copilot_v7_goal_r5/pr4870.trace.json.gz | Bin 0 -> 24493 bytes .../arms/copilot_v7_goal_r5/pr4923.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r5/pr4923.md | 44 + .../copilot_v7_goal_r5/pr4923.trace.json.gz | Bin 0 -> 23272 bytes .../arms/copilot_v7_goal_r5/pr4926.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r5/pr4926.md | 50 + .../copilot_v7_goal_r5/pr4926.trace.json.gz | Bin 0 -> 35994 bytes .../arms/copilot_v7_goal_r5/pr4977.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r5/pr4977.md | 42 + .../copilot_v7_goal_r5/pr4977.trace.json.gz | Bin 0 -> 18708 bytes .../arms/copilot_v7_goal_r5/pr5009.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r5/pr5009.md | 44 + .../copilot_v7_goal_r5/pr5009.trace.json.gz | Bin 0 -> 28260 bytes .../arms/copilot_v7_goal_r6/manifest.json | 20 + .../arms/copilot_v7_goal_r6/pr4804.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r6/pr4804.md | 50 + .../copilot_v7_goal_r6/pr4804.trace.json.gz | Bin 0 -> 67779 bytes .../arms/copilot_v7_goal_r6/pr4870.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r6/pr4870.md | 36 + .../copilot_v7_goal_r6/pr4870.trace.json.gz | Bin 0 -> 25158 bytes .../arms/copilot_v7_goal_r6/pr4923.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r6/pr4923.md | 40 + .../copilot_v7_goal_r6/pr4923.trace.json.gz | Bin 0 -> 21908 bytes .../arms/copilot_v7_goal_r6/pr4926.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r6/pr4926.md | 47 + .../copilot_v7_goal_r6/pr4926.trace.json.gz | Bin 0 -> 33743 bytes .../arms/copilot_v7_goal_r6/pr4977.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r6/pr4977.md | 40 + .../copilot_v7_goal_r6/pr4977.trace.json.gz | Bin 0 -> 20476 bytes .../arms/copilot_v7_goal_r6/pr5009.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r6/pr5009.md | 44 + .../copilot_v7_goal_r6/pr5009.trace.json.gz | Bin 0 -> 25847 bytes .../arms/copilot_v7_goal_r7/manifest.json | 20 + .../arms/copilot_v7_goal_r7/pr4804.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r7/pr4804.md | 55 + .../copilot_v7_goal_r7/pr4804.trace.json.gz | Bin 0 -> 64315 bytes .../arms/copilot_v7_goal_r7/pr4810.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r7/pr4810.md | 48 + .../copilot_v7_goal_r7/pr4810.trace.json.gz | Bin 0 -> 25992 bytes .../copilot_v7_goal_r7/pr4816.trace.json.gz | Bin 0 -> 6685 bytes .../copilot_v7_goal_r7/pr4825.trace.json.gz | Bin 0 -> 6255 bytes .../copilot_v7_goal_r7/pr4837.trace.json.gz | Bin 0 -> 14145 bytes .../arms/copilot_v7_goal_r7/pr4870.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r7/pr4870.md | 44 + .../copilot_v7_goal_r7/pr4870.trace.json.gz | Bin 0 -> 25931 bytes .../copilot_v7_goal_r7/pr4893.trace.json.gz | Bin 0 -> 9036 bytes .../arms/copilot_v7_goal_r7/pr4923.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r7/pr4923.md | 40 + .../copilot_v7_goal_r7/pr4923.trace.json.gz | Bin 0 -> 21875 bytes .../arms/copilot_v7_goal_r7/pr4926.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r7/pr4926.md | 55 + .../copilot_v7_goal_r7/pr4926.trace.json.gz | Bin 0 -> 34009 bytes .../arms/copilot_v7_goal_r7/pr4977.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r7/pr4977.md | 46 + .../copilot_v7_goal_r7/pr4977.trace.json.gz | Bin 0 -> 19305 bytes .../arms/copilot_v7_goal_r7/pr5009.cost.json | 107 + .../dataset/arms/copilot_v7_goal_r7/pr5009.md | 46 + .../copilot_v7_goal_r7/pr5009.trace.json.gz | Bin 0 -> 24921 bytes .../arms/copilot_v8_official_r1/manifest.json | 20 + .../copilot_v8_official_r1/pr4810.cost.json | 107 + .../arms/copilot_v8_official_r1/pr4810.md | 42 + .../pr4810.trace.json.gz | Bin 0 -> 28565 bytes .../copilot_v8_official_r1/pr4816.cost.json | 113 + .../arms/copilot_v8_official_r1/pr4816.md | 40 + .../pr4816.trace.json.gz | Bin 0 -> 15845 bytes .../copilot_v8_official_r1/pr4825.cost.json | 107 + .../arms/copilot_v8_official_r1/pr4825.md | 36 + .../pr4825.trace.json.gz | Bin 0 -> 19423 bytes .../copilot_v8_official_r1/pr4837.cost.json | 107 + .../arms/copilot_v8_official_r1/pr4837.md | 38 + .../pr4837.trace.json.gz | Bin 0 -> 17236 bytes .../copilot_v8_official_r1/pr4893.cost.json | 107 + .../arms/copilot_v8_official_r1/pr4893.md | 53 + .../pr4893.trace.json.gz | Bin 0 -> 22947 bytes .../arms/copilot_v9_official_r1/manifest.json | 20 + .../copilot_v9_official_r1/pr4810.cost.json | 107 + .../arms/copilot_v9_official_r1/pr4810.md | 42 + .../pr4810.trace.json.gz | Bin 0 -> 25112 bytes .../copilot_v9_official_r1/pr4816.cost.json | 113 + .../arms/copilot_v9_official_r1/pr4816.md | 38 + .../pr4816.trace.json.gz | Bin 0 -> 16895 bytes .../copilot_v9_official_r1/pr4825.cost.json | 107 + .../arms/copilot_v9_official_r1/pr4825.md | 40 + .../pr4825.trace.json.gz | Bin 0 -> 24059 bytes .../copilot_v9_official_r1/pr4837.cost.json | 107 + .../arms/copilot_v9_official_r1/pr4837.md | 38 + .../pr4837.trace.json.gz | Bin 0 -> 16787 bytes .../copilot_v9_official_r1/pr4893.cost.json | 107 + .../arms/copilot_v9_official_r1/pr4893.md | 46 + .../pr4893.trace.json.gz | Bin 0 -> 26982 bytes .../arms/cursor_composer25/manifest.json | 13 + .../arms/cursor_composer25/pr5509.cost.json | 15 + eval/dataset/arms/cursor_composer25/pr5509.md | 121 ++ .../arms/cursor_composer25/pr5550.cost.json | 15 + eval/dataset/arms/cursor_composer25/pr5550.md | 101 + .../arms/cursor_composer25/pr5610.cost.json | 15 + eval/dataset/arms/cursor_composer25/pr5610.md | 130 ++ .../arms/cursor_composer25/pr5703.cost.json | 15 + eval/dataset/arms/cursor_composer25/pr5703.md | 105 + .../arms/cursor_composer25/pr5715.cost.json | 15 + eval/dataset/arms/cursor_composer25/pr5715.md | 64 + .../arms/cursor_composer25/pr5840.cost.json | 15 + eval/dataset/arms/cursor_composer25/pr5840.md | 114 + .../arms/cursor_composer25/pr5863.cost.json | 15 + eval/dataset/arms/cursor_composer25/pr5863.md | 118 ++ .../arms/cursor_composer25/pr5884.cost.json | 17 + eval/dataset/arms/cursor_composer25/pr5884.md | 101 + .../arms/cursor_composer25/pr5957.cost.json | 15 + eval/dataset/arms/cursor_composer25/pr5957.md | 135 ++ .../arms/cursor_composer25/pr5976.cost.json | 15 + eval/dataset/arms/cursor_composer25/pr5976.md | 94 + eval/dataset/arms/cursor_grok45/manifest.json | 13 + .../arms/cursor_grok45/pr5509.cost.json | 18 + eval/dataset/arms/cursor_grok45/pr5509.md | 81 + .../arms/cursor_grok45/pr5550.cost.json | 15 + eval/dataset/arms/cursor_grok45/pr5550.md | 86 + .../arms/cursor_grok45/pr5610.cost.json | 17 + eval/dataset/arms/cursor_grok45/pr5610.md | 67 + .../arms/cursor_grok45/pr5703.cost.json | 19 + eval/dataset/arms/cursor_grok45/pr5703.md | 82 + .../arms/cursor_grok45/pr5715.cost.json | 17 + eval/dataset/arms/cursor_grok45/pr5715.md | 41 + .../arms/cursor_grok45/pr5840.cost.json | 18 + eval/dataset/arms/cursor_grok45/pr5840.md | 49 + .../arms/cursor_grok45/pr5863.cost.json | 17 + eval/dataset/arms/cursor_grok45/pr5863.md | 49 + .../arms/cursor_grok45/pr5884.cost.json | 18 + eval/dataset/arms/cursor_grok45/pr5884.md | 38 + .../arms/cursor_grok45/pr5957.cost.json | 18 + eval/dataset/arms/cursor_grok45/pr5957.md | 71 + .../arms/cursor_grok45/pr5976.cost.json | 19 + eval/dataset/arms/cursor_grok45/pr5976.md | 63 + .../arms/goal_debug_promo/manifest.json | 20 + .../arms/goal_debug_promo/pr4923.cost.json | 107 + eval/dataset/arms/goal_debug_promo/pr4923.md | 40 + .../goal_debug_promo/pr4923.trace.json.gz | Bin 0 -> 21516 bytes .../arms/goal_debug_promo2/manifest.json | 20 + .../arms/goal_debug_promo2/pr5009.cost.json | 107 + eval/dataset/arms/goal_debug_promo2/pr5009.md | 44 + .../goal_debug_promo2/pr5009.trace.json.gz | Bin 0 -> 25587 bytes .../arms/goal_debug_promo3/manifest.json | 20 + .../arms/goal_debug_promo3/pr5009.cost.json | 107 + eval/dataset/arms/goal_debug_promo3/pr5009.md | 46 + .../goal_debug_promo3/pr5009.trace.json.gz | Bin 0 -> 26044 bytes .../arms/goal_smoke_verify/manifest.json | 20 + .../arms/goal_smoke_verify/pr4870.cost.json | 107 + eval/dataset/arms/goal_smoke_verify/pr4870.md | 40 + .../goal_smoke_verify/pr4870.trace.json.gz | Bin 0 -> 32718 bytes .../baselines/claudecode_opus5/manifest.json | 8 +- .../claudecode_opus5/pr5509.cost.json | 53 + .../baselines/claudecode_opus5/pr5509.md | 117 ++ .../claudecode_opus5/pr5509.trace.json.gz | Bin 0 -> 123340 bytes .../claudecode_opus5/pr5550.cost.json | 42 + .../baselines/claudecode_opus5/pr5550.md | 95 + .../claudecode_opus5/pr5550.trace.json.gz | Bin 0 -> 104735 bytes .../claudecode_opus5/pr5610.cost.json | 75 + .../baselines/claudecode_opus5/pr5610.md | 78 + .../claudecode_opus5/pr5610.trace.json.gz | Bin 0 -> 118376 bytes .../claudecode_opus5/pr5703.cost.json | 67 + .../baselines/claudecode_opus5/pr5703.md | 109 + .../claudecode_opus5/pr5703.trace.json.gz | Bin 0 -> 99697 bytes .../claudecode_opus5/pr5715.cost.json | 50 + .../baselines/claudecode_opus5/pr5715.md | 83 + .../claudecode_opus5/pr5715.trace.json.gz | Bin 0 -> 81319 bytes .../claudecode_opus5/pr5840.cost.json | 70 + .../baselines/claudecode_opus5/pr5840.md | 117 ++ .../claudecode_opus5/pr5840.trace.json.gz | Bin 0 -> 149218 bytes .../claudecode_opus5/pr5863.cost.json | 61 + .../baselines/claudecode_opus5/pr5863.md | 145 ++ .../claudecode_opus5/pr5863.trace.json.gz | Bin 0 -> 150735 bytes .../claudecode_opus5/pr5884.cost.json | 64 + .../baselines/claudecode_opus5/pr5884.md | 81 + .../claudecode_opus5/pr5884.trace.json.gz | Bin 0 -> 148200 bytes .../claudecode_opus5/pr5957.cost.json | 77 + .../baselines/claudecode_opus5/pr5957.md | 118 ++ .../claudecode_opus5/pr5957.trace.json.gz | Bin 0 -> 245801 bytes .../claudecode_opus5/pr5976.cost.json | 66 + .../baselines/claudecode_opus5/pr5976.md | 103 + .../claudecode_opus5/pr5976.trace.json.gz | Bin 0 -> 174344 bytes .../JUDGE_REPORT.md | 36 + .../pr4810.r1.json | 40 + .../pr4810.r2.json | 40 + .../pr4810.r3.json | 40 + .../pr4816.r1.json | 40 + .../pr4816.r2.json | 40 + .../pr4816.r3.json | 40 + .../pr4825.r1.json | 40 + .../pr4825.r2.json | 40 + .../pr4825.r3.json | 40 + .../pr4837.r1.json | 40 + .../pr4837.r2.json | 40 + .../pr4837.r3.json | 40 + .../pr4893.r1.json | 40 + .../pr4893.r2.json | 40 + .../pr4893.r3.json | 40 + .../SMOKE_goal_judge/JUDGE_REPORT.md | 22 + .../judgments/SMOKE_goal_judge/pr4870.r1.json | 40 + .../JUDGE_REPORT.md | 51 + .../goal_composer_wave2_sonnet/pr5509.r1.json | 35 + .../goal_composer_wave2_sonnet/pr5509.r2.json | 37 + .../goal_composer_wave2_sonnet/pr5509.r3.json | 37 + .../goal_composer_wave2_sonnet/pr5550.r1.json | 37 + .../goal_composer_wave2_sonnet/pr5550.r2.json | 37 + .../goal_composer_wave2_sonnet/pr5550.r3.json | 35 + .../goal_composer_wave2_sonnet/pr5610.r1.json | 35 + .../goal_composer_wave2_sonnet/pr5610.r2.json | 37 + .../goal_composer_wave2_sonnet/pr5610.r3.json | 37 + .../goal_composer_wave2_sonnet/pr5703.r1.json | 35 + .../goal_composer_wave2_sonnet/pr5703.r2.json | 37 + .../goal_composer_wave2_sonnet/pr5703.r3.json | 37 + .../goal_composer_wave2_sonnet/pr5715.r1.json | 35 + .../goal_composer_wave2_sonnet/pr5715.r2.json | 37 + .../goal_composer_wave2_sonnet/pr5715.r3.json | 37 + .../goal_composer_wave2_sonnet/pr5840.r1.json | 37 + .../goal_composer_wave2_sonnet/pr5840.r2.json | 37 + .../goal_composer_wave2_sonnet/pr5840.r3.json | 35 + .../goal_composer_wave2_sonnet/pr5863.r1.json | 37 + .../goal_composer_wave2_sonnet/pr5863.r2.json | 37 + .../goal_composer_wave2_sonnet/pr5863.r3.json | 37 + .../goal_composer_wave2_sonnet/pr5884.r1.json | 37 + .../goal_composer_wave2_sonnet/pr5884.r2.json | 35 + .../goal_composer_wave2_sonnet/pr5884.r3.json | 37 + .../goal_composer_wave2_sonnet/pr5957.r1.json | 37 + .../goal_composer_wave2_sonnet/pr5957.r2.json | 35 + .../goal_composer_wave2_sonnet/pr5957.r3.json | 37 + .../goal_composer_wave2_sonnet/pr5976.r1.json | 37 + .../goal_composer_wave2_sonnet/pr5976.r2.json | 37 + .../goal_composer_wave2_sonnet/pr5976.r3.json | 37 + .../goal_ds_wave2_r2_sonnet/JUDGE_REPORT.md | 51 + .../goal_ds_wave2_r2_sonnet/pr5509.r1.json | 35 + .../goal_ds_wave2_r2_sonnet/pr5509.r2.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5509.r3.json | 35 + .../goal_ds_wave2_r2_sonnet/pr5550.r1.json | 35 + .../goal_ds_wave2_r2_sonnet/pr5550.r2.json | 35 + .../goal_ds_wave2_r2_sonnet/pr5550.r3.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5610.r1.json | 35 + .../goal_ds_wave2_r2_sonnet/pr5610.r2.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5610.r3.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5703.r1.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5703.r2.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5703.r3.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5715.r1.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5715.r2.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5715.r3.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5840.r1.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5840.r2.json | 35 + .../goal_ds_wave2_r2_sonnet/pr5840.r3.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5863.r1.json | 35 + .../goal_ds_wave2_r2_sonnet/pr5863.r2.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5863.r3.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5884.r1.json | 35 + .../goal_ds_wave2_r2_sonnet/pr5884.r2.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5884.r3.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5957.r1.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5957.r2.json | 37 + .../goal_ds_wave2_r2_sonnet/pr5957.r3.json | 35 + .../goal_ds_wave2_r2_sonnet/pr5976.r1.json | 35 + .../goal_ds_wave2_r2_sonnet/pr5976.r2.json | 35 + .../goal_ds_wave2_r2_sonnet/pr5976.r3.json | 37 + .../goal_grok45_wave2_sonnet/JUDGE_REPORT.md | 51 + .../goal_grok45_wave2_sonnet/pr5509.r1.json | 37 + .../goal_grok45_wave2_sonnet/pr5509.r2.json | 35 + .../goal_grok45_wave2_sonnet/pr5509.r3.json | 35 + .../goal_grok45_wave2_sonnet/pr5550.r1.json | 35 + .../goal_grok45_wave2_sonnet/pr5550.r2.json | 37 + .../goal_grok45_wave2_sonnet/pr5550.r3.json | 35 + .../goal_grok45_wave2_sonnet/pr5610.r1.json | 37 + .../goal_grok45_wave2_sonnet/pr5610.r2.json | 37 + .../goal_grok45_wave2_sonnet/pr5610.r3.json | 37 + .../goal_grok45_wave2_sonnet/pr5703.r1.json | 35 + .../goal_grok45_wave2_sonnet/pr5703.r2.json | 37 + .../goal_grok45_wave2_sonnet/pr5703.r3.json | 37 + .../goal_grok45_wave2_sonnet/pr5715.r1.json | 37 + .../goal_grok45_wave2_sonnet/pr5715.r2.json | 35 + .../goal_grok45_wave2_sonnet/pr5715.r3.json | 37 + .../goal_grok45_wave2_sonnet/pr5840.r1.json | 37 + .../goal_grok45_wave2_sonnet/pr5840.r2.json | 37 + .../goal_grok45_wave2_sonnet/pr5840.r3.json | 37 + .../goal_grok45_wave2_sonnet/pr5863.r1.json | 37 + .../goal_grok45_wave2_sonnet/pr5863.r2.json | 35 + .../goal_grok45_wave2_sonnet/pr5863.r3.json | 35 + .../goal_grok45_wave2_sonnet/pr5884.r1.json | 37 + .../goal_grok45_wave2_sonnet/pr5884.r2.json | 37 + .../goal_grok45_wave2_sonnet/pr5884.r3.json | 37 + .../goal_grok45_wave2_sonnet/pr5957.r1.json | 37 + .../goal_grok45_wave2_sonnet/pr5957.r2.json | 35 + .../goal_grok45_wave2_sonnet/pr5957.r3.json | 35 + .../goal_grok45_wave2_sonnet/pr5976.r1.json | 37 + .../goal_grok45_wave2_sonnet/pr5976.r2.json | 37 + .../goal_grok45_wave2_sonnet/pr5976.r3.json | 37 + .../judgments/goal_mimo_wave2/JUDGE_REPORT.md | 50 + .../judgments/goal_mimo_wave2/pr5509.r1.json | 40 + .../judgments/goal_mimo_wave2/pr5509.r2.json | 40 + .../judgments/goal_mimo_wave2/pr5509.r3.json | 40 + .../judgments/goal_mimo_wave2/pr5550.r1.json | 40 + .../judgments/goal_mimo_wave2/pr5550.r2.json | 40 + .../judgments/goal_mimo_wave2/pr5610.r1.json | 40 + .../judgments/goal_mimo_wave2/pr5610.r2.json | 40 + .../judgments/goal_mimo_wave2/pr5610.r3.json | 40 + .../judgments/goal_mimo_wave2/pr5703.r1.json | 40 + .../judgments/goal_mimo_wave2/pr5703.r2.json | 40 + .../judgments/goal_mimo_wave2/pr5703.r3.json | 40 + .../judgments/goal_mimo_wave2/pr5715.r1.json | 40 + .../judgments/goal_mimo_wave2/pr5715.r2.json | 40 + .../judgments/goal_mimo_wave2/pr5715.r3.json | 40 + .../judgments/goal_mimo_wave2/pr5840.r1.json | 40 + .../judgments/goal_mimo_wave2/pr5840.r2.json | 40 + .../judgments/goal_mimo_wave2/pr5840.r3.json | 40 + .../judgments/goal_mimo_wave2/pr5863.r1.json | 40 + .../judgments/goal_mimo_wave2/pr5863.r2.json | 40 + .../judgments/goal_mimo_wave2/pr5863.r3.json | 40 + .../judgments/goal_mimo_wave2/pr5884.r1.json | 40 + .../judgments/goal_mimo_wave2/pr5884.r2.json | 40 + .../judgments/goal_mimo_wave2/pr5884.r3.json | 40 + .../judgments/goal_mimo_wave2/pr5957.r1.json | 40 + .../judgments/goal_mimo_wave2/pr5957.r2.json | 40 + .../judgments/goal_mimo_wave2/pr5957.r3.json | 40 + .../judgments/goal_mimo_wave2/pr5976.r1.json | 40 + .../judgments/goal_mimo_wave2/pr5976.r2.json | 40 + .../judgments/goal_mimo_wave2/pr5976.r3.json | 40 + .../goal_mimo_wave2_sonnet/JUDGE_REPORT.md | 48 + .../goal_mimo_wave2_sonnet/pr5509.r1.json | 35 + .../goal_mimo_wave2_sonnet/pr5509.r2.json | 37 + .../goal_mimo_wave2_sonnet/pr5509.r3.json | 37 + .../goal_mimo_wave2_sonnet/pr5550.r1.json | 35 + .../goal_mimo_wave2_sonnet/pr5550.r2.json | 35 + .../goal_mimo_wave2_sonnet/pr5550.r3.json | 37 + .../goal_mimo_wave2_sonnet/pr5610.r1.json | 36 + .../goal_mimo_wave2_sonnet/pr5610.r2.json | 37 + .../goal_mimo_wave2_sonnet/pr5610.r3.json | 35 + .../goal_mimo_wave2_sonnet/pr5703.r3.json | 37 + .../goal_mimo_wave2_sonnet/pr5715.r1.json | 37 + .../goal_mimo_wave2_sonnet/pr5715.r2.json | 35 + .../goal_mimo_wave2_sonnet/pr5715.r3.json | 35 + .../goal_mimo_wave2_sonnet/pr5840.r1.json | 37 + .../goal_mimo_wave2_sonnet/pr5840.r2.json | 37 + .../goal_mimo_wave2_sonnet/pr5840.r3.json | 35 + .../goal_mimo_wave2_sonnet/pr5863.r1.json | 37 + .../goal_mimo_wave2_sonnet/pr5863.r2.json | 35 + .../goal_mimo_wave2_sonnet/pr5863.r3.json | 37 + .../goal_mimo_wave2_sonnet/pr5884.r1.json | 37 + .../goal_mimo_wave2_sonnet/pr5884.r3.json | 37 + .../goal_mimo_wave2_sonnet/pr5957.r1.json | 35 + .../goal_mimo_wave2_sonnet/pr5957.r2.json | 37 + .../goal_mimo_wave2_sonnet/pr5957.r3.json | 35 + .../goal_mimo_wave2_sonnet/pr5976.r1.json | 37 + .../goal_mimo_wave2_sonnet/pr5976.r2.json | 37 + .../goal_mimo_wave2_sonnet/pr5976.r3.json | 35 + .../judgments/goal_v11_train/JUDGE_REPORT.md | 29 + .../judgments/goal_v11_train/pr4804.r1.json | 40 + .../judgments/goal_v11_train/pr4804.r2.json | 40 + .../judgments/goal_v11_train/pr4923.r1.json | 40 + .../judgments/goal_v11_train/pr4923.r2.json | 40 + .../judgments/goal_v11_train/pr4926.r1.json | 40 + .../judgments/goal_v11_train/pr4926.r2.json | 40 + .../judgments/goal_v11_train/pr4977.r1.json | 40 + .../judgments/goal_v11_train/pr4977.r2.json | 40 + .../judgments/goal_v11_val/JUDGE_REPORT.md | 36 + .../judgments/goal_v11_val/pr4810.r1.json | 40 + .../judgments/goal_v11_val/pr4810.r2.json | 40 + .../judgments/goal_v11_val/pr4810.r3.json | 40 + .../judgments/goal_v11_val/pr4816.r1.json | 40 + .../judgments/goal_v11_val/pr4816.r2.json | 40 + .../judgments/goal_v11_val/pr4816.r3.json | 40 + .../judgments/goal_v11_val/pr4825.r1.json | 40 + .../judgments/goal_v11_val/pr4825.r2.json | 40 + .../judgments/goal_v11_val/pr4825.r3.json | 40 + .../judgments/goal_v11_val/pr4837.r1.json | 40 + .../judgments/goal_v11_val/pr4837.r2.json | 40 + .../judgments/goal_v11_val/pr4837.r3.json | 40 + .../judgments/goal_v11_val/pr4893.r1.json | 40 + .../judgments/goal_v11_val/pr4893.r2.json | 40 + .../judgments/goal_v11_val/pr4893.r3.json | 40 + .../judgments/goal_v12_val/JUDGE_REPORT.md | 36 + .../judgments/goal_v12_val/pr4810.r1.json | 40 + .../judgments/goal_v12_val/pr4810.r2.json | 40 + .../judgments/goal_v12_val/pr4810.r3.json | 40 + .../judgments/goal_v12_val/pr4816.r1.json | 40 + .../judgments/goal_v12_val/pr4816.r2.json | 40 + .../judgments/goal_v12_val/pr4816.r3.json | 40 + .../judgments/goal_v12_val/pr4825.r1.json | 40 + .../judgments/goal_v12_val/pr4825.r2.json | 40 + .../judgments/goal_v12_val/pr4825.r3.json | 40 + .../judgments/goal_v12_val/pr4837.r1.json | 40 + .../judgments/goal_v12_val/pr4837.r2.json | 40 + .../judgments/goal_v12_val/pr4837.r3.json | 40 + .../judgments/goal_v12_val/pr4893.r1.json | 40 + .../judgments/goal_v12_val/pr4893.r2.json | 40 + .../judgments/goal_v12_val/pr4893.r3.json | 40 + .../judgments/goal_v13_wave2/JUDGE_REPORT.md | 51 + .../judgments/goal_v13_wave2/pr5509.r1.json | 40 + .../judgments/goal_v13_wave2/pr5509.r2.json | 40 + .../judgments/goal_v13_wave2/pr5509.r3.json | 40 + .../judgments/goal_v13_wave2/pr5550.r1.json | 40 + .../judgments/goal_v13_wave2/pr5550.r2.json | 40 + .../judgments/goal_v13_wave2/pr5550.r3.json | 40 + .../judgments/goal_v13_wave2/pr5610.r1.json | 40 + .../judgments/goal_v13_wave2/pr5610.r2.json | 40 + .../judgments/goal_v13_wave2/pr5610.r3.json | 40 + .../judgments/goal_v13_wave2/pr5703.r1.json | 40 + .../judgments/goal_v13_wave2/pr5703.r2.json | 40 + .../judgments/goal_v13_wave2/pr5703.r3.json | 40 + .../judgments/goal_v13_wave2/pr5715.r1.json | 40 + .../judgments/goal_v13_wave2/pr5715.r2.json | 40 + .../judgments/goal_v13_wave2/pr5715.r3.json | 40 + .../judgments/goal_v13_wave2/pr5840.r1.json | 40 + .../judgments/goal_v13_wave2/pr5840.r2.json | 40 + .../judgments/goal_v13_wave2/pr5840.r3.json | 40 + .../judgments/goal_v13_wave2/pr5863.r1.json | 40 + .../judgments/goal_v13_wave2/pr5863.r2.json | 40 + .../judgments/goal_v13_wave2/pr5863.r3.json | 40 + .../judgments/goal_v13_wave2/pr5884.r1.json | 40 + .../judgments/goal_v13_wave2/pr5884.r2.json | 40 + .../judgments/goal_v13_wave2/pr5884.r3.json | 40 + .../judgments/goal_v13_wave2/pr5957.r1.json | 40 + .../judgments/goal_v13_wave2/pr5957.r2.json | 40 + .../judgments/goal_v13_wave2/pr5957.r3.json | 40 + .../judgments/goal_v13_wave2/pr5976.r1.json | 40 + .../judgments/goal_v13_wave2/pr5976.r2.json | 40 + .../judgments/goal_v13_wave2/pr5976.r3.json | 40 + .../goal_v13_wave2_sonnet/JUDGE_REPORT.md | 51 + .../goal_v13_wave2_sonnet/pr5509.r1.json | 35 + .../goal_v13_wave2_sonnet/pr5509.r2.json | 35 + .../goal_v13_wave2_sonnet/pr5509.r3.json | 37 + .../goal_v13_wave2_sonnet/pr5550.r1.json | 37 + .../goal_v13_wave2_sonnet/pr5550.r2.json | 37 + .../goal_v13_wave2_sonnet/pr5550.r3.json | 35 + .../goal_v13_wave2_sonnet/pr5610.r1.json | 35 + .../goal_v13_wave2_sonnet/pr5610.r2.json | 37 + .../goal_v13_wave2_sonnet/pr5610.r3.json | 37 + .../goal_v13_wave2_sonnet/pr5703.r1.json | 37 + .../goal_v13_wave2_sonnet/pr5703.r2.json | 35 + .../goal_v13_wave2_sonnet/pr5703.r3.json | 35 + .../goal_v13_wave2_sonnet/pr5715.r1.json | 37 + .../goal_v13_wave2_sonnet/pr5715.r2.json | 37 + .../goal_v13_wave2_sonnet/pr5715.r3.json | 37 + .../goal_v13_wave2_sonnet/pr5840.r1.json | 35 + .../goal_v13_wave2_sonnet/pr5840.r2.json | 35 + .../goal_v13_wave2_sonnet/pr5840.r3.json | 37 + .../goal_v13_wave2_sonnet/pr5863.r1.json | 37 + .../goal_v13_wave2_sonnet/pr5863.r2.json | 35 + .../goal_v13_wave2_sonnet/pr5863.r3.json | 37 + .../goal_v13_wave2_sonnet/pr5884.r1.json | 37 + .../goal_v13_wave2_sonnet/pr5884.r2.json | 35 + .../goal_v13_wave2_sonnet/pr5884.r3.json | 37 + .../goal_v13_wave2_sonnet/pr5957.r1.json | 35 + .../goal_v13_wave2_sonnet/pr5957.r2.json | 37 + .../goal_v13_wave2_sonnet/pr5957.r3.json | 37 + .../goal_v13_wave2_sonnet/pr5976.r1.json | 35 + .../goal_v13_wave2_sonnet/pr5976.r2.json | 35 + .../goal_v13_wave2_sonnet/pr5976.r3.json | 35 + .../judgments/goal_v7_train/JUDGE_REPORT.md | 27 + .../judgments/goal_v7_train/pr4804.r1.json | 40 + .../judgments/goal_v7_train/pr4870.r1.json | 40 + .../judgments/goal_v7_train/pr4923.r1.json | 40 + .../judgments/goal_v7_train/pr4926.r1.json | 40 + .../judgments/goal_v7_train/pr4977.r1.json | 40 + .../judgments/goal_v7_train/pr5009.r1.json | 40 + .../goal_v7_train_r2/JUDGE_REPORT.md | 27 + .../judgments/goal_v7_train_r2/pr4804.r1.json | 40 + .../judgments/goal_v7_train_r2/pr4870.r1.json | 40 + .../judgments/goal_v7_train_r2/pr4923.r1.json | 40 + .../judgments/goal_v7_train_r2/pr4926.r1.json | 40 + .../judgments/goal_v7_train_r2/pr4977.r1.json | 40 + .../judgments/goal_v7_train_r2/pr5009.r1.json | 40 + .../goal_v7_train_r3/JUDGE_REPORT.md | 33 + .../judgments/goal_v7_train_r3/pr4804.r1.json | 40 + .../judgments/goal_v7_train_r3/pr4804.r2.json | 40 + .../judgments/goal_v7_train_r3/pr4870.r1.json | 40 + .../judgments/goal_v7_train_r3/pr4870.r2.json | 40 + .../judgments/goal_v7_train_r3/pr4923.r1.json | 40 + .../judgments/goal_v7_train_r3/pr4923.r2.json | 40 + .../judgments/goal_v7_train_r3/pr4926.r1.json | 40 + .../judgments/goal_v7_train_r3/pr4926.r2.json | 40 + .../judgments/goal_v7_train_r3/pr4977.r1.json | 40 + .../judgments/goal_v7_train_r3/pr4977.r2.json | 40 + .../judgments/goal_v7_train_r3/pr5009.r1.json | 40 + .../judgments/goal_v7_train_r3/pr5009.r2.json | 40 + .../goal_v7_train_r4/JUDGE_REPORT.md | 33 + .../judgments/goal_v7_train_r4/pr4804.r1.json | 40 + .../judgments/goal_v7_train_r4/pr4804.r2.json | 40 + .../judgments/goal_v7_train_r4/pr4870.r1.json | 40 + .../judgments/goal_v7_train_r4/pr4870.r2.json | 40 + .../judgments/goal_v7_train_r4/pr4923.r1.json | 40 + .../judgments/goal_v7_train_r4/pr4923.r2.json | 40 + .../judgments/goal_v7_train_r4/pr4926.r1.json | 40 + .../judgments/goal_v7_train_r4/pr4926.r2.json | 40 + .../judgments/goal_v7_train_r4/pr4977.r1.json | 40 + .../judgments/goal_v7_train_r4/pr4977.r2.json | 40 + .../judgments/goal_v7_train_r4/pr5009.r1.json | 40 + .../judgments/goal_v7_train_r4/pr5009.r2.json | 40 + .../goal_v7_train_r7/JUDGE_REPORT.md | 33 + .../judgments/goal_v7_train_r7/pr4804.r1.json | 40 + .../judgments/goal_v7_train_r7/pr4804.r2.json | 40 + .../judgments/goal_v7_train_r7/pr4870.r1.json | 40 + .../judgments/goal_v7_train_r7/pr4870.r2.json | 40 + .../judgments/goal_v7_train_r7/pr4923.r1.json | 40 + .../judgments/goal_v7_train_r7/pr4923.r2.json | 40 + .../judgments/goal_v7_train_r7/pr4926.r1.json | 40 + .../judgments/goal_v7_train_r7/pr4926.r2.json | 40 + .../judgments/goal_v7_train_r7/pr4977.r1.json | 40 + .../judgments/goal_v7_train_r7/pr4977.r2.json | 40 + .../judgments/goal_v7_train_r7/pr5009.r1.json | 40 + .../judgments/goal_v7_train_r7/pr5009.r2.json | 40 + .../judgments/goal_v8_val/JUDGE_REPORT.md | 36 + .../judgments/goal_v8_val/pr4810.r1.json | 40 + .../judgments/goal_v8_val/pr4810.r2.json | 40 + .../judgments/goal_v8_val/pr4810.r3.json | 40 + .../judgments/goal_v8_val/pr4816.r1.json | 40 + .../judgments/goal_v8_val/pr4816.r2.json | 40 + .../judgments/goal_v8_val/pr4816.r3.json | 40 + .../judgments/goal_v8_val/pr4825.r1.json | 40 + .../judgments/goal_v8_val/pr4825.r2.json | 40 + .../judgments/goal_v8_val/pr4825.r3.json | 40 + .../judgments/goal_v8_val/pr4837.r1.json | 40 + .../judgments/goal_v8_val/pr4837.r2.json | 40 + .../judgments/goal_v8_val/pr4837.r3.json | 40 + .../judgments/goal_v8_val/pr4893.r1.json | 40 + .../judgments/goal_v8_val/pr4893.r2.json | 40 + .../judgments/goal_v8_val/pr4893.r3.json | 40 + .../judgments/goal_v9_val/JUDGE_REPORT.md | 36 + .../judgments/goal_v9_val/pr4810.r1.json | 40 + .../judgments/goal_v9_val/pr4810.r2.json | 40 + .../judgments/goal_v9_val/pr4810.r3.json | 40 + .../judgments/goal_v9_val/pr4816.r1.json | 40 + .../judgments/goal_v9_val/pr4816.r2.json | 40 + .../judgments/goal_v9_val/pr4816.r3.json | 40 + .../judgments/goal_v9_val/pr4825.r1.json | 40 + .../judgments/goal_v9_val/pr4825.r2.json | 40 + .../judgments/goal_v9_val/pr4825.r3.json | 40 + .../judgments/goal_v9_val/pr4837.r1.json | 40 + .../judgments/goal_v9_val/pr4837.r2.json | 40 + .../judgments/goal_v9_val/pr4837.r3.json | 40 + .../judgments/goal_v9_val/pr4893.r1.json | 40 + .../judgments/goal_v9_val/pr4893.r2.json | 40 + .../judgments/goal_v9_val/pr4893.r3.json | 40 + eval/dataset/results/aggregate_results.py | 133 ++ eval/dataset/results/model_comparison.csv | 10 + eval/dataset/results/model_comparison.json | 1838 +++++++++++++++++ eval/dataset/results/model_comparison.md | 109 + eval/dataset/snapshots/pr5509.json | 7 + eval/dataset/snapshots/pr5550.json | 7 + eval/dataset/snapshots/pr5610.json | 7 + eval/dataset/snapshots/pr5703.json | 7 + eval/dataset/snapshots/pr5715.json | 7 + eval/dataset/snapshots/pr5840.json | 7 + eval/dataset/snapshots/pr5863.json | 7 + eval/dataset/snapshots/pr5884.json | 7 + eval/dataset/snapshots/pr5957.json | 7 + eval/dataset/snapshots/pr5976.json | 7 + 795 files changed, 36031 insertions(+), 5 deletions(-) create mode 100644 eval/dataset/arms/copilot_v11_deep_r1/manifest.json create mode 100644 eval/dataset/arms/copilot_v11_deep_r1/pr4804.cost.json create mode 100644 eval/dataset/arms/copilot_v11_deep_r1/pr4804.md create mode 100644 eval/dataset/arms/copilot_v11_deep_r1/pr4804.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v11_deep_r1/pr4923.cost.json create mode 100644 eval/dataset/arms/copilot_v11_deep_r1/pr4923.md create mode 100644 eval/dataset/arms/copilot_v11_deep_r1/pr4923.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v11_deep_r1/pr4926.cost.json create mode 100644 eval/dataset/arms/copilot_v11_deep_r1/pr4926.md create mode 100644 eval/dataset/arms/copilot_v11_deep_r1/pr4926.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v11_deep_r1/pr4977.cost.json create mode 100644 eval/dataset/arms/copilot_v11_deep_r1/pr4977.md create mode 100644 eval/dataset/arms/copilot_v11_deep_r1/pr4977.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v11_deep_r2/manifest.json create mode 100644 eval/dataset/arms/copilot_v11_deep_r2/pr4804.cost.json create mode 100644 eval/dataset/arms/copilot_v11_deep_r2/pr4804.md create mode 100644 eval/dataset/arms/copilot_v11_deep_r2/pr4804.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v11_deep_r2/pr4923.cost.json create mode 100644 eval/dataset/arms/copilot_v11_deep_r2/pr4923.md create mode 100644 eval/dataset/arms/copilot_v11_deep_r2/pr4923.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v11_deep_r2/pr4926.cost.json create mode 100644 eval/dataset/arms/copilot_v11_deep_r2/pr4926.md create mode 100644 eval/dataset/arms/copilot_v11_deep_r2/pr4926.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v11_deep_r2/pr4977.cost.json create mode 100644 eval/dataset/arms/copilot_v11_deep_r2/pr4977.md create mode 100644 eval/dataset/arms/copilot_v11_deep_r2/pr4977.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v11_deep_val/manifest.json create mode 100644 eval/dataset/arms/copilot_v11_deep_val/pr4810.cost.json create mode 100644 eval/dataset/arms/copilot_v11_deep_val/pr4810.md create mode 100644 eval/dataset/arms/copilot_v11_deep_val/pr4810.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v11_deep_val/pr4816.cost.json create mode 100644 eval/dataset/arms/copilot_v11_deep_val/pr4816.md create mode 100644 eval/dataset/arms/copilot_v11_deep_val/pr4816.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v11_deep_val/pr4825.cost.json create mode 100644 eval/dataset/arms/copilot_v11_deep_val/pr4825.md create mode 100644 eval/dataset/arms/copilot_v11_deep_val/pr4825.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v11_deep_val/pr4837.cost.json create mode 100644 eval/dataset/arms/copilot_v11_deep_val/pr4837.md create mode 100644 eval/dataset/arms/copilot_v11_deep_val/pr4837.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v11_deep_val/pr4893.cost.json create mode 100644 eval/dataset/arms/copilot_v11_deep_val/pr4893.md create mode 100644 eval/dataset/arms/copilot_v11_deep_val/pr4893.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/manifest.json create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/pr4810.cost.json create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/pr4810.md create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/pr4810.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/pr4816.cost.json create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/pr4816.md create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/pr4816.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/pr4825.cost.json create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/pr4825.md create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/pr4825.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/pr4837.cost.json create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/pr4837.md create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/pr4837.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/pr4893.cost.json create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/pr4893.md create mode 100644 eval/dataset/arms/copilot_v12_hybrid_val/pr4893.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/manifest.json create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5509.cost.json create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5509.md create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5509.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5550.cost.json create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5550.md create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5550.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5610.cost.json create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5610.md create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5610.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5703.cost.json create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5703.md create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5703.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5715.cost.json create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5715.md create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5715.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5840.cost.json create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5840.md create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5840.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5863.cost.json create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5863.md create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5863.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5884.cost.json create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5884.md create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5884.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5957.cost.json create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5957.md create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5957.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5976.cost.json create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5976.md create mode 100644 eval/dataset/arms/copilot_v13_ds_wave2_r2/pr5976.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/manifest.json create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5509.cost.json create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5509.md create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5509.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5550.cost.json create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5550.md create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5550.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5610.cost.json create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5610.md create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5610.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5703.cost.json create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5703.md create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5703.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5715.cost.json create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5715.md create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5715.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5840.cost.json create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5840.md create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5840.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5863.cost.json create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5863.md create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5863.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5884.cost.json create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5884.md create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5884.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5957.cost.json create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5957.md create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5957.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5976.cost.json create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5976.md create mode 100644 eval/dataset/arms/copilot_v13_mimo_wave2/pr5976.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_proof_test/manifest.json create mode 100644 eval/dataset/arms/copilot_v13_proof_test/pr4762.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_proof_test/pr4777.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_proof_test/pr4834.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_proof_test/pr4849.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_proof_test/pr4954.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_proof_val/manifest.json create mode 100644 eval/dataset/arms/copilot_v13_proof_val/pr4810.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_proof_val/pr4816.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_proof_val/pr4825.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_proof_val/pr4837.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_proof_val/pr4893.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_wave2/manifest.json create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5509.cost.json create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5509.md create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5509.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5550.cost.json create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5550.md create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5550.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5610.cost.json create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5610.md create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5610.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5703.cost.json create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5703.md create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5703.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5715.cost.json create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5715.md create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5715.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5840.cost.json create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5840.md create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5840.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5863.cost.json create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5863.md create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5863.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5884.cost.json create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5884.md create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5884.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5957.cost.json create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5957.md create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5957.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5976.cost.json create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5976.md create mode 100644 eval/dataset/arms/copilot_v13_wave2/pr5976.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/manifest.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr4804.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr4804.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr4804.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr4870.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr4870.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr4870.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr4923.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr4923.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr4923.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr4926.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr4926.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr4926.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr4977.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr4977.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr4977.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr5009.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr5009.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r1/pr5009.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/manifest.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr4804.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr4804.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr4804.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr4870.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr4870.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr4870.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr4923.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr4923.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr4923.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr4926.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr4926.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr4926.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr4977.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr4977.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr4977.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr5009.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr5009.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r2/pr5009.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/manifest.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr4804.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr4804.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr4804.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr4870.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr4870.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr4870.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr4923.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr4923.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr4923.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr4926.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr4926.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr4926.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr4977.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr4977.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr4977.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr5009.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr5009.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r3/pr5009.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/manifest.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr4804.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr4804.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr4804.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr4870.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr4870.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr4870.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr4923.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr4923.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr4923.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr4926.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr4926.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr4926.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr4977.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr4977.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr4977.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr5009.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr5009.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r4/pr5009.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/manifest.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr4804.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr4804.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr4804.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr4870.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr4870.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr4870.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr4923.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr4923.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr4923.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr4926.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr4926.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr4926.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr4977.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr4977.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr4977.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr5009.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr5009.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r5/pr5009.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/manifest.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr4804.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr4804.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr4804.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr4870.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr4870.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr4870.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr4923.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr4923.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr4923.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr4926.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr4926.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr4926.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr4977.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr4977.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr4977.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr5009.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr5009.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r6/pr5009.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/manifest.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4804.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4804.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4804.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4810.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4810.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4810.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4816.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4825.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4837.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4870.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4870.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4870.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4893.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4923.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4923.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4923.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4926.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4926.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4926.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4977.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4977.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr4977.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr5009.cost.json create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr5009.md create mode 100644 eval/dataset/arms/copilot_v7_goal_r7/pr5009.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v8_official_r1/manifest.json create mode 100644 eval/dataset/arms/copilot_v8_official_r1/pr4810.cost.json create mode 100644 eval/dataset/arms/copilot_v8_official_r1/pr4810.md create mode 100644 eval/dataset/arms/copilot_v8_official_r1/pr4810.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v8_official_r1/pr4816.cost.json create mode 100644 eval/dataset/arms/copilot_v8_official_r1/pr4816.md create mode 100644 eval/dataset/arms/copilot_v8_official_r1/pr4816.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v8_official_r1/pr4825.cost.json create mode 100644 eval/dataset/arms/copilot_v8_official_r1/pr4825.md create mode 100644 eval/dataset/arms/copilot_v8_official_r1/pr4825.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v8_official_r1/pr4837.cost.json create mode 100644 eval/dataset/arms/copilot_v8_official_r1/pr4837.md create mode 100644 eval/dataset/arms/copilot_v8_official_r1/pr4837.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v8_official_r1/pr4893.cost.json create mode 100644 eval/dataset/arms/copilot_v8_official_r1/pr4893.md create mode 100644 eval/dataset/arms/copilot_v8_official_r1/pr4893.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v9_official_r1/manifest.json create mode 100644 eval/dataset/arms/copilot_v9_official_r1/pr4810.cost.json create mode 100644 eval/dataset/arms/copilot_v9_official_r1/pr4810.md create mode 100644 eval/dataset/arms/copilot_v9_official_r1/pr4810.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v9_official_r1/pr4816.cost.json create mode 100644 eval/dataset/arms/copilot_v9_official_r1/pr4816.md create mode 100644 eval/dataset/arms/copilot_v9_official_r1/pr4816.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v9_official_r1/pr4825.cost.json create mode 100644 eval/dataset/arms/copilot_v9_official_r1/pr4825.md create mode 100644 eval/dataset/arms/copilot_v9_official_r1/pr4825.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v9_official_r1/pr4837.cost.json create mode 100644 eval/dataset/arms/copilot_v9_official_r1/pr4837.md create mode 100644 eval/dataset/arms/copilot_v9_official_r1/pr4837.trace.json.gz create mode 100644 eval/dataset/arms/copilot_v9_official_r1/pr4893.cost.json create mode 100644 eval/dataset/arms/copilot_v9_official_r1/pr4893.md create mode 100644 eval/dataset/arms/copilot_v9_official_r1/pr4893.trace.json.gz create mode 100644 eval/dataset/arms/cursor_composer25/manifest.json create mode 100644 eval/dataset/arms/cursor_composer25/pr5509.cost.json create mode 100644 eval/dataset/arms/cursor_composer25/pr5509.md create mode 100644 eval/dataset/arms/cursor_composer25/pr5550.cost.json create mode 100644 eval/dataset/arms/cursor_composer25/pr5550.md create mode 100644 eval/dataset/arms/cursor_composer25/pr5610.cost.json create mode 100644 eval/dataset/arms/cursor_composer25/pr5610.md create mode 100644 eval/dataset/arms/cursor_composer25/pr5703.cost.json create mode 100644 eval/dataset/arms/cursor_composer25/pr5703.md create mode 100644 eval/dataset/arms/cursor_composer25/pr5715.cost.json create mode 100644 eval/dataset/arms/cursor_composer25/pr5715.md create mode 100644 eval/dataset/arms/cursor_composer25/pr5840.cost.json create mode 100644 eval/dataset/arms/cursor_composer25/pr5840.md create mode 100644 eval/dataset/arms/cursor_composer25/pr5863.cost.json create mode 100644 eval/dataset/arms/cursor_composer25/pr5863.md create mode 100644 eval/dataset/arms/cursor_composer25/pr5884.cost.json create mode 100644 eval/dataset/arms/cursor_composer25/pr5884.md create mode 100644 eval/dataset/arms/cursor_composer25/pr5957.cost.json create mode 100644 eval/dataset/arms/cursor_composer25/pr5957.md create mode 100644 eval/dataset/arms/cursor_composer25/pr5976.cost.json create mode 100644 eval/dataset/arms/cursor_composer25/pr5976.md create mode 100644 eval/dataset/arms/cursor_grok45/manifest.json create mode 100644 eval/dataset/arms/cursor_grok45/pr5509.cost.json create mode 100644 eval/dataset/arms/cursor_grok45/pr5509.md create mode 100644 eval/dataset/arms/cursor_grok45/pr5550.cost.json create mode 100644 eval/dataset/arms/cursor_grok45/pr5550.md create mode 100644 eval/dataset/arms/cursor_grok45/pr5610.cost.json create mode 100644 eval/dataset/arms/cursor_grok45/pr5610.md create mode 100644 eval/dataset/arms/cursor_grok45/pr5703.cost.json create mode 100644 eval/dataset/arms/cursor_grok45/pr5703.md create mode 100644 eval/dataset/arms/cursor_grok45/pr5715.cost.json create mode 100644 eval/dataset/arms/cursor_grok45/pr5715.md create mode 100644 eval/dataset/arms/cursor_grok45/pr5840.cost.json create mode 100644 eval/dataset/arms/cursor_grok45/pr5840.md create mode 100644 eval/dataset/arms/cursor_grok45/pr5863.cost.json create mode 100644 eval/dataset/arms/cursor_grok45/pr5863.md create mode 100644 eval/dataset/arms/cursor_grok45/pr5884.cost.json create mode 100644 eval/dataset/arms/cursor_grok45/pr5884.md create mode 100644 eval/dataset/arms/cursor_grok45/pr5957.cost.json create mode 100644 eval/dataset/arms/cursor_grok45/pr5957.md create mode 100644 eval/dataset/arms/cursor_grok45/pr5976.cost.json create mode 100644 eval/dataset/arms/cursor_grok45/pr5976.md create mode 100644 eval/dataset/arms/goal_debug_promo/manifest.json create mode 100644 eval/dataset/arms/goal_debug_promo/pr4923.cost.json create mode 100644 eval/dataset/arms/goal_debug_promo/pr4923.md create mode 100644 eval/dataset/arms/goal_debug_promo/pr4923.trace.json.gz create mode 100644 eval/dataset/arms/goal_debug_promo2/manifest.json create mode 100644 eval/dataset/arms/goal_debug_promo2/pr5009.cost.json create mode 100644 eval/dataset/arms/goal_debug_promo2/pr5009.md create mode 100644 eval/dataset/arms/goal_debug_promo2/pr5009.trace.json.gz create mode 100644 eval/dataset/arms/goal_debug_promo3/manifest.json create mode 100644 eval/dataset/arms/goal_debug_promo3/pr5009.cost.json create mode 100644 eval/dataset/arms/goal_debug_promo3/pr5009.md create mode 100644 eval/dataset/arms/goal_debug_promo3/pr5009.trace.json.gz create mode 100644 eval/dataset/arms/goal_smoke_verify/manifest.json create mode 100644 eval/dataset/arms/goal_smoke_verify/pr4870.cost.json create mode 100644 eval/dataset/arms/goal_smoke_verify/pr4870.md create mode 100644 eval/dataset/arms/goal_smoke_verify/pr4870.trace.json.gz create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5509.cost.json create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5509.md create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5509.trace.json.gz create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5550.cost.json create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5550.md create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5550.trace.json.gz create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5610.cost.json create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5610.md create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5610.trace.json.gz create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5703.cost.json create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5703.md create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5703.trace.json.gz create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5715.cost.json create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5715.md create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5715.trace.json.gz create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5840.cost.json create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5840.md create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5840.trace.json.gz create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5863.cost.json create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5863.md create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5863.trace.json.gz create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5884.cost.json create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5884.md create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5884.trace.json.gz create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5957.cost.json create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5957.md create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5957.trace.json.gz create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5976.cost.json create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5976.md create mode 100644 eval/dataset/baselines/claudecode_opus5/pr5976.trace.json.gz create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/pr4810.r1.json create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/pr4810.r2.json create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/pr4810.r3.json create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/pr4816.r1.json create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/pr4816.r2.json create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/pr4816.r3.json create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/pr4825.r1.json create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/pr4825.r2.json create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/pr4825.r3.json create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/pr4837.r1.json create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/pr4837.r2.json create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/pr4837.r3.json create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/pr4893.r1.json create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/pr4893.r2.json create mode 100644 eval/dataset/judgments/INVALID_apierror_goal_v7_val/pr4893.r3.json create mode 100644 eval/dataset/judgments/SMOKE_goal_judge/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/SMOKE_goal_judge/pr4870.r1.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5509.r1.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5509.r2.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5509.r3.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5550.r1.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5550.r2.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5550.r3.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5610.r1.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5610.r2.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5610.r3.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5703.r1.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5703.r2.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5703.r3.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5715.r1.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5715.r2.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5715.r3.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5840.r1.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5840.r2.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5840.r3.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5863.r1.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5863.r2.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5863.r3.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5884.r1.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5884.r2.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5884.r3.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5957.r1.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5957.r2.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5957.r3.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5976.r1.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5976.r2.json create mode 100644 eval/dataset/judgments/goal_composer_wave2_sonnet/pr5976.r3.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5509.r1.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5509.r2.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5509.r3.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5550.r1.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5550.r2.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5550.r3.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5610.r1.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5610.r2.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5610.r3.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5703.r1.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5703.r2.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5703.r3.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5715.r1.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5715.r2.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5715.r3.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5840.r1.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5840.r2.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5840.r3.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5863.r1.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5863.r2.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5863.r3.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5884.r1.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5884.r2.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5884.r3.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5957.r1.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5957.r2.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5957.r3.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5976.r1.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5976.r2.json create mode 100644 eval/dataset/judgments/goal_ds_wave2_r2_sonnet/pr5976.r3.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5509.r1.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5509.r2.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5509.r3.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5550.r1.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5550.r2.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5550.r3.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5610.r1.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5610.r2.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5610.r3.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5703.r1.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5703.r2.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5703.r3.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5715.r1.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5715.r2.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5715.r3.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5840.r1.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5840.r2.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5840.r3.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5863.r1.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5863.r2.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5863.r3.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5884.r1.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5884.r2.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5884.r3.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5957.r1.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5957.r2.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5957.r3.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5976.r1.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5976.r2.json create mode 100644 eval/dataset/judgments/goal_grok45_wave2_sonnet/pr5976.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5509.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5509.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5509.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5550.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5550.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5610.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5610.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5610.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5703.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5703.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5703.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5715.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5715.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5715.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5840.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5840.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5840.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5863.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5863.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5863.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5884.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5884.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5884.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5957.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5957.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5957.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5976.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5976.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2/pr5976.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5509.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5509.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5509.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5550.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5550.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5550.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5610.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5610.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5610.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5703.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5715.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5715.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5715.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5840.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5840.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5840.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5863.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5863.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5863.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5884.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5884.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5957.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5957.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5957.r3.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5976.r1.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5976.r2.json create mode 100644 eval/dataset/judgments/goal_mimo_wave2_sonnet/pr5976.r3.json create mode 100644 eval/dataset/judgments/goal_v11_train/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_v11_train/pr4804.r1.json create mode 100644 eval/dataset/judgments/goal_v11_train/pr4804.r2.json create mode 100644 eval/dataset/judgments/goal_v11_train/pr4923.r1.json create mode 100644 eval/dataset/judgments/goal_v11_train/pr4923.r2.json create mode 100644 eval/dataset/judgments/goal_v11_train/pr4926.r1.json create mode 100644 eval/dataset/judgments/goal_v11_train/pr4926.r2.json create mode 100644 eval/dataset/judgments/goal_v11_train/pr4977.r1.json create mode 100644 eval/dataset/judgments/goal_v11_train/pr4977.r2.json create mode 100644 eval/dataset/judgments/goal_v11_val/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_v11_val/pr4810.r1.json create mode 100644 eval/dataset/judgments/goal_v11_val/pr4810.r2.json create mode 100644 eval/dataset/judgments/goal_v11_val/pr4810.r3.json create mode 100644 eval/dataset/judgments/goal_v11_val/pr4816.r1.json create mode 100644 eval/dataset/judgments/goal_v11_val/pr4816.r2.json create mode 100644 eval/dataset/judgments/goal_v11_val/pr4816.r3.json create mode 100644 eval/dataset/judgments/goal_v11_val/pr4825.r1.json create mode 100644 eval/dataset/judgments/goal_v11_val/pr4825.r2.json create mode 100644 eval/dataset/judgments/goal_v11_val/pr4825.r3.json create mode 100644 eval/dataset/judgments/goal_v11_val/pr4837.r1.json create mode 100644 eval/dataset/judgments/goal_v11_val/pr4837.r2.json create mode 100644 eval/dataset/judgments/goal_v11_val/pr4837.r3.json create mode 100644 eval/dataset/judgments/goal_v11_val/pr4893.r1.json create mode 100644 eval/dataset/judgments/goal_v11_val/pr4893.r2.json create mode 100644 eval/dataset/judgments/goal_v11_val/pr4893.r3.json create mode 100644 eval/dataset/judgments/goal_v12_val/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_v12_val/pr4810.r1.json create mode 100644 eval/dataset/judgments/goal_v12_val/pr4810.r2.json create mode 100644 eval/dataset/judgments/goal_v12_val/pr4810.r3.json create mode 100644 eval/dataset/judgments/goal_v12_val/pr4816.r1.json create mode 100644 eval/dataset/judgments/goal_v12_val/pr4816.r2.json create mode 100644 eval/dataset/judgments/goal_v12_val/pr4816.r3.json create mode 100644 eval/dataset/judgments/goal_v12_val/pr4825.r1.json create mode 100644 eval/dataset/judgments/goal_v12_val/pr4825.r2.json create mode 100644 eval/dataset/judgments/goal_v12_val/pr4825.r3.json create mode 100644 eval/dataset/judgments/goal_v12_val/pr4837.r1.json create mode 100644 eval/dataset/judgments/goal_v12_val/pr4837.r2.json create mode 100644 eval/dataset/judgments/goal_v12_val/pr4837.r3.json create mode 100644 eval/dataset/judgments/goal_v12_val/pr4893.r1.json create mode 100644 eval/dataset/judgments/goal_v12_val/pr4893.r2.json create mode 100644 eval/dataset/judgments/goal_v12_val/pr4893.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5509.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5509.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5509.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5550.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5550.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5550.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5610.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5610.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5610.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5703.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5703.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5703.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5715.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5715.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5715.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5840.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5840.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5840.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5863.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5863.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5863.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5884.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5884.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5884.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5957.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5957.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5957.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5976.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5976.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2/pr5976.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5509.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5509.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5509.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5550.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5550.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5550.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5610.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5610.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5610.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5703.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5703.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5703.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5715.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5715.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5715.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5840.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5840.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5840.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5863.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5863.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5863.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5884.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5884.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5884.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5957.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5957.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5957.r3.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5976.r1.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5976.r2.json create mode 100644 eval/dataset/judgments/goal_v13_wave2_sonnet/pr5976.r3.json create mode 100644 eval/dataset/judgments/goal_v7_train/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_v7_train/pr4804.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train/pr4870.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train/pr4923.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train/pr4926.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train/pr4977.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train/pr5009.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r2/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_v7_train_r2/pr4804.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r2/pr4870.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r2/pr4923.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r2/pr4926.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r2/pr4977.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r2/pr5009.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r3/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_v7_train_r3/pr4804.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r3/pr4804.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r3/pr4870.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r3/pr4870.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r3/pr4923.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r3/pr4923.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r3/pr4926.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r3/pr4926.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r3/pr4977.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r3/pr4977.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r3/pr5009.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r3/pr5009.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r4/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_v7_train_r4/pr4804.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r4/pr4804.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r4/pr4870.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r4/pr4870.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r4/pr4923.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r4/pr4923.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r4/pr4926.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r4/pr4926.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r4/pr4977.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r4/pr4977.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r4/pr5009.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r4/pr5009.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r7/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_v7_train_r7/pr4804.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r7/pr4804.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r7/pr4870.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r7/pr4870.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r7/pr4923.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r7/pr4923.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r7/pr4926.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r7/pr4926.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r7/pr4977.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r7/pr4977.r2.json create mode 100644 eval/dataset/judgments/goal_v7_train_r7/pr5009.r1.json create mode 100644 eval/dataset/judgments/goal_v7_train_r7/pr5009.r2.json create mode 100644 eval/dataset/judgments/goal_v8_val/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_v8_val/pr4810.r1.json create mode 100644 eval/dataset/judgments/goal_v8_val/pr4810.r2.json create mode 100644 eval/dataset/judgments/goal_v8_val/pr4810.r3.json create mode 100644 eval/dataset/judgments/goal_v8_val/pr4816.r1.json create mode 100644 eval/dataset/judgments/goal_v8_val/pr4816.r2.json create mode 100644 eval/dataset/judgments/goal_v8_val/pr4816.r3.json create mode 100644 eval/dataset/judgments/goal_v8_val/pr4825.r1.json create mode 100644 eval/dataset/judgments/goal_v8_val/pr4825.r2.json create mode 100644 eval/dataset/judgments/goal_v8_val/pr4825.r3.json create mode 100644 eval/dataset/judgments/goal_v8_val/pr4837.r1.json create mode 100644 eval/dataset/judgments/goal_v8_val/pr4837.r2.json create mode 100644 eval/dataset/judgments/goal_v8_val/pr4837.r3.json create mode 100644 eval/dataset/judgments/goal_v8_val/pr4893.r1.json create mode 100644 eval/dataset/judgments/goal_v8_val/pr4893.r2.json create mode 100644 eval/dataset/judgments/goal_v8_val/pr4893.r3.json create mode 100644 eval/dataset/judgments/goal_v9_val/JUDGE_REPORT.md create mode 100644 eval/dataset/judgments/goal_v9_val/pr4810.r1.json create mode 100644 eval/dataset/judgments/goal_v9_val/pr4810.r2.json create mode 100644 eval/dataset/judgments/goal_v9_val/pr4810.r3.json create mode 100644 eval/dataset/judgments/goal_v9_val/pr4816.r1.json create mode 100644 eval/dataset/judgments/goal_v9_val/pr4816.r2.json create mode 100644 eval/dataset/judgments/goal_v9_val/pr4816.r3.json create mode 100644 eval/dataset/judgments/goal_v9_val/pr4825.r1.json create mode 100644 eval/dataset/judgments/goal_v9_val/pr4825.r2.json create mode 100644 eval/dataset/judgments/goal_v9_val/pr4825.r3.json create mode 100644 eval/dataset/judgments/goal_v9_val/pr4837.r1.json create mode 100644 eval/dataset/judgments/goal_v9_val/pr4837.r2.json create mode 100644 eval/dataset/judgments/goal_v9_val/pr4837.r3.json create mode 100644 eval/dataset/judgments/goal_v9_val/pr4893.r1.json create mode 100644 eval/dataset/judgments/goal_v9_val/pr4893.r2.json create mode 100644 eval/dataset/judgments/goal_v9_val/pr4893.r3.json create mode 100644 eval/dataset/results/aggregate_results.py create mode 100644 eval/dataset/results/model_comparison.csv create mode 100644 eval/dataset/results/model_comparison.json create mode 100644 eval/dataset/results/model_comparison.md create mode 100644 eval/dataset/snapshots/pr5509.json create mode 100644 eval/dataset/snapshots/pr5550.json create mode 100644 eval/dataset/snapshots/pr5610.json create mode 100644 eval/dataset/snapshots/pr5703.json create mode 100644 eval/dataset/snapshots/pr5715.json create mode 100644 eval/dataset/snapshots/pr5840.json create mode 100644 eval/dataset/snapshots/pr5863.json create mode 100644 eval/dataset/snapshots/pr5884.json create mode 100644 eval/dataset/snapshots/pr5957.json create mode 100644 eval/dataset/snapshots/pr5976.json diff --git a/eval/dataset/arms/copilot_v11_deep_r1/manifest.json b/eval/dataset/arms/copilot_v11_deep_r1/manifest.json new file mode 100644 index 000000000..f019f0e76 --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_r1/manifest.json @@ -0,0 +1,20 @@ +{ + "arm": "copilot_v11_deep_r1", + "engine": "infermatrix-copilot CLI (shipped pipeline, pr-review@6 adaptive-depth ensemble; issue-answer dry-run)", + "llm": "per .env (DeepSeek-routed)", + "dataset": "vllm_omni_dataset.yaml", + "splits": [ + "train" + ], + "kinds": [ + "pr_review" + ], + "n_items": 1, + "trace": { + "AGENT_TRACE": "(default)", + "AGENT_TRACE_IO": "(default)", + "AGENT_TRACE_IO_FULL": "(default)" + }, + "moa_when": "off", + "pr_context_mode": "(default)" +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v11_deep_r1/pr4804.cost.json b/eval/dataset/arms/copilot_v11_deep_r1/pr4804.cost.json new file mode 100644 index 000000000..85b2cb14c --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_r1/pr4804.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 1136.8, + "split": "train", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v11_deep_r1/runs/pr4804/run-20260813-120442-07e181", + "recorded_at": "2026-08-13T12:23:37", + "attempts": [ + { + "invocation_id": "inv-838a12a5", + "outcome": "run:run-20260813-120442-07e181", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-120442-07e181", + "llm_calls": 82, + "input_tokens": 1203030, + "output_tokens": 130757, + "usd": 0.766696, + "source": "spans" + } + ], + "attempt_usd_total": 0.766696, + "llm_calls": 82, + "input_tokens": 1203030, + "output_tokens": 130757, + "usd": 0.766696, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-120442-07e181", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.7667, + "minutes": 18.91, + "ci_minutes": 0, + "input_tokens": 1203030, + "output_tokens": 130757, + "tool_calls": 117, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.8222, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 82, + "by_role": { + "agent": { + "usd": 0.734896, + "input_tokens": 1150354, + "output_tokens": 114809, + "calls": 80 + }, + "reducer": { + "usd": 0.0318, + "input_tokens": 52676, + "output_tokens": 15948, + "calls": 2 + } + }, + "cache_read_tokens": 11038720, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 4.37, + "gate": 1.96, + "review": 1128.36, + "report": 0.06 + }, + "time_to_first_result_s": 1134.74 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4804.trace.json.gz", + "trace_bytes": 55624 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v11_deep_r1/pr4804.md b/eval/dataset/arms/copilot_v11_deep_r1/pr4804.md new file mode 100644 index 000000000..7878b6bde --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_r1/pr4804.md @@ -0,0 +1,38 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4804, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head 5d3e53170aae) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 1 finding(s) below | +| Security | no finding reported | +| Docs / comments | 2 finding(s) below | +| Behavior / compatibility | 1 finding(s) below | +| Correctness | 1 finding(s) below | + +**Validated:** +- [sweep] read modeling_moss_tts_codec.py fully: stream session acquire/release/reset symmetry checked; on_requests_finished is wired in gpu_generation_model_runner._update_request_states. +- [sweep] verified slot starvation branch: _decode_streaming_batch buffers codes then _finish_stream_request(..., None) pops _stream_pending_codes — buffered audio is discarded on finished-before-slot. +- [sweep] read stage_input_processors/moss_tts.py: raw path keys pending frames by external_req_id; mixin cleanup_finished_request pops both internal/external keys. +- [sweep] grep put_req_chunk consumers: raw_async_chunk reads the mixin's _put_req_chunk (incremented after send_chunk), so initial_chunk_frames threshold logic is consistent. +- [sweep] read modeling_moss_tts_talker.py preprocess: `max_new_frames = info_dict.get("max_new_frames", [-1])[0]` drops the old int/None/tuple normalization. +- [sweep] compared audio_tokenizer.py header vs new audio_tokenizer_v2.py/configuration_moss_audio_tokenizer_v2.py: new files lack the required 'Vendored from @ ' header. +- [sweep] grep qwen3_tts_request_seed: no remaining repo consumers; tts_local_seed only consumed in gpu_model_runner._explicit_talker_seed and produced in serving_speech + tests — seed key rename appears self-consistent. +- [sweep] grep AudioResampler usage: patch.py's new resampler call mirrors existing qwen3_tts/voxcpm2 patterns (AudioResampler(target_sr=...).resample(...)). + +`vllm_omni/v1/worker/gpu_generation_model_runner.py:799` [minor] — `model_kwargs["seq_token_counts"] = [int(num_tokens_padded)]` is assigned unconditionally in `GPUGenerationModelRunner._dummy_run` (vllm_omni/worker/gpu_generation_model_runner.py:799), so every generation-model dummy/profile run receives this kwarg. Please confirm it is safe/ignored by all generation models with explicit forward signatures, or gate it to the models that consume it. (evidence: [sweep] read gpu_generation_model_runner._dummy_run: new `seq_token_counts=[num_tokens_padded]` is unconditional for every generation model dummy/profile run; blast radius not checked for models with explicit forward signatures.) + +`buildkite/vllm-omni-npu-ci:0` [minor] — `buildkite/vllm-omni-npu-ci` is failing while the other reported checks (build 3.11/3.12, pre-commit, DCO, `buildkite/vllm-omni`, `buildkite/vllm-omni-amd-ci`, `buildkite/vllm-omni-intel-ci`, docs) pass. The check result alone does not attribute the failure to this diff, so please confirm whether the NPU lane failure pre-exists on main or is introduced by this PR. (evidence: [sweep] CI read: buildkite/vllm-omni-npu-ci is the only failing lane; attribution to this diff unverified (main may also be red).) + +`vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_codec.py:?` [minor] — When `session.acquire()` returns None in `_decode_streaming_batch`, the new frame is buffered via `_append_stream_pending`; if `finished=True`, the code then calls `_finish_stream_request(request_id, session, None)`, which pops `_stream_pending_codes[request_id]` and drops the buffered tail without decoding it (a warning is logged, but the stream is still truncated). The shipped `moss_tts_local.yaml` pairs `codec_stream_slots: 8` with stage-1 `max_num_seqs: 8`, so the default config should not exhaust slots; lower `codec_stream_slots` or stage-1 concurrency above slots will truncate a finished stream. Enforce `codec_stream_slots >= stage-1 max_num_seqs` and fail loudly on exhaustion, or decode the pending sequence once on this finish path instead of dropping it. (evidence: read_file vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_codec.py (_decode_streaming_batch and _finish_stream_request) and vllm_omni/deploy/moss_tts_local.yaml (codec_stream_slots=8, stage-1 max_num_seqs=8).) + +`vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_talker.py:?` [minor] — At vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_talker.py:1462 (MossTTSLocalTalkerForGeneration.preprocess), the diff replaces the previous `isinstance`/`int()`/`try-except` normalization with `max_new_frames = info_dict.get("max_new_frames", [-1])[0]`. If `max_new_frames` is present as `None` or a bare `int`, this line raises `TypeError` ('NoneType'/'int' object is not subscriptable), and an empty list raises `IndexError`. In-repo serving_speech.py currently emits `[request.max_new_tokens]` (a non-empty list), so the primary serving path is safe, but the defensive handling for alternate producers/tests has been removed. Restore the old normalization or add a producer-side shape/type assertion. (evidence: read_file vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_talker.py (Local preprocess) and the diff hunk showing the removed int/None handling.) + +`vllm_omni/model_executor/models/moss_tts/audio_tokenizer_v2.py:14` [minor] — The two new vendored files (audio_tokenizer_v2.py and configuration_moss_audio_tokenizer_v2.py, both lines 1-13) carry only the Apache license header and module docstring. The existing vendored codec in the same directory (audio_tokenizer.py lines 13-18) carries a provenance header beginning '# Vendored from OpenMOSS-Team/MOSS-Audio-Tokenizer (configuration_moss_audio_tokenizer.py and modeling_moss_audio_tokenizer.py). Simplified for inference-only use: ...'. Add an equivalent provenance header to both new files, ideally including the exact HF revision the code was copied from. (evidence: read_file vllm_omni/model_executor/models/moss_tts/audio_tokenizer.py lines 1-12 (header present) vs the new files' headers in the diff.) + +**Verdict:** COMMENT diff --git a/eval/dataset/arms/copilot_v11_deep_r1/pr4804.trace.json.gz b/eval/dataset/arms/copilot_v11_deep_r1/pr4804.trace.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..366ef421d5865412298086602973442633b5b1d5 GIT binary patch literal 55624 zcmV(;K-<3`iwFP!00002|LnbscH2g>DEccs%9$HdCMlA7E5ocfwlndX_^~Y|S$j!Z zAP9gGMkGK3pk&2cXPu9@UwFUdR#iXH-DrT6WIHow?8(>!(2wfs>guZMs_Ks$lUX{+ zH;&lHyAIoklQ8;#4;$Gs!MCR$H=^q(DZDHAq>5eoxfdmQG@r~Ocvoc02;M>!c))+U z!~Srue=ykY4u<`mong1X9}Nx$8&D`;_=%VJ^N3pCn9b(&wulxR=kTNO^DF8<&hsU< zoh)ZFc#qv<5+&d&Z2OKkm`M_Io0zkWS_@ea!j_V)&dJCot` za5tPB_IIevWjc?_BGdl%U?2{i{?5+c1_GM}@TpHY57I1*LeDR#PoSD^|DZeA zJ{b&;hTBKm`$Qb+G7BO>jg#Ages1p{4zUR&rDm$zf=^zN`9akCXAaX#OPbY^v2m8Cx_Re+` zOu``!NP}wkV1Iu-sA?hVPyOk1$DbVR^uy?|9}V{o_b1`O-e7yUeK?#3!Q>!_4hMtn zZJ4`*!|iFX*Y5{|;XyQ*Le)bPqQTDLnrH%g&f*{^kW7ez>0SVc2*Um0cC>vE?e&KU z%6`zQq10qI8txu$n-C2LySv-N^<@< z3n+gW^d~@f!CrW9xHkxcX)xX0pA7p4{@y;U^Zw5E&f(5*I@$Gozdz`QlfnMM?xa8T z55ozRv(Pl$-(GXxfI~UJ*$=k;$=+apx;^wKJG+x2h)9kY_Ma31IRmA4;;ycJ^upObdNJfFN~*C2apqgAIN1I4F*Af zZ-27Ad$`k|Z13#v9R$Pe?LjmRqaYj{?Cwnu2VwuP@9*!y>fGCh3EkVB?oCY~VYF+~ z`nz9Wd+%Pnee>?5HxEsa;Cr|o4yK3?fBP`l=?|d8p^sF)>+kRP2e1_U5aw;qpZbTp z!^49iEW(5Ra2P#ozkK@l*KdwbUOqdnz!U(_i$dsSZ`=2$`~6^N+CRXh z570~wfUf(8`-A9UFc^e8VX(Wu=kE=Md;0-&1gl^lU^=IrZWa?;gOwg9VwaoqO$;|8 zVug8d8O?DU8PNAE5<8f-|Aw)Dk-%mKYZEJf!;WFIWZK)a9WmD{N?ezZFBR8{q*$ZFE3yJojv>c#j}4Lv(}&ueSAAv#^x^#nif&W;snaY zIpYX~9hAL&cSIm0P!u-RDA2y;uuC)C7g^vZERScfaAvoF(LC_8kWH4;X_P^${xXbH z7;2O-AG&?NjPim_<0Q^6qnu5mX__Hq@0W3g&HB9IRv66*s3K+1U7A73z+dJO!INLc ziv@hV_2)AdEVGRF?a$(iB&P;)b`t|=fTN43J79DFgO@DlULL*A+Z}47@Ml+1#uizG zas^CGHi!Cu_=qBalNcuc^s{uCgltSr08^M|D0YoI%!jww&>Vy;gd7CC5XKUAf!)Gy zKM5js3G)h_q=`b~;Bfb_i=qgO>?%q?L;)CJ)_d%EmeQcG5|p zhBrVk-I)*dgLFQJaVx~c1%t_hxrjc%nh9dW$P$3a3RpNmOt3Famed^3UJkPka|g{8 zmp;zSb(F=^7`hLmX%rMaPDe0LC`u8Poql_o-+)AR&X8I;Oeki}(>(WzBKL?wdW%~& z1!?p4NYDz)B_e5#qpd8OMShO|=25}&+c|ETw+xnWa218$0kS+nCb=i*B*c;3#2F{j zi^b9_k>e2{9!E~kTP{MrR;dLK7fF=&zU?pty&5SI8WhN`IhH3G%-Q7;1Gy82;wf79 zCO$9)LDD&`gdC?SVFEx=RQB5H^?DulI!&TB%D7z|AXQMnfbuPHnrK9KVA6eLVJF)$@ZFxxkLwO&QHITL^|PRwj_3q*sW zuRp^vcri@FJpKT<4q&d5uz7xdn*`z$4k{OG3xN0*5NOtI#YvDwNd1VpDRc`|mW2E(tnVB!nZ!j5 z!=YDV(K032vsro(2Z(M2BZe&ncKx#r8?jXy=UfAc92ked^^wgS*a*6xrXWASLb`~G z*4c(x0DC&^4$j-B{qr#kGsGnh7a;?5ZXxMx70U%MYJ)GCNCgG}%9^AdPw^9kPB5P9 zoS2SPz`GtonJpOlG`?7>WduWX7CfRhfJ?&2W}t@=s7MF2Aiw=h@I4@=bk6>^TySFl zp8df72F&&QZv`+Weu*O9-$cna;Z?0y*vg>46xyEy+d>+}U9qH=?-&-Jr2)(>7G|hw z%Fl{Dm^?u;L@|539pJ}qEapRO0GC?MT4o0=U}ivXUY?DB1+{y!Wn@V0wu8VMS!oDD#4SE?0vN;Q$w>qF z*D@*c(J5}J^d|Qfeh54Zs2SE2>^dp%7ZNsR3>X)Sfa%^=cQon&)e|Hx+~2-qlbH|P zCG*30nUl06_62H&YKg5~PAY!5L5U-W%^xJ9%wqQ-U*S-XSiN|LFw^owQkwcPE_sG~ z?03et38DzeN2nIS7}Y+~LP`Y@0|gM^JQG|TVVeVS4Qm%Sr0~39!<1X4;<8)Od{NwX zfu{ih0G^0(w`fzE&in#JnP@f|_1i=-x!Q)PnznDreN>8Bd33?3F2^b{O6o-J_OQe~ z#1`N05lCl`oKEz|AL<%!S`iIZ8q2M~UX^{{t9rvGb+2|7@dDe4u->^&5Kxc^orpqwA?7IQ zOMLovf%;>?8K*=JK2Z0b15K=ZCFF2#UMvvW<|edB*XB6`8Hy7$`verr9z(7sc9w~b zGCzkcYJpTG#OY&FPWoJb0{FuvFu^&lQy8bvYM5HWB&Q>n(iJhGg{A3?vxh)Rnxq4b z4%*+bm%y(=u6Hg`(4WQE5p8xT`{O<Nt%HYk}RX_N+cL=K%U zp#&q|$R(9H=MoG|7_gjq>`!MI_u!M2@FrJ8^{mC_+dt>Ee{s<2(0aNdtp+9v?_<=oG%(72$e zEP$PB0y-RtG=2Ow zj26Y^w@2(-RO5*!!zbX^_$mVym5XL!HRG!pikw6_wDgbI_V(c+f5zP;XIpHr&7WyI z`S#A}SF==YZXN?GJlfnmOa6yF1EJ|61)=eO7>KvaS;77XUU$1)`ahO9A-1?B^!b3x zmH)#AOc*U%c^l|`mflcJb_`rRiwkTJ)C*BYnA~9PP?_gxfGrBkN)APbHLd*^UHaEC zEYB^@+=_StVk5O)2f(uw)>=>`XnUzEbp~_rn-m!d6J7T2Yjx3~|F^fgRHouz((0l^ z`R`$LSxw=8vdu+@@ZZklVw1!FWfqsp`2NKVF2K;gp1)MiU(OQlr=eV5K5o<>z|vj8~AXpA2dzz>1 za{OIgC1`w(8Vn#7rjIt4v2)ivEn7by6RYPIJwx^2`UY9)BuI zytzahAW@gnX=7ZJmq2z%iLG4f%N4z4Tvc4gZEP#!l(5EuX}q)2fp#aYEcHqfOt+&gNcOO+&|#N!PhY4i1}(g(KJuxhVGtjFW2^MSS5yYvd9ryx(^A&}S$|!Uk;x zVVL{>gd*hsQRS+Skg;5>mCn6kn+B0aSaKG~>C+{wnK(zP!Bg|r4EA9ziUFH^0-urs z80Hq}E8XY=(y?_~k5iR^0!0=!WZTwRJ)MisZhR2IeJGf8$xhDl`3zR>$UZU41{p_L2Ti~*l0;)M)>^%(iqkhtl#D@K%wlGr+putki~F6$AWQSh?~j=-NBHx1X2X^X`@Cdl>lx*6KKo1 z#=uw7!alqeO^g_2dU>ur?g-MdkEg}a0zae%)(SI0;-5IF-l7&5$0|zo>}ldS-M+a5u~QvS zhiN2EO-K}Do0}X7>?U2#`2IBkTDoH4k`yuC7m3U(PGzJmzH)@k&%~r!`S#s+Y(Yjq zWzYvzL;8Rfs+rOoF_ad4;tsMfZhEiY93NK(oXeq6cSx=@LMejIqwGTR@%t{QwGL-d z&XZHix0&*cfu`^6JBnRktp;SGYv-@;Uew8HYXKq`(Dr%@X#a5Ay@JkpKm~%Gjw^F8nyF*DDQEOGU9PRY$(-aj}8?RgfJC1)E!KI;4f7Xgf=9 zV6D3pYa!#J5GvccK}}mhpzG%-uSa2N69U1vfOQgpFbvhzL4(-YxedTn!MXMg>MfoS z2Tom|eLr%lTU_HNB~>VbWsF63W$cb19b9Z|eMmh=hfH z;df^+*Wv&~?&Y`;97ewH*dq;cjJCvjs~ifV0P|g^%2x)TpRb%qo=~G4+6tYi57D4j zo(G&&wAC}?OrV2Q7Zq~RqS~b?rQK z6Nsy-ck-sV!bzI&xTE>3-$*rbtn^61&QLidEf@7#RjSl6#}8F9w~AX`)3g;#WHuy( z=$bZ%0j*XvvuNX%XmBo;XDJ>v0gK4e39MB*GSyu2Q6K?qFVSa4Ap=+OkyGjBaOZ@s zAoKIf23j9}L0TuFwYW(gnqG&EouWDCXybJ?KNN6RDSaTjpx;rL2EcZJmcJ7Uk~j+) zSCy5qY|(ctQ7dfQ3UxM8hN)5ZsloQ?Nt^uMaG6B(>h4L@mp`#@!+*Xw=T zwxU-Qj(GHnWVd8l0tP0u3*K~9Z9%&S;!~+)S@ch6MeWD*RwDQ;IfnrQcDUmTAMn>h zzOO-64+*l`xZ^?dAX9i!(-X1!QcMoZIC>r~pYaezJQOY7m*oQZd^eDZ?N0n-o==H&`Yc@pfDf8hh>g%^n3f;I|qk*6n-Ajes45B zthT*7+}q!CwY+<{bD%Zc-yR&$2m_Q7JXpKhpdG!XmS2m2Jo4!S;&@sghs z_XSQonT=u4I~Wd!^6kRU&{ajtCSW6F*x!c#p_&fLzd|220G$93crrZz!1)W&>)@bw zIB*SjyT8A?ufp)@#CqYwC!Fatov986WKLTq3z+68RImH`9PI7w^G69quaB5}zw~F^ zfB+Le$!XKn(<`kCnfy=v_nRveA z86*U1MM?{eN^he0;<5nk8I}SNC!cyEr&y+}NCMj16^;AA4$}pQ1Og4*I3j+di!Uq^ z@kl!sJy2AzKaGl8Z@!!rF)UW1Re`GYl57}J#TZ>O<3L5geP=|Ta%wP52H~0>mdJG_ zNJX^3(>nAD;YD@`HVWRQ)Cnn1TY(Yro-;v55#wIKyQb#o7X>O~bG>b(h6|2`LGN%+ zrQzV%<K z@+GwB70BflQ439yP{d18f>*sbf_<=i^77RScJl7U3vL8wyWw`UyFJ+N`+n3W#~|8N zaGMo|{?eL60^98NJAQa^12jY)K{NAIBG_4A!@OvS41DkoJ`GhyBEOY z{jGZhF--1J_$(oI#HPD@!*FmAL_xpTi}nt}a4*=|-aiD6(%;+JIZL|T?!BNmOP)M= za!*ix_elHr&2(DB$f3QLcXCu4a12mBp#3 zHG&eU0P8b>!Q30Aoa ziEJ%v&C}#6x+R~v_R+oO4W@LS zg5Qhfc-;O@uk`az-tRxZJbwG)o%ia^^B2F2$AAW4DLx1XIUS#l(*@{V@zxD!v5P3{ zQv65vdK>#dsV>8v7mXTWG+AC;@VKDaEjq(K`7{l&+_dy zyyq|9K_k#d?q73VHsBNz#_a4UzdJxUJLf^SEResT${pbts53!56ygFM zoo%3D1$P~^8bCb>!Qmct52cO-uS5xt;TciZQioSKdF>wBJQuAtBQm4rMHK{xIyu|u z{WFb|79P=Hm}xK0{TZNW=W_UG|`K(lYN2-GhAil|AP@c<}ovk&}700K2V ztXcs_AP|e59*Oau!#Dn)K;U)c84QIRgDk6dK7{^mc;|g|k(xfBc1;;Rw)Pz^dX7;<)kq)87IlXG-t}P7 z7$ti*84zr_j+)k(v9d;dNwqCfC=dt5Wm{4Ktz8eJ5oqpVXNX|~)|P@08N^`@z@7J^ zG8EVP~)n+-(~~(F4g$P)%GG8e!g`*ukw>QYgw} zK*kkW8$mBbqXt0N7ay0!Jk32k#03UyzXnDPA}Ng*Bn{rI7m@Wf)CctI$*VVfSU4WB zv_?OnVWeD)uzdJo2S#l*p{KgtIIv_lisUh()zvb)YWDz>z&{G3Y*~^y!g}-TEIWoAOAZGr_hz-P|+}S0G zV}HLh+`iA6wEx^W-nXrH=yV(>OMR3@cB}JsbKtmkSM6H*A7qS<*CqcQnd{9}m`HfU z3EA;k-v>qp`RHFLi@;!KSOy~G0%SCnzRbF7V z!2!Ow0Ojt+&oJoEg>PVaF#0a6LWa^YhlXTo(JJx;g_qG^^CR}xoX!+0P2CHXF7)Z^ z0k_J*-W8HhxsReHb9;#IaRY6e%LlgcmZ;TcBVdej&)TKsvDe=ztzQpW9UNQwxKRCW z8gxT5oP@9eMNAE)BcuX;3dz&|&@{Pcp{!L>A66}&I-g|uBWm~hhlJYQ!%ugST-FMKC2 zq6t$3QTf6SId=YW3W7P*c?=owh}d zXw=haZzb5rxH`xZjt26)32I|q8DP75YUqU#kQ*HZvC)t+XJC7)v7qA-wf(-{7GOor zhROEV!b=G-ZQmR@yuG%zIFYeO*VuO*v=xJwF#0x>m%}n_C##`;`AVK@UQ2SXQu2h(-{0K}4)>@1!$GeX z?nT3c?ZIIHd*2!-basmkM{;&iPG)2!FRK~v@o1e!q$ zBr;0M?-&^eX!u$jpfTN2(33}FPc-XQ34GPgKhcUySdjqz53K>=v?bpv#sJMhiDc2k zLoe@VWHjptV4EfVnh(CAnY3BO0KaPu_uXFZYmOq>!LNsah{f+kIfWJx1d zoBW8MP?(qzqf_+iS8U^*Adl(5bqVO!t<Od8(=g@*u+qV13K0p0IoWNU1TYgevKFlPsAn#8h$V?m<`X4<PVMVOs z+mYr7qR`#jai>BQ>NHnygh!N=f_C&i+o|`+M6_ zuQ%A)-3@}Ef4DQOm6Dp8w8f;Rvc%SQItReib~*!+l2nFn?TD((+D*)KhPV3Vy ztto^a4mW;W2t2Yoirw|Er# z2)33X%Nf-zw|tf0($z3;P=vAD&CwI1;fRA2s;ydnJg5|ke5)X4db_h|TF@qdiKu1s zeXDaTuAFao6x<^wvnz9+hE$nc)yN{XEexo7VZJ9XqJpw-VBEYvD{j$4eIQSB^!5f~ z(#+^}d1htBw2KbNsX3ab*{!!MV##BWvIdf{A|5sFMbJ;i0}|916xcipP5R(rWC^TR z5iC)=W`PG}w{$yKz9rn(%q*6+inC4yrR@aU`>=@p;UMhw_75hLzCRfT(@E_*UaxU` zMXy(jPi}t~MU^M~$BrqpAnMQwN7T~tVYq}9+d&*df=~PBJ)Bt{B_kAkAq4egC>a0P zW>46law;y%Qo2xE!f^FyiyCAksTRYwD4}G8ftC^8Rf<_Dh2My`PRrCfm#%}9rlIx= zb5wOgHKGbVycQLIv|NSSR%3N|CzOp03_5T_Ni~t}b0=R4{T`v9t6Y@NHk>1$)^=R_ zENhNm-C9YgxrGs$mDa6mM5QTX3U0=_w`GU@fx6;YPo08KI<^6IPqWg zKd7t2G5%zovZqy-^eS%k6)iYOimIk#`H9bw&0e_w&xrrispZ`dNrh!F&WAK`mdMz0 ziZ4~HoOAqs`KQ;LZcqHdL9o5Izt34c$X?f57<*Oa;SU1^d3wLo2XU=~5whfh{Nx+< z3@A3kusnvbaO@5236$SS#@_zSy{~`rgNVIMf*u1wW8Py=F^M5o%Vo~d5PiJAp*DZv zi3UQEp%j~Wy1?j7`K7~t1MwPFf?=!0x5l#g@pP zkZ=9+^4W{m$1l1=GS2X7za}%Drb1-Ro7`g7YN*1qOZqpIQR5G8|s4z=_9%afnq{CdKEfBNp- z)7K|2UmUYH@7S|9ub;m>dHLowJpII;zW#6akC(5XgC@q4$3`D8^+=AOVjxE>rWCDv z1{mo0Lowna%>l;iUo0_a+6B7qQx@?>lu^_+s8dKv)XprP$HF1LIymg=URe{l{bn8C zdn{+}#^PyCr@6chAs=gg^)Eie#c|=|OERvFV2adJDMLD>^HQ*anvmXFIZDdL?TLOy)p>(lqlQgEbVEit{<7tH z-V;)f1=u`fo7Wuf@Gd|Uyr4#<&4?6k$V4@k`&=&``UL9Ud^46Hs)-P#taRTcw&Bn+bc}CeWrD1%(^pgVn!31G=OhSQu9hD&qqY*BOOH8G< znJzeWZnkx~;(46&1RX;6rJjZB0`R$9W?hQ-Kxpq#z4oe5iyl`(uR+@Y9bLJ?WRqp+ z51HS-&2^w9ECNtZ20GG>CkmRM(1_m*jvgX;A58vC)tKmzFIo+8BG-q&oxH(9JK`y= z`x-ir1b^WmJAIZN$X}zD6HP1XJ!&P0YrpQd67Dr`%D{Rb@+YWNo)(z~-9YV^r|3q) zSeqo)BZfy!>et!4$83J(?T_W{Ut&Fc2?qT6%NvfVS?< z{Txqcc}p-$ITh~d3y-@l@Dh_WopnV2BPz*%x6STz4EmGJQ;Qv@o=qZYfa~-K%`@uJ z?1Ju%6HU`^SKFGiUt=G<5@n5Tm5p^Yw0N`lDr%W^@s2U`g*^#)H(BPIfNx4w4t3lp zC`4vGiVPM?-FL!=faiXlO~p-Rc*-sGtpwQ_W;-(D(tA~p7OY=^?t`KZoSuNpsr(oD zzIIUi3Ze-B>BQuMlRf{>B7VLMTc98EUv0-i;8o0pQ09cHtOX9VPSVclT`nj*PL*ui zbFZami)UemQQ>4rEmueyH z{M^MjvArJ_Sqh^oZZ(X=c?nUK&a~ODj9WjTu&|7VqFYjAL#y=;nfnnYst%3kkWP;g`p-d?wS>Gy&HmpZb?W>b&dlZOYRPwbJ%35 zrUo&+EqlkW6{fJb*4f5OaYG{a)jVRAM)OO`yg!NLu`cC!5DFA5#cPm>e6%&WoD`gJ z_ITRzmY5Pb8*RG*%J~VijpAoa_YX?MavG5BwVcy{r(y!pJh+`d-1RX5DhiM{ zJ-&9dW06tWtN^QRI)dGIa@>VSbdmCt1~`y%IHxz9fU78TFLxW&NlU9{?-&%+TUm0J z6hPSp=u*=FHaxYFRF|dInkV|UR_3ae=3HeJ4&n@3sV)N~526Y^nW}2`FIC#+9gP$S z#MLc0G|AhpUSg`pH)FBKwy2_Ax^A2!4u(m8e7fDN(t-NwRHqA@b-Ok+N8txfWc}#O zEntnoSG8mohfr^*qt#-Rzion*sXEto;*>1VD zdEpVTYP}s9u{!v_XQupy$0|g0rl230?#L#U!cbKQPSM5W5BiA8M%vD%dzdH_g}Ih& zOP|Gd6?n{*?6cGng3UBt~XaUAj7#+a)3THLotiV0qiKv&_+=)I*bRU||Mz!EHP z#X4~T=3+e-K|2q&S2IN_)H5Mezj{T=Ig`|@+V@$OdYkP!$sCE4(wDdD_(e&~RBlaG z)zAh$hmQZ+aNPXGzlGKR8CCl_+uyra#|TNSYV$6I$$u63z`Fe{(jCaVlFy;dOgR!^C!@eUAwR7(x^FxtwNkGs4kyZ=(yKQlK zk$qCIs}BwKh!36w{6o(MU5cN)M8l&0K-F3!M=>%7*S}r40*IiGF{He9 z5pbH%DZvdL37w?N>kQd>0a9NN9XkdOUIdlsp)!{WI^1eYVaHL;gHaVqWhdHcUIS_w zJ!WO-ouhY__GlN)MY5pU<)DrT+8rA5ms<~#<_%O@_)s?42c2Dk2Dq}$=@GvS z>s(YW)5=jI_Q*poeiDQ6U(Vxbk5>#*U7kSOyP{KTkE6z{kN{h_P%&eoC+=`KLkcG_ zOuK>@3Uo+ofVK(%s)Jl{MBKTAOrA;3jOIa;{CKvJ&2!QEoum2sZRs(6VVsiHAFth8 zq?ikkv%n2;PuWAEIxtPjv|L?)nq}{HpM>%^jpF|-~k@?w#5{k_t=?^V|l1aWdm#kz_SMJy8bPAH9`x`nr11pQu zP~)S0uix*>5|pK#+oU{c+YKL9G=6^N%NLJmJDA2xR z@8zra@|6(~|D648#QNyn)_TtV%7zp`*bL{;mgFLWj<>e(q71QLyziMihPFtzhZ6KD^j?&lhiy$FA^?hGN8N$6D!!-JRCN&RCF^-8u3b*`|^RkhBQqmFc9NGYQ&$8V74Ek5BOOe+2iMPgMXj}xnigqeJ& z-)Hci+U(*8=y$A2#9E0MCSR7wXK4 zTuuRG6p*>3)Is>l$8#=>cNL{vRjOZ^)vs!+n_H0u%C;0lL47-liJW2>Vz=*JaOXK1Ghp4#^2fRcZS^mKa zMMEZA>KUZX8<$TDu1CzE|X(3ywMJu#D30m5IfLN(VN=+s`f;It<+1!Vk8{yCMwQ^LRn7hXRM$6Y-TanK zx&qRvbH5bdTMtnqH42m`*KK1G=&G3u!LQ2mw^i(=Li$dNM6C{8VU@JDtgbpv3TeC7 z-RIj^-j*VT5<~LOtPcx|5XhjO5@S?`A^}hC-rj9zsr&|%~Q^Xo}E{eWf%b}j%(?S$7P3QFG`OjC6Y)BHAj?FKf+&{ zzIy4;rXKb;qFzx!DdA8bL7^Qy4iIeRd)PQ@0a25Rxr*fPQp%R@sT@>8m6T8_jk2un zvXk;HU?3E}c7;g1g8R8D5A5Z%ui29v*(LhLX`=$YxmE3N6I5*)9(+}Z?}9#R z-^m~pI8g3ROI?mSX?HZ$jlZ$Alh4E={_z`oMuC@jtoXDjxHscx7eW8=8}<5*Qc5RINDV=2|?B52GUZ4>FB_BEKv2cjvpxXYcP+B%QV!5$}F9GSJ$m|kxlH~Nlz8C zBQ{wPFLgZDSh}d5%h-iiZ99ZT1DZOFihDh{#j9?@nSlS0}OcAoB%{Y za@Pf%Vni`*SI>TZ{*?Va^B0#7KHamO-Ly{_9wIbw>kCRln0gk!Rz3z8G96>C7wZEi zSBe77X=25(py_e1W>4H~bGCtZnc;1N&FAm3+Jq#A!Jhj>k&f8};CNstepZX%G{`1F zT1+b`z*<8ykVav#qgw442&|+8GmjGOkL5O|f2!zr6=aL-R>nKWSz~pQTz;KV5_;UY zVj-jI^EFy$UAdI+=8gtywg2fN7!(}qgfI|@dJKcAt%4ZJHzt_!UR}8(RXC*mqRIbj zO|DsCnLrFDbXHP1Ic_?rIbq+UI*wxG$>`&Wc@m;7Ux^y{u!S0nC0D-ME)`|_^4Aj^ z+xp2zW6i&sJi+Y?y>hW&#tN^}xPG#6f{@6q0^t(#(sr!|Vn39=2ZT)A`H|V^eE`8U zR2(!_yk2WPDhQ~ahuIXUEzt~oSqVZz&1;Kbgz#kaZ20mB0m|rLEt)eXcTM_pvo1Uc zELl_oxdAbrpxU6L84B`LFe9U#D)E(j!>sT%GVPXanB%TWJ)Bup_8M#tm%6>t=G@&kwN5I$?3yA=%c=N&v?jBun;Ow% zM^kV49-=CcB&iGScCOy_rPI%=b-uz(!`+|p68A5mb2{QeU;X@)d!d_7=}PLmy5^xH zN4g{~oHZLfH;hkSuvp4K(fdjou5%7vv8+-N>K-nEn9@ge9mvHf(bD8$X%t)+XvCGc zuF&?axrr!^6bZz65iKN-60Xl$FxxM2t~81q<9^7&oC5qX^SX6!>4#>A9K0>Itgym} z$h~SXwHV$`Q!oK@8BmTVikO3QQL!1~$Vr+7J&Nt>Py zXOX5DMVJ+Y0lnwZNB!xJkUaWm0DDIkmukoEASs1ES^rdje{SN^g(Nk@Yv9NwdL2esIOlkqjBln1RcIP#@bkd)0Jc`dvxq!tp&AM9$Z90!F;uR??Di{H#% zk#sOeeU|<+=RO*x%wlgar=+h{by4ev8nw&zp>aaq<|r#Ke6Hs4B5_ zS#$w#0JdkTAM!vy53dlzP;~DyO=a{Xea7bwFaB1FA|2GwLd`tF3%jU1TEcLQt|E zW}h#om^y$vpYQ`TSS1t7k3^MDml-lgei;m~l#-zr8@XBVO@DgKCC?z#-^)ZKJJ z_o`c~rs>8FZ43%+9QX3puXq%<6$;N4Os8?(K+7g6@I*t+BLmH)4sOIWaYCof4;YYQ zI31HyfhN>;1*TIa{T!&n%;ltINHQb?t1i#IV^@_1<(i`|W#3bq{MMGW?Yg43#CBej zCqp}Ra&OMEoyrH=u?GF&&I)^$-9sr==%!Yw_>Xjhh`m8*$2%^gKzY_>n;A7Mc51-u zRHD=@yWj}*LMJL}PS700BB;ZCs#YZs;a6U>w@M!}X=+*~d*_dJbdUUHY#?olHr%v~9R2=Yj5zW!w?9z0TQ-Nv6WQ*u)CUa~i^Y#Sm3#P%nqCtV0&qJowq? zsLqyY>}s7A%9NPpjs`>flk7!0pz61}tdQIFwmUWxH{BF^n9+R+v^ylP>uox|+|l9V zK^ND0A9So&J`Ic~7u`tx)}A?RTaMWg;ZX_b;ChOEo=B^1k>;Li%V1yjB9O zuTPJTY%7*5)0A{Y@1*>#dia~sSi1p2^;<`N(*oBkSJNz^0re*y@(ybzCABGBf{dX| zd4jBfN>ocQ2S9M7WR3d0e!CL6KuPlfl@U6n4BOVSGFPcxyNPQky>2zz_+Yb)8g5*B zq@%I9>hY4bO?8ao;)qpy({5;-*y@uJJ86o*_#7vaC*{osJVnfxJ&xo_Jem$9ts}+q ztn5o2D^^bE>TKp3Drs0NEap1qwV1L`XD$!<^;GxtU$g^9VCcjC&cVN+O&xPAQ0xn{ zGhuWLKNNgVyiG<^)V9cjYRWbfva=-0 z-C!h0*H*n0<&J7KYEGLY{!is2C(M!fx1w1CMUL<<`-459qg9G_Ed`9~H!G_N+mPJo z)rw?4K;@;HmZl^5z#74iSbq4RX8y#Z6r&Njf|WTR)|p}2Z%(VsgCPfI{gvg1YnNXc zT`H+udLi8V)W@v+xek9!m^jjTXe4KuPS+TLTr>Jt5>+LTaNVd~doJwYVzX7%-@47a zgy}V=LG7kfc@yHP1qJGTj!IFWF*~zX&J*NXiRi7ZFl?w$>s4v@*qH6k zfj4!{xk3+a^dqg=j*;w5Bxl>)Bx@I)notaJIzbb}d51Nxx3!S)Z!m^OoZ*y;%O^|} ztZ744t98e)Jlo*I;+xFU;HoAkL7f+$ZuxZDI4c=S9L7J3chTpLHWHvB@%{!g~PmAO93bS0(X3)Zb^_gbBi(`wW(xQ!@Z_Cw5A+!A71K<<|di_Z}I zYdI&CEG2cm-LkHX8wlfT;Aj|SKDpK7mnlIAFFX1ue%xsh&ZCd&XEpfKNED#YhL+wy zf4zRzy1E`@W@r1A1Fi?zXw3<54oxKdO?e()qBoB}-8uCm;n}-7*IAH_MgVuGis|2fjn}S>wc@Mmv|pRbv*thk3My*Nkj!x~5vAb#+ZEt6^h&8z&(nGWxMYAT)M+fnCMo&;-5YS14X&XO+w9f!)pTD5nr2)NwNyH!cq zo^Y2cM?F4vRPEKF$PGN4OA5^_e2q*q3OJSQ%IV4B%>APx=W@%B#Nl%S;FsqcX0A3Z zn)sQXJ8M+QlFCv1p_*tx`94Smud?fIN+?W3Q*aoa0m}>AgpBBDoW7v8X)mPvyEoLT zX{(6Opy+!#M&wn`h`cfJWSt`LRys^MH3F4RYn4(ECyQmVmD2s$(g}<56g^Y`PC&80 zDWC>Dl>@yZ=ku-_3e;&8o0gb=jq(VzR<8?)(82O%icca|^94~ht->Ze*S$JndXw;L zrM(QTV^XgfRHJbWv``8GQQ>S~9u--AIgg4eNGLR78m-V(Z6U65 z*MuJECa0DMyUC{-1XRYNy0Fn<52k+C?g-?IE47=qZDm?2aoei+f3@+xVRZg_4*Z1# zCE*G7+-arKP+P<7DpfRev_7=${$=;nqwJnwaWEfi_@6ghDT-ZD$lgNda?m0t_Meav zTR@`&Q=1i=%9gjW@dH7*+|kUkT*#m6B*3NI1n%M~P` zdRa{d3LYSb_WFq1d^FygcQ5DdVJ35FxC%@xy-K6^xJ^tP_38r!jwgYp`pX25fnCfn*Jv2QMs&rlIeOn~=w%>9QX-rHcv>>MZ9iUzs3x=! z{kwCMs;{o9kCgZoOuA%Vm0dOPNP5+tYYX#6^~Z`bTlW4Jv!v4)TZYprnp6#RjOrLH zmZ47x0yr8tU0c#p$X3mnZD@_xHdpKYQv$ov(eWPq0a(U6xL~hK8i3>et0N3Y} zlrD2rz*KqdA1+T5%uao3n*HjZ&Z`NUwHHr;U~Q~HpD#(C5Shq9e&8AET_XKE;v>~I zy!S_vHv$J*rzMd$F%B^K)-%AHd@`$5>iprsLVZEIi0Vj4H}6g6T#JQ2U>5(dZq*V^ zm%10%eEw)xZ_?sP6#)Q|w*jivhUcKfhu`JVyVZ=Mmoj~TPHi&Jf_|(?{*M)BZwXFs zk>0eR|9m+|Nb2sF#XnuTD5f094>mOFBDbQle4f;nZRp=>r<%8wW*-LE#Zk7=O4u5- zO@D>5l=!7E{YMX%V5{GjlCv?f?^1J@pEd5kgrgIH`1$nIcgKE8+6r@095pPbDT{V2LJS5^*06o^k3TM z-KEW&Epu3}lUv{*c^oeN85H`up?|Us-2&k+qiPfDZtZ4MooYi5-Q*n&*?oQ8?jOJ1 z%WW98IiT>=ma!cy;-UMY#xy$`XV&@b%*XJg9C&C{WwZDewi0Bpup40!s8;yFC7#_n zyIIAX@5h)q7knIF7CrscE6=`UD;?)%-mcvoyqnZh1A8n$T0}4nu$JS0hP}$Y-5SFy z!TCh<>>P%qK0Xrh^nf+zSY@xcK`irOKS(YZ>}E&omtm=fqb)39u%FEF5E3NQ47NX% zYw%>Sx&(E3Qoz|LQ_3<*u5=(rV`45sTrJrC)KkN0srNNK=w;>WN|vK9AiFjStM`;n zKb2^zCd(91@3CB@q`DrfTYaglkimYdm*Hd)m*MMkS!T!$a=2r}HDA9kKiB$SmM3#7 z3Zg5Q&=RH7D5i=WHj9uBxGIt)lsWj)(xgBRX6Y2aFwa|E=@LoC(QacM6JH|TXZhmG zh_}^)W}Ah$%u+grY{9Z(0yEG_jzD$NWV`v;xE3-K9BV>c@EIWeWmK8Nl~CG%n)BG0 zp>r*NOUu$*<0`1oJTfp)>+Pr!%*d3sS5?724DI3*I4{U6dj$sZP5 z&|OqL?xjaJ44^h>pMQ$my$p{-SwT4-{-GQX#a%;s53+Qd{mq!u-ZBCCIEsLW*mgSu z;btl~9{ci#>7ps_Yc<6qA;;jLf*DYI%Oyo=QzS1Vs98VAJoQIRPg~t8s~(AQrzAP# zCJ|JKGU@!Rap$%po-xLRSq3FL7}X?;{429#IdgzP4H4?+CVjLgz%uN6u@Wrtr$>qT z1RO1^@piR)9*8Gzhg;R3J>%x{6All+t+iUs=vf(Oca3@)1<0@iHEL^ECb5V+R|`M5 zYH44%i8j1r`^03cO@`^@eMcsG zq=qUihr1VB=^ITss`Pp0O}ISlHKy>;C3$+3ygzT(Z3mf3bh$!C(z>gs`&eIJ#&HD= z@iabfS2O)!e5clgrP31~D`W<*9eSklrd~sRY&ru_`lERm@Ks;@sxN-k7yl#mMdTJ= zmBg<~BC-CLYD3VSHhOESF>cfKS9Rudsxx18ng0M?#vSBY)orS?s(p1ua`J|v`2sec zzIk_f?Xqi5%lR&~l`V*9(FRKB;uHT^cSgK2nzT;qU57ucu%SE(4pzalAw|Iss z6D@(ymfXf)7O5AeL0)7$gGGHvXLN$+(@{vUwaRC++>d^4Yj609@ph3+s%KIRNYN<(K%hk>ed!_{eV>E-yf(m(Y+u zi;I{z+1ZAV{i-4&IZaNoRSY7q7hFcc)glE!hvx^OI6hI@10gJ5Kx5(#Fxo{8A!c>{ z2{Y%5t44VYyM9?%Pi5irI;xO-JXq#MItLsUSvoR=tCP#fHLXds%wUrt+!03#_)HTS z@V5mo%uj+b2q?czic0`5?-?|N_uV#Kib*uNl4>o&6c2R-xi**z3vn)L27V$UqC^8S zyej8zRAHP4z)vFHa86BfZYA0h?((naT9U0xxS6 zHmCI)_Tw_1g)){;w>VWm^mqY~FXWskgF*XTMpadfdYGO%>o9h_I9z}-AeX$1~VFDq|Ypfx8Jto^y|tSZ98?ZQ1^ z_dqvhIrqL&EpxU3`qsz-H^!!BbN`6OG5X!O2Yiw%pt zweUtqN>??RTno}_Ta}#Tg`a6@Bx@2#R?nS|v7UA< z80!AqXfT_Jd@)s{mf5Sg-Q9V#QRaS=SHRaLRW?pF_%-Qv8piShb$8OtaS;!6F(eEfA)#35=E zkw!X+{vh^^xu5tKN@hcptXS{pcjKVrXW~9l(H_yTN!HbKXPFn(Aus?hxx~mX!!u6Z?;qNR*ZM?BOE%g@ zp=xz<@3W1utTE>4y#t;ulKJSbjQOC@jX=WRs|e{TkFRXzw0~aB-5UzE%_wB$NSx?d z$c&-XxI^UD;o5z&_F9MPDgh79RPLG(c@%G*z(H~I>`eh(ce6>wJ_>|?j*)b)bc`zA0@?njPK6T zE%g$umptlV+&byB$LRJj#*n}h_)U8p@P#C0BJ-nDF0$mKkF?TS3JqF8xq#Fxe3M;8cpgiM~#|1r3!l8hbTVJCR2&-bif-qd9F&rdHc&a-w3au>2Y6+amXw>yD( zO$zv$)bDL4&*IjLl51J>P-R})*iWH}Blyq8zM7EgA--PSnNYN{VUquV*ex3z=XkNY zO>~tv@MC=B@akH}_FDq9{svsuY1)G6bd$$jRd?aa-l@Ui+N{gZ&h0vRPlx83R}1nh zBp)6&K&LFSg>w@NmK$-+&16+)+`7B>y43HbNdJ# zKh0rZW8BK?=E18>v>>B;2?nggj~1e>if>WmZ}x(D6n-m1mlw(WZA3_Q;*-wzNcR#` zW6w;l9^0`0{T=o$3#A{;XV~Qs4O-td5%DTjzRS;x)w{q*3a< zm!jcVKjyx;r^v>Bt4{>AlKu`7shI8m8R)&DFEeB#3Rx@CTH1KMG9@P(FPG^~uyHlVvSGr6*B zg6b!xA!drpAR^zmP2i1&)6Qy5Pp9bW=_^l9oAHzlRfF`8lsim1n-Tyhw>tbqSLD(t z2l0#123PR_ZIl~BW!M;#Fvxjv-f53}jS5v`vWy=4@fNY{FK0YnLN5Z!H>LZuMo^Pg z<8-_Nf#Y+&;g2YUn(s$6S%JJ=%lfl;9pMh&Z_G; zjPPPIJ^~C_Ck1mFX8=b<4b`w#s{N zVX~aHX*okBs;8^pu%CgRX7VB%JLb);G`h}|nxu7k_hCi&rV}i82g667xHSOO5Gq=G z0JK(feM=+yk>a0E>4CPZ$PH5OU-$?9}Do3$`sl^&|Nra5Sx{kp*@ z(ontPD9R~brxVy-D*9QSrsaw#5EpsH#??1jC5*uA#;n}Zzg9q&iDg_~&h;UgrQa}c zghmhEj^*)dWG#K)S;^#$@M?V|rU>NDakPRbwd$>Mqg8N?oEwtF{AC=5Ku+Yml8>T$ zCu=Y_lvO8B^<3WoZym1xFnR5HR7hrS$GADd*dCC_$viQtV=yM+)G!#g&G|75<{KuG zS&{HY?f*HesV?nP&JEwT%M4zrS>gc~qk)?mzjKQtiPsf(bF*=?a!dc(3>9b3O?D@{ zt(7dmpk*$sjY7G`~dxfn7>L_Cc zn&hu3M=A$RSbLf^Uw{pezq+SSVGybsV~Lsp1SIFxF#LWQ0s1p({e|A;(HbbV|C`@# zN0H-7k3~P($PdYcz!o#~iXck1p)nT>07NThHAIHhKv?x0w{#-YT?Rpv=jaAZoe?pO z&3?x~0YRFC=nN18uf`BICyuZQO-qvaLdUR0*rmTbcAmO#O z$~)&XXy8+uG1kcDQhgH1ut)c|iq|I1bB?qE*WVoH*R}p~H9QZ!HpvM~a&*gQz}@rr zYX|>5(g*4jvFb*rC*jV!Mmu+H;h|uHb+_8FOrY^c^f?%;IcF-mMeecJv0vT1L`d4~ zmK*M6+8@pBX*8(%K9<|R;q5CT`Rx6x8X?%Or4iBIV`wu*84D!2IZ9QyA%R8(QWp0H zvpZ>`S5fR9?jMgL&rh_|uk_H<2RHhtKivU$Gn*TIbd7yIKRuhjx?IB16;}9zJ3_Yf z$}m|vs4mRkm!qEcdc6)i(z`jo2bXQySGjS43$5#fw#YximbSX0%j+M{pI&^3i(|?^ z*k*ptq7M`Z3D0#Q4)-^})(6&^?W`A~Xkb%83w;+Q2elgRxgyL)KepR$yvR}V=(ITe zQ0ATOo73GfNURZDy&8E^l3S}*+iaP2%kJAN@M*SO#|hWS{#h&N5Wt+Q_{nhV)a>>A zUjD63LM3%oU3BGK9BIFwSc5-r=!7$MYJSBs=SfVB;k%Y~^Q_43db;PZknB(^a(!QA zoptXV@0JOs_mtb)OI!$`OS1@H9I>%6(6LaKqL5LtP}6N-qrK*!oTmIQ3AT*0=KwGc zfLrdUvu_hAG)=?Gb~x9S0a31|(Ea(Sbbpo|!!CB-gldDUSK4b>FwH&gyE~5N1ag^699^w3F9&aoeegeSC~?!UPnKDz%3E|TuyK&AcvO`CX1+wES13JTC;zp@b|1LLiW!eW09LW zqGnrm0Bc((S{DRR#r68Gy`Gu37QwD4O+6#E@+zwI?mCh&X8Um`GPa!mxD%OA@3OS^ ziHvmIvFwNS{>~Lo=dTAaUk_j$2QWf__<8{I^#JBeAHeX|c_v9kDh779ww&l~V9SEo z%!vM`gB+u>I>^z1a?RsC*|9jy(fc}YV9`ydHyFBh|FfHipTFGC-L(Gv#cbyjp1+t4 zf70_8eS$g;%4h_-V*f2qVe~f4UhZ)gW3~T@&SDIjbezSQy*}dJEwltixe4Y~)eH88!meofSB@b~QC@2HJdqfR}e$0QUo8`-%R@>z3 ztnp=%ya!w1E9`nbZ*=I{_D46K9%|<{!6^x8g>~9tNOrZ>l> zPpQk1p8Ts<8G>EL;~&#>wyIvOW%|py5QRHz-IwRiFcT5nvzF;TYmACl=2vY} z9^y#fXrz1R-w3<5G(b{{a^Oit%DwG-H8ZckE2sVwS%{@|*uPH6D2(Q7RCX|58V7)f zMac_9d%T9g8jv~9@`E6w?&K zMj2k1b*Y((q=6lL!=8bh2^17A=kr^gH&_Mv7rEWztu=}8Ss(87`+c(14#WvXnKS~H z%y8%G=Rc|)^Sv$Sx%Q)(;b_A*?zRg8WzZD$_2XD)dF7IBR50iDPqded{5osym5mkG ziCC>^;30k9uQzc(!DA3P9}3l6${Ro(Vc~X&g)&dpM`1LXnBAV8B0kY{lGiss+s&(^ z&-}=~e$d%bO%$u)Ph$HTwB4Dm%!N?zp0KjEBl^AG$)T!4e1oT(91O&_oEgCB!bi)` z?6!kkUbG#GqRLKDO0e$3qb6O<#S6Upp|7MsoWh#;WU0M_eB#G5%yw3L+|@*RgshwB zTiE&J==pIhRlJGVKhZV_I1HBoo)shF_;<0B%el%^EFGrFw^BDDQiT#1em3zhDo5Aq z7ANP}(X84)87pq9xc5m%eOK$+)ZATGQ!>?%OPv>$zT8K~+OJFLZgGIRa%<rgNI>qg%e#vyIDKg+c&jV@O^XTU6jXFt_?jBru3G~?p=&xZsow9EC zA_nzm>v}eud+9uhxA^tpcoEGKBmn-HZ;|*}6!{hxe8ho>=aNdj>tSzk%O;<`%`?!A zZ`c%_Wa#pwLBGGZvvZb!0vUZ^{qj$*7wirW_IHNP5PG8zL9{4%dTc!D!qWqOeo>Tt5IF3= zMI-5i=f|fjA7xO#C<*$40-gMwi(+xKwRLlI)AI@JUYcEO&3NH_>z9|$Uc5ej(FM>| zwO*68`U=P)8 zBrjFB02okVoXeBrZ97oC13U!rEG=a=UJb&&=7G;FX@}bpUG~3oZv#B;u`F>XR>*tX4SygDQ616 z<;EOaC_CYaFR2}Vc_mEN&1D?$skZXTLO2cE>dAxQlUF%A;iKy)Na~{1n1v>CtP8eVW7wA)re`ef0k68%|CdTSh7I*V z0+Zbbvn7a;s%8g&mUoErXMT0J`0cpVd)#ZfPQ5ioY&9dxq$+wUv>vH0smY1b3F!UScB}-w|eW&1!C2%pYPB@6>L~Wk1}G{W1YD z4tWjVl^ok)UfkNWrL|!0NTYgjpuM7J&gEOQ5pqIcHxUWwZ87}IdMH*H{98u73M5Z? z6oIjeuRC6TL=;Henx=94u2ilHiHjlmp+lE;fv(zI_2S>KAN>F+J?x^M09~Rm4k+`5 zOaoGSp3X7&AquV*DX@LTU;ILd+1}h=KyM#m>1p0&8+d!NvB#Zl+?72WU+6$t?Y72V zqkme*o0Z2ni3%or7Oq~no=<@lbx;C6KUatKYZtX|LD3+rzjc8Q!b{A7IDw)yDiM*A zEzn6v-tH>Qi(bf2uk{ZAs>cbp`qkvQ?k{qP=IE@;G58J3UX$a(HCIrnlX`D}>O@1K zodb|&I`tBUEVq5IV-#DT0`Tfv*zY^Fo3(q2>&LMUfiyZ$RA>f@4NdqOiAPw&4Ns0H zLLvr`hYSB~!%U)T6BZSmF1CkuK_{Wrly(pvmE?PDr(uh=39%lDo146b@*p`f&IOJw zM#bi2w2@MRkFuS(+kI;mUHHMRIDaRQ$E+3gF3`6Z1c)fO^fZQP2*cm7-$8)R(gny} zrF(?%D^aHhs9APr(KRr7y%W75KA7GjUC{BcEQ2OM9JzHf)7gf~7s`n#Kifb7Odzou zCT3Y^qf59_ESNTmyKm89EDGO}-w&T~bMG_jFO+{7Rv((8 zj{I}ic^ph9CFwQ6q7>68p|w;G5HDXUtYrm|*6J&So^W;h|GfqyrjuB;v6Qw2|14? z*~cxTf2~E-Nb6r0(bq-vbrF4rMYPwnb$wk#Ul-AT;366f5AG??HLebA6*aePETdt6 z-DR|Wzh$&LXlnZFGWxX3=)9_fTAGykqUyrSH92{_Xr6hrtV2c1zF{3|uE_QAsdT$9 zxO+9Pr>_g|>;CnhxPR^Ke5UQ|s}T5g5&fquqDJe&*G2So5q(`mUl-BWMf7zM{p&5F z;a8j9*G2TFFQPRMX)6YjiaEl3QD-12oroI_yDLu1H71ctxselZ+H!4JPvmng%j~)r zh`x{;n=&1q>#Y#n(?q#Gb7~##tSBmauj$+LToVhX;*en!~(;qoH)bp z%X=>z9_$&xaHv)lB&S1|YNjC++`M1gQ?5I_=8Al_?#dQy%@lT89m~0^4STgXDAj&t z)mj4oYPEaHZ&msV>NnIckPl|Xt>))y5?%{Cr@T~g;bVO&X5`J>#Z5#pB;-N$> zkrHZwK?M}V({2XdbP=n~le$*?wupXscx1Npbo1mJekc4rW3lU7k#W}h!EU&-zqh+R z9rk*A{prr`E^KSN6BUk);nx1A>G7VS+D%PiaQ1Fzk3HG#>{DnqI}_1(o;O`$pdieR z2wL$I;D|g(iOM|@?}hK(ES|`^K{~@cWfX4ZPXgJ%%Oc8nN)c8UI?FgngF}RA*zXL6 z94h=%M;)Eo?>wjmKDT8=twLJQL>hn0dlE&xYB0NmQ+n^$bcgwCtDk-wN?4xUAC|RZ z?N@`xu~i43!X*O8?gt?K5MH#)(7(w1#ia(1nw^uA<7dAeAT-VS*rC{?Mj6*ho4G__Nxex?mx} zyvc$+so=|2yItP4U{;)FQDno9K_o4nDY*(;yV`k~znL&3nORRuN?k?hEJau9GMktD z%p<}nNm!D4uoA*n+bIKjwP~aI@XKt`HqP~o(MWQBK-nJ*@kGRX996T?)ZncqGmVwl zOyB0KZIIu(1zZ-*B0uNAbr=>Snsw5uywzP`*A0?A6_^I3;QhBw@t^bdYATV*h4G;R zaM@l7w2Vyx(9i2&C+yba(m*5jL5W6XjzM54X7WZI%ad}>?efO}B58?_Oh6dhaPU{a=``Tz=vs-AyEK@I1 zh!}1lMpi>k-!uUAjZIV*D#2ho?^x-`E8nY(#m>294#7@~z77wgj#u$8 z6f{2*d!#EBfTTK33d6fN^6JtoXAJYt^XfX6D4ixjYv3MjkRrvU1=eCvfFB=*ji?aJnqgxen z((RD-L8VNddHj|M`_c*i`OU8IJUgRJjQGu3omf@$eg^&ep0en@97q{4 zk5az9aE)51JRmF81u3doz4xfOA-q85ItmdCXb1Q-HgirxEn335!H-3{kN%^Yy8h*t zSKT-BBnIkVPDDQYfL;2?O4!qPY?h`AVUHOMhMhz9WH{LE9Li-@-oH_9Wq6Z{NOc4B z?}z@Pz<+_9F6cUe6f-HNY&^zXg;8Gg7Fjx96x@X6#bG`kceM6+x-CD;;%nHXFQY6% zyVY2v`k~^es^k5-iH|YQegFbOPE14eK)gc!c^hS2@gCSN+8QQ!=^JpBh$rY*nTOHx zb(VBx#qn4jy5RXij;EZ!zd{Z;kl8sv((-BIDWe-*-Vi|Jm}d?b5%2@8Jpn|!hfQbv zN*xaq@QOobEzQ~g&)&N(w~Zu;!momrcfw?wgiyFc9`%`Cw|m+#eeto~v+;#mL4gD) zaZQnIf|RWFC}Mw|2RO0Ma2{-*#MW9f(%*x7p;u3KOQqsN4 z=_`F*!zGr)ix>;)Uh(}Ir^7`a3{8oQO4$*V)ntJ9eDnEa#Y(jEFJ@ipYnSQQCSB@L zKYc?1q$nh-Jjt<}(A9Q2A)u@6!#~aF3VR2HVFUs!%vq$U!#$a!DNXs`|NPf~=a5B% zdyILffU+8#7^~Be%D{q-D-^s!(-j!<>PTfaw`F0)PL-S&3{L(b?sO05){r#?7@w8k zK`ZLn$9whGmva%4DE{J#=F)|~eqgSE+i3K*vSE0>0821CQ~UvWlvX|zYzJ$X5up&l ziFg&#z;!xqumTZid5!hzXbaAbujJH9&~5xG2ebamvaVBQve#)ma}NG? zI)5)mr&>S6cM=TBMVL?|I+vyQD!Z4HQngT6{tz=Jq+?#rxK_Ol*|SlHc*{|)4FuLJ zyAoJaOjQzN1(r=xRsoN*kz_i7A4egm)n&w<<0H6vuW?q4&zCBB<@JKw{0(vG@Vs>7 zb6yp%zC0&(%{zr?&($bNfHab}DLWN8b81cNMCA0N%sL|;pz8x)6za*4z?>lN+*iBf z*xfDMAj)^$C|i5oJvXx1c(gG3ccp&d<@~P43)PLMgD%etHysXv!w}Z>iwoV=hNDhO zT}elnVNo5R!?zmlc|aGsF68}Q&Qr4OVoZfe0ZKAKgT30}jB?CgrKPJyibPm3mQuf7 z#dA@K!A4xA?4hLN*?KY*PK_aSaOvgl!rW0itlKTW|0MW3{EAQ~&Q#Masr`3*> z&ZNFw{KRs3#xrgYYOBi>9y5*eA=bPlK~kK0^%he>nO|n;=0~YndsS$+N|2~?MeT}C zqttzqPD4{LQRgKkWJsZc6j{8n%2#<3&y6W2%D%-;O9w|=&Q=ISvXa+;%=Va{mXimC zovW9_t*jNxkgF3R&8i*(``Lo(%8lz$c8%)36a89;xL9m>`ceJtzJpUg{aBm-eMcN( zzB#8K^^3XJ?@XnV80S^+h?T~uE2oRA&-k-)>dG1&*~WzH2an3fdpd2sqU@_2vs>x3 zU9O>CAN1?BgA#EQf1jO**5~z=@SAMKGI#^I2F_5fT!!eeR{t38NZCulNU!V(zG&DA zIqR=i&su8SGVKnCNg+jCLrc`IK8Ld>gIKhV@=qPP4@@(YkRRI5_ z!e`(`0^Kp=YmT~Nl(uA?eNwC05$iB_r2vJBj7!8DaeB_=e1Ug6G4dZc7AEU7AeR!K0=Mp$NP5Ws>DVeQqZ(HtAb5n+e#`K+DqbJ$*zW6 za|^qamjc(-$QC$|@>YFJ-T)e8E_CmdI!rf+swyK_@VqyT)qr-qNBf83Qn?r9rM60Z zeKyD{`^$%LbXr3ZAPhwe(`rAtK(?ms02hh%gED2`c$`*-9A1TTR7e?OYdD5)PhS<`6_ z=@>tAnAc&^n&Vt@F0Gh*Rx(rux_Ou+E}Z!BKYj} zfR}pZx~0X3d}&fcL1tyq4~~O^T1^p!gIm?Zq;#pA!NzZEgt%CR?5WDdYw@TUl^b*T zPeNz9@x*J|4Xe@tU5eU)Rn5URQwS`qMcGhzZ@|tw!+_f@6)>)1O?f^ zcO^LLYNqQ5pxy;V)wak>xi#v@S#%UE@8smO-CjvVp6Td_-o#X~?A5?ARZ7s+zm_%P ziaEAxn2LW>)5WphES*~i#6uP#e8&qa$Y~-Yv^{>Otz9r@Ba>{2tj4F z+XkvIVJj~_B#(twxubuw8cc!JS}oXy8n8WSzm#)MHr#cr0k`e0(L!(P!z2SA$Vsu} z3M3rW5WYf;p?Lv_@hqeJa~V^(3EQHIIHD;Q%Hr-sRPQIf#+o9Nse$#{xQheXDotK- z2EVUXOUDwKSmc1qK6n^qX>Lq8EG9OK?=VR)cQTW7=yWk(mWpG(!m3MBQ#MI&&^k8| zWy9ip&E(pWOo{LS4?l*Lc7<#7sN~q&Uk-=qJg;+#7P=F~iAq8XkOZwKlkpub-x!QX z=ywe$vWQDXRca9>(i8hsYr3x0>CHCwz##g<`(EvmB5}AZ#;upPs@DzfrJi9qS$$V* z!4na~qhpQgZ7a-@e!iPaCtIxH_Y|?jyk*NRX0;PjQQ{QPo#(W{t$og ziO(UwLUeux8b*d3i4@rk&_H~m+W`1N4v9jX{JP0?NGr2wTM4

P?xGn?Gd*?E6@f zDy2$lDzP1tj&gueXSs3VxnAj~M2XL*B=J`VzNy;s70(Ya8^N)fIagRd1Pn94*;=23`-bteY;NjIY5iAw=QleX+z%Yu^OCE&52;=`w)nGN_oVguUSo1Vjs^A>$;2eZ#0)p7@c$Op z1+%PH6O$JPqjEHu?lLJD*p-Kr;!2BP8F+J2*&Bhm6QiYJ7R1$w#Abx$H98vIf6;~} zNA1?BSE}`6bR5sN2Lu1d7jw%01req=nmU);>e*f80xN@-7IT5)b=VC>RWCw`E~nSs zj|74HT7;kK7?Vecs4S;8Qbdtvzt%)2h|p`*3#j@XDv_Gpf}XAFXoWjbd0~?b1Afb# zcCp0O&Z7G8n2K-CX0%nu@UH@LDJrc|PLSb&E8`UfQ3!F%2t3@?>9C|)_G z@H8e_w`eOu*!2-nYS3lgpE^!Qq>>?vd-@wWwlZS)**gDT=(Yi7t>&nxj@EX;H|i7} zvzUm8t5FnGM4#$|R@zN5gf(cX_}HLFSrJ)FDwQofqe`K(8>kDaLaF}5w0Ds*8p(vE0f`r#@DmMcx33o#B~{RxBb7Nc=lHAPjfq;$AOXj7c6 zad%diM?&-?<{KnQ^oClLatE)&reM*gzIkSHL9cLXp*yH z2#V%_XIfL#3T@4Pj{Q8HjpI8`7;~yOo6eehY&S-cX{;NxTI5TXDRTQIjYg+5cmn|> z!^vv?UAYa<)N4c-NA&%!J9S&5M^#6owe8geAsNfbZQ(_T^Vike^<;m$z8WFE>^&D; zkeQ_mq1kFR!JuQDowuc3S2IYfnPsW&vts^T?KX&<5KR#;UO878)?VXv#xZi5A9>OY z#lJ#0)bw-{>E%7&Yild}Sbd>a^e8y9Dr88n?D;eiGe?hjzG#fbs=DD)BuwE$toWtXFNo@( zwY(4%^{TqNQGN&-;Y9rVtlG=@W^`GSeH`Mxt<4Xxj;L-%P-*uhL4fwUQ$riPO@=7t zB;dFF90>0fbUUV}$OZ`~75dG6rZv7G6^gv#R8(o{rydxA^D9%VX$-OT+&&z#c{Z#H zv>KEVRgzW}QMRClD?cp>w#g_Ppx0MfB2b_*o#f6qxxro#b@{W$%?@fv`#<>k#J{z> z*zAc@s0wbc(n;Z3BE4+jwjCvD?gld)Pj4%}IX{uDQ0We%y~im9q6&~?Vh9ls*Pwg6 zxKNWi;cPI@OuGKa8OX5bi;LVF2NSehT$siS;r{}A%(ED%%4f8PrjtvQpJ_#Le+s8U z^(ZPlc+lAgJpeJa=MC4Hpkt642bIoY;;?KnEg(!&nl6v0gKo!T^ds{L=L)5zJXS`0GaX@cS^S=7Ov)7tyb^j@Kd>J+AEezJtB`ot5UuXiA=O-`WclIk z_NfcJinE@~h}LRS1Xi3glxeMW^;pwZJ+CWlv`$PL zd!Uhfpp}(mC7ep?+%I^76HAv~ntN2PH#v!Rjpn9eD&@ZN{>g5Iv`m?Gb9ZGsZ>jSJ z_qO!IVjkmrZlox(%#=PBMrmq{LulU>HJlSwdmO*gm&JB zd#|+XKDYw>`_A*i6Y1I%`{6sGk|7B%g5GU9y1dFgX~nOTJUz288P$wt!mC!^=vAL| zfSHLs_F&T|CfrTE*?jsRnQ5WA7k1q&JvX^aLahhX9>;SgzZnkM>XvrV(iSaCK15H9 z8{23#WbkbpOT76pvp8H=m=Ac_=SeLI&KX`hwf-owoRsGa%pnZ)?D)Iye!@?AX%k8s zGMQpBoDrto#zXb%(J@aGKAqJ18%=0Kj^)mD2+!cozZecD=0~ez-(MJSNg`Z~;5sj~ z?X|A56jMmRjU;yQSnLr_bw27Z7jzkSo1rG=Qi53VthTv&kgtMw?Q>kCRfo2=od(v= zYOo$$rGxjgDbiC8LjbV>hx&5Kv$vEtn}@#1eSTi+5P_ZNfflIoK22vl`M`8Ox+E5h z>@fs>KB!sgXyq=~Uhp$hTi)v3ZaR9}Zc4je9J$7}cDb$HcqWU|G`;)`k2AJO`(li& zv40-CtuDKaokaK|yJIsTe4gKN@}M`kudm}-E5CXbpQ;7}6cK1_Z;}}U_3ZNjTm@Mz zl16D;4s}6lOX(37F~(_HYJNigYQoc-X%c_s%m+sGp0pQ>67Qdsn4q_S={d6khwAY(nYp(?NB(SkZM`~W%A&I?hO?r~j0ewb!7@sw?;dC9X{?KVd zEM0w9_~DhCArL}UO~8(2LbWZD3#SWp2^b@*&Y;4s%Fj`;$;YBUs}y0^78+#G!<=u!M&|IvYKNde$ z^itg)@m=L<`~G=j4@B2``E0}=XD8nI{hDv`sfs@~`ldGD>XWk*=lovhuw8XAL4V$A zyx(d-p9s*N{wX>BQ<85r{Mq8vJ-?y;1lW00W!}^y)~Ur$S8$N31fr6+4adNvHYfw5J)v~jO~J-V@fJu0qyRQs9Z=`B_R;~U5{H9jg6 zt6Jv8Jhl2KIrXMGc9q3wnl)`hX%A`(t}(pTTZovR%132u=cajkrIm>$SFVxO*5~du z8c#3btmAi>Z#)ZZ_}AG4X+?L`tiK$M6OL*^4NyQ@vBwgJNHGd|=H`;zznI@ugLZgj zr-B8`^7>B8#Z+cr#=of`5`0|wKHnqQz~U1Vy9Is`%S!^kmb8lLVwCw3h$O6hx=3kF zq;7BPIu@PM*t}1!;8O8sv+ha|Ir@ceVJ#Wu9Z_fj&y1Z4E5n~MuHKRh(ItA3JJDRR zX~oWHJ8bBTcBI&aTBZxlYD@KzYRg7Ze9dZgLX-jrK>!twc#zH(#!LE#S9Q`kQKs^J zOT+N}dh}g9!jw0QslgR}ixDbJro-WAFp`pdZ7rkN`2ITqNtzeu3^d&j<1s0H`dCf~ z2FW*x)BJ{vZTd;A+u-L$b+mDM84vDsgKCmZZir}`pxM4^^Npv2c>Kw=`D8D$HeY{e zu{d=4!)Rcd_As&fX*x_Ce}A?4iV;7#Heb;d&u8pYntR*tY-XCt%5JZHt0 zh(2cXX&?EK@62pETu@r!DIjU#k09EK1fSGgNw$=oQ*bU}mxbdK+s27=V&{u(+d8pr z+qP}nwv!Xvwl%5wr{-p^=Ax^*cfA+=R&`f(uf6uO=8$AtCB;&x>}J!B2S{J>WXtHu z7A$fJ|6EiIpcpAsz<&A|<(T`BZ{|0SB;z?l$7UV*RN)q(x)1zHXu5xLclCfm+^+b< z-R-1vi~Z%r6$O|v^qOH-q=o|h=kob`<(^+1h4VoCiH39WMmD++B)q;qeB4l< z75$OkunThGEU{9&!I59Rd9EKT2E!#vx<&*`>I0rS;n|3hHjjRCzi`H)q&t*x&0hgk z#%NF^*)Y1CNJV5-kk4^K5t_P9Ayg!mI{4N`_zxmFrx3CLhWcI;*SLN^%0{o}E5S*d zvA%lF-R25`w809k3@iO6xtAn8)_wW&pQa90jAsq17K(OEEorx{YsNY%xLKrGnHm%J z=R?3H>Wxp$2lqK@W2g4G$p<_hC%ueFWCrzYH1G}6KJr9~d_UHZT)b!+jJ@_5qGAVt zjrQ0qeFN+E1G$vqX`5gOtBxi$DHtRD*u9;kzD#Sxg?IuMzG6*K#YQxUo9bHT(AcV` z@!IS-eQVNOxb~ph+5FBTYR5VKcVER1E19~RB=t&2yCLoCA5~ER3dlV3KqF6=S@M39 zm2l`XYeb@D=9H`U-C0JsEyDh}Mn8+N!4zGlCYXaoa9H}yZC!Bj4V$jnRwE7sB60&z zj<^tRv$6AX!33Ha(p#lq3yZ$A=U4)!Qe5E%9X`xsXJ|qFUScHye@PJqLb+KT2$}zX17iyvKBkbAc3aQBKVplyU4Q6CHmCFN6V7%yt0Kb6Kb0 zR+JG=ex3yf|JjqE9!`EQ%&$1Bl3jcY2WmlY#*y5IA($I>`4JIk;BSh0SDle-MQr{9 z4U45bvIRiqr~4pG`Yp47DC#K;st1Q?$*;+=AAR9faxr}o3x61i0X|V{2Ov%a{ zZBD)%8I}4vOwo6gm7>+<~=2+hi_<7^nwM@-73Ya@1qy`LMM)^0BxQb z>|`QMFPX0W^K#`PIV0b~x;XcVqPG$EpSBbL5i4jQ_lCbxvMq|)&CzYuNhwQOK88hP z*o7@#0;}{ndKWA0t~=$U*w{7tMof~&HU`$6oFocP<@=wp=J2VvJ%8kUu^f5FJ>_=G zA;VOXJJ!2|T?#z%)UKFjjF-am)+x7Q1Nx8O1%K2hQz9x9VgS@ozEHN`pQaEM=kp+b zEcsHd8%aqvFd_QgU(~s;W~^@Q6i)0J(~y*d5`R29qf^46Go2zCGoRo2eR09M@!R6|KjzAnmLu}Nji}^~37J1AahgWLB62aY zu-3h8(t5}3IVwmuGM=|;F$))kX0%MApZ<#crqUuym2%wN?|-NozZq>A)+1U+)v4xQ zK%wccPu5LhfW`eNDM6xX`({ZU)L7+s@Yo1C;)MM2WnRcnc0}&Z8OJa0; z6$W{eL~kxkzHGOg4(@^;`?k0Cw#F{sqgF*Ny!QBo3WncCM6eR<3rgQ<Dtu|+>Pq#-LXu4II*7iv)Y73N=uB_$ZZ- zQrZiG26vZTBB)+!VqWbsz1tEY2`rN2-!uk zVw`}yWoBAuRZ|*B|1td%%A#=xz=AqNLt!-3OJdHvC_L9w--eGO9!2Im))~s2*n2_} z4;j#!Q$RzNNd%M`3_q|^$5BYTg=_e^W>+fcQ;1?t^eW2tW@1ag!ve_^SO~dfuSgH1 zJ~2;$6|5nORGzCST6ivBq$!;xLz>tH{jGgKQW*vb=Z9iBfvpser2I<}VUgc3Y?T@I zIzY+=)D2n?9P9$Pwq4d(Cb4#vue@i^A|AqwOc^YQAE#7L_!n@~z*twMU=GbMh`=&v zjOUu2WPVQNi=`hQefp;+(%IjqPZaNxrp{)fk@ewKV}XrVTMaV_)(f!H!ir31aN>qX z9%y*v?mB5dhfrFkC*-t7a5sZkO+Im}2F9p}>(Id*qFxD<-zMgvBdBdCK$BwCOJhtJ zbW5))j~~WJvd>L{f|P=t6p^_kNPnhnK9RYUacUlCcC6~7R^wiLYhU_Iej?e$8}7Cg zsa-*}RTVzmzq2{c6zrTRm3$dhyjbPVY);mN&4(8NtJ_G;(uC+eLHcO^CE?v<`l1`{ z;g|}JqeXm74(MY-5Ekg_gEBunU*IgOEc9Ea-83dgs|tB(&|A*2qSxl`GglKfJU6h4 zlKxkv_Ix-lY*`P3iVtpJ&>i?5t?~ivT|Hs}(4W}Dm|eF-mv!H7H+8C6E;tZIQy5f<3+C}OY3x{ZL=})KH z2Lajb6~T`!mfRg_(PG=(KGf9s5XtYr-F`zSkS91HO;kIEWaD}LMk4*l8}t=s^$;pw*WOEn=03j4~`>h zd`J_I5en*3>H13snjdA3@sFWfwTySeoUKi#TA5T5t+;X2jSgk;7+`LAc52`&QHQgZ zv{#irLfpe8%e-ZT)dqWzQ{@bLsII8AS-1gy(S23vyeZ0t(>Q~={vkyMo^Tg(GeGGU zqYZNI*lH#uJmtkaO5y@Bneqy$fP|=S?8GR-HDrDj0iU`N%^(dPV?T2 z;go;(3&=yG)tRDl?G29>&AcX_{8hL_hb52sRpCmTOpR7}-a=KaVi?*US*j9YRBk5$BQ8+{QTe<3iIvo=bxp9_z zSw>RQccN!$lL?DZ@kAuaiMSi=fyNz;$n&#sTw0G9o#w92qF{N_F@3)@Md{tRRs%Bn z+XII%igy)D3$YP~;Ts~?Lf#-e*GVcUO6!R6ui$3Lo4ez^b-&_SB4Kz&|PWINpP@Zr@`kl7d`hHS}1cBH* z_yF=P<1bs?#D?MWwVXYhmF5QbGPRRpZA$6zP%kcmKIv%=YI)N9bSIseDGR3NQAGZ7 zmq5yZ8d}tdg|-}vjA9pW@h%)nW@NXVab@E7S~uYB?h{G56=(>JE&sw@wPBI#P|K5k8x4Ax0x- zkZs-s?TR{=n7EAY(nt4c(bJGKl=_Mp7_b{5a>CwyWq)Au1Cm0h z*}F>3NLPCq%_aYJ`{qKWq)|Lm#j1n~tk=#bhH<>629tm<4z{6zjX9cMhBv=v$x86R zBd6IWRh970I^8)^Ix6ws>aPjgM-H?9bakUP24R@TyT64gbxKibT%Y^n05}ka!yet zCl@>09~j9xCF^G1ocY5Q9(8>(s!hh@wxNer_ivVVmmD((U|iZ_Y1-m&P@?$HnSI9f zWv3!`x}!J3$sqUO?Yo7!D`r=k?E97977Guqhjr$_ys zmU_%P+ubHf46{P@CsX%bw3^4EIxp&l>J;3v=NnYU2(2yHX|i}nvvhOR64w~oehLshUc zTls;X42BK|oIo|L>NQq%$SAoSi;38Ec|WjPIn_Blx~EGKCpfyOnbM6u&I0#tjR9d* zjvuR^oJ*9UBAWWxh*@B<7@X1L?IR;Kd^rKtLMNS8u^bl(m9xg=fYPZ7+6P#6C0uKZ?vQSeI}F~Gb71*8aRCRbGoE2=NKZdbj(%7^E$ zjVRC=V)MO7&+Lr8sC@MFAIdn~>~nR9!YR2qWJiS+$e9=%jq6u;e}v1UGf)}NIPbbt zOyQ*!Z+oG20GdGok?MJ3qls;hju}iyg>6<;rx zqlO|=i;)c%8Dkt6{#z(&U#Wa;GV8a!$XCH*kcB2zFA~)Fl)bJls&p<~LC#415bsgM zx`};0MUe4oWNiRq*52kKCw^Mz)TCj4$i_+3wQgFu2;p(IzC1WV4DWb|fd=gGoRMpm z`VLq9XvFE(PVf3x8owy`6;4QOT?wveaeujFz3tz&LB?rJqoj6;Ak}oD?d%p`UgJ_$ zLz;njOT&9K$CdYd-wJ!H8r^O-Gu2=-e^x)YT~=ikrDKdcmUvJLR^qwCnnegIv8ghe z#DGfWw!(K|pT(_tmbKCrRK1%%4F!00b#+Ve5$TSJp5xT!7`CC^o!xR7y#+f-!?T3! zgf#Iuo5H5K=&BCyoc9Zm^PT%?{_;fe?k1pFGwPVuNiFJf+QD~QKvY2^U+onB^!{I< zYXMLx!sylZ;cg16tQm%hO9FoknXTR27UN)hGZJbktyu5|s-5_f%aPmErw41gzI&*W zh$dZiG1gb>5by5lp|aB(y_g$Rv50aI{QLQ$vgZ6y=LEE|C7JBako80DhcHiVG{qs^ zavZ7C+mPtIkCLo=qBF;<8Rnzx5YoS7Q7!JM40(4+EC*=L+S|yB@kk@LfpV7)XHFK+ zIHhveT8_`H`c~)t6)yQvlkY)C;7!p?+MQKehjYxm#_YwsdU1IFO=K|fwH9Lm!b7Gs zAc{JO(_U!Bni(mF9faU5!iQhYO%@=z^N5;cACz7b2{DG|5O3GIUkWg_f)VPh+S9xE z0IMbYOj#RptmSmWAMS7iMZ=4doLsI+3_Te45yoJYik(awyZhc9I8ln$bR7+5&a^ED zR%SC04$qTL&+F?@*Q%qJIrR92K2SU3`9LB#Seg!wsiX*&1rCZEP#DI0Njt$JJvPyy z;-385HLwB>ten0uYe)h-g>ilUSMDHV%~0JQ7J1+*vq9FZE5x_)#YCXmGYac{4#=fR zM&`IlM)tVL`5m&}KGPAy_M^*r1EZPPIS8D`B80FKh?8!hZ zSw4SK6S!=c`!vXXJVOl}PbwT}L#%L^bd|pw1f$8k!&=8w!+{Oq%?#@4diEp7B;F`6 zq}+5AbK<#D=_Zean|F{E@VhlCF;uA!?wPx8{{$;st}}GIy&fH|AKA8_bUr>VB_k7S zEmQ9EZ@!;*x%1oO01Vx-BfUAs{x9lk7<=jmREI>1pzCjV#l4RpOL0o|zSAmA&rQ_a zGv+eQ6p=#_U-IL^G#{W6UXkI*|A@7&8h3yx&Y+3{mplFMG$m+A z-IL4}v&q90DkvqS^wrThK8iC3a~;L7fhP;N2sR6GF0Qr}oQ+~gJc&}Wh7`?kzyQLIRZtIgX!FWf#POJLbeA+LR1@4SSf{C21PSd;p^ zsCSH8QvPeiQg$S(bcdp1k9!BMVBTCQCMAa4#!15yQUNS@n5WwL`Kq7v#2#<+h)+0R z7fDJfgs>MN0Bs%pFoNK;M(C+S;a}Js{BZVAKKik(5}Bb48WxTZ3BBT(Ck?0}J=z;i z_!}5+WGeOtErn)D;y0=YV1Z*GP}7zOTta!xE4?^Ag zgCiuRdI)>9gHIh{&OHlNAUTeaTP^zQuyaLR_%rG+lad+*-!ygoa^(`4^VPzPA?1zI zd~ycV#v{|LbrYk?)^c|wOhrCVsBo&`t#20dC?71pQ0pL23Mnz4cLGDHz~g% zCwvnJ(F-F=c>8^X_BE_Jcl|%m#Uep}%uZekX!xY++@_(3_mSVf=@sChI;k0&eTaCs|4~nC+*NcT zoKM}y25r#7sBS$h>2FXz*Wd!!hb=w>uV?QWW-iMYb)5)rIbldeJ&mIPXOx>X_y-`3 zWtQjzdyzXqqjC6AAK*C<#H&;D|JLk~Gm&OA^pEv7Ct>XX1VdVfYDd9edBU}(*x97A zG}g>+LFw_`fo2t7SmROD>E(RWkad%Ah&26kEifSac)=shl(`D~+mb|=z--cqijJ8G ztP)3EE}M}B;xrwZ-a5k%@$v>l=Wtr-lyMZoV=KH(%z68L9|g*bjXW-=j)|6;_C@tJ zVX3C(L=U|0`{XwHSfTT5G&q#X?2O}V7pT^wM92;l9SG{DSg#@jK0CqKSW@(pz8nh` z@|o?tH|74EE^tnnGnZ3wCw^>)Un!NuDW7I0!a881x4U+X=Tpt&g82L{WVsD%EK%TF zM(P}9No1!AN-5X$im;XjnsX}X5jAr?EHL*kb^83CDATuy8bbY&J1`K+qF6ei4y^|2 zeMElw=WV{SX(KEIQZrB$IXI!aox<<`4T}G>(_DBbb2G^j9rKLagX$mwGb=OKu7kGC z@a*Yur0K^m$}|4hBh@$3`-Macgsw1~7&jv{y1U@YE-^`B+|9Uo+2RE2+iX?bA{^gk z51PaI{R=MRpc)Z_IV(uUs!C*XG-6^!XO;V+9bBPb)m1?ZQi{lco1?@>SR-{{|D1ZD z`2xA|uGU3@v0uN_df`}HbMi+8=2?WZ@mrLxw}j8-d(!%nMuSQ}@!)J}J>>C~m2%{Rfgni%EZTn;(cG#aV$WRg1QT{E@Q z2FGRM3dp|~$FlflmM&oRX-#-t)9}h)hko9Y&WGZCp%rTLdrJsZ1P&oGsVs!7@$q@E z-oyFwV_z9pB6ekqI;%nLw1!ifp9G_M*26!`&bhAONnq^St}iJ2IGiYjm&}Yx!TQ~m z@??hCC`wFY;pC-%v3ti;3>BEo+&fxxg$qKrNt(|V1%7Ci_$K37jiZw9L>bIE(H*V~ z=3ECERyQ7Hf_#zW;Zwbqcauw{1moQz|JtVeBup}^&dIY`%nn7z%8YU{KxWR*B8NA| zGLa7g?;Vu^9l_?6r!CBJ(ma%YO=UzmR%S-k!t48Yx z;S=^c(?t7^EDp?ENT(<$mc$Ktq1|XkD=+e-jD@g0S53mg5NVv!p0cwjHnWf0gKx>E zP=)oa#q%|YsHP1wb5;B*A>h$bXz;)oSq|M9vJw+(GzX!Sp4A;zW3}_D>{Aq`)Rx>y z%)qaH3yVss;OjJ{DXFL;2RbeKASC*bm*<>2{6qPeQmUU}{q1HUwh8gu8fCti$P7(U zKwo&)rvoXhBAykSWE;H->ZF-?D&v@z?dOeHHooZ9&_~w!U9fs;I|Djb8Va7w@QD*G~ z<}_{)BxBXGcLQ?MYCs=*)r|LPLnNjU&ZEYi$m#_P+Zcpk({-n3(ws^+C{2XEfMte5 z;E9NoAz~D5F=4cSvwx^1z9Zu}IjH3h%7xx0r9x|J;rvvZ3aLg5`1&$Y33B$`mV?#0 zSj&np0$Dhz?R!Wsb2wjs$XLGA6uGoNsj;qmgepqSlx+VBTggufR>n_&a6fOXnKz4W zoQ8@_x0HtV857a{D)I`WtmMxb&`uwa?Wmo5_-;292*nVnqi*9rnLP8e1nifww^h|k z-%cRJwt9IgoCHwGX$dW|BKw*6mKnB#MjjfnbqegeJxDT@qkf!Bdge$-VG_}yIU_eO zuaMD=a*XkL8;(%_Ayrepvs)@t#YU;0e~XXwX{cX?hluLaMTq+vJm(Rq=+P%)pJ9F%Sg+N(+cY?O2TRM6ss> z;{7l2gzJyUV@Nv3b}&M5VTZpcSw#~$6l=Tn2zXS^9-+GCC5JXaT}q7;9>k92Qk8#y z9nNd5x^R=u6`jaa82StlXDOj`^|&gw3s9Nw?+rNh1sD`I=?698H;swl6i1;Pm5C2J zW|$NF(OB`d!FgJ$RZV))v6hN0`&XNIu-u;lC(T!ZSfP!b=cyd7bC{Iwk@@P#P)Cnw zhnf@F^?bN=FlMpTrd+2T8K2PgK_YVp zbw^5Yp%n-HfE(8gouB~158c8js*3`k4k&#+u@;S}=!C^2HO=Vr68|cok1fn^>J}=eR_hxpf)YQG-=D)#FLmXWZZ8FH)co9ibo0#YY#( z7O>_`O3s(fqtH(tJ5NXk6Eg80@q)h8^=a$7=!gE2hLRTQ;J zS(x}dHvLECqHHb0d(4+DhB8~dtVmidbg%(X>|vIXz#BuYVh=ZkOAzv5d!g&DK(r)+ zhvE&2={+>aT4g%_D17PL36VPuRTyQ25-mG^e_u294g0PCi#CH@Fs)7bDwh!_N?n&N z1iH#htfHgk@f6{Oyt3|hhP^=EF4Io{MXs4>LmQx1lzezDfqeL!Eg0aKXmAAu;CiPm zs#G~1UjI?6ryy$@tK(O9tgQ1`r-$tbuF4WNY9bU2zg~5HciW?nuM+U&rd%ZplWvJHtRA0&W-&1UK(pONw z5llKVNl^aebc(y7f{C@uYVqFO7i;gc$lwUZ1h$bN>@rzf286ta(ZHUd@yCe{qdx7p zKjZ*-Y3w~uzFUPW? zx(0d>L~jQ{bICi4nAuUSh_`o}>Jy*jOqo?O6mOUO>B!hBptU1xKL09R_i!yj07Sg3 zw)Kt-cFQrcQGI)LY~!P%_YF^?pcAtn=J~Xw%EtR~Q*it27~RYMdPcoJ%xJaTI1bld zy2hd-RZ}AhV;Sg2UqAG`pa}-ACCy9@tAk`iOG|r0&^hsP;0DU4-H9Eg>oaw{+Lhiv zn2MK*{=o}@V7f%qkfp#>-TCU4Ec0Rpazmgxz0^T`iqC6!_N;dMD?v+|0cxXk!s5ZG zfjOwH>@s)CGCXDD7LKb}Wh7mNPuPI)MLSZJq!`6cY#l_~A#(rKv7OK@^xwAdU`==G!jP*>gM6}q`3ZGsX5cK9g)67<7BdJ(%hr=Z)|-e8FYj*5 z@SdG1sgE?8wln^ZAF^2*i9ix7NGVukHgJVX8X^x!JG*Y(z-xoyR%VFO9XB}hEvnjjv0q()j1|1Gs19{qYr*wE~t`$IqOKW4wbvJ_6BmD z;TonU6qW?=PXyT|Dg1d@5lcxp+$k4ygu*T~lo4onb{R%2UH>kB2*;wkd%(mDVV2Ry zU<1b21@uNlL(}%mvutCGpZm=T?oE9zKzokC!N(@4fjPQOn4SB%exige`vERRvSYnc zM{1w^Nq554+P2Ytd%xcuws7~>Z%y?0+zVc>r}+3h?j2G}#E^UQEO$=z^ms0KyiaZG zLYJ)A=yq{zI`8?5ky6@{jM$1b=y-++=N_v0{7TM$zK`MUI{RFo_~E|MiMYQ{#q>@+ zZF$~b9hTu}BsUd*MqKy2o|dHWc5*k}2iJV}xcp4mEu7nIdG(BC-p~B~Y5(d8-)~Mv zGxWfICPG1|WIiJYf6|f@Oz(Q^xQhlfif;86rMwk1g$z_=%XD~hF5Rnc`D&*6gggCO z3h@_QJ%lYSRpJ1T&+!tRI#ev0E4$;}|8txYzok+=AfZ5ohk zWf0#JA*x2_>7@8U(DcRdMc=NYb3C^@E|T11w9=khQOGAkZDA2?3v8%GJ9XT#LSDZFuFxLsc-e?NGqyX|9)+H`607g5xzLFv4cFR<|tQI7c6B$gslH6HW%AzNNqE0q0F&dS~4ZL_-b{ z22(Db+v3GV#Y5qFjl-L7b@b0#K20fz3_!`hfNV>|C&TselV+Vqs$R52|0d?z`Q2t+c=#fd{amMm|B{ zRM|iGJ1$g-g^eVTKQ(&ra|CsGL=D<20lfKH+w?^k_(Z~mP)fx@^m!2&V_C936A_vK zGi0sQ8_vLG8W_c&&B+A|QD$$)4TEPuh_g$jj+8DKarV~31Rtwg$Ft*ee;nfR*yryO zI&;sw7&Br^Btc*))+UTPAfqDyua3nL%j~U!oa4-1DZs3sjnt)}GVQfsv2D$zh9$Ve zS6&SvbOy=A>%+h5IHMILOEuT+u|J}ioHWczSZ#}|=Ya3dKK>c0Q=Ax)JwAmksKofO zX5OMze!uAYxV_&^G1M@72Umd6A+DS-E7B&d;=cTPKKa6TpGaQj-Krwv$4@fxAmXS zeo85@yYp_8s~(_@O})~4c*v9yVI(y^2V4%{Pmn_DVeWkIDeI&MAjERBwL&5~h{KMs z+S$0V5LE+t2`zwF=B4ezdV9Ve)zI}+0bc`?In%|)ITyoD8Vr0rZu@xOx3tvsxZU2p zpyeOO1ZLBLITKSm(zwrGX4O`` z+2o%Xbn!Ap$NRX|>G^q|;d10;ue3_QLqavu+}r+)z**xK9;RZ^@lR8#WWTayavp3U z0#94#5$J9aRd!0*hW?C5jU5f>%&**UEk;tU{Oy`Y;|o8>>it?K%ZfMaNaXlN6c%HF zWtD*u1SzJoHj-b<)%P}>7yBRZLX?zwp3ecDD2GL4@|n{*32*xX;>kHR;ShiiV(1K$ zb}vK8G(o&2(;qjU=2xbgUV8kIgAJ@7k*H)Ky=2~!`=T!GsHuQo*o{9Vula*~6j}mj z#>3mKXQDJY^SF6Px#9SWqTL2dJ?SVW8?j7WN&D6YwNN`=2wBR{QB-FQ`^67#3(}_j$#I~)}FH5 zid{$9^;`gj-!*0r7%uSz$6D(&{k{6jN8%WfL2uu)@&Tty`2Yo94`QBadIv>eD~T7w zTF6Ly#*kJLU%2ahE@ZCoiHNNv;`Sw`cv_XKpVhm*elvYD;V~v?%wm1!JY(kadHt#2 z_Y(KsC|?|WCT6B_i!D=TLfE`1Lz`O>St$H4)VV~;NHl(V6Y5sI{b zg8AycAhYDaXea|_m01xUk?MFqJItq}jv?@fuW-9EAC*jW=jmtL-xN<`A@MxjXgYF< zBpL?KhbWGtdMArcGUDbIm2x~ZP)hQcLrQ9W4sA3G#`LoD7s#zQ5W$mS(FeEh29?&t zCW!-4ezzeRLk*ZkGB!>uwm1!?^=E1rl*mTHNLT%i+vurNnO9HAGBj_5k$@UjjX(*Q z0*DN96kk9<$WoJBoEBl-ol6kQgxO+JB@V~Tbj#k-N%X!PA4VF!Ef4EP^3E*mF3S!Xot@;mhynV-{eKeG5p{ z^)R64Y&8Lmkv2z#Fbl#N9DTU@fm(_noG=GdD17OR*QSa*x51Sq_?>@-?d7iWEYsb) z4+a}4g#+ve)P=4<*y3V7r&rA*RzAHm6UC#_18M*SfHBz|j3U%x9)-&4r2j2uyhfq} z0*=OOTn=i0Py1+u)Rt}*METI&m`hdn~c2<`N`Ztm?)U3 zy4~?`K+Y0vK&T5E4Fk-w3?|`Dn!2iD@%*GZIzgTc72zwoVz5fRw##2A9I` zC$F`2BE$~n{Bh915qY8=rxZS{C7?Z&HZ}`Iv#50e#Vjun7tuKY4;GDUH+kN`t33uW_UHyU0K3<=aNDy}< z(T~Lvh_rl+tI?V~Q2C+TW6BD%R)pZGLo8_as8VUk7T7gm7L(6Tjk%UUxAfe>;B#n; zJtVs2R8{J-ES#ZCDMk#*EQtptiF}G1^d}?1dLR^s#Qz9_isSotgwTFluN=)IOQ7Dc z=z&mG*}fHkn^o=jN*|XpQti7j^%djA1-?B&V@N#UG?2j<l;l8H_h--&7j>b5=#fzUA+{F4K9kuUBA!MRq?>uIRBzn)D zEADCDuR!al|L9$Uan_vIXa&&QgfC!+oeX!g+;cHFoTVW9x<)9C8c00$7=~S?s)GQR z!a{1H9+LTy1F_F**FLRKvSP$w%}W@heqv;F5-jUjUOG;|DMtN109;)~%TTGkY6K-h zxo|4t_@vz{F2t?Cr{g5zO0xZxoz1T9kvNZpJ*bx{0pOa4<~S2R zR2rLuxT!P{p=K2#BhjKiWR-9|tJ_O2uFzuu*+a<9f{nE@yDv!5j=MxMVjk;=ebL(* zTHc3Y&nMYwraM!82}9Dc?N1F`WYQWPSXp>rbKi^;9}@n6#DtA^WQ@4rmdNGJhT(R@94Jfsv^6b+<{WWb{q6zp2cKxcI5)n z#Y3YWAJOaXjA#&t+v6# zny~tIKU6QdnTyo^QtjG6P%NBkE{?_z7ds_xj>r3Mw3wJqwXI zku+K}Y&cIxt5DXFW- zp0B&zH2`2RIw1aB>;MFZ9P{i=_8(sOA^N0C!9 zsikm~a;42&voD8hzD?2K>v_k_0nO4TohgQ&rN#H5V7-AKq3=iu({>&w9F9lK6;jk> z0J(%DmvyWm+kP1^%A^E&vKBD!pFnEIi?*i<@pj=#hrYVK4N_2RvV0Xb9&Q@UXr4cd zUH!#EIE(Vl$uEVRZi|p;)@l+QyQfbX2-8nLQcM6zja>nX@IZ-)n_R+xU67y{(iF*Fzz98u%(Gt(qjNXmQ^8uiPY1xGjY z^)PWyiOkE#L6!mFrr|J_GQfq(d94W%+{*Oc!G z=xDN(*Y`gb*s^s=c6%H;9Vc-w+SWi%!`2|ES2e$n(S!cqM`|GTh1=a)#0WrM4oQHc z5ZWtI)W5%A6mLLIaiOq>XM0{=M+kJk9%Sm?#}bi$b)WXK;k^`tZaeN|zrHG zsh(_1|JzFVRvs=3xE}3uIf%L9%FlkFTd_GvFcaJ5lP4nn)e{+@JB)FVQs?4+)A3?t z>6ZB@K<_X?{<=eFcayRb(lbbopSL}vn%^ZMuOq8#51GahIEgFp7oLbIBsNWObdvwl zPR`?h3jyXwx#|D2f+{xv`UH`&L3(tO$n$U~*v)nT(5jEWL62aa8gi2)_%i-eM(no# z?@jvIaNbD&?EL2%QkNsJ2De`oKKl26mi5K#?~K^}r|kI~o(K7V$}YQz-SfEJ_H=uy z$(V)q%;F1RNo@5mWqU5RwX?W1%v@@wh!H10Sc+_ldajOv5-mz{G@h4$X)#&F=J}Iu z43EtYog0HK>JGVZTPi2EpYEE#%hs=icgjP-nTRZPnbL8yDrd&5Dy_2BJNxxHhXDKI z?0n_m-_jB(>%|^Yp2HUusU*P790Sxx*a$bjoUL)Ap9Bdm3p}W3rjd3_ z+e(`#fGd?md<;5?rpUfTh_{2(O|E4lV`S?}uNF}DPc3O7-xi{8QRh$z{7r*-Z@E6w zEbMP~Ex%AnOldH_g2U);K@WO_-Z^mTTg`WOU2fS2N*c_X<$Q)~zhLn_J9~~5*E;36 z^~o=}4fa`}BL%97@Z+j?Z#Nw@Y@g9r)s!YphFf?C3U)D#K`=oif8!XD4c_Je7*I9o zd!P`ScLM&r=)v%;`2YV;V+>%voQS7jBHX0xo=pg1X2X4*AsVSdk)vEbPan8>GNw9$ z$}q_o@K6+ItW;~?jrg;RKqbP&lEGdq>TzRs+b-BzWb_QRczMA7`VA8cf_O&_BQ-Mi zZVOr)`y#{?Chb8|w!n^&6CH>FP@AD&v78$)Z$T1xSB%mJuhbFWbPfXZtla(Z+`akA z7~)Y^w>(+G{MsSwB|nOc8riAnenla0o89hA>?qU5)D$nD-u8UmJsoBkCfGi51~Yx+ zMPPzzP1IfRDi)Tl;x4-?sngF|T*zky<**Cp1gV(6%Z$I(7U!L7MjMtS>*|)-i0EK) zPVQ1`PkX$qv19=;;o6GV!PDsBm~hjU8l(h*6Z6LTGEL6o7Wb|_A}+hyPdbXQ@Z6dW zmH1}&4R?DR0{W8d_O=bYQQh%Jhi-A~+IG!D_axMB`)af!NgF-NQ0eF-nv5k(Q~NpvCxEH_?8?$F{Fls( zb7KCXiz8J>mC?)#fb60~*U@n$JJPlB63&)QKN8gIGkRS&Z^V!l&A}}YK|3>=wEJ)*ibs0964zRA z>-WJWxzC{sq%{*D{{`zcC|%lKM!zwR#j z8(oJUG&W%@AcW%f(o;yH1|J$%eOEqI_r`eM*?IqW_McX#M#L_A_3HJICZJB*4qU}h zxBOF=eW+fSt)vZHI4X55|4Pai0ig);f-yKPP?BD}l)dk0^UGntNHkYXP=5f0Xv0ZA)Giy_#N{}v5|ZvXOw56DOpC^ zt*qPZ7w!G#&TX|57o}wbOGq-F`^9agdRAMZN0E33y%|>{-ur#J!wKUI0>NBik&+r* zlN{Kf!p4xh;~nmTllHnj{v2`Jlb{^bmtn^v*w2v57vF+G5k?yR$H>zMor_=h;fvp~ z_t?mEi}ZwDmZ&h2Pl$3m+34gXOpmD=#N>m2uRhI(K#UP$I(!6RrnzSae69P~doF}Q zC}!Pv)%u0BhS8uHj|1MJ=S3JI*`z_VWt_!is)^=rzi#vd{;vK4mL{t@LcL+~_v$Ds zz1fVB-=(np3zl@cI+=r?ijR&Wyya2g*8oXJn|D2asEo7T`&Wt`qt)N`NvF?IkIZv* zD=3`~=Jh)-t~XCqU8kCMX7tbVB~Ln!dEGvKJg0l!Lmz zAv*=0TB(*9yk_h=dtWc?>e4!Ai9dVYl$P(Wr)WIj#cZQ7)^hxk1KhIBnATzPXY{Kv zYq8ax555=*5$Wg)2x|v!Vj4DP?Jh)(MZdb1abB}AKKw!pW~6Wi%`@C`kj}5ydr1QZ zkvVe~3h@@6^v+_`NjnLaJ=CZ-U}+vYZyfw(P5YJqGK_pOq87B`lRw~B`FX5Af;FGM z(;CX(thTY#5dw8{MG9QUAcyVM$NHIGL&QiLXGAi_wKJ8mZy1k6xBU@w){IL0jkkf1iQ0FrC*QXnP4fK%Uf~+M;{II`y}=7 z1}bC2Ltn#x-}_9poM5E}645O{Plim&LQ#a%1NJL`o#%C5FiZ;h78^a{kK9Z&Y zPW<4=v31hkD7fCZIg5}v<7qe53K7{fb200qg#XtARM%*2vOZw+YaSOC7pCA4blbh~ zH+}YXBk@dBr6?MFoDBLs*Z5=t-T-@otD}}=R@WTALo7;N8ffC(eViWNIwQ$Qu*1X4 zrM#d-wJg2-_ps1wi*Gc}z}T(S#PiOV*N69TXC*QV#n*;)Ufkho-~KTp?lMakeQlR0 zS}Ft1NW`wAcNU)P--$;_KJu8}2K*)Ijz?qw`soA< zlT1u;(j6>b>CNwsuX>5EeUc6DT)r0SfVyeS6Lg0!V5S%D3|XfDLf~UvQY2#Q5*LQk z%F_p=z_1-Z6m1K`+o!$S${ijT?--#_!@g^_5hx=-)B{cuo8#4yJMuil)*e*09bPf9 zY_i|YF^jC=u!+pC6Gcz@Tw_=&driH8zbHnV+CV@17KxUz@|DDV?3=cXV1EdCggd>^ zES{2us#(mH$}i^ARf_EJLjMm`c3MMXLtUnLUE^yJL45MK&-2{f^NZ!W&$3If@0jE$ zT4{cMu}R;M)w}-aSt=!?>c3h>y~J%NQk8K~4PnU{IGStVS%pgD1=u1gvUWcRlor3K zqtW`$<8_iDm4Q-~=|AME!rg$mYJV?p)U?TFD^X0zd)(&6>r@?#iObu16ep>NR6Gb4 zp}~3VWiKA0qG^$^l}yFr%{B_|JpP4Qr%L4$g`%50P_6EWQ=`mI15xh-Q?q){VC@=c z#?<~_6jr{OcHBdoYky5K6EX|5Y(fr>#sI?y?+X?*_|aZ#)wCJ+L{|3%qpk(r4q-Fwyc4n0Ua%0u${m zG0}B^H4s_zJ_!^3$Ub1=0TT;MbhgAqF9;pS+5?~HdWYNjfQbqwx?5smfN4-2`;VjW zjO@tt91yRZ@Qe4D#kFEG4%>A;+qKs7{Hm87%r^zOJ)Xz?z-9&fRz zC=7nD>RtLOS}?9>`YyA4_n%ob42Jd1Aan!E5B~_+fw0CBv;FXokqe1d>$tx42g*JW zxuF?4K#J`z^rG=>xczM>`towd;nFZEJhQ%x@bbvF02R^L7kxqN3EAVSb@F@D6W=uf zhPHJ2Ec?J*jV`Y=9x!eg2EmRFFt!&wi2qBr9H6u4(PkPG)zQG#dc2fXPLloD#JV2& zk?FG_06Bc~Ee|~3a(CQv$04KLqup}5w&hNJ%Qx}f*=Y9z*R(v_23c+M-S?Td-nXV{ z_ai5G%)4*ZJ!`hv{jHr#BFvh=!cgcWHrjuX8`#bi$ASMnW&k)f0Qk+GO_HQK7Un-<@j<-EtS&<;%p(Qp`# z#>=_l9VzF6+ki7!j%#R{&C)^R4$19P9A=l-*XW0sKQw0ZX|f!oiP7>534`ZGlFk-a zr$d;AaWjgI`Es0QZR3Cb>%R^0Sr(6PV9dlA&ZpN#ngNm#u2Y*c=_hA=VCTl)pf6)_ z1v_p4Izt<@F3EmIi}X4(=2JuyV>E$wqs+*Fp^UHLa2T)Oz)G)DV**V{qE_BzzmA4%kRAta6z zj_Yv}FXAj+9LMNUFJzJ44P!kt@%HBI7${YCECdnwyKAGrJIF;{W?Qancda0W+v51m zFF*C(ynOxhn|J1QqTQzu3*&|NME>IJJsF9CWK0KKekQ_!54s?B#mtU9qVwbUu0Ngf z))RmiHp+=VwTJl0+Ime(;+zQXQ$ z;~0o-G=A+;}mD!LGb5?#xeCeodo zRN^7E1t1dShN*BNkQh3foAiuRQ92*IC9bB@@RH2uSLxur+HiyT1hqLVW>M)EA1PrD zNW1curGfIpR61Uj<43qF`3BM(AVEo!j!KUu)Ser-C_o$PDNUXmI4Zywx8ESjL=^N$ zoQEC~2fq8Wxt#MkcT`65g`D7hK_EVICZ|qYijV958^u{6gvDJ@G!=6a=1H)`NfVAr zRfP0|m@lLg2Yq&bPs~hA7!9bfdc5K+7f1Ln{!x7@n+`7-2hLXz*x+K}jR9x?29>{} z2Fpol!)rUX7^S!mM?Y{m6qhDQQ#0|aR*(g8#`OFk5|tI=#bQnngFn*i*+TF*`g0;) z^wdLzn}=5mZKdn@ATpuv56wOw(l;aDaTqdjl1|bH91PqeNCsHUVo7f;Aj@$G4E&JA zeC&9>sH`Lkn8V}-;oT#69)0X z>1)>y+ZtLS`SgAI`Uf$ru}3!pUU|6;g#0ppM`A+x`4j_X%k5#t#BUGjViViB$~gi;a3%aLD6Wy1s&1qr^uJY89z-V^@sz-+xFmc0@e!&5E|yofC&QMHvKAOA`1k{ zC}g3pKnBir2{1>4)OT1Iq+T?Pj>;ekQx{~6pl|!Orvp*MOy+o=?>`8Le9I5)1~55p z1x)s*fyuG0a7!rpf$5c?T20*gBD10Cw zxgZ-h0LgY}Qu~7vvq2Kw5=+iFbvrtdb|sqEarOszztmH4YhCr zJP3W)4;}C6p zkF9iQSvoKg8U&H`Kwx5?)5txzp1p4{1=jXFUQniZe&Cs54NL(uy})A5uJ@8345Ojv z4kLf42NR$Hrj2qQ22AMb)&wReboLA;o6hGpV8UuoWiSOu2kT%8(Fen0yAr)&GzfrM zO6IJMhB31>`i< z9wx+t2n&=w44iz&wwg3~(Vm6cU|Sv!J@Lyto(C6|21c&wc$V*YyRzbbkPaPp;M#0Z zhY`mCkZhUrP%!esMpis>_AD=L%R0Djg~M|l7MQFKB&wHd?R+Z@qQtd=*aneS4O{8@#8zbBR-|SuaJKG*Q;b7nn2dNfHp6P<@ z$f5^=lF!^Ga_892-WB58at?KR=imiY0~3HVumet@cT?geer%^<-;QG)m=FzY)Wts# zm?F24!*l7r+T*PhMcZCW=(=8(Xd^A}fQ%prqxP=61B=pR=*CIl#f}b2hy{)tc@M)o zcr0`qZzX1t5VyyB$zj`?EwI{a)l%zOI;t1=W*7v!n`*t#XZ@iU0kBi8sWvj5z_*-- zIZgp%jhY9C?U%=kfS=r!=y_G5#~i<=6n6okU=so9c0En0l?)>bcnCk}>rN9$wYC%T zgjx?^Ub2ElATh_3?YtoWz+k;Xm6Wegq$Al~lk`7m(eDDmCcN|HbWCPah4 z3fzaXT?9aIn{-6Mp0%y8)@@IdUsXRa2BfDhHUX<|dcGYvJFD{2em`{5p{KJ1ax4g0 zXg`p~6-x#*9VXXz_UlH(+^roSs`gsX3*ow|A0|66y%2DISK-zj*r^-F3`mRaFd-)y zSRr?RdjK$nw(t3km(!l1el`nsy_W)z9TT8>ZaUC$!S=2XUIj?5uLF{2dY&Kb%8A>3%X9q1N&-s@BvegVQRqF8 zU<)wvnt|l)mwO1DXj>?SZq>QfML$!g4yMR-eG7JSS3Oa`@5b>kwA?t-9wxl@fYv^c zHqXI==1s=hXukx{XWsU#c*R)DT)VCoXRc%VuIsrwJLILofTf`~NE1tYm?G1&xGwUM zS!)Bf-=k!ETiY&xPbl9?kq1a!(>*ZHF;SA**|Qa>{$LoV!ywjq@^d8c!iT$-BClDl z<^HyN^&;BZJteej{`?{HO-~0XC$ubYS619kt<(jI84PS!dz#?pLvtK^5I{kvtdW0k z9BZ#}7s1vhb-TidyTESiU=)~9U;$^bD>-dOXIE-xIZ>MWgT(0vwhl(VW7?i;J&;KqIt?0yC3nvft?OC#_Ex|$lzht# zHC(ulxe~nHZMBv)z#PVL7)DxAiO6Zd9*S)U;5uxQl=ke)ZwFiZJpe>1#wD0;-MGYn z!SFqh8g}KvF>uLE!hRU1T0N^DnwDz?*29=;;WG3ZjTX+H=^YEVb$b9h=2h;b(5l5c z1CRocXM^1>fGkZzCyDJ88t8z8NZjuE$u|aBS zeI^;^T@9ijdLSry&8~R1zh@TDcDHvPimFyxC-gL|xb6C80Q~CCB+qgC!O%~9Hyr9n zp3Q7CjBNIBBrj_86Zd?V?b`!+TQ(1|xRQ~>#S|$uI03KBM7Z28V2W)gNu0z1G|=%6 zL{x$2eJFjnZLvnNmhnU+`_#1t*0y#Fpm3Gom>>k6tA3t9w(=dz-}!bzM?My(SXM#j zK*Y~$WIdElJn+3ne%ufDYfq0tX9wTb^5qoykw&n!F};9g?e5w-NI+l;Ew}HcI)y8p zR%ZGB15v#&Siz3_`wgu1xzFL2i3C_hm*;4VfN(3Bm{-Ny)pQXiei%f_U^q;43RfFt zTPw64$aE38Rx&61O_do4;1xCWF2u^iJc4{Drw2Q+2SdivzE&qrDBxM) zL-~kfg`!4gJlHSy;5q)b2g$7_PI0`jo*WN$F*0q-3H)8T2gi$&aG1uSmuk5OKmwn6 zj{8v7bm*(uHD%vnX8!g9lU?yk@$H~4Fag&90+Z$WyXuGr{U}YWpr3fQqlFS+0ka<} zLLFGVs8O(WqJ3w32)8$vXJi`KK_vAKq8mB?*B)&FalaK%y@dxtqTIrbL z27A`GZm%Qqs|mdV7ieHj-RjtuX=6UGT{(yT&`Dg(5%<9Md!MYg{ylsqr?1K&@EmX>)yET9a?4`pAnnwf`vXCaFG zz}r5mwoLH?FYq-K&$Y=|yR&af6b}ZL-A_YZ_G69&0ka;+fyiH>cyRWf?!jklYfYZ( zRSkh|$ilj`Yd9ZEQYD^X7yV0H#AiLhD zx_U*9rHP7hfc3M1Nw;_9^5TIr9NO+6cKTW;v%+cF>&h!QFdQtra-ia^qn& zm4N33kzcpiy398%z^k1TM!Ejbj)Gy*_qE=w91%k2fl^i5uG5^kHFASJr(R>9nv00J zKFCou$rku!6ZrJd-Suh$F3}$bea8b{PXi^yf*@p(|3H$hyCP+j=kM8KYg^v7x;!-e zRbtdJ5V>YrhaE9JxaZl<`6kmO>Gy|hkcK)91mS__c@LDi)keS2CgVjI?3vfIqOH%9 z=T{?`JkQnm^t-@pAW;o>4uFom)Qv(%uNNmYz%ru`MD+YHY9xBWo^=pj=mpz*=2f$T z`jM@Hk{@DL&~Rse@h}>s{bVp?UaCD!hy|hLK9ttgW~|v!g$350nLQmx4*T6(EawwF zoT|!k<^iChx*o!_ER)2GU3qdi>A}EHlHoAWint(|c{cNH_CO*oPqCx-`r_>R9_e0W zxw@1@ShG?zYGBOE{wlU(IgTbpC_)giYi9@l;V=pPlsQ(vuXFI{r#6a$2QuASjUi{D zyL<4ar6YZe=jWbJ$0;v8hvi@AM_V2UhVCOAO2At4=hjz%UUFo53r%}H@=tp6q zgB)Q7=2G+?h`VS^Q{=I5pMYad7=1e2f@+W_W~b3uJR;jOEp*`8ISL^3qQ2{3dPdzf zMrh$N=YcYbT8$x3duB9jZiw89A9O05hHJC19zGb^c)tXWy(>vf27ctYiRasW9fS}A z94lhZ|b^@D$R53I){M}ihvC4oDC-)4xRomWCMR-H9!dVJFp+w z8Hy@Ap!=y1a-wj1V^mqD0#2-6TgHum3I!1yga5y|tKDrIhJo+W2QVa363K@d3V%pL zfwn8sZo_`;-ACC8u*q_UC>}iIPk2;#|@c0;FV+;s!{1&=DSUrR&e1T%}{ngyBwEmLUh(E&gkfFTiib{XGSe(4 z&lejqBpAf|Do*n-hlRX8;Aj%sHcBSi7bfFxABn+5nZ(Z0+QmQX`54+kGGX*C^`;+v zO}UqU1kNCbgM=)!eN)J?^IEb34Wh7;jdfHjm2`7N!|a<4DSJJeA~U*FP4nhGHMoO1 zrLScWTHN!lIH9eH!mhJ|=E5|E1k?V}pNH+W77_nxw7AG633AuD4mJYgv3V_?oX=zN_ zaoejAcz;8w@WU~NYV1R91I@#rTKS;}laQWH##AjS=Y$_aIH(KP18$ooqqtpB0#^y}Fy??~Fne+Xu-?KZ# zZ%^_2hEMwV`3ot!L|X^I1ONin<9Pr8 literal 0 HcmV?d00001 diff --git a/eval/dataset/arms/copilot_v11_deep_r1/pr4923.cost.json b/eval/dataset/arms/copilot_v11_deep_r1/pr4923.cost.json new file mode 100644 index 000000000..1483c34aa --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_r1/pr4923.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 1642.1, + "split": "train", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v11_deep_r1/runs/pr4923/run-20260813-120515-00cc4c", + "recorded_at": "2026-08-13T12:32:30", + "attempts": [ + { + "invocation_id": "inv-414cd709", + "outcome": "run:run-20260813-120515-00cc4c", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-120515-00cc4c", + "llm_calls": 102, + "input_tokens": 1144475, + "output_tokens": 204843, + "usd": 0.806164, + "source": "spans" + } + ], + "attempt_usd_total": 0.806164, + "llm_calls": 102, + "input_tokens": 1144475, + "output_tokens": 204843, + "usd": 0.806164, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-120515-00cc4c", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.8062, + "minutes": 27.26, + "ci_minutes": 0, + "input_tokens": 1144475, + "output_tokens": 204843, + "tool_calls": 204, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.9746, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 102, + "by_role": { + "agent": { + "usd": 0.781107, + "input_tokens": 1136468, + "output_tokens": 184061, + "calls": 100 + }, + "reducer": { + "usd": 0.025057, + "input_tokens": 8007, + "output_tokens": 20782, + "calls": 2 + } + }, + "cache_read_tokens": 10067712, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 4.34, + "gate": 2.88, + "review": 1628.03, + "report": 0.06 + }, + "time_to_first_result_s": 1635.33 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4923.trace.json.gz", + "trace_bytes": 24025 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v11_deep_r1/pr4923.md b/eval/dataset/arms/copilot_v11_deep_r1/pr4923.md new file mode 100644 index 000000000..809998fca --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_r1/pr4923.md @@ -0,0 +1,40 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4923, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head 3b2d87c9c3bc) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 4 finding(s) below | +| Security | no finding reported | +| Docs / comments | 2 finding(s) below | +| Behavior / compatibility | no finding reported | +| Correctness | no finding reported | + +**Validated:** +- [validated] read qwen3_tts_talker.py hunk: `talker_mtp_graph_safe=True` + `has_full_cudagraphs()` gating is consistent with the runner wrap decision in gpu_model_runner.py:209 (`if cudagraph_mode.has_full_cudagraphs() and (has_separate_talker or talker_mtp_graph_safe)`). +- [validated] read qwen3_tts_code_predictor_vllm.py + common/qwen3_code_predictor.py `_setup_compile`: with `use_cuda_graphs=False` on GPU the wrapper still runs torch.compile and warmup buckets and skips `_capture_cuda_graphs`, matching the PR's 'GPU torch.compile only' claim. +- [validated] grep `has_full_cudagraphs`: the method is already used pre-existing at gpu_model_runner.py:209, so this diff introduces no new vLLM version floor for that API. +- [sweep] grep `talker_mtp_graph_safe`: only qwen3_tts_talker.py:327 (this diff) and fish_speech_slow_ar.py:198 (pre-existing) set it; no cross-model blast radius beyond Qwen3-TTS. +- [sweep] grep `tts_local_seed`: serving_speech.py:3694/3706 injects it from `request.seed`; the model comment (qwen3_tts_talker.py:334-346) admits it is silently dropped under full cudagraphs — basis for review_comments[0]. +- [sweep] read gpu_ar_model_runner.py `_should_use_async_omni_output`: async output requires `omni_prefix_cache is None`, `async_chunk`, and no spec decode; the yaml disables stage-0 prefix caching and keeps `async_chunk: true`, so the new flags are coherent. +- [sweep] read gpu_ar_model_runner.py `_maybe_run_eager_omni_postprocess_before_async_output`: eager postprocess writes `hidden_states.last` into the buffer before the async snapshot, so next-step talker_mtp inputs stay available. +- [validated] grep `omni_pooler_payload_include_hidden = False` in qwen3_tts_talker.py: set at line 317 and duplicated at line 353 by this diff — basis for review_comments[3]. + +`vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py:?` [major] — This hunk disables the per-row generator fast-path whenever talker_mtp is graph-wrapped (full cudagraphs), and the in-file TODO concedes per-request `tts_local_seed` is then non-reproducible and MTP seeding is silently dropped for decode batches > 1. `serving_speech.py:3686-3694` still accepts `request.seed` and injects `tts_local_seed` with no warning, so the API continues to imply the seed is honored. Emit `logger.warning_once` (or return a clear error) when `request.seed` is set for qwen3_tts under full cudagraphs, or document the guarantee change; silently ignoring the seed is a user-visible regression. (evidence: grep tts_local_seed -> serving_speech.py:3694/3706; read qwen3_tts_talker.py:334-346 comment; read gpu_model_runner.py `_init_talker_mtp` (wraps talker_mtp in CUDAGraphWrapper under full cudagraphs) and `_talker_mtp_forward` (generator path + scalar fallback); read tests/worker/test_omni_gpu_model_runner.py:317.) + +`vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py:?` [minor] — This hunk declares `self.talker_mtp_graph_safe = True` unconditionally (qwen3_tts_talker.py:327), while the code predictor still sets `use_cuda_graphs=current_omni_platform.is_npu()` (qwen3_tts_code_predictor_vllm.py:39), so on NPU the inner NPU graph capture still runs (common/qwen3_code_predictor.py:532-535). On NPU the only thing preventing the runner from also wrapping talker_mtp in an outer ACLGraphWrapper is the shipped `platforms.npu` `cudagraph_mode: PIECEWISE` override in qwen3_tts.yaml; `gpu_model_runner._init_talker_mtp` wraps talker_mtp whenever full cudagraphs and `talker_mtp_graph_safe` are both true, and `npu_ar_model_runner._capture_talker_mtp_graphs` confirms the ACLGraphWrapper path. A user overriding NPU to FULL cudagraphs would re-enter nested outer ACLGraphWrapper plus inner NPU graph capture. Gate the flag in code (e.g. `talker_mtp_graph_safe = not current_omni_platform.is_npu()`) or assert at the runner level. (evidence: read qwen3_tts.yaml platforms.npu PIECEWISE override; read platforms/npu/worker/npu_ar_model_runner.py `_capture_talker_mtp_graphs` (ACLGraphWrapper + torch.npu.synchronize); read common/qwen3_code_predictor.py `_setup_compile` (`is_npu and use_cuda_graphs -> _capture_npu_graphs`).) + +`vllm_omni/deploy/qwen3_tts.yaml:?` [minor] — The YAML header (lines 6-8) still says stage 0 uses max_num_batched_tokens=512 for low first-chunk latency while line 48 now sets 32768, and the stage-0 inline comment (lines 41-45) still says 'Prefix caching on by default' while line 46 disables it. Update both comments (or restore the values) so the stated intent matches the config. (evidence: read vllm_omni/deploy/qwen3_tts.yaml header and stage-0 block; diff hunk `max_num_batched_tokens: 512 -> 32768` and `enable_prefix_caching: true -> false`.) + +`vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py:?` [minor] — Add a unit test for the mode-dependent gate. qwen3_tts_talker.py now sets talker_mtp_accepts_per_row_generators = not talker_mtp_graph_wrapped (line 355), where talker_mtp_graph_wrapped depends on cudagraph_mode.has_full_cudagraphs(); no test file is changed in this PR, and tests/worker/test_omni_gpu_model_runner.py only sets talker_mtp_accepts_per_row_generators=True directly on a fake model rather than constructing the model-side gate. Construct the talker under FULL vs NONE/PIECEWISE cudagraph_mode and assert False/True respectively, so the gate that drops per-row generators (and makes per-request tts_local_seed non-reproducible under full cudagraphs) is covered in CI. (evidence: grep tests for qwen3_tts -> no changed test; read tests/worker/test_omni_gpu_model_runner.py:198-356 (existing runner tests do not construct the model gate); PR body Test Plan fields empty.) + +`vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_code_predictor_vllm.py:?` [minor] — The anchor is exact: `use_cuda_graphs=current_omni_platform.is_npu()` at line 39. `OmniPlatform.is_npu()` returns `self._omni_enum == OmniPlatformEnum.NPU`, so this diff disables the wrapper's inner CUDA graphs on every non-NPU platform (CUDA/ROCm/XPU/MUSA/Unspecified) and keeps them only on NPU. CI confirms `buildkite/vllm-omni-amd-ci` and `buildkite/vllm-omni-npu-ci` are FAILURE. Please confirm in the PR whether those lanes are also red on main or introduced by this non-NPU flip; if introduced, the platform gating needs explicit non-NPU evidence and scoping beyond the H100 benchmark and the NPU PIECEWISE yaml override. (evidence: gate_report lists buildkite/vllm-omni-npu-ci and buildkite/vllm-omni-amd-ci as failing; read platforms/__init__.py (is_npu False for cuda/rocm/xpu/musa) and platforms/interface.py OmniPlatform.is_npu.) + +`vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py:?` [nit] — Line 353 re-assigns `self.omni_pooler_payload_include_hidden = False`, which is already set to `False` at line 317 with its explanatory comment. The PR-added block should keep only the two new flags (`use_async_omni_output = True` and `eager_omni_postprocess_before_async_output = True`) and drop the duplicate line 353 assignment. (evidence: grep 'omni_pooler_payload_include_hidden = False' in qwen3_tts_talker.py -> lines 317 and 353.) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v11_deep_r1/pr4923.trace.json.gz b/eval/dataset/arms/copilot_v11_deep_r1/pr4923.trace.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..2bc462c5be5a066ad9d082d4b2f28a22978caf57 GIT binary patch literal 24025 zcmbTdQ*b3-6z&7-*@ePTNu+sTPfJpViQ-kGYInwQ!2d-`_m zmsPuV)n4DXNaEnY{>Os&zV5sHwJEhq1ES-Ro6~Vqfb(ovClj1FNO%O8g|| zDdPe2KLu7B)q+;u`qWB0zx+M}Gc&EykWx+m5#%4b_lXX0k0;7$Z>H@Gw8*}H>=L7Y zy^n$8@GXXy+j0@ld#6gXy|E0e78=F3OaK~_c3msrX_n%iEd@$Tz8=?#0=*+|(`)RP zl!hx5JQ=Y1K9#>g=YU-!q3<`^?V%n4E+%u`KNabbheTDv%FTC2kIKRW{?8D%rc&(G z{JobN|6Ehb{g@)NHV!hKD6YpGg29` ze(4N;qoIwta49wC2|`Q~yfjOdN2iYgt#S7oepn{oy7n zK6PcznNP1_Pi@{*)U08=#Rm`4Wx!ORNUhAXIY#Z!@>HkC^}hjXjMBma!44c#^}0AH zenMMcD%x@o7eb1z&B%Io7syBsOR8!=EdN!q5UUvNTyL(w{KX4d{a4@`OoZrM^kK93E3Efv7gM6D&vI?pr}<`jp;E&64>j^WJ%P}8+AP~+c-8v z%WI`iWq>H5%&HnG&~+Ayz;zlsC9M=^Cfu}b*sRO>c?3r zF9oLBl68;UR5o44Bv?T&rChAal~A&`CtSuHl$+*%_mFgrGgtWic%bQ>!=%bDlcnQ% z*t)b|knJh4oUvsEFN2MgezM3EXGUGYGRay?WDxCHxd^pRDvQy<@soDjSN-jzr(xjp zt}3gc-|x-MQi+g`#OxQjLu63Qa zztB|Z3(g&f_nRT=i>ctb=AN;z4|;rLaTNr}%{OI%aKyjA(BU(lddZtFZ_ff&H&f|f z#;sLW)*T38;Dd~y$eOmdj6}hA07h0o#*Y>ek8)&6-Xvn;(1d3^bP7~7;>rRESh*O5 zr&%+BC~FV4JphM5F;XP2720KjFWDC*m zof_rAu0I?*6^0FZeSN>x_XwIN=#3i_#QjhOqd&JE`Ubs4-JYiTLO~8q?WuAYz16u3 zo8uybUR1&2z*La+cuuNGlzT?*Ea%V1FoK=(pRLmN0-Sxp-IdV#KfRNOqRWH~#@dj5 z)w8d7qQg93{$}m1-yPW{w??*{+rT78X&0fmkedQIid2C@1lBAE@2<{#GcZFQxQNog zG^6lK`N%Mf{2>(@jnE@<;(Tk41@B~o!Ogu8uRd86i{t^wI;_Db6WsE{eK;#FBL(ZP z_MjQx#^`zTwjVIq!Qn1nQ(?wCfPi6(H!6`g_^&4+g6L-tK3hnEU8_SE4=jUe7-te# z>o2i)H$FRz$IZ0PynrW!tA&=msXfGN4GC+%klDjD_Hu#0A}DCQ*^j}DA#|sg(6aKsCYCIXj(HjN*S;(~xcR~+K%hIS&pz_%NwDL?>S zpnJ~+Nj@T<)*(n5)$$8Hsj|iOn9+I=hc&t(w5o8xn+2YD{s%opndcX&%$w^in?DkH zn;663Seah_$Uheq78j;S+R{E$g~nA5uj)D6|*(3B6d z2a`4+xesM0;QJE=oP}*czC#$QoSa=r6*utngJ0q(%$VwSh@GZVFYcjbtaXyF1~zRS zYV3j6JM`k23b;ozg$`OE>31IUpm*ouVPd|9g4*RcQ;cCxyK`Z)U7)Kys7NzJGM)cU zWOnZJeQI`EWvTdLiFtQSFK01KuDcz#FOi)WXiKXamcSo;oMQn0D6_yvVGd)r_;g@H zI^UosX68sR5Q73Doaq95X=;Uk~JIRkIH;JSVkK}zJ>-hj1aZ|5w;px za!|~U#;VL#1SM)x)^9M~6EmL6D#s%<^goUok&euU;OHKV;}MQbN5Kc@RD+ToFZ8Iw zHvah^Nu*67S92dc@CdCfV&G6Xwulm71#!Wr5Y8nJe0cD)H_DBLkfhan(M!_PLyUN!_?i$HfP873z@OUO9sO!K0f4(Wu$sWu%ud zklbJ@l(OgjH9Rcu^R?YlHTQM5J!NfFU{DI{d0&h$x>1a%?}BTURYt4B(dGJuek_Ix zZDLtcI0*w*P5BTD22IhApnd(y3sk2->-&@87iiQqPFfOHG{G+(K>U~VWx=Dk7DrGG z7}$%qKFd-eedPJ&&NWmuWSi4&0$~wIF&&6MpF2@4`SRyrvA_pghxCKNJ1KWt{}YP{ z-T(9P-TAI5=*a5!{YR@rpLM#`P(GfUA5VvqX~>b(9${=pXeC)*Q==QKL}T3tJj>+4 zgrw6KOaa+?mC1$0CzDO$dFNVAN&R#NdE!N%ou2*8k6CAlACDlm$cceGZ^?%b0DSj2 zT%!{9pr0sC=sq0j&z3*Hu?Nv15tvI0C)Pn=#9k7$^N=MB`pkGUQL%hZh_muiux%PL ztOwGrawth*ALx05TloX3SC8WGosfzv=j4Z@qF)x+jB8{IwIFshvT~iQ0Xd(nPO>$u zf|AH&xk7;iQgLn?VuKL`u~k-AE0|*v6HMgI9Y06=2e%Dv>mihAxUqjh<=Lkn`P9q= zHfUVjqiJqjM$oX`JM00^+nLm+We*n=fekinCgO{lzirCKD{5gJs()+if0NK)YMr0a z>p@r%hZZ0#j6jWg$utb_ijWDd8bI%5FAt8Ciht(~_sbQq8J$K+FjN|xziIA*H_J}C zf@|X)T3h!d*J~?>N@5IwV!;cK%2*iqIl}!A43(1sCzK7gXmWED2G4e-Jnu0p$&m~0 zhUuZT@8RuC+;YES@L(j0SyW9Z5CnYMd)cFPxqfx=$NC`d9jB_U4vdF1&!*}WFv!MQCt+~l`@+_(@8jpE^7N?*EJn`L*Y4gMbM!el*d@o9S zE^M(8>3nLqBXL&JW;})X{OAOIRzif_^5FZhzv@)1mGivP2;o$py|Mc zb0>x<&r%NrJL>6fyXX0AlL*;MyVxVEANt zwfD>w&Z3obT}JvX)rF z9Lx4Q^{MAWksr!0V(>_-5@Rk58{TKO+LLY3EN{ah_B_kM*kj9{rt;ZQsE(1t^`;w z8`-~!IVF@-ZSZ#uGr-_4^B-|knzK!g*cC8}_4I;iQ+SK3M_|rf%&qk_RQ?AF!bfI96nmxLJ_7jgkBBqQvWu&QE6j(b`HX zZ2*3Ot@JcXYj8W=sTu7E?j}dh+ z0e&=eJ(y-xv^bFF3;3>j)H)=bL@z4jO7~BUQJDh<$oM!a2w~2w{xW}2lMc2fglC5z zP>*jpEo7%aaE5ytueLM;(y2o-`iMpqED^>-aKcY{PY(8BJ263Na4mMhNnId z>`aBY^ZkW>r)|fHwI|Y)>&?c}{Eh9PQq?Ikt~70R#0)Ec#tB^1{iLxTSLR55DSyW- z1bS#buZcL!*I8?5qC9I@bIlxE67xXrD2(ELv2^_{mE>rf;G|=a#=K-ny0hqhv-Qwussld!o__rGN`7OufbBE8jzvk?}_G+$?@^25=4wDrvx ze-tvSGJr(&*Khypkm$r&Dz~0ozA1=NIAWNNNr0u-LMzP-K0yqh$GesexuPxWg0#~% zqEqkGfu-+4GV{qiymy>?cXT1IHE?t2VI|6NY4zeIO0+#5^|~~jN#}p|cvXa15G*Gk z4{E)ETwCS2m3v%tjz#H9+!(FXQQK56_fn?G(g~{R1{3uHu)7`LQh3e5#!yu2s5&xg z4{$RUXlTw05cdjL=Xo5E`d_J_5;8B;{QA5=lqPcULlm8<+qgg=Rx7l*np>)zC*v*E zP?GQ`=*q*CiTPE~sOcyI|56ERK}l>6(~|4r1#NMJAJzOJnG?eO_d#*;-#s)lZ05p{ zZIO_uTRKe&k;xqHn6WfYbGme^I+$<~MAOCact{eLb%E!>d`I|-p)5K#Mc#rpoD+S0 z7vJKLG+SEs&?mM%T+II4P}lB0#FPxf#7ga+tXMaq-!l3XYpaJ-zX6w@04cmvv?lx< zo+)<)q3MwB7YbxrfRQNGY(jgdFk?Afcj=i=8}=P)tT_7RB;Pvj}?34ThZFy`0^mnwvgYWhK zoTL5kn0*gOpKwg)aIkY3%M6 z)~5H{)}*WBjY(z(@P8TO@IS@L1M)sUF*O+lljBgnf5M}>rOnz#t#7>LaL@(u*vplN zaEg{pAD{hlmviV7&~d`mQZs3Deat%=)Y|d&S47^o;5C+_%Wb7sV4o0i+-1pWk)(@~ z5_Pt((#^kDnzfUPO68HH@kyDW{pl=+3HluREwXD$&0v#OgU9kq8=B@Me4LDZ7(n|B_0)d zWBH!nGGmi)DpexFKOPwMxS^SUbj6!5>E8>k2!X>`5!>GPa@Az4NzJhHtN}5;!$2`t zCe9(VVOvNc_Rbi}r(CYwNz&kO_bfi<-dn;oxG%~l3nPc*h6PcrR3Vrg`5M#}**b5^ zjrwva3u=R)tM1^t4MP6frh9=|qwwc2xP%5gNB1!rrWrrtQz_42=s%n#H=K~uLUIC0 zbQI29QW+V<>t@v9NTa~9mGGpQRVlht{9)-v?gY4^lqj}hFcwLTju`N)cJ0z_ zEM+hJFGhfXN^l0nKx@}eXj&Zem{IKgUAf2Z%xEKQD=EjbE(vj;6L5LLkooaUd?r0} zu(V)t*bXfD{>a8765EjUnqOI^Wj^`WAP1Zb&j+4!1mp|Ia%c@4@I|%sDs2hx3yNP4 z(W*b~w3|7}Sl8(`r-k&xb69T0FQ2Ql^i(2G34RVi%)Gpwg(XXB}&a&?Gm&taN zY0X+|pCY62ok_vfLnKvW*;bq9lGdG&D~YLjBV{I-LNi+p0XPxX52yjQC&~x_Fsrd} zDVu^Zime}BNRYpU7mq+tB^|x1%LINp+**I9QB`=xYFCsv$5tBknQE5gSS>yCOqtSY z6FpW5FDE$V-CIxpL_ay8nb41|)(o|p<@Lkfwwbo6aX1q5=0z>h@LrABNYV-m>0@x( zuh-~%6Sp0G_O`2{%63;siQ*jM)+!##f)fqs|0c|VwaT4e32ik-*%_n{VfsX-{hkzm zsCb|EKR*j|_U9!sZi~w{*>T3uSPTBA#o%lDmYHR~8;MANQ1z|1RRnU%TaQ7~P%~0j zu6TT%`4j^9l||=jLI}7y!!WazRE)+)L}{E8&{HI^9&A8SqeqOgVdGtp<&a?{(w9A~ zG&y4C#cmGB8R1wAqk39i+EHQ^QycnmOoC;)1;49}Mmj zt+=GE)C<-OzWLGm#Pj1|%_#n5=Ftkb`e@44mAyo|nypw(*;MqZ zPdSs&eYH=xSaK^ASHQU07VxX8Pu^ips-bfKjQfH+dL#mZ_GC*uBx-fG=iK+q?oA8aPfjp2o(IYf)`- zhb`J^pF%nHy8={VG}mAlJn%OA*y$Tx&?(SBDn(k3_#}Mgkl#tm`jAPgp9Q6op$;hJeqr27bI=p`HG}cCmU9w zuV$=Huw1#Njs7X)%b}e3d3-5ayYdElH$v(ExS#SgdL#KpYYq3c2xUGu2Ep+;B%6wO z-{UeMfQmvT9?9VJzSMiNAnNw0SU~E3`=~YssAvxMwrjK69wU=?QAB^nU2kAI3ASXjem}CXJ#j5%ZB?rArx)`ocow|IzrPA33E5ypF+#Z%?arZoG>}4UdtwI|w z4gi~a)qIzJ+`GyFibnSJAK`(0VOPDo{L+SBG}?FCLhDc(>56_r&5HUGr-w5E<%jSf zo}nJoIcjHWoV#p;YA61}ioX4rBxfWRkSi;3=zs(*soiWcN}Dbt4Fv6IJkUqjo)N!4 zoEN5n)@(Xu!v*$ttiO_wTDTZX!IS))woQ3usauSJoqYO(_MYI**tN9?8=D*7_vjkD z!u)zd?@z=aw+cNn8;Nc%ncGKa=yQ6^*^|9MP6Crfc$cT5vpVnTpXV`XUK^Q=Lh7^w@#q?Y}u*ewS9Jl%e7)38D{W0UIQ3rdv*+ zaWs6)Bq5*xMQ>|Cy8hRfT)inLVLeImyD*~Wu~va==w`oI-NK*-Y!3#=y`1@UN^l$| zds5o@W=mN-Xmg2rF|6xVnYTUwQ*=is6y$xZcQUZX!vo_1yx@Wn??pHrryI4>YyL0G z+%=*6N+!9O968GD`DrYb6Rb|E1QK^2N=cC!LYi?WCV+?5r0A0qL)fX7CD}N$2qdFl|05!+m)S#>oAT2~%5-sOpxxLF&(JMQ@EJ#n!A9 zMU*`fa3Q4Ebf({qK;!4C%el2L&EiA;VD{o#3R8PnX*`g^`#yb>hmHIqmIc!`-_v68SnFEEJ9DS4ua^NyOk(bs0!9um2%$*~ z^p|*OMA|Yni133*2=ick(>6)eJTa^x+FqGB3l#*dITS(!f>b(}+^LR2pAxdy-`I$0 zuR?iI^>}iOghtO;n0(T9)stb!W=eG>F8t%>)s;`qz2vgh5NeK?TeShTDLEiSZjVc4 zY*vzl-LGM#r3W8V&X9yqZ^c1_(%e|7FI1CJHCkIDt zqUJSd$THzDn9((}9mvfb-SxJmuYJ>q#eS&o7`JA-^)Kxm$y!S6c1Lnuf)Lmx_Ai8T znXrGDKB;&{Gz7k#P@9mC0VWRBbO&yKrq5y9JUQ?(UcMyyEsupwCby(WV?IR??)_4& zp&>%S{#hm2xB6idj~)C=Dd-HKEoET)>sR}~i+u*D{xJrOVpvC-25ca0GQrCJEIwCt zKKLyb{oJP3@I~w|2gx}S{7yEb{@lZljK9{T_bdaMx0(gdscvM1R+}VPno*BN$@-HXBS}-`=mdG*(yivz3!gUa3=^R20}7o4#ZJ z3FKuue{fl;lZ?;_z|5d@@}|EWtF%K{tkaZ#Tyk!t?*sA?^?SyQ-)FC7@H6iAxNa}q zgI#9vGZF|Jc?29o({7lIDelNSO$HoqM!ceoeZek=G- zRG!ZoT(l8o7Zx( zHBqjO;O4zW7vowLQ-u1q26`eGn|`${ZZm=^NW4Ta1*!G}p5XlM1crUPuVyZXzMpmn zX~s1C`23Rud#mciINSAt0tEvFy}v$N8jOX0eBE!q?85GT;`Y8>vKtI@&lw2?+-*N- z?|wd-1J|sX3nRSgzW`XXqcqpRG~?0X3`WK??|enybA}t{*AD6gc}Y`MJ*H|b2MOO} z$tvxeib{J)TJ@4M7C!wNzETz(xas}ZUi(|hJZ+6x#>C+-0s0O!xqBn%i1PAk&j4}Z zxsfGF7Xo*Obg7c!=&HTUThvda@7jjg@jM5_CNt9^4yXfRSQL^ko_wn$5+Ta4nll3t zKTP1&2>Y%D-k_Y6LVxjuKN;K`xswQz4MLsY`4rB{1%5d@E(FzN214pJ>ZlJ0RRX$K z;B;t9qm}h^mw47oT}|2vVk+*BvZAi*q{1E+kkj4GKL^Klu&U_Nv0d+ta{XupABU8Px@`ZE zH9;ZR7-4dnamj2`$5YvGC&i;HAXj@O;--2YR|KA*Qysvo2y*CB&gkf}dxw4UhvxbB z{54ZjF#(RKyU=U;d5WNw!Is+YSDu~JLEu|y#_R?L))j?xqEFEu#rc;%Wz z*R$4eH#)z(`0-ZA;I;c?J}`iLaFb*y@pdt)i71qlHz)~##4VEIn8S=^eZD4t=>r13 zdY{R&hBT$xy5yIUdhrtPlPjt4Xq*6nB`!n9l%R(1m+v1y9MBFoU`EO`LpKmNG)nq8 zc>pL0G?R&jm4O-q>?0Oz$?^{6eT6jp=-wRnW3wXuP_@Z8^AN-6rK0hJNhLMhCOD*j zc8f*+QeCgbV)!#$SgA$6ApOeK0ZxRhzQL)#{k5RWlV8=JKw8kw@~n9S#gi!iPR3lU z_jo$|M5e>S?Hdr)BAvkn=KMLDd-=vLi14PX;CHGbgEfp{S@-hsApv>&ZwL{37U+}k zeagSIpkSN#PcPFfG1TlZoV#hBp04LDgA=Qf(z+A~j(y{&bZv99`e^-ExAa182j*4L zXRpeLNFDA>`b{7QgU4I)XJVC6%gT=nk289?qicDgqOi=F9Y!x%pWef&X7(qFI2_$Y8-D;%F58do+;_H%&Z6pUc_? zL%#m@D{VH1)KJYa_H=+j!%qiUoq2YSRpOTP`Qv`Q!aK5u8IxrYaBF8c0 z+T@f0Db}Lf8tIFR2yWT|Y75fJCswks^sLmiT9U-9r1I9YML!nam)GcVyoGAv4T@ef z2Bw9mv@##P(rg@5PlF293}wytNNay@Gvka$2e($h4JPG6_dn^!6AGh{{JLr5afhihb274-Ic+=5eO>I{Yo>qLCW z8!SHfXjwDOM$o@c8;S zk!JGQt-vS4&n#S*8I<`iW>3RubAL_Rhsdc->jarond)|lRAcegc3DE%_2 z7ODyCvgqq>DQ=+8|LbYVx@wm!b7{(XVUD@^j(kkKh;!(zQyorebwXMRo* z)iJd6QWRjq&>h?zFX{ioo{TB*mpf%gx4Idq)_Ja$SlZ;jDozRgIy)#L4jSZ zBi89a%+e1uAyRyqCDPGmzx4AmG4x3&x(@NYgxq%`(fJ>S;zEdLoZ^RV*TO=tD0EEf z<+=Luy!A@7uK~wXZp(aF9ZI*iiyAN55Qa`IX(QlA{ig5hV3A$>StBQP&j%sOzX>ln zf)Sx+Z=)8_+porc*wbMSh$RzO6{Q6O*wYx*-X*z#G+ygv57VI=QVlRw54(+e(K`(0 zCDkG#?Vc!a@3KQz1p%Sm^j`;;ArJpG?VyqOc#HM#K#_Hc53BU<)G5_ygZv|-nA)PQ z92}gUW$k#%boHn(>$S5g6F9ydi$yJ0aE3{qngG6>)FgVt!V^Vck;CE^ai9J7@0Gco zklEkPk?9@U_kAFdq(V0`<*rzzuKlXP^(wLR-VK2h`IRn46hTIw>jOa3JE**iOy&=FWB2VFah5qJKr>{&}fkkZt&YGc^n z<;EUFIz=X9Ri)b-E2TmR+zMg$iDTIkuypeN=-fA81^I7XnwhamAH7dX#J7JR0$G$# zESJGV_(w{Qi@qX-ID7^H#V4YNFb^K&}Lu1wwj%A%!wX5J0 zhVON&_8~^66GzhL0^x$$&9>iqL@0f)*~U&iF;CCbYPn9AIQ?1(Q30%l@u*sY6t>Z?bzFNX={3OB(2ZBZjdN#&gy^6!9+Y7w2Il{{B*gV z#ET9B`m|fHPS4CIIc(}&`O*?t%17`F-HmPfo9lUf67RNRnO3|ICdztNyl&ul-asO> z+>b(dZxlgF1q5je64vUUo#Zom!U*dN7PmKXdK*66mNt1Yjru#F_xKxYzL_^{ALm`j zsi-`O{(Wol77kdtP4ar_zsX-6>H5=gZLk`drp2A6?V+-th;04jHa=k5Lwq)dbeADF zH}X#*+_|UURCl!7x>G4q((geM{7i1_#%=E!^hUF5{?l6?-ZRv1=H&1s!0C4-Du*gp zHa1~yl+c8YF=pUW2i9=WS^#sEYvx-+k!B){V9&G7nor(ybC@FWP1>GA#H_uV;uNCZ zhG$prVaddel%pbX617El83dW@>84BkvXoffY+3g?ZL<&eoPbQm z`%1T^@}%HZ9L%Fe%UcRiQyk~bz(qY-Q7=f1VbU&lFc+0;e_1fi8l;exk>-9pYczKSHtO9PBLpb15jZ^5SqLPiq4fMcLMlxlfOJ zei+ZHbBjo)1f*%|`7vx?P)olLt?Is{jXqz-u|vC`=nJczD%0xwJ*D3o=3k2l-gK`w z%vEL51*FD(-9!Ldme~c$krCq9A=hld1|$UIo^MG6V_XxyF2>?uE-mN0CaCr9ya)(W zjCP_n%1+O{kdW8*=SPWRD75JloV;DkwLDU8j<@efoNtn*TU3$b2Xf1RKw=UBnh$zm z4H$fK&_(U!xzF659vsZ+hFW4TI?ZGp4DoVXB?&S<|58Jxy;2bI(U_Ozxh(A|JagA$ z=!l<633KBl#TMH|BIn&a}giD9<_c)I|ae3OLUGfnrI%sFx-22 z%yExEN$2||4#4)Bwti1D9e4?%##Hs^0a|$;?sFH4-V_hkr%GSBwr0LxO0^Shfhrx+V%%g=YZE~ayJcY7|T@9~8SrYweR#Al zh4O2UX9^tR^>AO>zmKbK2foiq>WYeBnF~K(iU$1dt_V>74U#cli2qWnIi&SHC5j&n zTkvMVN0B-lvJt>dfGxW-;V`o2DBPxTc>;Y4{RkurTYOpPSw1Dca2E~;aB?w#>L!7W z`nSbs@2>m{@OcJP@A6$j_pE~DR~_5;q}gOC>suY|SQBlCozVses~rZaxOGpZE@7$l z6&zO`UEH+0v8~SW z9#j%LTipI)Spp~I(3IO{bLxL732}(?KAq+&i|K; z`-v(g{Ga?k`TOucHvc!z3yAykczcZ#m;Z16)Bh(X9K`>R0!7V&MVYOLTL=FyV$Mn= z9#zPF&k$YyFKBE9(m9CvAPN724jv^V{C^z!KjEZZLh-Q=-Uy$(O@`eR5|FF#iM#I+ zm--~OSVpU|z-@I8i?Ry};>9;i`P~&lyWvalN9yb>x#nI4Z%!%d$h_@=rkv!L=sb~% z5GCe7R{Hz2vtMe}o+-2pNgZ!MFpdE@D*$|`(Md6i5#-v;27Swxp_TTo5{UG}{OZ4W zA`*a>OoVEAODzmeY1|Gm&nS~DX;7%YCL8p|xub(_MXe|zsZ}c+!$nk5P@@60| zWr3zWL)Aa61{GLnJV-Ldg!VAd%smuB@iLPlO(HZ;Yb5~(Dg#i|Q%p(3-N~@7#FkQ~ zOr#9ZcCZ6?m_u<|M2*Tr*m|J1IDFoDK-WW?JEQ?Xp8sy#sT=N`p18_BfC2A7+8BUns5Wz<;EzPsT-5)a^^2)8QR_cssFh%Vv#?lwX(*ikFPt|)OIViiKBMgN zdlMEG*_id(@MH+Z{~N~`s=|!8VI3JD5M@><&N^3;eX26;PIJPG`9K)!^Z#FdpgHjo zq(Prqa#e5~41B$Ydzi>}9sOX#l72j_opk|WHzcTNW48$v`$v7^!O;6-K?di_1&b6f zxRRx-NAAKDGw=Jd-Uuisa8pfPOq;0lNDY>G2jxX4Zq@ zvJbPuHYVZ`)5<;vg_%Ph&6Yjjv`$^Vi$VnXq8i?XdzO{D$=<8kBA6%3&+6~8)IOn_ zdYG2^Yx1XSZPvcQTP5cIhHcJ^0s~*K0JiG{9xbdo4~IJqjOpf|`;V=-L29W~Wvonh zHnx3fak$b9b7ECFaR-z)G&`)l1XB!6i*kS-+wN`^9J|X-IxRkejyroigqVU2la0O0 zx&s8ruVBXM%)?{}D4)Xt!WovS#=MZJc`45Gxr7NR5VWCT4SbnBn2rjI@!n*wlc3f= zo!0QtG-=F#ne_>XZp~rYLxo06+3tgU3=)V5Qqd-3npCKh03&y4q1OXnDjI@+C zUXjNBXcb7X!93ckG`}DhK|1iC%K60}qx|nW1`<_ZL{Cwtf@X0_Mu5F{E@YKIgPZ%} zU6rA7rM(h${czI6-c}W2)N{*sf912!U8L%PlD8kFoHvR)rL;dfR2%I=!L|!ZV&OVd z_yT~DGGOUo(PZdRchzfDPZ`YE!hV+|;r+r&F-&1;?2cud+6EVypM5@b{d>4_LW@L9NhR+EXZ5xzu^nD0s$*U%G z(q)=pVl{8Tct|2w3Z=m0ayIxIAJCQAIn5MxxI-8r9|=B$CX5sJ-)$+T>stWxZIgaw znX@U|I4ZT^DrRAD2o(!Aw)hM@&!k;9D^!~oq1b?#1ygYL*FY4Rk$1lt`T@92V<3Kq z20Nwfd-VdwT#+;*S93bOnwfqLLI-bc;fkReWJF2~b}*agE*v>F>@o+O3cnw2{CiMS zIzXL%dxs-J7?qr$723}_KV)~sPpYI$k=NSZ`fKlL!Wi7}kXeP6I?C~>B>?uAs2>qr zm4*E;iG|>#2Ez6rZfB2(#pagqJZl*ZP127oKjC4u<0Nk|8{;ty=8+@YXU8{Zpyq!bx*=a7s9#m3ZsBTfVqB)S_|_ry|8tp`gxLi&Sk z_gcTjsuHZwHUaw12W$B|ov#mOO+W7qb?f#O*j(eZHob$Vb1z+0j~+=B*|nrsfA@*A zGK~C2wOpC_m+VT>sZ|6SQcGhYMrS?4sOMPNP(7jM&`2|t)l>*GiZLP`PHz=G(TF5>D<5drs<2mP ztaMnbk^sU@OTY@y-~&6TkSNQuJJpj<21iU#@LjX07{z88(%upk|Iyu$SKcg^TUMus zecz@?a4;Pew-0u7TF0p|xGC+cJvfz z5%`1{6vC3r?YSnu`?Eq8rpNMK5ebv^#`e9 zYw}^@r>{C9Fa-=}=*?`MHmXqMn7r7OtTomk?HikCeEDb6SFFCRS_@kq=~NFAXAbW~ zOq5P1wJtpu>Zi7O{d8;7#t zG%;E0Z!wE4$Ky|OCpIS`Nem&9!~miTz@9-s-U`<%0RA|=S3ZAGcf9TJM#&XI0l$*4 zOCM`zBQh~p6Z~3U-C$c)9~!&B5x2p@z|2k`x9k>}n^l^bwX~pDIuhRKKXDU(+uc(U z2j{KpTJjoTgb(@3U1y3%(4n1gn*edOg_6Z<( z{lZsOR(lR`Aq}QcrtY&{#Rp92C*K5WMhw0rjQ;*H_V0lUW~wcxQ&H`_Ou~Xe$6idf z{${?M5#{G7@iEb!8ol^Qt7?Zy^DACO*uxI#1ApJ)osS>a^%J!{MpvAe6{zlCnhv8o z^W)`P+fAUNoV*uo4e#I;d<6U3FUN*~kev7*sc3OdkdHBxTBMFh}i?JM=p$;kw~CtzcYQeiE3;U z>D}@~;Thc*QJ&F(aOFcqn*YVOZ9*isby67NXor$O9k&?5FtPH+1fk7<8J{!0y)n$} z$HzBh{RyLoj!cM#L$(VXC&jB(0q~G)RHAUXOVIp!=_qTIKSC_o) z2_=sX+u{vzy4xCCVM|G3wrA$0;dtr>4d43f4LPMWM6&+-e^IIo{Nu_=j7`~#RXVzs>5&xPCtYFr38 z9AbQ*cnlh{M7$4pInu3mM6$Fq1&LV84T@2Vydq}-&JN-9nkmg9xO{gU`|UfOi-~URcW3pdAXn zEZnMzyaUW;uEHw$G$s-r4UtZCn_7RGoAu*@l`*;kjx1Q)x2`GphF2al zSR0GrQ-!Ky@g2Ek+j4}v{4C3mFzUAU?}CJ=>sq5#W$;?{q`H3U<}x~jEa`BPh!MHPBtaUL$ zL2oeoKS6D}OW&Z;^MNZdFhj3qQ$z7v^{3?KKj<~FT>ye&ZoliF@gps>xURllLcL|| zuXi1rrq-E``_}fYXu-eO4{aQNPklo@4s6#2V|!P1P*@55Le-st4o>ehyOFAWNJV}7 z{#^e41%st4e9MzNwPCG)_DlY=B^>{C1t9!k=`TgSJi^>zkM^rdbNE2c{v(KIsq5CX za8K#AwLM$u;F!$aXWP@a;p^dHe_dW@Grut}wa8+1`x9yL-o(XK8QHiLnsrvuL;51J z&Rs|CxYO!o62VJue@JD$<-M?K)wUwHc5}Y=EJ=q}XYiwiM{4XGZaRQIf&Hc1N=$M% zT#9F=>jM#+E}>U*|4i9%=vcch9J|Bhw#Iortro{_UVPi3onsF3S6Z8@#WW*NpkGhL z3R%1?vYS@R=0D&53A$S2IOK{2sZ+zT+9#bXk;(rf^BfH0v&)*2ET*mFe9Mw7!N3o` zBPCg-vWRjFMbySnwkVjYtLFL?VU;aIAD?FQ-BoNUW=-fK_|2Q=^I97U&tVzRcsLr- z)6pswE2Z5E4Z~O2WyjAA%eyjPm9wbudPCJf<8vBS*U&dpARMkEuVp3_E()SE&&pS}AlEZ#oD-tl zkPtyPOP<&5-iQz#GY2|6&|yJ`YC}5Ik&yO2?MH763`@g8&9%1nUTi9%^%!S`)atXq z9>>j4QgQZKtX72FNNd#HIn$RWt(1|<@IP{w6TKI=IozQmNMOz~jX#(e2U->MS0k5i`#8J<%giO&yTDm38SaeEg#2K(t_Kl5AKvw73%Jy?(pnTTu@ zsryh8o395%p}6N2M7U5DNh=il3>M0mw(7G`Qwo=`!q#G8+!-#1iZ)x5A zPWFfSRdPI*Qr`~;nARh)r}rH)tzgpHO2MSLPy8dmUgWo|vU?|sQZBkxmf)`TdF0D; zk+}sne)w^&QF!9QtvFEklNQv|N?impntc`vb$F`do!FKG6V{+^ux<05bPQqk+~KA* zbC;B*4$iCAx+s*Kb0arGfMVOi-|VOGeO>tT75KC3;J+@2 z@qDJcx9PY}2&#zooJGZUiF;;;((Nel}r?Wm)@WJ}YE2g${73^Hdj?9UMG_{zbsJsIG3wP}7g zef@4e)XFqOWE{CVv%w|SOZp9W9yYUa+wVi=HjMC|Zrtn7#-&bQ#as{!IFng_g^s~` zNE#tjfUmS&bar`(8vy4QgVA`H&XOz*j&)Gr;7O2Xqv^$0U6@Ajt``U6SwEYc1poCv z|5uPdHi`T1;GJpE9S<*qYywnC%t813QfxpBer0chA84at0Aw3XdcOwqipTlu9tXog z7F@>v9F7B^<#DSUC%{U<1aQ=Xf#SimmrUTzfHkTX4Q7Kh8wWJKWR?OpyL`TOr%bbo+&ANOGJM*;97O!j|#{loY3Nj(av zG!-88{D1Ce>^Chw3Sbez0AvAnRXWJjte~G3K*!Q@s6-u{M?& zx5w-h1l8oUIFx%fXz_^CVV^C=BgIrJijqVor!Ri~vHjxN^PgV43NO<-IGIRstLz7o ze(~fj6)FSi3=^XFHVF>A(2`6hsfipZhJJk284ms3gZPs4)?8cUdtKv8U-n#ZE%F@R zHyq)}aX;vTqMQB>#5qXgar!&b`|t4#@QzG2>J27@cG66}cUi%jG!Bv1^r8oBGaCfQ zz)Jbo7~;!4#gn5Cq{D!y2l4#C1`zxpvfCZP%*712kFYhb{e33_WQ%^*np_O0!E6NF zlmeNLfFlt#0pLD`8z#Xq@eE=<_%bMp&V&ANh@ZWF1MQf$JV|FKPlBiYc$Q{B@$gSJ zIPU?gyoJ@!xGky#|Bx3~`{HQ&KnJc*<~;yI^4zLg7W6IvsGvi-7m!Gp>t=ZwdqJV2 zaY)y^6ea%fcj9W6ZlvQYK&mHi%QvpEJHLE8EM~eiE?z2a@_&Dk4d!=NUNmQ>3VSWF zAJJ}MXpnO`L;;$D6ibNGP*N9IdlKNHP}<^1S^6Zvw*r0<{P0hJ0MGo4kssP996a<# z<6_Ql{2p+W)w#Ya1jKLNDq58-i-%YD5G;6w2o|AGa@5D@908l>ltDCFc;ZDZ{&-q! zB%sGWd>~sUCW;19xc+?$US6>9f9RvtsjU0@!f^0*1)&Y}p1&E8252aoS9HT{PyzS~ z#HPIr;c)c5kALCPa?#W*{FcO>a15+&=R_XTkEhcyVGMrBE=SWmlvd9t`4{anQz7tB zwOo4lRoD=5l+o%Sv!p?ZCIIqGNC`x!WPWN>#$v*qx-Zif&)?05Ju;dK?@ zp&U<<)K_6L9im}Ubc-NADd#AEZyU5zdqNV(MWL!-h8bp~P_Cdxf;r@jCC|P<4KQ{E zGe?$1GVOFs+R2$ZUsrs7#Bu$II>c3a-X)rkrHV)>U4#3sjI`0U|jY#6ZZ|Mqbi8MUthn zw>H5M$g?VrfY@0LAAu3g1OqPH5=W6qlWsRvrYpO3IFdGG+!?nYK4J=H-h?B!TSnsC z@lC~@6d|iBVi}3htYS%qN=jzP$KEs^B7%3`d#PjxMl3|J}V6O_>Q8lCe;) zph>`u&X}#8T3{u`(JyLnYEh(G?4rFNJ1^qAQ8x+0cdVNP*bti{N`zu5_@Iqm(M{kj znNUOk^Dld>BJ@_1;gSORM# zS1Q8Y6J}eXCoGE%&=HGe9g?ICjf#Mi_k|=U8r>#}UhXg??#;P-?juA7}(CU?t9!fqNPQfi5nxB|zvhLvpSgb{jM zpGjaqz2mw{#ikZXKmx)_{@g$dmuHe0{TH^qeP&!~Ww{`J@%eYPBEbG>! zY(2l4i0>(7%ar1}iJ#=Im6KtcvJ>c`sDdU;%&yuHh$Ur(+}g8Eq%d7jdtJt&IxI06 z!mmc}$%bG`@FsRLJGRE{rkY8txJ*p3I!@xD_RYz+!jS4lEY$|PQ`cb#DF6%?VfMsF zMl{ww*d1di5}Rk$t5I<-WGxrTklQHQ8b>LBm31?o@VcxzjL%Rf`Hgye!I5tCYW?wE zJ4BMNXWz&Uw@b2)Tkh(x1lVMnZEHAWFiM>2F4wh&gC_zl)t(fRX1Y-!@tsG+x4_X- z36XIuWGgrl5tarUxiyl~D7B!Q!FH_GM@R!)YbW-lk4&SqGj_L}grS$sBdt}9 zLv1lYw5{a;mc6j4%viTpD*+X7W|iAhYKaTQ8-sXuw{Zt0Hx_t&$xjkesnvKm6E*~^ zPl#>94zinM5l>~uX0_%W5&<4kx+l|tOV*fBH@huP;nHqyfLC#Zg%hha4@^d(FhEk< z23k|jGs)A8i$vDq2xvfS9)hZ>JhvneV z?#a3ns=088>wO1?K>sYIXoa>l!cKv(1K_x=lNC1WN^QC8>e`qC5`p8sT4OIr(zelZ z*m1QaS8iiRNv>5m#Iy_zNN%&OyyHg409+@#b(<|oP#Gumws8j=XDL{_3D^&Dt;|EkhlS7kL6p&~SuyF|#vR1Q z!6B?vUGiwTUM(F0KPj{kms=-jEj+3Oi_=b48+I@jE4(7c_k|?Y7{U`fuJD5CZ=Cqk zOBLeE)oLaefzJure_LnCbp&Hr-m79(hb1dR=A7CSCxJ%1uIzZXF0IvWVEQQ`wk^Z!#)--@qdQS;+{4B_&jd#V!<*bD<%B+e! z&KNLLvRn_ zaS=vFDrvS>O)TXd(FFxn8zDlWYK&F;3O-oXTp{juTs(vwh&HwJmJ3_4MtpUby5s`v zJjb@R@?_kRE={_ziRx;Pkz1WL`$_>?qor*w<5#;y(#B1q%(DFe6INoi4v%BmxZ#oA z+JK0iV@X#SE9>HT$g76gJ?#Tz2KczqXX19xfxMN?7F4h=R&fN(6>-D2H6OrN(uSu_ z>aay+t%DD|)0MGtahBF9CZGYP!5ZFEAkT?w zMBa`&#TW_e1V%~{`%(}YP#q{0w#rNbx4d$vwibCwVq);&T_n1-)&nuC)^0!60~m6mxzU5xV#jz=PH)~e zWvTdqN3JF=j+rOLINv%GC|QSfVw-g0THPc7QDEe$eFa>FY~HNN#g3ajm`$CG$%1lo{<5qbvf(s@tw9!NL^G`Yy|Rz?2ZpCH;KY{nzZq-mPBl7A0&= z?Ih?W@S3FCR_JAz@pVG!*w*E2i3T+0Zuf;F+gPV!+-|Fhl-RtR_)-5k&6k}fQRmt!a6Kg$X<}Ib}SnlXgS|n(IC71>{1pMmrBor(I)dizrbwUn`AZHGS8>S`97Fi=wv5F=aMN zQt7&BH>pJvSdkIDVPAE~vN3z1qiDy~;?gNu)8Xba{=Ai&h$SCxtqjY(T>)Q;;POeERn4WpKF7{Um*iY5RIcI{+p+hCDRQs%&O z1_f1TL>Lo=5sUPmit~hunp^Ddx<|j<_zWh@^DVV*)ir@4%Nh7Ik=VMyjfpvnQy$AS zsyTxRo5JFO>-&l4!K#{B$(IiAviG~>TNHdPNj#%#u>@*^VTa_cyCAD1X5BcAV_UbG ziKxIxx38@2A8Y9u$0EJ438C#$jGSZJ>zZm;t2LhSBDc=nx-88imL{F7BkF7jwE6{@ zReS0UDtUA4)wgKg;VulpwcEVyZ3C|qU}`ms>y!#3#nsl9x$MN<3{1wv)SapTgy2G{wAoud zB^IYOb{N<#p0rb&hK)2^?l54Zn%>n;N1-)>yRAD6=vXM#Wjd|(afu2{R8 zAe8UcygM{GBR2J)3|lh#j0Wkm65Uf^l!rz{%5I&*)vC)ou}Z-TuEP`3fn|DMu?S~a zBR3hjY)ITx>_vR@LVvkbxe^JxBJ>n?5HetuMcaprx_F+C>BLzLnmidOWwhQ?O$4|d zO@8jmb_{mYTD{GUh~elUk z0vZ4X%E1}9eK^uf!C7%r967bQ1-KON(|UE&G~o#C^sR#vptaJd>mnA#-3AK+#%}dIcvl<$Y;;CUH zu>T^hwm$zy#oDMYSo2n>20qe}W_nNOG@0g3ZOb9p8{$ZBe6Y;QrZg__(khB<$hhHV zTWu6GrbVV>sa0J#f=B?XefHxpx#o(0wR~uazUfvcYs}`du4>k4izHo#B&Ucs{+QoAc(ts#goWANG7{T3MvfVTrN)#})woBNg-ZD&l(w?GRcdkj zZ8sIX7DYBxY+nb`HSfugYPZUWiD=`qQuT5cL%XPsi-6lA0L%8BL#0fVbx}8RNrR}v z$eOmdO&h7@zM(fo5*NZey5SL=)&fbddPlhX(^;;z<)bbzQq<)glhhr-NoFJG5%BPS zkkowUuwFh?M%}d7Uv6yIX}RPcfljCkGXRnp$hxf$EQzJ+;$gLs%5*J~L};-a(w??R z3+|fxe(lz3;9z;#&3Cf9;;5sH(orodfjk9!MYBC^orz>4 zmeg`NE=F;D|374ysNkIY1n9Y-=`B%yn6WF}qq6fpxFJzTcApE*r~Jm|8BLys;mvHchLq zgzRviRZW2k;z%y>9g#oyz z(M?LK5S(4JwYLN+DK-()iKuI=LPj14R^e`&0U=iMNjEG(Q9z7tQ)Y6@g?V`HY>mjn zv+D3fvTd0uvaZW{B9hqF>^t+C&+V+!A^eg7Z1Qr1$ut{L^y3`n8IH4ez3hEE?d5ZX zFN@*no1?0x*4#$i)X*(LLUo6(1`~;ho6W#rG#aFm%dWF^^nlQXZ{^ltcmw}^ O`2PVtU_U?w%>n?L4gZV) literal 0 HcmV?d00001 diff --git a/eval/dataset/arms/copilot_v11_deep_r1/pr4926.cost.json b/eval/dataset/arms/copilot_v11_deep_r1/pr4926.cost.json new file mode 100644 index 000000000..b5aae9da3 --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_r1/pr4926.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 1148.0, + "split": "train", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v11_deep_r1/runs/pr4926/run-20260813-122340-f64b1f", + "recorded_at": "2026-08-13T12:42:45", + "attempts": [ + { + "invocation_id": "inv-e24e4a33", + "outcome": "run:run-20260813-122340-f64b1f", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-122340-f64b1f", + "llm_calls": 79, + "input_tokens": 447093, + "output_tokens": 139238, + "usd": 0.380591, + "source": "spans" + } + ], + "attempt_usd_total": 0.380591, + "llm_calls": 79, + "input_tokens": 447093, + "output_tokens": 139238, + "usd": 0.380591, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-122340-f64b1f", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.3806, + "minutes": 19.09, + "ci_minutes": 0, + "input_tokens": 447093, + "output_tokens": 139238, + "tool_calls": 136, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.6687, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 79, + "by_role": { + "agent": { + "usd": 0.355174, + "input_tokens": 428029, + "output_tokens": 120842, + "calls": 77 + }, + "reducer": { + "usd": 0.025417, + "input_tokens": 19064, + "output_tokens": 18396, + "calls": 2 + } + }, + "cache_read_tokens": 3952384, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 3.66, + "gate": 1.61, + "review": 1139.89, + "report": 0.06 + }, + "time_to_first_result_s": 1145.22 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4926.trace.json.gz", + "trace_bytes": 29557 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v11_deep_r1/pr4926.md b/eval/dataset/arms/copilot_v11_deep_r1/pr4926.md new file mode 100644 index 000000000..3ab1ae12e --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_r1/pr4926.md @@ -0,0 +1,44 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4926, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head 35aa741e1962) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 4 finding(s) below | +| Security | no finding reported | +| Docs / comments | 1 finding(s) below | +| Behavior / compatibility | 1 finding(s) below | +| Correctness | 2 finding(s) below | + +**Validated:** +- [sweep] Indexed accesses flash_attn_hub.py:47/114/293: all are tuple-first-element unwraps guarded by `isinstance(out, tuple)`; consistent with flash_attn.py:76-77 FA2/FA3 (out, lse) convention. +- [sweep] Branch flash_attn_hub.py:46 (`isinstance(out, tuple)`): safe unwrap, mirrors flash_attn.py. +- [sweep] Branch flash_attn_hub.py:88/267 (`if backend_kwargs`): only logs ignored kwargs, same as flash_attn.py:69-71; no behavior risk. +- [sweep] Branch flash_attn_hub.py:108/287 (`if ... is None and ... is None`): raises only when both funcs absent, so a module with only flash_attn_func leaves varlen=None for the masked path. +- [sweep] Branches flash_attn_hub.py:117/296 (`if attn_func is not None`): piecewise wrapper falls back to varlen; logically safe when init invariant holds. +- [sweep] Branches flash_attn_hub.py:189/205/213 and 368/384/392 (piecewise/mask/dense routing): dense func path is exercised by the added execution test; piecewise and varlen/masked paths are not exercised by any added test. +- [sweep] cuda/platform.py:124-144: fallback order read — kernels missing -> FLASH_ATTN, pre-Hopper FLASH_ATTN_3_HUB -> FLASH_ATTN_HUB; matches docs; only ImportError is caught around the kernels import. +- [sweep] musa/npu/rocm/xpu platform hunks read: each maps HUB -> FLASH_ATTN before enum lookup, so no enum KeyError risk on those platforms. + +`tests/diffusion/attention/test_kernels_hub.py:58` [major] — tests/diffusion/attention/test_kernels_hub.py::test_kernels_hub_execution (lines 56-58, @pytest.mark.core_model + @hardware_test(res={'cuda': 'L4'}, num_cards=1)) is the only behavioral test for the new FLASH_ATTN_HUB/FLASH_ATTN_3_HUB kernels, but no Buildkite lane selects tests/diffusion/attention. The CUDA diffusion lanes in .buildkite/test-ready.yml:64-145 target tests/diffusion/{offloader,cache,batching,distributed,models,*.py} with '-m core_model and cuda', and the recursive tests/diffusion run uses '-m core_model and cpu', so the L4/cuda-marked test is deselected. Add a CUDA/L4 lane for this file and extend it with masked/varlen/piecewise forward coverage (those call self.flash_attn_varlen_func and are untested). (evidence: grep '.buildkite' for 'tests/diffusion/attention' returns no matches; .buildkite/test-ready.yml:64-145 lists the cuda diffusion lanes; tests/diffusion/attention/test_kernels_hub.py:56-58 applies @hardware_test(res={'cuda': 'L4'}) + @pytest.mark.core_model.) + +`vllm_omni/platforms/rocm/platform.py:91` [major] — Buildkite gate reports buildkite/vllm-omni-amd-ci and buildkite/vllm-omni-npu-ci as FAILURE (verified via gh_ci_read). This diff does touch the ROCm and NPU selector files (rocm/platform.py:91, npu/platform.py:110). Logs are not accessible through current tooling, so whether these failures are pre-existing or caused by the selector changes remains unverified. Please confirm attribution before merge. (evidence: [gate] buildkite/vllm-omni-amd-ci and buildkite/vllm-omni-npu-ci failing; gh_ci_read exposes status only (no logs), so attribution to this diff cannot be confirmed.) + +`pyproject.toml:?` [minor] — The diff adds runtime `from kernels import get_kernel` and `get_kernel(..., version=1/2/default)` calls in FlashAttentionHubImpl.__init__ (flash_attn_hub.py:92-103) and FlashAttention3HubImpl.__init__ (flash_attn_hub.py:271-282), but pyproject.toml:70 declares kernels only as `kernels<0.15.0` in the dev extra with no lower bound. The docs (attention_backends.md:228) pin kernels==0.14.1, and no runtime requirement exists. An older kernels release may lack get_kernel or the version= kwarg. Add a lower bound (e.g. kernels>=0.14.1,<0.15.0, matching the docs) and consider exposing a runtime extra now that this is a user-facing backend rather than only an accuracy-test dependency. (evidence: pyproject.toml:70 shows only `kernels<0.15.0`; docs/user_guide/diffusion/attention_backends.md pins `kernels==0.14.1`; vllm_omni/diffusion/attention/backends/flash_attn_hub.py:92-103 and 271-282 import `kernels.get_kernel` and call it with `version=`.) + +`tests/diffusion/attention/test_kernels_hub.py:92` [minor] — tests/diffusion/attention/test_kernels_hub.py:92 skips FA3 execution when compute capability major < 9, but test_kernels_hub_execution is only marked for cuda L4 (sm_89, major=8), so FlashAttention3HubImpl.forward can never execute in the PR's declared CI lane. Add an sm_90+ (e.g. H100) lane, or adjust the skip/config, so the FLASH_ATTN_3_HUB path is actually exercised. (evidence: [sweep] FA3 coverage: test_kernels_hub.py:92 skips when major < 9, but the test is gated on L4 (sm_89), so FlashAttention3HubImpl.forward never executes in CI.) + +`vllm_omni/diffusion/attention/backends/flash_attn_hub.py:?` [minor] — Confirmed. FlashAttentionHubBackend and FlashAttention3HubBackend return True from supports_attention_mask (lines 55/234), and forward_cuda routes any attn_mask containing a False entry to _forward_varlen_masked, which calls self.flash_attn_varlen_func (line 147; FA3 path mirrored) unconditionally. __init__ only raises when both flash_attn_func and flash_attn_varlen_func are None, so a Hub module that exposes flash_attn_func but not flash_attn_varlen_func passes init and then crashes masked requests with TypeError. sequence_parallel.py:416 gates auto-pad on supports_attention_mask, so SP auto-pad can reach this path. Fix by raising at init when varlen is missing while mask support is advertised (or returning False from supports_attention_mask), and add a masked forward_cuda test since test_kernels_hub.py only uses AttentionMetadata(attn_mask=None). (evidence: read flash_attn_hub.py:105-109 and 147; supports_attention_mask returns True at lines 55/234; vllm_omni/diffusion/hooks/sequence_parallel.py:416 gates auto-pad on supports_attention_mask; tests/diffusion/attention/test_kernels_hub.py never sets attn_mask or full_attn_spans.) + +`vllm_omni/diffusion/attention/backends/flash_attn_hub.py:96` [minor] — Both Hub impl constructors (flash_attn_hub.py:92-103 and 271-282) import kernels.get_kernel and execute the version=1/2/default chain plus a 'Loading ...' log on every construction. Attention.__init__ (layer.py:94-95) constructs one impl per attention layer, so multi-layer DiTs repeat this per layer unless kernels.get_kernel caches; the log line repeats regardless. Memoize the resolved module (module-level or functools.cache around the version chain) so resolution/logging happens once per process. (evidence: layer.py:94 `self.attention = self.attn_impl_cls(...)` inside Attention.__init__; flash_attn_hub.py:92-103 runs get_kernel unconditionally in __init__.) + +**Additional observations (beyond the comment budget):** +- `vllm_omni/platforms/cuda/platform.py:126` [minor] The CUDA selector (cuda/platform.py:124-138) catches only ImportError while probing `kernels`; once the package imports, FlashAttentionHubImpl.__init__ (flash_attn_hub.py:92-105) calls get_kernel with version-1/version-2 +- `vllm_omni/platforms/cuda/platform.py:124` [minor] The CUDA HUB branch at cuda/platform.py:124 routes FLASH_ATTN_HUB whenever `kernels` imports successfully, without the compute-capability gate that line 149 applies to local FLASH_ATTN (sm_80+ else TORCH_SDPA). A pre-Amp + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v11_deep_r1/pr4926.trace.json.gz b/eval/dataset/arms/copilot_v11_deep_r1/pr4926.trace.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..51bbc3bf88e698a29566717daf66403116258c77 GIT binary patch literal 29557 zcmaIdV{;`?yQu4!Gq!EpX2d;ePZm4Jbk9!?+@o}{mz5XmK^K^`|2H5KJ0Deb#Opa@d! z`P(P#cUs@g8~#G`QJ>0$;r;e>Pu#mRZt?$}XoLMSxWwDrr{j z;|fChy0npUD-oq}m{qYP^{kP_aLcbeN=WJWmQx=vs`h3yfap0L=In?7rIR|C-i%wI#OT_565fr!WLVyMQ zqeW*5fj3@0|`hl3~;+uTt&E7l4&X&&l8--{bar6GJVzLhWc1X_yv@yHA=HOa`OvlX{H20gAd)IXgdSK zW64x-nXDUav6ns?Yqb4~;y*odBUj2rQmQjlXxx^^2L9)*W1FXHCB(yM@S9LP#w9(v zeQ0FhD1{x<{uE2tlG-7NV(f7vz(e?27?0M5apmB3aW3ef zf7QSJw%SP3VRuT+BNKn96W=fN=R%u;0oPFmg~Deq)}AU=C?|9@EAdR&8c(> z0YAoQgU~k0XbTa4NTFVdZCVRo15x&Q4_|tei+~d&1Fiu0lo8=u579Ci`-NgE{0(2^ zD4)N2sHLDI#9+fOd(3noqv>B88UIsW?*W? z%8`iNa7EFNBkoAY69#S*CXrYX;Io(JZw)RF+*c2%1XgH&?iaH zu0M!mV~Pp=FDk3vZa|nH?63$t#^Utq_YEz#AYc}#_$Po5xzL^6%71~Y-NJbgtc|+l zJVrdvjshPxe_y5syiZc|K%iz+e~`oW55zvWAKV~m7o5BI`7lPL$O@|GY;4q>mi zBf_d%@zK*eT~$r%@y`+hEOw81~^|tQ;P=qU*jWxAiJBv%nae4*VwNba3Tm< zTzJ+=D_N#{qqlLB@KI{q=4z3ZA&u3 zdL`?@)@j?hP7v<&Bl*ThSabXS0{6PR zIc+SR^)rk0<@bGP)v7XeGmiS4XU~x)*@Gs*B(Eo#Dfr^ux~^Yt9rq5m-){8#T=8q9 zHUH6Jgm)dfwwBNsd%PPU8yO9L5zi5- z=owM?5*rLzVJk^x{>Yf04DvHm&Jpr{1l{bOugr#ubiW#_yjtx-K7muca*lpss5G&s z+|!Y;7?~8xgPkXQ=3(b54hA9au#2Yiiq#3`|v9&D_y`tK|S_Ce5PqW<&)B zcSUkW>$*6M!Wj)`920MCar-5)*+l0Vbd^A{tnI2DLEuYoU=b{Pcr44Ld(J@EovsVE zzOP9$@0?=zGI6xb`zjG6mi$N#wI0V;ZxC8;IAgSx^_`rJ}-WWx|l`) z%(Rg;;V3eOw!~^pZVCmE5h`xLy#wk8h(cX-hq&A0L&Xlpo3A4))P(#c#xKJEW=BSt z;Oz}k`}vkHpF^>_=g?#nwf=}T7D9Dx6cwA^-(x^`I5F+!VII^21!V~pbZZ$-3&=PLt}) z!Cm^^6L}|be9{;S(0~tYh{t>adLd<*WaK#`tbg<5Me~(9emxBR-2_R9Giw7ru1o=8 zYRHiio_5>!z~i()8WbjKVQA>K$lQDl@J3MFV42QGTtDiuZCz(oBQo5Mkq7TCx}7B7 zzl$#9iQj`SUwtk@gw_RPrZ2k`tj&ZnWL*%};MUN~nf(i4e}!;K=6G!Y9y>5Tv%FBt z+756j!+AnBP!ai%j6Fe8AWq7vORM$=TmuCJZp*d{0d+n$T| zKH5w-M9fvCXwfn?2wsL0$VfB`lY1mzax^lNIv$!;;@Ia7XT~8R+zf#oN5Bcooy}+g zSM9cW){`FQVZk02}P!ltOeE zCy%&CeKl;M16M(`^ySPw0|S0%Z^tC*ou|w+jWp6;I5B}$%r66S{t#$LfvK6$VJvV| zM}foV_4slqk+u_LQ6>rh#D^$CW^WD$_RhiKzAXraz7x=T=uL)8d@5ac_@DO?{ZTA3 zYsPe^Q1D$IyI0-(Nzg}9;Sqtf_*CJZkmPWu>wwdb`+fxUodVm>OSkQQ@;nC9 z`b{mYg!Mg!)ml1H{H=q^fLlA^QG{Uq?X*P@gKiEa#RRMz21!Aq6aOKIp=W1Ys^_j_ z=Vu`Gw&6+)?itKw+_BwU)ao6~rEs-dE7o`lN!I--;b)9doKGCfQR+S8C#+3Oi>!s3 zcff)LSc)>Ypxy|YJ%Lnx4k6&HMfs)F0S4b(Say7$KZplMG^v3BXv7NT3VM=L9>b{Offu-@wtTtF)Y3 z|H&36MF<;3W7fU@>h!%y>{O8X;7+A%OMrBf{N?`JZ@5hB=_4`!{5@(@f(^iE^<%vN zETWNSY=^j_hpFyUR59Hen432&(~Bm_sv&ve+n`1YHAi67K04RCh}6^?aJ1X}r<82m^5hy}eliTLZfkLDK-7*x z8wv5(hx^UPAwiReDrJmZA$GI&VxlOQx5v*i;S*HXdb}kRJY2Mk)h!~IY}pu8 z_(`~5Mj-^OC;BSOYrrqQ(;};Xfb~y2ATP2@?8SV;O zR6{I9GrSB{)FG9CAcc~MxcdPM4&`TOYDX{~Jk})f6_nJ-CX|s`PVBQDhs~ljD!Q6F zd~r-NbH#;I&QIE8nKPKhf~AOA=_`I+A6bFpx5kDOsYU z^#bwwDh5sVk8iX+LJk%Xt~F&`BM@C+&cnTa;Xp|eFU7)$LmJ3CO<)rZ7A*g>JNWW< zl_}Mm9NXoUzwc@o(XR*ji5wr8yq$&@^n6x~T8~)Jg*>Ift)RU%)etDy1_T_b*l_q@ z)0-!EaRQ_H){g34LGFMuL_~jq)!aKoTx-Gvdw?y@j-48yplc&hQZS>$RI-=e7;;b9 z#HGj4+mkpqTYim_@F(W?Fpp((BoDmJx{ABb?eMng1I>?(QoBclQkzw6b)Zj5z&K{% zB>W|aRh4-Ei+svxjj|nl2E2dw&%w3N0UMe`XoA@$r?m1*CCosU_Hn0;QYSpw(nxeQKeHN7eZ2%F}*(i0s`DzkY#l z$h^m3s<|iM07%SfD-KjVGIk2-3b^uAhsN!OUb-oK$?K18`UGziIR`#jnpGLtdADs@K8+8wk0TmQ68w*1A6#^*1Uad_PQI4=Cq@eeI9Rm76{Hp)<8elQ zL((YjTR9IUY6@+{a+`x^nlQM_q)2q>WZ3ZlnKnTnF z`;L9SLUjwW1W^6VoD73A5QA{!`WvSqJfNj$J%9||-0cVQ=0jtDkDzA-yd;FT;+;A3 zeWnpwgA0^9xCB5ZH3>{%oSZLCr1%>`JP|3PO(w)`xV1ysy=g0DJG5Jyu@MToVs9|? zT=k$&s(*)d@fG@N!HK)>!a+W_Z2yTU7hYLu=F1cL9O}<-=xPh3+ur=K4AdK9Ddc^1;nhP_Be8+d2MSW=s0aLZTusfBJb7`(wAgK)Fw^ zuWA93Mtj^L>}}S8cKoNM|B#-a`|VpP)**7agW~hp`j3rsxJm`5!Y6%vy)WMdmGQ97@}BkmBNlPsltQG!OO|;{17~dQW5<%u`_-vdC{;Lj*?7e?qnr zZn5yODx|fbdn$ijmZ;he??g@G5>`1R4i1km5ngTPWNqcTPCIZC37tI4eg&DvJ@3B! z+>00^3Uk($EjS_9EL(u(FM#YbKm1hTwpdeNVKfAFYMmQQ(D=$xsV8SAg!s zU@!KGJA55HVxz9g&FZo>ngCMw17j z^zNal$G5bMB;??XrBQI09avYxt83mt>6v;Y`c;uhahSw}!Ku8Iq!fG%jyqkJ+R-uBIGzKb<@(7z-_6{d5D(V?!?a+O+&jT!J zz4d`~d}CBeb?=T$NE(<5cX>y>UF$darl>Iy%Af#BZRlXGZ$Neg4P~vu7zhhACN^AA zZPhEgXw$`1v0HFw4&j~CjXHns&C5;ViRiyKZ<|{j??k=`+z$=_CTG&>O86I-?nf70 zj$FA@&68#E=F|>%JXEG?+J?9>%(#3;W5Lfbe$Iw!Q1ibJb?8PVd5zbm0y&l-|Dv(0 z(Nz1Q&)}zM(ca(Nx$^SzxVpjF8eEM*$x9MmXOrds9#dG~%2`|F1kU4wfUZg_49_9Z zReMcBdrHKET+(Y+IJoLozSt3qR!9U&hcP+RsilD+3+|6dQL0EJ-f3rK8%uV7C1h4v zC`+&r0Ii(3z|VKuz<>iN)IGHfl?1JLA=waJG$BZix~QCB%3oS3-$_q!Zl&^#d%4&r z9x6dta=|X7-L}Pu$jyv*VLwn%b^b~wm1@!M2e_%4hkTBM6|4M}!%nJcVB6NqU&Ru* zq5r7ngXkkH&+yyZ=WeRN^@YeCM8h4GyRG7BV0;6$z+NvqFjj#8J_=$K|8^c!vC_7h;LjKKLOIm8 z{UVq&f$-{n;|(xk-VskoZA|un!1t2!$3LwA!**WJA#@r_=dB!!jI*= zO)dJDpuK%rT|E3rIJmr<7U)g8NBl0XJK$U?g#^4fD0uAn^E>lW( za}8K|;a|7o;`~LHxd$B*G4hKtiR|}`gdEwRcbyCQX+5d8B;YCT1QkEXqD#;)WZvqI z88yFeCg8Dr=4ZS!6v#=EQnGF9UMh_##d)X;8vitn1%@Hmr(B!h23b4In)QSN1)Oh@ zjWr5Bl=KXWt5;BL+Bj?2c^4)0H2zg3D+ILas4yCO#OtS+iDF3bc$$b2K$Lzgg*?tD!n_$7Xcm&GPDFsLXq=%jbzEzzM0 zH3Jk78mS@Hmi=KUGdqAzhf+Id7s@*+tF(7 zWMWChnbF}Aut<35dQuMmSK~Jxg6QIDa;6xgYJx8&h}@LaptW%+vASs;cQL+fg(kT< z=yt>Muo4+wyT0{IhabnJ%@Bb2es2)itwrwSmXkph8w3TM9YNUpBZn3dI{>q4ryVqjv#bOI6_UQ6ef z<3g1dr4GiK*0Q>%p9hpnG$AYn+`a?x1YvbAA)6T&d9B_a84jLsQDy=mJ(nz*4}Dxt ziqfCKr}vQkLa@=Sh)@n~3Ssx|A!lU-Z~R>fs-cpwIkVJdvw!C}G_k(;3;vAg=dvUe z@lgd06T#pC$8AIGePLj`=J4%;%}hs0HO!1v3^zckL%(?$>*ygd`|&Z~xT0rqdKN@E zzmu>uJ1ay)1`O7Wo@yihOT(^{tzYb>N`X_Up;*wqL#m~J3U=pJk@mMP(FVo1TVRl6 z*FrbX?5?G43==u{N!#q;@kbBs>DRiWDLX#}RG&MG%C`d*-CzEg51S*k&3~oVp z7l17haC3vxQy&OQ>nsbgwHrJ8;7Cjvc3Dtr&+Fr%u-Su=&oIiJdV!LTBLY23sbC*v z78nf+@pO>>$K7;=L1v@U>E?jupV)lY<}=kG-98Os`4c0>L>`(^{diJ;zVHJ`_lke_ zh{#cUg;P_ZerU?=)V2iaqFIdWtQ{(I6naam@)4AnRy#`|lyV~kH5ZkiyjYSrI|^%3 zwGQuC;g%H1F)|7QQ|iB^rLPPd*ne_EVDlRMnxvfIIy16OuK}Qx1>!q}(<94fm`w^( z5O;gtpUmG@YK2yLEMJqdzLENd`!BYh_OU+n81rK0K^I+r7iR=e(|9sD@{FntUD}&C z5nMKX8Z5dMR@iyv#?NeBiWSllvp78*^kNk~>!|K@cE1o&DJP*wdS*Jj`|WrmyBPE+ z&|IshciiO%NO(&ScdW}KNDu#sa9?t&>Y2vtikTwm;6^Ma9!5>~`qS`}Ud(dJ<VepyNfBlQSd5Pb%-mbr zps|#5m+DEebScR!6}$-cYc{4i{oPNomKm&LKaYmo)}P>qU6m5bOFREMh1h0N4Fq7> zd#GT=4cvub1L&GO?=8DGx(AE>Dh*&O8mViGK;7)+Qbu#p9D`=|X?Nfiye1@9NYu;= znw3!@b|9T9{di&O85D1yH(XT6{ni)#nT+t?tfmIEvp`OO1dbh7TTE!7o`JM#a znhTw9mXq0u6=Q=*-H3x}48Qp#9*QL@7mr0l9Tq=hxp`2v60bdQP25j{53Vl~fP8HV! zCey~)4?_ugB=*WU)qO=aErjcUJu!ZH9T6}K+@oZFU-sY=&BWm3GPRCD=Po$l-`i7eBHbnXLHYA+2xi@Yz_w)OKoo} zRN_F_fE=3JOXcRIkUg8_=5#=EQ@DA+F!VY$bRB{^7h3!#_hWwi>y07?$-2(q{RA76 zrQ(0aVUINeTv|p}1jk zVsg~gL>Z#%eVwjJd+h_GV9s;$!p8e&n;PEDS^ej7m_qjDgJz7)-)GfW`+cGA0jBvf z;dbFE-Q_d&_e7Ra(<&W<*w%?Q>t1xNP5K<&eAWIL{^l_2=Z#=>a6j(wvXrPAp`Fr# zQUch|n(#U9zs>l0QZSova}^P?AXm*kl)iJEPk?%sIdgcar)0b8TPEsNUO09Ed7H~0 zocY26^p|F{gW7$XzU5Ym!b^_{g1N8;DOp(8_k+}`kcJu2@zE)(3Jd1oXh3tY8h?9j zMb=ND^hr6WI&`){b5+*_&2+}EV;^NGQ!{-6@b#L7zM6_}lBx=kxgi`!@Fnt2!U~n* zu4e?!GAK5F`s*TeEoJjPepJS{x|d=b5$slj-DuiD#$fF~D%W9(HFL_O!P zA|T72G?gLGZ4GIBgmw?B!e;Aj)DpW4tIM<#;b7w_HBHQ7&e9vVRwp;aX7h&U$aw%g z(Rqm}0kL8{=hiMRcT!g8FpcMJ481$nY;_+S6A3KdsKgvYN&!L$^NLxS9XAK(%A?Su z3-NEu4Gp^^Yvzc{Q*ffmG?O6p8sZxCC~#7vT_g#a(G>%}GBiuib8%hpKJ9;&!QfU- zT>Dpr^?Cp!?IwEo8-|_YVbkso*dAsrC9(X|HoIP+A|=F5-E|>hyZ7Z0o4PcR4#HG5 z6FXM!A^e3`IZw#% zJbk;DKL?jPuYjd{)g+vpt}=dTjgl9zRUy(ACRkvj(R2DyYJjWc}fH7OV&IkpZI4%0P)c`(+-BOa=@R{48y9~iW7 zph<~_tap-I1^AN8ql;Ta#vP@2d)xph%>H2cNJC^CBl3AABc9JMoi3wYqm7T&`4+=2 z`7oAx2TnI8;k|eZYkX~r7J+)|e-FS`u;Zp}s!v}(FZV4f6k@?Pcon}-2FQey_6zWj z4T~F$VWMpAp@}CeW6wjmhJX$YUZ^6qm_?FP#x}oRW4f+Eh@jHqQ%3|5LxUbA)uSA8 zbMTGGpGI><%FH3j`@T}zT72J06BxsNifCl)(VtmIKV>GEfwC;LxG!v7P(T(@+(M(} zx+qfd+q2!+QFgsYttSNBpXurr*3`!TK*zP8!gZSNZ^{wSssb-g%nZ%?L$5z`6e5Bu>Z9(R7tmyD)vSmU+&SF}=n->|z%<&SD&tV9)#$%h zEpz)!5lXU`OSsBhEs$hz0dHfaYA5NOl#9)tMOZW^w4$HKuuCw?sdA91pD!dRup-~1 z62+4T3`*3^#cjh-kAa3-_qKkvCF~gxC)QV{=?j26dPpJzQ+p@J9{5M4E@Rs`V{89leMh%l12$FhQmSH&}#696|fz&ap!n6n*Gz@cG-9kE=a!NajMtMU7Feb(9%I)vm{C zCF-gDR>)e*mKs3A!H?WmS_T;>^h=2vLcp{i=7c95<5a5eqAz4557yivZkjF;;`A}- zicLrMIN_hn;UEk_Mwbk_n<6uflV62^LG)~5`xAWM(s(1n!1Z9nLTmmj@rDXOMi+}!~=%)N)SxdNg$`kDZ47GnAot$o|S zB}bKqhT;VV_{HQ|wO#n~_q}<424mGz--x0o=sZx1?r6uC(OsR8!yk2OxBT@A_EoPx@IKz4All&Lxq7B;8hT7%a` zTQN~rA7QmkKe33<-NAHaDN7CoaCFvi-%v!*`=NFJI5rml7z}C^8gNrMRV#hfx0Xw% zeUCH8iP*Br0@qvnyLq8noUkM%h-gw?K8dw`lvuvGOrl-?U=(_&_Oy8;*$v-z~1-g;SWKRvI}u(G2Nl&8ZD zTp%T7kvMfCaA;;y(Eo#-#SbXE>v7pETUXO0dP<`kM=!3nVn&8AF{lX-*Nkbn)HPPR z60t--yY(lSR&|A|yhjs#x#dDx|Dfp2s|p{7NPgTK!2V~l=iZXI!dI1{FYeMBXZRL; zNu3%Rw6Qn9USD_9%N2gfV1FdA~HBhN03e9&Vp0%}Zs0nC02q6)iz z^sY$Jlb2ggs3NqzfYWu1zj-}6{Mp5#La9x|cv1}7!slABE}SGe$5tKO(Y{qW4$M$A@6V7|*CFa?GYxnP9r%mTC`3O( zEA}h$)V((K?Sl#qb?(Rz=~Vn^Kk@E<V{Z8n({1yM z`uMi~B|aKt7xqUZn?n0us4gT>QRa}=Q=_22UBLHOoskmLY>>c`C;=OvZ{fjhAqJPM zU1E*>9*_N{R~^@tQY5@Szx3~CWn7(h`>@CHM01@T$XdJAN9cqqO(z-sx@>M@3v-l( zo;n^1wk1Y{ZMsJ@;!s+!T3GTq<`R6kKjYn@8MxU~w8fA2t6IkZ!~UDjWFZeUy1xfh z{bXmYA_4LC8t)F~(9im_sWe@lKl@s9z0!dgVuIF7!gI4Vmn3=!$sx*{V{c?`AK_3E znEy>C zCU&(^^S_q|mxEzQbeRzLv<@?XSE~h+3Xi0?&CN+$QB}`AgFSi2v5hK+Bzn9kxUI@F zlE9Gs&!UkkW@-s;Ct3fu< z<58>Gkjp5%>-?J`ynN+qEf*=iMM_>i*i@pa`H&Y!t9hmMXTQSae&fXybp4W z-of{>MZN8hEhdu)O8f=d>*=hg9)FdRI_~k0G#Be_w`SIMMmRVcCF)OAv6Wo@J(ju~ zek_~w6|oaHt^_$G*2u3RjS(SkaK43q^cqF7FR1g&VA)>bJWr&^OFyt4_?C`wOfsM= zbgRp(_5CYToQ{(11nXhG`=(~dh65slN#Bb#zL|~yC2&5GIpu)}XFvLr{lbo!Oq!g$ z6dex>!Q|Vf#uIx=!VviTf>Le>RI``F-IhH&8ttHWb)XM*dg)Ge3?;mldB}y&E6Zlz z09PO+8x=gkhFC^oVk{zj|a9Eqxb1h!m#8GMPY*X{*Hl3S*;sokT| z=Ng?xpt_oBJFeJw;YP?L*VvyRs88jUKR;*8#yeLDs=uN1oY=v`Z_d>$*<0F+Yxq@4 z`+tvP@l`NOTkt9j;E^E9H$qLrbxmuV3oTTeSKOcZ^vO{f+yU;z!mA7=OyjtMt=f); zrFix4p0u8qNCMc1>(DN`JvH~sV56$QGsw0MH7+lWq(-mV%)wCm@P*G^wu#fOvFh%) zz?cm1BVMCYjbJ_kTs1<9zes*UcgAd)Vo?BvjHFXHEf!GwJ_;#bTUB1Djuo&|bkd+8 zh;Iwc$51knUh~-xLTy7Cp^*Wa-N_~7PrmVn;H{!VlB`j7OR7)1aH)2e=aCfakB>+d zliuo|^(FJ6)`cib(m~`8|4C%dx4)hOe^s1;i4#YDs~avBdt7RPrbD5lb!)S$=v7VX zcem|bg9Gkfghvv^ti;zgbi{UuJ7cd;FZK3!e15cvx9eK0x@7#T0E4mr&I+VshPtr~ z7gviT>I?N9Mi=|ibqv9N`ZUvz_n*Nv{S-}K)p9*^%8Bk_n`I=fb%~+3M&lyE_SD?U zuc`mJq9=Ow%=Fni{7ZK#lS_f*3k&QkGv?j5ku07mf(-cD>e6#I_pfTZAg^&pM>OAF zj}h*t>hn6aYcN_nF|2HGb$GGKbNJ>Y$VbCzl%l+wyZRaIKD)v`f=S1tF-CvIs=AMB z!9UZ~x2+5?>5GLW5pyE4RO0@@{^bJ99F7?q*6?{pd|*dnp>syptGX=l^*aY2JkvVY z+wN699v;eBD6WY>+u1TX4;+cG*6`D~{o5F?jblmnW!Y21#Nhd(qBk>ovb^ww!B;(_ z`j?&_bVoImp+HB(yDftU%P{#Diqjr>vD;ZKCn~Ne+-3wEddYR;O)f4Kq-XiB54y#_ zt`i7@syJ4J^+7DcC2O#bCm0;Xba>nvo~p&$m6PU+#&OOgtB4A1hiQM2%B*{kz)G!j zPtT}R3&y-(k5U2ZZYmSaEJX3h~$mD z)NEOa{%A-g6i zE2`i5=LIow{3pvQ=7RYtSt@=*he7;2*ji@9>7qIQ)Wwi=;AE^0J~S)BWU=SEp7&67 zEoqram-j(+8G6*E+Sk7N;jNpUkQUiR(kTIv{kdRD0&EkFyr`E0uYP}qmZ`BSJHOgC zSQ7!KzVtrl$L@BS*j)l{F1^+?3-gFR7!;neOv937pfG2k6hV_pVa6dD6=Du~byond zxO=NI;hK@p*mWP(ymZfNHvUbvh}&zO{?ST1)XF$oNjqlmQJcTp*wFuhywNF0Z48>u z|H7LQdoLQaoZduTsj2Z=d8_EZE?TKqt!2ue+e4F!`Rf(kV4Vamus7xXi(?^j$EYU9%Ez0ZkFGlR=3&l61_0uH&;@19?x$+t@sms zyH;|jFU6@VIqD}pI0o`=>Vf-YnE=L(;bkms(2bFPIAb(X`Lf%`XOR-#2*37YDbK|{ z!*fl4bL?66pTprUIq09Y71nVooj0`=mVYouZQM4!dvC$GZ0}s%blzSk*A`t{lk=DV zI4t~(c(DwB`Xu0|R6!Pf@@?O9IkCj{E5P^+vs6bm)%XW!$7G`zp=QQN4rvfF(V(d_e>sdVnC+0E2x7jYjw#r~pf z3SYZ^rMc^78t&@4I^u^?ecupY0T*V}gvrJu&_y?zJ6`I9&awqsPG|E7`SNUP! zjY5*8>YClt__6?Jo6^&GbL}J;K{Y8qQoqOqr+ih4-Yq`j|8ae(A;;q3!z?=!vUMe_ zyNl<)JC0R!7UUj{$Fb5q$^3ornixM{4fB_<&&sgHU&kIrui zlM!)6XLR@Qnz)}>Jxa39ioPUos&?cO z&ic00$h+sEr6)gicz8CgXK&5f(34KC4Pq6NL3tU&x5}`;VQq3*$xizN|5U-8N$IJf zM=3YR8ApFBlH%`DzWh-k&N~c~+z?KGfE>x5%GIuJ`uSKH9HZH@bRtzceyhhqWnGE3 zgk5vd(muHY1#4NyrB&a&NY$|p|NXmotlB9ZWTq>((D3;kf4whg-Dpdo2m?cp<5rce z*TxVjRJ*tfV#W46#In|%Ze9E+^&0M^?$9{yZc!8Nb{nq} zc_`4A$Jo0K%wvcn$rAkF(=e6X$nmKfSdyInC36{cQ>5b^!HjTU55zt=&Mjwnd8U@|Cu5 zcRS=W6n#Du%{bec8_K3|2W`reDZFcA5Ss24mGk>`x-#?}_Y&*#*J$8Im?LD4*K;^) zItx!;zchnX0LEs9B!wv_JWa;$Q)9SZMi8;BXaIvyO=(|pNfFgKvDNrZE)o2f*N>am zNuJguh$U_Vwbgsu>h0%fCEZQXVK8y#25i-^NJ(6=uKa^8{`A7F4J5fOx0*UX0wYI5 zSKo4fMmAIjN4Vghi%tTSQ|r|8ahGPMg>vP9g>Co=Kc3{AE_cq(_W<@N9KlZAtLmx; zexYtOEqa9jHT~Q5Wnk_hi&=NZ!a3f-g3^(LN9_)?Q9hTXZLGjFAoUZ?AS(k zX7UyHzf%kz+xC+)*f=zK68!lBk2L{&78jSCg#Dqg?hN)Wuf#G&EU9aP)32Qu`T^g{ z&${5G!@&~mXS$n$RW(>W>z7a`;J+BM4IDi9C=KjBYro%nmxMH%1$t-1yE4n647NIx z9bH0yn>xzBCQW)J@$;pu0-T@r$0u?0Lv$_^t=`yH772;NGq&ALbGii24|LeB@RZxy zR-#?o+)5sCuSEUgpZ4*Xsb`dg+rlVv+$vl7nQna@#oviB@ERC+IEoXO-v@{hMI)N? zYy$34?SaD=OK_0|I4smUy!*Vhg!U=leH*m`=L6b^T_X4_uV%-$YkY7JT@W@QTI42Q z1kz&&xvGwz_vYv`2j! z82-9ezE9ui+H@I4Uv&<@m#gNWS%Z`QPhFTbE^mBGJx&=Nj&N$dM0_Po#01t=@vjs{ zmP5IeEibNXItgZ$cPVXl)MchX6tOFSIpU^79bx-nRfhQ*uT<}HrO*X`kIR2A!@h$N z9Z2g4f$Dh6W?qnCox)NyYnN;>wy}nciY=knu6GXHy$wHVFX|cRU)h{#v6GkKbXEJ) ze*9vUJ?C#&{_cxNsBOZ96_~zoOL-yt+8|cCz zO$1A@ayvHp4;A!h<#uX-a_FwG4|Jchv2xdPz7j-PxmCVgbPjRgb@wwe^*^QJtPu~1 zv^xt;O==r8g6h3XW#@ldmYamTgDBPmvvl=N3b@kL2f58S7{3UJHJi@!!MFAcse;jZ z4+WtyfsHRZzTJVS%6|!ja}nW!9+p@PKlPI_I%>6re0Z}pkAZ(X199;nX~wKhQRp3P zc!Eo3bGwW>R#y6jIZ+59zeczArsr_&C-a8=t)Q>h3h&JWCQ!1yAdPNjMxY8wZgf?(QhmT75DM&RO>$3MK>l?zb94U(MNtXBQ~-2mV2UwS^tZo(rHmlN$+r z3VET^0E3j3fvRAAeS)U}lt#CBTetGZcA9n%LPpAsIQOXy&224Bker+fc}w#fK4hKG z+t0WYQI>E3ht<1|NY;XNWNn$nfIq%<{Ze`_o7*Hgf_`rwqi+uiPb~%AB)HfLHyf}O z+{Bxo(b6m2+W(1sd<4j~FD~WuFCbSuXYEuFQDNKz*1>#Xdds_bcEXS;Uo=-xT)IcU zEV^(Ht~YekHH*_P&ufl!7jZj3SH9oBt2E!uSV($2T|d&CPJhgRrjI(XPHIk@W|!?U ztF63jLCxH0SKH8DB1Yd@9hlEKPL6&@?C`0iIADv8ZKUQu4ICLFf!_~7p5VeWVu*!$ zO%m?B-k*E+qqFt#q=pNXK!_hQedh)^?U+=2=)aSE6;wpVdYvvm_@8J4%! z&2Hg~3c0>n6Ia>fl-0Dv6N*v_@jPdieYAF0<**gRBXM7I_wfus7)&#Xdk}buOb|_49(Phf))VDKYJ@4U@`gm!ALXQG zMtL(8ad>O_Bl8(=k>=ASEX*!V;dS8zPOjsAfmWtNOSX>(>oz#wf%hpviIHy#d(B^3 zxinu)!R!|8?3wW00F|`k(vTWGTW`XHo8nVK@Q)~PWMWHrm+VaMx|Wl?mW}xP(eRtZ z?aW;hm+$w`R$PugF-pz9b$RK4v0|o}=n6!)_o;>{+6S7x^xUD`EsVcR+Lf}U>Wj#{ zVJDjzQ=72kqcSx(N@gM=te&TJj0T@LZl9Y1Llmfe+VEOMW-t0&o_oCgx9+HH?A*eA;``ylp6`M-2Vo%i1WIt) zz)eh2p3@#|SQ&1JZmYa#?c9q%1^?geO=P8TXltQ}z63eTY+B1TgC& z`E4POJR;j>6q^I_hjbJkWAHfC$lIhnSfvP#+zg*t&=clGMYrfLw52F#0KxA2>&wCD z{wTxy*So*siAt3r5!r*I21GBh>@HtF?^!}x_FeCLBa3;9M6zV1{^Ha&W8 z`euMNq)+o`a0=IKl9B|jYqaGb3@L{VmD&zi+M0ML( zHBp8`ANvy(NT(%xU2(P%i&JNmq& z!>Z?3Y`|25NC|Fo9D<8A+*Dwfs#KN28Ae(Xk#bOF1T|1TXsA7RGIJQ$AUvFhXIF6S zYmi`l@YNfC5Ux=E^g;gi!9Hit1a4X^>t#Sb!WkajEf6I0Y@(wEF4eACCmOK(g0YHZ zd6McyC8&V_;S@V$>G{7DV4utc)-t3oH&vYF0$s50H`{g5s#1OsQz#kYbF|7$0-#I0h+|A8SxtUY7qwb%bdqfM7Yy|mEKjpyD zx%V3}5zXSCc`({Q9X70u@brhFCW)twKbMA~q0}+yIc+{Ui6OTJ_;+BQNvoA?}w6IpWtAVfY;X~v3gN0_`Ls3}TP zFrq48Z!yt=+xivl4+e}|zuY>n7n!s+d^Gs5n})U2>5W<8?e8=`MGvHfXgf^`M?g0T zhD&#XiXsPdhb;0#h;s)VE`OD4Um&j83_ZHPC`UWorp*>yFY)Q@&oj8x?3SC~o7Oiq zm3Hp}orR&x!7=&~V}~R80|#hS`Dex>NOG9`AJ(Puux|xaN_2&HbH$_341 zjv&hP-Jnqc$G2HiZ^pR6&PLn~g$c&(BAvA)C}s2RlpK$)Aya<&xmiJD7|A@-Q)b`4rN!X@w6P75E!k0y0+@9B~XO!+9j}5Ku?`S(kcVrV#x`A9%n?d{nGKN;xW%Ow3jE~`v=s`{5BHdfN#uNCI%r-d4Pg5hQU|;>^Ys^bQ zzNS6@Y>(SxQjr0S);G#H@mW$9dJEZ)7D}JcK)nptv%s>_}S3@&@fb{d^ zBvb^w6U^9=M6S&u2fJ??{?ah8M5OQ%q2dFZl8D~5MjRaRK>|d=+CU@zd+J?S;|cec z$p5v>F);U_whT(%9#ip~ioG^FhNT}S&G0#PmE7s8@a618T3kvvN*7ZxfQLmu47Kx3 zoW5#RZJkiIj9WuOSjqOL=RhaN4q@Dw>_L@kL<9s)luO!M&s+}Gp6rQL;+GudS8BHv zXl-fe8csAF2*Ap$rd z+?{@*T)Z-c%B+d1lWBcAvMo|#q?kM%T|c4loN@XP&rJZPhg(uNBfCafZoQgH=UtT(gylD=*dolZCYx++rysL(J=2Ehdcl1h=qJ5`Rz4KUEW&9q!4cN(DQk~*O zn%*5OnV>Osl7w?Q*O%V(tv0p5FlC$ zXjg{`zzj{L3GCj~O(zp28C17Ngr>*is(TsYIhnRR4oR2(qbq4Rz=r#t!XkIwi}{9H zhgc5>$tT7`BTxmH&D5{r=y}W=HMhflD$(|2gCpi|=vZ~8y-LDO6cIyF@}G@Szs_3t zadIp+&pKH4{(9HA;on16!5OPouFl4AVyz)ZG?R8eaot!k-Wn)Sc@5$qDDfs%(JSjen?v<~{1 z%V54O(B+U6M>Eb6J&Q_93Z*Xe0m^63*X%5VjZQq zO!=tMV$N=e8HE{BBjuN>-KQ5UI0o=SAOjJ8rjF0#$AXOVgia(eD6{64TpVWaVkR0Y z>8I;cVj>qJK$rQ@{-8M9x(<0_q-sIVGzME$($`BorE;vc(rJcVVqNpiP zM;;7UBxDw1i(Lsan&M`RN;zd4uVPWV!dS4(SL2+^fh{X8haf=8{xt#uhsC{__c1Va zHZlit6B!@FGNhC_0un?abtH1DVgfmULUAen5;Mc%EfUFrdxch40YT$~gZ5}rdv@(S z3~yTF-7%;VQHwe>Ae)vu>(2u98|OpSB)?wZtzSB+nk5%Pn8N4dd;ZneCKgX@UBiD63k)^GY&Iys> ze!ZW%auEZ8mQp#-actMb{%MxizfR`_7|pDbh;g^7Sp{I=>B9nG=of#I^y`;=NY_F( zfYDWoPobs-6O69V^SMcuZPBYPIO92|f@2$${&7iX@A%ROsCXSnoYFGO*6d51!ZOPu z3s4yazieLr1y;771x2mW}&`swwxw9Hw!IvPo-A)!u8|B2K<@YP7IuGtz zW6C54SvG5r<(Ctt4=&Qi@?^yr(1dAq4CRNb;L-xz5Vp!_j%zxP0yBT%rVqi8#RPXA zVp|aScsr)H^W~qRb$BF%Y%6Je8gHcoTw04t;;bSf2;$$p`cKK+n-&PGpbsGkH%Txe z6A_8Y46YAL|Bk?qE0QbW6^Hs(*C!@TlJ4box2>b^ljfDoY-bW^-Ep4bGP?OIpK=ri zB@~2ufWd>$qKrD8U*RoVOC}{ zIamMb!UXlj#5f5@s;#8qG(b`0NIy$IwDoEs?J%k^_nQ^G>*wa}dYfn5yD+e2!v%LG zQ3`ffDhj})zDzIF)Q-yHta06$t-8WoKT68U$e!c!RcmjrW?@0^npR zrv?8lKjwm`v;;5GW$;to^~+%oDGm3kbjn>q#3bT8D{TuQ+pUubM08N!#v*95rpp-o zsVf|jwlb_HZLNZ*L|&bbvtdK0^4oq$CJL~n7(ydzWOb1^&I{DklNIJPGt+`PvT7)_ zDb*jeIH;uN<-6m8%iDAQ)WuPlyPQb)&{y#;WDt@;g5|XV`v{rVx-K2<2Jv>2X(N?b9xMmx`p6rp(RjLnT5(@(O+w+YbF9MHIy%b_ zz42DJg0nN2yWyyp>Z;m|rte&_tzKqi(6cnw_0V#9KTBdvoD_p3XQGeuijE#5BgkvD zPU=)vr^po$T}((y4PNXT-zE)rvOtD~gmsmGL9ruNMm4!$O*5eHYVS`VSM8)*Gx$Bp zn7)H2MvtylJW5fU1NW;r7DpD^?umxss)-2`*Q?-r%)SfEbNoi$wECT4C{J&{q=xx; ze_Ql-lSLEJYN#AWQWX1pTWgNn6x4#bVHZON8QZuxKduDHE5TaA3p(9#t5;(QJSa0` z0Y4>53Q)N=L2Zwmu6&EcHiGHKoapTJqyr6yCbEdcPd_9C?K-$XrdkFOHX^Qf)ogn> zd~i8ouE(WCsYF;;c0rFL8$WN0wPqHy3-XxbyI+O=8OJOmudOl2Z ziS!f{O|`3hUDFPc-L(#E#x3IPC&z{vF{1RQ3923g33-@)z&NH}etIez_f`{VPi=Op zsJwq9)R=tA$gQpV;<3JUzr-Rk4g*n-RR&$#jrkzl9+e;??^p#j*XoS@o=966;jIzy zDyWtjcY&1e_gl?Rq4I^fMn8j0@W*D15inJi$Wwv_jPgD?V7+Dl8k+DSoy@7c zm9l;X68@qDI=n(;@|Km76xn8;m5s$Ao+vyA?)MB@0cX*A>t*6|w_C%ENL1m#Z{vY* zwV6>+nt#OG6#V4P;m6Ad5gq#CEX5}Wkcm;Pl?BEfe_m(-+y20P>b4xZ61uP?v7G1Y z9xzxpr?1bB1Fa_*?JXU+)US%l@BzP&kq^I|Ph!1Jsygo2HVIaUd-{g7Xm5|+JM@u{ zvR8UH_=4&$`mIg+b!le8U6LkXbn=bxkd2F#P37q&bkZL|<8fb7hKR=;epxpD?<*Xq`fS^f%$)~_EIw#nnW{#3}R-Zj@PA#pQ=ZlZ1 zd*n{vYw($yL*FZed6SS?EQu-X8J)z5AU5hVK_wAtZb@T@C|PWAD7#uQ!sGyVf;Iom zw;!m#f_KIg!!D#S;+`ybW=0u>&}ZR>DK2wy{~X%?{MHaOGrXKcoA`#Rz|<=|HNj0D zHH|&avPA|=Iw82+l{29y+zy##4Afprgh{-^_F|jyV#si={tQg>ynvup!eqMtejQvV z?`&K}US@XopU_2Uv6jHyQM=6Ut~;#P1Mu>r{DzFR`(ltM|xh|9hscTq+#7 zRMfP!Vbmk@78|^n@At4a>1eFop<=8YSNEVbnM$m!IWP;*vo!Art`ssA)l3?6ux8cN zg7724I;_{>?YAkqo9~|vH!5~vRJZ;+#~L`@!7A?7E+euAK2V7=c%H;!TCz1^7cf__ zpG2Q+)aPSTCe@S=2s6bmEsSi(R@8uP&BKk*-mbKe0FyP9+QBL9nY2$HbkISg8z;%< z6vnY9rj;mEi3X9txNWZI&n>6Y?$W8!QvZt8i=+eekjn${|(6C_u@B9 zAa#5{dcJ}GeIZKTF+2L-@_$x@8~A>FHBldV94MN4=IApa>N^1MmygW#avF;Tz?ba=pm!qaew|F9!kIW{THFl*GJ|%kbh`OW!IFM zQm@K`wzP*$@>Lr}ycep3pW0GMm&myTe_lL?Raj_VwDT5E?q)%U4g8wu@X zCrGZnTiN;5V&>Lk`Q$maK%ZoglqrCv#?qO`&gEqnbDezzwpW+$SJAXTq?RNp0pp0( zbt@z_ueqDfDyl3Y9OmqYvB7-kp(T*J3nnfgL1xeug^?@)YNK-q=LJD#KjAg|lKbO& z>h!xmp7(DJ2K7eBSB=YHi>tERBrvVynYvnlmC8GRP8lMr_ENy3Ufvu9&OJa-&%cq| zIOkcY*UOB?E?efcd&lfw_|{8ojM{s|&fW|eTp;o>E`L4vLP!s2JEb29Kh{6fOwub% zeu@NqA&|%XH}uIV9VBGIU$11sU?M+0L2Twgy@B(;J|2$Ep56{#Qr)NLSer9h9_)hf z18m_CsGfC(fpOWj%6%xGuujfiKxy==U*d;^r@m}gh&?7ybF&b#| z!TpVL(i@b8muT}Iu+|;HjRvCYOr-WHvCaQKWyh@8CngmUdlu-j*{zKkZ=8vyrMMrU^Re@*M?Uh{4a)C$pR&dryKQGJ;SpwLZoPYAf{NA* za7KSuaN?z;pK0C#>g->fl!k_%XXlC|x8p{BDhVc{x;)}~G=%f`wLMQ3QjlM4-}tP+ z5^ODUu-!cNmh8u)!rTCK<*7`G7cRgd2flQp60esmgRmZzL)L~Eu#R7l#IK`~KOc^k zZpVb9zLO%o>NERFAI`inC8wsjZ!`4SCBRU%P1{K>A>_IdI*fu@E=1o09&`gP-|CG| za}34F+f~^9r3hW!-Hm1wT+`wSyYL#usiEzn8VC29JII^?2NQFm4ho9c!{|d!@I<`B zazqt}D15IM0;AalNkejhpb%eR|JB}PuvUecR{%cA@r=KFZvKoAV=Cs$AHcG}CRtRT z?^%Fjx`X{$=nfN1JM$5zH7uqcGT+ z-sNi;A}0?)l7%zPBInfFovmX6+h61*a>a3%AaExF?^?@o2@?G&tAVA+!2;BQ`m|ND zvF@qxL~KJSs3g)9n|Z>7)#QF~9Dx4_V+ghz!Bb&5F|j2=?g7$yS}zu=r}h96{po>w zZGA$CXI=ltMx02dLG+6Tz-}2dd+`OxKM3Q&OzK;m%X^q$W?0}OA0hTNqD~+B7t-$a z{mX?<`0Zr=&_+%9McZCW?wkqxrORQ(qp7@&;R<-{~@h{Mj0&IC~Xn|nHoj@xVo?(Umw}id1FbU(nPqynH&?21z zn~P(C2!erk7fxA`H3;cs5--1g7wVL0*&cUdotSLygzV`;%U#0`J| znKUukM;bQiTlgOey3>9*A@`7Voss+MRw^H;laNp6mCWQugMO&*g4)Bxz-29hL@15@ z5D(3SPMo@Z!%Te;!x4Y^_#JdjgnPHNF}?2MQ<}`Hrrl=Cv*t?_8|-wW$W}Z!Yh_Fy z0J#V+?on>fO}XjQ@#Ek|=)JMZAYy+h#MG(yh{MKxDEp1iqaqFm-bkp6|PlGgLG3ai; z!0Iw!wZrQ$BO1NNC?<3TxV?3=eS7!m$Ca?{3i!Eg`|(=x`Bo9;6UR!E{(y@k&k=pe z%`nV0Zu%8(sKt(KvHRsP$P1Mf?H|vLC1K3IRVOWmDMUALii4`e82e9uM5;rZY}Aut zv(0gtCLkqeMxr&~sOlKO$kq`kxJs5?qPLSU9-83dzR{zR@V2Z;p)ilY@b6h`qq#fR zd0>sF`P91wjag4WAAgc|R(=9Bdw+YJ$G819Q~1{P`uopSkm#juvxBZDsJ0ZF8#|vP zcBKZf9HAiwMY|t-<-rnuM2(?UyniUsrfKiWf>Ew*aal(9*M}E1nBG0%g2yEv`BX|2 zAU&m|uQ}nI-2gheW!FK$6U~8ZsF%(CIJuPR*oXHRS6djcZI__~$Svgd6v>3>KEe^i z$!Q_h6kFi{hxg*1ewGCdz3RiTpJ?0?0#ia?c+_9H?-tX8REK7S0UK%lz44>N7It!$ zW22E_pljd)n>+Yt;P%h$z}lflhN(IZD;$u{?+egsU>sSx>?ocBaC5uHdMR}@KEQRz#cjS`3;2~Xi5!mSz!^^aW zPCq{P*v0dNv65P0P?DYH8xLUT4gN0?fS_RNz2=PxidZ5_jU)B zp)f-}vET`Qt*ID8jSVzos(Ltt)HT-mW3iQHtzSpUjlI$|$!E-v7Ele$#$BB@KwA#T zTWi1a4*19_Doqb4fK=Rl?YoS=GxklbI@(VcapD``Ss_g;%1A#yuijht#C!+g(EV3U zGQX~~+I$7Iqe4_wVU@yh%KiSW*W0fLKmXh~Y=wb%C6orNA2QPVBK z2NuJ>=5D*dEjFKjnr-#>&R8>C+5qI!NjIP8KVZZs?Wmi9=n)mO$dP!Le*xtD`D?4q zKzR9z->k-VTPp|yp8EGpkf?Ux6~Y0roL7K@F;1VUhMUbe*OcTP2>O|aPCm?#7DlNC z0kvnxfSvmm7=E76YDglSooZl8S0Fngv3ya7*`Q$^Wpya1h!h^Wx}}^3%UZ@Qjp4Lz zc?x9pD@-sl(sk9O(-PCVzX)*rj!O7t|_A! zSbKA8&mM9})cI$J#^0t#t?GZZ?v3=`+6(ax@KedK+l{yywnOT(3mkol^Y#<>msk&j zqsm?gsWT^o#E}xwE}crJiW>$#0sN21E=er^0-B zOgi?XyYFn6>?5JW%d@xii%TtlZ@3p(K2)&$UT*z{{%vB58uHCzL28Q`VZu%9)5_rc%n(cSWUq2 zFSJ#iGTGM8K^zoUZS;8#fyj3jkUHsKduzEn<#!+BKW&u@Ld0(1Df8rRZRNXax%#lx zPIY9d(@^gNnl1gpMvSbzvh8uI&HPG|3=c8v4O-r>n;kkb3(xWTvd|STF9sG$IcuIj zl80ojpTGP169hqFW#&b;`>qR`zDMqIl1dN&Ln?C?LU+Hc*_a|KqA zD?1iX|9yqL)yN+og^hof z?Why*xV6ZGo^XeFtcgkbQqF4WBHIkaudaNBJFdUBd{r9Hufu@ak%y}L3VUJl- z<1F7h4G6vFF76mwUhTHscC$rT>6y~|Yo|cAa^8q}udQ{9}e*#ajU$}4;icRNeXXM<(F z7GdgksmBY*UDOoqkF=*H&z=LsPG(BSH8J=loDDBO?pp2uq7Gpzrr1 zGj*+lyZ-2g_~^>i6{5taWX9fppAdG9UBOm(YiS30r?dquSyz=dvH}S)mV{q5<8lI% zXrWpUBFq5G(w5Jj9YzkLny%F56V0?(stTxdrhf23E39T}^vnVR7Ohf=9A>`!yh-*T zOGHQ|w|EV6B$#-G@v4I`=L&0Do}&VWF`8||Wa#2%E!gkQyzih26PsbX3;AhHm~m|sxA~gO_&w9V zm0urR2|0&*4`9R5tKXu#=wpBT;D^Pds&axM!L;kO8Fkc1O6PtZo#e=&F)|C_djx3x zK-dWxJ&5uD5p5qY;HdnW8sIj&Y5Nv%7S^pC-*87cOWJ|iLb?s@XXZiv{;y}dUsZ(7 zji(+}2$^}G9lj(V@y5q1tv5KE-}~jPWN%=lCI|VK*Zn~5-9kUKd?-lmS1|Y=R@9+@ zh~G@0m2yu(^K+~0@~t3De;Zg7Q7VyZa}OW-qrVmE6w?0D63$QtedpqbAN>c^kXEb( zAqxMuU0d9ocm~){C!>vR!g)C61m+f*8&A)m`fL&tv@S?O#+vPnXy0&D`xSt?WyVOutY(ST=Z@YSCCGGZbl& zVY63o z!XsYDWTcds*jy?EQsy)Wp8G)V8+Ubx4R4-Y)7udL8*2&8&OrUFn% zND14HM`)t|`_mhVJwUI3=7(h#TcO@2@X5an2GZC$vL|E{tPHy^e|P%+iqxAOF$L8! zMmU}|JCk&Tq`%Ma9|p$@8R2bzoT5-^Dv;?h8dx4zteuiQYRc5wFJ=RuArmaZR1fpJNs*$1oQsW7b`Uq6;sRG(@~3 z-_(GEIBhy*4PZRi(5YE(67!I}(0Ldv5DP6p%K<}zn}oy8gb$A+3LPaFJey&Iy!TeA zjG2xP6n!Z&N686kZWz}akJ0+8Kv{*-zfY%*5ERYMOQ#hejQ)pi%4o>GJ_L?;&eEtm zmK7aM5}Zz7z))s_680)aY#H2If zQ+C8CrJ_Q8@#AETj-3jZ`&4j9B4v_rwt1r&4nocpwyLj93Hqz4HYJ@tKL#V42{=!4 zXgdw{Nfap~gH`3f6MapecuABE60lS>jXPlJOO-8jH7+DM8=g!$LW6`aRk8(dbrBXy z&z7kLr@lYj#|pMTK*IhL3ZVe9;@*>erMKgyIiqJRO9g}HlRFcq>b`};0D9{F%kgU5 zgcTHs^eUM_Sn`9y_+K(%*Z>bjP+!aY zhgr_AzbhPUi;2%drlog>D=V67H$CYZMP6p}Swh`Yj=zGJC9=ylRyLphitn6LujKeX z2JJPqun0XUD1lqq%fCN*7g2rWFo)@4&LZb{`ChL~Dj9w0n<@Wv_YOi_vFK?Y2(2aO z?E5uDCGZcC-N>r;{m4S6@WE(1(EMZ88wTOR44vW}VeAn#nk9|~nuIiXCsNf{>7GIw zE2|q6R^h()iHIT&6~f?xO!f*qW*`*_Q$ed-lY~GqrdQm0SE?JLUIbPs6ix0nfhNMe zQH1ukmry$L@dtxb8;q*eP7F*d!#YG6o(O<;W{^)+To6p=8WI-%Tqm|MsD(tr3EDfh z(F|@d3g?l%L^U2~AOnZi{e57N$bYnrAcT*k6Ei=&p-<5~3MGVdHGWB|FcyfSumVL<3lF;4DAR}a(ghG*S(-3&H)xZM+LGLRwW%I z>0n|;cu?2}=X4pvhgdT0ZNHLO1lv(V3KOS$M4Au0sKP;9V3wWqxyZ3R#s)0FZYG%5 zc$qwU_QNnXGcvPiryJJe^pR3+hKN$>ld#rsuni(X7q^ciW-)zI@vNC08Zexs{l}ne z6aPxpXG10tr^{e-!~a)S8R_pDj2Eveym}-Ou?vbV5d5IJqCdq+SXzfVr*{WyU%h- zeEgfUGo~X`b$ynC++If;NT-}p7}`_3+44-N7iYj>ilZy z8icqk`7Z4eVmXr*uyrkQPaAx$>r`P$@7(Xn(gvyTP@1mmWIl_XUvWTv?Rq0MEIs_9 zi}n`6v`fE7EIqe&l3$ALB*NoS7nEOO0QRQ#g2au@o4YC*p}9=P7T@Dqfyw%^GNmiR zGWEzW^<3Q-zSssTpz!^an}}^EBOOLDh9_|gv8s%Trlpaqypr{O%7UOL50 z!a+!wklN3wXz$A8lcT4~)3pH7hTnyl62ZQEd00;rTfOh~^n2VdsnqKiGl4czRuA&N zr4C<$cG|sN_LRj7>YddGz7@%<)|f6+O28K=k*`s>HWJV(P+DLJs1vxfxL{nuI4~g>RSNL|OPXmL>f_l9 z(bC<@MN%RL(CdpjZBWeu9df*=+SF*( zRFqlnd)BBanrkK;VCeOh!SR-{fRc|9t{A+89AatO48e8^g;_%RvK$80Cjz~|}B?oE9Z=r$JH&IkZ)_>%;5DOKm`oNfVoqIH*?-?TH(!3qR90jKFyW(h|9CJ&UK#(#un zZVG$^a^ltHWrLUA+nRm8Z1LEjc6Dicp1q%acod{a;;U zF3Xt+T*N~$+FE+ihy{Wb1Gf6@+R?W++G6il*jo6ND8~A$6GyDi)2$Eb0&t)*yf0y~ z9FoFY0nn*-=d@a~)PnkWAcxzN2Y|PEG&>Qx`CK5vwhk0zr|ht0k;#KZSez~!as{dt zU40k@dU5IS*R@SpOuVxJS4kGx-&RCG+V?vb5KYS;MQ8L#Nf=bDGO{LTou8;F42N9D zPdD;gH`3eS^nqc!I$CRE9Z`zxG!?OCIjneoq;qnDby5yK&jwVzls_r%1wdDHMbMqL zo0Uaaj{IN`1Q;OKo`b+8G9mnB?PI$K`eVo&R{^wO%^w$Q!}baDh`3UBPjQ*&2;GHu zR&L9~U|S83kj6G;g(g_o8fC>RFva&u;DA;RQaZF34s8!&SPo9(;9qYmJ3GH@&vjSb zQgasxLH|vvW1|-J7EHMAFYwp(nMc{6m$E|~tl-?Br@@|N>26o-_#GdBbEeDpJbgXh z>me2EaoG~VNe($eV**FUhL~awrqDj3&8SFid6v>a6%3zah}#eWGPd zLa!dGo4JWllA2^QC=c~QNhX=VK@}N2oGMP^q8Byy{J%OiXt|#G5T@s3jn_TDGK}cq z&)0XajTFhK>rGBj#^#)j_9rTL%HFl}ovjsXMsV!nvPZDoDy#t}bxKEu!-iK7f2_77 z?o~E@qVRQtuOy!vQA$MJd-O%M&LCSU+>fzbjV{S3Jw{9P>|3`ugoz2@g6{7n>kO(? zUyUz*L@NFWS~)CZp@kT&FZ90pG2gCJ6m0SIW{2}*hrjh1&yJTzo+!h_#nEK&(c+n( zxT8~B!lF)lRS*5EA}6#s^wo!6w8h=^lSK0Hp|@xKNK>*n>LWh@vmn<$Fy&l2p+T`% zJsDg`7E%^@Ya9SgzL^~NP~hj*4n3ro&(zmwD?%wb`y*^^$){0%;`+ne_B zF~hvyXUQ?tS)-9Pzr%FE(4i2wZZuNRv$*ozkk|LmD2L_@g>gzc*;ZC)Y0exKTqmwq)dlSK87eR?9YXA;Mr(nHN_-920lf#&AZS|X-J zKw+X0q71k)!xB*%WP?K=uFavTy{jhte-t0v41uGCbMey4q()lOf~s>O!zyp5AmpF7 zoxR{FU7nW12ZqbS^2ci}cH7S&RuFu&gyOr88g>hl+Mx%YLF_-qjnfCO@<3}bEJv_; zrW_v&+Erc#&<`MM_oTqGti{{BHau=rOi>{4GteT=gWqDhB9Q&nADnU`Nw&^pKp6b| zAP0-nL>`9@1Y|cw-ue#>?8uLhSI`ZixSc4?o?m=uVPyxaSPpmerenkhe=cbKyf&yY zSgEV+bpeu@COhN26?&@KHlJv?qBVz|7;KdS7Hys{g$Jdw#+0D>*Oi@wJ?&mp^I^?X zn9U|RC&8_CsCI+=!^nnS?%U>V{U3%K#mSCC=6}Xi0IhF3#LKQ;+c#x(39M)RvO||w zW&0D4RAu=Wi$UrJesrx?Z+w(yN~j5}-8F2r8|ZS?T$+^Y-4k5W`O*|9l&2cy|9oWP zp}_i?uY`T;fB7?Y+LeE;J0Yagh`Bqk@R5O8D-|&nahZJ0lS&odbt;NH=_(=x73@i~61xHE!{3{3 zqyN5Z*S}}j&pRLf&R=O3B5KpDsgV#OcLokk72VvUGFx#SYG5C67~fSx%s!3}h)SlV z8K;i?9KD}D_t_Q`Gn9!!ifB#Psj$}5BiTj-ly{8a|85LXEo7tATwyj#w7#%l;7$U+ zQ*KG@Q{#WnqUP+nnLn!z4BVk&_PaANa?L2kNr1{bVSM8`mCQI@3H$K<>0Au{nc}yj= z4uRD0P4kE2qvp)7O|geNV%O}+7|=zbuk!C*KZewSs+kVG(EpzUu3oB7E!of?U+ZSo)Sjg|| zQ>*3uPVejTJm=Tfo9)*}kEfJM&*8*Tn!hA#WeLhq@N`xnE0SOmWJ=Lew~-Lm$0PTA Wk=e_4Cz<#QBHd{|X!#5bD{wf&u)`1@?W_b>X@p-a`CAi&H$)kjz1{2&o47>Rz%CyKrZ9(qLnL{H}f} zwkHldf-#0t^wVX(`FMq|x5^{bqYgyg_{<1^_@PK@=}O(c_KOP3fkt(y+ra8(C$?aZ zYdmf`dq+UPHMT)1@heg=wgiN#|M}Wl_`pLj^>(wT{i5WcE|onYs6l zHHkYOAkiI}IoM6I^vl#YKZEwZu*%*gld$9)&P}q%(^qrKjt1d=@WX_0%l~B!A!41y zqso*Ye%WUC?&9pmGm{t^I;#>SM6?@U5%BeXu7A36=Nk(>&=Mq^hu1ixFDegRsC>J! z9)6R`lz&qRjT1Ba@Qp_dvsBKK$Yg40%rg7p+jVnL@>){G{hkwLQ+RreN`dFEg zHd(;uQ{np{OvBIb^FU+f6joq2A4xeiR9d>6TfpH>1tVI;(L4;<6P1Kvu?l*igPEa3e0Xgaxb&n%J>%%0=?VeJV#W4Q~x`&B-j zt}x01R5)nr9X|ad?gyqV1;vS)@^|E)O50Md+yIx$HyQNpe0&;-4f0-WBou2zkYq(X zXjO)fqKGI&K!&spazbU?omF-Kh2u8`Ei7c`o;6{S(xHQ{qOE!~P1kmn`n2T4>YydY zi!L<1At0DJGD&m5Qw@h%@JBpuAwVEu@egs}51G$d72mS<2>xb1GRhcDI}!hx1^Q`A z>4{%(T=Z&U{?Pp|5rIQd!e|w$7CTXx0D)T-!r$Z1kSp4~riS1#MB)La-FF8^xKbo- z+IG@OXD?tV|E%R5&~Q^=hLn+)uO_G!SvcuY93cEWh2v=f$x_fjxC{)E=Rh_&6ACbr z7-XEDP8r?WcA3#Uo|0eTftn?R3hDPhn5B#Hh@nyBqmOk5%-iL~cEJsy@i^$`=^?m0 z!FLY81z}T5O&|^~nyfuc1jnF)Vx=hQmuTfbaQ=`$$}IU0QSK+O0VB|X z6+-rQy%g~P=G0jb6y+a#iex-jPRrgoY^y1Lug8VwPmLMBkG&=QXIjg=)Wc0=0N17p zSHLK~``S|_WtyM z7tMJfx(yG-lY=&RGbP1{IxHOh_|D8TL^*iw(^4CzF7rjM;no;Sb12m<8(JEG(cEVhq}AC!X^og>^D zoUFwOa)XpO3Ihyf(~aJ&d14y*rp|s9J_MnM0b&NZZw!hWzmE9MgM3+tN=~Nje^Pec zQOhd``ywml@hW-Z>YH;tQ3qg45U(}n{~}xy_Pl#x%?adEFxngJYq+EKLwHZ^I~0P> z`wT9hZ#7wc9rOtL_3i*8x)*;|t0j-D8@g~P3d0-Mk+J%Uy&FEVa@Lf|59hh^>cvx^ zh1}X83}yH6Qh^=wLtI147ZWU!%ZLp)lkKFqevg|syoAFH@qnaho_>H-QYJAq&M!NB z_tT9p+pO*19fm2#XT%0zK!Wpu`;g0(2_V!qv!)hl6=iM23&4a^7Zv(`ots(NW0tY! zzsWXt9YnoM_(jNd1eMDjt~G6lbDWk(Fewg$;^ycy{0W4Cp_zGyNuv7}&xn0@>rZ3WLUp9`=dfa4Oz%X-jApDm~1oUp6Kx&u?rEuE8%ChAAsUM2ee|zB~muU!yyWD!iP^>h1z%PG6=8xxW;{@ z#0KoET&Von`l}m3Sea|&=-@J$ccp!+i0a`Fn3pe)T+%BwtZ-%(FCvX4-YDdomI#>{ zSt(S;N6Q(?5V7a+Zwn#HG~D~lHE*7A_<~L|Uzs+x$P<;Kj;ux`R`D*bunTuvuGk&$ z{*eWTyEVZRz~#ftQ{%M52&spHRXTJ$Xz&NYyC}D6{qI0RZDeLzR`$FYn`HBdA=nSX znjdW_Um29aJEcS={tYtk2-blAip+B%bi43W;LqNx2tHwp#v8ETQO(XZBAM78be{GQ z!APzMYw|-eDk1z0NyrSMvo&`|SM0z@kg-%01rRCx3%rgOiYKHJiFwNiPrnq2`O#=S za5N5Nkv=r7A=vGIRs?gQLk-L%5&F0jDR&-%5sgv}5Fa08y~o>ysXjk^$JZd( z5iZ{gxa2a?gp&XBq)}cL+!Cq+KyisRp)ifL@Dnl{f?t{$2VBUi(N(~-QN|RshAI6A zgM4CDC^{lIZi^L~0^ox>_p+v7;I}lO)%pW&(*b82Pyk4r@`q_uW{!qhd!Ut@{<%o< zRwkNVpqNfp$&Dy;3Y?#iR0z0aIcD&W9!%mnVG~Yg&)Y_^fKzG^K@8K^m(9U}KJ9Yq zwOSvciw_^wp`?MGa3W8AwxA>9=<_{l1GY*~>?$1|hZG6-Wx7q{++Y+QQjn1RYBYZD z^yn}M>^3MEYZn8=qxG`hXKgZY4bQ|yt`K&4MasTx!p%;w`R;_+j1*!Vb%j`L`%}f6 zdr{`?4?5|B-%(ZyWq%hhQop@?u%JvY>=2S*3Gz4*b|uFN0O_Ypf){g1Tm!V@r~XBR2&1{rSstL! zyjLOpz{{xhL^^A}OYcx@ld2cf%fhgHrtHMy0H&Lw&lla6p9@%+y`J$Ylzv8W*S*!S zKTK3ZzC#&gYyz-9kTp|y@v(_AWR+CJe5xfLyp&?>IKNm4J0u{{7-I7`s-iNNQYI;-?7`^)(W_=5M>Z{g% zuzrGfdwwVbeR+Yz@L%|=3PTt||5ESs%y<$$V3V^UhHs zk`9wZ*KIi3ez0Wpo6Zt~`v^ctDMx6?OvJR$tLFL({0i`*ZWb~D98ii~N{8_q`tuJ8 z)B!IoEaHMZ9PTY}_l^sBumaUtdq@`wn4PDiUOQZbRk5E(MT~Y7&vo+sp`?v4!3Urz z@%x{@9fI&CYdNXn2w3-|2ORFY#?g8L@1(O3#N zHk_UQ{$6rpsy2n72mYDsRAsYV(csmZu4Ebj7BcX3lB}i(_%FO#WFkpBo%H}6Hzzq>Fq5#z`xoWp#GKS&d6y~QWJliqXVQ-#spiQ3CJ5~xdIF+B z8Otz~(@nZY5h~x;?UQa@PA6PawR76(p;UF&pr4;Gd5eHQDRBZma5K_rB*MK`PWR`# z|L$KQ*neu;T9(PfqN4!Zd#acCX=2>8AMCll6un+W)3LEEBOHt9(b2ddL8FSl==9w^ zuT?q3Ao&f!4}XsxSZ2od{!Hhl*~jp7x7WYDju$%TJDHL3{0Enys*;#sagNpoJ}}TnYK1$M#9D5 ztAv7llhO9G1aREg3W$NzYjCQoT<|ehb=R3oOf96Eglm>STj~X z16G;u@1H%NbM*G-0DHJ80Li?BE!Z9}1nmXY8pgNy!Au27GxT^=4kvGJ6|%>~LfR)6 znOyq|d&bk8%=+V$+;^8(Y>on0$`gn!dpPao0#1mdt6XpOi>~!XHlSPV0j!OlUdWoF<;&WOKr8|4LgZSbmmizQ^<(NlvN7>)|I=a+_;7@ z^5O?T>B|m5G^U`gEdGwCN`?#b5e*+Y_I5hQN!^CIt#F&))ECI_^TrzKn}+(t8+K-6 z^0XeDgIGDUS+t_gBQqgFS-f8l8j-?t*mHO+DxPtFJ#OVbMR0pm>u-!=$34dy82t=( zld%|jc($3~+be5dHa1~#m1ZC1S2=nI^zwir1>PbvQE3EMu3-0)&bK?f8^4cp$V~T9 zjf8%?jchxBbhdCoJ}8KoyUUN&)24U-w5oP`m%}RD?7~ zktvQZfUg&~lXMuhK$TnyI5573l>JwyQQQA~)*ox`YErTDv3YLQJfqn-m-s_2ZIimD zn4Q=rKEfj*J17JBz#`hpd_L|kwFkSby$v%MwM%o;YyzKwEuk{4ScB*(K45I);j%*Z>6URyA!19k=8R|J@4`mkX2b{gs?G@B)m_L+|nLD{=LPeWvj& zeB0Xl+{?gdAXGcrpL|SE17WbT;ZmC3CBz?%xKRgEvsj|gn% zRywhEVXuYgQMxG#7dRny@V?Aj9Ai3>*!l`DR4J2oq@xZI_#I|&HvA>T9L8`i6sISF zZ`<4oIh1M|$?eKYfzY8O6;d*(f+FqJg4%#!3q|7mM*-qQwbuY!zuy#DLNGydk!}B& z1=BFg#(zqdthq1*hw{9xH4Oju7bPaFE#5`e1z(S$9?m2{ zBp?~C==(C{!dq-zjSRiH{#8#JlUZ4sx$95qN@vjpsd73c1Lj4@63fM#>KVA`LFKHw zXK_$Cn`q=b@tamXe3lu@?p#A8AK>DB(^2H1G)cb8ejqHQxqj4>0muC(V}GilQ2%@s z^@-tM3WYPx0YwBTpko%;`zpx82_W}%~%z1$WNJ#>nrPy z{<4*f-1JVcC3qlrW%=_!*?57enoZ?feCB9;ft^4DlYv_B@=iFY)F`oIe&6r>iM zIDX9#ng2Y+ZA@(jr*aC=GW1 zU|)0i!KA{7;(Wb$9kW{{ZWR-Mgmew<-6kG<{Q2(dSkqyJz{a-0!L9A)mp)Evm1y=l z=+oI}i=sRFc?n{RD&`c>#cI_jb4*ZX?<5&D_mD3L)|GRIBCRrmcJAM6T7aS7GREv_ zJ(J;n1JB*{Bg{`GTj2+wY2KUiRW|@#kZmZU3NVlLh<+fRn*2GR)@dgxlN}td3BU%LTk@IWNss++8 zB>H58{FYaisNmGP0Qds}4R5MtkRRhRQx%^h?2hr7l0^$S^x;BQX!T!qRD5GZDJbqF z3j8t82Id))dFwFu?G3^WE6)=~>s z3fh4Fx{|I9(?AiIRyg{z)>8Gir0&00rdeq#&IH-r$wPe;%EJjH25!)^4;s7>*$47| z17mvL9+0ru2%*b$MXh`eH8yQYl$rdGmleQBvBh7xMJ$v*kK9--??$X$)~M-Cjf_-O zsSNRlcWpZTElNKh2j{2vZ7P=b{+trKkF#WVXbOITuIravqkDw>)b!vnYOHQo1Uzv7z_CTh(AK1rr{+&HJKMWlK@ zLUxEmG1y=czyn)nKt3>(GS<|#*rTr-Y|UK!6*h#a$N~j2y8EwI&Z?KzN}lDRc|`M< z`;N;@h$isv{l!)E)ohV-cS>@OSb@w03XF{n^l>h*7=GY-q%2s15j_jYv<9e1^J{Y7 za1RYOtdlTGQPtYDIE%`!@RPTA>rw#;m`JefGP8!FW~V z&NeBq_u{d(9u)Lj^EdS5key(a=(DAkTyIgdR(WjB<@!@0Tadl>Dk)J79hIk?h)@=z z$<}3*=#IofGXk$2d)Da)n^dw#lS|Qr`MBza4C7mxZHrh<=@WL4>uF9y_k57np;g{0yv6tND!eTj9g&pif*(=& z=$}$HtpdJ@GE(-iit>MW)U!(=eQ#qf_xs~SA@kVaYyV-@(DZnJ=zUdik3_IlFVe-{VmwQP;6?oMSte?6d z*ZRCaa+7xuzz7YX(A>&SCSt|gqivU*z0W_u2zk23=nUVct?irDj0#;gvLNX*A-Plz zJ%E=bb3dQ0SA4%{Q=r&;`$XqO@b;s5of76eoarql7^Hn0fj(;WAe1d_jnJ@pObz$QO~6p^uJrWPz`oiw&Jwv+P*2U>5N8dYE5dn(jF& zehwdgb@g@lk=6ria<9_PNMJ}O*X>wD%kWd(aOP+=V^u|rW5=s0`BYyX>l+$R?$bO$ zay5l|+udpaB5QbqZl_AnqVX)}M2*llJRzrzJL#zF1FA_AA0fYt*v7Iw-4C4arM0sq zWQm^Wpl*tW(UyLb>95=2DYCYKtx!$yx!q9M_r$}&Mxna)H|swa{I`i!@e=KjzDJ*S zGTf&6#TX}Ys~hb2%bx|h38HlUtez`-&grplu#Wi${S|aIBqr2|T?WTAyTxYU*ZA)4 zd_Fd70W5Lt0N>ul1HelF!5R?dCJh`5U(pkxDhgEP+8D}iZx>9Z4hYsGB!+g5yu+|; zWYG?ga{qbe{+jwO&@~pPAWu(sp0e}a?}7V8hQCcbjXqJO!t*80Ck%e;87(AZPBgsm z-22)G_$?Pbc-3Ya-|TR+&(*cwxTPBgA&CqL)06a>y(r#TRj9GE2#E%GS`5eEYGf0o z#rFSmIc0g(*p`$+Rr?hok2L#XF~WCk3coa$I^t}Pi7Y%IpRC~lyZumPvP(dp$T@PQ zOTFW%R($$t|B0RsBnoy^xOpX)|Cwz zmf8`>Eq9rM|1@J(yKj)#6Zv_oaaY@GFl0VvFvR#;$#uJ;a8hfm^CgWrT&eq)cIZuv zj08ZJc~4iGr$b?yRKb|GcWS{-S6!2r>u$GFi{=p zOBM8FoFlPk*Y_J%RUDCl!1wv1(@7RKlK~G?x7)V&`_IznN6^ZQe%H(0R1{ISTer{G zd3#jc{`cPH$-)?y!UP@0?=X#$Gk?>2uMYK3kc|bYXX6h5xnw{ zLSf3BPGKL1G)L1d=S8y;7tmN|om`_sn;FK{(cHjPxYEYg!F8`?T!`pXB~;o1e^Uaw zk~7F~x%6uYK@La|Rus}IneS5mX=)5lR%nngmfve}jQMdA8>IiPhOpv1_cX#M+nS)w z&mpM5vfJ^8f6?G-EV+qFEroI;Co7kmuQT6|Jvkujzu(j^CsJCo1LD71CHEYxC4~04 z6|obKm^Ipu)Sl?g6xB}xF52`U8^^8rPfiSWxKI3KmFhr48ui2oK4EzVn2|I@FVC0f zV1FqZCra(&z#w_pP@vwMVEHt zIs8*?9@!=({QWd>B88bHJtL@`6)2Tfe?`o7QQw&`MHdE0J8Uj|YfqPT2ebr=c|1-q zeZPHV5$W^0>3=`0+~{|DfcD>ea(rHkr+^=?Ysfo29*_IAf_rMuv>tAoED{yos%^N~ zX7Td%kV9u`=`r&7K^W`=$4`dSGdPUGZX5`Ehc}IT_8oC;cg)#TXEcZjxy$1GKA zz=yqa2qd#GCI8y+!-DYhIeay-%6;om@fwEf!{DfqJ(yX$Y)F2JR58K({Ttu{(8B7 zt?c3d{s;MwU-r_3zoZReawwERyD7U{=_mwC&!$fH7ze%p|t*2DrieF z{)9vvShLlbUoYq?PNBl<^u$w3d;hP~y1Bq08PPy3i>JWUT_+UodziY!4C4PFS87wW z`x;81%8RV5FR$mHyt%rI9}RjjR_r|jSO3u}1VXMd%t>;bty0enu8lA~Ms$ta-U>7I ztgkioi9t-?nk815)2%|QjKM$S${C|3+iRj4K|@kL^jMN3yVhc?*k`H}?4=W^SYp!= zc0S{NRIZIeVLUCMy&5|6((C%hE-9sa{o zm7b(iH^0h&nmZ?9di?dU30^w<4zSJ0OXwQv3N}9;0yGq=20mxW%}eDR&nce2LPsL{ zVdp8d5|)#cfXv!~U|8;q{&o5Kzwc5yVbX{t>)v3N%Qu47NX;f>qBW-=m-CY>WyGhDd#6 zXT~UZC`+!_#K@p*NaO_idtLf0Iis$kYm_v?VAKNaf2D^_QCuZQzk9qTB?<(yjy~OyXeX8) zxxh{A9T(`!+UA4XlM+xGNCU9-k@yK3vxz{vp&h-tieAsVuk&$__~BZyckoz}|<{ zgJ)NmZCBNEs|uEgy7Z(6ErKO>3=P(8{*T!oB7(GU>8PuSSDnN*-8gU;t_s z7B9L{m3ai}(quvOE_jryr+Ge%p(!T`tm)3($lZQ-cP5{c&u47ix6tNJF>7}yequ}5 zRv!b=WeVTSDC{OrVQ67Tm8%WF$)mmahufXtWI$e0C()8?A~dE>=QYu&cn-2=vT~L>7pZWyD@UO2n^7hY{axkb;Ye5e>P3m? zoMM{eRHWRa&$3Kyo|L$edTfzW6RHmX;SL6Piqz^x*32|-T=;McDeAxo#} zbzH7&P>di;tV@<>myR5VW%nW`heWmwP`?tcTrVNWnbF4S61%G0a5*dZ`(tmb0?f8m z-IUG*?L>@n)`Oq8VN++C8~6j5cR}HzxqcvqnN05%*$~%y2C61qKk=)N5N2$Lz{dQT zuD1RvCeO`5pEft4rOOa3mzHK+OcUcDYY5sb&>kG`YH8S0l9=n2h}vv$we4(}BuAYY z^2U*ma!g`?(#ax9F(_w>ZK<#)p$8*idn`bp;nBHf&aH}7dDZHli>W?`e&~APxZ8Z| zjT$JabS(*~odD|33JE=CEK5(T=3~+wRJ=&ncc<*7P(@|u>S2sqK@eAd@UU?K#8n+T z$L|X>5q_7cpH*at7-91K^ao>OfI}2Nn`)zmQo)iEGNH7#TM6Yjz=d`ghNK&J*7{`B z=7HB(kCcI2l-_eN3u_YFwR}~;`a?B21QZ?$rykCtp29kc=!!Jwx0X*^*8X=cj^c{# zJDzxziy4VcV9vPuIu2*HwUn%5byKo5ATK?mD)GqN(Hb%ioC0DrOH-Ou<{B+}E75g6 z8Q<`<8l6`F#nsO@)u{Kye%nNd%4)1eFCWwXJzhp5S0klY z4ckUBosXYBUB{Rmmf9RItD^S+*nrK~JSfRUrF%}D4SON1Btla)HU(K-c#}XMFH9(SQWrh%y zeZ}#}HqphH^KlK{F>cFfVL*b}=e+d4#*-d$n*}zov^jCPb>q!EJ13XEoK=->WV}~Q zvKuqp4tK6mH)k?o?YeM2)ak;d0J z5;N6pbaD`vkj^qGEz@sXVe8=&&8XOFNFdvZpVw7;gOS>Y>?QS~{^c7tLzeOS*AF$V z4~z%-iJ6a2qi2uINlO(HkN78Ic}D!)k>f}a%#(oH<~7n8vHVp=GS4VDnS+F?8tF3i zAxi6c7u1VR_68{D%s}F5eqUjWOdr zRntm^NgYn*aCZ}H+R1mJ2rcRumPCx~_;Uqfzu)?5W>cN;S$93pfAr6WjNpYjdd=AR9kKaE?un^AaZ z**KIbbs#e*!oAvMrfKq&YF(0%W_KxT&jn4`hcmbSF&~ts-075t&2YOoV;kpRvJ5^i zxmW4$d%F!yRlluNQ@T=gD!rHHG;=qW(9D9b6}hSYF8z1Xbn43eVAGEGuUe_Q;io73 z>-UhS)z%!-#a6%ZK+CD~X$W4)!d~NRJcQ^R^H!=c#Ag>*P=XHGQ|=m5mn%o90WXj9 zX^QXvV_&$>(t?W}I~rYHl?JQZdYZWh|Gl1lo

*^&LU`cCtr36krBSX~u|k;Yer zae1jtp2cx_ca|?4pLV8ma;2i`AT5y`K%$!;D4gH)`^SUW%i$I)W8Wx0{QObvPq?bz zl)op$dlUrZ5a70xgTnlu!C~E3=DwW}UHxcXCpcV*0;fCRc$n^X!cgr6+NX|IYWh7S zLHy}-&`w8&H9zOQk;SJ;rk>-Ez3M!XEW5+lc0@C8{&N6JUkT?ekTdYZCSyd8he@0c z;@cfYH@S-D{-@ir3EJ%P|33SFq5b)=wg(bMAIU=g|CJcVwl0cyMH2HAFa923*+dk@ zx6VyDL5Rs<9GmJ@K&0D~8*|ZmjxoE=c-Qak<~oK)v*EQ=-IO> z-qP_kUts(D27BbSfVu*xoigY_^BrDG*ppnkW4u#V4&oHLHipZCNR17{(oZTZxO}hQ zd^j+L{xz)i2)GMd%^Wy4JLC|n2ko&Iw+QW>4qJ7k5RJIrkMgcp@b$Iq*duR|kzJZA z(w_4RH4-)>(rg68_r*`im$kSD*nm(U)4tj!H6s&0(Kq11Fch^MFl@TmGxP+8tn3y~ zBdt1f;Jl-62drN=Nu>_y&H*IrIr%Lnj~=yO|Q5 zZ|uSeyhDa!L>j_JmH~f(b7gb)@b{}9ttOxyRtzOd#N|kY+qrymk%9Q{S+KLL}F4}Y@5@A@qQ!)Ju=B`uM{3F?WR7`Z4>{t!_3-Qm3p=UoQpwIiq+KvJ5$R)%(She~(q1LqCZ&e{R2rN$MwU`ei zaz`%m=BitjAo9%U#yJF^tWnt8_rG35(R3;E(%3k&Hjc@CY3846I7M*X$4eZA+r+Rl z+9rz8wE6@A+NneC)d+lRzrI?SxFMe}>H%;IKQMHN)=h#ag3;l&^BFEM*v?&tf;o;}h9{+}d;cS34g_7f(O(ZAva{x_`s z(2OU-#29*xC!D4D>$?qC)mCGVZE?3i zdKxUZFYFYHdH}6W?#nQ#3}B~jf_@$$(cXAL-)daG`1OHQ6TbJNz`Q;<9aF9cI~LG+ z$V2fZLBtROCcMnRMYBJMy#`}5sRdPLx>h0<=_>$Ku!W6_+T(%?lOS+s5Hdjx;39td zQGY{X90sdxU-oL_za-rufGvV9uR%GAlfc#lC#?q!PEjC8_~4{iwD2$F_AK)GtW-6> z?^QdbJXBa_VzHk+fCD)5+&vrPePvzCc*|E;@i9XXZK(PpOO4AHAF`aLcH(lS_Nwgl zE-`I3WA5`rF`^G_ccp;NcSruC-*<2=g}(xl%PU%aFWTazvQD4iH;@8x-!o`uCuH)& zNK&tGvKq?r2an;U1z?SVd&J?G;R0tAXkLp`&l?E;L2M#|l_dKKVu!$TY*b5m1gr}N zF!jf~6=)c4_NI;hdIL^-7FQK#;2+R9#Qo{6Ful$>E#y1)OmxD$eQ(N^;vmsMNNYMX z_W!pgzqs1psJnNUxYqWecLk@|3)c6|y;Ge5oVtO6dT=c#M7g3nZYMMey$ch6cRiN2 z_?msN83NOg9N_EnlchltSi8o*nB3qbokboMP!wrTbZ|j_%2d5}ImWLOg1Himhm|?= z6vr7l9LQi+%>4uG{I2Lp6ey^z_4y`|Rqp^Y2lyfXgh_97=9yWr{5A$sSf&+kFm3T% zt2F?n$-q|@wCM~4Aq))D`42#C~bK_k#v~8Vnca#85fgovz{N z)j%jOaP6w#SN!4a#-ZVCeL?9}do}m%cSW8sNhyqJ0ZcwFGXs!EdnJ#-kxF@jM$8u^ z{s0&jK-4yf2qlYSJw>=z?$+Z!A;u$why=<&e~@^fIxI_8h1Q3WVY5B!DK0MhDDJE{ z>dpoSD0g|W?kr3=bM-Z~x*QY0Frgp|4bACx(4%Y~e0{DTEmYdS`RZK(H1dUOpFiJ$ zaDV7DV^)^Enr?-%7J%-O+2S}W76dQIk^GUpk79tV2{-M5 z@6vo2BRuk&kV1L)jiWl3`UMUdFzM=-xhT5&)6su@HKYsqK{GHVt*m7_x#Hi0KGFi( z(DH`SEYpPkrTTP7P=u?@(mU`fU)QfzE236u_rluIoSl3tk?+AQbJG#GWafEjsn?&b+^f>)0Yol`yadd>htH+yGLEM~8T zxv`=-2)KZbCcZsSU7{C^T)!efVhdAan;rJe$MIlnx(6 z&3IGX;{E*4KH+oyzUk+$lyZ$pqk(%Squ?h`2gf)32tL0V0})Hk#`%1Pa=??MPAI>= zxv@GU9eWP1VV~b;>It|XPh&i#KNnbad5od7OXhudX-T~|`{-5cHDBbvg=VY;OCGzk z7;upsawCggk#FYw;gqo#@L{5Wl@2fkK42uhZHM4RF&|{E`|Z9muJrkYJ2V-WtPLI6 ztZGYR|AI-k&zM!-0e@}r{STv5^A+=b3r}eWB=71KrTqXK^gcGf0RMqOXkl3u zO!6$G)f+5L{!dA^!f?AFc{QeI*9&^YI(c$49$zZPACq6E~Vj4dZ_f7Vz?jj1Fyx8-ZPtFEQxglcqHFU|}2wD{&cjIp4DjwcTB>4UOPEtk}( z?&P>N&hDZNIwB}?r6YAkoe{VAl45hD!q&V{xlmdES5$=atV;r>pB+#WiH9m9k(gtg z+gMbFveYZ~!_^fYeZo~QhBZWv@t;44cOP;gs$$mw-8y3=t0HUEN?CMyj+g}`ot6tT7*>Dfxaq(Capla0PLm; zZ-R4=&x=J+14V^Q@Jqi=Pcn8sNm>Ae>W4F*X@40wDIrSz@82yXq;MwWbRR*v!QtzI z%!A;B>h7bQgQB@~*G~1|e23dVkdfUr_w!d#kW*N@XI{+hcOg;>^#~=Im@gZj!uv$- z3>klQNSWwzS_tJLY7)mhhYd^ab_85F)D$J=a_izcrObxXe7-WV0T7O#{grRv7LDis zU0718-5MTOq__*A;__bf9*CcX)eUvaM0h_FJO2ZXm)7yb+<6bDd&A^0*VIC)+;?Nl zY&U)GdD0Bz9S8yeSWZj0KpjiMH>Ec3}Y&ZD5!{NFa>B$U5xMo3|^Gv*0sn z)(^Ta^nCkq&1AXsna5k4D%6=NoEmm7&1R0PNlff@bY%ZR#~`2+Llo^l>mgI&>OVTA z0ub9>Y0GeRO<^&+sgL$4()o}%y%67)0_IG_MpSs#&H|@-Q zZOBZSf=!l@;X~K3`L;Y=8-DIZ%}k2Yb91~NY??#v1*w~3S2VH#gt3hMsL=R30@5;?Y8mCw*^8(4K%r8~yF z3=UW3ysP}xXPB~{^Ig(+?%k9=cF5!!nWrJS9n6<6Q>aI*aLNfmVkhR$f2tHrJNrFd z`;h}8D27!ETUw*wfGp`965}c#c8qN-t*cy9iTy^UlIO^b$(bDs5HLis8oLU9^*XO0 z;F{X@D_wscfF+Y*l22;9pIdzUqv zlBnNueZVy!BFAs8GF3UfVy$p^m&Y%RfDykBy~PCEz$(!v+-V#ufkGX_aZS z4}({f$c0p|H?TiJ(>Mc~_M2)y-8SG9sX>YBGA+F+_|GC~eZ(L4xjt%=RbBJx+^-Sj z+OMy-ky;?XpijW~u3Hg;0G)Fm2Fx8Btho{Jzb_P81e2SGTHFg0t`W}To zf^9U4Y4`B;b!CB}UtCSO8x^}nVOPlmA6NeA$>kiHp8Cdn`=taH4&qH*zuEgK2_9jS z7(?HZls8s`${V3=`I9td%`6pMFE;ytkHIb%i$6#hT7bLY&z?$er;?`2KL~l`wzW7I z)-({^Mu$5pSr-^4$6S~JIm1Dlzw?(+&woQh(+r}T6?nfUu%MmUsc*>@tKiW99N@44z zWf6!If=GME-1@oBaEP_l0*|0~Jt6mClv}l~!bDj0SYN~6CO&NQ>2SW>&Bv%;L9y-I zCM&U&BtujYZW+;ucPP=ObB<^#5_dudSTlishGa(Mc7biCT7=?d(l#xHeHxuk(As+< zQy4^4xQb37d>j3FdCi2k*waXp-EQkpE3n;h{j~Ui(^R=w(0{d~GkYHx{9O>cuA=n} zDwZ~6hXTVdGkr1w;Q8m*aIal&7ZxMhU9HkL+V&b~>%gX|*Oe?y)^3q#wE6aqKhf^l za5Q&y1<)DD{Rw%Rm!ler@y#i-2Prn`VrpH^4?U}7ek!MV>OAa2{S&a4YM3?~$zmj>6`Dopfx%za;-WZXM zv1TFEP(2%GH&NX0X3L0yuCD!q+)5vL>I`MxCOUJh0K~t_4=J@K@lFhUTG|39 zv<`4omEWmZ{xDUOP?)&GJVK_s+|w94Xd--As3s9V?U_ykQ8++mK$s_RQj?tR9OC^x zdEr5Jj}49=s8FV+ zl{yZjw~0LT(-2xT0B>E*76%vmucnwYK8~f4cUwLb?1CP1L2PfpTGB9J^r{*|VmlU= zCO+$)C!YVSf$P4>WT;NvCs$aOEP#8MiM$qWsW;Aj~*ILo4S+YRvCVs~Q z-x^V0v>u&)w+deR$0+&V58Fp51ul8ZNypGpwu?vY}rKgbJZG{J+D4)n?;xtz=OX_{v*-hRPq(!6}rCnfL>5`ILX&Qs7Q4vL@ z!dp~EIm%MIx0l;10+3~}hQ5d+c!i6Eq0L3Er8*YWWu2wTdn~7YN&g#e9z3ou3gw31 z$lEZ2v+17XBXHO0(D2~G0ptgFtL7Mju`v@CIu8pH$|h!|BsW6w2|8{J!#J1lK0=p?`LN1dHL+Bo z4BbgrGxq|NITeHYdcjLpASo_oYMWiGE&~!FbjkzaX{>b6%T#m`k>(&&IUtO(A-gCNehyU>eFC0d<#W6y6L=Cr% zDGDf+Eh4S?9{~;u@%25^ z)(tL=1c{`$&!VIgNZR#1?yEg1;KpAml6-Q_@6bkK8+(P6Xef6EbdL_bA&@SJs$gq3 zcr<{42fmIx(P_vcrGnAAyR?a1%8RlsHB zs}Rh%Kn*(82BcKMd9a^=k_=X7I54(jpHQ%=0`BVuj9B1zL@KDUI{-?ttrhTi5b7W` z!Jw0n5C@pqlXDdlm{u`6#rcla?~U_>bd!#Z1T9OnU_{;Ox!yVpMF`^!(pZ^9cS$6| z0RsYcPe9_9S~|UU9^Y-MkmMUvl2VPdNZ|?e<+Z?s+61*!+7e7A4LuzTrNB<=Bqh-9 zJmJaxXm`G_Syr8%J3rX5Oxien?l(1XshMiWDG*2j0}Xsz?M@|8Bx4^Ka;H-WX%)9$ z?I(Wj0|Qy5ILO_4?zq~N-SK9!5Ntuv1tRUDwdZf^ZAd~9i6l-^9d@!i?z;+vq1lt) z&WM#DV#@3`S;*%bCkwG=VkB_PwgRq&=Yj$hc-scgnHF-84iY8@Q74czDEL55`>~RI z-;=AHtIF&aNX*|jqYJoQL%H*SBek7E8XXh9$HcbUon&bcGaqz3FI)j7zy!4)?<#zL zWw=l8HdTnh|28#}RKwi>=qx&vJIR2M)F-4yrY26g84QH%1d#|_UkWDoVYAEbsQNvJxhNMQ}dK&w5SljYv()COsG8%g3)Z7z;jGXW*&$ko>d z=#7EywXGqmV#x>IK*_-hUI$tUCfn^tYfwV3%!y=Tw?NVX7n{dq8_5k4RqZyAd?Y0! zx78U^76k)OcoBfR3rL2$Qu}gGq~uGvGF6|6z~6NS$`)DR%OZ&zg)JmnU<$PB3(uEk z>zun-Dlsr>FzA#?$Wb}p)7d0H5Wz|pxY%{%eB-=GV0b};lz4}gWIXN$Qu3{RLZBCh z3=BpdcAk3C0hqE3C}}_L1_eUX{4t6H-}v~LXeC85(UGQafVQ{?bkDZVK#Af(=xZ-w zT}RH1r(CclxwjvYq>H4w?lXS7CM3o-rzF!*9i;NO*H#@UeF$@7w`F&UZ+IH3)bk_V zDV2O=cSh}p-+^th+D!80j!!jzYUCVP8gGaDfKRzfYaMLO?PRFMK*oXRb$P&zz(UBr zJz2a55Gl4&yYso;ZKl52INZk@9w7t5z3p5z(s!BftF1Ng)PyP#N+&AlWOs-Ipql!7 zA|uU%KdR`H)O_=En7VN$$?MV@&@0XYFm$(d+Q}%1OlUwPbvf+-Gw?i(@9)RmU<|uO zwc9Zr>EM&YeOMW-oyui=fE>0@**cJvM$AvOH;6magp4n-h=-^46O#pf$Xj{rocX(r zCF#wpYy_=5eG_zvqySV2>2b65!6XrRsf>Y6lTLPrSRldb-;YQNbg=qhlBaeIrC?Ji z8P*KS3J(JYO(+GP?{Y1TzpZW;4kGL<<%dxyyPyQ9!KIX9KOTo5&@0oqY|`@%8J?1x zhGjKt*s|KlpshIsOt=8Q?S0@f(y zw{uVerd%EvA-2^Vw3miz09=KqU1!b-2e1R6*i*QVtxOaOtiN)XUZG7x=UO*_#JKi! z(2mLm2)lq=1#+tt(% z*7GpEs;!MH#y!|au? zdN^;Mt`(mefATi=n#*S1IHh{EVWh$CG5*#EH`BliBToXHOouW^IN%FUdj7qu!tbfw zp1)ulr;TIPz-2<>o@{A?u!qK#Sj%Z!_U9|m0|!_J)W{Vu;=YY*-zyOLO)XGX@$pkl|!OFta%JX+Bj^xG#Ia=l2ji-4R87Vgi zq}sY@%D{x4A0=sE!ft6qSw(73C8C1WMGI{X?CwA$l-QWnwa(T0Qg&zzjFS3|*fyvk zrQiuia`bc;j2MW(fNO6*r=Nq>XPAO!5rIz>N5bDYJZPGw!(kJZ*9zj=#`3<=X6u5- zQ7kwpZc4FEUdONqtl5_i)(lFXvT}U8cJN5{5ovppg-P$M}?ir;zbO$hS8E8RzTZYH=AU3{bu z=ofoR09#o$1y;UWLrIw~hr{hqYW6CV0t=_KO(pH2Sdv?tN&!}UNRx07@lNFdl@kTJ zx_{qjiQ7siyMq#j9oVK|@|tJPh~))sFk!GA#@|}UV=DH97lQ`U>AS<))Y^z074+)v z^4f0Qyxi%vXu;d469^`v*S#Ua(;nNf+iPo!nmsCwD>WD^=AXC^w>5U}FSVm?^x#^t r4ccF>S-!?LBx(TYAY`C= 5.5.3, which includes PreTrainedAudioTokenizerBase imported by audio_tokenizer_v2.py:27; AudioResampler is already used by many in-repo models. +- [validated] seq_token_counts real-run producers exist: gpu_generation_model_runner.py:316 and npu_generation_model_runner.py:299 both set model_kwargs['seq_token_counts'] before forward. +- [sweep] Indexed accesses added in audio_tokenizer_v2.py (cache[0], in_projs[0], projected[0..2], wav_list[0], codes_list[0], nqs[0]) were reviewed in the diff; they are guarded by non-empty checks or fixed tensor shapes. +- [sweep] New branches in seed_tts_eval.py save-audio path are env-gated, OSError is caught and counted, and audio_path is propagated to error and success rows. +- [sweep] Resource lifecycle: on_requests_finished release path exists (modeling_moss_tts_codec.py:564) and payload finished=True releases slots, but the GPU hook is gated on num_scheduled_tokens and the NPU runner never calls it -> finding. +- [sweep] Platform parity: GPU dummy run gained model_kwargs['seq_token_counts'] (gpu_generation_model_runner.py:793-799) but NPUGenerationModelRunner._dummy_run did not -> finding. + +`vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_codec.py:728` [major] — `_build_codec` now unconditionally tries `MossAudioTokenizerV2Config.from_pretrained` / `MossAudioTokenizerV2Model` and falls back only on exception. Both config classes declare `model_type = "moss-audio-tokenizer"`, and V2Config pops `model_type` and forwards legacy keys, so a legacy v1 `config.json` (e.g. the default `OpenMOSS-Team/MOSS-Audio-Tokenizer`) constructs the v2 module layout instead of triggering the fallback. `load_weights` then tries legacy v1 names (`*.attn.in_proj.*`, `*.ff1.*`, `*.ff2.*`) against v2 params (`self_attn.in_projs.0`/`linear1`/`linear2`); the remap table translates v2 checkpoint names to legacy names, not legacy names to v2, so those weights are skipped and the new strict missing/skipped RuntimeError aborts loading for delay/realtime checkpoints still using the v1 codec. Select the codec class on an explicit version/layout marker rather than bare try/except, and add a legacy-v1 codec load test. (evidence: grep: both `model_type = "moss-audio-tokenizer"`; read audio_tokenizer.py v1 config (permissive kwargs) and configuration_moss_audio_tokenizer_v2.py __init__ (drops model_type, forwards kwargs); read the new strict missing/skipped RuntimeError in load_weights.) + +`vllm_omni/worker/gpu_generation_model_runner.py:81` [major] — Confirmed for GPUGenerationModelRunner: `model.on_requests_finished(...)` at gpu_generation_model_runner.py:81-82 only runs when `async_chunk and num_scheduled_tokens`; the zero-token early return path calls `_update_states` (gpu_model_runner.py:450) which does not invoke the hook, so a zero-token finish/abort can leak MossTTSCodecDecoder stream slots. Correction: do not mirror into `NPUGenerationModelRunner._update_request_states` — no such method exists; NPUGenerationModelRunner has no on_requests_finished hook at all, and NPUARModelRunner already calls the hook unconditionally before its zero-token return (npu_ar_model_runner.py:419-420). (evidence: read gpu_generation_model_runner.py execute_model: hook at lines 79-82 is inside `if self.model_config.async_chunk and num_scheduled_tokens` and is followed by the zero-token early return; read npu_generation_model_runner.py `_update_request_states` (no model hook).) + +`vllm_omni/platforms/npu/worker/npu_generation_model_runner.py:?` [major] — MossTTSCodecDecoder.forward now raises RuntimeError('MossTTS codec requires seq_token_counts...') when seq_token_counts is absent, and the PR patches GPUGenerationModelRunner._dummy_run (gpu_generation_model_runner.py:796-799) but not NPUGenerationModelRunner._dummy_run (npu_generation_model_runner.py:780-782); only the NPU real-run path injects the key (line 299). Any NPU dummy/profile/CUDA-graph warmup reaching MossTTSCodecDecoder.forward with non-empty input_ids will hit the same RuntimeError. Add the same seq_token_counts injection on the NPU dummy path and verify the NPU CI lane. (evidence: read gpu_generation_model_runner.py:793-799 (dummy seq_token_counts added) vs npu_generation_model_runner.py `_dummy_run` (no such line); grep shows seq_token_counts only at line 299 of the NPU file (real run).) + +`vllm_omni/worker/gpu_model_runner.py:1874` [minor] — Confirmed: `_talker_mtp_forward` adds the `talker_mtp_accepts_req_infos` branch at vllm_omni/worker/gpu_model_runner.py:1874 wiring `req_ids`/`req_infos` into the new `MossTTSLocalTalkerForGeneration.talker_mtp`, but the PR's only worker-test changes use `_make_runner`'s `SimpleNamespace(talker_mtp_output_key=...)` model, so the flag is False and neither the branch nor `talker_mtp` is exercised. `grep MossTTSLocal tests/` returns no local MOSS-TTS talker_mtp coverage. Add a CPU unit test that calls `MossTTSLocalTalkerForGeneration.talker_mtp` with a `req_infos` dict and asserts the emit=True -> emit=False stop transition plus the `audio_codes.current` / `audio_codes.accumulated` writes. (evidence: grep tests/ for talker_mtp/make_omni_output/MossTTSLocal (no moss-tts local match); read tests/worker/test_omni_gpu_model_runner.py `_make_runner` (model is a SimpleNamespace without talker_mtp_accepts_req_infos).) + +`vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_talker.py:1673` [minor] — Confirmed at line 1673: MossTTSLocalTalkerForGeneration.make_omni_output emits codes.ref = per_req_ref_codes alongside codes.audio, but every stage-0→1 MOSS processor (talker2codec via _extract_audio_codes, talker2codec_delay_async_chunk, and talker2codec_raw_async_chunk) reads only codes_dict['audio']; per_req_ref_codes has no consumer on this path. Drop the ref key/per_req_ref_codes or wire a consumer so the stage-0 payload matches what the codec consumes. (evidence: read stage_input_processors/moss_tts.py: `_extract_audio_codes` uses codes_dict.get('audio') only and talker2codec_raw_async_chunk uses codes_dict.get('audio') only; grep 'ref' in that file returns no matches.) + +`vllm_omni/model_executor/models/moss_tts/audio_tokenizer_v2.py:14` [nit] — At the top of the newly vendored `audio_tokenizer_v2.py`, after the Apache-2.0 license header (line 14 starts `"""PyTorch MossAudioTokenizer model."""`), add a `# Vendored from /` provenance block matching `audio_tokenizer.py`, whose `# Vendored from OpenMOSS-Team/MOSS-Audio-Tokenizer ...` block begins at line 9. This anchors the exact upstream source revision for future diffs and rebases. (evidence: read the top of audio_tokenizer_v2.py (license + docstring, no provenance) and audio_tokenizer.py:9 (has the Vendored-from header).) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v11_deep_r2/pr4804.trace.json.gz b/eval/dataset/arms/copilot_v11_deep_r2/pr4804.trace.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..d2ed451a8f9c12d5ab798ac71da3062cfeb509b8 GIT binary patch literal 58245 zcmV(!K;^$5iwFP!00002|LnbscH2g>DEccHWzRKflN3o&&kVET_%ZRC#Ll*qWbGws zfgk}&7?A)CfRYt&?{z-ne&PL+TaRv_yU_q4$#!O+u_t2_KtHOhtE=nP)gQK|i+GxD z9XTI9cATv+@`LyIu$8PMe0%y~E4U7#%)Nq7d9h0`b%Q7kmeWN5@3Leaz+0#S5BN`a z*dOlo4+cBk!Em%U-0SwIgTb`F1%=X;7rBwQ45;<3#bQZsvtYG#4nHz4y`uiZG+krc z(R#6f_t-r)zDnFAxDJCGthJ1NEV32MVtibAiMPyQT*YYyzvtc}Mc`HmJd6(dBLp;9 z#aQnedhf=|DBKb+lgy2y#jULN@#FbNrSDn13Kwza&K6W*8Dt)T>Lp9{^Yvih`a!UA zlOYzHdb6u}xL5?f0!O^5Tt zU4MGmA5odhco`H$=KY<)Xf`?+`orn|XtcM5z$P>J)F+(J;=~Vp*UPBSK9b5ocd&CZ z*g5JS9*qu&IO26O3j{SzZ!`Ki8t#m6E|HY#scsuSxmn`Pg5EzcZhyzy9}EWv zqxoRC;|-_tU2n(hkNTs*t`5;)ba2=VAfV?YoTUVk4$)43esJi`b`FOJy8+PiXyDEF z_k#WXojKyFKM(c?`=e>#?G6ri_Gcr&>E2)-?0JW~ql2B@10AB_aBtW@Xa>_Ni7$Yn zZ9vVuL(l66{axSRpB?P?hy7q4065d(?r1di{n@V9KZIG_J(znt`*Yu$?N4V1d%J!B zc=vW@2AYQZI}PUzIFtn(SVf12)7{#a2)zD$|6t(njQoB7UbM>`{Lc5oOR_6<53?2L9B!Xx>x>sxcBk|HzPG!7I6Uy?CVe-^al-reEaKb z_w9>c-n>2OEq!wyXNP-(ndb)wGobms*O{hh%u@aOZL-O+r{Kir22 zJ{;`_{SgrRU^*LlUZ8`te>iB!6`sF(`uEpwj!#}aJ1)Vrw*#Z!gYI|xhdcAx3|P%@ zcYiuP7|iFR!y(My=y0|(8|@4SJ}{18XWv8OAI*1W^S$|OwDs|vwz@?~d<~X*7>R9e z$-5ZtM8pfz*=4ZAePlr2lR#`?$_DI*-bDnv87xk${Ec%AyQQPNJ&VqwZ@zKnK{mS- zzc0KjIE%W@TZ0CDo zH(IT`v+&Wm^Z)&y|JU(jCyFyC3A$c#u@0P@OD}Vty>h$-K!<1%TwoVb8C=Mc|8?&Md$^E_LFF+Wz^?@p1R$ z{u=6diR_*y^3xBJiU1s!b*1C=eaJ z{^eK4hhf}yVG^D7$_F&SDBi8Z1co!cCHNy4QX0H-scm-_ucIt=W^si3Le_Csi*?Gd zhfxfpZe`x$DoEUAwsLS!cc6R#^yGIO;Ap;sV@YB4&*AIsBKG_OC}5%>!g@9{)e2w(;FpB-9#6&`L?6H@P1Why*6FpkplbfPa~#mr;7Wte zm#amP1xF5W>ylyxz08hy^$wjLmm|gZ(w@1KbcWHotF`yVOcClK!C=)JRzu%H@Et^5idE>x_D!4sZnyElMcQ{QEis~aGP&Cj&>v-s+uj+h?MEtZepq-ZEEdlX zVG`rH1DrWXybRE?^3oJWas&%!0eD&`jxw(yVsQy=dPs<=1H3v|WiSm_H?ZnboSmYj z79$wJ#c}TM?01|3=iDkG0AzxWGY`{Cfj$>V_r~?;;~wrZ%Q#K3l0zd#Jj*v~Go!;% zyW^xW6K{OeT}R}5tG8|UM}OPm%Y!928Yfn1T?{liBh1sC31)F#?EeUw+F+z zxeRBQFvqjS8kUChOA?$Q9}fJdYd?%naM^*Fo&4YdjDUGq+q^hE{LIS|zq{8%9RG0t zTZbyX4N?#1DM2EY><`9Cb(dar3te`R9Wpv;fcnZQt$Ju$IOi`zgOjv;3PXoYZfQuN)_=3388_;OZomZ$I z!cdVi(-g4k`97~bE7pL;da%Av`>-8`kqg87XG#zF3)pSopZ0sbA-uTpuH6L;nZ6P) zD;}eFSgFkmkT(PF6Dbh|9a`Toaq#(K?V&<}a}mW+H(0KKj{^v^D_TDD@I9^mD1`^u z?^l5!Krz18z?7#zPRNv)@qluI5L;eb-_8n!?jD|2D&Gl;IcxYGeEesj!p4x|CZ ze{q=!9)-*rKH~!;iYhagTX|I3$zms<%2+C06Na56zR@8B@*{P~S%jdY+`o6;68~63On~h_?)}e>k$G@d>E+vbog3jW6vu(=~7L ze}}#9GLElsFgy-e;M8n`<$ZNeT190V}6SOB&&r~92FHXN~N z#F&3scp0v504;%i`G`8D4Hg@MZgEv_ChXRR)wF-O)7?KjY}42P%D?ADf5wt;xu8oO zV>O^XpZJ^tH?Ecf@ZW%w!CLaW^KgN?WQ3*^Xz!?k$qYu$rH8!Jc_gbk^CVt6e_O9u z?)c96-uWBs^4~pjE`jp|Ne2XhMR*mMEn;(rCS*AkodB&kXfwD-Lb=l~0^F)SpZ5Ak zCxVa5v)kXP9T4U(0yl(d%aFq(+X8_#|GufY2Ua(sS%oVcRZyzu z2dhPVn-^q(rFZKs7fsq%teHY0_st=4YJaDxn9uT zcmw-akUo;5aTdXa2ZHYPz+qjKHeMQ03vl4wZlh?Jg`i?x!`A14W=;Av!Uih#8<3P) z4Thazn|_gq+RcUAHcAzw|2Zz{_{;zS_z?6D)_GGJCV=22vr7PP2KrfmAifPqD6O^) zINlsIa+CoPppsY%T{qZ8AL&%Ij#ScusaS&uhuV-JtDhKScI>1mh|A$fI{?UjY?NrN z09ZjG&+y~(0Lvtxu*_h0g)y|HkP;>i^IWo_3DF`0H3r2TY<~tCUfNxN{IYN!Ax~kU zu$wdFN2DuZZ@Rki+wd7TjQct;ydYzxgeAVfm-db2x6UgR92YEN`0-7|#DSd`V*(r# zpdcKf39JjdX%~O+c7OfTPd_`q4Tx;hT%dD_=D8=bX>9vp3avyi^$vPsK>Ywg1r#Xw zi_~-}!GW-wi*BN4Dw3mPzv+t0;%V0<6Gj6R(&9jR9hCm-j*gU zoO39Eaiq>Mf1RV@znw}I$kQI?W33N@Q|NT(o3CfXRyQ1 zB&0n0sP(HDlpjAn29|X6`0-iv-_A2wfERIc>-;w=)$2v({5QPrcDwX{EOA0s!fgkX zR1kV*MBq?*DA#f;1zj-!36N?!#~`qwehNhf6-A!IP6G|1bcJG8P01gGv6@SR5+6>1 zW!5(SECzvNmPKecs48^^Q}CNKMG+#p&R46h-KuM#=BrV+xlz}|-T!q)U2}l{%WS#` z{r|-#U3Gl_nkHQp!haK!t~%t;Wzjv0IF2e0P5ZD_9UnoT3r{FJ+mTS>#7(1cwF=04 z?}$HHa(i?ZIm#bGv^$$$IO98LdhB*4a%o%It7wY^>sH=l8^@@>rIb?Z;>>x0#a%oFqgtLFt+ozGxf91dAMWv)dRZyfx{0Sr+vKv+qr|C%yT6ApRZp5KSl- zh>i&%k5Xp>gfe&4nrW{Obx>mWlEK8`Xw3q}6c2$BXc?)q$VxzD1w!{q)n?112xbPk=$3o;`$G)7OmR1a4GsYy zGZ06sRc|ukJh!kVF3`?uo%Kp~J4E@Vm%`#oTDZ)51Ts1vZDPaOR@>OOiaI@8h1Nz0 zs&^kCWmx;RRwqco6PH!3LVL_`>zzp91D5DA$Sz~j6)_Z>Bng=2b%Y#Yg6PSCcLfLZ z(%T3&#mjYc#gel{R5_X;aqyfAUj(zo3G%MefU_8sB-hr^!~yoTxb3pS%c2;D>J>v1 zLvmWItB~UK`ppTAgt}x;2ufJIdXSV;woXbrsq++mhG@4e8E6Sf0;wrGFc6cxkCXbH zgDmZ~xEHemz|vVR_@PyJ;^Kxx_V8n}xE+I#3d?vxVqw9bDd24t#|xYq>>v1J>j1H8 zajCVDw}xn0^pb0oFkumrQVp`rHE-?}L_%2N%H9J0CEVIex3K337F>^(%0&(Mj8$TnvMI7 zi%u^D6xam150fQi)yy>RAq(} zbKsK#gbhrx7#{DG!k?2&{zv~8l<`^_IZyOD#=WTP2|{v{$04~8U62<+mjd+ zP>Uf$_+Po0O^QTO|2mI-Lkz?#lwpW|2wRArrz#+c8jm3ZCkd3pH9`Y)uY%h#tVj?M&}Jtm7T#c_T@5xL$@!chA~{ry zRVa95L_5UhP>GNsFTU6$kolAAlANu|tWCBjOYbURHzq>}uT<1eau+NK-D%oI=O0H> zaMXmuA!mBv9L+83GIyHEP0@4donq&(-eg1U+(eK=Bh#0P#schZ6NlUCkcGOUVU9f6 zxh1N(S9#)6SNvikoHQg|nxt2Vhc!giKy|9v;zv!`-NF977@qudwndY6hW?bZEt*FB z4Qri7Kp&A0!hWT`he!VQu~W&x5R0gFn6ys^Uz&5}lDbvy9G=(^$~LYD3Gmf)MHYmkvV3%bz5-e6kQ! zd#{I|NzM7JV3%MY7n=HsZI*e{vkmsDJf>Km$eL#ufYzqqEcxnBevJmk>$`WT=;TajMix(WF zh;Prfu7Y3%5=ikerWHy#H;R`M4z5Guzg9`CT>g9_9Qg78E=;%*cuYp%EMXx&m&pt7 z8ap6cllys7C(gFry=FR3Cv6!Kp}4)I)3e?$X?Ro4D*P$%UM%#wu}F8w#976wbenx5 zN)Hp=sfdm{r~vAW02zdowCrJ2orDv`OE{r;+`{`CR5}YIXQKL1CcFa4Zs^1W`v6$qw4cb{#hu%|7)n@h1Sf1@D9fKP(Yx|Z|oM+&JZYYvG*1f`kI<-+){ zIc75s=KW`Wc%Z096OQUV0r;_F=SPD-fgb_ZNiLl{$nk~dGc=q1{f@qcvV%)FHeg+e zga(sl7JLpff)%Ry41|r&wBu`TDk0_@ek!Tny|G2jj_}9d>$lXNu(eSl6wuKwkW678XCT3 z7Zsgf$>Yr_9!halkvrzDKB;BMF6oQ_y|XVWrNDZ|d74Ll3Cj)@0_4Veiaj!&1WD-MK#o+h~QUvv629krCI;bo0Cf2=ZCn!5? zfbuBhl)|6U(CEP?vSMB#N+K3l@d9?tjWjs=l$!)9(M_qavE7{Z_cuVTupMZ;RH-ca zdsQ$G$WFc}kc8?YMPTyHKe z3>iuiH(X%35tc<553j1>AJ(K6>#PB_ISp32GI73F7~${DkKx=Ad~_0Vx1QFyVS z+@HIY2pM-PR@X2V@Nj2%h{RVerq?Eg|;r`BGr$3}2($P5=L@N#i$*c79Z*-`-XsSQh0iZCC^DJ~>F4ufc zsV|V?%6#2}-ogIfp?tgYk}UKVO1^0W^uy6Xe+RLGf_9ER#0K<%41)UZ_4$S9bhy*Q z@Z=bIh;`t+| z7#EloInNa;Wn;N3#TA}=qNh;iG>>NJG0TC>JYWa$>N1r1u-VHaexq&-i$pxq-bxRY zHP)L4*{uswL>7Y1LBuLh)hr9UmUT9~0MVR*3R1uUpHvndV;*(FH9f4+)F(kgW0xyW ziDQuw!A8klDU)@~dCf6pvTM!Y4w^jX#GtOJIr>G3Y_YjsA8Go6u`uWz=4sj)xqI|X zZ6S3X?)DC70h56ui(T?2L2r1t-{qy*hgODEhCs~cRpgoGiBU#W0P{f*z62JHcFdeb zEj0T>5jTnn-2BB6NZs9&S3kdSPTszF;j~bZa(4ZlV0UM*-}k(rO&NAc^W=0Pg}P1x z66ta1zQnP2Z%~XuM`r3ge}l3D?mXB9IWPz=#4Ofyv*0~B)TxSyMK5z#Oygj1 zr)lKkwrNyHFudOghW8@G;oS&v_$3HYJzq06KUQ*x@rRFX=tkqYg+(*ZWt48CPUo^W zJl!s(C79l$lH|d`w7&;by5HaNfCtaLox#w@s}Kwxe4h@O7vHBQE$TgF$?*3fENC$m zt8m3Z35x}1K_ol-sUKXQKY}9g-;3z_$eB!zU%Ytkg3ji?`0eRW?(-MFy?XXyG7+b5 zCzILQ_m27#(CXp$YCV~>zf~*!_|pCT$5+R{ym;&W{O0+KpC%JP1F#eigaZmum`pIz z+zYpFK*z>FRmwZmz23n-P^e3A=jEV*>H6Y=4|8RiC#m?_ff0NlupgCy6!C6>)x#2bd zWy48tBWM8`neVJY0Y}Rnrt5b&I-hY*MrhY%L>&jlT)eu0!2q1p6>6s-yvt9&z@vmByImow#FwYhEkh`tqxBM2 z4H~^x4AH?pLUh>g?DXaCv6?O2bg;KTr+I{q0~eE&(A^O&_;y4&^wN&=xZ`j>7#Ah* zBecl^dl-)T{eBxQfmlp35_Ygp*K-UwLo5fxWavL63&%xaof2hXa0Dr5Eh=|}W1!9i z^&FvjfX--#fH&NAmfm|8>PZL=_o#a)btHHtN_cprEK9ge)WLD`+CB7XtXgfd|8Vo7 z3W7tOoNe{~8HZ5|P1tCL>xHSe05ozXHf+QG*X8Zc<$v(lloOy0lwJ`vaZv=Nt@nZd z2tc5wi&abD2n1r$(<3q7bNI&p2?SmTuEtO}c(5g!P^M12c`SbjWz`RpBcdhezbR{< z{7meUuq)r>3jwx&o*?mo_Md>J2w%yG5Dm)t07MpDqHRn4s0^1=2O1#umRIn13n&DY zM6x?~*!|?jSM-?U7w7}MY8#}GAr2@z3R3~CT^D`OH21JGgs=fCOF@VX!utkr zr@bJ$4wE>dDA2Pl+t$JKtSlALPB_n-Iug8XFzgI=fV=I0D0(27i3mY12=fNT4mM&* zp(qUj861QLdZEh(>c056&X#fNq6--qu=yGoHHf4%UXV1nSr?{k3We1H{d)5A8y*&p zM=UM&QWT33mJd5167QxKJ>~7jf+e$2B#$wzu9n_ay9bE$A|JM*1f~m*E$g5VA5oS9 ze)jo0BgE`VLza*-kpqD=*+FVWKJ*7b$@VTO%ytAJ!w(K*so9phn75^w>IxdfV}<}t?}Ap4ls!CP)mJ=e=r0!4c(*!EVL@`)F! z_?i=*cuTB5h@@ql+lp`8g%@2A!~f$F7&goTm+#>TaF~}KMnGjL`ovmXCTH+QV@#PK zi_I6YmkG`XEh{rf!_L!T9Vr|#FXt#NayC~)WlyBXGLKJ7JweR;ld&@pi*mF}63702 zXSj2hHEI6&XnAiQvqsB$7_HS&ayqCdYAGcT*m(1CtY#fBvUqayTGN?hpSjBGNSZ7a-9cQhxUi{#+8p9%Q%7SpS#O z>HZF)bAPzkIk+!6>HK4m;DI@meThK58#hbVc1U@FD!TsH#V(V{;VGKOVS2^~u z47`X*$jd^yA+tBfcP#+`8cNc(0uj9^w2@AI<9=9bo?m{F_T@S`()2@h1t?V%VTvFs z9&0LS@{_Ely_5icEUX%azNb&2V%#4m%s3*t+QzU5b72R9pc6Kpb3C&Uy9iK!f^l^tVy}qXYj8`XA4>dK?@NAB3tWM9d?y?}f{h>4dvzxY;Q9bR6L@#-&flLtjO8W*?uFu6adrq z&B)>Q+REZY#vWXApqCxmlEF(DeQV0gVQIFL%}~GkS$Z|_WY}eRfzgov_l`#XJMts*-!%?9y_EFOdHcJ2v%~#)|8UUj`Fp|eU}tbRgT1eT zht6!V#wBMKMgBKn*%1FJJR^d_Q0WvHCjOSKSBn4-L1D7@?EFaQs){AILm`t9Y;n5p zL(^h4_R1FF!h+8)s_gitJ4U{jz2+}-|4b^^0eiDvevYmgb6qm3f0ebIS1Hx%bzRei}GzTT36`geFBaE%Qi;m;t z!SFGBhRTCIg(v3Cf9DKTNY*JDjiE_wrALP+x3}}sZRfEvbx#z76oyjImZwKt|GWo# zC}!7hbqB>{w|gX4dGEt?JWvLJ1`Dp=^HK!3h2uf1;m)uPd&kaje{a9nFMvl(D0AF) z{v!NsoWGu1B;eJdky{W6zeka9%h>l89Gjhc3ANiD*rA|E6XMc`u~DwYNwM8D5grO1 zGh%dk{hs{Q5+6Ij@xlwNZjP&rBS6e-5w41=-2nz(!e544M3#UFBlNWu)G{sfvY1y@ z;W?pEQ2(hLFL0|tFFl?C=_#Lh&AQKDynOoWPbcp4r>}ncFZa#QuV1-uU;OfhY{!ex zOYr~O&w*RtcJaIm5;ifE;eg!w1He^hu!}5(0^5nPaEQAUXxK3>m<``tV$+qx>1%mt zf>f~(bn6%`W5shXOy$>joo;G}&aNiw$fd)-95byqXDeZ8HDpJ!kgrrxI44%|?MMlP zvA~6_P?&i01r87uel*ccyF&f7n4p{Y(v&q<{V7A)*-%4u{Ha^VhH8@6jOCBI3RS2Q zrsWsXUPi{{_jTAajmWyi7&nMWY$+nFf`u=(6GPkWd(Vt*{>uYeAj{|PG->DruP}Jb zGUiq*OyJYwgM3o>c?RSU#NNg0d~#o3MLQ+CdWDQsebuTar&Je{Qqq1f+8@n4Z*M2) z^#-He-Pvr`KO7A!rKGwhO);siEU~pw=Kz@6s52lbNhCcW**Ge!MEs7N)9E4zR&Gk6 z$^G4mLKxw2VtO>|mL$cJ&C>b}6AAbPKFE|MLw#@dwy14*JeA z-R8u5W7t}TPU=wIV#~)wG@-X}tqpU7++L|UdJ>s?c<-g!dCLz6mC~i$<`8SQuPlOj zMwKi^j@sTpOd3ukrSu5irAN8XN`HGGI-sEDU>PU3?m7!4k3q^BNWu#6)cq`Q z1L!B=^9cDD6xc9F(mZ_c;<*yKoQ%#&RIXXP4&;_I-)xKUwbK1;+VW{qA=h?hJNvMR zz2U&`_4W^@)4n$y&gRp~b=<6Rb453+#gp6LMN#Dm{}|~q3!)B#aOB-;MJ8RFuwn*? zi%YJ0II|-E2`l(Q2%-;Ba3empohQzq3SB^G*o_ZELE*gji~~F5v=e>%T}muG**RoLfKfupanOG17iDX zJ}(q>^AP2;E$hhVl^qu`%Szx^w^9%eYlnbaBJu*tYz{=fGqy9RD-se>#=C z`#xDRG{$+K2F@ZpgGwnduzb$SndA41KfT_3XX*_OW;=WP`^@S=_PXB2vsYyvem_u< zr}sO35Z5|`!(9$8fAWp<3@A3iQ+W;^h09ZQ=clrNIYy96|P07$8Y@rGbOJg z{Qm0X$2Y&8IKMxA`}XPUlUFZ}oi}ftXK!9Ve|7Te&1-mi={$Y?U(P>Xy?zdw7$1`l z-mmb0A7$l1w2CQ3>z)RNOltyL{|qnv#XLV3YdnyCfuZ}9eNm>-6FIA?okh3|xi3YX z{FN-D2jtJZG}GY8aHPg#FCx`DipuYM^K~>6X=0$r%e)umCfHt{VHL&V9J0D=V6Dw) zwJB-K8`8r&puuFZt0b~VotF+PdoODFu__bT~(ucH~>&=(7~UPP26jKORvQ`Ka{tBiS_U$81UzlhmP|sz&~@Ia+D#zw0OkaA_wId_ZB^z;RntJ zjH#h;;)~qH;{0{boaut3C{#kc<)d^G+5A-0AzP@s;zUE3r=Bi0(rA#48KAAZOD{$5 zEVl%+l=3Y|>cZpD1uij-<3&gGKc2)#poBqO_1j=4!n}BZ$RStDr%D3xD zoz4+T-M1o!fU=lE3*v4*^l}S*D?)aL$&U1M>AiA93)U||_dcf%oSvDu(HVbH>}vK*fR;*H%g=Hq6B$H)OH7HJ zjkehUu0fn_6hEORUk^=)Wg3v}wXD;CUNM1aKHSa^AA29L_D4RwDhg0EJ-)WIV~|nN ztN^R62ZG&oa_oi2&Z+6rFyKJSVNS0F0hdu`U+yNVqn1+5+%f1-Z)wRHQUGNapgRX< zu;Jw!NqJcst+}FaV`XkyY1UO%;vm*zE9GT?Or#fDltlPGsSqk5CBCAKIZvhPkXFy|) z%{oa|TBae}Mdm`%#du4cvk)ebWiayX1(+JP)o!;75!l%G^oBJ(+b*}(FFXP^UvEd+ zSsnb}GhKe8M8`;Ux}YEE;mA6b!cg-LtfGs~AJh>QjkK*z_h#a%td?v`oyB$;dqq`~ zbOUvEn+p0i;$}=72b`K1b5~yne0>ZlCUJy!z9hFGGF=BHb*UsFY5*2sxf|Ar4KN$) zF$mh~*xt+(sZdXcF#pxdV&+UzuWH|AS*mTet0Z$IQgpf`ttx&|5Hpo)sH$q(z~|8M zUu%JzKY6#X`ah#;UuFAy`|~kEQmNX!jbZYC4m{vK@78!x7(uE-U$5I3^Q{KX`r%SL zNI@%nmCD>6V`mU_ceM(r)8&=|ikH!pi%yBzj@8%gyQx-~-gJxrnIC*{^B(^u2oI_^c{`6AbS7;;ws{(Dy_@Lnk!eKYl!)&o%0YuQpc%-~?5irfC zl;DP3LMQS1Ize`hmvQwluw!uVMNm3DROC{@fLmoLj4ah$7*&?5>_j{DYam}n4_O&% z=NO%(JeoyQku0ckIpi*ac87-i<<^6wc@32YxvM4z9?E0qy~?gY1Ke2W^r%B?>0DGU z(#la9_Q1s`eiDQ6U*>VN$L9>vdHk4f3Zv~kr&D8(qsELQ0j6+~pBWQ9alqjMDV)I2 z?FwQj&>^h>+A0934syjYapwXu>62V&&4VWS;cP2erlR$amgcLsrHAx|=ah{8xOQ(9 zV=h2ufm=K^?fXJ?pqrFwx!M5L$%J*u-_S0ZYm&c4S5bTu(Rdstf)DxFKC&+P`Q(we zF$m+g&M&dJP3FaN8u*+d^Ro#hbT*5mKePl&CizZZvXVtz*H@AggXm6li~R z-pN<*1NC0@!9JYo;L;z$dclraK-6R2X zP{n)9@&i#t#cN10(_)%WkGcbCReVRUsp?}ah4jc6&(Vom|5bGn*$`}R??PP$bNYq+w(fkuX#2 z^t%M!Q=45J0sRj15}{HegvpmB5;+EH<4S4c%4p+?+PG5MxKi4XGbAMWcbX*sE-nT^ zHkVTX83kmPlsX7s@p#U{csHk%t5WqVz4}#Ub^Vy|yI7xps%pI2teO&!&G(@i6;r(&{4otVo z?_feZqOXoQ>`u9p&nvt9-OGw#i9f@N@)S*dByTe)m9#1#efjdWLn61h1u(DPgi;Ho zz{kWTLEsBw5^syvP!64Yc#88@Tr}LAlwsE~nnwv?1dIs#H#8oOKSa%aIN%a(Wcddh z6eXn^*HXKq7IrKKL%g@tO8{opDu1F!mdUXhZnWb}LoekV#1=GEw29e!)kbAo)cBnm zEG7X{o8=GpHr+_0N!ISM`E zd4SI~fB^<|$OP}u)m}POtaO&Uf7(h8b9{*^9VKMRp0;1s=i6jgB?VQ)WZY&P&QNi~tqKMtb9M(P7a`u1As*Nu-6EBg&~C;4ejA zz4R7y7yBDiuc)9DaO59Bp&dL75Nzc;*f=Wz$tM+a70KU)lr6(k8B|S`lu+gxWl`N` zC*>_*AQZlKiAdax<6LtRK=bVBxhfhr>`9L7664~OQGwpvD);voRBaj_e9aNx1%1@K zlR+qOpxm7nx*T=V?kK7ozcICw&%`4B@tgCE4ql>L@oAQEG~;I%LI3d^_4<}lO4fNg z0X_=h5!okGzFrz6KRg(ga&9$e9j!py)2vsbNtg3gxF}h)@CxSJl(#=lC(6I``PFSR zWyuw*8?4LWhRnAWoUR;Ri%p`ojDQ)RM8?yx=)(MLU3Vq$Dblhw*~H$h^i(oC zVv{BDQpID1rHks>j9qvownbRfpsB*BxYvU%UTxEy@D}DnG@~z%7xQ^I!vpV15P&F1 z;ktlRJW)*B)w5rpKXv||c&p2Md-u#>H{}x^4-p!;@dc$JOg)QVOCM)=WIDuLFUAK< zt`y9apotA`K|SI=pFMH2&Dj>-Wrnv6*8AUOwJAvqgFU-Pk=$$oaC8`opXDMr4YEnR zxVn@AtTogR(&Sj|8?ccW3i2?pGx{&3E3jMmG;i!Yb;NaEv_?&e3y+Y z1~SSqU*jh0%BFnRcQk0!{--;^ki*gB2m^tro?*yq=Z6@IH#(T|UR}8(l{lpNqAvcc zA=fOiOdy66@|Bc4$90aH6X!cr$5D(t8Gks@Jt6t!D?tq(Hc(@*^tZ++>tDcP$ghXx?2$z_bwrw>K`=RtbAY|gskIY8x0|=(3 z;-IeLb))r|LqPRh%%(tXiDux-Mi6RhUK<1>geR?MEtW?JP+A9#Xil5lhV*A=UFZlb zSX2$U0WqGS+MuEt3UVr#vDQwR_=>$@k$DQ4c8fOjahIhYW>%%W2HV4>Zf-OxJWse> zgCdyJO+&4*M}2}Fr_6x!nkD9nIJa`#J8V2X&-dNdNr{(T)5+3eD!v;xWHwb(Bbw|e z>Midfc?FUrRiWL?)!SS;{k&S|&zNbr`y*cB{v~uy%el~>e|*YO=z36kF7;hq^U;wb zRT3A$nl+Id+9!9mTFZl?ca=13eh#i!R4EB{_m@C)>7%?3&CsM@9P6{@V3~Z!UiKE zN7Z0zF}$57a|F!AfpYXHVh+w#$!3U`kBOns?+H_So zvpB|6gh@siPpcdgX3h~&%-~$Bl**Hqx0M!H-1G*UV>oD| zLM=SsrZ$|@;GGR?IG(H`d^ zBw*~s(dJN>>ITg9A}{6c*A;TBd3m#Q6Az6U*L`-F72@`&6c?(Cz* z_n%K49?X!x%T^?uD>5_TkJ4vxu5BzQtz_GY$*zU}n4fY2!7u^&+bm?_AIh4qHdQJ3 z;JA7^b^OG1Nh=~|$9Zf;4&G1U=r@%DH!G2p$soBF-WE~`3FQyAHCBv+j!JJrgY1iM zW-m!Pn4><4|Cw@(Mj^A98_a1D!I9&k<0wCCLpg}Wy2XBr(6rC%jDcbF5V*v|eF~^b zXX}#S0^k5_FJjNckIpM4fF3lnUXMZ*r7q?tKV9_T-07Rn^`-s!9se>gHruS@~mmk%?m$ zf|Au^_UU?#sRKCpgdJ$GN~T5}NnSc$C&(Q6G8jr_AkfyME9UGu49N7vRnfb}$qJr` z#_$KhU{4XD)(sYNG;&^1%>{yZ^E~HoB_pY|3sII7f5a_^E<-z2 zH(k)Z>c*<6yKzGqLyk7ay}b1+9Ea^3h37d;r(xAVizeydiGrF39yFIaxOS$A37s}S zU_d#;>FAsaG@-WbU^-P&{XiW@UrtJfBuz3f>T>QKv#K;GSGc-z`(D1uZ*LphuI==e z*v>2RWGG%I`{pd$DSeKHML5`f2{Ni)DV72H}wDC&%j z4?{iLfbducLL*P#4A0A!{krL~I&^<1&K*(PO?)ppnF{k_9V;ySG=%w*A*$4%S`J?s zP8Qfa*!MZEvSn(!T4#eYC1$y!z|j6Adyxug{yV>{knMUq9g~UcVG2FW=)MNp9a7Zw z4!JK!DtvtC;#%#4+gH`}?8{ul zvLyYRACk5MEetT`y|PgAU2*=da0uP!=pB)Ng;Pm&4sxem2HPm(Wm)8+B1d;!()hj` z<5ju;kJIA%^k~VpV#qReNmtZPir;w;f72Rk*Fc#6R*_$~z}3p-G)rheeaT1OVa>#( zHu-CiF>+I$ASYZ@oE`ef{!)SbZi93IJ&;${O*5%Xo;k(|V% z&LL^I6boP3mvSqXJaknya|M+&tPK`(74yoOvQK9&_xXCNJH{`X2S;G&!~W>tU(lwG zITq;bi(_WO=xA{$_?~#1JWWyAA|I+L+KiA>Zhc`pesOqa{IZ8XlROxKg8L+8pygrH@RQWASfEvj&PB<6q_nb3(_P6zxh17*}sr zMiZtXxi+d5$$o&!OF1n~NAiINhaa*0@LtXQiLMl_5xIhMb3Uvy&9q;iR+$e&4)ppP z%MTlu-*~!IQn`#mxU<(sul%{3KRQf|bUqr%EYlhqBao|~{*^>kI!M^`)UG)f=HX(! z)x5u^&D(_O2Gfx5rgM1{;;8`(=9E$=OV2-D0-S8eG#$pLswB*EQtCZhxra+VRLcJ- z$RaP7-Lr)nb{MH+J#P@aVZ&+}@=Y-57}~&=Fyg2X1!}W1E9G24uB8*bl@*3H6)K}D z%^qvB-C6LKUvr+L2RHh$(riad_9l|EJ$_8qF7let8RB?~CW!NnQ@`FCA>n_)7#`rm zDHNAam?&7$hRRm!j%Il_!H2~+UBt7iikt*hQGBZ9(`w@^Whk*2{|w$mpF7H61mo%( zMh7=eLR{wWZD|Qe6w!kIRC|kH5fzIVuUknnebdoz8K^HktSP--`3kl~`Iec4LG2m_ z(QdG#SY;rq5sBed5niCAqqBRPcQqeDosC7d*4V((l*8-z6U^P3c!J0S&r@@iwmh_= z@frEiObzYTu7;yKk>NmxPai?9^R(U*Hts!?fqf|VNHtQ7{=GF>NWm+#?@if26T6v( zMoUbgfi+Bo)1D5_zqDDbxXDQ|(|lRen9^|r16f*%$Nz#Rvh@C;_o*IxpSJH}X1jxh zO}DLWi%`QaZ(_qQwCw+6>syiQqexd$7`|ZL>UOWq899wcHG|uj@@3!0jK!7^-2!sA zTv$9q&R>f;DP<|CitUzl<+%YryatYjr_86f>iK0#5W;20AHaszk1x@i%TFJz`jLq2-JEN%K)_hZ z{b@@^Mr*L79-DXTD1n>B%57KcaCrp2Mf7RlF{seaC2>`lMekr9t>86dQ=6`+R%=~t zXl2!`t@-R$0$O+If5YK_-JDuBqgJW24^$1!jk{VwEbE4qs$oeQXdF~!N1fPi$M;QA zP!su7E3$&ZPSa3i+Ail+er8RFcQ-Ya{P^anb0bfJF{XUb?=8WSw)h>3%Hy;u?^+RX z*@Jh>lCnABHdT&#eCVj!n~x&b@NkwC>REV$Ofxxfd9o{OB!@NkPkwSPHR4DtF(&|i z;os16wJE2GpQ*XC#-%K&jN?7uhDtSU6Y*&jeaFCvv>X|c)+U~;QUrdH0aI3uAWx^YNht`U)jHdb>HcgPghhFZ z?yD3~gKnM!y(H&zS2YEyw2H@un17A(2((tM3y7nG#my9-M6BioQ8b+&n{aLS>iF?Z z#Mer@30lXbUNb14#?jD1DFn1PmUBgB4fB;n^5aHDuzlewGWxQ*ib_bx6k-~0&{a(# zu5{OgdeBWyEgyE1PZbC#jV15GM29(;>Rr1dkT0&(uHUwE(^7%kRw@3gjrR>>@Yi$T zFC-|5NU-NtE0u!U3T9WTqM)PozHRm|yQdyy_XLZD`543hyxB@o>1~dpel&)iu;PysSSJztg9-e^rc)XR^T~oQ-*1BJ19(D7BS9! z60LT$cup*c`Mb^R)ZaM22Y@A?E>$FWog42uAo9*)$D<>6A)+XrI};tv!9?CTOkN74 zF>Hmgko2@(egz@eb7n!f5Ehyk54?i}mtp=1Oq&TnkjRR6F?-i>=u_zGO14UOmb5?* z!`qG*w^Tgv+s+&nvtD`U7Ff?WDETZS&8^5SIE&tjKFxk`muUu+^QU`EHkyn4ON9Ay z0}154Y$k&o9w3KybHqJIcJUdZz!I%;Ra6GVvw#0>yoj0eqp)_9ETxIl60qn*|qgWI7wX#*n<)Vd&T$>Rc7i!7? z*5{*`E_2L*Df8MtY>_6Io%&Qa`&B=kmlHHAFP;LynplHAUy(c^GLeJ)z!~aoBK=$9 zBjq;ScSn*p0taf+lE|AF&M^7bGr*gC(yNu~{NchveL=fO-jR@Q-s#M_5(|I8EdF8K zswJ8(bT8I?{%DtP(&D6w0D#Eb0M#mw=b*%g-=*u_XeOtZB7J~LZ8Fb->ei(A$5ODj z1gE!(Z(7iQx?UnARrkx{pDtY#Qw-#L6Po-Yx2&{$o>Z2t>ECFloVS%`9|qUOQMS=a z*c!C;c!i>r_?2V&j~*_;R=+JJXKiBNNz}6N;F1I(nrPcgO775tcNJ71c2y|Qk?_!T z+r&^;kwrN+KlJYqSH?vAoSGu!=cvFN>GOLzDif@0bk&jol=OOioOKrlzx>bYn}T2d zr?Po>Y4dj0DXiD&EpU)D^w-`33Vq$sKiP(Efbf@bxrwG*yWUi%+|YeDc}qiPUthQT zhi~_C8-{HTC_FZ0YzvFH?|!H-&5pvERX)4$@OV-RJTxe?S$qpy2{KsNjj#w*%e>ho z`fjb=Y~sy#W6aD2AI6tOPcM6g?@PAQv2Nz=%FV&OiQRl)4+Tg$5ljQD}NKJB_8Yt(S-xM*^%?puu#L%78W_MpDfV{36g07 z+aJm`=ou_8K~>lT9BhKF&iOn7HnVk^5L}d_Z2;;W##KqmZL8qyVeS; zca%;)m1vqzmM);)VYx_2wHd41e5tIE!G5cj;bak;;p=l*W+)7DIMU8FU%x9pSH@o! zp1BPLF_cSai85#uQ$-HzMMw)=B}o#>9DHeMQXmJjbc$b?=dG%Afh5hfTU$rRmq_VSe8s+8agQus7jh_*Sn1?A=AMzCd3Ax2GU=~r8(RPr3t7$ zkF^;(8^v21mfng`!5qzF4Fi?Yjtaqyb!mH373@RPEo|>iXDUuR-vnPL<^VO42$F zHd{Gm72l;ep~BuN;+?8S;4KzzZt8rxSm^ef*6WUAGjwVmx+CWe_aR4ajMS>rnNcKv z7;HfeQFS>=k8T)1ZP4_8irc-kh(lQ+4?O%M4?Gli4XHiI(rxE&+MM>*5y;0u06fIB z+i3{bQ@Qckmp}9nO>tkVE*=Rv1_w1;0JXPlQj{`9@-l*o^@Ge)eZ=&%<*l+Dkr;Lg zl0$A1K!qTY!OseJZd=YXhL|u*qht%CnkIpFrFSf64lt-7BLDfAKAIC?81~&T7cB9o zONsde94({qcDZ{#h$nA{+ti;qYGYVBv4}fY zGjDd)QoeAAlx1z3Lo7vGX$9;SXRSy_1xR;_S1R{XC#K&IF+-Wg3X#jpC}zoov2 z+~TW}_*F?H*8eKs5Y(WJUPCp;Zo2-e&U{XF=BqCAZ=lQA4|$e#oARtGUu`EjxuIaW zf{mxI-(6n0?CR69KTM_iSJ9{b1SROpYCy2-*Qh;)>QbTaSQy;Bb(+s|5v>QP`wuX;PA{LJ)!7*<~=hTE!sfaDEUv$0tg=AcUnWXiVGzM!Tpc z#B9z#q33+D)hOJs&C9}iDhi)hQHA8=vvr!qOTb~4#A8jkI=KvN(;5Zq1U4DM9dV=? z&oq$%zsmjGVc(`X3qyREwvlW1}y)f$8;9;yhkF_<|P!c^3pd675~ zB^r>&t5Oc5^22ln{3PIpQ)-gAm1s-6pTb_rHH>CT*r`6d$E)b5a{1I72e`I9tPQ{L zG1y`i62a#$d)L73E$4&f*+V)|R+QFW*Pr%NY8f9aW`t>wxH5?))Vl~WFUt}!HcXLV zl)=0S)t>{ia%G3BoWONOE*IBdCo+Ap zz)OR|`m}!I{ICuezC6pPTAWHCx}3w)BIa3c5$$rIC|e>$=SE~ypA4kk`*neWLH z@UwSC2P@;%ZCQ9WYnm#rilt$joB|g|3=!2xlVz|ri9|9CwdpAL4Ip%9g9QbP_WcR=~265zk&|j1j0!mt9NiETm zhiPj`B$h}13`%IQ!XJ{V^rEG`Mtp9v*BG7BUSk2|sFZcgzSEfREp{2R$S2uaM5A}z zT1;5Xt%WnGSofBO!}#QTi`nm&-dhYzf5yF~G&%nZ_Lj$5*mRSwlF7ac|79}<#!)DIf}<0st5S7SL&)FlWRs=ZL5@%Jo6GIjbudv$+F++ z80#t5f}!ruMuWvdM2fpNacKMSDcUI$4+f&N45m&Smr!mpjRl zTN8t~VX#oMM$f|tlOb8&E2bP6K*%1Z)Q8WLTCLSO)1vc$-j;b~s?yB}@CYkh*K zB^zy{P_;R^_u1A&)|ha5?-{3ybUch##(Yp1Mj+wum4x&>kFRXzw0~aC-RldrO)F&N zNSvx!$h4tUghQl;bL}o!dkv?0o`45uDz#0Bbj2GUI4G|B-g2O;ZZ;{|M}hECo8Mlf@yKhgEYqmNhK7+6LG-Cg}X!j13VA#V1j|85;Z`$L4FGR5;GCx}7B11m=nGG(N>r!rED5@n1v?1^ zLc0SaQU+ra@}zJmxoZv3G0=7|%mSDn3c^$P&v#D0xT#6+pS_m`=V_dR+=lE5VOxr-XPWY6vOn-jVGj+6^WRsAP>W&-wr+5;Z;N7xc0lpU1skZ-U&K z?IYxVTEf2Ou$9-&gEyIIK}K~WJg^QwT8OqXzD1G0nG5DY_$>`xUL^CkF(K88Px9}P z?j@$ioS9xVwqgJKJM3KsN05cVe4pBJEl_`{ZJ@*kAJD7W<1b@T1aI)c7fsOzHGUAGFC56!=eJ7s9tF@mk?nIqSBkJJQbi( zjrom;$Y?01BFy__!$zrS(8* z#^tkuO=(*s!Y|SG28+f+satrN>!~|eId$r0)n+qgt3&vy)IvRhtJ;oA8k-6J)g14v zx(@vSFDBy=;DL2gFy~nFPN#=X?Hn4bsmh9}ty^*Ch}Z2VxSOp{ofmvg)vP9W!5wUs zcizHeIb+kZ9+fD2SHE$71bSM?i)_p@Z+4~8=2NPZ*5%!YCE=SqSoRKv5B9j#0912S zw9*08sJXtO5&b~tpHIvRCYp0gu8s?7$hA#~+$t+9vUbnve7#t-Fkh8=RI#BssL6g^ zV-%^WUJ4XtidX3bwwIEAR;6j#P85iXyh82jo1}D%K<`Gc+)}@mK$eMRY*EhDM>31J zVc-b05xgBE;#bRB`mVE)$!o`})sg5TkUhuI2A+UTBLd|=mNmpKQYpHU9VXf@gR8!PMq}D`*XQFC` zs!OUcNTmVDP|+;NRYa>7_$8rE^Z{*F4WU=Ip6BGy6t)sX9p`bp;(MNQ`04UoUNr%z!JDjQ>ongIkP=glzuei;Dz6KVa0-lb~|l-mE{ z+wJJ&c&^7{oNVCvWI}LO3yg{&N;aX<7YqPID`qo9n$gEffDm{!CaI_EHf={l@+j7S{095e?2-etv2#9%;wT4E%*9lgBaHqP z$NHrCxn#f-=yaRg=W`*YqvJ)leC_BIhCaQ`0WdE#ZD!@zXjw!mB3h8~GKA^EQblGD z8SPca+*C>)Yfm8BXpcqMHHU{O2lIPMK~qZksxbdAQa=P-x~KwE*1>#` z@Y>oGo%0zq@VUttD`a!2K8a-5xq3c`{c-G~{m z_xxSs!+#Hqf%-(Ox;E%Zg!49N=VMz$DCl5)+#FaY(D(z!91I%HS$?`j?y=XQSKhqD zk+j7v8}4P=A0_N*JSfLL7TdoT?JG|5nfq7ygkZapMnroLq0JCwERf`uC{^Kx1R51c zSsV>!c2Y;LqSzh6KORJ$pJ=Dw7@?;QZu~)g`Uu=jZ*Kg-Huh$5dM1Chg@j`$tcVA< z9NAJU!(^$T+Ax1tj(Xbb^*YXx+RgbLxNP0N%Ekp2TALhgk$;3OZF5Ci)Ia*4Uc3*p zW6D3+cD&RH-qS%yc&<9)aCZZ2^}#y5on}WAHEib4Lf=`zL5+rcwi9OKAKGm@UgRkG zbXw@YFY?az_37?tBvuQqT8$zpDXdkgt+(uSi|*SS@Ts?4#R(f^|Ev^r2w)~F_6)a9 z^|*D2sMff8seLQ*H}qub%ofr< z^RkvYHdx239o6ifIt{2TX~U>%sxIJNG90Ch5=4G4zEj%w@8}BKMGh0W=9+H$yKD=$*a4vCPleZ z3%R?aV~GeibKz>O*zy_Q&p)Do^J(29pU2~I_v3sUd{I{SNWDXZ>$5}TBD@Zw{L~A2 zU1bQY#l^6}Bcd(=>2&xBSdGQ;P^>aS7m`@y4T06lDo>lAhN2_2;td&A^k z%27g)ui`Ysy6L}xk3vK?7CwsnwAH_ygW}8jsW9YH4>Gn#GXHzJPW1RttLua*OZXoD zOovH0SJ_~Vze1NX)jF#80k=eqb2-7CfC65|m@J|!vXl-kSDO7xg}-ykB4q#kAr`s0 z<VWyu4j68oGfC#8qG5M?+lt;Awk(KE zKha<3$k8h2jvN&z+dSTp9Sd)c+Shpvi>~wD;L)|a`)=;Z5GNu5!W{{oln2 zqb*L+cgGy?UA-+eewNSUVNpk6_poesO%y!D?oh}L{Pq#m82T~)p=y>ddfIH0ud>D$ zN%HP(g)gz|X3^--vl)-BJ>A#Nt%Fk#(gy3a#gJ@ksok;IiuHoZ*_YB4WYU7%;GR-j zAU(xbZ!!eijK@F3@nTcGO3U<@4au)aejy5XoTe|&t;bBn;hsjO`=s_%d~SYKCgnbX z^tDF1x89AgYfA$pr6>oUl*Jrv->aDU9K2%cKaqu4T8F*sn2f?`zD8vS^QAEWd@M>{ zAll;^0&76#oaF~?h^HV249+KJ4_#-#=1)Fdm0fxnSr{Kbjzq5Lz<>O>Q>xXks3mLk zDiVQxO5=Ym>K^U{xVxR#YU6FmyPebaea|J_)!zSUZT;$bnfH8j<4Au!nM_u<*`;*n` zu73QZ+%fNMDd*Y`7FwVUZ`^Gg1i3*|)K}fH*7A8sx>iA-*FVu-F7oR%+$(DOu1A=4Xa^b<~+3 zo7WFoJE}Ovs>PF-u?B5>rYn6RREH;QtZg~{ULE96)*-$@?Q zPmZ45u~hLUaQ=z5LBOHEo}sT85y#)fPA=yvjj^;JM~|d#LZk{MF1%#wU6fqc`7KV? zv7=eFhB7wXR&i%fM|D?CZEE%|%PE;E$fYWZN?q<_ZS7a3bT>FaRk_A_a_SPXmj+$I zPBg{X)Hv(diE`XrWdU_8qgZBB7Ju3j>oa*7Q2&vilCc3rwTd#g&)q`L<A zvwgi-EZuk+h1-06I9^1v1_^*4(`^zzvn<_a!ACemT$@ztUJrY#TW9*|+dKo^_{N!I zkPKa(H0byDMx(O`6v*Jc(=Yz?db8cZ!TxCI4F~hHsN3y2+kSAp9jzBAj6Zqud7%6L zd#Bs)^qnVz&d%3@z6b>Z5be!M#+G%Il z>pLwhaJD7Bfx_@DwQ?J;K})`+rOW#gYzu_m;QcIEWt<)xJzeNMVE;u{^g-Y_|0NoU zr<@<3u6&e-`bA057jn=k-Z{%wN88&sH#a?x!0yG##rA>=r`tchdiLV=@ry2io>%)d z0@V#s@opU^Fz_iTH7kH@hC8>j@NQ6H^e&QsZ5{y0O%kHL0d((pp51_IO6~X|8ri0- zMCZtpaKiZ20(8?k=Au7#ULBupIX^r-es$cTc7A_#^5dIdPn_SMzJ2@j^~tLj$IhF# z&a*eKpT9bJ_2xA^y>y7YBfVBy`AQ9r}#3YN!IkD;)aDr(d zr;38~l6aw@RT?6kw`hzi#_Pv3nhfr=x)WUVOQfSP!i7M^71+(lWc^0XvVpHt%oKpj z#vD^9JK@Ba)DB->36phm8P0gBjeIicWxR&TrZ@+X8u)G+$XUDRgQb<1*oeY+gxOrNnwu8$`Ut`-N{8+v9g-=d9>3BkDuNI-9k;a^rmvBBWq((08UdCI2< zv|YUEc-ciXlepDQNiF1;1>_5qfj5?!{1wV@n16j)IQCE)Y(+*#kasCf&D0%7&73*-ndFbCoUiq@z^1i5U1 zK|1nwS7BasefD0f9{^OB3Ag-J=ehPTa)|oqjLR|jhGn-daN(L2RO+NU8lXJUP-t`j z(oCmX!jR>54n|tB<`jTe-@<<1sobpXQ(QfcCInI&KvAL@C^ppLt0f*`4c8($>IjJ# zKaK73Zngy+dR#DnQcvO=2*iOwBYZ78L5+6V28hHfCk>(dTHW-z& zB%_U#5vAM?*VF--9>N>j9%?TZHNccTg5AK56d!W0>qIUH#41W<@rLHn9{Q?6u<-$%g4m@ zY-4(F<)%u#RZ?SJx0RJl0BTwK{(_NiJ2c)&@nfl?G%WgQ(-OYh~G zp{1g&tP+gcR|bq%QTcp~R?wI@TTd1602}lxfic&1Jw)ZwHn7wVOId{{V!NQz1KteH zlOSL4O@1tSgo`k{bwHw!_Ds@9X>Sjud;ymYJYysE$IiQLCk>&(1(zIr)SeJWwZAY7eFONA{oReotWeo0%*m1dg+ zE!Kdcm~Y6?CrY7XrtISTPv2tG*-v>CS_|7s0p$YJb0`0mvHPh`uhO&#;L0>b9=0i|Fej`ZruegW^EIT zJ9k?~yMwx>zb>OsyNu4uI;f#Z=`YG5ysXJd_oD9eXcz- z*VEU9_jUjJcig}BMxSZ>`YHr|T}1ybi>TJR@O2S=T|{3O(bq-vbrF4CME`n=X!zBp z_jM8d>5HhLleT0aDVZbm7gYw5B8a%g*DQ(BR^#6D0# z0Ooq=zmg#5c~bv>_TGKDZ6wJTeif{|6DHdvggXej)n`1md)l#m@v+^r@r7DJ zfg~t#Op$DYlv?Xi#QwDpaN<0}d9ZnsFE3R<0VwbytH+X>y*u3^P^fEGR%TY_FSm-{ z7SRv4kIa^yZaw*gbHeW!i(OxfjI$00{?H2p-yJ!o8CWCF_u*Lc`&l?PhFiyX)8dYy z+Uu&q;B3DY7|;AxNTJ!rFCrSR*Bh-cP!QHe1g*qV;D{WgMCG1|=fd}HGVaT72eS#{ zlu@`<+#kpa-Yk;^Ly8!6p|b{e$&Rp3j@7aq?iK!1M;)!&_Z(CMubVQWHk&lFIBEPd zHRTkU)nInnPwBm1(;VX0R$u+ik$ri#dtbJSwcp%5o~^q6WpjxgGbQrkP|XbD`#zyq@NOZn=Pi1+rs#gDWDXJFGAVyRF0_})bU4GPdITnD(|onF32R?E zW;sr1+ZYd;sje}dnv5`9aK|r%c~JLy>14L#=!Qb{L+|p}!b4*1Wr2kuYN$o&3tD-r zHX~4~)kn?B2RVjc27;r0P$`dH6&2&3)e5r&3sb~R7VIetzHBs`dA5b(hoeQ3l>3iC zBn?JNuJ)~2Z9KnC-6r`fHa%fd8b z8w4~eF$RI9%+ktI7cqn}U?~F`RyR&lTQ|){dZGpVB34C38?4JK>_|I4W=HJA55g$M zDK;cv`Y#8)j9NMOo=&@xD@>JaK-!{0ytIvcK<0AR;5TB}rFe+#_a1LCgi5(?@HCXD zfG9BEQJ+hxx8e%ihc&#XxU7FJ)vt^O4!gNR%o2K$Ld5U@DWDoMeb>CJA$d`wo`-z8 zDI96ZTmjSil%+%vK-xAm&+#iVSy)!R$zX+z=1s8^P+LQ-wy8}wQps;I0(}D!hxlyF zB>}7o&Y@(8Xfb2LPxv4bpuh|)lVP#3n_{RCzg)JOcrq&GO2>cNqh^Y`UrP{pV79bx z6gE-WsMLqD^G*qlyz;)vSjw1N#UYeo(YN7Y)bJ)AhJxnr#2zW70+3WoOJSIcBdu=D ze8mudoS0^=~1x?O^O1Xk2R@6vV;W1bOhTl2?^4teh)k!xZ-)EiPQbd!C=0^Lq_DeLVE@q zb7U1cZK{>Xe(EY~mV2y&g=DbXDpG7MGesVEqoKbd;gu(alNE5r16t$iIl3igCD#Q1 z7KvVnyb2^I6VRMg;SV$V!bArso`M!STcr&RivLVP_x~ahprpcw^dpcO8@Y_p6>1^t zP#|WXC2c!%UBp>J6m8PHpc3OUUW{YLNf0K2kjxf#=&+5TsC@BR_f?BkSk|Ar#^2zF zP;(7NE6GEuasBgyzAP?)Qrjb~ws*xEYj>?aTLuQMu=H}sx?L%gE_$40qMUSsU;p&; zhqpg}=)HgO!`ts)_ujqu@VdxupxWYoC~87NEw0m-Yk`fc`ji}jWEL>20dxU_BC`wH?H}GdjeG-)z-L=@Ydb+p2GAk^CwLl1I#=lwaPsg|AR~Kvt>>QuJjt z@6q~-FoDcvGQ@5`J;0~2niCr8vLUP+d@a)c?0>SVF2Dc&hxShw(=ky0sxR>D2gZ4f zti*Wn&X~+*b77CMZKo9(&m7xtMRJ?v`8V>T3^S<+s2iaFco@%@_+KEWGfF2gLnOtS zae9hag-N|qq6Oj=@}IZKqAi{S+eKSLACtZTS4kM4KSLfy%hxZ{w*2t)G`n=c`-2p3 zIe~u-x#K`)C(x5dObbtlZj`)X0F@)o9Bv}u2TFaS6HODVUT{jC9s=-+OJ)V;42Os_ zkdW zGn3%2W{Z!qb;|Waawn}pF$oi@L?<%!UTO9+R;nfn%N`A9eqysMHKD{EP&W##K}JeEQy$m6I4HQF|b=lBRt-V5xN&HbfJURl22Cchym z9iA7qd?u^n(U)OjSG-e*_neQ1Ltu?$ZAy;>W=^GOm5CgElBhG{0lGf$pwLc+45o{; zb5pL4sk@!KK@`urak}xidv2uj$#`k>uM73S%f)q%6V**-gEr&B%|;`TFvRWp>8b8$ z!&aw-s-z>#D9;Yi;;Dvn9>|4`3wgd5aY}X_jHw7Iz)7ZfVXsy=Cyd$iq;RxIl?XG& z67=hNybzff9Hdo3CyfWoQrF5SA)h(fP8p`X(S)$C^39=;C29XvMZc|bwer;FNH4i( zLRGRTaQi~WN?rMu(ZJo+;w(SSCPC656V(-_Z6a7xS80ayK0zA5G^~@-JKXq$=BcZ+ zT#-_lw6}|&m@dzF#`i%@b(q3qrnWyMnm6Q<#teb7udvd>#?hv;6$+8ewIgI%)vv(aOhI|%#>F_jKyzPLKh_Zr77HGIQa`(CVb_m7Rr-I^ z5}TMO=jfAuFgN;{DPIzRf1ua#X_wCKn(CLBMwRX*L&ZtLY~UuB!^ zQl-se4)toI-)wA@n49?fxGPGZR7b)m*_vT+0XYU1DAx`{)L6-X3}>XQB^RXE)&yVF zOois z5?}2NiVE5IC6t}ohH3#PaBOi$p_I{U3X)2DJ8BEZxNawFRfXS0U3gYcgX<{xPpYHK zztxa4`!1QVl`!>yPW8|oiJZDrgQ>13g@@H7o#8MU=m-4Z=uF{g? zbEg};P%GP(mcL~~lNJiPRu=VOJE*8t3{lv)Ro+ZWmC7Eh{kFo0^HIo}%2K?Rj`CLd zGKYT>In#+JUejz?r3&a!)HbYI4t8l*-E`0gd7Y$oCEF+3F4fe_gi*RH{I&Nnxvs3B zU>o?Zu#UQ%={g3ecR^9DE%H!qj9ao7ErrTk8T)KD*D{f3IzFPmVk+J2RhDC>q@b&Q zEpx=>eQZ`JA(iGb#}1G`Jac`&<#@pcMxmN4He5F@7vqcWNp**B;M?VEvb9|L?$3ad z5>zz1ZD9%%cjftq1j+Tp6Avxw7%(^5kWz*ym zZ|4RgZCIR*nOs|jGh#d-!%twOZQ&X{E;#n~SEEs~$nu=x4P94pqLQlxM1r>`)5$dr z-xy5B=yweyvWyE!RVooB(i8hgdv#sO)0=MXfkFI*>t64YB5Am&#%ogiTjWpAv!+;jUYpgM2c<(WFS7#O#pl$n?xl}eqHA}q}5rprG(y6_NL6(&7abI z?faOLDxpkjDzlxCjdFldXW8Y#bG^||i4tGTNaHDUBisR)ZfrIxGBHkBGo4M_#DKB{ zS!R8#vO*zsMRXEhLCei)0%xO<$&Z2!u}7iar%Es&aa{fx&!q7H=YC3#MC7+)P@xs%$f3sqf1 zB9ygKpY^rkD_LtUCakgO(=5=4XegQEzCI{|N^dv{N^Ve9rcRY;lsAn{{KNHpdxLoL z+nloTRSe_H8JhIj@8po$LuyxV!-}39b>?&OYsTv<9s&mYa%k1f`rCo*e&v`<=pf3n z-!}e*>q)hpROd`@`qWP|`+-AwPPwZ4kn)vdiLdpxC#~1_hARh@SYU3EO-vI6X1Gj* z|F^hZFpFk2(Rp!URE!4G9VWQ|yLOXOTxlLG18=&exe=5*H5wXWK^&dPY{r;gqouL^ z=VhpKRBn@brJO%TC-LmMG4LNA%n|<=RG55g>RfKidv~1+tPEOOE(DEN*KQ=TdJ##q znP0a*(F)vDBK%axm@Gm>6*;w$B8oQqjV3xlgkGy&K;`eyh*akm^lV*4Yut&71DmE8 z@LQC$^C7Nx7S)Hxlzelx;?5ftDVd7sq%1O5-r?HU90(4(Yb0v7iIOd=%x(LZR+*0ZFDg1DI1_m18(RivRP9T z1-Qr$ScILf@kd{y5^b*+*64;5X-B*-`RzOgkt-QK7it{7`V$7Z-SyZ;FGdF^1E^x9;?TQxE#^fyJ^?$Ode$$jrMM@q6o=YPQDAzL!7@UpRV1zoAvbs z@wn$)NI_qsxF`9_<&VW`L^1Pg$zG&Pr+n?~O&N^8_)JrRq>NxZ6C)rUx# z>y5U#Dx+uPYzfL65H{I-J-bD(jCz0{tsJau0(B7eWUp+KHPP1O;a(RGT(&7_jvDQ# z*BVi_lKK&RrZmi|`LX7_Em%#+zLT3@K$EQ)Udqw@h3^OPvLV_*%h|<|U74*~Ew{wk z*QmOiVt896(?;(Ll_kr+ULAPUiieiv(Qla1MGp+FjDRJaAyQ0iXKX=0jnqLdCrn!i%V*BvQ zW@uQY)v7T@lv!F?McIKHuKl#2*bc|(0KLA78iBGZvuWmxlU>*grY?W>_}W1YY5xcB zPyAaui}jv3xvAjlJelUMCDO|VPTTP?$=qN@li5|tH|IyXD^#k(Xzxivfv93hQqhD6 zh^w(Zo}Q|pPS_iaGn1}AvIjEk`SdjN#zBCV(^J!UDg0kxjv2*3H9q5gXfi!R{h3A- z_h+yxRF9(Eg9n{`&;t;mJulg2f{sCI98|J^j>E7;w}3ItsJk3b2h~m{=uHGw;$vY=L)AK94jNfoQ*NMEdI!dNtt1RGocsl19#)*gS?xm3Ym9>(5g-y(#&;4 zH$PnNE^UG5aoUrJXpK5UV97Z{>DF3Tj}2wj{krCkR^i}TXLoRnDVna=!nNInVy8pP z{tVuD2R!l)cx4H!gt?^7{el5b0F6L$zbqZRG<#HTzH$=fYOkB}t`zIa>Zf}v#Hl?^A;+vac@h0TP|XJ&zBTA%1q&7aVbrOvE9PU<_386tO;d&rLHMeuOCuYm(E`$ zAB$cr!qv{ou=h%{>Vqqm|E_Uf7)aNl*pI;NC0oBC|U=q<&`=Fm4e&(A9r;$mkw(1J8RCdr(^ z2WE@$8HreQA4AK}jhdy3*3NR}1@D>4@Yc_E-PY4qQ<(L9%hi^3ie;VJJ(;Jb>1BI( zlG@vp&)Z0A>*wHYb=XDfB*qsxEyo7N=Lan}3wnd|`XZh;vZGh=sj9hvA_k4+O}fTF zJNse)M?qSNq*2NiLtT*D5qsh+2jYwglV}yWycDxI8mT6Y<3Ou($?xx`5-BPs-7F z0m@1vG0#j;hqAO5>Oy{XdJ3m0^f;~5wwfXsV8(Fe+1sCi7Q#7=(q;?avmQoEY&4ci z`mAz|8G}y=;uZRok^kxGKmX_dz;9$i*rnybY?|fBm@|~pxF5MkEaCQ~J(6k2nxPPz zChQL@DSo9Bd)f07NQ9Ob5u$Lp8p>D4P|*pSt8`LpJXo&cN$GQ5$oh|f|?Wj;w ztp&YAgjv9xAfPj#GPRyosh#HLtJlgaAw3Cd?9rB5RAxw0Z&L$LQrMu+2#4`mO&R9v zc@)gfSvtlEQ;xGFzK!zQ;yf0jAIZNWM4 zSR=8?$iyZoFT#Hp3CCQlCX;K?$$Fd)et+}{t$&39-5h=r7Qt-HH?_5h@p3id2NV}$ z=f~p5np(>9Bfcv=ZQeYut%3MjFYAr?#F!;t8Xg(Ek8N#x+ga}hwZY9 z3HtMv_bJKL5%OD^L{@u=73 z>S)(t$CjK%Mfw%ia5EK-ba&YPVa*!YQX!bKFvn$L7#uI33 za(t9JR=$}R{nXl@WY_ER*p&vODONWPg*m7UxFYbDPazU|N*@((JGaH{mA6dPxnhp2 zrXIW3sNKCpvu@C`1LIlfz<=Fsh*nfb#rmu9WXPl@%mDdHEBCP^AySM&hTL4x`KOf_t_l51m>TZSS`qlm|ha(wP7RgE=HNJfJws0 zXUl}fMCSIUu3}Lswas<1flH;E?W!xR$idHa3TxRYS45!&jEtR#TZTWSY~GRs(E+{4 znW)d$v}R_s71mTnGm_6j1?fUF+CqM$%CJ!t-!NL85~ZwzV1SB5JV@qC<2C(aA;~578bMFML492!6@+eZ&nv~_1 zvw82M^zq1^%)6ZHQ9c{a#K&E7`j@BQSu({q_Sr&ug8WrJC?dgAo;Ho6j&im4VK#4n zG@fDW?YSY3Iin%uRAV*gdz7bmJ*o}_RBp64fF3XEqjML%V2nPTCWCWS#%W4?2Zc4; zch^jbsAImE^-&)A)=X!kCBX{MfJk$H1W`@^d{S`*Y@;QfuuDjrGI7wJ=jMjl8Bb6c z)7b@ZkOtP(Vt`C4m7@QTtW&M8+s;#a>{wwdU$Ud@ueGfdLT|V+nLF=JH*aqU%=i3m zws2OL+{lBzlSIDn+kM|O6KmKB?Yd)xCx)Es81z=QGCrE;6e^11+)I$;BM| zzFH)$eEmNvf6rSvFK-s`j6a}52gb>Wb~6LoAuRmdIK%X51UREylw?unf?XFXL}-`i z#@QlChSx^CI0G(<_4mhdN+JuQ(r}`Aw$mNZoytXgEl_5#S@k|-{dAPXqvGt8URLHx z`<`a5?#;ZC1WV3ptD?VF7;dXh6Fi@-NW0=0LAVoj%x3ayqW}jd6r@>-us*$6n$fjm zMM&3<9Vuv>^>khE>YIsh596B(;F_YbW(Up{E^}j!Ku!4gP{ULB%~zT?&6);;#w?c4 zBqk2iGQaN8;flNneb%Acin1oYWDtWgkreC2ZK3WLu~pXWlFcpXtQ5sYH6_-NVkV;1 zy_oA>)Dq(s1gS%4Bexf1baE`2=X_x{iX2e-`Kq4~g6!9vg7)qNh-Eza2nyWi$_`mI zQf!Dup5=y#*YAMX5QTnc5IIiLxx?JtN|T>0F(`F$0v zqq=Gm=Hd9415RkhFhU0u=?fftwLqr_#AKT$SNsyKX7OOK0{IgW#_`>1KEywJ@c2e| z#erUBm^)>e%(-gLAB^XW#CF0r=tSQ{y3wf3D2$+I7t<%9-YKo>e3q*pWJ54l)AZL> zlKfZFuyl;+Jibh_csNd+SIUgR4s)_pcViL>4PGAqYH2a@ZYsGRUL!ObNjmw|x4DbL zn5dw(6ZRWPY|r^>6k*+$g(O2yjKvI{JwU}uCdmaX^6_~18|L>}&a!IY3AuzOF$hyK z0h@fu?yqT~P85J`?0e3eU6ZaR+`wb+pkEsRqm`$PYDo3`i!CzuTo_ror%e=zTQ3ASYeKO_rJR#_*GrrewKTEzb)zGAaoy6VU$0@f=3irU8OzL`Zn- z%F5HiYnYAy9C&_`PM8`}UC)`kmNz zPgCQ6UnNtQ8_pK$^c3Sl-YycnZJaIs0xAv}#!pY1$nel9AGToWBn{=MUrj9PhZoO|ptX^&Rq#tzEwIo?P$D1Lz;RFRh2|;+`FEb$%}sPs;di;t4W**%bPJF##mQ=l?&%2fie*6qm*MEK%UbJ zF7(yee1-YUfl#N*9*1Ed=9q9XE4qX+;!D-bt{^L-K$r=N(#6hpJ081ON=u=LQ@#}7&Bu5#~?rr+5;PaF+~q$a!qvBxQbKJe>h|ZAa`KY z(CC_RhuhYs8gat&WSxh}3^~SNvLYFkOisfMupUI8rH<_csVNi<{i-1?>z3zPA-Arw zFk=Mim|byZv@ZSyAGNf-#JVlh+S2E1L*2Z)Ffl4!^l`v%r6o>c==&|V1KZ5=ahqkT zA8=D4i0Z6A=4PP}^H^Dk`XudH4t32HVliz4qLhIa#$m1^JrW@! z<2fjShzo^^NzSSv7}YAZY0j#}bh*g1{i{B#Wqo^&NkFB=IrV;DDTsw$u zW`V+^KuyesaMRMNKOoQbIL8x~%zpvDP3Q5nH>s*tH+nltmQ=CDLSNca$*OEA;Y9KT zs^2YUFh#jooNuV@3HVzQ`ipW&F*9YMLK(b{n*JolFw?xkHdmMlrfd)(NxV=YUG=H5 zIqOAZR!{%>_GeNB-jV#T2ze5$i$r#%I`r)5Ba0CD|IZDyvJQu0zeT8_LvriEr# zqpYKa>R6PTEfW4d*RAOrwg(`_GZ`&O_qqTr25%c!m^8~5lhlk1p{37EdL%OJHL5u? zZD87HcClN=cPjUXQ|&Ud%QebusU(!CQyw@ ziPNQWEh=+j;^QLXaVdpR{+VuKmDkzKep3Yq$(19#WmUeZoz`q#OnQ*JKfs*kt{sY~ zwt0pNiQ3ETNq*6)W=$w`zJR@J3z19F^qM8ns&+tCSPi8&dFd81bnF4WB@W-nJJeG9 z)0>PxCd$iZ^QPz+I-jLWDO<8yU!a^4QjGD{SG^o(y(j>5i#GK_HET2fDR`)rXB9%G z!;A^ETWkZRmJG${<|a=v^JnR0E8L5E$h9&IBbF>c|0N`PbrJIfOX`FtmsxUXQLJKoSKszHb64R5$-1CfVqWF|&Wj>ytZMzq z|AuKhuBM8GYMQ0gq)6CTi!q$!ELn*x5oOeQy)>^yY>^>?l*=N+qUQ?MCAZQ7#MkD- zA@*M4V;0aw7^Hz7}C_ovXziG-#v&gv862XaO#S#K_F(|?+> z7}Z9Eh{N}P{T^0EhODJ5JJ;r!1JaOJq@QV>I66qb8ON1rZf^$bXPg8L>=u#i68woFwDL3xd6jYQwXBkGJSM`ha zsgxu*43<%IF%?uwRg|rKpUl@^r*GEni=S3?dtf!?TH?W=lxS*`>|Jmes7txHi}|*| zr`BxDI_3G(T2p38rKdkyTYRDD&zq2v74^En=5g``yr9*}l?xrZU~6L|1wDOT@teAr z+84V4t}c>!?`k|9&aQxHd~0nk5TFyCf&h=#DF)2Q$H)_)*Q)iKCgX@a!2dz@t}XgSMktyyq1GD_yLs3ct^fQCtKnbm76*#AI`-p%bULF zJBaRanP*F{FBNU!g_^T&v*Hz4jbpBena5+J%h|wBT6M5TO!~srHQi7wvpLiktoqOD zisq>#%9$i8$Hh_M;Mm}ncT_pPY8~~Clyj`sX-_Gnb>OSIN*=L4ww7wje%VI~Dc#-! za2vwTQuRDG^`eKWUf6g74|aH<}qtlT@;Hw_C@v6N1>uaD?(Ykf5(>4cF$$=2r{=+!T?&`QwoEI~D5MrSorFYt4)&8pS>gJe7+ipcR#*Dc^%d^)x zffrqtC{`-NDLrl!{J{mao@~nbhZ^6Ap%7*=Sts2tq$led?BP`)$_XE@;>B57$~MTZ z2ULW#M)jg^rw`j&ukboNaCsB4{ehRu?}c$c+o@S~q6!}9bl??X5HfSQdd{#NBI`i; zIzxI_m@1g<`MZ9iHl_b=uZO&GLmWP`<*pqd~+o0*`mjq5|aCDVFS~5Wv%sSZHMU2 zTln|WXTZqYZG`8%pww6ep04XV-$dq7qH;ppAhT$1bLmc7Q_+*mrpw50$r3$LFUGie zRJUYlA%3nXkJ{>lLTSWvs1+fjG1xLTd`r0XFJ_DlA*?g#-Y*qPQ%5OfLr%L2C^tag zyB=z#pJ^d%ufG#eTdEPkv-c%gysf_nkmjT8v+e9i+S=~Mmej7fh1jae^)@B#9$aHr z+QF)%?H%utR?S&fVHZRH;R4Co)U?lD$&vPL0jiAKUq4ZgT}vcXHptB&Da; z^=7HH-{_R|cI{=QHrP{NzM@{{aiD9zC@Gk?KMCrZx!K}aAI_#vmj)wia+Dh48;Fp$ znHox|r?1by=h}RVZD&8y4t0|aDc@7VL3;~>(oSd2c7`S0%-(g(ar?72#@4~bdJVT* zYY^LobaMB^O}e%w(LLH_Bp2SLQMTtaH#R9$(6T9%nl0UaOZ$_CuEncOwo!Q|$m02B zK_PH;eZDmnmw@-c_=4TZYGZM-b~l+YD>Ng~d!?u4HWB4pnQpbM>~5cEe)#})@_K2b z*UPry-5lGk#&Z`cEXQ_d7~oe>k<^wbR=P#@ybU4Vr~v8%visLr_@%>;c5!+=9rUOJ zr>D&wL#!(kL4oRmWynbphg5ZDo>$zz=i@oTLnJcVdU-xe@y8YZ3D3^X%iV`YibF6p zrYdyU-(;@ZkuPRdmN;WQdx9@+#}A8NWDV<>QucBaTT;89eDWuBItl%@L_p+ANA9J|z?s zb?Bw)oYBDupEzNQTYG6@E%d!NS+wqB@>X(Qo;Nz z%nR`>@|5%t>V|Fn4qafoK)EHAEugbi<$dKy&rr?jE(&Au;wg~WLI!i`h3^Um!9NOE z#KJ$IP{%k=VJ*Vtj&d1*qMlGoc*_}--xIKus8yOx87GOL)f6M9xLFn$elun6QTPgg3Wd1(vlKr-B-0W-G+y_YyT7szM{w4cH7%9DC}iXmL^wtt=Z zP`r61BfVb;Tg*6>A?Tc_K#qvAU?*b)R_bQxK$LmZ_MBFrqdeV8R~6x3d(zBXFb+$s zN3?;|pr3SaQ>S8KuP@iin$}s>pX8RQD%fFTHEoyHv&qYcH~oXzVki=LE7ggF-DHR* z3wJ-6%&sVv5nGnwMzKY%OO@Bq3()8CRtlK~T=l{?{@mu-Ofe#n6ximT=^5g<&a=IKZ zk_p-_heksro`pjnNOXCX083#0lqP404Vi&{JrUxjxUvK4A&@r$*g&B|X=RQ8kwXqx z*=0*0qdXJ!MrdJtUSbAUo4~b1hCb0hmVh~?0Zm5)jhsC~W0#8N!0sYs zzdBpZR)|K2*5(g9`z_rpniZ(u^C&9VB} z*s5MK01A`^SwR~o`kBF0-7KXtRU$&+ZTXPM+d_cs@rkCz%w0V%o2>_aB0$?6uN5I^ zJM2)`-R1nFmW~iRGG$StL2Gx!A7L7hg4%+7@#qKmmndAxjW=47-vX&|ph+F6Ps+-p zWUEKTq-D8yR5dSF^gDeW#9oz(nj31$eh4RWoxoVqHKhlCow)dHM{)6)MqGThNnCuk zNnBKt#y)LTL?5>*MgQ82r;V7V!$1VyOP;}48DgHXM|78e%? zlPYS2w|4#(JqCw4JQ6`51INsJ&5%IyQVKf2O`%t?#6*fkA2Jz|Trow?ULjAW3YxPb z$-=Z11>g*!L!4rm_jFE6w6G&SH(gomf+Gg zbAsScfnplv3Mp}J=}{TrP1B_GeCF21L;0@RR6;1S8{L#XFH?C-*zi&H2I)(+h7=;? z%HNsVk1$|8*QyC@uep6c%?)%%A?!_Sn<^A84H8aTWxDZ_ z3FXwxhitovJLY9SDXg1{Wmx7vEoqb}bexYH$|ZeNY-yZGvmBda*c z8;bS{9pwr4Manc|nkh0T?F9k~w zEjYH8#wbZh@xYT)<};9K1!(=00gxSqAot=zeWV|dW&$VF;A4U+3u)IBb%q{M=O}EZ zLW9P77os-MoMHr~F7|I$ni{d?bA|$gn-bEsa(?Mg15vJWgn`_|c^>n8xuU4}ZZtmA z@fH3^d@kL~euK8Luy|Z{~qdzKRdvt zKpMbH*VdI4($q=yS;~1>sOjsHV~37SRBO$6#C(<#T6rc75;=@(&aQcBbpWc(6zQjf4(V9ncB7WVA<=siSX9n7~1D8%ye*phpYyjTGzo= zB~6t6T``ZgmaGb-Q`=}xR*BB>(3?$$ z;=ZEN&-#VpkV6#0)c(+}EDcR8xExw}AJ7iZ6{m78wymTqb3U7ALsVJXwzB2M3KXxP zf&!-t4}W63pglp>P9sHQ7~*A(QyJ4ZL^G4jc!Fl80gQY#Aw!dTrownCn4gU_WFEn+-X?hMR2&s>f9q3CO5o#-5_beFSL z`^2+4tNeI)B*o{t!1Jm(qZT`vIYGuakJiPrYq-XSdJ7%d=99v*))nO5_Vgwc8o*}z z%X&M^4wW?ZST<5sJrdh!4aS1lOp5bn773e;kzscw&&WAdx3(UIJ?3my)==a!0tR2! zSr8(GNL}Q;$6ck`lg!pI*`}CcD9mn>C|8I|^B5QYjmbZb`ZNcKUVQm_gR=P^aZxQt0ZpGrttc}{prbSu@s!l-b7~N+Zo7Y#MXK*^k zrZ}(Sl}N7_PbR${ddB~9pjx6iK~BAs7IFynV`-W!)uDZ^^K`5!S@FHXawoN(N>{T5 ze0zw=f^uAx9y|3g?n&7@X{#@!N0J>nwqp-$FG6?Q&M1kTq#yfUtw++9wU@n+w)_k) zWx@d6c;Mf*RhPaBkwy^FnlVS{enQR-z)v^#zI+eJ34q9zji78yUM$%wahMd+Jxkd>n~a_o)~$u!ZrgZ<92bb;!AER<Duk19LFq6+!HNe<(|CnLJ_Mm;G@c}!%VPzSSPg=Yk`go(JQJ9r_#T*X=HQjpos^82isDNKRpSXQEhS0dE* z7*Atd2&S|icei<>P3=sJ41-F07mL|k>psP{so6BcilCCc>@g;u!+Y~?rOmBQp)HS| zn)4=hI5T=bzs!)_1h zORco{t++20<+7THoJt+5+=H?vVX zHIb6ZqocCkdrLMqk}}mKss<=&r`g^DGS* z0kqBGyd?ioXh`xesSuVW5%QVg4lDUzTTtOJK3l}|^GsgA@f`K?=lHv2FcpJVI#`V7 z2xgo=C}j#|a_n?4sK@R^zQS%lmzSbXp-5qt%HxxU1Wadf%|3^?qtRk>JVpVoM`w@Z zt|#j!|D^1ShKaGWrBhpP7UAXoYdmiht11?!p9^45WC>_i^Qn?lA}}|P)h~^1p&|*< zimiy!&`I=ngMZ_@62lhiknmIZ1X3>MnLV~ny7>wg1* z>mpu!9L}z$APOwo!*Lq-L4Izxzxf4t^xN-Wzj*)J`1Z~B-y3g!G(LRy8oTrA_4}9a z-n{+r=BFQx?_R%qZDLXUANF>&Kp|Q_d6#e$1ZcWXydVs+qHW_Xb{j*F#C1m&!e?AM z28a6z6bAH)pvWU*26J#R{x8OdMTrkR%En8RYkV;r3bPeVXiAx+q1_bq=1l)a4O`P5 zmJ>56Rr$<#Cu)BOjGXTDPqz-jJEgwk7~3%6U7_Sqr~uRCPKHJDeL1C)gjI0J_Ecf1JBT-PZN@npiEq=i{Z7D+y2>}5Zh4$hGl zI7v;ixXg$C?f5rXkqb=keFlp7f+@FceE8|rPs-YVz=)K$lXxnhef8Dl_uv0u{4H7F zVsyXy%DA*l$1|;Bfi{|UjhC~Fi}4Z*VIYhw@!jkS7YB|e>JtX9h&QLFr}!T{%eeDg z07w^)M&p-XvVOMDXV5X*2~7+B&xQvy>W=NWg<@>H`Dva^c{joGAC~sjha7Z|_n7)U zX8b{rs~Lj8joX}~C!Hq=^i3@XW#%(HC8`;4f~Zx*bnB2u`o*Jb|j(=+dgJUsfaBX;nw?cbpFDNEzIx4)JC%_ZXX*FqIBzT&p zz{f_*R^ouX$nNutddd+Y{T zU$Zj*`TAY&{foc77EZCR-n={7wc~<##WB{PFeo?~k0U;M@Q2!*@UZ`1Zwz@3!f4 z_SyV;c@CmzHk;`o;>*xa2{5o`zZWR%ciH){4?lv`J02uG@)j<<)Nidj*u3iQbc(xE z-G9XixPtyCn0HWpC?u5I>;aXOO1nVHv+t<5e5w>WQvNaCv%GBlCEGImouQVZ&9WMx zzJPaz((>EY20=R{B!3%^F<AbI!0~w99scFIk zit$*R<9Y?89p(l=wfC^{m1TMz(jW0oGwa2&ejJWz(Fn>|1+S;Tab|ev7$_pZKq_bk zIBvXReeJyd-e_ECN03iyXqlaWzZs6V`pWLOf$7kUy^W%+nlW6y7N!8VaCxz9t#JbQz&bIi$g~?_g0&*4*-H9zH_Px-}TPCd@FUpW9#fxjQxM0z6*clCN-=<23vjL`EFnr(g zhC|nZJGrqi|HivHFyD++!-*X9JBts1OUV0v*l}whg;y4M0?kKQ3R^kM1V)yN^Gx7q< zb)Cct@Y6*y$0+ONIevI}4&y)k_n+Utes{>3#xG)!+)xCZk0)@H-~j>?outa4GGBoF z;2qrDKyB&j0yyY3)-_6&=r@1{vL{%8NvvT95nge^kQbDjo=kg&=ZE51^Ev!C8GM|f zJIdR4?GJB$cx`-m_xiQbz#$sy*qR5}<{4~oL;*YyP=@sP89TioSz>3teeve|H$VQ> zc=_Gym;dn|dI3A=_-TJNo(w;Zm&$S21{TvEjGvy+&KS=LV|)F>7Mjwuy7g*@V~tdXKsV)IvSO9{lk7-Ctk7sx00Y=-7=QMw*%8V&)U} z@BjJJyZ?Cq_QlKBhi{NPe<1RFDX0@gp+g~y9fCTO9tyezCDsP1CYkpZ&LR3j95T`3 z-)+Zo0;^-YZQJnzCum!J+wNP3?|%NV_wM!EpWb~iFNO!D`QaW>I<_U|z5k;9S|kXx zTA8)payFZ!y;YhF5nk-z3_6X=zwLi<$$Drm*x|?ab^IQ!t*sLyf_oqoe_l6l*=_CT zC-6rMN7Dt}WTdo6-zR6NYLcbhfJG=Ml9LN+9Ln9S9!B|(gI z4sgcVDqakY0i3^t%S}Fwt2uQFVO>U`rJ(lniuf&A48|#lAsJu-uOQo9 zo^z)-+JJBd=xcIZYoqn!ES`hx1JVroEnf4>WImZ)lQ-6E#8Ecv@yH+DJDvN#anVeWSja&z<+vd{8j(%rJz;y;%(=Q zUB{?E{^w5L;t+EOw8Das(^qPd*$6o@cE!o~uMySSK5&E;>) z17;2sro-nL$LOQ#4eDV+CL6MS^L_3z=^A(h(oO4Ay|dg?;cb>OlfFRj8U&py`luO< z>@r*p7$#6$)QYKxadh3#_BpH_&e+uj7t;*5Yj>Q0VdIupGoh2gfke*cm>X^DsWYw; zI7&pa7HSqm516sZHJuRZ)cgf=kQs{@m~o0sJohP^!vPEf>C%PuXtr7yx%-zv7iPE! z3^r=8J6IM3fYIK0O^pH>bwf9785;8F+LmXx3_am?1K({Ka$gv0*h1I#BCnYrpR=p> zYC67zg9sgji))8@FzEOn#3GyQSLfrwIV|yDvO*7=w~OQh3oS*_<}V<=3#3&Boz$||t`j~t z=)-pjr(`4lK&b(tG>+e1;2}PqVm7~-@cFI`uuLSL+jZ@L zgQC|LbE3%T&z5-kwk-NAcoD&aAZ^D@PAL~hhRU+@Ypg<2OeC-mp#h-fo}!Enfy8Z6+de7?qi*Z+m!aL z=Muv-VF8X;B9DhAxsx`}X#5*(eR5}~ptH0vz8{k-VTd9G4kA_B2B_)bfardT{c7k= zfT0I3+ovie9SCv9-m3qp@cprzZ5o~GpLB# zLm)nO8Jdho$>16!)UKh6Q9($LB&NUsXo;vPH5&CYQP&SVRwE?#Q5MjKo_w;pl*4uy zv;^n>uOP3U&t`n^oP0<^&bH0dE5$-1#&FNKlY4x^|^LxWD{p&UP8$#+~Z4> zYwKK^^D`vdZD=vlAKjrxgd@~nvOj;Iy1$(*@)a_bGZeQZ6V?zhji=99B0)`xPI{6} z^74n}G|-1?TbZmfon`)QGFdwb5H*wh3Qd}58R}NkOtV_cfv>Qdx?QQ9x^z zpr*I)j7x^?W(x%QfX1}2WK$mh#?tlqB7qLZf2V_%bq2eA65>>`);Av(TF(jb9r zYDkB9xn6i*oA6l(G6oEW=e!N`$@V1Jeb0&iYfrx!n}C=Y#8GxgRwH35f&zGq$ojGW zbd156VQQb#!^*7k%S{IZ;!;ARi-eX&9(mGUB#*(&T2uxo%ls!u8R65PrJf#&z9<8u zCidw78Ru-#OQ(1*NLKV5h2Xm6crRk`3FK;fYWH4!n{)rn1F}%0$=&5n2FHU_qhhWGp@fAto$T3b8Bkrlu zNX#=+fVNSLjd7}Rf}xSOv!S0h4bEz^8i=fau<4*PAdvwbUZdlo;!Z6%mJIupjLkGo zakKTVk}=4#=_$jc^6KQhWf`Gy`jrXnV`>UP-~0+D1fGpXcK)XWe}a^WA{)pU&?6C& z*Eq$L2>wr^N7cE3=SZX0o$WMv+9m zr#`_FmyY1>D9MQNvgMcoxwCzXLJNnuNM3v>i@f8wB31#2*`&m*a9Z&!C>j6Nb0J-h zCzC7=IKKxljrYVT71Sl7PBMZkT=SA$&kx1Y3hoJ9>O4V^LnzwE!2n}8WTJ0rSu3}Y zF<9a~z;bZ`2r{* zriiDp_F(M_O4B!Lnr8DBr}ff|7GC6(oB^g7Py^k+NPt@og>B(^tz;y%TCJQ$G8d+Bz{@FI z0;uK1JN0X6VBuv3OBk3p;9ij@7L2IbsnT;AnWKa}?>M zn~el_a%zMT(i_6#O;f1WIF+ylT#Gp0x1v9h@)|bDWHP&A+twP26#Q+yMKGt;@&Y%e z!fZ(cYS!?&t*-?IU+dWQY^^8w+G>ig#jWI$z~K&hLIfN3B!lc&c)4E734cc@lHV7J zOX4SvuV)}L71{(!Dt6wmvz?Lut#Z51(A%JsC0(`V9>CzkyT|2$?S;^ADsNA zFzu!1lu@GG02B1Ts{SE)%xxfsmtg_R|X3QrmK~suh+Z6uv zWY|%kE9uHk$xTz-;FY!1TN(QsRVzQaCuB~X<>|?D&A5bGCA?o2 z#hgRND@>8chO!ZgD~HtzV0^%H6Nc~RIt23)Gm>2e3m+YJ`&KMb{8_4(4>n&b;VGk; z&QoN@X*pAjn2M%2);HaIZLC7b4t%Rc&IECG>zQBq@xbrO;!|+hlA9AKf0xb4X{6p> ze?TNzjs0D?igK~izI(x|hYVQALY$@RQr@5EzB$Ex3O(n@b99R3Jj&N&UR;TRPbu7C2qz&0F^yIe@>Rq)a3s*|tR_h+obpq?jhw^_0lP`F+cjJlM1T6t zrkO*-b_L)LdOa(V7Cj{pJcH~89BW;Q(JSzSkW(@}pMeLd)uV5XaLV4x4rsQ9e^Z0TML`9{6{g33 z#o3OS2k{)%Q13$(U-ZXkD?IaYo|T&+>GDLEZ|{`;T)v;Fg14T=?StW5O z_hR2(?Awcdd$DgX_U*;Kz1X)G`?eMPPVc}Eup2s^Q{$NAF@{~)yGrj}rT4DV55rX& zXiu2!SsLEENjFQ3aVw|6ZpSY2z`d97-b?tNcnPapeD5JFJ%r2K{1#4g-TzhG^LzKz zz5D7Ny07lIZy%WJ>Y8I#=VysK4b8Kq({=Chx%c>dI3Ay6Vf)c}dp1AUs8B?ZsM+*; z_sd7&epxx+9+T_kjys_aTL*Rk@Ij{ky&L4ccY~~u^p|ykY~JQ-xr!3Lhw<3kruVk# zy={6A$)YwC6>^*k(kQ#ePjXk8s-j#F@sj-LD z*n3~@y)XBW8hc2My({D%QezLPv3E)QLqlrZ2uzJtN{)s}GDQB4*BBJ7lRFyj9R|Lf z!vHP?o$^x2r8v;zgZVYdOYIBe=>Uj04Ap`bPujklNaVjNGG2H9`1Z@@G* zGn*!wnrT%H^%g{!f0>d~X#kFavqJX<$W5?|?-HU)`tBpVIK=a#jjPm#L$N&=3r>z< ztk9tfi9*v`DCP$zIsL3KpWpGxO>uLJU7h(fG=j{tHXpx~5=}zgAZlqi38oEn5r9m1br3Mg`?x7n7 z7$l{ep%I22n)AQ-*7`2a<+**=zI*SU_3Yo4c@^8&a~|msVUZE?kG!xTgitU5nGOUm$bK18Z9E`zw8ho z-1N853x*DqTw8U|Iyl8Zf95)v{r^#)U|zOx*7KBl&;ow7utv7eVUZc5GIR6TtH4Yw zt^0wbrdll~X%QYlv!_T)zO^-J;z9nA#n=-me@j*_qaZwlY10jtQ6On`SN#0Cf)k>+ z*>`%m!~W#|_MFjS|HPL;deI6cnsAFmp8D}8BhPNh1%4h&9%$VY7kZQlyS$R30}BhsueFZW2khErF5hKxn*ZVdqZ zYeT;_F@>EUY`;j0lRzFW_^bYtMbBUn2r5sX{-cvR;#=)aAGBE{nv}b2crYb1%ujsu zBR^wYg6qYLN$Z{zD}K!8=Femlpp}0T0)Ntubabodu@oep z!wIx``n{dcZ@&2sr~=TKw5rlRQQ+}X3|p%wVsR(#09ujzhwZj&SYl+Pe#|7ht+5m1 zO7?0#*m9a|q8}uS1eI)P+0@V2@Wt%PLeQ$xen~Up5FF!m2`J(??^2%vCrUEL&oLx8 z+wQ0NvkkNi3O7dGTR{pili!;sVzZKe?3=MrP zvY%{m4wAtWH8#=00rkO+qQIm>tb%H);)gexyXo2galNZLd$PBMDDP>LcxB#x zJe^5s@bzfMc3`Ix#@bM!A~g5$BmhJd@@;L&Dx`n z!&{VlmejoM7r9=BXQR6{pmw^OD1&j^cdd#?JQ~pm3-(uj{p@;(pz+^p2fPP3FzSUq zDuzX$yI!ANVn@BdUz;ss{5DI@4b7;j`@lIVP;@$3D>{qEDCUM4$wy(D!hGag_?4C3 z&Mp3w6r=`Z{QgNJqh_-2ifk@L)Jt*h%3dlc#0H~otNJ)6dEY>@;jrTm-L?0_3SirezXTAW+gou>5Xm(D-oyd@|jFSdr}jCK8nB;uBm ztq0fB#CKOo*!V_t5gSzVACy1B)-CQG;53pW&xgHu7cHqLxnKAdK5svRQ`>hK);MFe!6u5^Qht%o(@)2 z!%$HIyHDI*-3OBg!_iHCqUUHTTJFP0mVSnX7Ahi%Kv`TlI>H^EkdO4#;pHW*(J>tD zeJHR9c+NeAD5HtBCxEB&kf>Y5AuGMN&}4jSO*$`_>8@}OD-=f8%q>9>YxscxIX3Pu zNj0v2*j7wcB+RG~%2t-ok{+jHE%HM>n+9w<`D>BQ$IXNGJ#DXx$45Be&NsIxRN^@t z?$0x^-p8%;MG1>G!Q~qK*K-q9&8+Nllf>lx=j~M_m*tW6y@z-XE8-T?x-&*uA4go0 zLAb^=V)gz{Fyj0bHM6>%81?X%o8?d6A!_MPNDxPznVk#BlUbXfVv};eaxhpkPK-KD z?0h=_YYfxJC7Md?42}NZcnzPNN}}PuPP0mie?X%)H`VxFoa4=HkX8Q{Rou~{cMcU@ zN!fM^U=~QleT6`zK#0K*$_hlQ*=qB zJODwqrYn(h{oXDwbYAnkVZ@?g$k-e3ac&JwTL{5pn1>UOW z6oW2ASv#fwAn8IlrT=z;GQb`CL=AopV&d=?YsWF+#~lb{>fPAdl;!_coCL=N`uQ1E z;DMHcphxHvC6tCc56NFTqEa0&T>sXKC42f&Ei`d#bYQiQJQ~U0W#`L~N9;R~&cmc7XO#Q0fDDO0`ZZeqI0K@&hdLHf1Ato8pHURW5j6alPxJM)y|wO_)Qa3!za9j=&xk@_MuD zx;3Z~j!r9wk?dsBV&fyea)$;#?@%n3 zA!2+8B<{5_rHg@mJ-}y|0Qa4T)XRg~T+vLYekH)hk+@KGxs_NM6=uz`M1dGOxh@M{ zFiVhdt_c+IN7Vg^FqxdR)WVyKy5FnydWrpK^W+`>KlrcM zCR%9hts^K&#Z$tHUBRO(Ed%y4alSO>Upsv^qKniFlkD2oY3w^e2Y$#=KacoMdvlE?>L}co|`T_ zGn^TcyV>7qxYvg3Awu*oq+OuNl!*Ga(Z4_lyu)X|k31LBUi!*q?`w9mSRvgpy2cXuuuDra}Xy*)dHisgnmzPE7#HKhYpx&rQ z+4z=bw|MAUbn4K!ah?R{+7deTFmE$@)kzO&tws6FG4+5{E}4bP9u(7W_5Z%A8PY1% zs-*OAhdntR>KH#FS3zPr4)1ci0p%9G!-aSlL@;!0t+s3sktXVILwbV^4_@Vl-+iW+pY#r%>;R~E zRzUyzp-Jab~x3ovFd1%JrGPR58 zSVDcLY7g+Ui(LZ$-{AUxg9=!%NF8(cWDcU$*&p*G2`(GrmlgCzM7S321-d_wTGKwj5;}oa+0!=`Ow-n|H9IZgI zKhNHn4DY_!`xdkMqNcz=?{cM4#vi|L`DDYC*|L86dOY4fLu^6|Q@Q zm=p0>NtM^$@j4|Q-`mshRa@jb_wEt_^CnM7zu254czOLnp`j8W$}3-4B<3M;^h{ljSf@ z*NGM*`VAabafFcSPB?a8Vb(}3M{1Ais{T`e3xqt3%&>d5KKw*~;J@z-T$!t@3hW|x zwGqs#*0-v}L2~eWRVfJ{x%qqk$8_C8@;qwc3P}Iq{IP_UnzH2TZ)P6f$|82A;;oYr~vO1o4>{&?FX74DSjHy{%ME6G{wyM!x)aD9S;+EmA0E z*q*n8Cym5DxdkOdB(raP*zG3n9DuVkZ5P)}pa}_h>3;g}`os+~TYpR` zOlpv{&KQ3*(k)g3Fwm7tMtZ(s_o}2Y51IvELQ^Rmy8LVJKC1d~Hp4q(jkZGk&{rvG zW=3)DW&aiJgVSt*(Dt))J<024-A&bDi?qSnxlUy!yIFaGtN#iRiFD%&-W0Lx7BOye zrOp)OPfkZKB>VJt8Fv^2=eT8)DPrw5<1CfmA0!JVEdIXkc0MxY*vGt32aEw~HKIAQ zFZ}E`4Rj2K2|K$A#z%KuIxHl7}> z5NpQ}sjdeLGd%Ch|1@x@M?b}OpvQfxCFNH`6mZ&`bOTID`;Pp<21+~Wgis=Lq#Zw( zL?ZWr^_T{Lke9~XeK2yhwsoa$)j3HJV52GBY4!}jk2vHp!+Qy%IHx&x6e&3oq|wx+ zmlX9M&Wc+#{*=rczH45PtP2=zNoBqe%(aagYW8=Rv-@7kdNhvWgRW&+tNH#*cYAEy z{w@fRRg>C3pT~G|McMMlF{C9nM%wk#8nij{uI@&T6~#IvIuJvD-OqL;_Kg+ak1aRJ zp3N(Y{?g4cmOuN$Fy&5i=3&*MP==g>#IQ}Ne*0hRLKvy5v8{MXl|?j7Z*)C+|JX4r z(+}yA_|U1Pl*+NcidTC7K}8-tp>*o5ztnDA$|2NUnNt1tCYeHL9!o^BV;`r5?cZ&l zY@6R^L8V1=CiyJ0Rn}mzGyj_weTN6a6BU_oMs^D=xg8;14`n>PQN5EkkB?Q6PAbup zubtm4;iySnIOUy11|~d!W5&bmLc3FfB|(fhc)0x~cg%JKg$75zFEStDjYqJ@S@e~Z{4AV~AefYpQG$=nUjeEQTLwa^cy--qq>S%De)UmdR z3(tIQEx?5rj|nScC;en0$f@RMGn@%W-{0B{oX>Q(4;P%{@8Mgf##SvuEP+Ofl{-zi z?1wg!cPzU}!qY+uV@V~vb+>l5re$>x4XHNdKZ}6ZVayCLyr7NKO^Q*L-xy|?`1a{M zB9{giDwYarW}JQSN3QkJiN3ZO?HmIn1&(GVOzM|JqBO_1xGy;wM- zNFn2p8(ITin59z8kL(%m^Qc=+DAqC;jeZ?T#dT5wDU3{onaC@egs%Q)t2HpG+)RaI z#8FvfEJqFru*BHTY7x>$GY(I^nCFLY5%obw# z36}+X3*cjKoG4&&p_*_?TG~IYhoGm?q&#WEe}2uC=?Ak%c_K;XVm@5N)uMgwGp>4G z!3viSQM*Q7WUEGt4}*y9Vw@}z=FGU_$yuAUA@g$TlUgL`3$?+Oj(v{{d^`U6bKSN3 z?49)GI_JV~8b;K=RQ^=P9U4uBVm|w?s?$#}{x-Shi!a;xLNus0e0`-dzrje*%xA08 zusr#eZQTMnhU8$UD;@&m(V`NOzfjy7dhlS4RO2tY&`T3o*Q^chiG}_l9iEEXEx&trr$v@3JZab~Hh*;n)*`EG; zF8+reqlPSs8u7fsF@vHv(i{%ny{&gmMf1X|U*(1IqzRjm*{%ef(v3LptN=GIwomi|*wEUE`FwIRY@a+tJBFm6@qxJ)#NZ*&HM$>T2v{(@<@g;nmA>y7;@59ey-_v< z;|H{;-8l3kXda$H6Vic?D~T3 zqKCTvqAHjSoYBMn$23o9y}qtp8nk3r=}A;95vddGNnQjZ*p)9f*=7U*8Yg`_cfRkX zd-vA-rP`Q4l-S+GdYmbsI8)-d-?y6G8O3M;M3hJA-BjX=)%8 zH~r+Fs|P2a1Pl6Wy~ZrjcSx(XWp>wr1pHtpVfQDkMkOTxeqNRJ`YoE?`P7XQXAM_W zXXBoKEfIX5NCsxGQW#JjgPh3%u zd2v)Bnkq0x+W(eycmE5e>3mZvhcVBSV6%?#)LY#`?f7qSg5ta(V|Y2>pwD5H*bT-9>umP;7DyY`D`8EE`meQ#dgH(dqOX(INFwV9 z$%4z>Y_UDgq?mpU{9T~Ir%`fNG4H7BZxZQ_m3&iSd=#eX(v0!RdUZiZ;-fI9>|dsY z!R&10jFS)ziB9O2kfyN9;ILEU@iF0xJRmHPqhbW?V=#0U9~Vj3nJdam2*DeJ_s2`0 z?(Pt$PV>d>RnO^u!qbyEWF|vPKjKZ+YHNSJ&5^jtE}3e@&@(qvUw$5XSZ!5QQ&Qr5 zPR7}w&lT2isXp@_GfHHhZl-y9H2rKzGdL!4EH z0|aL;z05cQCTUahjD-n-M|g{>w1Y!hvrU{FkPt(XPlzz z>aG>C`KbK5Tz{D{{3-!AyiX)kh;i8;90i`9WH5m2k z{E+vG{gtgrfJyj2A5BtpT*_CqkzRr-X@{M9kqD2Ax*SZcw-Bn4cAP<>==#zkf~j$5 z)lGid?#j*ypI_1>NKamka=6ag`edJDzo+B(Dh>wqQ9*Cb{9Q^Z-bCHmyl-upi76j) zbgeyUcAH5dmAxeqr{4t)ZPZyk8j%8%I)in5ceBUWdRRZg6pQpoH+ z!v?#$f}$^jRJ8?XO0g5i=UK7au{05!R%-Fp0=khb4u`;GRSUx0Q$FUo0ZEmHc)MJB z^ulT!mANV+8r0WWq-qQv+Smb~=^LW<*|OK$o3+@Vjzvl~qS98d1Q3>1?6t>vK=nD2 zXf#Iv%Wg4&{b@CbL)0qiWu}Ez$OE@AI?^v49n>GAG09yJi@)bb&io4t{Wr(h#+)lB z-y}NM+ALq&fqa~br4wg1{m=y4TBg8ZJsLReX}heLV;>_HIn(;XaUhqnu1Pxp+36Cl zKCfWv;oN*65m%bJ`pGgG5yX1HTW9j`TA;_MEu)|0#vxsmI{dW=a-L^c@Ge0}?8Ckb zOk#v%a-jX^n{a#)`3cfwLDc+eko?SjL2{M^BJlUf7YA0lKV69g*xd)2&I*prp=}4! zY-@8^psk7IJhS;q--@sK8g87QO#?pmX;27fn6QVl-E0fMa|N;s$eI|t6wg%q!d!Le z@-_iWZ@Bje$TZ3!pCFwojsab?M}xR)H{W&oacrrFjQHg1_0j(nxK-fTimZo3ENpuj zMX-to7JfK!8SQ7`Wjfe}6L=_Rasq_(H^fh+KaP3nuf4r$y*u94cq2}c-Re3mXS0kX z_7`SkH4bY3yx;;OwlyVnB-d#}53lkeRq|UDGrJiyrOc7Po0HHXqz<1v8Ki$LM&7Xd z%n4wN8#br0BR9TpmAYtn7>|?sC1kn=j+vi6zc+i|Q2LN#oXF2LkXh3u$?YGF&gOzc z@wzrymEeQXEmsO!978dw^2ksJ?Jt?j?bguer(_61F_le!SYGd!M2oZI&y?!i8 zL5MMa2f&p?3Lqu%XvN$qae$a;#)nR_?;OYKLr4bm*hy#*7XW|s3T?HhU?Ndq(f5bv zcLvkpVz%oWH4eVxhE~m$<&p5h1zE;5bc&Z0cy6!un}LgX+-e=D?4XHY%hwOiu!|G} zUSG|MC3?QI_bL17$(q(* z0pm^~<>Kz?y@_(;mWTD^;!;zU;vw_DlB8XGe_WlAuc^vH4?*vE^%dm&=TeFWQYtj! z_^$r;j$m4bohIta>`9?pg_UN@)&kTC|%tvG13R`x*Y?xoQuT?cDw zc1s!lT)um&fJN(|<=Mf)h{H}5QyRq9rkVz}>UeorWmD8!j#W;wu@jc_HH7?Ipind) zo|mK3v`kBNRSA%i|6Onq^yVicTOm-z4C80fKw!+i%~Er_7x#}D&dCP)Usl9SRCQGx zlOMoWtDJvb8#{%xRd&joztjv6@$W}NzmOTcZr2azWOZ5+Vm;0_V)aiM!#4HPyp~wO zvLYA6nXFX_b`Oc+`@~K&)!nYmGs8G9p$cep{XwCH-o^a!U1741ZcOAarVB-}A~BeP6Ml zA}dVXB)M73(@K3!og>`8vB^J;S%vVLp6Ax|`Lv`xFn_Z9Tbiv%B9Wd12jTaNnm#^_ z>3Hn1ou>>Wnb%*NblZkc#RWOU>w{EDS;Gvb@kEwoUhA3crxvYl{|VhL6;+O17^Xz8 zREY#khM84_tUuBAjKZnhh;0U(Tt-RE-^Nf+U?EvOKuBG(yG!_>TvG`7?^+$M3S_Q8f$or0@RF6-@RAiCsq)y?67+;+T6+<@Jga-Pp4 zI4D}EDsCp_+e--O{8;G=52e#LpvMm^Tw|T5AOX$e>hoHobP4l#(Fk2FM1>ZGcBjG> zYc~NwGiiB|zYvjgX*8CX32)p(IkCxru6Y$vRE$JCabSkvP|u?KLS2=Pd{3#;n%m^K z_<2*k8ZnJhjm8$?&g_z|d{7q&OaH#yQh7kEeb!p;KcpH)<0f5f!IdKcv13w6um5cI z;Y|=LG$czfn6@pP-OO1d#K`CKuXp}LxX8AwE6*EL9{OP4`1P`4wxFlgfX>rV=A6ee z&mtn4Zwk0~%PdRN6GplQNqVng(VSGWe=zs3uex}CdfOTiRZCY>XHb%2o?`(6sLF2t zkSkuol|mnapRTpj9|Ir7Vj$Nv=JTP?ok2?>?{4mo3u#_%(+exuJW2Vbgj3ZDo|lcT zZ#60)FdSABQU6@qBa6Fti#rz6t_^x{1KSOTM3t!NFg@dPph%iw82SUjD ACIA2c literal 0 HcmV?d00001 diff --git a/eval/dataset/arms/copilot_v11_deep_r2/pr4923.cost.json b/eval/dataset/arms/copilot_v11_deep_r2/pr4923.cost.json new file mode 100644 index 000000000..f0244e5cc --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_r2/pr4923.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 1269.0, + "split": "train", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v11_deep_r2/runs/pr4923/run-20260813-124627-247844", + "recorded_at": "2026-08-13T13:07:34", + "attempts": [ + { + "invocation_id": "inv-ef1cb63e", + "outcome": "run:run-20260813-124627-247844", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-124627-247844", + "llm_calls": 98, + "input_tokens": 772321, + "output_tokens": 150441, + "usd": 0.540588, + "source": "spans" + } + ], + "attempt_usd_total": 0.540588, + "llm_calls": 98, + "input_tokens": 772321, + "output_tokens": 150441, + "usd": 0.540588, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-124627-247844", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.5406, + "minutes": 21.12, + "ci_minutes": 0, + "input_tokens": 772321, + "output_tokens": 150441, + "tool_calls": 148, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.7732, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 98, + "by_role": { + "agent": { + "usd": 0.499102, + "input_tokens": 756353, + "output_tokens": 116703, + "calls": 95 + }, + "reducer": { + "usd": 0.041485, + "input_tokens": 15968, + "output_tokens": 33738, + "calls": 3 + } + }, + "cache_read_tokens": 6169472, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 3.16, + "gate": 1.86, + "review": 1261.93, + "report": 0.06 + }, + "time_to_first_result_s": 1266.99 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4923.trace.json.gz", + "trace_bytes": 24331 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v11_deep_r2/pr4923.md b/eval/dataset/arms/copilot_v11_deep_r2/pr4923.md new file mode 100644 index 000000000..dfeba53bd --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_r2/pr4923.md @@ -0,0 +1,40 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4923, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head 3b2d87c9c3bc) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 3 finding(s) below | +| Security | no finding reported | +| Docs / comments | 3 finding(s) below | +| Behavior / compatibility | no finding reported | +| Correctness | no finding reported | + +**Validated:** +- [sweep] Diff hunks verified in situ: vllm_omni/deploy/qwen3_tts.yaml (lines 35,43,67,95 area) and vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py (324-353) read directly. +- [validated] use_cuda_graphs=current_omni_platform.is_npu() matches qwen3_omni_moe_code_predictor_mtp.py:21; CodePredictorWrapper._setup_compile/_capture_cuda_graphs confirm GPU uses torch.compile only when use_cuda_graphs=False and NPU keeps internal graphs (common/qwen3_code_predictor.py:532,552). +- [validated] talker_mtp_graph_safe=True is consumed by OmniGPUModelRunner._init_talker_mtp (gpu_model_runner.py:208-210) to wrap in the platform full-graph wrapper, and _capture_talker_mtp_graphs (gpu_ar_model_runner.py) captures per-size graphs. +- [sweep] talker_mtp_accepts_per_row_generators consumer found at gpu_model_runner.py:1854: with the new False-under-full-cudagraphs value, seeded multi-row decode batches enter the scalar per-row fallback (gpu_model_runner.py:1852-1883) — basis of the major finding. +- [sweep] use_async_omni_output / eager_omni_postprocess_before_async_output / omni_pooler_payload_include_hidden consumers located in gpu_ar_model_runner.py:1534-1710 and 2020-2077; the async path exists and reads these flags. +- [sweep] Duplicate assignment found: self.omni_pooler_payload_include_hidden=False is set both at ~line 317 (pre-existing, commented) and again at line 353 (added by this diff). +- [sweep] Stale deploy comments: qwen3_tts.yaml header (lines 5-7) still documents max_num_batched_tokens=512 for stage 0 (now 32768), and the inline prefix-cache comment (line 43 area) says 'on by default / disable explicitly' while the value is now false. +- [sweep] CI gate: buildkite/vllm-omni-npu-ci and buildkite/vllm-omni-amd-ci are failing; the diff adds NPU compilation_config.cudagraph_mode=PIECEWISE, so the NPU lane plausibly exercises this change — attribution not verified. + +`vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py:?` [major] — Verified: talker.py:350 sets talker_mtp_accepts_per_row_generators=False whenever talker_mtp_graph_safe and cudagraph_mode.has_full_cudagraphs() (talker_mtp_graph_wrapped=True). gpu_model_runner.py:1854's fallback branch tests `not getattr(self.model, "talker_mtp_accepts_per_row_generators", False)` and gpu_model_runner.py:1870 recurses with a single req_id, so under full cudagraphs a seeded decode batch >1 still takes the scalar per-row path — the N-call hot spot this PR targets. Decide the seeded-batch contract: either keep per-row reproducibility and document the N-call cost, or reject tts_local_seed under full cudagraphs and remove the fallback; fix the 'silently dropped' wording accordingly. (evidence: Read qwen3_tts_talker.py:324-350 (new gate and comment) and gpu_model_runner.py:1848-1883 (fallback condition: decode_batch_size > 1 and any seeded and not talker_mtp_accepts_per_row_generators).) + +`vllm_omni/deploy/qwen3_tts.yaml:?` [minor] — In vllm_omni/deploy/qwen3_tts.yaml, line 43 is `enable_prefix_caching: false` under the unchanged comment block that still says 'Prefix caching on by default' and 'Disable explicitly if memory headroom is tight'. The file header is likewise stale: it documents Stage 0 `max_num_batched_tokens=512` while stage 0 now sets 32768. Update both comments to match the shipped defaults. (evidence: Read vllm_omni/deploy/qwen3_tts.yaml lines 4-10 and 35-46; the changed line is `enable_prefix_caching: false` under the unchanged 'on by default' comment.) + +`vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py:?` [minor] — Confirmed: line 317 already sets `self.omni_pooler_payload_include_hidden = False` with a stage-1 payload rationale comment, and this diff adds the identical assignment again at line 353. The duplicate is dead code, and the PR description's Changes list overstates it as a newly enabled async flag. Remove the line 353 assignment and drop this item from the description. (evidence: read_file qwen3_tts_talker.py shows line 317 `self.omni_pooler_payload_include_hidden = False` (pre-existing) and line 353 (added by this diff); grep confirms both occurrences.) + +`vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py:?` [minor] — This diff sets talker_mtp_accepts_per_row_generators = False whenever full cudagraphs are active (talker.py:350). tts_local_seed is plumbed from request.seed (serving_speech.py:3694/3706), and under FULL cudagraphs talker_mtp is graph-wrapped with a single captured RNG stream, so a user-supplied seed is not honored. The shipped qwen3_tts.yaml now uses PIECEWISE (not FULL), so this is not the default deploy path, but when FULL is explicitly configured a runtime warning or rejection should be emitted until runner-side generators are graph-safe. (evidence: grep tts_local_seed shows serving_speech.py:3694 `stage0_params.extra_args["tts_local_seed"] = request.seed` and :3706 default seed; gpu_model_runner.py:1854+ shows the seeded batch/scalar path that the captured graph cannot honor; the diff's own comment at talker.py:334-341 acknowledges the silent drop.) + +`vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py:?` [minor] — This diff opts the talker into `use_async_omni_output = True` (qwen3_tts_talker.py:351) while the deploy YAML raises `max_num_seqs` to 64 and disables prefix caching — changes that can shift per-step stream timing. The PR body's serving result reports `Streaming continuity OK rate: 21.88%` with no main-baseline value (the before/after artifacts only cover `talker_mtp` timing). Please rerun the same serving harness on main and on this PR and include before/after streaming-continuity metrics so an async-output/prefix-cache continuity regression can be ruled out. (evidence: PR body test result shows `Streaming continuity OK rate: 21.88%` with no baseline; gpu_ar_model_runner.py:1537 `_should_use_async_omni_output` shows the flag only takes effect in the async_chunk streaming path that this metric measures.) + +`vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py:?` [minor] — This diff adds a model-side __init__ branch at qwen3_tts_talker.py:346-350 that flips talker_mtp_accepts_per_row_generators based on cudagraph_mode.has_full_cudagraphs(), but no test covers it. tests/worker/test_omni_gpu_model_runner.py:374 sets the flag directly on a SimpleNamespace fake, and the Qwen3-TTS model tests build the instance via __new__ (bypassing __init__), so the FULL vs NONE/PIECEWISE gating is unexercised. Add a unit test that drives this init logic with cudagraph_mode FULL and NONE/PIECEWISE and asserts the resulting flag value. (evidence: grep tests for talker_mtp shows tests/worker/test_omni_gpu_model_runner.py:356-398 only sets `talker_mtp_accepts_per_row_generators=True` on a SimpleNamespace fake; no test file is touched by this diff.) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v11_deep_r2/pr4923.trace.json.gz b/eval/dataset/arms/copilot_v11_deep_r2/pr4923.trace.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..d8cf28ac542e4fb8a35445983ff7f42b32a675c1 GIT binary patch literal 24331 zcmbT7Q*DG`)4S`Uu4-3R6GlS; z{m%jUz3#bo+Z1c1e4}ocn$>U;fqD+C4B|QWUWi+~UViUvUF^Au>Y+peB5S2el1*@P zL7DCBI{`M^HcyA-mS^_9r49gslKL|e(m0g8hye`}TDh^C_dKa+&vA1se!9azDA!Xo zKsoI<${@b@4psN_G^zMPk2~FIzhCyyTE%f|R>aeF1OK<3Eo_|1XK3VwVT`-m|iU-!F$r#o{lT-c!|JN5$T>N;&{#pvSY#q2HCWvM3Hxg6)t zN_2AD_kcrsN$R>ukx4d!*uS?e0~WlP#-fR3JHEGB+FQM!vTf+k&kjInA~3LJnyffW z?opr`;e_?}`ZCL8-Kg+aYbijlJUfxq)x+2b@aS_Z9#$q=4o@IK$j;F{SWW@z!(eMBr^)!$V6T@`$(K3nC z=4yNDtO$;jU?4t1s?B&C`)esj_VCZ+Zz`yV3LE`z2gT2>O2^}mJ1@ojeJE4;}X^ECF3qW z&`{fK6zRb%%=<8{-Loo2A9q&xp^p4;sd63*@hOUm#h|DolKIN(=0YzbE3iqlpfPo~ zD67_T(AX-Q7$t($UNQF^Aj3;jPt3{KGo;B+qEn<5gEFr&eRZV78pFu~m3;CpKxk0j z%l2{UMp9AC@1> z82+7@#28pIL>&5&tC|7DH_AMd?m05WqfaqOe+29sf`J(}gDEXJBGx0`z?h;9ixci7 z6yqX9)Bu+tipfprY7>VAMQ9l2>X@!lieY{8O=p8ujqSQ1>BkAA1KS=9<#{p)YIvnq z)rr@M=lPyIH)%Q@^XZQ^T|Ji{^7_T(b*i=U`uW`Wejh*D=lZ{@-=6R0@_v6x-=^pF za}zwD1RH+1F;31%C9Xx*Ey^)_xLmaSaAi*2ud?s>f7kt9ml)f#^YlOqO0{r4QV&XL98T%dV-nL>8f?Ob72nNS&Zb<4coFh%YVpkp zN50mHVW#KuP5KKRC}e$)tLR>k-uw3O9O@_6s|U_+X6ArT zLU%R@-v2fbtS?Bjmp613-XKnsuPD%>aYVakhb8P&uE!%~aOj@#alwWeZ`bAL^xv)@ z=wfegn5*ZbBn)@T=T6)D6SK5sj0=K9YQO1cwl7dPmH*kWCto|>JQPslB(`gaPo+Zq}mNKEj*Z?q~jY{{|EhKko;XKVs!H?k$Z0 z{qr2RSSb0chzIpZ z#X_0l*+_jT16g$)|!}j$icAi5-2q= zx-nb#J#xIi7bYTPd)YazeXhAKn8fPA0Oq(4mnbWy{#Fj$A-2Hv2H}yez|$dWKXN~9 zdZRDdwhM0cNm%g9xA z)^PH=k5(2GiLF0l&&$upP&~)@R`6*u%`Wd@ffvkEYHXU?9cyJ;bMWp!kv*_fhk1(- z5fb|w)4__zky|aFceW?xn;cgZVYU5-Q`I?Zl4FZS08&mC#Wk3#FJh3t)aca z%`Qp3-jq10&I^!5K^N8_cW?Go7e=SDH|z}#rC&BUMZLi-Z#U>Xv|=w1M4}x4WzM^+ z0HbBn)<%xGjUlRR+C9hPCUGlRk*nOc3%?Q8okI_s0YZ!gSR{C}Yw`Y*eryA-+?Rjzcz5IEah|O&}=Kcv+; z;^8FMR3A0AO=vudi6EZoP_O?h2p-vB8c2tC4^uZm!ojat?kRL>k1!7XB_J6>5uNzFPsTvR1K^4VpP(Es&r8M=8DX- zf#dhUY|bA!g(FTyvD>F&2OGY4d#79E2y6}I%x7Z(dx2P&1VH{6NHr1|=-x(rz#{SD z(0(L|gR(eAkMkUU9CZMwD=2!Dvwcgekg{&g#om#@5%ARW0tu2`3uIRws&or2huHfX z8NKiMKQ!rx_YuU|JYL_I*;9r*;Nv(;FrT;(8ixi?zBAxQ2v0Uyh~ahQ>~U3|AX!na zY8Zp1GME7^;knyUxvGFz!F@(?f%Wqw8wVL&fCJo9IA%adekeTBuOVknl_N+wr?0vN zZD@8laF+zW+QKgJu%p~7%JMVtPJEfedv&OGO}Bjpe)Zq7O|rd@szTfqMaT0HABoQ(n1~m$s8t7yMaD%mZ!n9Eirf5W_k%yr=cUtSPrebi5@xE@8Pu`q{-PzR znabMn7(c@t*}Vaq)VK%s3*K{H6(TqWb`yPKR}-gili?UGGR?r<(70y9su1wF3>QME zl^5h&fgonkTa$#s1Z9SaS3>YS>lTFf_X(HTk^KHHSfwyDBFt5~H)M+HdjKrlOd0!J za!tq(!N>oCFRx3NI{Jx{$b8CxF%O99@IQ1-%wkvC?4G8v|Adz#ec=oolph%57AdVRn^Rn4AYg zgWC2nCzZ9@3s&D>7o4$YXGZ_{b|GY4W5F_m!!u~}e3`;niH!*4=xT<5aJ%JBNw8Pu z=Z_#o8@YO(`hFj>?|R${aQXdiH#LpyB;q6wsTfp8x1^lGu4NB|R68d8^zSvMpy6$K z(+m;>27UOcplX{~c&i&E1$Sx=V(7~uf$Rkz4b@S$E4J!))55#Dz{80>4#2^s^?6%B z>T6hU6}~$r`^iaV<2?xorf+xiRVLs2sFc`he1CZb39V&JM^AV3uR)`@xC_n!C@>lM zu>KTnX@T!Bp@f;^?-s+tiCNqs&Ib`{kL;ZkmzvVwkw}^2vCDcTk-gWH;YL1iA!9fqtvF8>w;U;wDjgFIX92l`saHF zrv4ophj)Z#5kvJnM$16CF^DY=7(y_Ji#-`hSLTivT5Y^_)T+@XWJ*-t&~D7T;)wG_ zKT&?ah{A+tm0dy^_mMe3j+@i=kf#lSd7^o9lo?xZh$CcaXa2Ja62M;UwXy%V7{7g& z+Fw$q-S|qycrrZzU4@No`oV-NT_Xo^4_@LXjF={(9AO9c;tkE?#sw+QJO7VfVM(f4 zj*=JVCo_bftMU1^6EMrUrLmyFOE%X{J4l8GzWMfk+(LJ+-B?v%j^xTMs3oF6e}Qw6 z!qiF@hQfdLiyuAHDF$CaY1RQDkxYzyKl6#RJ6L5WY?z)HrBcYMnmp& zXUQ=2I>Y3XRk>2_Gi3=h|4%y}rKNlpPm6N>zPBuBJRH^bTHE?Q|w zU@k%6uz06EwIT3}LLJ|bAe`cPMgkpgzeYwc_7MX#4M@w)28Bw?`O?n)yw)D8_a7gQ@E>3f3|U8NjISOGl@=`Tst^06!GrpfGePIs9eFOK>0Q z6f(PiLUDk^3`s0E`JJmGlYJw1_tFwt3Mp>jx;E_c`x178d1v6%#)^ z##KdXo&(9II76~d-cem+3z{DzA(d}>e08#oXU~!6YibsI*w;)Q3V{BrKtg6;5qz9e_`#s zh^QP3etg~dIiwi}&k;uiBp5fuCoK&NugD*EW~b1YAWB;+p#cVkrzH0CE!-(;tD&NQ zp+*LR?nWg`_!}Y5y5!9@`3K6Bwrt`zh^4;7Z?FyKLr)#1=W3V%vgG)~uYSccyj7M1 zEIA}B0F!qTc9e+JE5QR%bx|%INEpnNif@lxUI|5EyyOUgGl+(#sR#l;%n%8k>LFKA+IvS(|f# z-V~f^fs$Sf7z<#u4gr=tPF()T7?sR%8x#J@OhX)<^0Y=L_pW4(N*XLpuffGnr~e!$ z?_FMF^6Dnvvf815v_m6Z@!x!q+qDNdG0B|`%f76>`1+>1IeK|}&WrD|Fv*n%Hlo1F zkk6() zAV#(OWhjU;7f3M3R~3A}I`*RNFre0C8|sZWcZM;HmFu1*@$f~z!Hg`Pr035^J*lSD zjzWc-Tw?~pNa2m;RgV`GA!)FP*H=$SkF^Yk;pu~9fY6f-d?%$q% zr-O*V>_iT7fQa$YFuehH;snGhAaM2Pga_Mb)e{~lsAA}PXv*m-$yInFPqvPhL`Ma} z0C?C{rv!m)z_w3@7v$sS2Q%9TSV>p^{DqW!!;xv|mbkA13mh=;ds@1niHAa^A`8Ik zqnj3B5PRt*2f-NN(CT*_|K0YImv)VgpyQ^hf}wRatO#7q3)(TaZE#|?o{S<3sa{aS zrnXAPj$=*xshPenI6-cb?ppqQ9v%u%8z^>~{2q;*9GiI9mL}0G+vzmWr3$kv*>-z= z+oEovp|ZnKzOk|lY>szQv;s6|tXOF-YI*X%AZg?fk;Sq@?-Hkxn5 z2oAePR6l~mgu+|kiJzMi0G&1eQD8I&(_n(NMfcTkPp>B{ay6i$RRjrUy>3OW+s6T; zo*r#{W`92Bihupk^eUlnALnHq5h0zrCSK(1sDz@xypXyaFJ+5^Xsju8afQLOewXVb zOGbYdYp&t`<`8_-vi4!l)_-M0Mm3!a#T{#l$73LS&xJ)F|62fRnjDTnZkv&&0vmzO z*5mEWmlx1JWz(DQDvjnR`L82t%W6rJ0~~q#2i z`Q!q-Y&EC{4r}>_0Ac1n#2_)s=9E-jZXZqox@EneR8>lo?51;q#xrrL z8A}A~kjXBRW1*Qk@O@au@ArJX9(|jqhsvv8$fGOe!%}}Y}w>s7yx;JXUw!eQ`LHpp)Phm>D5)bCIrxI)|$TXZ5kh3)_ejTFb z_Q^Jn{!%~sv@TvN>Jj~kRDMEZ@T->$5?x?SPxmBN&-`53CZlMWKjhcdDx|h9`%SRVL>>MOIUgxt@qUo z6co2e!M2nZ@G8gRpV1xmE-D&HN$6R;wc8~G6Z^PL!P;k6*JjTBB z(^umfu+Z0UQKy4@$~K*MA4pJ6;I2?wIh0n*+Sp%<%~ecIytomoQq<14Wh(lkKLw|b zdb5^&R_DqYTB8fP6*LAwr&9%4voG{IU@1vXER}L<0e78?bzOZ>v>jbp+!k6d=i1Sr z=&P7;FJB}5<|e)vFRsBtCipJl1V@V@sjee3Hf_c$e&Ma3+>A79O3O6KQg$m|K7vPy zJY{WEM-S3;A5o4r=6n=sltxoMP|#=!Q6VX`h-+6XQ7IM6@9A-|4`z6MjI)#{WKc zXeXV}|1fH|$p>d|c=tCMw0K{tc&(3Xd={!@#L&yybM)fK(Tp6VTXaZBbHhuV^tDkj+l zk;Psg6Tuvv2*zP znoR2N{fAXj0zdTsG2~aJpb7rl)nBJ!h~lYxDNH{s`IE^1_x7>W^z9vjiezTQtY%%T zH@Rs?YtccjiZ?WSvpTJrrBXH1wb%DKIF^)`s%BW!Y#ypN(N|&PS#&$=7bm@OQ>k?& zjbzSBTT6PpdUdC{U9sl2isTa04_ykKNZhUq(9ET-muJn@ri}l(UsWx0Q`KrFdPrc! zU%NBSBn%i!P?*>fm)Eib1V0EmeZarn&h<75gphGLLRV} zw<6bK;*TR5pyMR;y&YTQNu)M!zg&aP(5NK4PvU?_( zdxv5A3K6=bX~tYgjkxaDI~?BGo3z!%=sBPBzIi?8b$)$~5C&K5SihRe@&B;CzVYJXiIxN&p2v(*_W2s93 z|CDpK=-woy5pWq_D>e?@;>Mbhrv--CL~{&n2u3Nh15pMGFf#|Aa517)C`h|b(9aNdHa_=4%59nRt z&SCETX!W##1H;$guS)d*Cb72eL_;Dnn$PznOiPp@c_{M)lVez{k@1rvXaVe}m;)z; z^-2YZ2|qfseaLqE-&}xj)fmW@H>cU->IZ#M4@@a#xghu)L=X4}+Q;evOu5L5((CpC ziB2$;>n=m>)FWKPrthzTwqjR~Yfl4>#0TF{^P1g&3GEql4ut6KrJ@ig6;m|0*GUOWi2tDRd)x^xMX5vE+j;C3SH`t!A~RYb6)Wf}3DN(zKXdIUM%Y zNanksr-S2IMi8`qOiByJgps(WbU4oiGy7Z4C~oSsMAf)|JcE7F4g~i?Yz$B96>JSKJ6T zV3zh|k4gJUo!ieUPlyiL?mVbRNJT-11CqD;@#4L0wp*Pcm-jJ_3F;=2Kt5Kk zP~<5onK{)ftj9bmbFNV}!}*k-ShnRcchFfmMm3-j#DrNS1HP`_2>diuJ6hh&e0}yP zxxkPsM>>JTFLRq{ToC5I-ZT|-hxghlO?N#-S56%5!xqn$ zn!$nPfl#T_5NbRWi%c&^u7{~P>T%=2-*zz_9ZOFnxk=v(w@Q1PX7}Ecp45JvOJ=U% zEBi*1Ib5v;P)5PGMC~N&qc$Hi_c=c`z`oE!njPD9C#0vlU$9{$ zv`0z%SiGn5z`B6d-+XCvg70r?;U9VAKJ~iZpGMF%j`q>)acnJ@KKOML(saEfSE`|A zgv?a5&@Nz@+&jK-fA||Q>|UDkd^uEUyndP=Yya=lk=e>hz*})kq(9KNBbv=h^jp6pvdB)q$jH0t9I88~cMM8- z!uhQbaAj>@FH2$advU1g5FTcse8Ubyw->mTV(Z2`#`ZeMhe7}cT_PxAz zo2kjp0IdMf!QYyR$NFW0Y6rETYLLyVuvK;PAWqP zVgJH3guICm>>MsqPvm+31~7ngH+pSwdUba*L|Dns&;Wf^m)yga|C4C@E5)QM0+D0i zH1>{Xpv7LES+rrX?6Sl`$AKY(InU8*_rT?Fa;((gkR$kyzfLF|834(SZ z#O5E!<(?~5HN{!4d(j$TSDBoqGnEGZPxpE|+rjYizM7JHg%o*M>{Qs$g%{hzKx@K< z3eabUty{pf?aFo|xJ047 zdX71{r&FpP??E&Cf%melr#_=Bg~Ho_QOL3*NM+j z(guL>MzKW2l@%w4d#y&ZoQ%J=f$}>SRTryqI|j4Y*>3T&*x7y;cr&0P!V2w5NrQ1m zwTJt>PfxFfjbDB4n-O_4m{!k)IZUNpJJ;WGzhxD-O%pj2a2)$seO1IMfs|^D2|4V8 z)`i_Fz;Ag)#2QJ}d|GY$__;`nyOXWLow4>qo2p0FLd<@l-AJOao^4D<)py0P5ieLZl>vuPG5 z1Kwk0&3s3m-(>M8D^DukhlDx36;^_BD2Sn_oMFzm+}~^SQc>GAm*?_C1PCmc)Gf{T143E`EopIzOy>p z_de?Vlk<9EKkERk&+GmEH1bHO$NT%keQQ6f$Mblv_C;9hz$a=KzI(--rqy;B6j!Fn zCvkdTZ@8OVt#d)Losb4=1m2aflg-90xQd|<)7dE9n*-9(5nvd{FzMts;n&*fS>cHG zVP2zDU1PsYr<;`INmhhwY=b67Toy617grPV6ZJa{F<|(izIboJGus0*KZo(V?)~-F zbx+If|CNqbS;%+>qDh+j;%S&{b5Njav=#K=MBM%yx*LFlil+3sF}f`62)~7m28_>9 zAOvp2ICBW-r`|h`wy{@9rGy=N{ZSJ6a$b#Xv2gi)V=|5Ho3wp-I~6_PfMcmO$?BqY zt|~Ac(!9EO)1loF@|d6G5nE-nos71Tz}Iw~kl918XT|60b?HF4y9T)B)S&ZQ+Uwi` z3SHMYV`Xr69VSvT&kf9)9&75l&zNDNhXeWxL-JKT9~#uh;&9>=<9aTWqht6t*zpk#6|8vrS~F}fSqYpT z#6lx#LC;_4((~&L(p_!>2#O}d6Mz6VbHp(snoZ+O zIH3nb0pw-j{G^M9b^YBmr*5sFvE!Yo1uiPDE6BsQ=(Z#93>O+yJ>btQLmijr<|sp-f4il~4;~b(s=~`s$Powzi)NRN8*pKqh7VkWvfYsCrCqdjQ1 zV4!W}<230|bq#x%&*}bz#TPTAQHOD7Wxf75_jwx4i=EHm$OsPdQ7nB`%({-d1~zMX zZLFJOB7vZ0Axe=9vE8DYnz;anN{f#23lXH2V))X`= zH-IujG80fld!ppuq|H=>nt0lK5Uo?r2w2T+UGL~-u>`73n5vngg4x48 z39l@-0A}eJ7UrXG7G5XlEZt&9pSvq7pPZcUbHKk@z3lg3&(tZ`YGpRA+eW@aa8;7Zkg%= zx&`^m^So3$aE{cvKA)dyXM-^L-hiB4E-FbiTD)v9&)Ix}BGI4>W3A`5EF~wainr%TzuckCiVO9H#-Tcn;HT zhD?sf5;8%101p!K+6%$#h)C4+#mMxLD#D$}^i)|-1q!2sZoW)ZrbxBuL&ijf-8Fjn zZ_!C`Z#422nsU65j&l1-LQmb81`fU4G5^@ws8_q7`k8x1np^D~+@_gqJN_M;Z`B&# zP9NW~mBnNAZ6#jaT$Rd=^M|zHwru*or(kYfzxTigmv`HOj?RkZ*0RaEjc76bI@LDT z&G6-0?NgU`S91T?^9S*?!m|h#8ae(IV`N8u1OQ2`FRhVt*AF9rWML2yK!O1vQENL) z=EZ2}9E(5J7{18<&Ho={5N(W%tB1`DHsMxU0@}>cJlwU2_gUBsYP~|-lA~c4nW^qw zuo0J-x`JlmfM(RvI&({n-#hW6=hhiLWpCZV@uJzxU5uuuA$Mw(o$^hHDTa^H(ryfH zS3jw5xSNA!IdS~%YpT2&b+6WlYeTDK`=ViuVC2~rOSYu;j44z1SRXs?gKZCc>io@E z6?Siwm{;Gv^=-vk4hWjbOMQx?IR*4DY#=B^xT30foK5t6yEaDk#QFOX>o9~-g_;0et{_KE3)*p zALi%(_7LiZy1EaIB%i?(rH2*Ph zOg4w6)_@laj&i|EB|>_q0;(b)EkWa#Zs5XHu_R zg(Aw!a3^vtd4Rxc|HUIy?v~Sq18Mja!AW8r-?Z_3O{hZd+X9koEow#jwU+SiPrNy2G5?!+g}=_!aKna-pW! zv|=`~J`SOam&N3Z&w3w}UE7xtn8zVaipkROBDY1;D!tW^s^_B3oYSJFvExSK zyh5vv6%<_PP8N%mD_`3jwkjk1$(&(p{KoM_t_iOEZCLo*YkhdGx}+bWtW9SHt8`Wroh`xUetpSJKl`OFdc0|n zPtHjgm!>Y-gesaMJ$5BV-i>DS&^De2#WOi?G|P|T?;zgM2%l(lz4;{G>Rla_S3{lM zEf^8USx6LlRYQPy4pjJX%lR z&7a#Lj;%#7&95I#`kb?&mkwCv#@|G(t|AUQMmNmU4;3uE7A3(`Wm61Ey zIU1Z?gEj8VPd6Tt5RhC+n7WP{^+9WG23C5WlxdHZ0#s@_K1_uZY1E>(cE5?v8s&f9 z51w{Aq1)~i`$IXjTzQ-u-fPOr>-c0lEq^tQG$@sqhjK6nvPztDbeE+LRZcTaBQS!h zN;KLRt;~7qSks0T33USooR}tBo|WUaJl0RdFu5j!rZkRF`FK z1PDgH9FPq+VYBtiK4$KaoI85*r}%XdKiSnBxA%PFy!12SB`-!2Xu&_aWxPrXKZ=oS zBZ48?x$hRr**UAz$5MCeBv6U%A-(g1li^EZ+e=BXA0|M3B^ha zB2kVp?rh-Jk$@mwPLgWAYT`@hM}2?jO`n6i_HO(0tYY!th_;jVP{4k$9VUzaTIb=; z^q*tq8V!G>O%uXIEr%(td zj+1*ky*smg%{;h6WYyU~krvgKh?A5H#J-p_#&pu7xel7u30t6ICSO@%;go$sF#kpC8;1f{_%weP6@!Q>P1;o*IJ;^A4j5_q$5wm!dqeA@TE z|K#Nm`rqmCd=m2W`QKmB5a}B@;ad%ODUFLzZ9ovR0D@0E=I{_R4+WihEaD)i@AElt zw4J-#|Eci&-eT^^i>$QG(G%aG~FpSL%{zbigBb=6L$MT+V2*a zd@D5iQEKx42c%XI$u7f_o`uIh{=<8Zy#I$m-ah_c9mmZ77Sa8ue%=hH&K5?WDS$au z2z4$Wf4}^{1Q7h6to(NsGb1c^THyDo(C0h$k7voA@S^R10{!=b ze)m6|0B|qg4$Hjq&a?~7?9?4n~3ii4!t0cE{3-QH(%&ikBOH4L_{x6#$X42 z^fWy)KuPB|2x03iGj$S63OY4wtJT?MrXUrr%7_w#)YpAVgjmT*X8QZ(6TM~w8e{w> z6tRDEX9Oquy^`B7)sp)wWSgr0$Zp93!4cDWg5LYOHj~6P3lQV@I2o_+Z|DFnzr4>o zV$98rVboo6AWQ%nP&_GW(c#(|Okx#gi=GEq=OExYYF$Y*43h!W%6jo+dK~Mx$&TPk z(-_@74inpm4B6lclpbV%HN=;T{(vx+*)3O8=N#65wM+6*$YMUsXMqFH-iMUu?f8`c zRmuw3mxZC_Rx6>ni4-rB_Z2jW1^I`N-Pkah*Ql7QTjYOl{xiz$aJU;EPIX@>mg zUOpUB2Rtki;ZVtOdtV8cD*j4V*BX4kC(kpY=!JXY5FtVO=Vj~jcz$z$(%0pNHxsB9 zm3ps1PlDBhNqY1RJ#VOv40AX-nMx-K8S~CF;ndj&D4KEWTX_#Y-=SaB31}ab=%7k} zd?FmS&*#mcOvDGa7z76|M}4->_oT<`{wl*g=Ep+*$-~M8g{<&jc1mD6_CqU)WHB6p z89$)hfv*7EeJ6R+D9RK2zkQK^AHvPZd*~R+a}^P{zgK_jl449!(}bB9h_TNTVq7G| zxc-phCnX|`ON5&g|Nr49Dl*a@0`vhDmsFFNHK>>jBNcSx@|0u77<;e(hD_aT8E!UM z^KR^80O5y|b%oEZmlOz9(gI;BN;{Z5v4Ag!9;~*Qk%Q^>;H4Q}?8Pn#!htd-w3xrV zPGZD!c}MqMHm4jszXi^vv2kVM9|8*nFN(@MR%{c)6BEgA&k1z7<%>B*mlt zmTb|GLr$zzUZy)rXfOO$#2ZB)P6{KJ827qeG3)3*ZvXQaZ|3LeaMGM%=4+rlPj+A{ zx6lVm9%AVJsv3*Gj>DQlS?5sEc;H<%hE&bGV1X9q;LlBHcbO-Dnx;{4BJu5ig_=}R zf4INKpBUY9pN)~c(Aj!lVz1CZ3II&#N_=ixt_EEbN1M6)@6Q7F359@7yc+i<6SQGV*>57c`+52m8l#1v zdtT8wB+t|NNQX_y}wQYXzCVWBj9>EMD|&@L?yMuZj%EawDzNXBaF*%>^Z%Qp^i$py>Br#5N)l| z+S)t!i}N>ZsdsyNu{gU(L4_{vG$n{odf?5^9Tv<63d)Ln13G%iig z@fDLcQABZ3MUyDGa7&W!``IGF@g9|BQpOtF(hcV=AU-A&hw|+FoVRfiyG%_sZ2#Q1 zF)v0*${AH$da&Tnl8}1GuKjj|SK}cQ7NzN8mIO|Q-6NvQq39b?!>pAjuR%jQ{X>+3 zVnYMn5c|@jEq#;=DsGknjaUX=S4*2A|8LA)$dhPC-{N=P3!}a3UZ0M-K#;te-GqJrvgrKIG_2C zozE**DfY2FPD8psH%i>-GZ@Io~5YZ~y$bd$41B;wsuCUeSv1i4Kv=sRJrFVn7 zAyzVPGU1-_3%*@4CvtUXGNY~~fgHwPo<7VzmV_AXHv%hYj&D`Vd77_Q612s7dJ4c| zc_>{%2GtdBxsXVRRrD^L`BYP#YNz3&lT{^k4jCj){Sk6RQX>fpi9mQ#m{#P6TTKI! z$iOe<*|+iP^16V2c1nvjFRbKxV?q)Y=v5~?TKHe6ty4G$x+O${aXSVoE!2i$hIA$| z9-`ypYD%7o+47i5LW*@U*aKST2l^9rneSYhPSk>!ekk>N|D5BSjIiE0Lo8pF%&2)k zBvC<=CjhIZLXYenlP28g8wm`&zr;+Yt_6B?ZA6yA@5aKB88f~yPANa`=41D5P{a{6 z96vegj0m4`z9T=ao5nN$YALQGKNR@atL47WAAyVTEyj1~4Blo+E229qBhM}f>!Iokp1=h0MCR|4sg43Ix)?e$hO{|Jy#Fs3CJX_!T+8tKmS1J|j#=)C4QkV#)7e(Ft74C&o$+YIWiwz5PC6@r&^D6AgbJL2F6xCFvvr z2tM))!(a82&Wxy#=P$8IRDkA(DpUwLK?D{$euUzPdr!$G zl?sM4xG+2)?!$(PeG)~&qUWDkYF@s!Ue)t6;V`E5iNMBnpYQ<%A%!buekjWELpwVY zbet6Jow8tN=&OlW9A%doNSF`SS#-(PvG1THLVJ$9{M9fX7py_^4+(`C5Zw4i4$&$0 zsY!u(7Eg&T3F?O&%7P?RRt$tmpN&Z)1T9+=94vz=M)con`Vp>_Z};n0whZ~fdPYPS z=?4#nL`Ls4{-_EPc7wk2NA^C$Ziq@8VDNY&a=mUUTwUH6i)RhtSyJ*PS>I_ULJm;< zMTW*3WlH>C0dyXV;Z_Ic;R~GpAir`wI;)?%`EdNc8?P^Pm_nu-1j_uemMns3H>9?9 z4Xk3?7igh=OiBDoL_U}>Al5#nNy`A}81xCoKc-1DKVVvL5q=Ws07}|sEc_FzbyQOL zQ5Z@2F`4yuA|ED5tBCWDsC-*RSY#2v|34&^2@ia{p{_&u8N6o{56+f4YOjT78mwOf zf|-4)%L{#J7~|2WxjLXH699I8n#%*_UWQ;MvQKk$A=ZVe@BzeesFm0o}{(iD~lLCE=1 z<2&2ZwPi!<4~Lq`vT<8?F#4B0Yx+0q;16`Sb=Y3kzW9)RDR9t%q&% z?Yyf^iwYVk{?_)s_N95}s=rki-#VWg9sKSX<<=?Heq8y~`b%7KC!ORCsym&AS#4oh zMO7ty1BMmnAA1Rzi7;ytYskzeu@uHsB>ATu7qttGwERt4H#XZ~*53@K5#xdftc6Eo z$L1CejS$gdOAajr3)nt5bUsh;WZk0;8gEuPb10WNbKQk^8=qdEZ{M5U?f9*~!@%n> z=tzV<6JSbMO{whOO1D(|$YbMpki+Rb$-0X{TM&_5JIXIb8sha zx=!j%s^P}yC8EJL_t!lq0vA2+n@Yk0FW*qt+o|o#uG&=~EjL*gk3da}m{E}`d+F2Y zNWrwyqO-hM?$=Mp%fKsu5XRSIx7BOES_e z-w+S+kLyZxKe5chgIGS`VTp%gLp;<%N6Puv?}dm_AP;zWz{3I$<%W1@OejJocYueX zI^f{}4+}gDH^M`Xoz6LbNbMrG&-{V)2;VRh8L2;=^>8k=VxN9}3#?Z9Ohn#Xi!)5!!VNy z?8Gm&S;F?b#bBQ!Wj4v}<3_73Gy529;)6=;YmLNLbNYB&vh}tpYFdV&AA*$P51@_b z-P#XMR_m}hE&P5dFYJ(Na0qFXgr+tLX#gG~hl=*!Nn}koRjB-c2P`n*?`z$L+7Bci zP4tPW^a1` zyq9{W^da-U~1Yw~weevW=}YB|l=r;(|Q>oJ1K2bHM2ZU=2U`zSJ; ze+&#M6h3D`5DD>sQKyT{2FP%f%%@XeKsOuUQmFhWk|v1wCViel$-`h%9x_CN<%TY62Y+uh;@1L)7I;2|3EcS<>vh?Y zEHGO_!!Z7!-KctNH|mBP4~CABQtL=>LI?(9Bim*{1WY|>N)StXZ|>v4x|@q_3Nr0W zE`y>bY%N*|c_g>bT9N!A!%J<C2pZ2~)HtHnNkpM?DJ$`HZ*Vyu&kJW{I>=*KQ8|&p=v_PI%-u~tzbl7Ynug>mWXNLYqighy+HOpCZ|2jkMJ$B}Tfja2 zcrct#vqh}5m+4?zrCM|PE_8{{Ehc_xX+tiWnO}@#CYiCu&-I#>EU7# zMTWgNdFxr!#@IJ(q&FX>+0>)KlX(iTh$mOM7w6t}{IBu!)SC?F@OdyAPvPR*+yh-7 zW-vnED~%6C7&8ZbHonH+d%y{T##1CJ@9Fc-Fdk)IJOf3&M_5NY;2*W&Rv4Ht zGBsQZ{6O6kkw^(-5mdw=@p_b>kb@`o>KTe2McZu-vLMQ99q^AUN zJTNhDdgHOZdlX-jP$_vi-4RdYYa5~{s19ic?;B5WWFUqiNTJ!UfFGkYo~FMd#{U}6 zXJaCy$zYTdCXw72yvsaCku;9O)j^J!?;Qi$yPwhebsof%qYoru#_zIGT*Ph?pa8(7 zZjS(SHNTFDb1nSz?}V2Fg#gOtS7XF&padXUZvym#knsTYNMgBnd}+};n|PmlMS#c~ zj>q`jN*|D2DQpS?K0A5hJsrmLH1nQu-1 z(Ve$?J1l0_7#A;9n*6@M%0}g#wHKANRHLp2^%3qCY5;i;#>hfb;3vdTF*;O2?TLqr z0<@u#vh<0EZw2_`_QO9OZg^&AjP%e&-e7lsGA-u(#y0rFD4dg51%LR(>MYBwW%2NG zyl+7(gs`{^6@SH)`WbWKFoSTka7Yw!v+1k|`q5(_K9DjJ6Gj6n-2T1=EiXv;KS*6_ zOl95I7lPmF1q3$Gu3r^X^u6m=6%g3tpZhI zPC+GeUW=EU-`HFsvs4XDt6eFKPB)Yi{T zsC~Y?e(r}%aB3#L!TRGc?Zq-lk}&MGtsl2F3?ott@4x;ehBbCcGN8ySZ5mWr?Qf*+ zFKivb3@!-tpUE(*Kf{=0q%`+Qy16~QEuxPP+lV|9R$c)tDvf?a+l6$Dnn|2nf+9P{ z(UiTj3-YO(6Av=NXLZ;JjSWJZz>$ISrIMO5MGD{mY1d$Llq9L{%YgTjgea;4rH}=2 z5cd@*2D}wYcp#BbYCk}dVZ>JeNl>DF1xEaSFk(?;HU(1D01}vmrUgcf`6db?>M6So zj3gI95CuFHO18jA2F4ehN8CI-N|IDUvpXmW9&QLE4eD{7k|NjvKU{(m$7b<75PVB0 zaR8InS;9gVwL&TIjWJ=kA0R11=+_3MkWO0I9gH|PBH9p2q47-(N|6vz3n_7coKcE| z#x_6_It?p))k>j6zSJFpNy&|A2^dYCmLwi*VwS)p6sSy? zd9W>*GM*-!1w3P{4NM%5`Qnnv{ea0RV^)bJlEk}H9e^Q;O~q2+2Q?_kNVEY-N0cdbQOsWk|hz3#$z8^S+imiqdsm;3s6D)XBh9`Vg0}`MWZ_yo;^aY;I zvNbJHZQ@+fVC8A z5pApE1v&_O{iLs=xK%D89$;Uk{g_KKxP_LC+^O~uM!}{GFK8&@B9OdA#WQq5DPwv| z7=ba%`T(Ug=?85vGNCWD2?Dbx7)6w?)6Va9tB8ksQ@2AX8wwteM9@M<5tU5}wyjK( zrWdOu0)tj$tuO-JpK)!Xy}(EZGFSy8-eh<8bvu9-%QuBm(14PRg2<1Sp(Ic$g=Smd zP)`Ti^a5#k(oRW0sR7f$emr;*Pc2#nrC`_2p-oLCp&KG8#6+}ZB^=L?20h1YYtX8+ z7sp{7B&OX`LP`xoMRH*uFp5_ChQi%~NN&pO@VwE6F$?x>3l#w`3Iet*fck-C{WN2` zABC*|0tDb-keG)Dpm4V|BsbN*1z$3eOpE6(3<6&p6DqkahOuz02)MS{);#Ko*oYqM8_wEw22kud6H4xhk`!MV@Y}Ig7up^un{d0R zDUdi~^)igOJzs8Hft2<|7WT6siHv9^B|Od^uy#+}PH?pnMtrw0!X}`bQj%)=893I3 zECWgUAdezCC3!2~cc%I((@Mc`Z9qaKz&87PvX0d1aG%_5+?=yb-F4hFnVsUTybeHV zgwflANTx|oBs}Vaeb5RbK!H$#X1*T~X)gHhqcwyq+T1>hnh9LO50*iM=PU5-y>)@& zJTP2geq1Yv5CtNekh2dE>6Kc8*{#yRf^c)iuAbL`hQe}?FXVx*n9L$E;ga&*nrjydP%T2kw$I~=(pi8O1nL$ygW|i*Juy;jN=ace6e|L5bPKnW%nv~I4z^AaN|avjmuKt_67cSFC&L@yOdn5wuHN`M6+V`5J>lF_#~ z?LxKdINug|T#aly!DLHzJ(ER4VaiJV}Cv*Ah8b z8Z-y3w$8N^I^-;qvY)i`JET;`gJ?g23Ii(9N+{VN-44MNhGO#+@Ti#ru6c`h$VhCB zX!y2itDrapfzK@0fW}t(vr#~<6zHb++?k^6#1w*@I7_g2$fk~o1}K^58onW z(Qpp2s71e1pqPhIsN}X~6In~=Y3v%X z1Sf2VVM4xjXh{d5ih4XYGBB-h(pc=NIo}h<(>7w>uH@~wb&boL-Ka%Clbks4pO$1v zCAqJ;;X-d+sMHIM?xzu#t&$1pHNb|gd&*1^!Rq9F5$(8YR)w4Uhh(z=y(KYRNh&8Z za$9Orpgdq4Hjt*(VxlCnNb5c6c&KEqbhN5nd&C(Rn}ey|%OR8zlgj}nh1IRnSZ~)> zjZp%gvM`deAMjQ>45nfTRm=_*V>Ds zzV64GMQv$bD#DH)Qp!CM6RdA_W~;hthk`W9E-j+;nEn?^0flCzaXEjpeKfZJ(hT^n1~l&QY%>3+cC zBy9x|pnwrV>=_wuT|nE90+A>~wG^!o0t~=k z_hi%RTU@Q2?^qs%Y>T|(Ms_$DOD%N7LSG4x`rG=$BLO2NQ7m|C&kc(MM#z1!IupoM z+MNn^TvmpK&YRachs|_JE?SEJ441w!EHYcG;b|oL{X}RcL@S`MOaNBE_tocfb()uE zJFYEbO!3WYVwxG^g12aR27BFzh_S6(JA}P3>G9A6K^vTa{qmYSH9Jm^au@T@IxL0BMxp7fs5yyWVf$u4>L zEqPwB_p#S;-?2?Fmk|)IQ57 z*llChjm;+2Oq7x<=SFibqkfZ9tHTBs1t#)v}eCkkjIc5qq+4Z*h#PAlRCig0=vL zl^oOYFtcYX;rWEqqh-Kd$3GWOtrykcF!FNp3cxe+vC}r&`gDsdN&$5JeowUtEW!8p+$~=4i0|p_ z7``%Qs3Wygu!(?LCEgQk;Zn~~Awwj>CfFo4kWgB0?ZQYk$FvNSvVKe6r9}(aqHsT9 zM5BY1ymH6RTD8rIKh2&i;d0Y{bd_NpK(Mthu$QL&Q0pj-!xpuJP(Z364AkE00M_k} zd{3Q2ZG8BJZfb_yFt!x)fGw0DRfO8Q|5pN>O~zS|$0A+l+p^rZU-E0U{OfeKp^`1wFoeyTG}R6yp@j^#HFFKX z*CYyLQmebB~2$DZl|h3Vkj_$hKAM(okfu6)6jJ z+6E*U_<;%RLCbppl3D3^5Y6t$>b{M?jM_8+ufm2~4p69%aa zR1#&aK*Gd86Yiw#rV60}+oN z9=`+1k9Mqsw^7oqzHiw)qMI{~#VpHE3jI(ry>0nzn#BnQ15+;ADG88(W3lpnD2elx zPzrUhQy@i~GCMpbwtk8_Q>Mis6yhOBqF5_w8q)&Vn?JW0tlCC{V|jLaB!qLXzQb?Ig9a^V{gA zKmtIwusV=ZQJ}Xj$p~dHjCy^b#jw@e0Gw1AuJ=^k#6_^$+aQ?OF(E}8_s#Hcy?Suq&7KQ5GShs>n!T8wD>Y>RcZg$J_!f1CRy{ld4JI&J zhYwA05Ru%i%RxqxZR!{bS;JV;SbEa}CK39WavE)YdaX=!9OziGwkkAhFNexv?!$6~ zvv9YXgRm)U8+J!-KuLg-)ea@Y4O82<{W6)(gpQ(qtL~scVTp{?56#D_f=KLKGHvR2 zkd3r(5I$zPu1SYLb0HYaTgT8-Oy$pdfl_fR$3q2@$#7qXoT=3(Gx26ymGIsik8j?I zve{@H6Iz#nr2NQ4Y-_P3GovJzv1t$dg`w~{w(ZHy%S;&=^`ka0F>Kl&?d$B#TchaB7RmRdB~VmCy*Zf7(kW+A zOEP$fCqx?&ZQbcGj4~0i9*g6ozarZ~MO-WP(8viiTeed;Y0;Kze7llon!eW1$d<&R z5Zk>kSG)^K7yp+Xh)!`8fg^to&Ju&*8~Y-KZRt;KH12~7Bn;S@G~;tZREF4L0=K_vuv z%D1H_-pjDzO~!SrQykTO%1hZz&I5eNU~gPaQ9jEibS&N_%+sG{?*`fXZaU~Y_M1(L z;pvy7MmCjV5o^gGr5%x>;MI~ydQ){vM)TpYQ47@Eqk07^IZ}(z@o%*e(`+)H&iwvh z6c4-k{Q5ec-Yo3}-csKS)nWqRHiUF6`)1qFm%T6vf_^{lw-jO9-Bm%b{pmGt;Qv1S M|5G9x23V#705Cm=a{vGU literal 0 HcmV?d00001 diff --git a/eval/dataset/arms/copilot_v11_deep_r2/pr4926.cost.json b/eval/dataset/arms/copilot_v11_deep_r2/pr4926.cost.json new file mode 100644 index 000000000..bc2d45cdd --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_r2/pr4926.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 1860.5, + "split": "train", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v11_deep_r2/runs/pr4926/run-20260813-130951-783171", + "recorded_at": "2026-08-13T13:40:49", + "attempts": [ + { + "invocation_id": "inv-95c41c43", + "outcome": "run:run-20260813-130951-783171", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-130951-783171", + "llm_calls": 117, + "input_tokens": 582597, + "output_tokens": 203494, + "usd": 0.569528, + "source": "spans" + } + ], + "attempt_usd_total": 0.569528, + "llm_calls": 117, + "input_tokens": 582597, + "output_tokens": 203494, + "usd": 0.569528, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-130951-783171", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.5695, + "minutes": 30.97, + "ci_minutes": 0, + "input_tokens": 582597, + "output_tokens": 203494, + "tool_calls": 210, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.9281, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 117, + "by_role": { + "agent": { + "usd": 0.539063, + "input_tokens": 560541, + "output_tokens": 181244, + "calls": 115 + }, + "reducer": { + "usd": 0.030465, + "input_tokens": 22056, + "output_tokens": 22250, + "calls": 2 + } + }, + "cache_read_tokens": 6977152, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 3.78, + "gate": 1.6, + "review": 1852.57, + "report": 0.06 + }, + "time_to_first_result_s": 1858.0 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4926.trace.json.gz", + "trace_bytes": 35479 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v11_deep_r2/pr4926.md b/eval/dataset/arms/copilot_v11_deep_r2/pr4926.md new file mode 100644 index 000000000..dca986c36 --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_r2/pr4926.md @@ -0,0 +1,46 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4926, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head 35aa741e1962) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 5 finding(s) below | +| Security | no finding reported | +| Docs / comments | 3 finding(s) below | +| Behavior / compatibility | no finding reported | +| Correctness | 2 finding(s) below | + +**Validated:** +- [validated] flash_attn_hub.py:47,114,293 indexed accesses `out[0]`: all guarded by `isinstance(out, tuple)`; FA2 returns tensor, FA3 returns (out, lse); first element is the output and `_run_varlen_dense` reshapes afterward. +- [validated] flash_attn_hub.py:108/287 both-funcs-None branch: raises RuntimeError; constructor guarantees at least one of flash_attn_func/flash_attn_varlen_func is loaded. +- [sweep] flash_attn_hub.py:117/296 `attn_func is not None` branch: piecewise wrapper falls back to `_run_varlen_dense` only when varlen func exists; consistent with constructor. +- [sweep] flash_attn_hub.py:88/267 `if backend_kwargs:` branch only warns; benign. +- [sweep] flash_attn_hub.py:189/368 full_attn_spans and 205/384 attention_mask branches mirror local flash_attn.py; read piecewise_attn.py and utils/fa.py; mask contract is 2D bool/int, same as existing backend. +- [sweep] cuda/platform.py hub branches 124/129/134/140/142: read full get_diffusion_attn_backend_cls; found the HUB names skip the `flash_attn_supported`/compute-capability gate (review comment 1). +- [sweep] musa/npu/rocm/xpu platform.py hub fallback branches: each routes HUB names to FLASH_ATTN and then existing checks; xpu lacks a selected-backend FLASH_ATTN support check but that is pre-existing. +- [validated] registry.py enum additions resolve to the new classes; _DiffusionBackendEnumMeta handles unknown names. + +`vllm_omni/platforms/cuda/platform.py:124` [major] — In vllm_omni/platforms/cuda/platform.py:124-127, a successful `importlib.import_module("kernels")` leaves `backend_upper` as FLASH_ATTN_HUB with no compute-capability check. The subsequent local FLASH_ATTN gate (lines 149-158) only runs for `backend_upper == "FLASH_ATTN"`, so FLASH_ATTN_HUB reaches the unconditional enum lookup at line 175. On a pre-Ampere GPU (capability < 8.0) with `kernels` installed, this selects hub FA2 and later fails to load/run instead of falling back like local FLASH_ATTN does to TORCH_SDPA. Gate FLASH_ATTN_HUB on `compute_supported` (or fall back to TORCH_SDPA) before the enum lookup. (evidence: Read vllm_omni/platforms/cuda/platform.py: lines 61-66 (FA requires sm_80+), 124-147 (new HUB branch), 149-158 (local FLASH_ATTN fallback to TORCH_SDPA), 175 (unconditional enum lookup).) + +`tests/diffusion/attention/test_kernels_hub.py:58` [major] — The PR adds `tests/diffusion/attention/test_kernels_hub.py` with `@hardware_test(res={"cuda": "L4"})` on `test_kernels_hub_execution` (line 57, def line 58), but no Buildkite CUDA step collects `tests/diffusion/attention/` (the diffusion steps enumerate offloader/cache/batching/distributed/models and the non-recursive `tests/diffusion/*.py` glob), so the new hub backends have no automated FA2 execution in CI. The CPU step deselects it via `-m 'core_model and cpu'`, and the FA3 branch skips at line 92 (`if major < 9`) on L4 (sm_89), so FA3 would also not execute on that lane. Add a CUDA Buildkite step that collects `tests/diffusion/attention/`, and run the FA3 case on an H100/SM90 queue or change the marker/queue to satisfy the SM90 requirement. (evidence: Read .buildkite/test-ready.yml lines 13, 64, 80, 96, 112, 129, 145; grep tests/diffusion/attention in .buildkite returned no matches; read tests/diffusion/attention/test_kernels_hub.py lines 57 and 92.) + +`tests/diffusion/attention/test_kernels_hub.py:75` [minor] — The execution test only builds AttentionMetadata(attn_mask=None) at test_kernels_hub.py:75 and passes that same object to both Hub implementations, so full_attn_spans remains None and attn_mask never contains a False entry. As a result, the new _forward_varlen_masked and piecewise_attn branches in flash_attn_hub.py's forward_cuda are never executed. Add a masked 2D bool/int case (at least one False) and a mixed causal/full full_attn_spans case, each compared against the SDPA reference, since these varlen/masked and piecewise paths are the most likely to expose Hub-kernel signature or wrapper-shape divergence. (evidence: Read flash_attn_hub.py lines 128-161, 189-202; test file lines 75-86.) + +`vllm_omni/diffusion/attention/backends/flash_attn_hub.py:108` [minor] — FlashAttentionHubBackend.supports_attention_mask() returns True (flash_attn_hub.py:55-56), but __init__ only raises when both flash_attn_func and flash_attn_varlen_func are None (lines 108-109). The masked path calls self.flash_attn_varlen_func unconditionally (line 147), so a Hub module exposing only the dense flash_attn_func will crash with TypeError on the first masked request. Require flash_attn_varlen_func at construction or derive supports_attention_mask() from the loaded functions. The same issue applies to FlashAttention3HubImpl (lines 287 and 326). (evidence: Read flash_attn_hub.py lines 108-109, 55-56, 128-147, 205-211.) + +`vllm_omni/platforms/cuda/platform.py:126` [minor] — The CUDA selector gates Hub backends solely on `importlib.import_module("kernels")` (platform.py:126), while `FlashAttentionHubImpl.__init__` and `FlashAttention3HubImpl.__init__` require `from kernels import get_kernel` and call `get_kernel(..., version=...)` (flash_attn_hub.py:92, :271). A `kernels` install that imports but lacks `get_kernel` passes selection and then raises during backend impl construction rather than falling back to FLASH_ATTN as documented. Recommend probing for a callable `kernels.get_kernel` (and version compatibility) in the selector, and adding a lower bound such as `kernels>=0.14.1` (pyproject.toml:70 currently only has `kernels<0.15.0` in dev). (evidence: Read vllm_omni/platforms/cuda/platform.py line 126; vllm_omni/diffusion/attention/backends/flash_attn_hub.py lines 92-103; pyproject.toml line 70; docs/user_guide/diffusion/attention_backends.md install section.) + +`vllm_omni/diffusion/attention/backends/flash_attn_hub.py:108` [minor] — The `FlashAttentionHubImpl.__init__` retry chain (flash_attn_hub.py:95-108) catches failures for `version=1` and `version=2`, but the final default `get_kernel('kernels-community/flash-attn2')` at line 103 has no try/except; the same pattern repeats for `FlashAttention3HubImpl`. The CUDA selector (`cuda/platform.py:124-139`) only falls back on `ImportError` of the `kernels` package, so with `kernels` installed but Hub artifacts uncached and `HF_HUB_OFFLINE=1`, model-load raises instead of falling back. Either probe kernel availability at selection time or document the offline/uncached behavior. (evidence: vllm_omni/diffusion/attention/backends/flash_attn_hub.py:80-108; vllm_omni/platforms/cuda/platform.py:124-139; vllm_omni/diffusion/attention/layer.py:100-110.) + +**Additional observations (beyond the comment budget):** +- `tests/diffusion/attention/test_kernels_hub.py:70` [minor] In test_kernels_hub_execution (tests/diffusion/attention/test_kernels_hub.py:69-71), k = q.clone() and v = q.clone() make q/k/v identical, so the SDPA-vs-hub comparison cannot detect q/k/v operand swaps (any permutation +- `vllm_omni/diffusion/attention/layer.py:100` [minor] Could you confirm whether `kernels.get_kernel` caches internally across calls? Each `Attention` module constructs one impl via `self.attn_impl_cls(...)` (layer.py:95), and `FlashAttentionHubImpl.__init__` / `FlashAttenti +- `pyproject.toml:70` [minor] pyproject.toml:70 declares `kernels<0.15.0` only in the `dev` extra (no lower bound, no runtime optional extra), yet the new FLASH_ATTN_HUB/FLASH_ATTN_3_HUB backends import `kernels` at runtime and the docs instruct `pip +- `vllm_omni/platforms/cuda/platform.py:141` [minor] `vllm_omni/platforms/cuda/platform.py:141` gates `FLASH_ATTN_3_HUB` with `compute_capability.major >= 9`, so Blackwell sm_100/103/120/121 (and future major 11/12 variants) pass the gate whenever `kernels` is installed. T + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v11_deep_r2/pr4926.trace.json.gz b/eval/dataset/arms/copilot_v11_deep_r2/pr4926.trace.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..135e57dae020212449580c60cd8aca24b2e854b7 GIT binary patch literal 35479 zcmV)UK(N0biwFP!00002|LnclcH2m{Ao?q@l+{b5%>fe!P^Ic}ddT!DPbJ%@x|iie zf&?g`L;^HGN>*k1tbXbrxczva?=Rf<`zhy3?j9mAb5c4ptt{3#RTeP>c0}xXil2_g zvv3?A4a84h8sf+gT<HT7pR&&@#N1(VSAgA07m4`zPg9c8W#WB*@XJiPc9uLR#82(x2Y z9onR&7X>t4r)TTdm*m=Tc^P@}WjMooMyE@Qvb-G5odpf`rypOxe2U++I@7*8o*0(p zb{(rjU;ooTUZ4H#?VCrB=?P(!OdJAHX*1Tf;nr{&ITNq-<9shP-rOdGI}Fs%stB z?2h%0V@<|dzti)qzSlAJe$TQst!wt2X;1H(x~6%q-Rn(yW2^7jwqf?iuBqdMQm8sw zr)%s6)mn%=bJEegX>V$|w&m%z+tFOugvoU58a-_c3-0KBW88;f`_9Nya0r*f@x|wrrq_tvDUT6X2;PD)98*( z$FsX#%^Jh1>Ag;W(lIS#YEJrA&vnKCmD9J}zH5yO`=%Q`bH{B122}x$+qZGkrcQUP znHHdsZ8&wPyG`nl``mmHRuG#BMU_(3lv;&)*2M6|9>+c4SV#7n{ z0@!qmnd8v)r_%~3VJ&)qC7NmKuGa;sg;gBuz5dt&42Q+E-D%%30rxF!s&`>Kr=B*o zt*+}#0N-4_w2?akdG`L*@a*ZE*Jtlqb2kr?F|j9l*X>M=o-ua2Q+HzOPIqe9_5@hd z1lQ5A$DV6;ydL1)c%pfR+qXPz;^<}vcupRqes4#Ro;-i_!>iYC-#vf)wgi*Y(F{$8 z^#KlP+ue?j7>}HLKG(`tp$^-OMMp z29)jxQp(L)h@l`NRv1q%y*Y{zo$g1TlwcY8>-x?`0KyDt6JP#Lyaiz?GFQ)o^WeMh z#ME0(F6G}B&eA&%n&QC&{A&`co}UO;&|ySS1|$`c%B;%bN5#Bxf9*u&6i$u;kiQZh=~)p zz6+y?NjRSa6~qEQ6AE1uSKeYNrXasTs>TAI6CL5MU}Glmc-#{20FFy<=1Ks>#6vk6 zi!dM%-<%!8Ta)F=nTg4z6I^%$y8ih6lSd-nOg1;FRUqNcTb^#(l4P=*3In$F%kg4BQ^yGIY57FkK%O1f;cqD-Bj=lEW51QAo82Bmp$}$Yajptm6X&BM-kKR02zzKi8n8}3# zKvyw>-)-xf9C{8+6JVq1!EOoHi@YiPH<)qk@ZT^Py7J^IwXYF`7@v~>f6#8K!u94M zjtkfw_)8cFVSM!RN&D5C_qgS`Ww`;OgY^!BsUN{+UI-_Kt@WDTXCL?m{Knl~5wS#} zyPCkbwN6_%bDx3T3Kuvxoc#)B1%K9COpS`LkQ_f8f4FvLM27B1F`YT_C1`caV0gJ2 zw-&bptJlzVt6|utAR_}tcP6kPxW1!swfvxc91Q?NXX0YzpiYQ8Ir3va0Db^?QiEq; z-C(J_`sf6f-T<0iuA%@i6J%f*(RL))_;FCS>nA|}5$reY5h4VkGORw_2co3)8x2uI zWCR4w9R%t~pc?Hga3QC_5N-gX7P7+4bJVqd+vxYiI9y&f5tHL4;r2L!8C(to$N`-0 z8K8$h_nt;k7@gq$0N-CBC!wjsW_$SErI>jRz>Nch)9lx9&m{Hten#B_k?Mf)I}qkW;$2iGVSA5s_0Yh@b#Hs@4uoD3Ad_%~_%*tc!WR z@E2~yjnDY0=D!8zKCEuH2S_oc&wGJ8yn>y%hzAN@`Dq6*;6#kQz`qE#twY~#o83Nu zIGgbeG*DSo7+R-o_AS^7gck6Ah|EHM3R#DbJ1Dt#Iy4M>B2e(9i>IzLOIiA}xIJ~~ zIT{g`0IdPSP2&lmPdIA>Q8q*jdJIx+La}n*xk*h{uG3!3oFy>kIerADmIA387BDLt z$Qu5&H2AMEVCZnl;=)^!vbaJv&kLZ|b~1xSoQ5j^83_g%9og+9KUiX9)-zr$P{@1k zsJ%|!)s&EBDJ}GRZXWYh?6g6zXh-2>-u}E;iQ=RZPOfG*5FW?|5dkp+Tq%Trn5jsn zMPDL1Cd&x#f^a?&_z{-VP9i~@a{k#7WR&$i# zJ|a?#{3V=SdkCqL1ru0IU=Jr^_(W~U_YBHYc=x5ZbU=y+?#x5^autL(0nZjcdw!%= z5e3gsN^*(h>AJDt3xg}~7Q`IkM|>N%=D_`7*bm3@z2lFk$ORfwXo++qS>P;SEHF2K zkx#=DfZvaRnFH(*$v~3CyfQzA#4>;}H~)ehTLIEKKgw);1;_*2AUdBr{{#yAv#>`} zYr(}rKA&nqq*RbH1N6eH>3Xo1s0@IDa|x{bhLi*kz{2;U32|Wglm`kRsA5BO zv;tzyN^dBJ9k+`@Y29vwn#<-!h+kqp`4xzFX{))IQVfd}Q4oNha~>1{5)W4(vJeC%Qj_ z7$?iy{4q=0@S;EUJ*IDYhvo)UPa40x1wfMg*lVFgGbZtLmv;w);>Drbvth!#nsaeH z2_tVv^C$5HJnk3;O!C!ce7*k5l_LQ59*brZ{YNv98({q>50L1JR@14}2JXrb#V%wc zcAy2Z3tfqQ(fe*&w$pa}@)2o!YAEqRrX%vkWDrVKV0K&o`ro5Je$+MXm`y^`<pwmda*|Edy;lLMW6;F&VPAX5_Q5nUrn0K{Ww@HbglC}Jp=az?sm!?H=n_&U z5pOwuBb{z`lJc6g+G6!3eBhBnh^7IsNsy5Ffi6duN^O~fj(T^AOqEH4zDroU<7bc5 z`)a)7IxD^J15ow=M06YtUq5^H;`ytmym*PDtGrjr?G-l?FI@{QH~xF191GIWVXsow zuK>bj;am`Ccwc&n2sd!SlmqGeRP$ASBtH-t4>IWEw_sQ2tR7N6ecZW=)EkG=${Ei* zo*91AxrQO>65eaFLJEd%oZ`kvdK*tig6w*9zL6U!ohtHS0BaEfOGQLb`>eHetM!@p ziJZ15gTej|_>PW?FUf5T8xOC=SrQ~4nTxs&UcK}$E~S9vWuV|U-a)H~?*!iG<#= z0@?^iNQ(^Q`ij2_B+LHDK#>>vYyCi)KFuf`ui|BZlfy47x!WWkG0gC@$37kOF{0B4 zUYG9S)GIl20_l=OqYQwZJANR2h(khxA#OEuKSYlqvOC1iulwTyu(;~q-}@`;-k1S} z4Z)z?I@-Dc?(}fwzmia7coA41q5V(CuD4iT9uLGZdXq?#g%HBjQ$we+=R*QXJJ&1d=6XiAIjN8 zOVJd6bRYcoIet?}1YJG4Xh{AsW$tteWr{;*cGk@ITzlLmM!`Gz!0t4b?C9)u-?=E2(ab~#2NE&1r zFoVTiYQSnuvN;<(GGmkz+?Hc5u;4Jw08B`h9?MH0*^5gOAd1{hMS}K;O`O6154i>V zI8u>@UZ+lS1XzHeC<|EjayPiqG2IS1<`M0`bZ^Ou0YGE#)poPZ(CBK3(_boFMp+sJ~0 zNH9V~XH^IFD6-0cnWM>wuS$Cr6#+~@6{tfY$44~+)?Wz{TK;#h&mR9ceEa0hqftwo zAtNnt`(&@+D#@d@>>WU7m~x}CCQz{vIEQDLlZ<*L%v)NO$R}rEc(q!z7%zeFV1Z(p zr-nv&I1bSPlTqr29HjxwG_E*W?^^}#fR-bTzy7fivfYfcQ`rsy`cNoB zU1ym{A5R95aa2Dj23hYES!D~P2qT79M~Z*QY%yXZ@oJon@~U;El_BO8q5VjSV<_~C z1CNyXlLTxNW@fTFK3d~NPoQDhs+@V*;6Cr$k^u!Pq9vYQ1L=dXlY8d5z{ylle2DLW zN*JI1bbf@`d=7|ne)PgRKl)N1f$q18^I>=}M#!h$LWw;}0Jja`t*ElA@H`23j<{nY ztI}So0`o2L%3D$}-B~TeIr5YYic&HG<{B_lHfSyWv40zJozng}0J9I6wa1kwoW1~Ni_0?$bWgcwqMV)x**r7n7Ii#K;f0L>ZHMt5nHHd!?lp~;cprP@ zT>@Pb=cQ6+N%eO<$`3|E4}~t%8?!uI{i|!<0N^yqW=JmNpiz#7dR-;?pbTe+TI9z_ zvnu--`1>UC=YBv2)mPGA?m`(|yV>vla;h+Pql&IwqzZK6Z=wp)zOjl8Y$j)jc`$&> ziQfNG-l*|ou=JD^w#85Z(3|nXo1mLjrG3{%L*T^dzf(>sz5(K_!Wm@rn|fF0?CaF? zRT~w&L)9<$ZLIgGa}sw$dI?y!DX2XpvL|E-N#G0` z%K2ull2dX`2=i+hN0UTuGT(sYKGQR+*qOD-M^ySOkHGkk_Q!>T76wrT=_GK|k?K5W zw9msBrS+A^Ayn7DMt23M{mUEAV-bkqRLM4C{{rZTQ^}?CDe(zNFXY!Et#W3B6nTsZ zAmMmdPo%yIv?QPCb-oILnY$5Kud^?KNjC#%YtvT2cZ@n9`K57niw_i>$@pBsvp`<# zoD&;e7hrmF(mpD_Rk81B6wYCnFD^h2e&$SQ$4T|VPq9;soQ)g&5g^KPk!3HUiic4( z_#=d7?8&L3U!2PjMup`VN!K)!>_{4@@i6c9t<^_~f$%O`c?oLf>r0TPM;q33B$G6O z5TpacLbn!*T_OSY&Ts+*+UZI1B>C1WV85Xhg-HaLB@=lE3Ic7*Kpf-T@%OO^$C%=! z9ApS62E<27JLVXPoQur80{z{mkg8NE*DtiQOg9D=EY(^!ujUku0*=rt0m{LP3{I%P zFMBMqQ3W48CrN1kSLX;=8?1qx>vWn}+)Nz8>NPM*&@^;?jlsTOUY6nt){aKJ*1ro| z_*FcRv+&{q_%g;`Fj!daj66TeG60opLV&o1bQs*;^P|lIzZ4H3iVtPh2POa@_ME+b zJWoOiiJv;=Y=+Ur<#6Itj`4_Wwl`la5<@%aAiDO%1m zFY%SPp9f2H&rxQ=!ilBlJ(&&B{FHo8SqDktl1p;UWr_&u#7NY@>j0Oedx7~aP_I{+ zA`x@0NSYmy)=MT?5=6#R|1;fO?)*w}{7N<&*oVDqXLpwEV!IujXng>MG$Y z;gghOMI1mn&fukpbw1EemXV{fx@O@G?A4g!#I;dAnOx=qT%y(`f%&u6(mO3}q(10+ z*V*xcFgsvFa?T{9ZghIeNp++8Zy6nsE>fd=a`%*LRwHZ!vFlqBK2Aq0EdO2P$Aqh z8^(Y^%0>fNwIt^YQx(8oN>cBYIi{>l&`~H-%Z80|vuucjrqQ{Uc!Q}|+$Exj31M|C zGs!ik5GnZw6-y>2nvMHP_MKf9Cl>|AzzK6HnX0oBkE)fxQlVO4_~Y;z6v4!eLNmi9 zgB_sVi>0fcz(>^BoxhfNMV7_x6nmw>s%djzHO*eCi|bfs8q2($UraLT#MhwnD7PYa zW~aS8JI#xryM6pA{^u{ch>GHQiqx5wD{ zgK~;vKniq-CQc+y+Lw4E-8L+hoJ`G-AO$n$^kF4bF-?`BgD!7XtU%p|7#Ec**;(>B zGzoK)3>ijq@*)*`yp))3MFOhUB$vxNG+;rg3R(wPngv#wsO4*7JU3Bw0h!ytY`n$t ztFznvv{N~?6R{7!-lhx}7vihB$oji**!}%u#DW`Q$WKsHN15&mFp>lnn|&gPQR;rj*(UuT;xzB$Bu!am-X^P$C?779k(4i*RJ0@`X8HO` zSy3r<*U?|MWU}C^l*>{_2Ihbb;ZmZlt3df@I98rK^y_f}XOeW1xWpMc%+D<_sHB|~ zg{5#o)@O34U_*zLeSG#|ZU;^UZo&41k^PFOZM|uOvAY(m0RmK8izdKNFDQYTVa&kW zNNkp4gc}LvxcOh5ASC#ZwZqQ~Ey8$o-ByG+atMkrmR}i~=GohMsg3u2!k> z06?s&nn#s6rjKeZvk$zXf-z8%ITko=af|8oN~c=iY}6|DZ z7U&S_`NXS3w60LJL8^x%R6fT93x~rn0^$rXq5j&B!hn!~v*~3?bU4E**HjfFjR9;e zH>p8Y#h$9xGfkdl?ej7%?ak{^G@T^A26m!!Vi8!2|xP*Ssua%ie$YcaHNus2B!r5q_P%_WH(2a;!DgHX$i0smn0DR0#I7) zwW)u$w}h|$%afpEz0e{Y`feMS9zU7nUt#D9i8N_YC?jBQ7S!RuRhSLY!$)Q4 zn1l7JC<3^pf0}az=8lV_`xCEH+TL7-Gv=cta%(aRQEwW$jm`?g)S9N<+b%}$OhfWp zL`U*#w7U?0j#~G9TxSjcB9}LE`m2+L+fiewiZ(njQ;gPB9dP5H6 z_ZSLBJjv$tDiIr#Du~8^Zq$?fh$F(LViQjEuQZbdL4T^VR5&=Hw8TqlG!TI#YKW`r zret3f69YLIfl)ef*rzr3f!|ztNy&ZU155JZGf9Wd>$r(G(-x+j)CpvU{ZT2li-KZF zbp$bIM7rcX0}+B5m|!E1t|T+L9_4c*lLyI^9(@`m1h+NU5eSZxIhmoCsb?>}vDXy}43(vU ziX9SKJ_&;j+=VQ;T$bxq3i$x+LeC|bP+|#w9y3kK2_`M^y@zR({LO5Ea@O^z0s}@+30GY%NVoeJURg_aa!ip3Rpyo_|hoJCtqOMZ+Vsy$2@uF+pd6!6p^)-BUo&qb#xGB%eo|tJ~ToxkF8R zhrEu#dRN%iToB#d#qFhkJ+4uRuP6pl!5^lMNe-qo`gE$|PZ@t%w#xvkAf-oI?D|0; zR|R*VRDjsI6(wgHI-H2IrF*3o&xf}5uue30tn5~JimMez>1ROBagQhNpq)0(h7ok_ zb6Yn)r?s43LgA@qi%I9stchvSvAFgfP=%>LdpK4@4dZz{YLEOly)>$9Iz=(l6ap^# z%9LD@cQ4JAz9BkS_33j5n8j!5`BGWnpk9oDP{_r7=3)IACS8WfGRcKF#f6ZuN($%F zFqyURmi0n5X0FU0_n?hB3XMU{s0t&DRWlNm`T<*6QWQ$g=5D8nrIP6|R1g9r%g_vT zOT$FWJyh-uo^AdZn0utyXFI4bnK7RvACw7CxF0GlAjPN#x8)YQ zuVm`1GYn3C$qhYXw$(-I>oYE@ip`Ub3Q5?NGb|{SmQBXU)Sky}s=r(lb!>?rkVTX} zwkE-Xs>`W7-CruZzMN~Zwv;@Fb2pa~9sh$nIHn~mrKAe(6qQZERY^RzGR-enp0cJk z9zM@s|I(G^s4$uvg|fpv1{tUUoH)*v`yS(lTyCAO`87FNY04!|wzj%b zg>3+3x0aP6H>q$2VP4|)3e424KoTNLlGu53UHvIDoP39{QLgr2RC2G5lI)j=rCbv2 zE>w7CYdeVszh-GC&!a6rl`oZ*bRz}u*Dw3rQH&EFRENDpCEU5{*;SBM&Td0@>$gRB zSp~VZVn8N*rJeZ5X;rQsBqUs0)fdU)$y#Nr!}|R!n50rkSet+AV#zIZ#3#BssK#IT zlR;(GZN=GXDJ!J5hP1<(!cN&q6 z6an3VL8WOfpjb)cZW*Qy`jd4>*O7x@E8=&5d&Md5UP*hj=aiT@oO$ft+QWHL%X?_L zb)_e5m0VoHkG;T^BVQAU!Og+L%km;a}`P1>#O?GV!k)r z>73Fflo+b6>%(uVqQ2AxpPZo?RHw%iKWc|`qC}yVFJ7UuDqf$mL=LE8ykQwt>+NNvX?wU{@-`Z}5Tw<$ zwiSP&-bk5k+SW<*e|rY9T@S@`y}bfOQGQKIBoo;Z6jipVgDup<)iU;7tT;^*Oy)xmm6v-xR!2XwF%DAuPGjtcnP~vpt_Qr}OPccQ-*AlnPgMWdsNwud=C3AIin2~ z@+aPs=!AF68_lPTj>**=2|g1qn<)C@x1E5N~| zjpJaTZjN5ECY=?8>j_d3^{cH2Z2Y$JKi5<;le<{(#>@uR~usT4$jSl!-Egne3{3K2$ajs-4Y&;$NqgNyA%}D+KD{5q%G(2@bf~iTNDPT1uvi1C`GHF$ zwZD9Ut8?_ZOk#N~&GJ+F{3MK*ba@Ozh4Nq@M5Q{ zbu7%OI;O34%`XHr&Nj*oZj^^M3=KPKmPR%VqoZX8)=bUn zc4&@-s>2C+`@;+?VWfP`1xCM9)x1kwbt*@I+-PCL_pL=~%@9DTN@dQ<8nQlxO3^N7_^eAUW*r~Q=5ynia z^kao;ivkJr)`qzBGmjs!2sY3!D3$rhg)Jd0047QY!ctLeG-PQ?chC^I;7`+V^cVT3 z5aU`s?1XW_tDv{KoCyqkix-!F(^yE*9i!Fhec@3(z|SECyy0tjz~zvMzSHMbVhEFM zjlv5&jF$H$GocigpCsNZSh1XAwNOWA!CPoh!XrbFHbevJ!T{8S=DX)FpNe;9PoIh! z87F4Pae9{S>2}v(M+6C;$n(_k5^0-H@3Vt>1SjN~Xx^ZQ8PBeW#go@bd>uYPE=+<8 z7Z%?fF1^ou3=R&{Y+j&p+}^iU-+13Q&I4jg!WmDxrZ@K7p3!O{v+YjJsb}@7e57zpE z5F%=cm!Wj|vc-s<(WJ_jW=P;ErwGN{M?XB33EV%6x4@3l0yd*%5XL=!^{(Fp#tK)_ z6IF0OJbR;esnG-Q7>AQBUx31K%TbBjg|xas!Dpoom1t5MR(SRW>il>TN0Q?zi3d#= z*)e7m161h}lp!CQBRZb8@O%K5xzIB=t#)oWGgGh9VvQGVzy_s=1g<=>ELFdLc=KLm z%>VHBN9~t?f20y*;!@8kwWZv151@Ydb2%OS=VIBQQEobxYSbXDBCuSj?v;Wt)L0CJ zTsy+ld_3@V227fgNGrZ*uKA|9(>K=^ADtA__A7GUxoZV3A8#Q4|99Vs$Ea6ctRli4 zNqst#pKdhFE)w7?V3je}WHk%h?#mpo#B6~t5ttk<%V2`9!s~=`%F4z<0Y78ab6=ts zeJO`E=5f~cS7D+UEJ?(;A3vd4{o~a&;L!6xT)%knviW)*_>BZfQUuSTOC_$*u1?ZZ zid(c+4`Hpc8}v7sG=S!eV{>18sY9>4P4C$xP^66_hegmPc|t?c_E|RDAG& za0o_z@Icg3nq5&aH~Y~iNgEa-fzaCQ@)5qSSdl83EVbPeBvPKEWE%PefByPBD86cSj9q!y5dF1=YumcsGUVyn$qxD zyA`#pgRV}+MykzgEPLH5rdZL+?9bH6tWmY2o0(=S-LLn^$N1-4*(@iU!zAgH{A06l z3{!dh;yI=UUr@DUrjHOg0boJn(w2QEo>n{uH!$%uT8bynpFMm3_WA2q2;kFK@9?kh zA3grt(^pSU3pMu)b^-(ZNb>juatBZRnWEf(0ZlVui)m1!;(TH`tU?^2mP9=LjD+vS zN|<~no=CN*hGim6DyRa)&*`=M9XB=J@A>l}34P)u6BU*Ctb|7gAdLmJe<=5z8uO~DQvh(`-_Bs7hd zmOl3k0Dl}VZX>i%-+@&y5XnzI#v4-sF@Piq38_%X*beW!N@co@(;wJLD9+vHQlQKe zDE;^ePKlu2njGu|wcLrWRE|`xFiSn26q&Y?)oP{vOA=9n9sxdxoX09vPV70JIsjXZ zxABDEJ;$aq!{oTA6;RWq(hx~at$I|(bLs9--Q|Ua|NpBj!NLV?rELvMcNQzSRCy?> z8gi)Ip|yD!TzR)-d?Y1aFX`GFyoS9t9HGc^J*F|n%J|L9qP4)P#Bg2>NT$Wl#|@F`<=!CAc0K!9YFb5%!5rk8yoLWMPP37@cYaja$duwx$wQ>(a|g8A%$bi}2QJ0xNXnt_ zo@#^A>}vgn(CT$HPJIKc#hEoESaSX2p&v{`+*$tkK>T!mlyBIL!6_^&`BKlL8&FRn zN|aY4GVS=IYiZikNd#M3`dyOvrZ?q6E? z+LrUsTNx7Om0MocHm-&FEv<(6t*?h__K$%aslK8@D@RitU$U=Lmf*nF@h8?0<<6S8 z6Hm*3qJGcz0EaH^!)5`y%fZlmQtkwvj8FCYIvJcjm^>%mD6D>qxnlk#ITHf=66A8^ zij{+uj?qy0KiEkMZF}-WB;nOoPJS|*Oxj2D&swGGA#V=lvA+B|{z%>)V-fpO@20q% z98e3-^yM{mY%pBvTJoAaUKlQ!MuI=;^kHoIn-0mJnmfVDnGIvlb8D8N!1VU1v6||?%FlLs}%bgCil`qRkAWQ{bN+!^f7+Omf1Gqx^=XLsL`Qy2DGvvCh z%N`ZAqK~z{O--p#$O}Xyb(qSu)QleF5XH*YyK5u=hZUXtAO2>o{C6-8?nrfJ1=*SF>CXHn2+vF?&-_(M&-^8{XMQE(Gk+2F znZJ7aAng~G^oL!Lq&LeDKqpPU4e=j6!rc_@j7yj(nY2J;Enp8kN{Vm#jSj&lp<95 zyXq_O|8hl6vx`w1QaAD8Ff~iij3`y1sKGPc8`Q);t-R=VAPM{()eJNe({Rm=YiG9d zs&73|e=uVlQ1=#fEyqy@f~*0WA@!ioj5a9g_zj^900Ov$*J||-1LLCtR^A4n`KGRz z&twH=z$ffAI860_EVjMyYFa~Ni~#Xa=ymZxNJ9X#k51*^tnf?D87xgBF$owR{!#JC ztKB|%mHFh=zMo754mNjA_c$x|KFPO$TOeX07RBa<7aRWfgQieW9*uzUWKfz=_SQ-W+z=1&KHdGG4?d=Z%gWU{!=_f`RtYf91dED? zQUH%vC^VVu06z39rU>%aj59Q`@26s2vz7_Q`hhZWtnAd~JPTvz&zg9cJ!~S0nhz5_ zOCCIdokBhUa(}EC|3lYm2%8V6hyTITYK%_z<8Gfsw#c6V=3g;**`c}HxI3Jikvlfy z;l^__a@y>A93T}oCzg`kCw4E3)U)J{J;Nmy0 zt}~w%KXc~(DgBdubO9Qj|9OC$B~Ho3mA(53MwvOclsSn{tQ|)A_^6!)d=neOu7CQ$ z13L7F_Y_}97y<6BJX$ihqOGjID*3zk5U&f?I?*4++qq>$FG==y@!@ppDEg8gna47X zuI02dH>=v+SEH-&FTlLa+|_}|Ehiy99hbPwGyPf;$9Hwo53Xc-I8|m$*&5<0{Ua;?yy8bI;3!YFaMY=)$I;1>^yZ7C zj)UA|YJ0tU#lT--l16dd%843xo|ciDmQ0mRCS`opSrzcS^6Xj~kphG1XD}R6crF|2 zT{m9#sM{Te%{*QaNdCI#?LPc%58T}@nun_Z8!O7FD(gZuk{4OC0z?5RavyS*ivW86 zxYh(=j)nzs*b+=iaaIb^XWpRRw5mH`o-?QVAn1tewMyY(l`_AYof6MQ_KU2MKo_4H z;tGfhv`Fa~B2x||8FS>Eh6*3d?Xjw{E$ttJNjqP;j>#ys^z-V2_40hzTAw>0dZX~@ zr{aaH;)Uzt1!)0D5d~xe*eC_@Q6np$>%}+oCPHq2yo^@k0WBN>E_T9sY+9p;1_XMh@;qZk^Y!C6+PKgQ&2sr{CIF zpx?3|VOVw~tZ^m`%ie^NMmR(zK3E#@!GoVFREw0;46>Z2QTb?rrwl51O7X=n zMQmTYJ?jJ_KUKxGQ;bR;%b{!ry)nx_|IJs6gsguGI*sUHDg4 zwB?(KmxeCh-}wVMZSq}oo9Wn{%qFk9l|HpzgOiq$>y1}v-@m6Kf%F4p@24wHMMvcZ z>bPKevp3l`X@bbeO|62PG0(|qtW``e8SS*n5GlS<5J4HL%?C-U0IHJtTbPHq#cvX~Fu!u-!rUWtVeS&UFb~0t zL-69)3|^T3meC9I_laJZ{|?a$^A8!lFb~m-uM)j5?``x#g(%X{#UWC0h*TUR6^BU0 zAyRROR2(7|he*XCQgMh>93mBmNW~#iafnp>p&}J`DGF5mp7tsS%)F#Z{`wQcm`sd` z+jCnjr*9Z;uRk5@#g4ak`M_R9g1I-bJg{qE<4yQq>U+!1`;Km2a>qhgSzSV2{05uef!~9>ar%5O80R*5efnPz*FB!&AWI`KxD7 z&-VTXVX+@fy@)?V^65GLuKL=;!)^V<@1giRy`BlGV=07InI;ubt^+91L zk35}hNa%y1t6Tst6UNFI6Zn5WNZ&f69d%^kGa~#rX?2KwQxneu!e5*I-2c`&2g*M*FDEEwQhH6tR>h@Ll#MN)3ZpFx}hUg;y<7dJ){H7m>${tZw;mUfI zyL?S>fIhbFtln~8qRmeD_oXJc%4JTIsc1GIvJ%Z>JqHhiLx2w~y8_+707ueDpD&5oRfgZct z2cp71rjRAuOLQxZM&V?>1CbUVQ&c*(O~W;NwxxkJJGHgW)a{v*es>2dEf2Y#Ov}$A zo%Tsk_78&cx2Dpw*N^A-Ur;*!(u+3Bz0yjr)6)xarNkr{(MM9dj*s@;TQBK7UILq z8N-VMfd{qTe#9?`H>+q7#@=~!9^ilJTq^)1IdCXeL#v~{)e*^?-A;fWz_r5JwI>gU zbbtu{>wo_rb&gf;%&!Xv&t^)2m*;Lp9=b!L3Be&|o;*jX7GsM|Cl)`Py?I^7lYp_zh33or2aTsKkkS{JQ`z#V8V!=^vS#l+dk3)jBuh4t*I zPIO{2p?yPehHU9%rf^z$Uf2@`Bv}i}Cb)F){8!9gb+IXWa$_1fa}Vbgi`vY;@~V$J ztHZaJc&s!Y@+mI;i%Sp$cm&uSk3_`$H;j-aJ|l%S?rq@AQUUsaPzl1KmOlk6 z$<(HEJY#hwdgVKDgyX)iGcXU`J0FmMZAmZ{S6x>ZUikH%7SLPY=bhZJ) zqPI$CYYhaJPZaBtnue%tJDqLamX2xaMyukSuQeYtw|vap@G((KaF2L|Y7qVQrL74k znq0zuPnLLqcRksX*8#}hph-uUC8EE9G)O076i#l>AIBpPb3;te9mg|w0tefV0#A=C zPG5_wj$dO*mOWFgNY@F--^oKQ-prgpU4z|xMCaGzdhsz)h?=9l&dM6`G4B8?VaVAk z$PPyk&Ti?JIw1}I($V^rw zyDDdNT(P?PXg15vVS#i3%o{jCOiMq=z_H&Hp1*qbT4=qd2KJ#b=;-kO;sc%wjq$b! zo){RW{ZTx=Tm`87P~95Xc)j#ryq`4b{~Ii2$UJ_5E$UF;&!CHQ;=vum#tg2ZQI&q z+qP}nw(VWEZDW^h+qSDt{oQ@<>Gu;co-%T+h?OxPX5^UPShxnl&A4^3WuPnGb%O$0 zuiJchf4e=u&%-_+VeSYEyWyRY$zGp;Rk-}RKd61Wn+FR>b0or|GdgvbpwjA5cI=mvd%VzzTSK{;QCd0jx#XFY4}ucoM%2VzK=*uORbhnLiLgtBjIzLP~;(MXII6R`u#H zx}iqY8VoMdhKGL7C+^a5on72KeDf|FoP0=Wc&-MeJ7093(x$ILhtQvqg^-^e+H~l5 zd0RaIu`pafp5VAKu@t+~bYox^#AUZLO?H{FH5^kpmF+SUxEoXTT+5lZVmDUZJvMS& zaM2szQHlJ?%an%laMb)P&cxsa`F@}Xow!1|t~{L_XB&S@mYO$@+e)!zxOumiMvScH zpD=_QZikHI8|4Mxo#0nv!j`9AbF-y`V{y4JvSU@n(%|qf&qtg|zng%PZ_66gr0MF7 z*f3IM^3QbLfUqiGZzrAPci!9P!Z!`MzgK?V{p@FDXPh@<_aa}4%u0{?GnBl>#CNsV zi>#G#>o!nIEZSuKpt9dGe%^>3R)U)yIg1Ni`vin<#*MGZ#Ch#DuQ-%Arr{xEDC6Pu zrnB)wiI0_~&qg7MGc(uNI=3Ro!H1M!oJtMnaLQFZscs37{Kn6194DQRF_3t_k;yh3 zvU7?n8FL#jqcBa?DfQl{&{Y#4&mQ>eossNM{H4IVK`iyRuCa_t?M~O*Rx&Iz=U({D zGGhOhGiEqjgX@V%#o)lb8h$WkdtLE$`+YoCZ543$w*$|5gZ=T_n;9^{o|-Z_^l;$f zz-(;~r`b5g^LYa*9Hw5~JVhHJ3PMJ>naX&D!`DBo!sPkAk{}ax5E7pWWw`PUt6h?N z4Hm0F^Kn}Ce)&97y1TwSQ{t@q+l0a~IAVh4&pDh{f%m)w)A!b+fRhO6NWA7|!zURkRADNTn zB26sZS>xw1ZpF;c>*ILtpwwpE+eZT9Tb-!Q?P_veT@J{)<4WF?cn4GW;vHdi+z~UxKb#HU_#Und7_GT;Ry0tgL|1=Ae5`FshrbU0$&7zMLS>^NO*6*A za`E@{zSY(8kl=|hwVDS;yfSG|>Gw3Uqf8t2PvaO{t)8`**C0N3W}{TNw?>5U&~w@Sj~WI(O3-qEhj@!^{t~6O7(| zqfF|_Vjpya;T)Ezhj5Dv6!(KO)KF39esmj9qX=>=S z&`nm}J@gleLbPIHOk5ru@Nuxy&^zml3MeU(0hVuwADkdy-9(X}@8LJVA&AFuPy(i- zSW(8{EMfFt-O(+S0>*bv9{I2cJ~NYZ6CffQ$ID&{KEcWma@{Qz5`Ky20j5d~RzU53 zbhzb>ry!cI>whp?xxbvvJ6i5t8^4e(l-IL+R3>iI!*GNfra=dG(>T-W*g^Nqv=9O?*nrb-m&HX~zj~TG+;fT~?3%*;F5Q z5dA^MkT)=8(&YIsVJlZ%i{X?GvdG^lQ2eBdcd)D(`hck`K}zh?DgQh6qTg0Ue=L6bxN|*<437=uKY5=u_vB<%O04|4R+rom<(E|81pSY znv`ew%wYr(gK>Mzajw~G^XO%WL-#w8adAL)v>s=0M3x3j>erwXCF=+TQF`rP9lldc zK>rKjAIgPq=V|X=QV5gx$Jp_k)PShmG{bgMi>A2L6@nZt-Vs#RK%br8p29cj&}q)2`*}-V6?|%WuL8)hc zDO1Z(1Zt$p&Qz8d8!(bWCU6tpN1&%g_(GC5R#OfnQTaCMgD_8oL7T1<+9t5twpc zvj#AzE#yCT6X=NEclTZ15;J2U7!^|GmX`}IYAk8K!<_x%^ZA{9`B&o=?xb)D)uPXV zOE^N;yq(R7-UzC5jQXkpxnKrhFmHRVy^ozV$zE_S73(zoN$erS^~vpe(Rz6C^LV62gjR-@_6-F1XG60r6Ji z+oNN>cmrzw76I-$*n;~*j6=jdZJKIK<^#$iBIVbYkPz~p2DjCqlxTkct~#U>QW^&S zkQ))?{eFzbDZx8n^J{2#awBO)+ZoOPS(yU{T;OA_tc0jco&`h;Y5p~jhzVY`@f`x? zTZ&L5+0tzJIkK!0nS8ZA5^L!(@@CH(q;U{QW*KNtf32v`SQ_X%3>f6VQqdfzt*X-a zdenfB&L|b+6quPQVi8-)d(U9aY4&;|bCyZ@`x?+c%p!2bm(_ehq;nP)#E_zLmCR}I z45seDS`4D0j)|Ihg$qioy!>T@LWmQh8Tk8?uh1N?$psmbvFX9aH5%3YLsHD2r?><0 z<4TK~6@RPKmKQ3TtvEBJ3Q_o9pXvaX&0C>c)y-w>{Tb~BEGUtxk-%E1F@aOpC}_h zi3(HP!rLrKf{_r9hDc|Ga$@wf;mCQ{JpVY)E`b|%D7#K>sRRUX=R%DRhxtpZnhe{5cSCW&V0XnPx8(v_5cGkn8nAU7A7~EVt?%4)vqA@>8CR9}BiX{IO3Yi?-*wE(EtZ6T7Or7|)9G#s> z^2mcOUD^8{;D|V-x4Rz;-=`XF2NQqJx9l}o%TVi+;OuK?u*rjyJfHmv;;>?$q*G8fMQU?4oC;^w$S}j|EChA$ zFj8Gsa1gG~>~H62DQ9kWUjtmf@o(C9EQ!0M6z!dCL|YU_eJVy%=0!xMCJ%(ne8A6YVm#stSQ7#vlcSfMbF$|j0j z%rd60s#U44*x~uk3y}!M6#-R23KY%QDb5c^C@19KTGtAx*U##quedjoU0hc^N~buN zAt7zCSi$flEo2il5*0Yfk^N2j=1(q*NbFOByX41qi^ZfU!j3iiNYcpaOL?&ux*~nM zC0kc_?7H&aaJI%Ud?XdC6oq2svy&mzo!o!UHE-Rl{C+n4x)N09pc!cYQZUkV&qQVL z^>@`HES?&mZE{ZW>_!ITf5*=KgQoE$VD2BbBhniGy@>PAQfewvKM$IZNLa>hc}ZE@ zBOMNlcmbTHcBp47nkJu$j5dZHD^T`y33{+?L?vd(>P&_DFc^m<^V znt&rcO}b7;GmKd;^r`q)hepz+Nvo)*N_%52`IpwX3K8J77-~9sk*5q1qj=d1L}?*T z5X^dph*$OXr-(U~I|L)#TAXdOd1OyT8|+=^Y``GCKsiLV>QmJ?qQh%LrpQJiI4zpV zx^?6@ds4|<{HK6Cwp2n_Uo+n!`j2k%;zg(~jhBpescwX+BGi1aLnf8xwx}9%sd5b} zYDmN!*?u76=vz0ab((KcvKe1?9~g(Wa7A|cD498x$a0T5u2g@j9ELVClO#}i$KHN3DE;N54TFghce4+TY_BZVy_Xp^H%#6SEf-;-Ezu8QB_ zAgSzn-e}vGw@K#Xu?yG;M9#JBW;6wkuQ4p`{#_nJ2g=|i{rhpzCt{}E{kk?87XqQK z1U{+%U-E;}FfW8Y^6M9d=q;ZYPGrVAzw~&0^CSs}_;8R8ETR(VdUSOmNgP!_KFrUT zb8v?|2JAr}xH={HRpm0ET5C?H)k!ev0E4ODo@300zjj^yENy4G+q=uyp(@}BM_dca zW8A7vW>~Za2{KGwD#NZj*t>E7@0acd1U##tqbF z$7f~SxZ1{1S|y>BFbrf3O7}2Cz}td0FiNMH{+&Ktl;&paDaWA1cd*V<`(2b>4}WB4 z5e0f6mOjAn55pW&RQmpBu#3tF%O`<=f*pleWNfmJWjrRj@8*GX4!!}*Jq@T6G5V^9 zLi#5}!hfEl$|OY#R`O9`-*dH%>EJ|in7&9vlEiG-1o#ekY8ApifxC){NMVucAn`#e zz))Ldvpn z38UpSbxNoLaR4fm`sJaRaGKex4#3B!aNK&F(sdRZYWO7k1qtQicNHn&ND9K8t= zLnX(A=7+8$@|HCO_h&=x7%+ni@LG5bPNCBP+01(To~vnK$)(TZsBj8v>|*arrG?2T zCyyrE#wcTEcViXF$k{L5&tA+-8k&AYE;~UkwV3!oKw+y}yDJPD5=X*FtJ&qlULjj% zPXcIlZ0qTzwMX)`u>@BdCe8;M8}Z;eMDT{5y^W5&UD5sZP=_$gbFyV zfLyvC9=V?>T}BS75littR^vuzkGLu*L`_&4uxRh3pw zjJC!M1CFIsGAuXzO%lV)zQ8mLXpu&UOqL6raoqDLrh@NJjgEbw+{RWy_<-a}ZoFL* zN>F9aV4%el8OH*vqgPmPlj~l)=vwWEZuuSZ{MK0@Bx0ZOFHU2a8P04}Pyv5uilFvL z#;d42OX|?Sa7sfE=O4a`oy9R_c}B$Cu~M)Hdr<`~U?`QT|4A}@bPk;Jdzm%g1h>e$ zp)^d;0_!xiT|+-^{<>+o+^2|1KA>Xq zfEes9f@&}mM1XJWZAWD6ipKQ|@W{aA2UQa8^sRN8(t`scK$ z-^-3Vdft7tO%-U7U$8IlRWI>#;O}= zJl_(iu^Cto1|=Zh{XwnBQIM(kP7%9ZRi9&dvVNkF0iY&wU-TKxaIKAb8jKA4FsL?i|Z?>RDTFZOB zXwfF*F1gVE8XnqyK#fG;?yym#3AubDY=zWLaW91~kO0cP^MXNifO$;nr56b8H8}NQ z3@u?acdFWSJek$XVgx7XB=6D7M*#9D$s8SojpSux*y%!g$Hs;#yC~lRRg$Q!&r{rJqIxN$; z*A~&3s%$J;%KxQc8b&`1l$v}@4z=`wgiAwwCyBspa-D0|VZLDMhIBDZxfhjTJ1@25q zZb%SA`mSm%;yo0p;nN6YCCTabz1x4{%e>hMZ!Gom`}w>)LsrlVzH8tRW2!cDv{jCJ z3{v!vr-GL}5-8`UNC7Xt&r+Zg>wJ$9-3KPsv)>pLdOjljxNn{4LHTn;g&Y2n<)K0D zxl#LAwJ@`=MJRS3PyBY1Myyfr!zbu_sIw|i8*~X^L`uoUPC|u;gz9%CJd6bRJx?<1 zDe$t%dXW_Wxo4NTny6z*1=%tx- zp3vzsp5wXFIM(Fsc4Hc=(y8ktA?sCWfDz%encK33_1SiifO4g@JWh>n*IY+5^tm~< zDH$Xe9NSV(#HKf((-rwpUPwDa>8|`wM!>hKk<1ULo5{7d2a{o@usm`c5Y^*<5M5o+ z<=>tSvl}@t;o8%^tIV4Ngftt;*e)CNnFYeOzWm60m^C%8c+H1?Uc^H>HekbE<(yca zGJc$v$s{~gdEtjHr{IoBGTgh!c}g%%F0tb9i@+pxR{lw`O5!fJ&&tg>+Ku|g`UF2; zgi@l-`PyVJtx1A5f_sTR1p^wQH^G`idEy1zvdYd6Dyi)$HDWB&Ev7}PT6bO+rAn}v z2AVzOJ@6huA&py)j&X8L4+X&xnNrKMgT$+AK+@ukKzGcb)dd~|iQ*Avah*W*H!|!8 zi?fn<0d+0^v!G@$h6FutI;{3LS!eWdKQ)@tE>7tgPVt!T?)|=BEvyDJtZj zDcHD^vGF|BasIcG$NyKQJ=U@R>QL~1HZh^WV@8GHq5Vfq{r`E@ufVqQ!-juFXmK?= zxc}dZA8|P~aQNTK^;GvxkfBjsE#IfOBV)Dr(OUW62i6nkIxjag!Vr&?H&o zNv;8(8UDv47Nw=R@x|RB!})znR|^M5dL&psY0z+G<^7dm$9ix9YkOMp62O7~n*=B< z=mZ9XP_6O=V1iR!qJ(MP)7ajeq@nhN5W3sSBSiRqDde>Ry<=0_YCBAiAJnbqvEg2GO8M5TP+wRmd|~`NzG7(RdX<+>0-k^l3~#iBn)~LTGs5vMz>AGP zERgfPegAkmfzCOef&lmx#>S_z$O*Aor-DPZd+argv zeDiL?A!6(oiEQ14&KolE<4jtPTdf*)LR1TdfdzxIqmr|`zsCA;j*JkPIZ z6x~l!Mm(d`&zrwLS7+oe$SgUp89ap@KaC<0{fnz$l=sIB%EdRxe$C!~?D&u~dE`~4 z;@$JZd*p?2&kEt462d>mgL{qw^Be~Jzb}8OJn+u|ryThs{)v;L9W(~x+ytOO8s7(G z!42C()zuQ7Vv$KMU>ee74*`2b7{}9bN9IJ1xkP!TR-f`GBsn0HKzd~uNAH5@Aohv6 z$Go+foZ}bYyP+iK>WdPi(%6Rxx~<%5I@qypr_B4)m^nDumR6}w6ezzt}VBs(y)-9a@G`}~u}>5!ZYyMHEsq;pG?n&w{XgE`&(oV(rY zu5=73-%urK-icajlS?IYO@j```YEfS-QL1fb#16Td=X z8%1u+3GFSI4y`fr!XNdO|Kty4W0++w!mI*fqYhBm#;d`!Mf{I5_dIG@p5rlddQ4W+ z+R^L1EuP$Uv*vNIG|-gG;|T|~1iyb5Cx`!YMLXFF7wnqt>9jEAc5Le;m?5Cz7BmcX z(@=5MkjwR72AtqEfHv|&sp%p5F-yw5rL&ff(t?ea!DpZXDz3|G9sG_uTpA&Rskh7o zO{DZCpew(&QD_bd=jp9En(ixdSIOd^>-#AKPf4z+k_UNKJo4m2zF=Z->562d5t6H+ zNj-i`oxEfh4q2e&xvN__)|B;v5x%)A(VN6HyMHIloX~#a8Roap#Es4YJP(6XxpLOR z7_+Xra2^mnfktHLp65V5N^t&0Kn&a`<$eiXxQk|SEPPK~*(5J6YtNWJEqgaoKl_dl zQ1dGHLt%Q0aGoP>%mFW-T3OhuYCK{ZkX+nqqd7V+>Wl%dTTZ#qgFYnEVAWdH-f9ab zQ0BIGwSU*3Sly24_erQ(XLTWi2UEZcKh!+CBhKr8t34_@WO zkT^o`uDo#qx54A}+38E1T4%MStB9uP7iojizvUhI4`=#-->@9igmre4!`IV6n6!x` z5V}%Lwe*;ppi1=p%c<@L613f~rv^vmW*Z3HI7&0&P!BQx)Y8Ta(=KF3n2`veOsrtkMzEUuNWj&N~OCXzezcp z!&08dmmnWtbIY`3;;v}a-6bkyBEeR0gtqtj>abucgqt9~%8>hi#bibX&5NbD=l`{l z(2Z8V`-`t_)DJ+AbodgCI>@dqviX5$)HQus2Gsh|-2>X-Z_lOhAt*Ar zx?lh^9+kEMV!GUMko}G;)Cd=S)|cKR z^Y5_jG+PAM-K}PE#gn#r<;G3=tH-`L@HeCW>RGeH!YXyG*Duq`wya_I_(9Aqj}+d=je=8N^^>;dkWQ# z&zp^Zg;&UPjM)np-UkonmCk8@CGx`v%YD^8mdQ&5Upi;Tmp@XUK9HrTp&gbt62LJQ z6{c9Kr}0NkM4irE`>4Kvv$j|yIyaiHlVsK}TAF-!nx^ccIH?S-E~?@35O23X0IDD( zLrl$69q|1{W`dXNqwN;PM7wdf&;~2Z7o6KCDl#)GJtY`1I9#)p#FpOK;3adkJs0%$;j^mKr_YAkUJe zr*G95Y>{`aw)C+yO6%P0_%WKgY4S8eKdaxsGoWLCp^D1gY^Sv==ciR?Ro5)fb^1JP z%L%R6R6o#CV_PMREqhhhtg*2ye2Z>Yr)^vBi8x?$!CXoru|U_M!9K(tTJLFObbsI2 zi57jKv@8GvY&Fw1F0`*0+xXvJ7!4Hp+>{zvb(HTsZ1utP>h>DB-E_aUOvzRjKJj1m zh(T3t)oXH+X=-+V3`HSWZh$%Ed-bMvtgy)leH4ZWU)Ic>J2%-;O;9j1+Hx*^J0BSM z6gushd=FJL5Qvg@x1KoDE;N0I=j-O^=q+&itv0b|8lwBazC`#ua?%#=+&tc(kG9xq z9JA&1+^mHfvT>iU5rkIdJj1@OJwJ$T_{?@%H&*%qL-u^U79!ee0dIAi?Zh$s0jzp~ zQJHez>4IokI={%YaP8IM)@Xbr=#Ne{aHBogh+6}y%A3^;HT4DtJm&JUde5wBZPJyy z@dF<1@fmZG2ZCWK;_^#g-03m3WFW76kTJ>sX;{?bDAjE>`u-M2^_o~R?9>V}V4d>) z(Lz>Q>cgU~j(a201GO^F4Lpo$pKsiBTcyP+TX=!BoK5NtkvICYXw?iJ++~Nk_1&#`?JLy?Q)rj*0()L#;nwlj4&q!8 z_+rag1>On6Q>U@;Cm(wGLld9IYvKH{iBF#~EF1T;T;m0o(19#<`wMJdspXuo!?u~G zV&rNIT)&f+A?nW;w*4`>j@GsrxL5DU6+_?oH(`tGVGu^Va9oFA!Z+CXWw?q>7PV)8mehWap``r=Hyz?naZJ-~M(ajgP#N0Fqy^bLig~ia{o=P*dm0+N9Rr z#yNBKUytv@M=R3Q(yZkDAI=odiry4;=8y|l=rZ+_`yQMfYW(r`A1~3`v_}fe_kO-J z^?JQpjj?s+88>UAf))U9#`lV_NrVj%NBS-oSYt;%lor&VYH#x|2CkU)69%`>^_tqF zc;28$XJNbGm|TyKBs3n(PnOQVs=Zd}p)|=;2PZUL<@~aRL7{_N^$W|rHkULmKjTE^ zD>THXH_Z=vU!5Dhz33@d9G`Tr>-~K~y=HDup_@ND+b=hwi8yE7l14UtNZwjmhzW=p z4uOERN66Rq9b5wB{eBogwH+0nVF+P|cOx~+%Rs&meEGRU2`W>Vt}1A0Dpe{jt!>cN zrmIue@l0$q01n47 zL<}G?{G?9rzD$jp_Fz|oYW_l^{WH6eF~O2$fEmLSjC|hQ5|$bmU=+TWjK`@?Q?m=4 zs}|H2a#aJt%XxF=jit0M!st)%R|Z(2l`L{Eoor-B{N6mO!5M`B=v0h)aKR}XQ7XZ?-*ka6M}9)OP8tO`Z^&H}9#=`&Tl(xFj+aGXZ#2V!TM{UE zg`3yY#cX)Su(E(Cb|q=uZ6N_pvbj&rluf~Ut`cBtK{XgM6w(4b^`5V47r*#b((I*htenpjn3ml4lI@q1`yVn^jts2Q}Ez2oD!)LFKl@AOawI@<{@uae{Ql(lZo zThKRWHK?4*`Eny!^eCJao_M}tEFLcZ2S>N90d|*|zqGFU8n~dss_YMcM1vRh@N-)K zo#KY2V-|kd|9>M(MJwTwC2v%=5|Z(O=zN3@3jR%`?KbQ!v>E6R;Nt|<=FS()yduu zU(-US+fI`nFdgynF0G)4#Q1xpcm&1jJj6%6&&1Jn!`Eez@D=AohppeO_SJK)>eebf zN$Ar?XI31W#YKsC?MgZYdsi-_fc8#GC-}#ATm)6iR>W@DLEnv2uOE+7Z}q9~4++vw zf4A>^E2^-hGq;!PWUHj1d)knj#3xAvg91{wm_e^UNoXqko({qeXg#uUHbr>0Ew!+F zCqYGz){vXPCrvnu0s^pp1t|Sq)<*c74$=;3eUdPr0#adI7vG{^Apue^^nNFSFOqN$ zNw_9^*f)+xMhY(45Ea67rgOPgsYy1}Nsp`~t`+Y)bFq%4+URtDy)F}TSbCX`G6W#t z57Z%Zk(cB?fpBW=pDu!bee?Wg&Baf@y$|+=sT=a*@&*9v+{W(gG zQKb?+l&7ph5n7-EZnn*B?&u=OtOsxZny=TVvm72cTw#!w-Oj#JdRW! zGe&9M(GXDJEaX4K!ifkBqW2ZICcpwWX0CMvqUyO>*B^D2sjn~eS}q6g?e61JmRi`4 z7Ttx?VPmp2Kwgs=JL*L@u20=EKl(*CB3fjqCkZ5C598vbJ*)b#=a97&`$*vM1K*^! zx~ggGfebZf2`FLc_OtA|B-Y~yx*SBnC0~uE+S* zIF1n#zzGf0c_82RZL(@3F@Vu<#svR?t|y9YcP8!-|C>p~D;16=3kM|aLo^;4j$3cq zg`UZY%_n$9ZlVUI;|3v1RPqmuO23v*g1jab6+kjt>O z*vc))9#<)3&7sJCe+@!v@63@ya|DSz00C}biy?3x%NZ6=sIk(3TCoy>oX;QW-z97U zwL`6Rvp3Xe4@CC6mX|j&i^XHsL}%GQBx^31NJ7KSCN|&F;@I?W3N{+MXT7a^+_oGv z_dIrQJDarA8wJ{VKh7ubRLsXmGc1H|FB(#!^H0UsaPGQsvSR<_eyo4$`v~<+#N0^b zJEQ`!M@oCh$*7o)S0rD(4yrjl3StgRa^LQ;Eqi#d(e|wm%tKP@JE(U@6MSxFnO+)} z#werPLSg$%;r3%|vFHTDE0ldc9_F)}Y1jSa#FR}3DV*5f+HTn~*rIuPtK*uZ{f3qGPG2OCL3>%BJ77=-dEw!ufn2bHuxhIuJJSIC(v=sc-v>onkZeCQe^NHVN^A!h=R-ii@50^E+ zn){A-dnY%CmCYS(9$wIk6*lyjm9&<~u;0!kt=EQ6k|{1AR@un&=WFn1-F>M#sbB$$iE9IMCfQwF&-A0_2w;C4&)+>Zx5gKXA!AX%enDQ+5GKMCTrp%EDN@63jYe4jZ3TcZ1C&f zcI%$Zm%Eos#^F`Z>gTmkalt))RDkLI(cGrNu4 zpZk}euZt-kCzDJMCo{XZrP|)_mEPXhcemb;lc?UGFTV|v!j^WXPN$uojhu0muD1p% zyNRo#rK7=@EW7yyUT+HgbvEs-RZh0v_l}R@`{kV4_MOj3A5`COwkzzKtecviouZvqwALLrxt9q)^E5Auth>rui--KKb(`_ZoAH|)Hd77T z4bty*J6|xpvu~+y8$Y-QJIF4ZhozB;3wmTFpLePcw1t}~H&YGP&aBSoFFn5mWEp!D z7QFBR>h;$LTF3NATQ~R+LJeIVZi$Tmijo!Zf^^?Di4-;`Fu%^XxSs<(-;cPT_pDmK zH#fa6p?ij(MbC7-Yg}og^Y;`91&5fA%ty-Z3jKTF#;tGT_x zVf#e0Y5cO7JunFbeQLd5UwnF>&o{mg^u1qO3wPpvxkO+~w?bAw;~#u2B8tqIWRO>9 zl2+{V@IA^k+{!&X%RSvxy*t~R&Nn@Dd@a1ENk?`&3fn38orxDo%VdA0ZaUj#a;*bh zQ?4n#8Zo~wI=U)b|9O{Pcd|h|AyR)aU zYHBw$$7gz9^Pj}nc34-Alo&ghlI-?9>TMFRpyoWTU-m4o_3!lwD^AJ6@0vY2h`U%9^bB7lR`I63voF*I`>;) z$L|YV@I9wVOt@uga$lNjK2TWe?|oAT%>vJCV}@iBQmOBd&3a8_Gead`MJ3dyN6FJ37J4O|-dw9`3$(8fuyW zGM%?H4Bb{?kMhrnbK|EMmpl~|c3PPp^>aVB3GW&12`z&{E-F@ozf^fastmk>X?=EcL`DktIGXKTs&1`}~ zhtL`d^!8c~1KoC_)uR;nb@{A@NqO+B_1fQQm(L%QuFTt-6iiSnmMu5Q{hU#^+w+H? zrI2&2`3ivxxe>IWr+jSOk8OKeTNYE@y}Vv`PIQKCONM6;mcmN+av7NEwgb*Q{@4R+ zYB|*Tiib?IYxklA5?W@r_}+FoS-8*Jw5~mEv>*vpqS)#AsjOqsYtF=LZHPdxLMQXso#H z_P%(`?Ui!A?O||zw%_wdp#Odd*qmLIMjJo#)EjZgRV-J)DALD&5`A&Bg-Hv`(t>t? zIXc^E%JqV_i=Ht`fyU5zp7#d|1F^-W7DqHSktQ;Mdn<+out}9-%EUrNB~=BiNEIURq3FWD2=ObO2$UTJat92lQW{j^J(J{&Opy-&(%Zmm|3F*D z5^E(nnyWQN28{$6=(*zCU_I@b-q0|?1uU{nDK(t&XDs_O{?>5TvsRv$?ZKsfKMbUKO=Y(CUc z&GcZAh&{x`Di`!YA$Fr19ZjX=$MY*?g|U|(2FELkIV>ZS3Xzi(J2EkqgPMx6U{VZ@ zCT87;+wT)(k4AI21hh~{!LG=q{)awtNemcx3;>iQNOttxY6<#kS`~+GJ=O%^iU1SV z=?2BObP7|kqHri^q81hH%))R2iQ#B6Fe;Qq)P=+$?G?(Y4O@o8F{a4HD4a#zcIb71 zRP_&q&v0omH}i;>%3dd)6!oH5WOt0_L|_2RggD}3GyIj|)Dyk_MA*&QWr)`TOX+si58KHtp#i6SdH?0^DPKNNJDCTNqqLXOO z6ZYUlFRGx^SzLc3(1=e&D0BNQo;+KaqEmZ|cyq^Lf%ghyL~>Xm`(}L2O>moc9%|<@ zO2&xm^M7WIYb2lv2BhiXIvCV2fV5XBhUpbjyw8?Z81fcn(op?fU~Y2U;AFNifo&UC zmb*04(@cyAy|TQ}sT5=BoD%Bm$1*j+@Ic%rpisDZ!N=@@{#n>u?68NR?UoG2@{;3J z5($Hv6e3u(jda>B=5L*3<%z}RVSknpm@*O#X)fXOoCTtVfSN6{&%r@im}{ zT^VC@)J7XHWTuk6PYp-{S!jZ+3eeb+c26`rE!7+#kI-av8!=qM8arfhHxttwN{!q5 z2bSq%dEXwi#pzX`2%WU^r9)m2P0gCBVwR6;2xpnBZYD24!#iTtn0Ue%v)SpOlO!iV zSn4nB%cMgUN)`mJ!mL~5PfVpW`)e#9++t?_&QNVQ3BwgwVRGLWo++xoUe1V}lu$41 z)gw`|kL?<}L)A^Hl8H2<^QbK;jVu$&Rx2#|UnGb1!OXQS4Vu8ugUXkhHh-m}3;+zS z2YqR!3qWzd5a>*(r|GuG(qZp|(xDJt{2mOr5xJD~L@0btatVIX0-z)nqY~kKIu7?c z16O5?3gw@>7T0VMO4xjmBA00zfZuZ|LZ**G8ec3@)vlc%KXW&n-72=UNP*VDKB%Ig zPnI3t-|BGXh?j;~-2ZIGNs<^bSBfOpY;ziw`QyH6cLT-yB>ODS5*p00Q$a!X=3uUc^n{xNVO4LsWqzhUk2U;C-7&319Nat`zAG z2+v80rqHaYIKmxQB7xVEf-h2mBOB}$*l^brOEGyug%3f49{_GWns({Nh#Qs6Q48MR zRGpeXZzAcZyvq};<(t-=GL7 z0ja=j;YdzIaaYti>r>S|9lhX$L|ZIAdq7)VAeF9QX>up5CoRa*`E}c=-FhOQ5j%0j zEw+htA|b2-P>h|AF;l0k3STh|wh1KzmQ0i>v7BL8_2ZAxxt2-iNS?4P5if78OATr@zyfJ#9$Nx`UcyGt9Z?nVq_w z^G8h_wKi^|!bUnT7|{ZRc0eE|2QB*2kC%zH6#uN;)|2qseL&DKlgp4n^bknpdK4qN zz#Mds&U4g|ZoKQbVJ!xo8@-j5vYGS;;;E;osh>6KrH^FH4Eg zD3i9iK#8(zdc^=2>CU|EHUvp8C`7WbcesGf6O0`>FUOC%oaAhQsWI8OX^W085pfc# zR|r;aq;#Mm&3^4Gb zjKZOTN=_7N7z>>F_;4*(3)Hu#d?6 z{F-$Xq_JDA(<|$EyYX3r`-*miAPd&>;Cd;N1)Ynii9wVR0RrubbRHi0Pyx``+$W`Y zbQVCaSt#0?D6%k*^rr8$&4Xz8s z8}kipD-K(?m*_CN$33uo`OF(IsK@gb{zy98G{=}yCmC(3YMCcD z`i_MGaWGURElA>&jsox3*@uH)?nEM&-P`9-f~8H{n6$8RRCruqlm9U`hf%!pYW7_t4zRb$uz5JAVjKWzR=G#W1FrR2hbByQP#Vy|Aor&YQ>SLrNiu@| zw?8n&kf0TZZyYhkQ6n9+P0QM`9_GgZs7+d=YQMvve>ahWpPPcfg>xAbxuK;diy37o9s`en1S%d1A94{g&z;Gmsw-ljNj=3{583KKJ1OC-AZT$6DU zPo-jHZBHsz1w&z}2~L1KrXWQe9KhJq>Ks-&d5pJZ68)`MmyE5~S?jFzq*bv)!&~9E z>rKxl1pzTF<>24$r~I!g=iHP*L9(p`maCUYbVTmp%yCvx+wTYqLv7QP2wXLcdn=ZB zMd6fg*#s6B3_@wHI*umSpuy-Rs!_#Oyv@X>aD;^#n?aPkofA1)HA4DdoyyLo{MZI3 zXOm%26-XkPR<|!N@xY1So)il7Y}L!`NJ9HVvrWN#B>$G|fzwNnOj+I%SEoq3)}0z$+?I4&0ssg8Rz3oquDRnZCW*(9 z9%N+DNYf=EySH;!Ulp{rWZmOw-z{BetjJ$sw0enk>>ygWpI*xi;Ep+T$%~RUU{}w_*d2@i(j>)gGxGBtsGQipWO>uGPA#)YX z^r45%L_-DEzH#F3ye84IB3ji^22#KU1y?UkHk}OFOC&H{O8f=xys%bq33Q`ao{1Dq zJx%*KE1ris%$>Ak9^3NlP(*?kU(>`~^0*OSxJ|!Sr%5l9jVedkF2}{|O8U^Y+R^tr&#h z!&-bqOrybrrM1iEwW8DSUvmAc%$!CXl=0Lh5-} zsS~|)ZA>?!zhM1=aJpwYtca%~!0;jki@{t$g58MGtd7ZDW6?u}qOQ!rl*E9ER^lWA zdg0<2_U*jRlDs-03n+zXvCk9bqElz>) znmG73l!;vaqO8{F-Q8R@-M!ECW7XV!F%db7&wJxcmg@R2xL(-Ye(LXMrMVG$U2zj! zG{2C$P^J?Nj&7y6V!Zb&==_#sm%Tg-7QiiFf4I|G_&mJ!Oufc*qbY^inbo$TUF`$b zIU0{nE-=tqfd+#cXYuISMS5dzW#qS3&ncv-b6g4KNZ$0a%5O52OB|puF`; zTbUjON)IEELF(ILlx~ij+xS{<%b>ducMLHLkrvL0ayUAOJS|OEG$Zdlgh&8rP>j_y zYRNqNx-+W%PD)=(<+dYAia~8HM5$02Vv8z-nA2Tfx zogs~^ZY@hUPToH#NyuC8SU^8rk+?Uu9)?3rEl3s}dzuSU4GnMAR}!i^3z7wmVzw?G zov~BIh3k7$`}-sG4+nuD=E6EH1ZTGvS+KF)mcH4#tsk*<tOOY z(}3#>V&cje0Mt&wMw;D_9GepA5|A6GSuHXxWd{+~BV-$^>+!O2w|RD$J{owaMiBur zQJwuTqzczujKWTM8=})3KS%41%-VHRJ}an^`O49T6Ae_w*+F(tgg6Ma3w%}m)G}2m45a{CQd@ZnFHVTJa*Z9jv zwA!ckGAR7;gqFQ4JeXIRZSSgTyLI)>k|D@?C5ufWuh!lF(n+7_vZ2G|qR?Oh!?Wa0 z`cgR$qlCE_#aL3}Y}Pto6fzlLVO-7FjaTk);mRE`O?64!KQ#Bw6ls_$c}+n$;Ou(Xpq)+3q6 zqtrLCQV2dy8P#>sEf_V#>B(&Go*$QP;T%GgS{t>aPMXy^S=E(@Yj#cU#;i^RVhrK< zPLkB~vO$9jlgDdA|6X8bPx0w=Y;j*Fn1iPRgSk$h>ga zFsDE0Ftm%B(xxFqsfWwWR-}5zTAN9Wwn15cWd1G$Sm%_2To@hN>_(#()Fak7{j?8z zwd5mZ=W|n}nzp+-lu8yiWUZa4X>@BbrvSge*&6+_H6gIj3#XRu`j^j9 zB-YGiJ4;xVgT+TH`FxP}cVfNAAHtfhduJa}@ z9W4?DIX>4!ysXJH6Y8c{hI%O34rHo#sb$4MoDGsdjSZvsyO1q9B%6w+Z^u`?BU-q@ zcyeJ#`kY?uz0Wil8%Dt-{SCd5P?Z`=*>biS#SAJ2U9H0ULNf}+W$j(S>6a(;2_L*= z|CzqKD1}4MvuQc7cIe6w!E4qlB_SZQwDI@?@s2-X$-0AcGma~I{V0j~HK$A=c@3Oh zW%Yr|5-Tc4yizJZ@&5ApBkPY`7A2lMR_h0GIIzn>qy-b`>^PjawABWsDVuDO)kJ_I z1RsvBv5SG0tqoz+$?;vE@qt-GW~!F$jox%@Hr3=7WG6APO=9-ouRWm;ZY^Gp7c?oj zg?l6^OmEo_{o4~jGHyw+Ku3PITVjOT;yb_+#@vz#f%9;5IW5w%&Bhd_^S#Fb-2*_z z;Ay&9*rM?HnG&&RjEwmpyfvt1304A0{W$$`pS1O@nbPE*eTlvJj>MEJW9>rQiz0KK zk^?Du7t@hN)+x0z5Hze5Hw5oS`9Nvm7D+R$lZ!*rud_Op%)qPw7CuOr?UaaX$IQFb zNHJpX*lFP7E8)2qmQ^JuacVi-k|ph5kL8E#x|*)9vj58LtLC67jjKk}CE4jXg=`!h03eB2e@=`@-kw>%cp%t8KPAlnNV(onWQn&^Sekef+W( zk90cGr1q;rERi#Qi9(?$VK!3~H6_c=p~ot#Nh!|v5NIu3!tuN2!m_k$a4f)<2 z405KIRZ>fWIyalCdaJvgrL(RMkkh4B3WzO^Tm&WiLt|UB-_rYs=C;-6Z|pgXyXM== z=DUmfa<}h%Zf>7GzJ2A3-V`L8o`qpkV%iaOK@Zw=+HZQkCD)HvSAB8q?a{$TgmjIB zkcln+gd1_!+}_+h6uZmo^6KLL@#Dwx?)$-Bq_=$0qmUS$WwqO2LFMFa*m47&D*Gp= Zwe813M4l}Vf3Sal`2U`x{BI@O%{ET0_8#vL;CX3N8!g6ERUG=#^;arll?Mc zs>S>U+8LUmi7tLI4d*8lBQaESD>RwF296JQ3{1+qyc)mN#h_s%!|!#6)W0-Ovt3li zyt!qR9$h`OD(3G7R{8VJP)>4XBMvk7I?JY=<03wAKjs1B_`97X`!QK{A z>B_u>bh71|4OKc1WCZlG9eV@z2VX)Kp6%&axL&lOlb?2|(YH~DJj1ZxhMwite)IAE zCe@0ef{%&xFq0gM%H%_|*IV$@m|@1ZY}KaEO~eS#L-QIk|DJFi_8K8Yc@MzmB6;b+ z%ibpHbijudW!mE4m(8#Pk8%Q=j}OCXwJ@uNr7Sq(gMrIZah0&Z*XldSoIwkQ2<&pU z(*`p6&+*OCZ(%9p$AhzeO?XJAonhCRvl4n~>_fFpX0fvPTXa>+TPIms3N7Tig5rA@ zI)VqKWJH#c0a-cp@kiP|QFdEQwjlWKsY50q9jHylQJB$;$t8*)DE?;A6(NADjl1cM zKS`ii*4-UBnKewn4NrIy5Dva~Hjttr7kpt6A&^X|vh3)LnR%Yt7q+>5o zicJV&ffdg)EwnnT3m~^G?DTJuG6BPb=a|@^r!lIF1!} zcYt9mnKm!K_b;ot1;JE&(dX&;?QZ3M8moTyJYCsUX8S=x1G_tTa(3aD-#*{;3jr1Y zcQo;Wc@AJ%MvxtW0~=@tj!C?NLWbCLFyibgBjs5we=%Mj9~kg`_>^6)Xuvo}W7de* zEu`%LaAo2ri{syahAs)RcU_AUF&dp{qKA4#D=^AtfL=0ugIuBq-fvmpRqlTGNQL-4wetiJDVW*a@^>`xZmnPy{FkwcR zU)i%9RLd#xzxhMS?Nd-!*;t$aWM*CHj36)A#) z!aT&Rh$d@N7|ofQTM=u$w2{xAGj5K5*$CLO`a(xZ|0?7yJ^@hy)V z6@kGrZZ)yZ0H{SJWx|QUB=lYHw<@Avq%+BrY?CS-T6&NGBYJFe*3KS1y`e}_w!r8M zSNmJgIY76IMd$)4^jI_4GaPKm#Fd-$0&-ogniwxV_^m7z0c&+!o`ZJ1LuViVTB~z z2#Gn6)QJ`la`wn_35Erl)F8chk8v>&vn)8ywv@aHUg_$NjTe`ainEM(mp zbsmZ~s?9k?)jkD^hS*y%&g`$P(IseVJbOz`7-X42Fdf`HSVHFsxC~#3c0azFzdeya zx_l@zkQ`?ofJO`po3UDrBebuAf5u${4N1n7Kj!UG2|`^a8qBFMZTrW&?~e0{i~Rj~#)d+0yC5jh`qSG2$&tUo|Lp9~Y!AQhyectKIp zvjkQ<(V`4FG5!RY#bSbHn(!QDn;sRY{f9r{&4{(6ok#1jneH?xlJPsNz6g7N8l9R0 zDj`-p2yQ%E>8GaK;S; zyK@VMe|bGo4MRlKg_<`)F%o9*c21(wWKx)nlEl>_K|_GQLJE(V2?1rqg!T+ZQ+LuU zF$o7CQWiqEnQTDeilXKDuJ*;bhCKzLgT6xtkL}CiCV)weGYcahlNg}(Q;n)IG`s== zMtyH2_bHH#X_~~biSc1$5i$ggW^kP1rOPC%C@8@dg!f5ggV*t6$lmG>vAkBG-WNis zlg<+d5)#s()0Mai)?odR`j#TY7xZ5Zh*m>sjo>lq5lHkgMM{z=#L#7)7o9o^k_|eG zIEnXVGJ}We&Qq|wEU8C5ltZvvm#ja=kjq#gjGeiG5K^P?4O&~fYYVfXsbyG)J|QLq zf6>>%8P*JXd@CTqo+53wb6S3+RrKob!2a9k10`_qzHauL7LGmMHJ~3c!|i_ z;2|A(ODx-nDt;b*%Z-Hh-O!fMp*92@9nz4%5?l~yh<#bIflHBJT?GJc=yt14nnkpA zI4IYY&wuMcaRcW>IZ9CWGz~pO{`_+^Dis9M6^ZM&_$1M(Y@`Gal4}d=iNxBN;iy4(!MiI5nX2uzk4>rjoEvZ8EkvO<@XzIa^ zMR+2QAp?Urzpe?K3wE7pR-nY6JDrb4Dur2n8&h`$pMG+d-t~7Fdu$4Mr zFF7S`7F+J(IV>BTKQcC{6^tp2ak==w0qF3ByX#{(jvQ4-t`Iho)Fo<)n0iy` zFj!uGrfp;MC2K@ya@V@x;?<$YWBs+)(rbx{E${sgB=v$zN_c1Bi;uiJvP4xhIDR0b z4*jd3>1Ci2LB-ACnWCkQ{{#X;ShapT6G!e^2hQSd1HUv@X%#waUnM;k>H;ToEU|z_ zAZ{VV`{99}huS5;{VN2&wy%DE6y~llsqUCM!7MkSlPws}I4}4seC$hcLN0!q;+qAJ z-+tKnFFY{%(B>C;vNYwVG+>1=$nXK#2x?J<_8FH+wyHq;C{kM6&-gM7XN8S&gBBpu za92XWFU}6*nv4o|`dcl*z}A(JJSkxPxL#Ywl1J$LI_X;Hf;9J^Qv1xx7I;voLyn58 z5ud`J*BD4C-(!sgB4Yj-X$hU;x)8!5N&(rya7f4x9mXp)MzFHVyGn2ry(S zj)b(VR7LHsr2O11EC5@sOraaxkZm5^VDvReg?Z7x6o|YlY^l8~oNkAm06&Bme=r?} zftpLc#{+(3vR#~ns{#@dtj@|om-xEFB8p~Ly#X#S7miR%2p23pp@guL@^{5#2J-jp zv5HSo^wjzNNnztuxuYp^vH1R(BXR1lm}6G4+*q69doXF6+(2H#KIf6G zJu3lc0eG~guhs0hB1(+kkSjth_xHx&-`{V;qBkV(MJ4M88n8St3@7JamB!J)a9OI} zMLBViYczGbGxSjRQA_ROTScFI+~yi_A|iet0?3PlVo9nCqD{1ds%Ch-Z^0JLBBAIH zFou^8|93&ZkQ`0>APr6}qeYU)^1($-Qn7H!V6=<3HR3huU5{Z_!J<&sjL7@%1+TYb zgGMAm2x#!PW28gA4w;}rNcs57vOl)LJYK)uNngf!aUOl`Cwh! zf`4q@y`6^mw+uMatZN!BFs#1Bj=k&XysmZ0Qf0Jr`ADE{Y>0hhKFrM+H>i1ut%nMpB-~mG@GfeJ6dA{Da%y z5m2;U(O=y3q2y^UGAoDy+Udr9tv?{*876-qCcO$mSdvIj2eTJVH2LV8lV7DWUH(~zmc}F%w(?s;$kfA` zjdd}AC7fIs620;%D zYA}+}8qOS3POE>M+~nEggu^08Uoq7Vph3g&?T#-sZLbDI?ZYQ@RllTJ81ySCF^H@= z!=`mMBl!+OKvTuM#6rR-7bqbr(`E9BtrAwJX{Y74LGj6y!$VBbr<+XJr0W|J4uJr& zRDh0#q>8`bIy2YS~v#Gn?IdI zAbsg9FzN{opy2eG9nl!Ur5Lw>^8q8`OGarsLGZjA1#15v z%KTuG-*Kzv5#Q0+RL7S9PiqkMQ@)f3n6DHTn~t-OG(6pzMZ)il(Bf6)?+_;8>T+$9 zN2fpomQX#aYO|D9LqmYX2J>o}8)A5EOFQSNCu!D1XY?Vmo$#u0?xH)608p@3Qq4SOtg%XqZ-gnjDGhK=3)j(Z2;$VUi$#HLY(SmXfb+xLM=9aXw?-y+fQ z%h*(^#wnCfMyjPg^NUQJ)rpHgm!eRK3PuO;Ql_~jKejH(E+XJeJ%LI%x=;1QNE$)C z`6B9}bSFup?G-h>8)ZB2`*3#ZyZsrvRVaU=zRE)MU=JlVh!IJs4+v_~Zz5t%qKw}M zZNl7JH?E}uLakUQ*8fjL!n{W|i!FUn{33$Xo`g3JjY&$nYLrzb$%xG(+Bm&uJ#BH9 zYgPSmsC|mTkKJQ%LM&8&AX;|}XgCmBs5R0>D0ec*LG7Qo8s*n73dvWYMs}gxq+*WP zbW^f@>CTo5kSQjfk|7En+gqNm(7h=UlmSB=-pVqm?tQuzWqaTL%5s&cYLls2xxS@n zdojwa+<|E9bZv@Xl@;!+EJf4MK*DA&e+;{%IRRB4YfV^+P0E4gsB7hi9ou0t%cQK2 z<>Fc~8%YX*qh?dfFgM_Frd542k7kKe=K1AEoUBZi+8BvMiYk#2j=yhF+$d9Mk*osI zLiG&oCZ(Tw`*faV9fu^@e8jBfjDs zbd^W;&m+%`p$(ESLk!8F{(D^V#_*B~F9;OpIc4Tl=JH|?wiS3kJ24!}1`(l-_BSK+ z!u;b_3xCPG2ej}TocT;!3A|sR;b@^FsXs&&IQ}Hl)x{!i(UJ;DCOH52 zCF^wAKBqX%w4?Q?9k?8Ky>>wrwAg?~YQ~0|i9A8i_WbbSkGYK834@J{_=Yc(xx%^c z({0fZZ7yrm51`@*IQj|6V>RJ7{aI<*;>$fvr5((zlz`#Q#I1=) z)F~h%KKVdy-RyAuvUBJ-S&N*aAedfg7J_nD#?qNEpwvgI96Nv@jv3qZk3>AfGTr2_ zy1H%#^BR7(qBj@E|EJE>x&o`uPEuFCH1H?H9R+dtt>`tdHcW@4a2%B z2z3`*T|0}~0gIL&yFn#CM$V~4%#GdgWVIy)pF61(K!BnXQ%{LC(TSiJCwC$8+-FD) z#$Yv7uN4RP#mZmvdl>29w^SF%2k(a)Tu17F)b_xfLQIh=2nvR_^lyD=n3rPlU zFXQ3_nV_rKM!}rw`96sq;UNz{`SXF^Nv(BTLI%xiqeB%?4yl7wFAj~u2bIzMf5(vm z1a6_f=|MKak@cC1?HBPvZWzdPQxTNl99m6vF|Zst>z+y|QeXZma#d^8wYa&fms|Dpk{Vk31=sxxV=llbN@GYx^{-WK?IXeizFHsBN7{n7j)|we(7rBzC$Gw16KZ zZiZ^sNu@21Ox~&&e%5{tw@(NXziS-iwUUWdi$%uui8YjK*fFCTGtqTzTN|w&*VG)s zA6A}6AEq#fUo^RmgiFcT5n?4#kR!#ch>C&zR@I*e6L+mH!Gxy0z+DgKBe@_z%TRw) zXbc-8GChH#!67v#KY+|sU{N|ci^Y{xU*I`m!;Fiu0;QZ(DuPUAKS~8@4r$ zF_4{hsRC9b_+xr_xx)C#gRTnjLScMAehYO4R9WZ*IIlGy7(Ckndi;~VY6qxs)8@Ku zsv(S|68{x>Pt$E|(5CN5H0;~IZ+*TgajXnQ&<*pcQn;$oloLXR0&206byVanmJZmu zkt^+DNATf9J%ggh5XOxJ4DvC!;O;c#rakR1(I}jLWOlK$7~`%xBG#B0hQEQW&IqSI zX`*oCJqV&oG$}iE2Fz8yE$M|_f3s(fUt^hYKuc%M-tr3pEWqwE<{cLM>K@oGqb%?v zowa(1D6g{E3tRXzwSs0O>PQ6Es7ka1hY7gmCe_4U81Xfi#zQUk>7hG#4;?z>#}1HE z+`YjAiwkgMV9{lja%8`}YipQ{3)RM{{Gj?%ZuOrtD3EJrf;2m-dxese z6#o+y>s!rb0wXmmm%J4INwcq@cRX+~TDPaw1j=Ja#*!$_aWPs z??D28Q*Ih$m_aQKpK2siM&1ziuMCVcj4$J(9+Rj7BDl9Gao1u*UbXv#$z7NAcANDL zaikmvbi04nxN+Y_4kA+VJj{=oVW3O7RqA=*b)WL4Bq%yON90u8F_{g8 zC8~y&0{kLqH4Rld{?aYWtE~A`lH&hP)Eb);*(jHFJI+7|aZH3lc^)d^BB`zS_`aUB zd>=1gwYGT2bEI%ym}7|(PSnWLoR1=#_^Pest>Xz22`GCpI=^nmu{yW`^3gVM#ItY* z(T<49|LNmS838}=%Q8&s9bQH)_tjK1Ff{h)@pge`+>K6d{^+pdPWbiOLjecB(1WTW9pibm0hmT)twc|cd+$T>F;J%c)>sPvVk+1)Kx?z2Ofc!PNM~`};4yx_8!&1}Ia1s-JY)Je@q}>SLI^ zu3lV~xQO{l>nmwoOJAy9%PI7awIcZVMXvstxw-uZ#SUrkPeT3jn~qXC!#K5@fDnuc z`cfClY0rSve{9oOU$^#M(j9$O-ARo@*$Pj%@9oZ~3k$qPE1NyAq9@2cmMFR56Az z+@ES5M#oxDsbSQWiDRH)bWy-t zFVtr9J`s$UNsv}1SA{drw*=2%?e2OWBtF@l{R12@ud`qqs&cCaJ)zavF`4UYvw26Q zT*KL;`gNl|mQq}@A8}ctRX4WNk=`vE4qoF&tQ_1bu*HF9xDyU9`~8;5(XhTgA{!1 zeQQ{!EqcPQOLp5qI9fTLqa9u)tdHGwoXVNw4#gZTSNC##BVTs4mtHA!#zgKXMIu`C z@MsyJrY10@gNJdW(eFG;@;V{>XgXZ^EKjLF-I>sQ%opy<+AHy6Z>KSw&Gq~8d$Vz% zHN0>Oud0dbQWCVHfNq$41iRUZFYW#@0K>^N#gP5tjZtT_+G~HFvb`M}goYM=hVFn= z{oMO5-CJv*Fi7zqZ~$8pfEKJezj*oi0`G}WpN}Kz7g!DOdB4tl{@3|@nr@o28$Vv3 z-{bTCFuQ~`-)b0VNO1K^{Y^AyfQgZ+NezQoe@t5Dqea{McCm4^)(=nA@BQfaI=w|y zF;wp{nSwQK1X-3bl_#sl+s#(kYv}McD2HQiujt4|e?H6(%&T%U5@u3!Ur(V_dvm*_ zk&q!NTg6)*DN2p?Q#(1lk`~jiMy{v~@r+J_lZl?04qhp`C|+h##t<-$tYub@J>Ky8 zEuC(Hd%UzE_U_Aq4IXlCTs$ZgOY7ns!Tz$Au{r2~1GQWr*>*G%xI`^g4kvKQhIGgU zL&Tp1lVd^(-ZmLlA5v5@6a#+5HYsQpJ{IXZ(@~wHL%xcG0a?1VI4mRD|t^N&OFP!ZbYPMKC_AUlp2hZd-MWjdyK*@5fiK z$NQPv7jJIhQwGtr8?P}DfVT`%=LGC0O)qb-*2?GmOM2dOAOdo1fQ#VsUk8L>DD!u& z9;78`0ERbo|6KW!v@;iW&R_-;qY=(#Z&VLRj~Gr1$m!Yv5hl3B^|-Pii(I>-cyYn4 zc-|I~xQ7+^0ksEs9PjIki-bF1%0cmDMKV5s>-%D~+VtjWizvYV{d9KeOpr*xhkW{K zkKgs>*XJDIDhaYym|WU;ilFk5z(STO2SIc{ z;_BY3IsW{BD?{8wNEAJEvm$gY5bQ#lD38;5aD=+_CCUR#McIGDl93BeX(lC66`f2o z$pf}%w3oQIY?kxC9$L6brM6T`O6k`+6z~jx*0qUPC{Uqn0yVxAbyL$4iqE>84roEv z$(b%AG_}q@x_%;wN;4Rc3zA;lpnRkts?VD!|Q(smuH1^HJ8E@QmYRzO_NUNthhN14k!L&NYr4M_B40ACQ6L1cU=cqVsZNCst z>p7~E3^TirHQL|w*massbedtF57z{hz_2EFn{}bG{MejnHhJ&k(x?~&R&^#Gy;Z~O zL>V*1EolYdc8yeK2l!i{OtcS%%IV8Q4|X4vtv`+Hf65)#(-v4#o7Y&l#HyxIm5E)I z>lLi}_Ndr%UJ0(p%InTp2GK(LsLB*-7;pIH_jv?*`PaxK|B`!@lI0ALO{B=~6+QAs zhjNA{o_$D*jnr^MHSr}ZWM708e>9X_pma(ij;J9}R|Y}EU93_;VB^VCPy4tTrc--H zesfpB!zxo&|24pN%JbvV=XVs<-v7t1$cDRf*UiaNruLD6{zLbwJBF_BR^QV(&hw8O zR==;K?_X5+v@efDw!P_ChkL&@+*S8Kg!MY*JKp-T&7;2DLp|(*i63WnaJthd16Udt zGkbo2)8;lGS+(rl)<;GP|!GJcJ!Q0_f*Vb39Fli|r zcG6%|t!2)bxHzz3TW`SEiU0Zj{B$%J*z4DACDnu4)IjJ40(|Acj~SMbrR|ZMlWJAP zI#H|AqQRzXWE#B=y@V=E;kP&89)c5=`p2KZjVyCU9%LXS=xwr`X5t{8=q;otJPeRZ zHX1A#Xx@@-LXv1QbNAULexoXQvTdYU2|bK3Wn>ci_q=kC!te`SyVD>2TO2kaK%ZM) z%n0TSGm8g4xXGI#D)_>W%HhOX-u#{cC-tnNCs7P^|9Y_$!a~afSJHpBDNs>=qxcit z(BDJsSx8L>(P%B0_;LwNhBd_gu-N|)J|%Vt(yH9*Wclet!_}ODNam=~oD!{9H4i~3 zH3jbtM@ld{N+|$7(Yk%50W248k7Vu%-Eo zLEfLrKtU#EdHC?J-h=uCu-0v)s9oi6*hpSZEP@`n|1&1r<#&!Stpd4Ej#=_Xh8deX zYf;XY@^l8IMIN|ii!(wd`ck?~s3)-FKxO9T$uyalIj(A{dqE|q8x zuM!yLB}Z{(o~P%o5YvRSUyanpriEwgg#yl7g;`Ni-uYT3JVGuD`vlszjgC3By?Xv) zyt8k(;acGrB4u*w(Qah4(c%|FHjmPsIAGTak%EGCuSUz(G(!ofA ze}dp(-?^{>`6Ws;EBDd@p8~>EWmad9R*>6wj%Nx_ge1YIOog0BK{Mc6qd`}NoH?-tsbF^RZ|^d zt&^CSgF>$XN7;c?qjk5@VsrK|HkTdkIYkIT=k1cellgG^{31!$UR95{K^8Wr#3X(w zSAxE~YNc4x<{6!>6|gkSjQp@GFfp`LJr@2os&F(9)c@H2n{cARSjp~Sll(&QA+Eu< zcli)B21ZiY=G<%Eop=EmV$<0xK??)H?<_W{8bgR^KmkTLDn*n1tbjH0aPj@8$QUY# z4fjY`|IXnCe!hwz1eK%2UT6eL3j&=}#Mi3Iv7exFs!#{;vMFT{FB>eb z4xh{9&6`{iJp{TMwT;qUNY#tMYU%U<2a{GJsi}0lrOK?W*49j_ij1jqs#CE>-B;X( z^Fg%>ihV2i{~#9y$W}Ba^>MiA=HWFsHpis%W$2|y(;4I`k{Z8$Md#EMK3oy_W#6ai zlt7Hm(XSQ&byVM5t59>>oAZ1%9)oW6l*n?XtU$f9u~A0?NUOdQcz{}v;Zk1s!2%?S zx>GqPC;$xOlrMEm?EQQS@FsB$4 zD#PQO0H5{wZb-(2!C#lJ8&yeln}O+HcQ&yDPQReCIajZCKFBZ}_!thpwwVUO>?)6= z4}G7ZK-eyh)UEE*)w5TiJqe2D4q7Jlj|P7)ndzwf)FJNc)`C^Ajy>A|o*iLD z={Zg}QnCNazn8OlZ(;+b>RO$@!d_5MR&;tgLRv1*8c-h`}AS<>cSlV zFSY-d@BjYS9BQ9Fxs`eKpz!PF`)Jo3dXZkan{804FURG6$&zfjA&q%qZqgzsWmB70 zj$E?;QQP#!Tlsd}+&u)bGC%qw(iIWA53^~6VG3{4E5`_yA)<9X&06&RJJhYU_wsH; zJDM=q4%UawaYbG#bx{!qPD+SvbezCm)OL%xCtr&|K1$IQ>+4 z@sj)k694SD^&Co5mOS!RF|7Pqh+`;CKjl6b7bsg)v3V4b`HS5OjdFQ~8V6_^ZKRM& zNbCE9icqq)*(1-4!uFJ?p2IGC;nWe%O2_OJpQX$(w986|2?cF*)^tmQ8X+mA)XZ{0 z*M~;4z@LTG|M;@0P_^5tcH>_(tV5CHJBt#aonfsrO&v3vm*he5A@egfC zj39(5d=lSC+w4=mFrgl%)4@@>qKiy5(YJ1^*myC~G~w=J-|FQ;6q!y#3)snEqF~t|YvpvPOSet>QI-sVaB|0nEzt4(f!-<5w1@~G?kSRckFAixC~ zsf|nPAT-~AFvvdk$7xG~*c9i;OSSxU0@@%`+4u{(as^`aV!s}JTcpDvPIvVC&iMD& zfjElAKYG}ugDq^I5p2Rwf@y5Ro^?9YAbxc0lQgX)WPW#LanR{CrwnUSq%p5C8k(?S z)-_kVJ$E}zPp(J$@e=*rM1Wpw^@`YLB(T4rV$TqdWYEBsyTT>XoKKlSG;fZXYo6i% zm;rImrC|~Iwfejte2MH%%yQ(RPF}_3m*rjbOzT^X8t|P*-hp1{kZJrTf1^grZhxEX zVZUv&%q}!xX31$_3yf@)Ib?MVYoYFwHbqxv0J@mKe~F@E1R;=|dPQho zHUfMDw91(82zau=>)Vn_-4Fn9Wiw!UnN4HH;*eEVzd=$G3I2DD+al4@=6b?#1ZfM* zxtKev&vwI-ukCZ_+0@~c44 z%q5rWq*juJFvi`X>+_kZ^C-Of`=h|?we!y#&1jVD@(4%7jo~Y>B2&L+DT7IT{lnRo ziTj0nskemp-!QTxbVDJ`VPZX(Hz z1P%{rSyF-v9tCj)?Uut4;Vw@p{yzQvj7C81N5JX>r^YcyPT&GeyKrUbjkxeyaSvDwxLQ=L=KjO!R;fm~6W7zWFvam^p((2dN<;5hWypn!|t= z5iwcg#m@RBf?s+HKU1Ixq^UF&cngED^+dkO+Rg^5<S^$&ws!ZZY+F+$oMDIigw3HY@yp>2*^DPN(S((|XIfeUm6o1P~?dXO!37BXr7vS_(#mOs0@)%x+f7&&ap6G_Ud*6i}K`mw`$^Y5CFYn}sc#8vj{8k4r6r~kAw!6RKp&3th7;5zfI z!9TX8;p&+jylq$g4_iQUW$0Y%SG@?qTbu)4o~}gW-#0UT_)hQdHDVk;!}iRJ>|ISJ z?KLBwX8$w7sO=kD!8Iqh$j-j`ZC~KCL!JPq-gK`b$9j(vX^yyXuZl^mcxIWY_TEAf zwibiH%leuTe8u5=B9 zV<2`@{{sH}mqKLS$4-!>F_7S+0RH70S5;2pzPzfC`~v|?(@(VCjGLFum)1>WXw zgUEDp){6)WX(_H-{jup9pUy|v)qx2-LdVT)@Z{^&Rna`^uYGY-<4oI*JUCZ4GYkBu zx@n6#)rtMX1g5GFXJwW<5N~T+M70LXYE%IZyCR=>&O@=wJpx~QOobMM^BixZtUy^y z2B+WAPIFs%1FY0ED-lfU^sU4S1Z_eJ=7OmjvkrKouG&v|GD$AlRW=?8c~`u<^6QR| zkgj$+ZhjWH2I{bKp=8$6t}AM_^twP!hVEQ=D({G`m7-mq{8KFo%V6eC;Y7U@sCzdq z!LC`(o$%;(efupzMSDDyzwXk=NWWL6H>2POTP*5&A$Z*6eI+A#d4LxQwbGG&*4)>R zl?mJ}=2nssye#A_$F_-urKXKJw-57FV&m=}ZXLzLNrM8bNN`nUqGyyjQ(kyuJhH=UB$&QS1cs<{NTd4f&2i&rAm#EvJ3| zE5KNI7Kdj)yq_(s%RzJ5#ZIWq5~GrPS=t#-*_r=ND^vq&-B#J8P~jdDyJt0G5X?v2 zh0hDgxZj7FeC}4>?fYI>%7tE$5os>W=3b}cH;5_HR@h9f2wN{R!ciiokVX)Ez-;PJ-Q6R+A72yj)LP6XW&z?*o z4sa4`e$m4(KCoG5kS)U@s7fPSTO+BQ82@#wQa_&b^F^Zo7Y=l#t2V39SeNlsS2MBG zX!T;|#eev$x;U5@FoW>NNc*{00?be-v|HNX>)Gr>X8j<|VRObq9pq7Fow@U%8_#6C z6}x+Womq2GVX6ElXDVFX$qWb4;lUlD|Js!b?Xv1kO@GrgFFb%t271TgBIjpkdGZ)>V3ZK0Z=j(a95?wt-^G z0YwlBl#qllNm*zRN6r-QFx^;01`zMm4?tRhP3>#6njXNYL+he<_bsYBLP; z&GMy4zS;mrqxlul{ z73+D<)v!=F>Ezg(bPcH!9u)_mi}kDvF6R9X#c|q3GviInGdA3Y(?W=qtxd5WlcHxC zmJ~cE4;I#L*{_67}-VF8*MYX9o=R#g%&n0+oBA$*vuiaq@t)ajqy45dRq zUL`88YnpYwSE;Rg8Sa#-rotnt@~Rxvr~gZ~NnZ=`0KCJ7v`LYZoKk|jK(&=zu4}HC zlNmC|3%8>~L`|Q|CplPC{bUl`ATAViEcr0dSi@+-sDHfJbw4OQ{r!kJ%o$i;y?1Ar zyL+gn@%8y&FiNc}Y?`}rL+QS$4Ym)5Z2N>hR!Z~7?NQhoZYMJHsEdU+f?&)XNd~|Yamb{F@A{IHWnQ=I2stIb!1IJwm zF>)}r_IBzXqFLa;rVMb4Ean#=>b4Ff69c4S-O*Dk86)go|u6y_0 zl^*i+X$0Q>HKdJ?+$&$3JSX9XM*T1HJ;Xy|53s+l0a!^D!vz;QbG@RduJYx- zD4sL59%9)Wk9E4%22i0uKm`^^(;8lKZkiDi%_0)*M-HU04XBbD?6%DR?i5V0*;&Fw z%fZ6f$H&W!Kdgx*j=QZ2p`1GKzD|tGM{X0ZnUD#<@~^Ri62i=ND~Z`-+NW0u-2$XB zIn(YFFYK_S+pKHpZuCSw@uZrB(cBwQ?$R82%jP-77NffBa56xNw2M^uC zsPLV7`#4+fDFd^5XwqprZG!~FHkK&TZ4N@gTy9}6 z1m#(rY3HDFl&QexeLUreB%h!OkTOU-> z!=pKtSB9Rut#ui_oxs9=W5SikMb;~cJ8!yY2GjA9V(oZaXIZw%68p6^I}6F!N~|nO z!r**7*Gh!NN&m_L)Em)3L#r&^&$asi%IR>{y_T=~WZfiZ%wR#T>L}CP9ycILf9u(6 zW=476(*(ws_kE@#o4mmG{^USo&$T_PoL&C3Zc;9*6|1GC8kTGsV(H_OU(vyoq}p}@%Q zDAuPexbnc>E@yn`DTElcjrVd0k3Gw^B%wF-e0LGP+q7GE6t7)+Cy-$mf5#D4+{Sjl z^$+F8Ec4PPo4OT~aa@kFU2IKUO*c8*-FD^E5y3fwCCx6EEqC9kB~M>jt4G7UA#~2x zUs7A;BZQ@N|4@!WWi#wP5OPh_8@+h8%(0GM!5hr_`_?ZUNxlICSk`e6=c0<;>L2dIp5T4PSxd3q- zKUEC#*a7qBYpY}315|;-8H+UEJts46zv^cq%e3YIv!Y4Kxda@YOA?LeSpCV34?Q`v(yqP**z*aK-t;D0>=jS&bdJ{u z8Sl9nMM9%ot3a>jp#mjzTNL@;4)L$8swGx}nPqp9NvJ|2J6yVUur;j0Bi~#agpKkE z<2HBAlcU|mg0LM#{LFmae^aZeo`|--O=;u231a^IT&|pTa&2M5y%X#bJ#^p1x`hQN zPMkR-8!A+jZLzhalVErrJJ>)+bxPO0XUu60HA~dtM$S~v!aK=2$C3R%9<~%fT@b~W zIF4zrT@^O(?QPmLEzN?@(~N{I4H+Rd1*oJ7=U+GPcYGZIM`_gSY~YrmTDoJIHoO0+;Y`%DNoWPqz{2 zd}~@&sdj$4Xc-b6|XXV!J$d5t^)Yo45O4MQ#+cw$;#m+rKbYJPb)ev|Sw zvbrJgc)lCi0ly$F*R?C!RP_HVn;%!VX1V}H&zO97GL2E=?;%bdmG+|zjaz-%Hnza) uZzXv$VCctHaPUX)1+zdjLjANtp>e0RBZJ=F@Q>g74}&cT)#Gmv!2bY6!tZnd literal 0 HcmV?d00001 diff --git a/eval/dataset/arms/copilot_v11_deep_val/manifest.json b/eval/dataset/arms/copilot_v11_deep_val/manifest.json new file mode 100644 index 000000000..4da07c263 --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_val/manifest.json @@ -0,0 +1,20 @@ +{ + "arm": "copilot_v11_deep_val", + "engine": "infermatrix-copilot CLI (shipped pipeline, pr-review@6 adaptive-depth ensemble; issue-answer dry-run)", + "llm": "per .env (DeepSeek-routed)", + "dataset": "vllm_omni_dataset.yaml", + "splits": [ + "val" + ], + "kinds": [ + "pr_review" + ], + "n_items": 1, + "trace": { + "AGENT_TRACE": "(default)", + "AGENT_TRACE_IO": "(default)", + "AGENT_TRACE_IO_FULL": "(default)" + }, + "moa_when": "off", + "pr_context_mode": "(default)" +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v11_deep_val/pr4810.cost.json b/eval/dataset/arms/copilot_v11_deep_val/pr4810.cost.json new file mode 100644 index 000000000..8b81e8d2d --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_val/pr4810.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 1089.3, + "split": "val", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v11_deep_val/runs/pr4810/run-20260813-134520-993dd9", + "recorded_at": "2026-08-13T14:03:26", + "attempts": [ + { + "invocation_id": "inv-bd7c06dd", + "outcome": "run:run-20260813-134520-993dd9", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-134520-993dd9", + "llm_calls": 84, + "input_tokens": 545604, + "output_tokens": 117169, + "usd": 0.461226, + "source": "spans" + } + ], + "attempt_usd_total": 0.461226, + "llm_calls": 84, + "input_tokens": 545604, + "output_tokens": 117169, + "usd": 0.461226, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-134520-993dd9", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.4612, + "minutes": 18.11, + "ci_minutes": 0, + "input_tokens": 545604, + "output_tokens": 117169, + "tool_calls": 131, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.6875, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 84, + "by_role": { + "agent": { + "usd": 0.445147, + "input_tokens": 533662, + "output_tokens": 105514, + "calls": 82 + }, + "reducer": { + "usd": 0.016079, + "input_tokens": 11942, + "output_tokens": 11655, + "calls": 2 + } + }, + "cache_read_tokens": 6852864, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 3.95, + "gate": 1.7, + "review": 1080.93, + "report": 0.06 + }, + "time_to_first_result_s": 1086.64 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4810.trace.json.gz", + "trace_bytes": 24449 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v11_deep_val/pr4810.md b/eval/dataset/arms/copilot_v11_deep_val/pr4810.md new file mode 100644 index 000000000..afd9b60ea --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_val/pr4810.md @@ -0,0 +1,38 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4810, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head f9665cc223c8) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 4 finding(s) below | +| Security | no finding reported | +| Docs / comments | 1 finding(s) below | +| Behavior / compatibility | no finding reported | +| Correctness | no finding reported | + +**Validated:** +- [sweep] Read new branch arms at mammoth_moda2.py:486/488 and mimo_audio_llm.py:1156/1158 in the tree — both `quant_config is not None` and `cache_scale_mapper is not None` arms are reachable (None quant_config skips, base mapper returning None skips); no dead/redundant branch found. +- [sweep] Non-test files touched and their coverage: hunyuan_image3.py and qwen2_old.py are exercised by test_auto_weights_loader_delegated_loaders_accept_mapped_cache_scale; mammoth_moda2.py and mimo_audio_llm.py are exercised by test_direct_custom_loaders_apply_cache_scale_mapper. +- [sweep] Read all diff hunk locations in situ: tests/model_executor/models/test_kv_cache_scale_mapper.py (full), hunyuan_image3.py around 222, mammoth_moda2.py around 483, mimo_audio_llm.py around 1153, qwen2_old.py around 335. +- [sweep] Repo-wide grep `get_cache_scale` => exactly one production survivor: vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_transformer.py:2238; grep `"get_cache_scale"` in vllm_omni => no getattr-style survivors. +- [validated] HunyuanModel and Qwen2Model have no standalone entries in registry.py, so relying on the outer AutoWeightsLoader for those two loaders is consistent with the PR's reasoning. +- [validated] HunyuanImage3ForConditionalGeneration.load_weights uses AutoWeightsLoader (hunyuan_image3.py:2178); Qwen2ForCausalLM.load_weights uses AutoWeightsLoader (qwen2_old.py end). +- [validated] MiMoAudioLLMForConditionalGeneration.load_weights is the top-level custom loader and iterates weights directly (no AutoWeightsLoader), so the added mapper application there is required. +- [validated] MammothModa2Qwen2ForCausalLM is registered standalone in registry.py:92, so its direct-load path needs the mapper application even if the nested AR path may already map the names. + +`vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_transformer.py:?` [blocker] — The diff removes the `get_cache_scale` branch in the four AR/custom loaders, but `vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_transformer.py:2238` still calls the removed API (`if self.quant_config is not None and (scale_name := self.quant_config.get_cache_scale(name)):`); a repo-wide grep confirms it is the only production survivor. Quantized HunyuanImage3 DiT checkpoints reaching this path will hit `AttributeError` once `quant_config` no longer exposes `get_cache_scale`. Migrate this branch to `get_cache_scale_mapper()` (or remove it if the loader is reached via an outer `AutoWeightsLoader`) and add `vllm_omni/diffusion/models/hunyuan_image3/hunyuan_image3_transformer.py` to `_STALE_API_FILES` in `tests/model_executor/models/test_kv_cache_scale_mapper.py:15` so the guard test fails until this code is fixed. (evidence: grep 'get_cache_scale' across the repo shows the surviving call only at this file:2238; read_file of the load_weights body confirms the branch is intact and still reads `params_dict[scale_name]` / `loaded_weight[0]`.) + +`tests/model_executor/models/test_kv_cache_scale_mapper.py:12` [minor] — Confirmed against the PR-time tree: the new test at tests/model_executor/models/test_kv_cache_scale_mapper.py:12 is marked `core_model` + `cpu`, and both failing merge lanes run `tests/model_executor -m 'core_model and cpu'` (.buildkite/test-merge.yml:13 and .buildkite/test-amd-merge.yml:32), so this test is selected there. Confirm from the AMD/NPU lane logs whether the failures pre-exist on base or are caused by this test. (evidence: gate_report lists the two failing lanes; grep of .buildkite/test-merge.yml:13 and test-amd-merge.yml:32 shows the `core_model and cpu` model_executor commands that select this test.) + +`setup.py:1` [minor] — setup.py install_requires is assembled by get_install_requires() from requirements/{device}.txt, and no vllm version constraint is declared there (grep matched only a comment in requirements/common.txt). The PR now depends on quant_config.get_cache_scale_mapper() in mammoth_moda2.py:487 and mimo_audio_llm.py:1157 and removes the old get_cache_scale fallback in hunyuan_image3, so if vLLM < 0.23.0rc2 is still supported the loaders would fail on older quant_configs; otherwise an explicit vllm floor should be declared. (evidence: [sweep] Dependency check: setup.py/requirements do not declare a vllm lower bound; PR states the API landed in vLLM 0.23.0rc2 and env reports vLLM 0.24.0, so no lower-bound gap is checkable from this repo.) + +`tests/model_executor/models/test_kv_cache_scale_mapper.py:144` [minor] — The diff deletes the `loaded_weight = loaded_weight if loaded_weight.dim() == 0 else loaded_weight[0]` scalarization from MammothModa2 and MiMo-Audio load_weights, but `test_direct_custom_loaders_apply_cache_scale_mapper` (line 144) only feeds a 0-dim `torch.ones(())`. A 1-element safetensors scale is shape `[1]`, so the un-scalarized mapper path is never exercised by the regression test. Parametrize with `torch.ones(())` and `torch.ones(1)` (or otherwise assert shape-`[1]` handling) to prove the new path is shape-safe. (evidence: Diff of mammoth_moda2.py and mimo_audio_llm.py deletes the `loaded_weight[0]` normalization; both new direct-loader tests call load_weights with `torch.ones(())` only.) + +`tests/model_executor/models/test_kv_cache_scale_mapper.py:112` [nit] — At lines 98-114, `test_auto_weights_loader_delegated_loaders_accept_mapped_cache_scale` calls `HunyuanModel.load_weights` and `Qwen2Model.load_weights` directly with `_MAPPED_SCALE_NAME`; it never instantiates `AutoWeightsLoader`, so it does not test the PR's claim that the outer loader maps `.k_proj.output_scale` to `.attn.k_scale` before delegation. Drive the parent loaders (`HunyuanImage3ForConditionalGeneration.load_weights` / `Qwen2ForCausalLM.load_weights`) with a real quant config, or rename the test to state it verifies delegated loaders accept already-mapped names. (evidence: Test lines 98-114 call the inner class load_weights directly; hunyuan_image3.py:2178 and qwen2_old.py's Qwen2ForCausalLM.load_weights both construct AutoWeightsLoader.) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v11_deep_val/pr4810.trace.json.gz b/eval/dataset/arms/copilot_v11_deep_val/pr4810.trace.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..e2d0b5506c4475ab0abdcb10d72152d593c18e34 GIT binary patch literal 24449 zcmb4~Q>-XJv!%Cf+qP}nwr!kk+qP}vY}>YN+njUnKN-DEXFYT(l}dLf{m|9l!jFOg z_^$!Wq#i7|vP5lcvuKDqA?kbJKg@eze__)yfX$$y zuz_cp2j{r`!yYp7V37T==Fh;B#*=Gjiq`wV-;8fg)1JOLX3;tRG^We!arxqb=a~-D zbVL@#y{)jo-X{c6NZg-J&3;*Oj%sGk?%y`>k2^Xw!T3U?VRFn{q#k&w-#rU*$sPc^ zGG@UqnmFlpy}Un$&8FCcpVQO=pqI{Xl~sJKH+z0YHW*@vHtnbI_n(USI&j94 zCs9<(kV^%!L?=0WGs=-UBH^mk!acs}EAsLDrfora{Hh$5USfijcqEgulY(;jnZcQG4*B4{$R$sfcNuyKfv@=)-Yzr0o4hgdkGQM{~hGl>-jmKt{Rm; zgXBBuITWRNuq6y0ypKrIf25d{|GB%&GOxSGJXxDUCgyvob~wTe%x#w_n}0}XFl0i6 zNDK2mf-uD2Gid^_lav6lL=)Se%Ufgzv;_-=)_6>fP#)wtW-}D;7z>mrI6U)|^UKT~ z8si2c?^>Hy~Wv+7S?;K=LKOSM-ba{X2yvd4LE+^Ut`v>K7#f-0u2N& z0Iy=#UB%aIb)cQenR^;h+lwV0#d6kBFjZ?c#;BUEbQ3Gs7fra^N8U21#n&Jv%T{O$ zwT9}?Zot(HJ!BYdw55jhu=g51Oj209K+kp4#E3d(}x{G%yv?G=pKui@rQ5^)^p*;4ytpHWSUqB3mhsdvHsOjJe5j*^yHZ5l$YT-`A?X zUXRD;*Lc=SwgGdOBg7Ca48mKHZHO^gJT7-p{AUj;1R9WB49slU`sV=SXe;LWR%CU9 zF-NNN-8*nZhop>+EF=(ulQQL?0vsaoR8W*fq*CzwH;ocdHIeV&8o#_iDxm4X#janRA8d)m=g+6|_2}rVpYPA4ehayu*UM2E z`vxI@pgj);?SZ5_=4?@t5)s^&B?rU-_^8(Dwn=MWCw}wqBmb{9v@)&asTLaS*OfP- zB|iOE`^!=LiM#>tehV3Y9{-XVr359c-3&cZW>M`4b%NRqN8ab7Xl#;fA;nZ!i&xb7 z(roShNfZTloDdeIsh*0r=%e_#kT0O6LBgnI;kRvq4*l2BJBq~+i0iQYgJY&_kxUS7 zwoG$ho_{!R+22NsOuhaw^HaodPkaEDseNCUQeNL@ogmO+txOTDa@l^<n&7wfs0KndnLb{)nnd9BSTIr%eZdXBKNkwj8r#~`}j*^m@xQoVd=#NkM zir&7Avwo%H{a`wxm^P2&EHeR6j)BD!`%#iS{H5cERRnKMF3AHYeP;X~eCcoXRX;c) za~6!FmIE^&X>V9|>^{mt1&59}Z!Y-n^&(M>6c%@cLYZ#xJIDBfpC9?%U`N0>9~xJj z<(8a8yGwa;%1aRobEh9yA1ZAxkPwoY>nypY!hVP&=Lqq*w@ha1?)chEBL(Iec}!fp2m(^SF${YiWbPS*&x7A=mClkTbb*2Mf_ zHSt9z^WF)hZM{z%ypB%@Vk<%#Z<1&sQ6#~Hk;ie$Z#Be%=-42)un_17UQPmepfap9 z04Vf^9+kQhKv*Lpg7vgdOZ@y|kobX(lIRAo^JNs!2RLAuy2ulKYBP)b;t_|JmP*pHZQ;*$5|5y&y4qp z2p*Dn=t1GajSRz$_KT0&hVhovSKm0R8H|cq5VmiGSrcBtV=B7%IRSfat8e=Ncl$k1Rdhl>+p}p z00j(#3Z7rVA&OqEN&r3?r*n(@?h#;qj_MSVOoP+7e2WR#Y!e2bHW#^U2)8KBW1+Ky z*g!^K(A)Xn--oRRm+NVy28bi{Q`&pO^^4;WrTu>Xg_Xl(iNvyIz>?0O07xJ;Y>5Qn zmk^V2+SR&1$SH;(w?OcaPD9S%HlY)S-heC{+BF|U zMNlwEtBsU!gXDPSlp`i7;T=@5w9b|G3pD>s!EZBY@sw^t)}y^m9IifeSNQ2tY2TH4 zgXq$*YWX?sljk$RK3twtEq3`>BYteNn?=oB zYGTQpkAzVP^`i(@MUyC1a0hD%duK?<9&BD^)_dQ{U(2Pxxy%w<`L`0Y+LP`I*iT7qD>xcdFra1fI0Y%i>ChpZ zG!BDWorzuR&M!;|jmrkMXJ^S)l+1?=dTtkUOJ1l@CL_I04G5 z4u3FiI9Y51<56lZG?da{$s3GFMTD87sL^Dg%TY=jQYnplZDO4QLRI%K#idk0V{O@)v>)daX9w>II4N21cVYl?F76}&!btoA1BIhaYB1kaLvT2UN&j&6 zkM;(CIiV864b$;&7KPASOZH|I)(Ox)l*~-T^(19{BtGZswi0-voHj&m0pv1Kc}6BQs< z?b6BzFhg8l47mk(MM%IW19Nxe0Jal%Ccx@nJvABXGY-hxiy|J<-_uy+=B{8xw0gE!qMvn@Ky%`3)M716rH_;%e;7yL-GN-OqZc_ni|f- z$}J{C7DlSo6mUB#3(KE>t;dOxNh?2bC5muTNni+G@{xIOi)Yj}XkRdOfwVF#Bd#(b zoxo(tE;}c|AGkoaTr_Fh|{?$z-s zmJT+E^JeNGXtiz1lcC~U$H6M9MwQM8JRq*|^Fk19_80|TwgLQ<8&3rJw=aB18WSR0 z2Nw7bi;kxs!EvXQgg!#g%RFkct{o6hD{WZd`X?JhtR&-c&g>v(Sq zd7ty_;+59t-xsb4^rZF|`!*g7m@~h(A??Ika##B6Ue*H2-z!63kS;U~-tLO-07|oa z9B`2Kf?UN9m^(wy6HBbU@QJG2!J=yO=3Wu>A61y>s9dnS5#o4k_P8ICX$5Yeg2Y$; z7d(aA+Yn^`7o+@Z-6{6A^eXITH%9g$O=zzkDRgI_6#a^*47B%Q&-706Ur~Ijf9`NO~BpqVtz;7->z!<2cv$t3H(KQL;tO=BOHKEYDoLc*|iZ>Nkchqp)`Y^eWz~R_a zdCI1^78WY9k$s$za3=x4?y818LWoQ7lNL0CUWl}i(!B)_C*{$ZSY^T>&GZL+;ZYUB zVt9T3l5*bf!}zGC(>z{P)P8dbo1{rA=Ihh?0b_F52~Ed>c@n(H+a*}hg?$L^w|7&f z7)Q*Lm?U89oEE>z6?zm)R{{{pJ&4vOMpd4p4bHQY?tTz*SqZF{Eo}}g1_Un=S1M{& zjfq#_pL8Pb;)0;4dv6@&j?pi~<$s)bc;_l;$)$AR5sCIyJ)SLgols=Q4_=`C?jZ>x zOc;!taPS>4+VcDMNHFdUOgN5d98bx`^G;r!a{$t@#|bXVG6xzaP{{QA2k{6g+r;+q zrGNaw&MMUEu=kQZ!L*38=3NttJ(1wzS>M8i+3Wp$e2h2kXOV|z^z!)LUtuVq%6Q=l z?l_sR~tMhi?H1TN1{mz;jEIHH3Ywlig{nzT49Rf>}Vp>|*pe`Ts8ed9GffOZCtM#Ig!Y zwhrU-ZuejiV$koSs%yFA$n0BElzd`k0|*LN6J_Q}1bQQ$ihyCrUZNhdngaMySBb!Z zDK=xFc*C1cBhj#)BmJUEG8eD^?32RQ92RFNug%uCUNPmP;5=ROT&tviDD8 z$RA;6OYH_NGpKTeBAX&B>XYf@W{yOU)9?0taiJBvXPGe1cTE}+f#Oor>chSE0>G$+ ziwAQ(@sN@F)|3taAEu!@FfmtEt~KKXP^GWw_{b><$R7g+hL5z|bBezTP%CkkF;tK4 znOtznDnoAMY@X1Tf)zDwU&;Vt!S$KrA@}7;nRzS8SZin`mG=)g-Z`1>!_|=3Sfo*7 zfkJ{C??_+UU;jAS&-hcAvu$w?Xso~uYIZJS)xkvKQo9C_0c@HL+XgY*ZsN-bP^F`` zpE7QL402XT0!y&U1qzD;>WsKSqA=M;+H)p9){T6CGk(%f`9|dr+4%aEUF`+sLp|7(uQHJf zG-(Sx;bXcRb%|Q?7t7^ax?y`^Z3;itszEJSO~QlFKkX|JEd<-AaVXSi7LvF%a`z4EKPLVer~^`9M3aelE6ScH55q$spBP`hzOOQ&MM`a z677YcAehB0ZH0hwgGsl33bP}e3}EKC0bngm5lfGHwJ#{t_Br1LiDEVI$t}L242Fqk zM=0|U=R|~!sX!;e-8qUJ6qSV%fl2a@QyCNfko01WQ{{O_ZOb1*ow)`W)0(}|%MIuW zc_p+TEOP+_DvAq=Tjz6M|6UaGd&*p;bmC*l1UT9~i0*fe>w~GW!_|;lDrV`26iw-< zA+6dV@R$%@0K=RTj{ut{6~?cTeM}KVAw0jHCmhPEUKlv^ady}QEIo7?b-1cOoTbt5 zB1l6FpwkdBvCJNN?ieHeU)za^{P8CXN}qo6ReDRD8ad9dSx;1mC2=7wNXG(7Zfhsf zT%7uEni4~zluA(w_e2LQNa8{@Gv%r}@awTpuQ|4s)-xSnA)w}GCs3ebfHxAg1hDR; z<7`h9xqy$vvsl$|6nkylprqop7^FyU;);95cChiZYYoNY$bOV(a3x_pb)v1EM?;i* z6dDw`GSBr#gW?QnGgn!3r?d_5PG%BBMZ`nGkuJ$t9Xxuis6Pt_z$_`G+=##|akZUnpo*UbCj8BatiB0S z5Y(fDg2w~5$RpuQ@YEO;4_#P8O{_+1wF4$KeRkw3;X$bA3x&nTFPt1g#w(jih3}~bpFYHnQy+P+?0h$sA}GCIM!rwNrf$~zSP$xj zxO>XPz&B1opki>oUHcRo+Sw%+044U0?%ySrev|$7ciwwBebCd7MJslF{1E6&?cl52 zlpMYm%h1$$ceSO)^#~%HCipKUWq^`AX;YSp_JQG6vy;aG=?)+4KZG2tF#|H{PXO$F zN>fOG1HI8i$bEkYWS^ZL-k3J!ni`W?*lrKYO>Q=?wtQMSJ-loV_||A@Z^m*^^t$fX z)5Ww}1?RDWIirImcPWI5rp`kg(V1hLVCJNpU?ZO}bs#*CqR$OkcRZ%#rJd--FaT&O zq6l$&jq=eDBbVswm!QQf70{e8*;ZnjDUeX9B4u2yxWpjG)rq4t3+u?T(X&u-;$lF% zEI9j1fn)O&9FV<5+I~9iOmFtvdMk%XL=njI ze4SVTJxsQ0Kc*{81`phz;vF<&FNvIV0=P#yj&B{_Lp;2ws0U==tRQvf=%w#=OdhRC z+j2xlI((yB=xO{>u;7ZiyQW%9P-z~_rCEAX5UY>`)ieOQq;d?Usq&Y2+DZ{~yxtpGIJ=j}c-?N*6;gI`mXiWtLygBRMDY!BAZe&^9fN`#?OKbc+6 zQx`zfg!C#5DY*1*w9B{)^Yy7@g0z~`K}lBppEY{#ciZ8g0vT~KY{|jE#0L`oiMt0K zP&A-GN&Q5IjKzDLhXaZcS8D3!R(qrr)yTQ}bGi!^u0N^6X=2WWyNY9rstX(a~;T#LRMdZJZ5ELwfuKkkU&=q=7QsDGz}V13vY zm<1-*^sQzR4i~_cwXE9o`?zKv@%~ORm2RG<+*9t`B&c^W9rxgfeYD4>omVSiCKA4r ziL2o_lHiJ*`SM*5uCN=N?RikE$W}f#Wezv8sK#EZbr*7yXp{*&}T(rjbb4NoT?+77+yvKt(gwErPJ_07aUM@IFZaC6v`UmZUDvCJNg z8^`pLi)J8PB?b7P8c9)t3IaMUi^nIXLaE49%MCU?^-^t*Ak5T|mz#c~QA>;tguBfG zrWQml)uxR#Tw&&uh?#^8Tg{XJSTpQcap`0WRDYpB?iTN1id)6>qAt5mG&kF9Qv`U! zD)fiDkb)NzjU`*Nj%zM8M^p~zwSgq5_5G7lh)Fhz4+t%}b}x6p9zbP`#hy%^9`O%2sT<2iE+;KYoJ{fPHgej&c^pvz1Dj1uGf`M5hw2pvJFsv7m z4;8?hA{=A0md1%OIY4YdSXzIyxYRRbpdt|(#yPPz?W$F(hqcT2F|+-JsY#Jz0g~c~%V7<h z-WnFsZ{hjviiGl17TFUID;ivGYntO;c*Mksd8PYj^i-CSt(`Z7a(R=w1|7<&m#JCS zi}qO1_sgsv?xo#6-&dMNvEb~$ui>#@si%Hb6Yv?QRIN|- zhP)>mi)o?_NM9heS}U6UIm7H>?n40p0fR7Lr|CQ%jqm*n^C!1G@(LjYmf>;0Brw|H zuaJxk(;7czm$}?vv7A_tgT8q%V8GWT(Y102O1DY)zHj?>O>d8REdh=;7X0-GPPzECat@y6B%whWe;B@8`F`nLTRM?aFwki^mWgDi6SvwSruV zSa!9aMFOG^EvN3Ojw}(8qj#nT(Zkp3cTH`qy}1R;WcFPV2>`x=w|Le3iTFyd#Xvn@ zvUr~FLuT^^?{umWcs;)F542rT$lN$e^?n>Y3x;_M6yuuv^TsIE zc>;bDnBh!0aFyQ>U4h>-j3A)0aoKh_YQNO@o)!~bcBatgTiiJ&r)7b|%}f9DnPJ;nr%zi@i^&%n@9gRB7d%in~Mv5xMN zSSNlEjfqyij!ATc%z)D=;9kzaB=GT;1(%J3Q+C^N&F}a#GPH6;oE!K*(h!I9Jkwlt zs?r=iFMpowg{K?7%Gqol6+Kq&5`+p#=2gE?S_Skz)RRk4D0%5*HwZDB_lbo*N6d&K zg9kMkMSPJVC78W1=#C&?&Q!1ikm~MSM2qWY{>FZsrCa(Iz}DRaiA=_De$0W6SXJLU zK&}<-Vm3k#=yl_?bDbzgT|n=#n_L+NHJ>`Nd_lbNMwb?aLs6X|HIBxBCk>-;(p1tI zOy}EW@EhsE>x!+eWtbs8^MZ`bApB|i&VZX3qVOUE?o~-F%a@9PJTvo7i&fZtcc>oc z(H4PoJr@QK=?Mb&`_%cJi^rmb`GwrbHc^4Hco?;e>7S@mr*rWx>c9TcSr##yd;-}k zz#LOiNsYH>TZY+!&E7V7>(?&}zg%ylh9brwX0?uN^s#6znI$Lr?Tm>LB}KMKYYInm37u-JQsj}9q9i(-xIG3G?wi3GPW61(1s#xms3@24)D=zSckV8@U-qMO ztvU8ry;83_80Mk|Aj_ubYYTS1 zcEa5%GfD5#p6OQUEVTr}u?=l?QruJZ6I|8XXVsErcjdJ;WlugLO$xIDSt+LN+lykU zPD-n!eT682))XZGrJCd`Jp*DI7Qaoc`f*s&gQS>JahB+gIm164T(>D_3Qi@$u{1uA z7cZ#u;pbV-6!L42VyOB^#oHk}7AR9FT9VV^p@JexNz$%8E>8_;%pO^Zg`p2q(J9Ax zPk7Px@}IrXB9eRsBnA-P^I&FY6j|F}s@} zOdF%xZTI)jj8#aZEN_!FbHy_f3HyG15EAmBvqW?6nx~Mu1n%o z?qgP{sWW%f`GK-D-EM{OuX)=%Rf?R}R(UEwaaYX01BG3!eD}J&fLauD!z;Xd4Ao(WUXb^Gb)3_UUPj88^X>DSHFniC3W4nwc6XNk(*`s8421NyzA$U*Ti@vB)8z00V#4|&envVJdC^CK6c zJO?p$Xa!GT4jlf0L0q}bae?tM@(6Srt!=J6Rq&#>8kTy8!vb>V7IP<{c#z`FgW(i! z*Rx9(R~P&H-hLJ=kXXv91WO)?)X0vOx0E%O4PWtbxv;C%8ADJ;3SUJh`%K(>2Y79K z0Tn;{z1MiBcpPF}0KEv9M_q&xT2>gT z0d6}#I-q9c$?kK_isrUeF@E0=RO7^wf38=p|K3Wd=^|WnTwCq~(E{83Rv5J!pUO{N z0_P6$w*dZ}_SJMTs60DYJf#68WSmfbQr>!O>m*FIcgq&0@!a*6qO%KVl%^>htEhj~ zzV>kI%bQdNaQ7O*d4eF!^C%o7a0QNzMUG3=7)I8~%P+pfnZHvEvSf68*&$o>_e*JL zK8=`fh2O4~)l!)lZ5w_N3cxN!j@$H2Yt?V^yoMa!F1)99K+7I#he~sAbHKlb&Bw3$ za=dWe?rZ7apUViN0-QpP{fPzt82`{M4PJ{kAsb7nn@PYmEOk!4(5_1E`UC5;M*gx)O&NJ0x`~B+)m2zF_fV<)TSSn4D z6WOE4mV~U=L-$8%<&MR6CV*v9AQxpkY~k`SdPYJ1n$ZkHgKbfmH}F2$oounr?kCYY zi}%5O3&Ezo%{UM%A>_MHM)E3mnJ?pqxIQ>?c8ffa-@wg6i9``dmX;F(lLw9*%;F|!S5A(7b&fUR1}_?p3z?-`6@#jj0|%~dT-$Hg8^Peb8Ml7bf6pCC$NPJ(_w zxG`3^@X}#608WjxnuW z`Aj_aK7#ee1*WDZH9sW_UpV$6HLzME0x{M5I@vVLmNdkXdX%cL>#^p|{SaQRqp{LY99|MwE|P>#WD=RCF&;hf z3i^14p-Rw@_`N&bvh2p6msQ;OmWij2ey@?3M@y zWS?%Rw{^1-OEnd1y$&Hh0pp<&OG`wU^a>oNqs1cy|HY6O42vCnA1-XRj zhx%%jf%pAD{>qxe^gY%D=+)vXc=7u={q;hZsh@Fr+vja3Y9?2K5gSzAQK3C$(2uvR z+@M1dA5*joQF-TUY0VS-rBE?BN8v2d^x;^}U0Rk|(6X-vA^v9jg8F@+_`T4 zTon6*@Q@$pq>;cgO{*;X+WyKct64$g_V1E3`{titxfXr#D3hK!j#bzK?TuOcU5l_h z9WM)LfJgm2p8f-x98bnBuW~&z8E{aKKD4K#?_MW{M~Q9sJRZQP!!G+`<&QSI{6OzD z!@VxBwUTjZhGc*U!>FR9@uNx>50<%F@YoqwE7__W+e!L)d@JQ=D04a9jc+J? z&y(ZE45Ipc9*%>|5Wtg1(KS79>WLz`s*qPNxcP1!1^%M^7&d_wxJ3AzD|&AkrI4`F z893H`=xrWW9O4_?A~q4(_|3xFFk=2*n5i;Ef15wV=4UJ3VfWah>bN zNy`%LVf4GaKQ3oyL-~8XzfhnBjM5ANxO)I?0L>_d+h~%+b_zq1r|tmOIu7M?_sTd& z{s!)aEy_pW>%bY9`?W_7uB{5uSY@%AFLGuW+&`u1e^@pj#`N;~EqCs4<@UJ8>Xe)s z4v|WmH&6avrs?bP`aUR+ilKLQaXqe#Z8l>0;{!VmJ8LWxMakJ49A=8}0^q{r%z!~; z4}P76s>{T3_*F}8h0%u)ejqmU{HQ1IK2aK46Wgpab+*kIBV>(c9*wb*}p!dpv6_1Ujm9rVXW(gGSd>-8@B9&&m ze4X142F13QQz}rc1+_m3&%H2gDAV>DJEw80?RB=^il*9!g5hVdx$%qy0;@WP$BrLF zk)xlpaJ>)K+2?li%^#ZJ7~<~H{IEbQX!1y&lH)I4Ax*v=r7T1=$+S&zQw8%bA2&Ox z{9>Dl_GE9irttwx0uxDMO*inR&+Y!XY$~H|yS3W@&#OXhzJO@hTH*gXJig4^0qFGp z^MJ$n1#0+)9I`i`!q3_7EKZ;x_hplrN3n#j$H)d#WD#l#Cg)FjN{)0d|6!ew)AjR= z&XxG{QNsrdkA+==73)1!FD&m=dHu)8w4~vq2ex5EmrtZ6k}D|LCz9k=;}?A8rSfOf z!}*!r?%$Q&FiVzapKf6JbU zoN&8^n<#&RZ^XFtQ+1L$8W-cC%Qd>IqCYFZExgu|rzY@$mXRD5XV_5jhICmkYOE1M zG%a9*a#VV%Cp*O(=5GKYaKu^fY*4*zf`CvGX+|2U%wwU&Lj8w0s7#_0yS6rGM!QBg z02x^sI&zg8%T^-kDn7GOagEbF_Bu#uLMazm_S(WKQ(DUT`k2PydsYfzr5dhWf|`IY zCpI*)$S@)<*{}+LRL5#ig)XlcLcE7nN>Kkni7&ZhUT?Quw}}y=o+ctVh9Xh!L$7zy z?;R~6%r>o9vnt`YNyZC9A;OE3ZW~BFiprqM<;^Xo1Wl?V(efR7bJQQZmMPj8QtGwm zmZFH2)JqOaUlZU^u~~ot$k!XWcvxez#hUgpsR<_$9XDBzx+gwB$_1eQTfBh^1rkqV z4Fjy|CE*jTa=Xb?M3lVzU;b%*Tj7~IAU{tQhWAVomy`dR3@oFG?R zBL#%4>lc4rKQg(arV*U2B|82BCt4t-x}1cCTUdlf4HP;I@j%6daDeJ|LFc-fK0#%? zg|n_<8H{}ZUDdUlg{0;8KI6k`pCyh~0L{^<7B^d)f-aR;wg?#!Z_$%p(OKDu_Q9%< z<*zIK^j_xixkB{RX+MRQv&xt+!%o6PT2LAIakHoUgUFHszhP~Ho(@^ORaril-?$#U zUEy$zygy-gL_3!KA8v6g&>1;TC`9p(?uB%ZddQwVUxq`endqtfzGPvy@Wfel478F* z=ylCk7YctP39=G9ubey%A~lrZKKUe0RN=SdxKSkR2WY!r|ETkFY6p!89zC>PUhC)I zzgmA6u_o+PN%~t6t~}QQ)!36gdwTS`N?SRDdC#cG3&1XWml(Wqnz1Juqfet2r-7w5 z2t(BbG|$)8qho;&4c}vzC$rq^o1M|m=yKSsgu-spOP4iFYqF%YCwM=U*V7Rjs*CC- zDq5%Mm@0qwHN_v;T0SZM8x2%{9}M#L)wLWX$7brKR%Yk3SSBDZm}{I%`m?WJKp_eW zOJNpxIkbZ7SX}Hjek0LRty0&!pZr%!i1Gsur5}9)UeXBzp@?e9wDM|;BxxuzGzm9J zzi1rwqx>;e6SsBlI+=MCw8zm{yn1KfiUW;wU(x!eN`&chSBYwmCkK{n9<{G5H%8(+ zk766HRHaMV3=|ItNBdD{1B59=!cie*NpejlBUq`7m=L)Y-AOoFvnSjg$uKMmif9?} zZ0WEvL5uNNP%{ZbL>1*^xq0~FtGvWnr#Z3D^I+qDVtlg4$#Dvq>M0AR9abD;b!A1# z8)QKZU=$u3bw3CEzNLjlA$PzgVi{C5m5r3$D8xUc3|GuXb>?d|;b-E*k@Dq+V{cfW z$w+_oNHQiaflCv`TgLfRbL<1H&nt?8FpHJ5RZ{#1Op(R_MTp8T)t-E z?w=IB7f~=PV9qaXhz`eq{dgn>A|fSEGgji-8H50@ABVEFtq>KhN>I8o6xMGbb*gj* zIPh|b)+cZDW*xLjq;fu!Pkwy}CaMv?wP>SIR&D82IxvZL7Yx%*w|xw7FRStRZRv0y z6H~(pVfL4+9rz|sYRCy37qW92{)OS}rVQgN!i!fW^nOvGQWd=HN>IU{uzUmFkpo!yCJuvtypd4LKVtJTc7*e9FwfnX#5QiZ4XC_?o#hm;tM+P zO}TzqbDUag{?&X2*kYICp)ndNTfuIJrU!MCXwBoN`~oP>`aAxJg)4yu!lNkF&Nm-A zes7Gh4A~(p(M1`jO}InlNq0O6a4q5y6@pkYH13bXpg zogUOmlndDV+I=QIMu?E=`yt+NN-6UeFR^?wp_Y+L+hJz z5u(N;j@7)Ji?jx$5lfsW5m2VobsZ~4DbUpDlB?6X3tfj8+xLvdm5=|I1+?Z%;~_ep z$N818n&-!J5Yw9~5?V*)Kl4tZyL8h&kz!5Ips8_jfs*wgA(VtoKIl@UA=xkE;ERLZ@eu#=>$j zm%=6T<{}|m_{GLe28*dbv56_jP4#x`A8K2ZBCiF}7yea&m(-pH33*W$P{_#oVpojT zc*l@=bvCYAR}+yD>9wOYaBuR2hoY2*bs`gW=f->zoddGxNfi|70-`I&2!{BjU}3<} zr|gF^EWeMp(_{^C`aHbv$IEpr*Je#?63PKXkz65Uc%u9rn}bLY+R3Vp?kL;cm;I_6EXPh@ zRO@pcnu#ZxUSYG%U<4|?YlRemwkpPmpzB+d6qU(f7hsyCSnGrd5~YDBA%3#LC~0No zDrxHJ7`cQQv1CrY3CgVaTs(>KHI%=-App??AJK|j5v>?x;^4Z7Wbj7;GBQBI-5M3# z6X;$N3JHL|rp50mz6)CNCifD#`e3H~lKg73Xmu`?or&a1u0)*Q4T9n&b~Gu3T-MtP z3DDAbqjgOj=kR#an`d#T_9RM(j=LohP06JKbS5l{1z=MX>G|O`bcl0Y!!chNr-;Xn z-a_U~5~hLvGQj{t!-8X@WUoD#3KU z+4Jut{M3nG&wM1Y=M@;;mHZ2NFup=?8c{W-ta53}&{@0S z4S-j`^SrLMs;YI9L=sDLn*YrTe2YezDA3qr-ceq_lRzAawYAyyo^Mo{WjQ2M&d; zpKVRJa43yl$xQ3oHz0G-7OTZ}nn?!G|XQJTja8 zPS|2%+4+u8qctdo0!^ULhjE!vrtEj{OWq~ZZzmXKV$m+KfH*q;_ab7d!NK+4oHq0&@~cRp6`;0Vt=;8P{kybCl8f*CQ~#7UEx?&i_(+yT@MRvlB4a!YiMle`xqGpown4UR>{y**F8|n(iDsORz!sGCjAm2 zLH`#iiwvy|{r-Rit;nY|j&z=ssc)jF!e#lDP|dEzX|e=it{R6!d+0ie^B3DHX{?f> z4RGS!ZzDQ_0^abvH(Los8t!IxHNrUMNMoc@<=*i40r+Sz+zP=c9DXGJq5 zfd~_4!B;3MKurYC|DQYO!8!gC6IX|i4w6P+^SwouE5ruWl@Z)xAH-A@gQgFQTAl{i z5}20)w*+&YMm`sQ3{_vcUzi`OCv;PGx0>3+*SCUNmU4oD3dA$<<#003&-Xp9zpk!% zb@u0S^|UUJcNxyya$knOQe+^iquoSY?<7DMw=;xOC zI!6i5bU)5VYLlSl z=*@rhE%)gh@K0X+YyX{PaQQE{Z-XzuJx`!Zo?wR@;THMdm-o;o$ia6&%YXEL?f5^( z0AGFWT=UYd_Ni0#zta9qeB>7X75lGxjuw6fw*Qn@-Xy=Yj<3;4`f{(CqoIA292n&7 zN)bfpa%7Yd*c4ZDZET&E3@Rp*nQZ%zvhO$&MBA)2RDd7nxM^tbfVugkq3S7~*mY%r zU6C)7LkvkYCZCKdOgI1vq0}yJIlQYT@(0COxAm@Cb0V2j{XZs&7q$TvqMm$kmuNOx zKuNr^O^cKY*xK^;a-lfTl{3HBednAgiIu`Fr+Vu!HOvF=a>egoag0vFUDT8>ej>+O zJ>XRL<2-cT0pf)f2-f=yxBSBd-EyA;{AS+H4c1Psx37#2cm$$7eqcL?O@|ClgVOpo z#ycKn7+y*3hzEncrd`WA^avmfsOA7R@EzxWrrkXzxq*MIe2`{oFv^R#z|l~YV*G); zH@shJ_tpIYO_20}sX;X_!3zZcJaSF$?1}J_(EdRj9|qqW(Ae08ywg;F$S$8>sxpxr zz^88{T;tzaeV)ag9^V%k?ykPIleH#WY6$;FklLX=rZ!q#2#<&WQHVRk!IB(nd3_^z z>xV29J`N?zyU$&pGF2tC;-%d(nk!|r7fR^Q2JQP6k;1ps}ndvAQTngx_ zZx!yaVHu;1x9_vrLN^>dSG~3#xudY#_34*PNmK+O^EfW)POO9!Z0n?;q(3@Q!njLm zL0}8@5JuysV<(sYZ@&ld6J-C+5}~-sJ#j`p^8PW&i7-er)SAbF2IGyp#kp>3;DpYt zy3fsaH;QH#29}S>_QpLL^g1Q5azJMiEmr582>ZZR`IV+xjCeYpt^UD-1=zuG3p$L1 zMj<5yHQO+iD|~IwC>$!hdIM+LqSn17;X`QEz*zA015TZ+B~^GN{V<^CsAv#M0f*Qz zlXj0!m)o;VEV|zL++ot8YhxkfX3OW;#Akl1l6tf%Cgk_+?R|FE@4f2f$bL5W>q(6J zNL;`MKDBfW>BqtY6vgtoBWnuHU44SHG~FN<0Ck@aV3Lij8oR>3nC3Nwz&N08 zFwV+7xe+^gtHK9DT#lq&X(o`w`e_GQx{yPwRu^Cu+r-f~!i=0pOiLjayTirQwF;5_ zqbVK|-}-N?;$n$hX6CXKadPCUq%^qpu@5C1&AWtbsKWj>4j#n0nB+MCM9OJzZ))F4Y zkJwGn9u5)5B>4{R7HWL3qZ`nX6!8|onCi|gC@h|lVoT7y%#SSzc^@b*NEKHcQS}LrZL;#`wLrf|y1u;lIY^Ah zo<%+f;T8>$&+w9m)HSdR0?2~IEkp2F9-Mt>iGLc=zaTu6+kRx zX5{;nx{(Q%tTGHP6^vCmTyxP|r^fuAls`?16)40``W1^LN|UIsq9V;$QdM(k;IO_( zlH_rMN2VN!Q6ag+ie6Icl^~-)LRx5gEP7<`xtv~|O_ z98alerAd||rutG%t7@d6N)6L7*z4&ms#;8Uw?j_ul!S~EmU?3aiE1kqLbq*$d#=0l zm2N}jSaHFEIi`K@7ks;ZwBmAYms!^RU+yi++-i$;gvHGJkIem==jto<9AP*+U#VW5 zaUsEKleJCVbX~{uzKF>R-E|!I9&rBJT&pTUYrd`k4t}5b*_|8Y+)e=w*DVj&_}4F@ zX{Evx;tQZ2$I)%e27Oz6W7z2G+y<_}&2L1F!Ua2q?XWKbD?Oc?F88c&3n^Vs2SvdY z%JKfSAY~_rajlkPdg2>mJhp-VA&i-Dzp-Uox@TL4VSknDZechr+13rqbR3g^Q%klD z#vSw9!fplzlPxUEw7#)*+o0)MmTP@u@HUxla>wz$&UJT^w#{{*Y|nMvZ)(ZRFu1V3 zoh3UusAQ(?if?Jj!Z7Z=v5r%94c#<7+Z11btE-r=0R4wK-xH;@az=3eDR#xN*NQymHcPMU(+`$nhNe9tf~5 zl(~80E_W@*zPEVOCyp31bPuL(I_#by&m#1jsMCr3xBwIEVqxF^1@xm3`ZQbt_TGJ? z;n`x%MK})okJ~dVVLE3RXXm%!c`=LICb!756D$lgIhn*G5RvnXG%++kahl%y51@{6GFIO(V2iQtOF%acU{xt{Hr+)(7Q~}u=v*xPe*q^Gx6A04o_GnH$B&K zzWPHuxLnS4+hkw^J74*s9a^l=J<9+|=za@}^>o_d_DS11=5Khm_XQ_7s*V=HNPMqE%z)P)KZgwNj_Tc;n@h`+=mF`e}u1_ zhM^17GOcfC!FZ;Pd%|+=zpe=Sw{BaO%kGoq@lV-VL{>OlWdXKpT8{XtU1!)9%duV8 zz-qH zhV^?SwzLY))j#d%m~Uy_gUI4m>brEoRd8EphRYoGb;sj`>*@}-xNz;S7nl}BZGaOS z-dAtjTHK|ow)TBGzI*GWK~w^OjQb=vZk@AZ=%(SC%EM2Uo{Ji)RJXaKU^wGKxJIko zsVw*1vB=uX!W7tIUX|+{#>szEd#&HEZl^zA;CBzGM`Wcl!5YSW#3|9byLo zns-fK+~h|And0Bho2$QZ*+X^8+IsIYRTiuHG(vuD`uO~}*~ z2(EB-SHPl7r|juKgCn`nGK+2ltlc-F=z5NQho&iR+*+UZxQ;cExjL}BR=Mvm^T*hZ zj#~ENx^1&vx_e)v``f9}O{jZaV8nOEs^z&AB7!;La))mL5gmJvh%7Xx<~@5)sWc5s?8$6kFhkwqSca+~Z-1 zhhj}UG#O`(`EAt$c0F6i9+da$DAR3DPS{Y z)BUQuCa9GH12Bzy5Io0kgmJ#S!37p zbZl|K?m>pu?@K+NVdw^)hc(=Yi&e_bW?pzky9yx+g4`i?97u4XqS2-5vXzvL`$P#I$xeOT2+G0Qr>}%H#7&aqu4cooQJx>H7Id#>%DK0sao#aaZI$#fPdGZ~p!=#~EvqHp z+6E<*Yyu_jGW$M5$!uraZk%k}2z0Fu6Hj*;yAPGUZg&S*E|&W@xcyZh?x1D5m{z|= z3b1e-&NgLcjwS9hnecYn?~2TY8hO+rnPMi5{HWZUMK+|z zvH6;ZZ;$4= z0tOGYFq+LT9*tod+Ewgp*X#{ah6WC2(@&g0ea?Lk<71s z;vcjZl%%iCFCzFK?7N0(Gub@X{1FfvT`zqy+TWy6^PFrNt>S#q@;OkThDZ+7L~mt) zU9k7s8K$jNtc5e}@vG-roXk@#O(JbHNrTHMJEeE>MRpZm;RtH6nu?j^+B81TU=1|5 zKi5VXqJRd2G{;xg$l=kMc9>=`z~ma=g4z3t21|m;PEtRNvLm<^@le86AIFJ585Y=9Lq)vbLPfl& zVP4ccN=~&u{Nv^8pWeKB{N&lmbJWIOkT&*2DPs|k>q+R({XCkV_}LVj8PJ{K6=Oq8 zKpWx{z^d|-A}+Q_1{}0S!)4}xnZn}6!1K&7^iE#?`pfY3vsW))ztyK<*G4(EVR@GQ znfRZdUs4Vm@QgGdgdY;;zzc&>=mdrx(kQcue?3Z5`E=q>Nj#NOfu8U)e=0NL3bI9_ z!{^c&j_gmg3COegzdXhnZn1C^?UTd@m8vaohnV34T_2cQS8|en|RNf^L@Tq}JD^DacpYpaK!;SpXtIZXBr$ z#{xxXnNHUv8zlmfPvUAC51vzY-$fK$Rv%p9BwO`ySj@aJE?%k}i2HaECFPT~50%tZ zqpt@25$+ZW1=$+uA_0&cYq&3fX~?Pzs6EthQMuyMQTR~9uL68=`;p1vhDUP7NDo8g z4s!QrSuy7i@&pw!G1Q?xFFC|-Qu}E$m*vat^wK4*5W?avRIIp+j{h-3VFux7VWt&n zn%TU_F{Rf&e`$L4W_`*85Hw2^RWiZ%5!zZ$!ln7u!rX9O|$Bbv?%ou`{l)Qw@4 zsBrU8woIjyO$Z%Tp@?Q*eo|nA1Yw{SSMW*tf&_!iS5~N!&-{Yl2F{q7p^nuADhiL^&{BB@! zP3c-%14@>wdo?I=%i&D7TTrr@?zo=gnj1oCWJO$ze0#(TMplQCX@g?y%7gX)c_=Yg zIhhRjO*f+rCqQ#pDVUL?0zSL$ zPQV13hTF&TxUpR*F<}s2Uk6I2rMopKS+?B?BvcAanVoV8iPf_$ zb2sXP<(gEwp$kfYSzG3K#=1aqKnB&d2gCM^4!Pudx@R)xZK^$lEc9$sFlO4k6H1=W zg<38tV!sr$I~1@LWS@H89x*{-=8a7{jJ4FS5o_2;vEdY19QK zL<8F}9cL$C0xO^wOs=zCU7QO+A^Npg9_Zb5EzhyB(0&_Cm?v!8&ZY{4X}ESUwgR7z zx_KV(REOL4ZX^>Jfkv-n5?j{t48hlBdcxJ6x@0mur@`|GN;V#EVPa$bfeA2%UEg%9 zmi|x@s|_FD5zjMv)jZR3whARnSZsYDnRQ*<^q9_DK*DD0HWQw`A&?wA1#Co-Gjh#N zAX!-4lzY1oOfF!z=mC;tvh9)*TbJKiMuXom!{VOqw1I?W=>Y_rY7eHvoNyG4yfGVf z0tu~JZkq0nfaKbuSH}}u=677I>mssP!}-7j2nXG4feGXn<^T$7LooT{&@kYx<@%jq zA~bLfXGcn&<#>Gzk2}tG;bbyzZ6(icP?PXDH#=CKjTVgo8sMg2@sm~?wldQFa zwZdvZ$+S$j?Rx;Ot$W-wiS{@0ft3WfVetfWWpW-zHO?6`#~Ut%SU|7+%6a)4hR*++y#tWqdz-EY*#38Zrs&W04J6T|B#W~SPb=d# znwuMEd08SH2V;YqM##HhggD^Yma`+DgDHh!cb1p8?Wlz*-1YOkcq7lt@ot;(X%FF|X-dORmtX{{Ex8>NMSXU%*(a`QpkVzdXxNGX%axH69kAn$(I2@Ty z;EuY<2v7hN%UwC|IOoFXO-k9aTgWu5_0L+=1cQ*}HjQz-CVdz-Fs4Ij{ zlxjP+aCa2;Gx{@!*p^RPte@T`TI|lnMC3M%JgjGGZmfX^f^&e79lG5T32^{32zPWG z#$b&1#wsLg?oudOR*RBM%V?7lXV`|!biK__Er~!l!0=e;bb<-cfSWrD`x#&Y_Axw@ znOinz*EN%xKoTx<+cF7cy6ynFZ5*ozgf+GTQ$#%K(i{i}JiaS#$7Q6f`^3_A^>4wQ z^<8((^gDyKZ6uBj+_@w5`;GMPNJNeoaVxY$Cx`$AIGD0K@;88LXZBmN=5`Z?I9pe5 zuG>Yfm;*p%BvJY>QdVF|jhM<~HZye*x|Umr}iEBd&d#GNfd$#uQ; zqkauya=^Z|+fed!P*b_LsUtiXyC8A6=Z(f)P$Dd_`0n&Ov)_5gWGr%v(F*3V^-t2; z&Acx3=(XTvx>#k&=9{W`hBx-G)Sc^^olFmzs%hGuwX5_y(QDDdmTXxxS(djxoEiaI z3mH%wPWYe;%JRm?G=0+;TM^bm?>Y^Gc;Fa-B0EyWIaWjI)A3|U-7V6Sv9`tJG*eQ9 z=^7p9G`VBy0&#BRbXHf`rZe)vxaFPn!6~9c8ncj@Jt%#=I?`dS+`E5O5mltR{2qE+~1r zYrBkZYBTxF_aiYLSwXkWZtPAW?>}Z?8TNJy5HV+c{{wGC06oigIz$s5gf>mn z-82Bm+|k&#Bh%vJP89-)6(rw|>JKr~7QG5NbJ(tlNi+|anYL@TPik@(txwn7xXyzZ zn<4iM%Nd(pzC6MLA)FnZEn}v~dmXOsmXpK5F!9zuXVIwhVA@8Xvcy61ukUcJGd#y~c{|68dzNm(-^Rv{BEts= zc~NKtqfSx+QZ+4WM|JVAwn1-lxan;<5zkrw^k1EjY_BUf+yv!ZfKjpWF$>QKEt`iy z6pF567D%TSv*p3}UBJj-z0q{xZTV!ntVghBq_!@UT+wxq)-){Lu)vzxRCVwi(+$Rc z@E$gwds_K+#;EnBo}`|!+$cL;XVY8$jBrZeJFHU$#rp+6csP9Vme{0@mz67awvNTR%a(SyR`uytcU{@SF^Wy*Do81>7_OjJ18Ab#gry>u|Wvj`Ro1 zvwKflSX+)TtzY=KY0`r7V71dz1h;j=^X^a)FZ8V_vILgI>wuA9j&3kh7LVBhjBKNy zm@v$|WzU1L{)w1+wAO}A;O$H=0g9gk+qkiN$iT|SA&*?E%kh9%z!;N7Mt1=vP>y<2 zA7mMoU1m)J!>b3EOw74!s}Gh;Br^D>Y97ERj8H)UVcZ2K!~%}hT6Y8{yZN^?I35VKaO{m(*cG)PBrsjm+Yv2UmSOZ~iOT}D+bn70 zuAevAsN&^uPtH#JO^qfF3O9E>bL@92A5BwS(a;b(UN26CSyCIB!~C9hg{1t zw_fPI>wMXbFaoo-k?Uo0!OS)(*_aFAF=u1{0}HGmVB^TJ1J=p&fTudHBX(7{u-B^f zueKlTo6aSBive5288MG4tjGvu3L^R%gt|eTu{Ie7APsc z)s?hIXaTPp_pX6ft6qnB*QEcogSocTEKg*zju4dV*}4f-!8dQL09)z0emHiz681pv z&<$ZQ_uhf1S-;x4BZy?y^O~_-*2pC{Ivhx@=jy=P*~S_F;aH4DzGJhnQ@ubCnPBzW z-hG3S$r=Tu#9d+Ji1m%$dL-9kpz66@Fft9$;Wl<4MYcUQnDD|d=z$Rii#J=eB2&Uy_&*hG%OEYloy!U*Wpw1s8eH#!nqE-)Zu z675=a)ND=SFuQ|}Ji~xJ^!UcrR7OCBqcDIWLe>c*JS*yoU8(5i4b_mju~kBHHxNgS zvXcx;ceFvov2KtIb#A0FIFS_w!X5`fmwS&;z_111Rb8U~L`qYbTLlplrghiMBCrrT z&dYchViktDaj`sZ`rNh*3!X6hfCQ}0V7n``+ZS`PwyW4N?%K}!1*;np2{@+J#qL0D z5VpH9n4ECrhLH`M!=g@h=jhC0yJ}3rSiciiRt?=Em>6RloDyV>{xPssZOy@BTsJ+j zx#nO;q32kx&4Ny9BJB~|UfTCfb6{rE!^iHdIymdIyjC3uJpS5-lC1*(3|a8y6c)7S0`dV=X`8& z(;9i5k_pDQY_Y4V)%=F$bKaI&o+~yuZ|gK7?V_XpyT`dM;E#<wCcO3_CO; zr<0mM`nkfgtb6BdRRO#$!by0Bw{HBesgMg3I~=sRp3QX2aP5t&!GmRM8HVHAj^Lg2 zgy#udzN^ev}XaDPj?O)hNKHD*=rpZmmA`N zUR59GdIt!_yGU!^ZyT?5rAoXe*yq|M3933 literal 0 HcmV?d00001 diff --git a/eval/dataset/arms/copilot_v11_deep_val/pr4816.cost.json b/eval/dataset/arms/copilot_v11_deep_val/pr4816.cost.json new file mode 100644 index 000000000..90404d600 --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_val/pr4816.cost.json @@ -0,0 +1,113 @@ +{ + "wall_s": 935.6, + "split": "val", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v11_deep_val/runs/pr4816/run-20260813-140328-27c7da", + "recorded_at": "2026-08-13T14:19:02", + "attempts": [ + { + "invocation_id": "inv-8e2ed7b2", + "outcome": "run:run-20260813-140328-27c7da", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-140328-27c7da", + "llm_calls": 44, + "input_tokens": 507301, + "output_tokens": 74874, + "usd": 0.297442, + "source": "spans" + } + ], + "attempt_usd_total": 0.297442, + "llm_calls": 44, + "input_tokens": 507301, + "output_tokens": 74874, + "usd": 0.297442, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-140328-27c7da", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.2974, + "minutes": 15.56, + "ci_minutes": 0, + "input_tokens": 507301, + "output_tokens": 74874, + "tool_calls": 74, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.5668, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 44, + "by_role": { + "planner": { + "usd": 0.000515, + "input_tokens": 87, + "output_tokens": 400, + "calls": 1 + }, + "agent": { + "usd": 0.274234, + "input_tokens": 500102, + "output_tokens": 55622, + "calls": 41 + }, + "reducer": { + "usd": 0.022692, + "input_tokens": 7112, + "output_tokens": 18852, + "calls": 2 + } + }, + "cache_read_tokens": 2892928, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 3.26, + "gate": 1.87, + "review": 928.46, + "report": 0.07 + }, + "time_to_first_result_s": 933.63 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4816.trace.json.gz", + "trace_bytes": 14821 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v11_deep_val/pr4816.md b/eval/dataset/arms/copilot_v11_deep_val/pr4816.md new file mode 100644 index 000000000..a859c277c --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_val/pr4816.md @@ -0,0 +1,38 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4816, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head e1dac23ac02c) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 2 finding(s) below | +| Security | no finding reported | +| Docs / comments | 1 finding(s) below | +| Behavior / compatibility | 2 finding(s) below | +| Correctness | no finding reported | + +**Validated:** +- [sweep] grep 'openai_serving_tokenization' repo-wide -> 0 matches; rename is complete across api_server.py:754,1017,1157,1248,1286,1329 and test_serving_speech.py:2533,2546,2615. +- [sweep] grep 'serving_tokenization' repo-wide -> only api_server.py producers/consumers and the three test assignments; no stale consumers left. +- [validated] api_server.py omni_init_app_state pure-diffusion branch (line 754) and LLM branch (line 1017) both set `serving_tokenization`; no other state attribute uses the old name. +- [sweep] read create_chat_completion (1157), create_speech (1248), create_speech_batch (1286), create_audio_generate (1329): all None-handler branches consistently use `getattr(..., "serving_tokenization", None)`. +- [sweep] read _create_speech_error_json_response (api_server.py:369-376): calls `base(raw_request).create_error_response(...)`; with pure-diffusion `state.serving_tokenization = None`, upstream `base()` returns None -> AttributeError -> 500 (residual defect). +- [sweep] read _patch_api_server_base (test_serving_speech.py:2339-2353): patches `api_server_module.base`, so list_voices/upload_voice/delete_voice tests never execute the real `base()` that reads `state.serving_tokenization`; the fixed path lacks regression coverage. +- [validated] The three modified tests (2533/2546/2615) set `state.serving_tokenization = fake_base`, matching the production getattr; without the rename the handler-None branch would raise HTTPException and fail the test, so the test changes are required and correct. +- [sweep] grep 'instrumentator' and 'from vllm.entrypoints.serve.instrumentator.basic import base' -> only api_server.py:67 imports `base`; no other in-repo consumer of the renamed attribute. + +`vllm_omni/entrypoints/openai/api_server.py:754` [major] — Line 754 changes the pure-diffusion app-state slot from `state.openai_serving_tokenization = None` to `state.serving_tokenization = None`. Upstream `base(raw_request)` forwards to `tokenization()`, which returns `request.app.state.serving_tokenization`; in pure-diffusion mode that is now `None`, so `_create_speech_error_json_response` still executes `base(raw_request).create_error_response(...)` at api_server.py:376 and raises `AttributeError: 'NoneType' object has no attribute 'create_error_response'` whenever a speech error reaches the helper (e.g. `upload_voice` raising `ValueError` for undecodable audio). Provide a non-None error responder in pure-diffusion mode or make the helper None-safe (fall back to a locally built `ErrorResponse`/`HTTPException`). (evidence: Read api_server.py:369-376 and 1374-1542; read serving_speech.py `OmniOpenAIServingSpeech.for_diffusion` and `upload_voice` (ValueError on decode failure). PR description quotes upstream `tokenization(request)` returning `request.app.state.serving_tokenization`.) + +`setup.py:69` [minor] — setup.py's install_requires is derived only from requirements/{device}.txt, and a grep of requirements/*.txt shows no vllm version pin. Since vllm_omni/entrypoints/openai/api_server.py already imports post-refactor vllm modules (`vllm.entrypoints.serve.tokenize.serving.ServingTokenization` at line 69 and `vllm.entrypoints.serve.instrumentator.basic.base` at line 67), consider declaring the minimum supported vllm version explicitly in the packaging requirements to make the dependency contract clear. (evidence: setup.py + requirements/*.txt -> no vllm version pin declared, so the oldest supported vllm cannot be checked; api_server.py already imports `vllm.entrypoints.serve.tokenize.serving.ServingTokenization` and `serve.instrumentator.basic.base` (post-refactor modules), implying the repo already requires the new vllm layout.) + +`vllm_omni/entrypoints/openai/api_server.py:66` [minor] — At vllm_omni/entrypoints/openai/api_server.py:65-67 the comment says to 'Keep a fallback for older/newer upstream layouts during rebase windows', but line 67 is a direct `from vllm.entrypoints.serve.instrumentator.basic import base` with no try/except or legacy-path fallback. This is pre-existing and the import block is untouched by this diff; maintainers may update or remove the stale comment separately. (evidence: Stale import comment at api_server.py:65-67 claims 'Keep a fallback for older/newer upstream layouts' but the import is direct and no fallback exists (pre-existing, imports untouched by this diff).) + +`gate_report:1` [minor] — Gate report shows buildkite/vllm-omni-amd-ci, buildkite/vllm-omni-intel-ci, and buildkite/vllm-omni-npu-ci failing. The diff only renames `openai_serving_tokenization` to `serving_tokenization` in api_server.py and CPU/mock tests, so it does not plausibly affect those hardware lanes. Please confirm these are pre-existing/expected and tracked separately so they do not block merge. (evidence: gate_report: amd/intel/npu lanes failing; this rename in api_server.py + CPU/mock test does not plausibly touch those hardware lanes, so no attribution comment added.) + +`tests/entrypoints/openai_api/test_serving_speech.py:?` [minor] — Lines 2533/2546/2615 only rename the app-state attribute in the three `without_handler` tests, which call create_speech/create_speech_batch/create_audio_generate and therefore only hit the direct `getattr(raw_request.app.state, "serving_tokenization", None)` branch (api_server.py:1248/1286/1329). The voice endpoint error path goes through `_create_speech_error_json_response` -> `base(raw_request)` (api_server.py:369-376) and is covered only by tests that mock `api_server_module.base` via `_patch_api_server_base` (test_serving_speech.py:2339), so the real `base()` that reads `state.serving_tokenization` is never exercised. Consider a regression test that sets `raw_request.app.state.serving_tokenization` to a fake exposing `create_error_response`, does NOT patch `api_server_module.base`, and drives upload_voice/delete_voice/list_voices into an error branch. (evidence: Read _patch_api_server_base at test_serving_speech.py:2339-2353 and api_server.py:1374-1542; the modified tests call create_speech/create_speech_batch/create_audio_generate, none of which invoke api_server_module.base.) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v11_deep_val/pr4816.trace.json.gz b/eval/dataset/arms/copilot_v11_deep_val/pr4816.trace.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..f1f518116074e569b335c755e9fd2fb981907610 GIT binary patch literal 14821 zcmb8UV~i$V^d#K2ZQDJqXWF)H+j!d6v~AnAZA{y?ZO{Ar=iN;<`)zMdZt7M}PIAB1 zsUnVo0r{T;_V3y^%6U_~b+0b}s3I{roa7@gt$T-X%!z6zf>^`N&5iy+m}D-Vg$fo) zNP2~$b|Y?$d5d}7MIM|&FbGX*UZW9gR+QrYivLz4@5i@pb=K>Br{VrHjogHpaP)%h zXANH=%;TwT_jRcRPd-S&(b{cYEFIc9y-KLAZg0($SIrT_K9Klrejm%7Aua+bDRoaj ztSbRO%-d0kA1-Wwg@DHEL%&ZTzX+XU;}gb!%IGIzC-3APBabip<E{m?e%b!H)`51H({&A&3_MZZPfn$|8CylJASj4YLo*4n6j)i&;q#c&@Ea;1cH@dpa`hqQ?xhJFXm#5Ty`~ z>QXn=$=!_lt-&`Im0a&A{HVmb`}p-N-r_83Z@EeO+FgH2%{ObtWYI&T)O##rv|ei$ zO6vZVYw@WIH;blZOKaA(<=PS5fyo>_Ze-j83Mv%0NK;0-EP)XkU?>BknhfWe?MRZh z_HdQ!Xu9=CVD>nEp_Lnrwiy`;5d_m=^NJJaedlf@y6|$8nT{XR*$_mO9a@o~G zasJLpLT*Y1Yr=E>ypBZv?8Yp_Oz!i1{oCwm})Fu;{yJ{x17A>5sefRIx1?(OC}qvbztIOWZ zDU#5X42X*lYokV!40a)?nRk2m-VKosKH>ug$tyrqd$U-kB8+SkhC+|;<2@%)`ARhv zXa#Xu@V}nD6EJ!$Jm128!#ldzHk`!!`8 zzwl-Y8_#zg{Q#K<>v5z1J_Y`*hO?QZ%SAU_gd9Fr{HkRdjJQ8YL47gmy>BT9OB3J_ z$K_oONJ@eib76^kw=7&?o21@1%{(;yBR%4&$iqd5FnG_A>r*p&O2>4=&L7A<6EmS^ zF1l_A;**HK9&8YXM*|3v%lgede3suQ2+!bqs3bW!o_oafL$uDRw@r#y+{&yF=Jv$%qXb z#m(XUHE0ygtPpnh0E#1UhtnPsLd*{_?)}R<>mwWNVjm0-F=q-9M+In<9a_ZRZ&dCC zax&jsC+oAHVf6)=5hn+M?Vd?T+V{yUzC(xF(Z~T~7-h-9SqFl-Xh0AplH;|*Y6{)S zgMB-cdG{S_fGov9MLU{aS&#?h`Q0CUy}}OU(EFI4_TgdmgF;n;?Bbn>XOgRn`*J9W z|)JzJU*-|9H%xqHj^RlY*Gv&PYg$VS5m_g#r#*fw2IV zCQmrocEAk>R5X;4ub?11$T|#T0S!Lb`~VkUEPTj@CT&bDilRF|g|jytL%xvRw8MFo zsc_N8KSWVcJaEVnVr>1;D~#fRR1$(1u);>M#H*MgQo|pR7*TLTS4ori7@~>bL^hO5d1fHww~rfzRAPxpcfkP0`D+8P(_vK zHI84gj)XYo5Qhx!egyghg#R+ZJP9&K*q(Zbc%~){+wU9T((It-zl%}0lHebu@GjE!5|j84I^hG+&Jfa<)1=(|)B&Yau0TYP zm^P|&3zSfb<r%MJYoZ4OZg z6ecl_X1m2tSV=?!lcHJ*JOs#Um00lBvKNF@&|&ypWec~(jie&DI7`gm?fw2+ncNF+ zGMqy8Q`>T&9Txl;8QjCe1L+NJ>@C(`Zjn_M`4-e4F}oF3*hDG4R3yKfcCyPcG3NFB z;oCGD5uq7D7Qo+2MVN<|L)CZnz+7kKRJ4Y2m@vA*TP|eAd6Dsh7gbV0Lk}&;zo&q z>Gt6soFCXcOQj(w1YH7O_b(_^p!4e_Ig*jF(mO#%ZyHNkLv`p8hlLgxdb1yiQLc`hlHxbOC z>z*>kJ-RmV_fJGKU;@gQSD6g8Z?!)BM2=Hb^~S$?romJ<)57V`r~%yh0Y-OJyKz(p zZjJ(`Eq;#w_3ADgp}GMQ)euk+q4K)=+d#(3F9NP7*?zmJ@~eFEt;u+gDjog=(HR+G zX6tR0U4%@oh4$+P0){(ol(xqJUYsHyMMNVr(S_x4;4Owflm@@wQKXY7uvm1n!DCvD z^M{=jpJr%NB7lU2T*>^Y?l0Qn5tS-0$Y2U5^94|g)BID2xxL$ zNJbG;fDt&2qMS{RCs@)&{F?I66fiph)iBGV`}-+5m-9!U`1T#B&A z4=T|(TO$HoR2xkEoJHB=iHxt0#u9fJfCGn7IKoyA@|02Vf$)dC0-5)oH&&_2iWYe$ z{)N+=X&%&z<}#hz6nJa|7Kwgiz@p4T0}sX|?ruIGNLqrx?M+Zp!Xc%tlf)6Zj^9jC zX;BZ$My3#?wWP6+fj^;Yc|Vd$eJfE+*t&l>23LTh-B zKEW%P(DZcpLXXG1`z2;@FY)gDYd2uIEx13%eao+LFM;fb<04G-zVCL-@K*PVIM_KI zBrfbQT@^YT$^s0zh($UelSSHq7uS+-Hal*}=5SZ=LLuhB>n>|Z#;~=1Rb%_y5hKP{FxS_Ffae#{YW?Sr_)N-qBl zXV);q$Z83hq&L9igK7v~&62XAMAY2CRkJ~TN=iW#wY-$9Uu73Mg-dRXbbkB1$`MfLa- zhH{uGA#2ISVbT1L>Pd%HxA*5y@O+P{-h!1L<5ly2s@nu|meMyDpPA?o zOzdb8-W8vXuwdAHAS?hF_@(cyBBwJOj zS`q>Gy>fDkcrk`aBDO}YrWIBIP|YuiE|;G8dN9INcaO7I$qVUK!+6WjU~R+(x*ds{}XOoJ(5+&A|kRIu|3PYxdbhSdZn8}CxbppFxm|0pz;;FgoJT)P9B@b+ zS|AVB)T z^>_c(T)}D!{8D6Xw1C=D|HA?qRmJJb`)EnO;zYWgHEn{RAM}P3sXTb1Jfs{11t#gk19qA zMsaMQrw6K$0Q=A9XPbj{BL6C@C&q}O*I?n3OO72e6kKb<4f`~6yYWi6qo+w5U0cgMRkoyhw#7?dd!rYt7 z-n_m}0q|Q~vecbVXdpsrU*3IrNQf-I_7VXxRPVI>$jG>(w;$lMrDdi=f056kaf$f0 zhZcnDB~GZDj+%8-Sp+m=g?B|LLt3)n02hmW7zs;S7{vqMPuE&($>4Kzr>C2(3C!x0ze6n`)k9G z%mX7lf%|p|CQOs|_YBdZ$VXF3O|phv$*`8VTMX#VMMUe%_G*+6JlJq9()8Jw*MD;u zJzOkY9ByoRG?XRk9h`1}D~G^PHusIAnQ!WL=cX^R51_%a6;I~-FSfB7^K5dS30-=&KRP5hw8(t)?gM1Vp3G{u#uQeU9))Um_}ynL=BlL%`8 zpqBpgQO*MJQm2=|XXmX3eTvYIris3jRS7RjmRdCUQLO~uqGBB< zr#^)#G+f3a!oUv-%LCQ`K+_yyAnA(28q(O|B7uZa##V~1q_%stO{jI8y+`9IC$E~Z zz443eRHLo=qL-G~Gf9iZ!?8tYG)GK|XRaP$)kw#o3z&9@dYSs4fX8>nl`8@@?Z2@u zv#0#8ceZ#S=d!L|3v~Ge$(LgpRy?s9M1-Lv>_dKL?~iEddPYi1kh&wd{4Cl?w*jPv z;>iR3&w&Re)0|d;j|V`#^A6`$Yn&cIU6)+)-$p$nB*@LB%|kmu zuodaj52S6So_u~0(VPo+HRH|b0S1Ye;Rl+Dm4ke&VOiO@(Ia`0lM_vQHgSjvOYxqy z9|R6-#!J+$!BfyHSnUYh7u>u-#nTtqrD^lSTsXD1^c3dH5KQR4ZF~}n&wwq2y&fnO z;WS-uutq?s#(h%p@9%uG=sir?+Tej^0V#>^WSAC^WiLKZ>nV~&m-sF(MLK946sy$o zfeXVRj_>pnY_^XfEHFvf_~WpLpxLJvy0uQdK2*W;p7?YZ&k=8T#9oX4_H%eo-lcKL5t>7 z^}TY&oDV}EwyHraCSnO$N73zIDAXBDKq5I74KhJ+-_rZt#k0)e(8N^Q=m2CUedLJI zuo#XOFE>L|BNV0+_CR3=6(x1G|97+U@E@nLt+#td0_l%4oPK)8g^dlt{)3zWQQuNW zO1{#?-x7#B+oc3;ZGZL3L=xr1DUz>7dQAdHOtewk7{;(-8-;o`W0T8<-nw8iT`WQ{u5~PJ7--L$oU01phi8U5ZMvG6KwTW{qY?= zYCnC$K%M zpF&LR^y^Pxj9(T-e6Z*{4+j2V5@aZCLcTeK5#P$%;w5nbN zUQ(>{R~(f#Pswju6Us+X2uJ&dCUNce{)N!L z%thnY{9ka82Q;JO#ig3lD{_-`XPbhOt6(OKsiQpshtGrU;4#kTtPucael8)e$o-m2 zaP5HK;5GsJu}ntA=+I@})^jQi_CW~}>ze0v5Lx*Ecg+zBL+zCMLH`muU1&l?8wO;! zr>*eJlg|~>)uK%%e0o_AHD>x@*}2)7+2uE_?^C5fY;&9Lvspa-%+;dg!mBkw)4p^< z0jqe&u~8}5Z89Isd=aQYG!!lpw8}^iUnZ{E<^L9jEaKfB-{%vNyQ$4n0jN3yKY-`+ zf$!boX42}c0E9rF&(Gmh(q7uBK=&790|cfKcP++|E$9)L8l-4a>H0@Y#HyVay;Ep{7SBg=)8qluj9yj zwnErCo{93o$+sHfm;N*E(v84N@@j=(cjUvkrMlZ~An#=VECgGJ^1i3o`lydOo7L0O zk#t3Yz1>49UJ$~e=RFM})1(dv?s4RPb|sRf)tBwPo~T)sYg`eqIa-T3AvkKnG@01F zYfnyJ)|+b0w$s9BUgD1*P-4-qkDf52kHmf(K(9xU5+T!FiPSM5?>?O0EO^VP%c1OQ`8~Nsvr*$>zdt`=5(=9grBxE z=e$IZL7q;TUE{$^k6SAIPeZjw&dTS_+fONZPDu?%F5X(tc4|_|N2z7ivsI~L2~|9# zzT8;3TD*)d-&k;Tz+oJbc=dq)0)A;E^+=E5WQWB0;Rw5Wn44V&;;YU!GNvt>00-yN zgFoyn+$tf#dJHfN1%p<7e^n;dV3PR!xvnaSa2g#9lZiL8aZ znwWcb1UW~80=Y0^e*QXUFX^I~Z}&AC^5ZR_>c?4(62}u~HpnReR(nY6-M6(q`_b0O zRGgjgHAdbZ_*2%44T|Vk+BhABKtlvw-%RuLq`K&NN?kn$n(!}ruo?IQtn<^>$Wb_p z9C6u59f$3n{n?0FHp52GP@rBVf0pgLr|X93Gld0#a_tvCW&u;r(#8L_^{7oF*VU-! zGC)EB<1m@wS^XBHQ{eUZi7m!E+fu?4IOGg>&>+IA+xdEWNl&IdthE~Wx7N^{{uGUb z(Bs*{P`SaxhAWE!Z?CUp@?7tLF-sj2e#zhfRJjcV-;7o9KuYcJ)V|}d6ZwWJ{`G>%rLo0b6)zWWOWg@ zuZkHbrenY1F^mwOi@#D60RJfb{?8zYoz1_WN~3Nk)%Bbtv}V?R!G++saSguapO-C0 zvCR(pb6_$R$!@K>J@~gdA=nD}jNU>Kp8g3Y%=4nY#wP!uX}69|$?Ep>V0KcqLVa=Z zObvoh(?m1W;Z3H|KXi~`NKngpMICU(OC8@$;hSDT*e2Xpq1{URYMb7E(+sdY)%_C> z$TX->8v=++sX?!a#{t3P-?cl-`&Zz)QWj7`5FgCQU~hliBv1%&^Om0XV@%tdp`dg2 zv#|Eq-FCDag%-M7sd~@;B&NuGK6SQ5#%X;d9zT?!B#Tzx>REb+)Y>XRL~pw}_k>j0 z%H~f^A=1ID^4%Di27_|ONkT`mIL0ejS$F6|Dr1=`@vFLP$UJ*L`pXTVKxmkhPxf*r zSI|avBJd@F5xFvFyhzOrfKM|*yo*VGo~vFA9L$;Y9*0E$Llkpk@HIUaq|zxpI_!|l zK<`HuilS>rpDk z&L#`7k7>8;sVPHMmc_Sn6O2~k$d=#Y#%1O--$dXSWY3V-C5^;U_6OF4-R#5T{iaUTG~bS(Hoa?nK2pNRqa8Z`Vg5v zHR8*Y@_9cO{9Ty#KK$zekZ@DYE2SLWJ66FudQ<9FhCG+~UT)aPx3XcsFKRT5LVR6` z#8*4jma5UKgrw~COt_k&ER5P7DsUD?qB8#;l|f;dC}(9kA??=DZ`-?0?MGVu1~0`6 z#qpp!xKMNRB*-1obDdWH+%PZv!3XP)dXa9{S=jM)8JM- zX?s$kqVMSN+$&6q94haq*m#u1-(F@JXxA?4gQJ2BKgRMutdpbZ>i*H8X)!nJq=G`s85kwRs6rm{ zQ{bOuTaF!1!fp$CM=R>nkB}iLjzG`%aa!BIv#*aCfag1LUoW@ky9p{niN}Bu<)PM# zu_a>7%AEE8d5xJ`=GevCK}>j=e~Ck->a(q2F|~xs_#X^XrfZ@)jQ)|en_tNAlq4&u z_5-;2Bx9eUTY1i>;(gP~#8mn$q~dNjOrJTb)La8xsch+^iOfTlQe=q4ToX!{tXp8K zRE%}7fO%#@YF?k5(n0J(b_w_VKMt;gHkVEtUfHcgr}V3b44x8kjaa9SSyz+JCPi+c zTKMHw;2hU*P5%R5tTfVF3iAGgU4zwnMk}=}HvbXy(;JxPcA`}=mj6Mb#s6ASHFR{g z{&NjmM(pgI{(l(Nz9FW2O;-DV{rJBpwo^-ske!`kS9q1)@Ur@2^Z#!BZ_IyOO}Ev5 z9shBOX{|>kdc&@HQSH4LRqdtisx}Vgs?k~h7Jhc087DUh)^3~)n`#neX6W2hclZ4q zn$z^}W6$F`-SszIa#^&B<&6*fbN3%AguWZ|{%&X#ZB@38^HYqd@sc8crjaf{;g0WZ zg!#r$cb{-UUh(&zO<26lTSi!hi9Q{}bD&dS9Y}=fY&Gywam#84c@y4<+Y{WJM@<$Q zw!cI;{UXFFWrB;2WrclI7#qF)Y!`a_X=o6@!xjelD)5cg-50_%dqck+%44g-LXI7E zl=PYoyag$x-3cIJs3leaTS{@fE33~tR7w_F&NC!Kat7RerZIa-99MY%6ggSsCP4wseG_B2qkb>*nHs92x{{PUq5I&=fBFLS`Q1|& z2BYSgM}zmM7P@FYL)=Zo5hdVu(Q`6`@LwmFR){Xb8o@DWS9=Z&6ZYsFhQ?%-ds^<8 zvln{aJWjk~O<1QMWM^IZMs=+R(1USgKAYBuNB@iAL?^2IR(|PuKc5?p($OrMdf|D+ z9o~p>DsK0Eef%6tVMh;(#z2#X-_GXH;Fo;cE->chX;g>IxM7=yA7pX4G7u@WXMo&Dcg-xn`FrkW(-Kk?mSEo9Hp(JUF){ce z;CP2<{)&>a70?!W&sU=6<`=<_zylFLKL1B_ur0pKf?5i|gS#JQZ;8r2;|{*s`d;0g zLrIOFWPGNMWN%F48vfj{39RB`m2g!tS;R_UQ~fMHYV>t`J#rn6%rYB%efKy8D822z zXl$mZPxVYp>bmN0rhjgjH+iW$t3x4E=g_;o~E;jHxkGue8%O$rna@@5g=peKlNfO zFRPfDBx!L+!8RI%Q^KEN_{98V`QcmFZyj%d?4>1;P8q*neEmtITG_LEmy;OcsVK>b z(M7WP#EO>bNDd=dEIGXgk%W@WhPdjas5|Sy4AEr>)^=pt6aNEY;g>x#ZYGp-prCAV z?8BXg_7tVwGEQjvWCevI_ZSt~C|1?DR9v#wrTFc4oG6w0%pqiSq{7uP<>YbL^8mVZ zUqNEStLO*KPk0FnA|-KtMes#d6{t$r(pg2s5Ca7OiOPmm7-m(Jv2mKeIvu@E8%qQc=v(wa!hmTs#>v1=Kyt zCxnqXMGoDDhzQ=ct3Xklmz0c>OlOALie-9$l@i@KdBvhqqV0=f2<6I+iOCH6t-ycE zEaYsKnsp4ewiM=Rx!@+!!Ssol{gF3Gh4`ILQE6?vy&ZCX@xblTfVG<`ZXJx-dj3K5 z^?@~dk-7ECyNkIyXvnvSxzI=if@!-_+>)&(=d)BwrKhrn*6R`HVjUZQ@FQglMDzk% ztUsZ^e{v|{BIp>>C4B=YdbjT9shyx`e|uF>)_D=vb&2F`Blx_ZnTOPvePx_|zV(e= zw!h$VMvBe|PCuFXj_b&T(6iZ^PGv&h<>up%p9NQTW;9G2^6&nYBee_cU?!aP>~;YXByA2J#-$SnO3Iys;f;`=NUXAjK0zeK&2ShI zdGw_wUx9{aQY~5^_jioQh7~k=E1sxfSHU)tVrAN~AdaCieU?s(ve?Pp{Ac{14#alh zwdfLcoo;*?xjc@Z5KH7(JN=-jZTcBkg$6Ln&SV^a&K&0CHc!c32glYs zM{}?+wOjbZ2V<5V!Yj0Y>EKk`{i^?C_d*k~l~#J~8GG#qF3Q9()?C7v{wjyC!ZCoG z8taq1paioC3^^Qk;Jlp=VJBfy`4zSvY9K_3)^r_Te^5|B5cZ!_aH4<@1DNh;hNQ+jOn`8cAi;!h)-?e)}g$ ziDaTDK8z2OIS~IQ_tZS@!X!IvBrJV11J`Vxg}AUZDNNI#`>Iykha`Sz8&BHI>DA(p zm3s}dyIpugMHDtck?x>04lt1sW4G&=&MVXXBl+4{98SFgW%jG~Ha+zzgWOFpEO9Co zNr#)^Jl;++%ur9Jt$eB%os`>8CXA7A<}BVpGAw9w;q>YTX!$AsKU^x1hO=1YoW;K5 z;RZ;f*HtoG*8?w^zV-&TVAe&L3y z#4p1_J0eb-Y7XSx$C{jap#~P`m%+@)iaz!9R*Y1`xHclh z;nE0K94zl6mE_iX;yjd*jE>$XqT^}Zsiuz4@J5l`zL#+;OnD0D9y#kx7X%{dDlr zZf0Kzzs*0aGvGf`3cQVz5`1jCEj?o+%CYB;qy+b8RL7fU@7G=&%!7|@jGjhF=&wrm zi3&xM$L~8{pabkae(DzGx38g_|>J8(CHsfsG#hW2YJV*bLh#IlrK$yd$3s#eRaF5cdfsuRgZKnRh*28 z)_+`_Oy?Kcx*h*>uqIq%>hZZ}soi@o<+}LVOSfIZ-yCQ{fdrZH)-)$Q7qAO8V|vzh zEKVPL>HW7{+k+lIx8)FoDT~}KH2mJNbzj5amnuu`+l6+7f>Ey(s%m#~5xXI~OT+mz zP1c|kW;1GFPkY&!GOy0MDMt;=!sZ@SQ;KL?EzBg9QD5;wsL&eLD}hXsGT^M_bkSD( zg*A4OHzZ;fDiO)*-&m-T(U5=7y5M>akvvip32`Y)gcPb?@nTxjvi8(ayPKS{XIMcvm6E3e<{?fo!7FdP#w=%DLIn(+r;bxs_8 zXiC;S9a$1@FEF;OJ#w4KU~TZ=m9KcZq=(@IUm@ApZUtMQCC2-3*$#Q)|i1j^wPCdxKSTHL$?9Rm8_Ux=zV5TfGXn zhC_qq6aG5_%H{s?c4r9H^jq2(FIhwj2ayi_~=H!Q5!r#R= z+wPv9{q{p|>s{0Zjns5<1bg?WvBqx)y}T9NfLt&rbmCAxQu zsdmn(Xqv*kpjJw#4J<%fc>1}O9M9#Ej~ouI=9Wnap|CFdx`Cyyz7bdCc#uO~SA)4s zPU2#AL?k72vwidIYH}mWd`tDP4)TY5gHetrQp3FcuO)(Tlp9tlA%b)@U&(d-!}`gs zf>qa;^CM}z>BtGMH>3ar4I%r}>1Vai(v*)&k4n_z@qpce@b&vC&)n8mZ|Sw=Pvxsj z1+v-pYTeExR0;iA{A1<3D7I`EM*y>stUzlLXuJD1%j+>nGKg#SM8~U_N?spR<*~MIz>}O^ikQGNo^~r6yF^F`+ zJ2FqB9Q_v~Gtf;LZ<^HyVl)(u*wx)j4+8eR*l5LnM zJysd$J5_14TH0EmWR+T6rkT}s&o&Je@B`!`DkWPJW+;A4x1t>M?;JzXoxP zHRNDwHhtGLh%4n&HfuE-x`4_Y4Ppg3!b;4b#kP~nb%c2ZlZ|^!}d|*qVt_| z6@+tk-PxV`DGu8*z0nbL%)~k!*+r6#J=xHJB~as=64`P0w|6nd@=`)1O{=AHwqiSp z>L>wXFL#k+39@PiQBBy(w@G0^D7F#jO+QOjG~{|O!L*q7Ve#MK9lZ6G%X7Cx>-EvN z_9I#yQ4VtNC~z&RjtEa}%Pfss?bh)NVUrUzM3U?Rm9r_y3oloR8 zGS1n`(Fmj6^}`##hLBlMk~`5>6S^d-0bw=;89m{%DKdcHX7F(@BzL@HCOQYOMEG4! zXhYRm_aVSN7f%6e*r%Gq8g}B4Esosk7|I>8+;)^boCZqXd49M-Y-1fiFq4TiQut6z zjy8{c5L4FS~0oy%)6s^+f~cP_0lT}tZ1iZ8NEe4VRSo4qxF?m%Zm|GN&YJ8 z>h{$TDhlucz0EFSo2`cdfVS1id)FxX=8_fKOi;#Y0PSkOTyqvAx~)zHG?5C}47m)0 z4o5-6IEY5?7IJ3)f&Mv%)uvmTC0a7cE{dFMg+(xwcI)W#9|F$Dw&BF7#5E<@xMT}1 zqlohU;}yP+Ru+z%n&wgWnyj@^R2ckzl{i6<{MuAGnAs7#&9#+j3|5}l`R7)(wQ#C@ zuubP1DUQHE@SlD-_V=z-&(uph>x+@whO|fa2O+wEO4f`b z_rm+{Pc$ziUb~bzbSyQqoA{jJ(tX}BJ(j29lZl-*KahL%Q_4ByQcYpB8_B_~>zqxBbJnYNDok`aNnsm; zEy=KL>_pxD>L+{mY_o%Ym@Kch*gnHpw$(;_1il+>!D~Yh+bEhg?2&BG9?>M!&$98+;b;(sBWfog6hc(+xRH7B`{2g#oPM|U_r5v4%dGbD?l=V9&c!6Kt+$1AUxJ~LY);V+T>zC-5_HQ) zApPJ4kwd(8t0qq7=M*dxWCIi{psq5c+*_WB6I6E;?dHz-J@={2tc>D>E&^2JmocK| zLKb1*l*6vdovT@R_m+scMT+5w8feIMs!4u`rSbJch})xO!$wf7Lco{2DlzVl*gmiWi&qw ziQ2=lyO7eOf}L|1d_VqXwBC}Xg6<-h5xZVbSc7folW49bLZvRp*|u!E(89=T90n;| zFa0FNCBlCtt_Bfr%K$W@-3D;kgugtvL7XdhjF>WtE{C@fAB8wx@+@-oJo(==RXaS;xo*f1KJm+jkT*~R>V`= zqrxei288!*M!9UVoz+s8?4)Ib~O8^%ydTXq@i_c zj~9pOBecS56nt}^>39^_+|(9BECsk4g9Rwl$3TI+`@-@VR7VpzNy}d=l4|XYiVu+` zh8QBA>so7Et#vlq(;Jm}k-9quir3L%Re7uMM4!D2?T@#7Z{zKfJaE;<)MGR7Q1Qwz zVTYX0OvN7C)+8EZ{!Sf9p7n~+95jcP4fmcz53=czrad3w;8B(4-`#4i8hZ`Fu>JiZ zi0GS9)Ml_nS2f@}e;C`GG-o}C%7N83!ncm732$D5Ff_S+F#f<5hZ-@0!?0(kMX3aB zcy6Z$0)K;w>^p^AL!(W4lYXlmw5#L5*))PsMITZKAv0 zW|z}QiAEn;Q+_%j&b}{B8`Sk0Qo3A~jpOb_@HJI|d)9Zoq6CxgdOe05*UPZrR_mk`K&iMG8?MUg zIof{X@~p%jnGWyf0#l?*Bq_>OD9bsiuiY4hYtc?Kuu8{$Bib$}eB@Gjcmiy>$S=T5bzVd&UEey?qe^CH`RLk6d;tN%I<#WTq;orN0=<)>1udxobuh xvobJqVsNwvbi~|nZxV|yN^H_$I`+&yrIxwvWFzjbkNF?5#C}ewaTh4a{|A-6{T%=R literal 0 HcmV?d00001 diff --git a/eval/dataset/arms/copilot_v11_deep_val/pr4825.cost.json b/eval/dataset/arms/copilot_v11_deep_val/pr4825.cost.json new file mode 100644 index 000000000..6ce12584c --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_val/pr4825.cost.json @@ -0,0 +1,115 @@ +{ + "wall_s": 999.0, + "split": "val", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v11_deep_val/runs/pr4825/run-20260813-141905-e75827", + "recorded_at": "2026-08-13T14:35:41", + "attempts": [ + { + "invocation_id": "inv-df91e568", + "outcome": "run:run-20260813-141905-e75827", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-141359-6be7ec", + "llm_calls": 58, + "input_tokens": 297025, + "output_tokens": 89846, + "usd": 0.254925, + "source": "spans" + }, + { + "run_dir": "run-20260813-141905-e75827", + "llm_calls": 60, + "input_tokens": 272835, + "output_tokens": 115896, + "usd": 0.288982, + "source": "spans" + } + ], + "attempt_usd_total": 0.543907, + "llm_calls": 60, + "input_tokens": 272835, + "output_tokens": 115896, + "usd": 0.288982, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-141905-e75827", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.289, + "minutes": 16.61, + "ci_minutes": 0, + "input_tokens": 272835, + "output_tokens": 115896, + "tool_calls": 106, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.5822, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 60, + "by_role": { + "agent": { + "usd": 0.269363, + "input_tokens": 265021, + "output_tokens": 100010, + "calls": 58 + }, + "reducer": { + "usd": 0.019619, + "input_tokens": 7814, + "output_tokens": 15886, + "calls": 2 + } + }, + "cache_read_tokens": 3252992, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 3.89, + "gate": 1.63, + "review": 991.02, + "report": 0.06 + }, + "time_to_first_result_s": 996.58 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4825.trace.json.gz", + "trace_bytes": 37222 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v11_deep_val/pr4825.md b/eval/dataset/arms/copilot_v11_deep_val/pr4825.md new file mode 100644 index 000000000..62e2f56ab --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_val/pr4825.md @@ -0,0 +1,38 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4825, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head ac3a2ce5a9b9) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 3 finding(s) below | +| Security | no finding reported | +| Docs / comments | 2 finding(s) below | +| Behavior / compatibility | no finding reported | +| Correctness | no finding reported | + +**Validated:** +- [sweep] Read both diff hunks in situ: manager.py default_components (lines 368-383) adds "unet"; test adds test_lora_manager_discovers_unet_component (lines 578-619). +- [sweep] Branch tests/diffusion/lora/test_lora_manager.py:587 `if isinstance(layer, _FakeLinearBase):` — both arms reachable: container `down_block` returns layer unchanged, `proj` (FakeLinearBase) wraps; not dead. +- [sweep] Non-test file manager.py covered by the new test test_lora_manager_discovers_unet_component; the file's pytestmark (core_model+cpu) selects it, and the changed tuple is driven through `_replace_layers_with_lora`. +- [validated] manager.py:380 — adding unet matches StableDiffusionXLPipeline.unet (pipeline_sdxl.py:110); grep `self.unet =` finds no other built-in top-level unet (only a nested rmvpe unet at soulx_singer/modules/preprocess/rmvpe.py:236). +- [validated] registry.py:427 already includes "unet" in transformer_attrs, so treating unet as a denoiser component is consistent with the existing SP path. +- [validated] SDXL UNet layers are LoRA-replaceable: to_qkv QKVParallelLinear, to_q/to_k/to_v ColumnParallelLinear, ff.geglu.proj MergedColumnParallelLinear, to_out/out_proj RowParallelLinear (sdxl_unet.py:121-177); from_layer_diffusion wraps these classes (lora/utils.py). +- [validated] Packed to_qkv sublayer mapping is available: SDXL load_weights sets stacked_params_mapping (sdxl_unet.py:827-832) and DiffusionWorker calls load_model before init_lora_manager (diffusion_worker.py:230-232,338-348). +- [sweep] _lora_components consumers: only SenseNovaU1Pipeline (pipeline_sensenova_u1.py:543) defines it as ["language_model"]; no overlap with the new unet default, so no double-scan today. + +`vllm_omni/diffusion/lora/manager.py:380` [major] — This diff adds 'unet' to default_components (manager.py:380), so _replace_layers_with_lora now walks SDXL's UNet, but standard SDXL LoRA target_modules 'ff.net.0.proj'/'ff.net.2' never match the repo module names 'ff.geglu.proj'/'ff.out_proj': _match_target_modules (utils.py:20) does literal suffix matching with no remap, and the only ff.net->geglu/out_proj remap is in SDXLUNet2DConditionModel.load_weights (sdxl_unet.py). FF layers are therefore never wrapped, and their deltas are silently dropped. Please apply the same diffusers->internal name normalization in the LoRA matching path (or normalize SDXL target_modules before matching) and add a regression test using an SDXL FF target name. (evidence: Read vllm_omni/diffusion/lora/utils.py `_match_target_modules` (literal suffix match, no remap); read vllm_omni/diffusion/models/sdxl/sdxl_unet.py:835-848 (ff.net.* remap only in load_weights); grepped vllm_omni/diffusion/lora for `ff.net`/`geglu` (no matches).) + +`tests/diffusion/lora/test_lora_manager.py:580` [minor] — Confirmed: tests/diffusion/lora/test_lora_manager.py:19 sets pytestmark=[core_model, cpu]; test_lora_manager_discovers_unet_component is at line 580. .buildkite/test-amd-merge.yml:21 runs `pytest -sv tests/diffusion -m 'core_model and cpu' --ignore=tests/diffusion/cache/test_teacache_extractors.py`, so the new test is selected in the AMD lane. gh_ci_read(4825) reports buildkite/vllm-omni-amd-ci and buildkite/vllm-omni-npu-ci FAILURE while buildkite/vllm-omni passes; grep found no tests/diffusion command in the NPU pipeline files. Please confirm whether the AMD failure is pre-existing on main or introduced here. (evidence: gh_ci_read(4825) reports buildkite/vllm-omni-amd-ci and buildkite/vllm-omni-npu-ci FAILURE while buildkite/vllm-omni passes; read .buildkite/test-amd-merge.yml:21 and grepped the NPU pipeline files for tests/diffusion.) + +`vllm_omni/diffusion/lora/manager.py:380` [minor] — SDXLTransformer2DModel.proj_in/proj_out (vllm_omni/diffusion/models/sdxl/sdxl_unet.py:281,296) are plain nn.Linear, which from_layer_diffusion cannot wrap, so any SDXL LoRA adapter targeting proj_in/proj_out will be silently skipped now that 'unet' is a default scanned component in _replace_layers_with_lora. Consider documenting or guarding this unsupported-target case. (evidence: Residual gap: SDXL LoRA path applies attention q/k/v (and to_qkv via packed mapping), but diffusers-style target names `ff.net.0.proj`/`ff.net.2` won't match `ff.geglu.proj`/`ff.out_proj` (sdxl_unet.py:835-848 remaps only inside load_weights); proj_in/proj_out are nn.Linear (sdxl_unet.py:281,296) and unwrappable by from_layer_diffusion. Verified from files read; documented as a minor comment.) + +`tests/diffusion/lora/test_lora_manager.py:616` [minor] — This diff adds test_lora_manager_discovers_unet_component (tests/diffusion/lora/test_lora_manager.py:580), but the synthetic leaf 'pipeline.unet.down_block.proj' and peft_helper without target_modules never exercise the SDXL-specific matching this new 'unet' default depends on: _matches_target returns True when target_modules is unset (manager.py:356), so the packed/sublayer branch at manager.py:405-419 (e.g. SDXL self-attention 'to_qkv') is skipped, and no real SDXL module names ('attn1.to_out.0', 'attn2.to_q', 'ff.geglu.proj'/'ff.out_proj') are asserted. Please add an SDXL-shaped UNet test with explicit target_modules (e.g. ['to_q','to_k','to_v','to_out.0']) and assert _lora_modules contains both '...attn1.to_qkv' (packed path) and '...attn2.to_q/to_k/to_v', so a naming regression cannot silently drop SDXL layers again. (evidence: Read tests/diffusion/lora/test_lora_manager.py (new test) and vllm_omni/diffusion/lora/manager.py lines 400-419.) + +`vllm_omni/diffusion/lora/manager.py:370` [nit] — Reword the comment above default_components: 'unet' is now a new default component, so _lora_components is not the only opt-in path and 'other pipelines unchanged' no longer holds. Suggested wording: 'Default denoising components (including unet for SDXL-style pipelines) plus any pipeline opt-ins via _lora_components.' (evidence: grep `self.unet =` in vllm_omni: only pipeline_sdxl.py:110 (plus nested rmvpe.py:236).) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v11_deep_val/pr4825.trace.json.gz b/eval/dataset/arms/copilot_v11_deep_val/pr4825.trace.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..2b315777d94a579e7ac78ba67147274bfab2b5f3 GIT binary patch literal 37222 zcmbTdRZJyJ&@GC)yW8My8+V7noxx>rcXxMpcN^Sw<1h?t+}+(>@Au~xU&3O3cYq@;>kqCy^;vRtuuGK0CmQCd0fA zz`tM4v++k-pnpdiqBM;l)%Sg01{&V$u=Rx&RIf9C`2Mrt?MJ{sS&p_H*rgYMFuVaN zYq{yV0Zc2ivW>mO`FBuIQFqFHuRx<5-_i3Yu(HmQT8KizFJPny!u_SsV_tp0MvEBA zpz{}RBH#Qt4u9f%Lfho{s8X$B2zAHLB-1k zpvN!MU}_K2i;ZFQl(CbE^c`y7_v=k?k-&M(+3ooB&-YNpX!npS35^vS3a0l?G_H`s zBB(A~*wAK;XU|ut?+LSqG>4bi8Xo1LDXnLxh`=7&PN?ONKNiSHrEs`C5g9VFa0?2)~+ z5Erb>JvIKM5VQ3~`u%yI6VT)LdFj$6UU1g8?-cjmW|bWu?-A=i;gz27wV%<>vb`gi zZEBN`f1(D8hEM{W=EpolFSpBUh;xXqyvU#u&9|UBHtE0#SGAlPXG;nYyhHb5&X`wr ziBEY}WYL7XnOKJspHd)pQjYDk6()q&b;Eo_cDQn{#_d!%CB2eseWgMAT8$%;wnexP zswmLh{jjq>n+Gn*gPcEraY$GE+52m|oO|`}ox>eRbW9Xuq)pB!w6#0o3$gu*cc&@x(eF);b_B@XdRi{1zsv2qU z)@U%sK2q7I+2;yyXnZk4x#bJ_=+nSfPKKM}@de)H%PPPk;Ho;C`;meqJdh##YNgNk zEq(W$TyfaV6%wvMSIS4K+@^BcTP@q-l$XqPJJ-l|2uU6+w#Ekh0<(wtfCTjs2y4`t z^arMafv-oH)0e>+Kr^GhTtER|As)&!lA#o#VY`coB zkil|MX7a_`{K|GXu|9aaVMvTa20JvyQe^7pAYN2md^}RS+ieU$@%@Fq;Y2htZv|r9 zoM{1%69hE$B=WZb9)rg{7OIY=S+gOvEvz<_?KC*R#{@Hb5K@hGi(q7~B`WZII7ii}g1sA|1^6OsX9l(pAlMv+LGh@nY(y`y9(13zT=TK@W z|3+ILX+z)jCU5!FnixHL>3TgMFSq^QUN5_~K-F6^#jy*8??fjv2DlJizTja+}Qf11r= zi{LP&-9f~s-Ulz`N1^^kzevzR>;qSVnB2-|7Z$)O!#7vyHrt3S{^#LUjPMPxI_!ah zg)AWvb#@T$ould@KlPA5fU`X_S;da5o8tu#7K_nx{!!=np5 z$ay!8{w4@5lH{P-{zOP5Hlq4)HSk~lWmWh&Bq}N95G4zqLeEgSuO+rF6_=&nCeX8^Re5dw;4@lQy~&zdydqo>w1k`*g&Huj z6|4y&&;prTG}rf;s2POf-raW^-ZZZ!3i^Z>jEcbrJY9tPeP7m)8hk!*KYd)Y6nyV9 z5uQlj#19#v)6D+XyHP3YM*Xa@#2bwmt9@Qx^$qIxH~QZG{tVDbZPNMG5R3YG;XAN8 z6=v9QT%TD0^e)z=HYg})@#y<`-WOUB8?Sm5)o;o}nSxhU~ z+LBIE5r_f@m-m8!76gr`R#4qya~TIKU98O2*I`Ia?4$k~lBIE9|IHo$5%K#J=*AieM+I;=7>XaCTZ zM({~0yXQsHFkEz6aWw$qyuc4B-~|tm8$#lOFUjT_lBxS8M3x9}I7<~^= zz|UQyZbQb+jL*V`TXIQp5|ZQU?rDHD{LbDa)MP7cju`Ctq-9Kd*vp~z{dU!|Ph$KRne z8Z_ck^OrA?9891dV^mrd$amqSa9MkIRY|oxvdCpJ|H8t^;)C9-Jjma)wztM&9#u_8HSJ5qQgt{i_pqOL&!5UWMcyN__?i|1Sc*|45d1 zbwEI82ld7!$jr@6@L0rW0GsJC({jCj$=emdCGMV0d6kz`A!S6`&o|3)#aDtd)VH@G zDwFjCr;OZ@8;aYG$lfRMMT9qtbmZpARc<*D(t;HlUI^zncAjLEbyHWW^w_XW1YaaD zvd+^5I<(Ag%d?&q35 zWC!GuiUXMvx7Fi8vszNj_ycxbyyI3Kdgep;#v4Sl$f(y7EUZuySa`6hY`Wc*`)s=Lmx*bCLpnLfuXNo?IV$z0^pgHZ{9K45XB4n$Wgi(I&%Z61xeHOvIP4QdhMj zk^WFLYImHPi&t0_uQN>m0cvIyQYKe#tWkDPkTT)4#(x?Uk6@wVT-}(;P zUDy;hNPy_VqN0&?JRL=PiW8g%)>+ziTWe7Ta`5_Fq5j=&P)JulBjxe);q6ZaIh47m ze~v>UPM(hX#){NS#+&(zNFcv4JM@)>-6jBsBe8d|N{}3d4JSlbz@)^m!=5m84s-uw z>1Ia#T*D3M3Kh~_WEnVx`4J?YKqFYs_RJVV!^^y%&|)IfaOUEn?h9*gF8!(ZfqzX? zXT;{}8((?;t(C@b*AuybUr_b$mv;{j@PT#$;I2G2Sj;rsea4u$t=!@D+PheBaIe;) z5%d3n|9if|nK<=DY9j~9a7OBzW9PKY~h1n6c3>)&<@V+Tm6HkJY;#W z4rH;MtQ4M6vVgtU9V}@r8V}dHK$CzaoqxiB2)`d8Z@Qn7=I7NZP+a!DXYKc=ci0|? ztZ@p5c`mU@ja9jajv22&U384F<&ikot<27|Fk-j6(D|25dw2}onP?lQ;?w`W8A^yK zj7}4%Q&plV9QsFrm9Fxu@Qtr3^?@|-jC%OSg)1}gOQY0uW)|BG0-~DT6Tz5UqZTht zBQe-A3c7IS=MWB=bfh62xfr+jMCyO>0}19WR#D(bY;Av0=_pw^`}Fyx)f$Q;EMz$^ zi;VVz<%wm?Fe@QS*K;vmxf&bZ$>lLCEJm@VZ=e7aWsZx%EK-NT4}%ckkR)G2qSg% zQl=__B8tZQiXLdu6Sf)e=Gbd0_a~xy~=?i0I%dCf~#q7lFEamr;x6KA6Sm%_WhMnY= zjR;F2`iCc#?RQA#h-Az68Gu{;#K~5oYozv2!El=jVUB2w=Gfto|Cp7FA@$ipm8EcM zOXcUxgJJ+xX*g1ufm$qk_lR`tP-`l&gaP3C@_c3Hg_B8g$r!5vgpwx;MS@KBKl1qF;I_x+L;BHYKxa}cE>hH}8-h6c^k zlTQ5pXCEJ|jypthvkb186-6*=Gf#i2d3GY}xpf=qG>T9w^}>yfkk1ni&%;Esg+=rB zsteiFJ!kjt{$_3gX~idgJrb0N2jQN5;%G%9++gRj^-GU(j#C}NKsU4-%HATHRb3M1 zK+(Ox!qrKiQU{#z{3|c;GqPCiWReZ*TD(hs|6Ri|k#bjVTT!Cn3BeL!B@WlR>BJ?i zEq9omSBuk)!d|>4mNniL{hck|mk1u%<$;olYod!f=W9aSR~RRcFYmg6qo(h*|7Jyj zSeqMLYIckoM$k5jChX1WG5rzs{s7z-C*g6kr_eE$0SsB=qaaGiC^Jdkj%{hjSkYDy zdQiR8QOY(ZfU36eef$YXN0c%vAMHC-CMyHUIWu}FTNd@sG%i&Jz95Ps=G1T#XjC^r ztLI!T*Zx}+2gT93F7WCg=H;3?zXx+lJtF(}5JMC;1uALqzYKOpsURLIXWBSqt{21a zmtz}YaSsuSbkBcAPP80JA9U=OA*Wc4hSC`&HdsZ2w4$|`yB1dU^20Cs!lP5r1pP)m zK~%Baa5=Kt4y|P_zkbn+5^M%bLX9_ob&+{7VgY9KMlf!qAmpvahOm91@l^(GC?<4) zoo1~?GkeLpVZDkxccqI@Izd@rsltie(hgasCrkOY$+7J039D!xO@w#5FLS*R{m3dp z1Kl((LQJEBTGeoUOIS-0aD79jA1-uXdNIN+IT55YXq1?l$})2qp1#5V%0;RmeUMqC zVKEEG<<;a;$uj@ecwhROK-T3lur?EW$3|V#VWN&wZ?M0GEf^M|ec*sFJ7l@e@ta3*IHOXMybXYtFZ$xg~z8J5VU&4Y;p- zlX|g$uQa&@rwBp6b%i6#ldxW*YVCibLWyME@!~>^DdyHfVOF$rVCZNy3S;neOSVmX zf>ID(ZsGqzb0OxSmAgVpN9XUcx)^5ZG?SJHnBhw4AF_?SW#|AwD&~~6t5BaPm80An z=+a0s@jdZ{BIbe9MGujA>~Y+>Bdh3})V7g~66Ba(XkECS&ZG*Qpe5Zmkmth`RuFNe9u1YC zXgXl2$A$rpEKX8TfLDFNTHqdJOrP;iGSUtxoa3{(BKk;B46iUbYiT{yout4Mnc4)( z@P9U2gADNT{V%=dId58IKLLYe!`E|ekN)SF1<#mEBqM00%Orcf-} zK->WoY@<$7^)l21N);3+GTQM))wWxdD#qn(NcMXqjz1W4+CX~A`3jr>vB`IB&ZW4l z->pJm%i)9d3Jh8+J7uXgG&w5-;cP8LxPrGpp@_C4I%z-kCcyR5atV~~rRpFfv*zqT z-J{Zuv)KDaD68i#q@6t~cBN@TrwVajl;$1Fr@9AaOxK{r#; zyM>a&8XY_ujR_9Dm7I{aA*8slOU3%OB)NHckDNKwI;ze-DH3kxnIo3SOrVGr0*w=G zaxTA!tW-exaH(I8)gYpr&Ssx61Yphv*XJfVh_RRci_t`-R~&MU;FL_ack%EjAK`jE z&&WdJu#38p#xR)Yc0$Or9i`>prIaQj zXldG>y+Bo&PJ*xd&wixpg}Y!zYq&H8B9m)?N~u<8wZtFVw>#{%ax`+m*`ooNo{jnN z^kbq;hjs++YULE5K=r;mtsM4K{9JYtnxr|?-(}Cke_@4sR?3ZQ@e1RLBRd_OI`LtG zF)-C1dcV>Le}R^iU2=a&_3e{@46MpX5caB=l#@#bms28i1X$u3b{XU~@?zqO#$ZzH zRPlhNSQMe31-lGQF*#dPQGw9*hUd6TM$puq9ts>o6q(a=&@?#u9;dRHnJW*Mz?HW+6fcoKe<^abu+#IgoR3=T;W z1OpD!2653mv?{y@?bE|&2jdk{K9=y?i`25x#D)9Hj}2+eg9v|eq75L!j)|1?A)V6B zR|!d4DEwbIYB?-LDWD&FnPQGHWEP0x=@Je$QzJ+yRcTIl!M9nP=#BCiH;bhmI}zC* z8kte54mM?4ts!|lBx^V$m)O0%*cV=Jn$TR^Gy&!3efh>7?I8sbS#2&YO2(a;w~k~A zA}5gwmIihTdXnZJY*WF@a8@>#vhp$Wmbw9uq@5UpmjHh51tmR#x@-v=5iI$H53|lVPL*k3R;J0bMDxG;2Y3XwPhbP${*XDYOCVxmOsjUlq=;xy-VkZIWg}?g8?P!Ml%2m zSW7AC!X3LXiiea8X+J$T6C~<3rN9x3a0JL{Ug%C)_}xPh6*}d**8Vif$aILAVnu2@ z>BwQBFLBb{FunD3CYHW~*mvV&MFd1ojx%WDfc@&YlP_7UT5{M7vtIkF?_bw1L1uA>V!8qpWz@wTt;qOak z3LNFO+6hN9=&~JKT2~Pr&F8?w4w8i=U4NLwo|04)?}!j{n;g;%V$WNqqB-|FI5o5y z-gXfF#qe*4TIjuU<&)D|p(dy8cM|E`H`OB!xVtY8ND^o`cu$*_ZVaUd(+HpaE27Sb zA%KzEx!PMd5g2o_&^0|_V4J1R+~I@Acf*O`MoyCwfk@i~=E7fx;t=zgHaYmNbgJzh zkDx%lG3OcHm!zWT9W-v50J6w@_G1yvA-r{WYw!7_pPpT1tDT-gRZ-TUIm)rsFUymr zH7KL-BG6!5Qqhcjn~YrOwP|-B|0l9-7?GCN_M7Dz;)i8ZxUtxU96~3vE>`XY*)*kH z&sw3qZnDkloOX;T(rKk#OjpZ>)(f;I!MH2yZup_+0u-yKLX37(_J!h5X3L0v=Fn6R zupHhORVO21S&9?-+x(9_M@yViRVV~?q>0!A=a+T~{nx7_qr7C+E7Mj}z18io%`dP} z;i%vUNq2tcO^Bvx6YwPzMS%5Epg`hdx>>s6t&Lv*Z8f8bPL5OhUjb1$)KqR7cp<4Y zIMYPqkq#LaUX9pUeNc&WxCquCQlhaLwvXmg=SWHhE1hBD_gxK{`i-a5XW`eWLKN0uqf+->EckFs-*F=GoMFV zN0&vd-_Wy0Q#_?e;g4UXd&bLQ(BOCAi!m{ zH-)!1sw@m||!*Uf)X97G|km*@n0 z+~J~xK`!ACxRw$<=oZd3-E`hw2ubLrvJcmBz-8+rStD1c+qvRTJhLEJd zkgOjaCOlApTum>XATR7hu$Efqh1b&OKBl8qm9A(0j)S&=*8V^XqwgUsn(|f6U?aXh za9~6vHnhsQW{Cl1oxTaGI;9{|)(K55iuL6%rm=_a65ywEfQt=b!laRunC$2jBkNLP zEtv|P6m4CiO=`QLtKAxz1R1S0@AwdP##MA#b`r^UY=Y}5xm(1w z5~Yfl2I(CyDhq2eeo{M=$*RGPHD9c28IG;}TLA!`F&cc<`9VF~m&*WijdRZqkDouN z7UejJSF2bYbT#GK z42~MQ_cww&(>#9bAh-7gKV?WoW{1IXcF&R8o>7RG`9(S@E_etj;uv@v7Sx`vMyLI~ z)r`+Zan!N25Ur;Mt4*t&Hjl(P^o&&Kx5zcsyz0oaVTSUb+}i);-5m9?9MD*Fs67Vr zPVk+H6T*<0(9AdCe{-QdyckqF&)B0}EwOJZkmKnE$>-tkN&z4*vJVLs^V0eoU0Tx& zxe)RYtU`i~6FlrY73F6NAs0P1y;uTxUbp^^3YnMtL8V~*sB!FB z;7A^ap!q&-p~q%1;w$V%*f$I^E6{XjsR@Zmx#%}!km^{<@ww-W zX@B^$k4~5~=tt9hKRPCsA&dp=uCJ53<3d+`M_xud^gQJw;*^j*`Pdj6-I7X{O-ew? zBd{7BZO|`Bbk}WvIY1hTI8gABun;p0Mu=FWTixkdBiD8Yul2T0F4&bzb37L3ms`O} z$XH(S(K&HUcKtOvzz?%0=69QGid%d(^!bLy*$?|2LhHcA!;0u)rJ*V-C@$`nc%i1H zNHI*EbIk=&q;)~}mE{*{7kc7Szd}bcBT+j9MG>#b_>~)B9WW_Odcv~fFt8V~{99~Z z8g|`M*NaHZiRSguPScjPiM-rmhhD&f+3ICz2LiDsD-@^b&MD z!4#LtrL;yec_HyZ1y)O%yeVzZZ!AU2q`BOkVQ00qxz_Dp9%s-+o{X|S{Tn{?kko9; zdjCxYG7i{cS%k)}AfUs^lL3+KtWQ6UvhS9p=%^X(3oIC&`Sa!|K{%cc14@5}e<-^W zpa=98@spFLP$6{9Tld3~mcx+ZlRz?0Rn+$lqA3kaeoV*l4fIt; zB}-D#!wS|l$Q(5eG|a9sg?B3DRWhMdE5C2Q!EguR2<{Xzwb-B*v!`WIBfh<0{;r&p z8u-6IQN4dHq6#-?2{(EK1VnLo-`-u`bJrX6`#jz&U0kzl_o&}!evm49CXxH?wBLb= z`f418ZEz_3E1@0p)=-CKxR;8C+w(hx6nH+SN%P5fMXxm?d)4@Jraqie@zaW0!Xu4S zZgxIiXhi65y(XPjG(dR8rory!z1bpl&gbp|KYP$b)?Z){1W&4oNa`;iE|H#H96ja_ zwlQ2W$k`t~WXy&=LFu{B9Y`fhcdGHdwpsP8(xYz;UnPicw z4KNjKmXaZ%!{2D*bcLscZj&_Ey!;pow8wf!Y8QvOOVA<}={hsgM|4ZVUIRw03jcHwP^}d}t zjMLNZeSf`zgnNDN&$AQgE_V7eXrgJhf?Ed#Z^=gly35qoDB?GW&W`+LBRa$G4 zbVSo}52mh?RPU2Ak_k>bNr_G&>%t@3aAGi852i%ByzAg}`z#6ULmpX`(}DT+@^%Y5 zhT3B&T0{Zfec>L)E5m4?@=`H~^+@FRP6gvRD(SPt5~ZAPv|NzgbdXxB+5W~jvtH%0jfK~HF)Ddpl!>wsQ3dWd zz3eML!4RftT;ECY{aiH*%6*`ilZ$Kapk)K6AJp;;S@{w0}TA|Bx2DImYOJZ7qnUh%LM|Ik36u-p<;T~&ikDUI0)m57sS2g8aHRW?x zACJ}LTp6Fsb=hkhQGViO!mUNBF6V^#c*;L*G+qxl7_cb*rjC?d(JFY#x5>yb|22Ku zdj%D5BWlKu6^B*myX7FIo%g7A+qguih0;?;Q0cIzef=kT`$6j{X)c=Tmk`+0e4N?W z)R8P__cTcSU09SYxv;q8u0y$%J7TjZb%DU?%T(M?u-qEn{YBAXm^L4I%wpwI@w+Hi z^`uU~PWB-jG3zidI{VSh&)QRGNJ^#3j#ZLS1BJ!E>^6$~3m7_zoH`GhA z8s9LJr<8u!F^YZM7`VgF0y8&GWTk-C!v!$4NX@C%EK;3Uh2lEwRo>X z7DXA&<%2(>xn@*n-G0>w(I_BMm_q1SDUcRU8ORZhP%Qr_TI5X5Q1nrDobmE8MayIg z+c{vxG7%$61}w8fmkOOqs4Dzg%WE5)DZ$(oV(#BF-YOIVRRw>cqpgQLPKFP*SE1LhkKN?d0uklXV)?44dsQ|DVh}}zax#o zvq!)9F4^$9SFeFl-n%CG5}9HUX`?4I7ztnkqcWvHN z4ffZFz1bG!ekRxXGcN2&Ap?=4-@relvdiKb=ZSRcnB1%HTO`1mu7_!hU+P?sfq0|n zRW)(N{%I)jzhzYf4Jp_$q^B4u=6al>o^cg)Rg9!zLS10a=1#n_8SBhTxKF2Kh@`M_ z)2oZ%FHV>Ws2O?@E5yq3lcH%rkExH<4#xrxM8@{ZToRkX2;a&YdF8+i9<*EDUpJl& zE(i$Wj4f(mqcu$su7&vMRqSrbc-naRrT^s~PcFd0Ob*(mV38BVU>)yMBihN86>L1K zg~1`;ck5$#_DklR*n`>T6H)FwA}E=3LXobW_7}1B=rjD_%c*~0=0r$}%_l}V{%Od! zk^|h=M(4se@j^d&h0J;<5i+GLPlimL6XVkmzQUkB(=@dYk{$O5bTW=vH)}?RL_Y4r zr02E#$9ijGjGgh8l(VW;6ZfR>|B@L?{x;;qT=dVM-8gWBiFwb+wAZ`cX32Y_(fPTG z!0O*nB!NE$cOAHpAL-#P-Esg6wAstn)V@W#I{xncOsnY^coE2qG_9i0vI#C%5tlpu z6P%;X=SM#kSn^k)nNN;fjD4=B+60zePQsH->GVMoyST8ix+q?;Y?NUMgmOKBz{~i| zYnWT0M#xT<%`+&c?cC~#q>jcRT`Sp zBsfYy7u(-oy#Yz~v9KE9108x>`*w}ZUQB(hQ@hn71%?;3dGN3Xb`9c=!d8%#Dr@|3 zUjqNetZhU$ZaMYs(sG^7tUQ1C0^Ro`~nhS+Yf$;mPVqW*`(^oRzFb>(h^*bDK3VZ9hp zR%14r(l~(mk7At+Z%Jy=bv2j2*#>1pJ&m;n*R!*Nb$J>;&%4|bBY;u^VihKPsWBQb z2MLk{WC(u7k%D!R`pI`9?Zynrw95jr-<-~gg zd`>lo0D7uCN9${WRz3;U5XYmBT)MN}tv2=~AJA*^>iU!qi~5w~hyd8be3T9eU*`Be z1zcG%>0FFZb872N@S(IZKIh#g2I8~K2SEv#8QYP}ec1|tx%W$v(}DR055~jAsxGiB z#R4>K-rsy2m~?FAKIxxfQNDWZ=*2@c_k9!ZF`O=yO!!yrzMms+;^08kpAwa2?$Uv` zkT-wio7l;4>xYsr_}iQy3#%`CwQr(hO`}Q6^+<>@WYp2GW~9+s z1Ef{lJflth9if3YRM`dt*o1ulU^kDncc^poV8G?HA(AEj26o#u zmp-k2^=-oXt$s}^*`Uzg3fjs_)m9Mq#xdd5Ju)fR_RYPOXTf7T=uLO}%`}AF-FvX_ zBv!aOkxysx*Y(`AP}DU%^(_G1rvUhTg89Nl1=7)52(~N8cy_(N`{mjNnt11MzT+wc zNIT503H`W&HyZ}Xp3Pbe5pQBA{$i%?6SnJo_P^*QeVsapErR=$L?p^QHukJxe%lkj%f7$=fWv>%> zu2MdpWc}YM`#;gg{HiyrfJZk0EjTpak(S-;nK#_Q9$A zn;jVAYqlkg^Ie;jfug~OFunLA&xf4r#opa}_D|l|U=5Bw(h|O} zckZ}ZuTV0PBWoM_nfO9e7>l~y`*us^^L%vpldD6->mnk&K4Rmp7l^qc+gm)aV=e1D zHw5Q7aY@JI=^3KJn!3m9bISoEgckdPaIwss-LwDR{}v5k7YzN_wD7**5PZrad6!S{ zD<2b@H!V1A1U;<$U%1`VPMx?IhCGevR1Qm@`-zQO;^j=snu|nS zoh8<`;WcTj#(m zV#V-ecF49800%j{OJBAFT7yGiFAt3p9rUDdM%befKkQtS3~pxwKJK0lbCNJXfy9tP zzi-EkGpF|{nWPS1yu_~yEILL*ri1ShNu_P4P0qZH4>FBZbJB^SNS?**th3F}XF8;c6McJ+E% zu9x>(iuMZGdM<~at<8>{`~RQnO%dCOnoA+1fchsN5(H!PvbjS5RG104J1YGf$dFB# zyhZ^^Cxd%f-F9)7XewDo72q!r~=!Z3PRz7baaK_s75ia8wH(~k(A zc??-!oPfliY$os&3@I2dO*ml;%QGbfcr8X2%VMH3UBGz7XNIs822u6BN37BoAi5JB7do; z{?iPcL3ZzTP5fDU5tWvyGR{8TWVNlRlevhn(&e5LXA#@DBn|H0Xzu+_XsOfG+xJT& zjS~b9$<|&|QENAhX!EmS?y=;k8NaR`sSkKiB)$_@}C%(}uq~LDR0z zo+*`kT6A5!l&*ZqF$b-cbIeN|JzPO)h*dDY!qlKY!1h;)e#>z{>7uG-8qrZOjqH9p zg#soxzB-u%07z7kL7WPy3=oZy^ep_B`B5;zMi_Vk2aez5+F}|YqwwO&r=}0=W4@cEoKN>jyG|oRfrAYZbGjZ)S z6S5kAR-G*sseh_KXHw1LX{BZh)=yY5?}Rc~z`d9a(!!ClNlzTEg6G;2$y`ci)Sq=- zZ~X&nI46a@x~!L3QC0o!8Mv71CA+*S%YOq^$*N>E7jEL&RkHe)zkU7sX?pPNAA0Mg z{*T+}DR8OIqPzUjGfQw~PqvRsxCQrO+{-y~@vi9jO7iGiwg3>+A)>CiqT@*ahdkn%2y&yW|m<4o{9SZ>$}9n8%de z_7NApi03u*HXGwA$TS$m8(r7Ekh*k8$MkK!!Y?bAGH~{A8SWXC7W3ml(~V3cZhFg( zn-!c^gzf}LEX!VcQ0qv^?KBtk**4kfex%Qg6;H6!nP&vG!Zy+#J;Vy$NfvvTC6p*q z_v?EUdHB)g33nYkL!G1sZ|v3n*9$}n16Zi*Gv6znR_oJK3q8bD#+=RGiop6ilNk8b zx-OUa?xl4>eHG-XCbwr);3tCFxx$F5{Lt*erUc&aLHCiWU#HBM&)(5B*=?VpbCr6? zP$5>`ZO=T|u044ck711+4RsCf0OePps`CR4mXQmxexvQYtNMP7v+isck$5}Pe#|gn zNK<|j4CW?N%-N6T;u<>jkUqT(zx*|H51G?S+Cxx3AQ>m+TpJ-$Y1yArNACRJveco< zGO5AprTq)cD{BkNW&`p)>KqP1RNbHFu5gn~`jv4dV8XF?_MT<_Z-Kp?JTzwX=U;!e}gU=?N?k zsCB2G@+g5x3cB{zTd@s;ToPA9`wASd1AlQjaZ()>l}+q&0SSq?rvA6M4&H8s>!boNHRYr#wB5B zZLz@ViJZf#Ts9n-Lb`UO_@xYtfxrDV*FE=*8Ue!JMdcCZS_RhyRF7!mI2?>xqyQy; zK$5B&HmbbJB|21B1NFS1ueg?i{wz@<($W4hj&W1rT}MM_`aLa|N2q=D?JGowqI7|B zU9p?3k9JZ?W&CAO<%eqg$Vs*pU0vCEzI7caLvX6?lIgX{xU=mM&DU$xE0w!?pw`E% zej-cZ(uL<2zNqXd69OsJFYD^+RgWU2j6tMHWmk}<;CvNA%FKxjS<7@2 zQDmRqN9Y;a5Q7ovd<|B+eTi1L5gdNo;l#?R)L2Q$zXH`xYm{aB6La8M$8w3UBNiheaHPds^05ACi1w=rM#> z(6`xpC6MJe%^z53;~n7<7@_%#HWatW>Mz&Q7_5mij1?|lAT=>Ul;VCo@1-l^T1IK8g3>q?+( ziW^&_l|=k^n104dVJm*R@!C?#WV0?bJ>Fkd#^V^B_#)nL%8XWa7`IKw7CVkkre`NA z=jS6*&Uq-nV?O6Vck`(?=JnM_x1xJq)j5LpZ*wnin_hV(5J$Tru$>gJ^6&6s8c@_W zvXpbe!=6_`s?X}CH9Jr|S8#zY(g-y*=Vlj@YN4d9G`?=F3{BzhrUbwAJJKQ?;6&)RSb*CxIG!3`@sVs3G(|S;;GZp7JfmCq(Nja{+gA#GbwTR=jRAK_nr0k>o&W}i8c!6)vTcHjSN z$N4S4!$vh_q$7^Xe%Is@rp+LY$4wX zjEYJddA8L2_$wE=I`u~=t=n>4eWrYFj*(E037b;GJH0R;&X4_;g8tyCIAihM_{k=8 zzW*9-AdH*-4-GfY_eqckE5qJ6TTME8oYHOaSWjXz%78af(xt4jUj041wyjrrB+Spl zR4ak|6sIU@Cd!g87k$d17^IXwBT1SI;ji~&?r+ccxAkKd>3Zdux1^TJW7ViVk%%AK zDFyfclLB~^sn*)JQ6MyR-^9BHBS}ptb%rj8=&D|?uE7_dT0oj4Tm$#;k{!iqO04ZFY~@U#B)Up)al#3C362`p50%t)H0w%&1%J|QElnbHeu zzYJnqW`Tu2R0r2FzW53&m{79gG|Y-~BKgVca;Iib6rVTsZgku%&Lyz{ttU0& z0ioSx&LZv-U76tHf$gE8R^ydFEA!a()S3B=ASrU$eHnnb?W9>;_B^;8Qr>t!JkYug zpDVt}P$CYChXfC&gzAo-7Hsg2o#)3gIWkTd2*OASCk`-V z;LJj<#LsITo1qN%+DOUkYtqoqke?~FR)BlmGb$6$=(Atd59{xeNm0z(!1Vf~F!j*4 zQ;9EDdkk!QlD{}>a^mgC&W=gV%38nuUyra!%gZepjFKmdMa zz|cGe;6hi}zQ&>}y)ET2La5eBJ{ern;6HkPi9-rG>#|eZ*H&MVD0vFED@FM2E!U)Q zb)w&S`%*h@N0R|fLY&8K$E4(D^+{Jv?bhwXYELN}r8`EJKg!!kHm1>nV`-}f$=eS_ zVC`$0X@Cg`YS2yOWz`2EGA`#S`kfa-mROm22`&E^TJSbDGhP;1b|!(6$}XydN|rw9 z;3u;xKlUUPn`pzTfu@?8BT`^X2TE*eGl8~nWZ?vem^x~oc*;;hCEVHNiIKA zB$RuzNKWo%OAMSMuw%Bc(Im|oYRkvbJm12Qaj#U2V}*N|*BG}Y0 zVm~Q;y#kh6QQPfHF)FD&aItxX|E51<&n1F(pM7i#OZgr8>qIBs-cyA%H&I)Ek)suM z&x@M(@omRUE#-kYcw!FOPmMJzdLAO>T*n~6>`r}T#w|mR!49&F)v?72OWth^>8()W zk1?{oAD%Tkz=8^pmEWdDi{I7WXynW<0GicPyG;)=qbra43~Pf@g>s%i;hc9BufUaA zv0UU}x;NI&x3GMZn>e`8Wan)wx092zmY$ergrrPahsJcH2;H?B)s>}I130qErF+cDA?R9o!O;YV@$Fs`U&`u?uL{M4Pb!uC4gio@pH|Z$O zeYWqlfQUW=^u*%<$FEP7@2b8`KkX#xwZUZ7(Vs9|i7#PcKofzBw&!W9v-_!-U#;NA zji9l9MMY!lf$;Vg*ry&Y^~lP<+L2D3ui(;a7X@Fp zT8DksW$}oP>Biqe{4)TJR_Dec&9-S=VkOTCQEsc85#-Fr2x{ikoy`agjhh74l?|8t zY3YFr;QUj?ahW1tEKcSw;)_St8Gc!J;Mt8WU*5>p&w*7xbxqVPjfwlYEI+oIzUfsY z9LL!(THfpa$I`&Uit>>JOv)0fR{d^clR)0H&)LOC*6yDFpWHckN|`aTv!6w-U^AP$3FPy;bPor(Y89BMyBX5|AI{DxMiXF*)@|F%P zYN2ZOrkE^GMX;qSG@nfKp=yaNu}u?ihLvK)Sx~tXlx$&GrguWmq0X+x(x#Gob$GF{ zSLfc};Jr8K3QBq^e>@2hhz#*@MOp9OrSic1xr8bQ;EHa)bmt)k&FzyXS-IgsP{LYv zmk&w9e7P_n9Lhc8&;=O~Z(T(~$}s9GDKTB*Fy?~Ra)qIar+s11qZj%Yo#d?fdYLk< ziY!(uPPPj!_Wi`Y{eyvrsCFe=Sq zTtsiN-_M(&fG)74m=`X}YDKG@^K?BX|IjSsyRFFRl!LDtQmT4iEvL{Lz`5!Jmc?7& zDRyP-rk2u&!a4yOmtg2w3F`lBLoC(aQ5OP9RCZzQ9DT%2_-wMsrM0F>;|-7Sev%Tlkz1Q~GOz%GC$V!hZl>(edL{ z%;^LS{!1U%4Pnxvc%`r4aYft0b@{lixH@ymDa!b1`VY&{g44`;N({@^=uBfr(V+qM3R4_I(JpjOZL(Os(#%QxQDEJYsKlbz zBGXgoo=Y{%k4k((>D8*-Jd5W67kyzOzmZ$>(Nitgq#C0?%D;1g**UOxo``l|o_=B= zrmN(FCyH&ej>fwwq%;!|fqdffT|>JuB}IwLf;{5nw57?MmU|wKEuy4?u3W!Zb?`k^smxg^Ds zJIZ#Hd1+}mq51|xti8#7T5v>Vo-gj%Y!9Ep2m5(@oz-HmJY)i%77G5KRd}&w6-2=U zix4=VN@2j-afDyPQZaD3zP%*Dq>_6blvVu?+{QMC$c2U?_xx1v@>Mq$?NiJjrzd|e z!JNXH81*zxmj;gZx;FleyIudo*^~tZt}TVy=85HqRtxvMpiuvsb8mCXf~9I`m4yMm zF=kcr@Qx1Y`}(P=rw$o~O8s_je5oIHfHajNBEWJ zcsVNZ>@9*@eH-lcM$UzeDf#|-8uR#EGeGEYK_ATsca*(EPAr^2YA^Aq&QUJPqP~Re zYrZ)TeOrr>rr6n0kf=(X_$epu5#{P3b+i+(oul?d8wk=}^tD3VxqV{HTaX6s*AjI_ zm7FU0TFO^@?zLM72JurK`G@Uzkt%F73mW6dn6#eGBML3j7TF=cpN$T?QL()B`sq&T zOf=X9G7p>VV^RxUow!hi-W<%rvC%8B({?opbQH47^EVXPU_2&C*Q7U5 z{~Y0iNNj7@kvf;2B`AE_V6`%E`opehL=;c>46i@SWrS3pNd4KGFO?xkDg$kNk`uM~ zBkigx0G-|2!Ch;fj&p*)fyV&+P?t^Gq4H?Pinq1>-WzPQE>kAsoekM#C%|>fC+LpF ze%9-(de0^5>;lcU_51Kss@4Ci(q%rX)@PS#wYKXIYv4LZB@wh1z=|^@jfC>na@7zeBwq5AR=m=J5!7UT)IyybKBMywiY7UoW_H@2PJ@U0qtf zZ3r5FcE#>Fw9z@q$L2l6s?-o01vg}8<*W%m3+=_W?}Aot(HIb+B)YikrG!);jZoOY zlRCFHeN1k$E_9gDTxTUE@5rz*_h_w1#AS0Hau!r=-cHRIWE_isdi$QuaE{e^e4;Qn zT$;gm-^hL3Yh6W3kZXok;=b|F#>eL6ZPDsuRt*#ntOOq3 zd{*0(L!4D$85d&CyJ1qL=7QuF55pl^vn>rEcAoKYVwmp`7tS3!$2Bx8MaF8f4(?-t zcTHakrxOcl1JL(poa36ASy*cDhEcTWsuGA`w=JJPm{c@qCinTP+d>hyP{**~F^@DE z6P`sxBV+^M)nY%I;WZD+UFef$UqO;j{JBJO+V2?>kFPN#_CG&FCZ{jKY@Jt_KJ<+I zruD+T(f}~LYVD?wJYu9ZvV_RYO-+zgDWwjqn?y^Q**lglaA($8(ATb2q8+$4&O?ze^D1m$3n_CA z+=42#JE;DaM61P*a{e9WF;?YMyb|Rg8_sd>043_Y_|JV;#(-{wzQu8 zorW<`P#&p!8X-$}+E{X_48qw58W0KBU00};eabpYBAQDaNAf=gfQZL%9OaE!cH>n6 zCF-lkyz%KVO(n}fc4^`~HWYgAC!Le&)pxW}rD1NG&KqRnZq;=H`p`t^M)7xC8*34z z7CUF^l)ezH6l?3x7Fw48MR@4F^=KNg#3Aj2u7kLgKxz4m(xD z0{hBLi4BnlkFeBskiQj2R?YApr~U$Cw6ZInPiW_3hwQ`45zyEl? zMRW~8a79;|HZ-iM>N1|s{pL07i1;u2Lj;d@P=?FXrN8^Zb{$EZPLScNap(X0`go7t=K9BgXTbxms8iy#c>aFRc=DIY^}qJ zJJIkY-A#qz6?p@9ZxZdVu788UaKQwkj7oEBd@6X<-8*k>upGz1<@k?kMxCGT>k6Y+ z9IdTP+T~xDFzykAx|3Ch3C7`aX_@J=+*NwsyMUv7-poG75gN5H%I~=Dw-k9uHe7@c zq&|(kUs?d(T=EvDRI}T$1~0WDK{dje+LeW*2Qzl*1{~hV+FY#A(t2v%@E>WZm$W0w zk8e*1c1MI8nNLkDKj8X0E*ZT51{a@j?h_*6sulW;aP1*xEAZg#!Mdw-EtW~4>x-zJ z*_zP56Sk&y;C4Zzo4E=nb(QQhXD%K$SXh5t-2VY@>b4gCQ@%7kR5JN>n;=@~g*F18!E2Fi5vY`mG5EUy<7uEk3U^2R|^{fzx3{scAS_QYGz^bLV z+5&yKSiA~i6-}mDvL^w>GKn~EN!MO3KhqY54jZQk<0Z2y_TN-P)%mm@cGO& zcCq@eTaTwx6NrEr`9_0Tj1RM#IS0+*1#h$O>lNNSeAo%gRGWAKg70P>N z{9+3D(c1T|UVC6s*P3c)lP(?q%PWfkGaq6#x8_R@mp~VR%-U1pxl_b;rMU{!C&MmN zyV8_@%ntNJd3p{4JA&?rBoxXC>_9B_iRhKnmlFusz1t5u;w%@>i=9*g3?bRi zL{6W=fu-MyQXlGMZ})>bbk@T9k+3kuNnWAd_IgxI^bdyq}WxoC{QHnxSS%0;9Y(P;IHTsu-!K4 zOg$m&cX@%b#g6-BY`&br4EO$N`eA)3l`9}kSP=4VFbWASb z1s+}d$Sgx3&!jNz!q`UFm#g&Mo?9z~{v@@n4B4PTvG66r%+1TayELqEwQc-jeI5Z% zeXs5?;%#@awuoD@ViE57M`TKIU}c+{gl-(MW57GIy~_4(O}Amli*0EmLf_x5(Z-JQZwqVwaxtuO46>}VR-vkwwacwx0sI#nLZs$LnK%5UfMKn+ z41lQSm!4+}W&;@%9|nDO*-I^pxK7XSIVyA#{n@M(>yQba-J%>l;>A5-Y!wX)`$Q(C z1pmSU^_0M>Zt##vIzQl|Uux>%v&}o>$HM{^iNIToe}JDX6Vqu7XEDk0x(RFK#l6%0 zBIxN)QOmDEk(CMfozwNmuR6~Kq}DwnjgR1H!kgFQ?8`}@y58PH34Dy$8@w1XyX;9j z4~3{7K&?&f<)u=$qd<&Q*5Q&7C}@~w{h62u$ty!^jJ}XRj&Vrf`;y~~ft1J{*ea#H zzV1I<%LJtk9k>&yJ0u17NQ@#_jDaF(YT)vMfmV7;OF zW)a)9>2u*A&zN~4gZw0cUGO!6%3yxO-^mcSL{<_}wvaf!j|&&NX;yM{3^hn!754Sa z(_>EjP5y`z-1q@_uHef=g**US&rMUKzaW6*PjR1_Oe6jdF||NddcC29)Npe-UY$>H zZlC?sU#BF7bDF<=UtNGJ$3~|XFC}MzeMt!IUuh0HBg?{8Ex0`r`AgI{*Ax>z1|ieKQcE_l+qZhgNF%1;v(jhiRGW|f_rURdx7f?? z>YJJ)?n}D2Id{Yk)J;#s)aJv9M|{@wsf9=U0^ANxR!@;T+pH~RzS3LY%I>Jt2W+7V z7_Lyek=>V%9R#`VqBZtheXFE`^QyZVlH~)?_-Y4IMUHK4LPMduwFJLYjVzkTDQMa| zgkk83o_xDHC<4$GG7!?Y#$I;gZX@skZa^5rrQ6e`gdNGzlKqM4*jD9Ko_szeM#KmJ`$@gr1!XTrihJA*{BztS?6 z6GhG5LD5XV5=XIG;B%ns=f}EYrpDw+?Z~!(ckzI4H>rZ5TFR)TZ5Vi(YF zNW|D=iGKj!x6Yacr zRVVc!8Hb1(%N=IwXC>@)jcG388A4yebgdK*9}Z@P?*(b>J?aF|>89E9aFMG#5^@p9 zrh20mpC^skk}IC*OzJGYLz$A5s3QSeHkvLSQWHX2Wjl4*3v39yyXYyLu>jKi-dN*z z30b{AGqVFTf(@ve(2gPbNZl0eM}gvPyR)@y+3m3Eqo(c5q(&7Jgc{4OC0N?l4}QOu zQP8n_c8uX*(86I^v{h^hB~w!^n{0G+4tSjKbSegv zNwk3+dy1^__;HY?!V0e=o?rhmdsVKGClx59$)1>`>!b5bA3~v4+ll3+kXNkI$HONo zTTxODvW!$%@v#=KS@(FfP-MMde}r*r^q=Brl`u{HbcxAh=V;U1x?g)(>491~EMhe$ ze{42%fz4Pjawb&C+c0uQDTo_2HXz;u>HK=aVJ+i!07)IS++<5|8=*#W3y-py1n!BY z`mN3(9~sGlnU7?T*}XdQ$kO;z{Lsh(N>z!p?G*J38{ z2A&Je|$t)*?gKtbUDUI3(6nO6NWkC%OgVH zxLPwKXyMIW)Xa)0@Onedn(h=`0bN_@6!RUTc;-t1U)Je@fk(+5P%g!<74NQQ@kl|N zvJ*k`b-Vq7pQ2tyN~-;qS_Tc9eu8?<0?*{@q6r_Ep_UOjDt;v%9~&_JUTC*x@J#Bk zDB=ZCB8l@k8O`>v%eR!xV)(G;9pbQ6Rlz&{6mKsQDpH_K&Ss+c! zw7dDa4i}N_kk%KD6SeQT_lWzN>NgSvv4aaYK>^;L;Dik^j@nOy}4NU((ee)B0 z9ISSKl&m2U9T02>-}VXLBTvp~BINttk&QFv>!a8dZzVH3Sym+{J_SNt{I^xs76Lut z6H&FfQ5YkoEas-RCL`e_#AuoCPxNT|2kn{3CysgE6y~&l z16FVjV=;LcWxa%SuHX5BAlnu6aY}9gK~!+!E~IZ=ie4qpQHg(}<)8MPdC**&tS|5j z+bqdphnHB@shje6K=_&;xLq7RaDlNWkGc{BQl6zPP>uoH5M^ z)iuS{3>YsoN0XO2Q0OPXcqJNt)Y?@fa5?%n6Vd5gV}`8moos&#%j@kwbk zK|yG0IYf?`2@KLt3YdRePmNEpQrl7+R!UnqWo{3RolL1k&ev7;9bOGiL5R9u949_0 ze~%|qY{@qsEd7d|U!!gFrCaUU3G*Y=NKxsK^W!{GB+Fis;wZ7zkxa~BghuBSd7(1S z#y{@=EY@>gXwOkfp-`_+lTzBqW;!q{zPJ^0I}O^Oh4$k0<)tAJEpHV*;U5EFxJ4ZrszO$sv6bl8em zfds0a2#JgfnvH>fMrFVWX|9gA>ft<7f}jegCp<~_Q^3Eg`bc{OwQrn}(2)-j$;hPL z%z;8b8}}nk{PmjtoX`Z}Sy!$wOneJu)o(m1s#TD(NP*3NXS?coC*4;gaV5)c0=Re> zg4{jm8qITO`fK4}nsF6gEgd9Hw=)y+_Jo36H9Cpb7t);EyS6Gl<+8gR>9e zRx;^MeVUgwdD_qRc;u*0#AA0rGZe6b94Okh+mnVQ0_~N|gD`>KYe9qD98W=%U0=+l z$7;a+Yj~=1s>*3C<$F1em70pKfHYFZ?+q&ogLEBU1E zBA4Xm?Ct1yUnS}Xb4*~6IJ0D2c9JMtB| zs6um}@X)R7a=fW^n!b9yEEv6%#K%c3|8jmN0!KFsyZq~fUybvbvB3)VYuup2goA}U*F<4GLmxGqC2POL3XTw{5Vp#VAmxFj?(AV&9w zq+ro}&3Jx~nueYoNZ(w{hqCAU1%>&4&)r9dRm{NYZoF;W<&ELBA|814I3v*45y+dU z1ZGB!)~5kCxF+qK-hhLE9U|{5ofa=sMBWifU5qc}u>@xbv-HVOoX8R)wRKur&;|Y* zefs1~q{gts?anQceyVt-zlgwGSr7sR8eWYasn^Dm%OX@^&6=<^^Q9V3-m7`d>Sn4H z5n_II)<^_%{A0F_o~h

8v^cNPzj~kedjb;8LBUhdF3v&3)3=MJFNe+TNB=JwBVd zu@Q1<>x8ljM##T3f*{O^A3{liZD@R5q-|=c$u0 zY88*l@i_lM*nEoL7v;Hqbl-0LYsA!_5bPgkexG^&!Ps==*Q*%;OF@+SC*zm;Bs|Ef zMcu>Lpbz2Rxk>&9XOngeu8p8qri34t@5p$69CTiyQSj8~CEu9QQ?_V=lr*%Am{C!I z&0&Z6S4L3TX@lA7yhzcd*D3};k9gIZvBp+=)b3)arTkzuRfuk*p}RawyH!{ss?P<% zM>}XwplmaS#gOegs7jSmd2yud-kZOvFxCKfMG=nbxoe3KV5XqMWh|P2}^f*4G=8omxj*|iABzM zG&tBWCYzj68jhjDUm#}iL+JGV&x8_i2rR0XB@+HI@HLj6Fu^uhA`e1dP+h(W9Us!{ zuPqIU1Dy+_~BI1VW*gAejnELXeTQgzh&qdH%S<%>rsd z5WS(zGsRSbAPk{Qo^UP4`5VJqFh}AO3|lKh{x9ydqqH(>OW9#+oLPyOGOVF{xaV|( zxsqWKYB4XrTPA}+sKVs`HoOKSo4=+3^zIk5k ze0WGjL17hOmkmlxOJQNtDXbY1xq!F>QtqEx7tXBeLsky{i^K{3yt3to6nzWdwfq#V z0IJse?B)#PE!ryaC_MN;>}>WQ>RnZ)~r(u=RjK8u= zr~8?E;m0c?d_`06C*?AE0Jh#3BPY}D*W1%qe|=}ATgm{i>$Pjp3}(u&>G{7Fe@B9G z>RL-($kZMjzwGxi|9PZ!Puj*Vn*1ra{C-Xb+$&Wsb{rJk$U1gj)HSS%mBOFnIPN<^ z%AX(_q_?e{Qlv%sN80Z5WUAfW_4kHu(CB)2##a!|+e_)|H!;|v_9JyrAQ#4rz6o6g zw4db4U*lXXXYf_8J>_^Z5nzXrq7fE!Rehfr$_*^F);y5#DTWuJr0#?CM7ENivu0Avx32wpYZsNZ(eF`UY z)1tQM-mWWqdxwx;XzaaEGZAM~FV(1RnbvOOG4QMl^B69+LJIMkB@mliDAz;xHUhAD zNE-n(}cVRy7sl}O2c|NiOMNrn(S8+AZVugH2LW9RSg2SA)@m0S>@Iuby z^l*8m6-!U@>e~W_>M6d@wQ(akC@x=j6U*xSW%sy$#tSOiIH?^$k1c)UQq3nB&IN!A z6(2Ped37SNlebe4AWpYF!eS&G+~;t}`+OJ9NES3|v5>N34pW8*5!bvIPo^`G^~0C` zSJ3oT^X!P znec3Ng#Ky33du$&H>E&0ZpG|;-!5J1fzHv9+xD$>N=-~{+=PO673OK#U81-yE|ZG) z#r?<3dE@O3W+3wSLS6oI<7W|Ft)>^{>kU7?D$ASY{jg8-`hAAT3nrTjPXA*cLj8Vy zIh6QN%G>4j{;7T&+wjnUWIOb;A5c>wM)0~ZQFZF-7Ozac*(wz^a(v?sxaTw81)Otmi_zo`1Q-n=dY)m$NTsB7U${$&@=Kgc_|yWPk`=+2@iSj;EUt{ zS;8pycCq38i;=ya0+xMr9}`Uw*keRl*tSeMZ)$EzB=GO|?+VC^);zlLG<`^df@Sa0 zBvQ|DywyR4w1ss|nF`VK4!Hf~7TcYD-XhXDF)9Pu1=8#;o_A=tI&(NCFD9U0j}u@) zzlng!X7xNa&60dPkhI`@r5RaM;22H!7Ur*bf5ik$xQU%{FN{G}VE_@=Jo@YI8kKyF zL1mVc+}t7nf!-_hheIwVV=86+C`cjR5M6 zT@Yi~LsM;eI*`-(#)`heawGOkm3Y5eB;zf4T%syH~~#;rLdi8TQ8nQgCJqCQ00Or=t)>ppOfE_X9?)0 zsig(E&il)7%s(29l(Sr!O+#`tlgTLexWK&J_J zGXSw-TffBD@<#Xz=FU@>A?kfEFJS(FjR_FwJhEPZx`D#f#)*Y_PgmR0WRbTU=?ad& zpQ0*qx$;0LJIUPj>6Qpis*iC5qYxP%-$Hft>kGi`LJPNl^JM)TRA71u+AHcM`Ft)V zO7L7Ws%`(2e2PZQALl666Mc$?Y&&y&%%ICLlg3XoA9C}5DZ}O}g^%YyQ6GLy335#D{gFGaK}%*Z;!Cl zR_{XKtF6dJNx4%cUq5MKzpU}ClUzTXQapINGR}q}=5*(36_xtFy$!9snX}`9iUY@H zBKTfClBW*L8OaQQ&xU29I{?#_Vm7we_6FA31;@COjTF5zT5T{0MB?)04={9tiA2bbXWR& z`%dt@Yr!2PAse_?`M#|@Jot9Mo@J7omdYac`I3P7ChWPsGva5HVew0zl&(HLvkZJ* ziw?K5Yc@o>>RnHjk9;gFA=fQD;=oq(HjadG%QCt7k{=xKR9Mst*GG z=cx@l`u=|UC@K3BZ{ki-NZPWwWVKA_aoFolnN6F`+alXEqE^S*H{bcCLQ_RHSHm4` zLIg;GNCmAX_G@L;F^j#ezS$>I%T+s?Ytauzc5SYE=Ll4zv5B`=i* z*i@A4;l4k;DFvY1@1*EMw>EHc(B!u^b+FU;Kfm-?stGtxK(f)~xA(fJ30#R?AJ+yE zHMC_9phR%C0t!_kJzon&02mo_teXuLw6tB0Yw4_RDrGd|mAd%JQNd$*Mrt0F2_F2f z65bI?3XDY0Gx?5_s3KfA56MXBYwx|O5j4?$Fv{CyA+#6L^NjmU0uK5ez?v65p^m4- zmzdh6TvB^eNj)NDurvhP9%5DQM$KT24^FI7)j+@$?QM;>W&1;_q|-3)SU-;Mdyj^8 zQuB)ss5HSkEEI&*api}bXr7EnsuA&!0E4BDW|tW1?l({7O-aB~=x|E%n^OWCZ;!6Wza>_q7Um`x4tl6FcYc_hzn0s>$|nOc6)pmqRM}&3G+o%KW_f z1yJ;gPVj?sk>NJayyYn@sNcWl7Q^YS2y;GI91IN#zX^j(C7bNhW$LOza*pUc-{r!M zeMPgk$`S|kSo+Q;U?0M~9BOR`cQy;oqC)bFqkDajEcu+-xe3y?>X7)m+ z6p!8o)DAC6zKCbub}5z(-t$TKcRNqxnq^747|bW)ZnHES0eZuC1zGKJ%DIwd zTF9DQu%2hS`S1bo;uOLxIAZ(OTSFuT;5D=tFli>-Y^qq*wXvmcj)K?QgRAOuBMTvOE6_?yN=ecHuelnC^wq9{t;5nR26-sXrT z^D;#vT-ga7R<-~>;i3c*4Mg0sxm#E{I$O0B>;Su=-VjSqac-?8;)=mL=<>|=*Tgg` z#Gq02(3J+W?ChTTmsEsVnl#DPdJ|nh!74X2tRM-6P~)S#V4Mx(lQ zg;Sh$QH0^tGWNr)jMeLXIQ#~@(W?H)BY6&N$ykU8#NX3Yh~WcO;Y5(BFi{-VXpPlt zR+yE`k49nN$Gin#eUKFbRaZ&pD4&i8mgg&hvhQNvTsf`pkmd7V%5)#2mrYNutdfMN zn@a#S|OZot~xPGTvM4(VOsAwn5dzc zHs@XzVuoq83~FwSIMs5CN${5URk=bc&e`t!9@h8^Y4;ocf;5_=!^PhL2P~8!_9K-~ z1q^OXzkfaz#7R1{A#SSqNHrGeL43ys@Ux|EpAOU$cP#~fL86VXroVm>i^Bbsdq3bn z;BQ5AzgesrtA^BRi#+B|$VFL>82e9c+o6?AU-9d6AeaM!qUzK{kx z*9;6?v^Qf$e$HXe%`)TfwJ}V8NiffaVL3<>yAuNDzydk?KE^13gCB>MI$U`?0wYVC zGj+8$TJ}~;LunJb?~$aMj^B*d_VfWM7kjFrp&a8+?~DN3ly2%c8AB>-V1&!aZ3y*g zwr^u8+MAdXu~+rKLz5pboi$DvddF^|B3s5~d?(a5CUud5>%94;h14lBYfkeeW*$t{deB9OQ|TY=S4a=4!fHY36FYu`MZPI;PBx zbdxBDD(lScW@+Bq<};m+`v#)AsnM4i-`Bmj9uD3gw5;5(o!s1 zwj0%@^D;op`u`3sI-=ekAw4kZ6C1ne=5^huvYE|GCDvQxjeuSD7foKc^(4TWxESDe z+XhWlSjEpI$rRlBTPc(G>Euur-TdYHSSH(wDmk@iwM6ji?M%+tf4)QjPmF4@#6X%^ zyQG*^yj)trA~C6(P`tdv{3je*0%ue%kjaTF%d^53HL#n8v8<@z>$f;R`9=`KxpvA1 zO7k-2d`fK-sg*Uf=0$+HHp9PJvw!)Y=FdHmi?P0SJzRRLFx4^FNW8% zHd)DArpBxt9X9^E1mOHfZLe##salfP#B2KBVWYi{RkdO&+2X%DSk24-XY&;Af7Ho5 zR_3z*t1kVHno-|dMOO*Mo2q_4H7>HKH{O@VDYqnc4n;2)1S5@E8=}kzsfOH!}8 zxWGbWlhamN?Oc@Udh1AE;yXx_y+}-NaXYZ6tKI5Q;dwUsm(N&5{a(tPF)rB!&0W(n z_gkpb7{nz2U$p&ZWX&X35u-hp6$|MUFuex1XzICVGtD@hzeI`p4-h9U!hI zo)8N4uLp`Swn#cU9^F}dB8hm}hMlf)ND!`D@K2uP;ezw|I0v;lq(*8;U8o*GSwu(f zxiwhN-K}}W{cgV{*d3an)DnmV|Ag!4=!eV!PHOlleV~&NZhiIU%Ggs*G8)pQ`~=4I z`S@I;hi<(4hMa{XfZ)w`)#3TNqj7Whalydt5zVH)j!b(|MDNIXSQT$DFEww6LZBPI zF6kNe209{Mw>!c7S8Wh96L6DraOh%G@f!YyB-(HIo7LxMyDKk_VAt3&lVJ83vhfYr z+K2D|Z`tef0IC!F(2H2HUw%Uvd532~i8{k$G*HC*M3#u?cIB{s_Zv|c_itttMdJl= z2w@Oq6HprYQ)w|4;wMWtw*Oka+YOOT5L=;1Fy_K{P`5BXjTiQSN~^-F`|bhn?XEmN zFNZ)ftF4#-%&EK!VR6@7dNxYMcChR0TW_CP)P}!I1H{q<0|~` zDaZ({F94bh2bC@4fWQ_P#!<`OiGA+>8|)w|FdUpL?A2s{A3CsSOv*mI%)n5+9HG6q zDXlTsNqQODqk~?&hsRUO7SDI(fDS~!7fJ8)#GZ3H>mOpBh`eU&NO!!NB5QmI zX`Mg)5~XLkH?M^U#kaq%@uzC?DXMWlnO9<+7NG6+I=HDNX9fUT?K?Hr*s$efbHBLV z?qeIsf4q=!A5gOzy4MK>-`dylKgJyojRdk#q=e#g?`6%xHy%xl&A$8u1R^A|bY&Hv zg(7V$oS8mt$h(AXt(8V@8!c*sA>1kvMMp(K*~K^Yl8`sAO}RmBjWk#e=J*`|SMr1P z?h2KZAACgV=fb8Tbl}d+#pK%zk(AFzjybjZ$q3uUhxWDzJ%5~eCH0PV$$>WWl6=h` zY#c3Jdh=KP1{^a{pD^e(N5QmAk|)N8!cie~4=lr_vT={=^5L~J(e%T<%`n%pd_knx z8jot(BYKQnk&MLjlpKnR3|4sNe&I(X5m+g=4PfH zhi8c)3#3aMPBoVYyg3-}IZbt2wX)2Nz+x$GC&Dz%PyXt7;kI6!x0}jVr!<}j(O`87 zv!alhH4J&*C*v9o9#*ql1b)cl(rCrcwm=nulVYlT?k=ot-N!wei2Ud8C7Zvz76U>J z&I2%-TxS6FMl(|4@{UGtg;}(EA-H6+<0^HyPvw1Z>Nh5|n9p%3t#iEYvC?V;M~V*b zZFX}Pe%G?cQMHKYu%l8zJU4g!3%Y-=2IkG+Gnwj`Vs3_uiJmXxazio`9ftfO9=y_M zH0GEe#wF=~ExK6tt*M05JR#{-i5cUTem+B$_<7MMiIcLFx}`ouBwag zz}Z&77sm^rEq~G$5?;mmjE(2i-&Xe+38%wCVDS#pdNH_X& zACNtdG{UttEi*X3N-F24EC*D0Q#<(&b7MhJWgLFSJ^E#0PW71xsdje{LC)|S>a!-a zY}JiKLH{+=xo1!wb55xV+V2=!*4Pk({DjB*JRKxB#HA?0N$Hsoh&zc#hCHl!{PV2# zOTd_5uk>|aq!IKPhsrXjp=IMhE zolo7Wi1nD2?Dg$Q@}h{V^J!=H%!QUV0~VD7Y?gMMa2Go=RSk^lX&Wb=2vxe^z% zb~jKq=Xg^j-+ma{m2(;+<$}Zv7cy=w0MC=? zr3x+n_j88vR&?Ggs@8$DhuZRm89aBCOTnMmAP$mfCfDOVo8`^WekNH;BIme_J1o!p zC>}r$Civ@+4X>CJNzfOlRFOrx^4Wk!s2^kmtdHurK`YQTeG)jdq-KG`bthk3MFS4i zi8I)`79edVwTlgx|4hBFNF~ys@!Iaw)*|R-M7j8gSzy>aV-g0QV;P`2p)VQ9=3b|V z*R3~X*s4xg)!W`5U&?ibpqm=;qPV!h947WhXw(nEn}qsq~D-`KAX8GqV!=< zi|atZcUdfQqy;U_x(WJ^XNEG;vClZ;zD}c0E=<3mP3s);kAqO-WV74qzJ5%ol^|DA zJX*Tt$;4~EUgyvN2BIU^A*j$J0S@G-O;1`1Zp$vpH6y04B0_!>hjf?7!AeSLPo=k*q2r84a9426RSn ziN~*al=DxV3CnGqr`pe%%)Ka&*#i8OwMr^@CTGUL1{WdhX0i$lSC`rcn@lj-b{eMx zlH8Q=Jet02KosOpxf$}dq=@PEp|dU4UqNjgtXH?i<>O7sAbX!CTz2In$i zLaQ-LQg{xfDNWBn5kPE~;rGb<-Dt9{|tnLZW8e~qq5y5%&I z1YCh4YTa;d?A#@bY7U`=&fkN@H(jw7Rae#Uyj+(*z z_J}x)fNrsYCZ`kYRzhQp^5dy40y2wuo>*}<)g=U~k1pnIOtyq`Zld4X6WyaW75~un ziO3%lh0%KqxT+>r4t>t<(S&@V)#vjL7wS8!v(x(opr&FrWbDNrh`X{5Q{Fz%oTYt# z+Q;8(X@=~tBBOsh{CO6>U+oZBE7(oy46#*H&}idAqCnP4-H@i77Q~ZioEi$pnA@tGVwVRhxO> zAc!o$RB`GgF(mzjpJVtlGM(T>M)As&9RWYb6X~(~lkb6m>|_VbDd3d@+dM}~;r|4r z4_oj*@I1b}9DSm@yhe~tXPx<^GaS#;={Oz*0|0erFq!@mPZJzHpTbihtVhHD63x;u z_3ytAe8$CM-uyIOPNN6XN5l>Ne&fKWxd#_RcoEH%@4SmI^G(A_V!!Bo;`~>$VD@PY zSLVZhFiPVAjbo=Y9K79UdiaUFIG`K|AV_~rr~Tn9^@nm9_dle`>FlaUtHVv?$8gk$ zb%t3{9@`v+k^Vv)^J5xA*k(NH=9pAMGaBPqMKin@V3y519X|~I@~`LT|9tV&H&4z^ zpQ6(ABPmT!GMy;}N<2;Cc|1$!r}6XxJJ8Wzx*r+qVmeb7?*JY&JI!<{__@O)t|QAt zhe;F+ozy5WvQN){e%3ud`|0`l%e=C`a!~MmWW{IVYdEx>hOdu5PW-UtF7X7s&`Ac4 zrNGoGW8Hr{LUgLlcV7-;&zaj|R`lc&-wa4O!n*3;f!U7}K+E5e)P9ezfIyJQE{Ee8 zVGv1{;m0({sFmIm3=r$y4`)D^M$LOO75G6-xXZ9Wl>+15|G)@4~FA*a@A}M(kp9RM_xHCY=2VVzya$7K( zOz?LvpFp}La3~1*^yE?SWE5W|Y48LjCQ$slAux>(NKN?O4rQEu&XTx%5*GdPpR3_C zD?Gu!goN>BDG72D4k*V%x)*?$l;@~t7+RjVPW0jLTnrvx-dLDWUbu0M`rhL0u$g(~ zy?CimCj86$biBB;^q~bYmB_0?euTTZqCkek|UB(5^dARNt&sXXUzI?odp>9Nl=NE_6t)T6@n-&@l1l7#=JkLpWV_w%_< z;8g|!8%QL7GXM?1P*j}!y!LVFz-uQqANs-WKX^qCx0bP{)x57Zhe)cB8KtwH2Jw78 z^&{MYD_>mZ+DtQ@WPj)ui3$%7MGGCJu-1n7d`hcdeA0!H6To0mamx54eZhml=e}mB zGP{g(ewzrZBQa1h*W7RZ+U5s{&u(xsYaT#>{>ukIyrQ&!rnv~C+Wu7( z8WBb0z1?8{mJ4B`zKmJq8u#x)7MV!!o$S9K!ydaN8BpXcj2_&M{~Or{OGAn)%9=<0 zZIqUu;WW~DI3AF2%Z~KAWMq8SClfQ_#T9^}@IErT<`Yy*;+ZEWvU4(u@;7!tI(28n zfwT^J2{JO$EOd3q;7Dt%>n%VA=(+@$BPAUlSO@5+JRFuFf=dHtO&T-N2qGMC4rc0} z00frL3IG`?b_yUSgtME%NQJ6QJxoXwni`OBo^fQfF=}-4ij6=ZzYH0T1L3$ZSg>0Tm5oq>_P%OcRi3n74I| z?FUHK%fcogIj(mKBreHQvk@urD!=2#*ao3QgL;9XZVRPe9BDC#BVhlHq(qaBrQ&Wc zP!de&Riq@MU6YdLY(p@Cvc^haQUHtwr54B88l|M#R&|iZrYi3DlSnl|iN+g)Y}gN! zK%lc#w8ShikR5_aMcQpnO=U2FI?AI^)!~F=dbQqG!;4bZhd1|9?Q|nCVVfcyx&0^) z0*4%}((#PlDU^%{`$lrfR)EBCkZ+Bccoczj=ZupMxfL-b!X;MrG?Tq1DADj;#ngVF zBsrI>SRUBoe5YU%l6JWfE$L8}X^A@-)eKo4F&R1mJWp>cnOxs#mhyCvHcBQM5sqyS z_d`t9*i{xSD4=%Ns<_dc^E_RKlXj?8)Ziq*jB?T#u_c@m16rQprr&dM9ZuN@V$PM? z4>_6D8U!jfJB1U7;0>W0dR2N7$l_X8z` z?_2>T#da*31R2U3i6$OKB`8^JBMpG}1W8DNKnuH6w;C`HV zDiYC3mM8U&wQ<13Ei{MJTSHiNQeXsNNPV$BGBaYN@8Mq zw{pp|(2cYQS>bq~4mYwqoSMpjlD1`ejN8PfZUFz^D3@sHtc>J-EL)=;Uqwp7xg9Hc zOl%4!07==GXAGAO+5--Pl~@G1tu8LLNTrqiUaXp+M6+`xQ~RMMYo%NTCB0i9*-gzQ zZYz#FQ1C=(>kty5HbzI=dWd+OBt71bQ{6XB5Ta4Q!tMz^%IMX+Znvy%QxZa*t>PPE zwidIyd&GIrb0k`r-c9S{+V8aW+}tC!K9{NdJ`)hf8yu{_#1Xd$X@Rw;z8dmLz#tpaJX ztu3o`8fnH9fT+<-qM_7QaJ45G*_DwcyIbuW=h$otBT>;9xO5E~gCD}noZH$flnxR* zNJQEb=}3vG!K=~c59i$;xtZE#YC@f z8-WB!U?R6CnS`FZXr)s~?^Y&BDWc8Y4drwPqaw!ZHWH03W?b-XW3)PUE>`_ibH-Ni zJLv#L_Tve+A~Jksj8^P8__t~34^u(Pv=m{5TpdOx#I}fL+n|z5l0GO6hN)&!q6x~9 z$^E$QoYE`RJEp|01L>O@Nm^9A4PYdpK3s#6b)l0g;#*54u0@=fm`RZ|K?#SXD<&hh zCrXk^ugsFvyAAzmV>jn_WgT3&NQ%0C=Z76L&!M-15@Sh}^u>S;8g&O6f^GKWTy;R< zD{~#RvpY7FK&8D@w&?aM-)@d@V4Ae#v(~H7~7mT*-3uBx8-g>dV>Q}T1iRV?mB|5 zH>acul$>@A!6X{A3^0(}dcckB^-Vu=X3!W+q8Z_!qDSiRe0SbF++jM3+Cm4fxK|yE zTmzJJ6l%=`lUqlV`XH6!n9HW5AwN!Bc^Awc67ZtPxz#%Oj$N!=v*=4LTFD~Y;CA4k zn@18Xnr(?GjeD%ubLbrujbOs5=tlB=!AVQElA0{rac-9ooBG04)qxi=(LhWz$y(U0 zS)NMc80@82R49%4wc>$l#E%)^7xLOLwsX5t2Q-T33*^0MmWctnQcL2Y#a|b>uK2-t?=W{ zt@k|b0YsLq)E(5W=}2uFtx&Q;M<9|w*Q%MbIA5Dv7MZO>v%Mbg@j=R!)r~;H!Qou( zsqhbUht=FpnjOcKl-h!l%3f9nw!f=K{`|;r$>p|Q2kBxVRICN-yGEe|LeEIKr?hZq z?dkw+q;?D@=NilX|*#yGE6SQf>1Cab&btj{J6{Wk?&Q|SY zv|A`?^QD@&gL6}LM;?s?w}x-4<|r8*CRhJ=zd*pg>d_3OJ)hRF_sb^BXL4wY7UsDUb)L8vwdCh1dx*G?QXK z${$}D7L<|XyM!BeVh3@12&p<_^z+hzfo#hetrTg>2RezGTB`U_@Qm%LS%=oE zqq#eFoHOg9&Ep#tUmr;Nx<{7ADn{DaZB+~f%6bnpcCJlB8D};k+W4Wm2L&Uc_-^$J z=QdU_fMzPmE0L+^f%##I+-}SK+#pSaQHhKDdIg9$9CpMH4MdVxoJit}^C`vVI$$Ng zNa=~vF0((|ME;DEL+lo7F$Nn(<;F>##qMkk{^ zm2Gn2gWrv$WYli81|>G{pHiVD%bVC8&aP7?vfKKMVy)PKC6f0=)_>Fu>|!TQz>faU2G+BXh{*V64TVP$1wH`G*mc0D!K3?>FkuEz#(fEz$iN89Ro zG7-^$a|5PrQ>cPupJ5T-k8EN_uWUzz1I~AA*Se;Jh-<;5tJI`yB%-=H?#H}k9-QrB z?KWmnU#ouFXfn}MWXa8bYGyE^wbB{xiwAe-4L8x|lBt|qq`6cLVLv}=jmxbQA`BSJ z4*Pp0tSRl4W?+J$ujL;&>|^EZv~FqUPdPutF{}_bjR&G&33jugAGIT@_Q8+foEg0M z4?KtK=6iKZQ;8}q4eC7B57*+}eZ0Xj^b;d{BJSyCbfD7(J!SiLd~e`CpZ`BN##-%1 G69fQZp%k6~ literal 0 HcmV?d00001 diff --git a/eval/dataset/arms/copilot_v11_deep_val/pr4837.cost.json b/eval/dataset/arms/copilot_v11_deep_val/pr4837.cost.json new file mode 100644 index 000000000..70c7d5275 --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_val/pr4837.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 1041.2, + "split": "val", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v11_deep_val/runs/pr4837/run-20260813-134521-66bf63", + "recorded_at": "2026-08-13T14:02:40", + "attempts": [ + { + "invocation_id": "inv-914f48c1", + "outcome": "run:run-20260813-134521-66bf63", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-134521-66bf63", + "llm_calls": 69, + "input_tokens": 358964, + "output_tokens": 107182, + "usd": 0.347113, + "source": "spans" + } + ], + "attempt_usd_total": 0.347113, + "llm_calls": 69, + "input_tokens": 358964, + "output_tokens": 107182, + "usd": 0.347113, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-134521-66bf63", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.3471, + "minutes": 17.32, + "ci_minutes": 0, + "input_tokens": 358964, + "output_tokens": 107182, + "tool_calls": 121, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.6224, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 69, + "by_role": { + "agent": { + "usd": 0.329313, + "input_tokens": 350855, + "output_tokens": 93022, + "calls": 67 + }, + "reducer": { + "usd": 0.0178, + "input_tokens": 8109, + "output_tokens": 14160, + "calls": 2 + } + }, + "cache_read_tokens": 4899712, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 3.93, + "gate": 1.66, + "review": 1033.57, + "report": 0.07 + }, + "time_to_first_result_s": 1039.21 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4837.trace.json.gz", + "trace_bytes": 20549 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v11_deep_val/pr4837.md b/eval/dataset/arms/copilot_v11_deep_val/pr4837.md new file mode 100644 index 000000000..459db6ebb --- /dev/null +++ b/eval/dataset/arms/copilot_v11_deep_val/pr4837.md @@ -0,0 +1,36 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4837, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head 78cc4dcbb161) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 3 finding(s) below | +| Security | no finding reported | +| Docs / comments | no finding reported | +| Behavior / compatibility | no finding reported | +| Correctness | 1 finding(s) below | + +**Validated:** +- [sweep] Read both diff hunks in situ: pipeline_ming_imagegen.py:446-472 and orchestrator.py:1287-1301; both match the provided diff. +- [validated] orchestrator.py:1290 unconditional unwrap matches StagePool.submit_initial/submit_update (stage_pool.py:932-944 and 1001-1020), which both raise ValueError for list diffusion prompts; the prior already_submitted gate left the initial-submit path broken. +- [validated] thinker2imagegen (ming_flash_omni.py:346-450) returns a list of length 0 or 1, so len(diffusion_prompt) == 1 is exactly the produced case; issue #4839 confirms Hunyuan ar2diffusion is also list[dict] with a single AR output. +- [sweep] Multi-element diffusion lists remain rejected by design (stage_pool error text points to submitting independent requests); this PR does not change that contract and no current bridge produces >1 for these stages. +- [validated] pipeline_ming_imagegen.py:462-472: ZImagePipeline.forward returns DiffusionOutput(output=tensor) (pipeline_z_image.py:729) and DiffusionOutput has an .output field (data.py:1186), so raw = outputs.output + return outputs is type-correct; the removed outputs[0] indexing was a TypeError on DiffusionOutput. +- [sweep] pipeline_ming_imagegen.py:449 calls super().forward(z_req, prompt=None, height=..., width=..., num_inference_steps=..., guidance_scale=..., prompt_embeds=..., negative_prompt_embeds=..., generator=..., output_type='pt', return_dict=True), but ZImagePipeline.forward is def forward(self, req) (pipeline_z_image.py:413); the unexpected 'prompt' kwarg raises TypeError before the new output handling. +- [sweep] _ZPipelineSamplingParams (pipeline_ming_imagegen.py:58-76) lacks latents, cfg_normalize, extra_args, and output_type, all read unconditionally by ZImagePipeline.forward at pipeline_z_image.py:464-471; dropping the extra kwargs would still fail on these attributes. +- [sweep] DiffusionModelRunner._execute_request_list invokes self.pipeline.forward(batch) (diffusion_model_runner.py:462), so MingImagePipeline.forward is the real entry point; there is no alternate route that accepts those kwargs. + +`vllm_omni/diffusion/models/ming_flash_omni/pipeline_ming_imagegen.py:?` [blocker] — Confirmed at head 78cc4dcbb161. pipeline_ming_imagegen.py:449-464 calls super().forward(z_req, prompt=None, height=..., width=..., num_inference_steps=..., guidance_scale=..., prompt_embeds=..., negative_prompt_embeds=..., generator=..., output_type="pt", return_dict=True), but ZImagePipeline.forward is `def forward(self, req: DiffusionRequestBatch) -> DiffusionOutput` (pipeline_z_image.py:413, no **kwargs). Every request raises TypeError: forward() got an unexpected keyword argument 'prompt' before raw=outputs.output / return outputs can execute. Dropping the kwargs is not sufficient: ZImagePipeline.forward reads req.sampling_params.latents, .cfg_normalize, .extra_args, and .output_type, none of which exist on _ZPipelineSamplingParams (lines 62-76), and it calls self.encode_prompt -> _encode_prompt -> self.tokenizer/self.text_encoder while Ming sets both to None. Rework MingImagePipeline.forward to the single-request super().forward(z_req) contract (route cap_feats/negative_cap_feats through an overridden encode_prompt, or stop delegating to ZImagePipeline.forward) and re-run serve+curl at the merged head. (evidence: grep 'def forward' vllm_omni/diffusion/models/z_image/pipeline_z_image.py -> only line 413 `def forward(self, req: DiffusionRequestBatch) -> DiffusionOutput:`; read pipeline_ming_imagegen.py:449-475 (the diff hunk) and pipeline_z_image.py:405-720 (fields accessed from req.sampling_params and encode_prompt via self.text_encoder); gh_pr_view(#4837) shows the test was recorded at vLLM-Omni commit 5c51782c, before the merge commit 78cc4dcb.) + +`vllm_omni/diffusion/models/ming_flash_omni/pipeline_ming_imagegen.py:449` [major] — The merged head still crashes in MingImagePipeline.forward. Line 449 calls `super().forward(z_req, prompt=None, height=..., width=..., num_inference_steps=..., guidance_scale=..., prompt_embeds=..., negative_prompt_embeds=..., generator=..., output_type='pt', return_dict=True)`, but the inherited `ZImagePipeline.forward(self, req: DiffusionRequestBatch)` (vllm_omni/diffusion/models/z_image/pipeline_z_image.py:413) accepts only `req`, so this raises `TypeError: forward() got an unexpected keyword argument 'prompt'`. The orchestrator fix lets the request reach this line, so the #4839 ValueError is only moved, not resolved. Please fix this forward call (and add a regression test) or narrow the PR scope before closing #4839. (evidence: scope — description says 'Resolves #4839', but the merged head only moves the user-visible error: the orchestrator ValueError is fixed while the Ming pipeline still crashes (see review comment), so the linked issue is not actually resolved.) + +`vllm_omni/engine/orchestrator.py:?` [minor] — The diff removes the `already_submitted` guard at orchestrator.py:1290, so the initial diffusion submit path now unwraps a single-element `diffusion_prompt` list; StagePool.submit_initial and submit_update both reject list diffusion prompts, and the existing bridge test only covers the LLM path. Add a diffusion-stage fake returning `[{"prompt": "", "extra": {"thinker_hidden_states": ...}}]` and assert `submit_initial` receives the unwrapped dict, plus an `already_submitted=True` case for `submit_update`. (evidence: read tests/engine/test_orchestrator_stage_input_bridge.py and vllm_omni/engine/stage_pool.py:932-1020 (both diffusion submit paths raise ValueError on list); issue #4839 comment asks for a Hunyuan before/after test.) + +`vllm_omni/diffusion/models/ming_flash_omni/pipeline_ming_imagegen.py:449` [minor] — CI at this head: buildkite/vllm-omni-amd-ci and buildkite/vllm-omni-npu-ci are FAILURE while buildkite/vllm-omni is SUCCESS. These lanes may be pre-existing on main; lane logs are needed before attributing them to this diff. (evidence: CI — gh_ci_read(#4837): buildkite/vllm-omni-amd-ci and buildkite/vllm-omni-npu-ci FAILURE, buildkite/vllm-omni SUCCESS; this review cannot attribute the red lanes to the diff vs pre-existing on main.) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v11_deep_val/pr4837.trace.json.gz b/eval/dataset/arms/copilot_v11_deep_val/pr4837.trace.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..454deb3f82a957222d7c5650171a74196ba5781e GIT binary patch literal 20549 zcmb5VQ;;uA@GLmCZJ)7i+qP}~#9=O=ue9}DSIV$uBJK{=JT^R05>`mn(ubLVubplP52$&7 z|MU~`-8x=?$v$T|^38fFBdAdQ+b{INkb_7*7hX8`K6(Jx%dOhj<#{tBIJvv{Eao-X zUr!$KfLnWb;(VjUHNA8$;osN6TmmE9P^>VaZ?a%Ey&H048=%+ue!bnAoNPS+*l`rjf<0szbSpeXRbm^XFK=W< zAtIHV2FzGT&%%_x`0-=Ms4Hb9{!2z6-T zz7y(MH8w&%~K1jZaX^>N6CGP>+Z zNY{$%eh&x*dOx6tb#A9HqRE!7EgF9v5z%B&^p5!%C_YN645=5y_*hw;V(@xJ8^eG@q^86B6$ z9Zw$dLccRcmPqFc!e|VOtht9W1V=bgC@1bKm1yR4O6_xt#KHcZoeykg1-0#-gz?BcHEWjO`s|7=eW> ziaE=vRxDEBmX;q-PavozyO%+uBEw`KI58N#rh~^{m9g(7sWd{EK-<9{5?D^ESf%*d z_(?gQg`K*nW*RTBkpNpc8Hk2Tc>(#zh$84C_9*Yp46U2QfLFqTp_WJ9D~FdVk*FPpF}5kz zD*~}~D02lUV2eu~vX+w$=@{1}3-#n=Z1Shf*$1M*TAe0!`hikQ8Gk-$&M*%f<9;7 zm-Tsvy<$6;u5x}cb!L7)gr!k?dfWc;! z>f`tS*S59_ED+*wjfh$7^ytT`@vThyuPC|%GyY$E^=eo02FP7+)6b{>mjX?ucJjsc zq$cHLVYk`b`E|5t#-V7S`!uR$h zCQB+l-sT1CCz4rYYDGg(N6fyHE}T4F+^7kxA?$Cim}jjs?1oif=bRTJL+;6{kFA*}$a9Ez98y}?M$+5$Mz`0@3hOgeQx{?7VQLz-g@1hz>- z*9*zm3x;(-sCP_A2}H7Z|5W3Bkb%z@NPs-jNi%SLr^Qb_VY?Nc z2R*SL^Ut3xRMqD+^tU|_r|P*G358Athd<$m@Igk+~W@&18{^PP9g)| zg;$H)nMd^mF6}NJU_M+yo{K3>mMJt>cMBecc^246@F=q$DH@amizz_1RT}I;6mY_j zA5S6HUq70qS>-d&ULU=oW#w$sWp}`3a)#4$5~%&m3<37|NC=BgDfx(sqx;fe#se-0 ziG7&Z_wFSWTyh85$t+kqz!gRJ)8!r~1v%e>b3gMRmo(8@#oNjP5)VE;Y6VnU!;GJt zlPfCGy6{)9NRz|~mt^>6{c-+!vb7isU=^_pLd?QhqY49LyX3_lKeHc zT1z87tr=G@I3!>je?<;t4IaY3#_$dZ&Z3F~k z|2#b;n;g1r1H)m~=yVW^z$;yWTw)`hG>H%+EgR2+Us4CO^m)HaGgX}MANc#SDhW7g z_8X?zWPo-BYyXUhIOMvAgybXR65ce_ui>B1o)u;N<4c^)=ezuKT7;&%M+okAO$Sp@rO&se=MqU3`yhf!~U4?%_-8 zE`T8oRf9g*?vlg0RfE&K85P7x?0bkkm7$PK z)`dr7Fm!{I3)%$m%E}nu&LRA<39q=DC3}%{y6et(^VIi>eT9E@@++x!K2T|zn4m7& z3@fhWNUJ*S{WHu9_&;+Ks`|&KICzbL-=4#p8QR_{i*e)Z_g^3BB0)nCWM$+03Eu ziZX4%<;SZ*4W$od70(TqgIfs*i_n3}npc+rbr*ZIBLtEWE-6%gNMrUWh*GmNwc~;7 zU=tiY&p*xy#bQZ?x0#2qJ^qQST@qM+THyvp)PYf2vPgoMn>Ez~kwEd{%Z@O!vIG_| zvPmxWrHWJOtz?r1j)7J{Q_*~i-SIC~NB{o<>_le<$~FuhEgKsfU=+8a4pY9PJkB zz~o2m1+VHBUJ)l(YX%Nb5XBP=%$ZT+2j?nw#Q%V8>>Fw*)PQVYV~L-Yyx!a3Yh&I# zLpc^Tcgh1(nFs9p0D$qJhlTBjjQ#j$#9f$+W7y!+os>@~dVhYhe-Ih`2j`*x8ey?Y z^kLTgGZMq3;C?H(@d5}SLF}i`6~!!nrmQXwal*8lpEt*sTXeK??44I46Qd$Hh|Qco z2lHqky(k$89=Hc2a^HNQRY0mof#UK4#mpFjH#WC~_Tv^%-hw7i@2~i9LvV^&V69}X z5owFOr~q^xUDya-=L{9Wp8o{U;{-841O%inkd?fi8D3R%2vxCxSwjR#*!cOB8a)l6 zg#!7K4m5+IFnOh8syM3FnI9iZ9C1MeZfe9ZgU7J)&HORKOF3137?df#=p_tnGBg^U z5GDYVsYbdFr6-$n4VqeL<^gG3xt4CrEBIdZe-y=a4h`=!o&&}bk6c9nuPbI1^5brn znPqWG@1FD^vqOExxxeJ%#1yx&{wS#?3cbWQYUl-3fAc|FSob)(BFmc5ICA`xh~-Da zJLlFgKp6eg6D}mS|QpqXl;wyDo zOyw1KzBL1-`={G{ysi+g8Pd=g@!3lZ<2-;;+T)l5^#*5JNbfjU%pNZJ59E!;2J|l{ z52PG(KV#)P4M8HboIh}zn*XYS?Y6D2DKZ zb#N@RfP`8uNdEqk2kbSVJB6+vh;8U(30jwXBU=Fz?N=o1%B=W{52_F>uQ)=xmu>EB zXJU73Glu5jg+f6;_?FA>@!0gS>IE+2!Zu4!t53U5wHIry#^AuYZmrn=A(Eq_wWim# zl9{o@1YhBI1tkr7ir_fBfNgXM@ww~;2M>8ZjFSk;@hK(}Iv7x7(VIESX^idEd%7S{ z^>mH`n&Mjnbyj>}=`(ac2cWe8v4x6nSdGjB`uWixZim9)yCN1%vm)pkZ;Yem2z*#4 zol+z5iAg*m=XXy$Vf$|fvH_k(sBG8~l<^rylP?Px(9#-yozi#z1HGePX6R}Jg41|@ zsvy3vq?j9AxORk|_ii0JdVeI|bd>F&n2|sLUNR{Alq6Jy{;yQ+ga}_M(P|ALC&v?V z>IIj!JhDXgDc+G6$sk30RP?ia44%TfiOXw&=Br>$#IN9=Ul;EhDXNDyca3U^5eQmt zwTjR)O%2vhHRwl=Kn+n%@`UZx$;ubrux4_0=kUisp)0#swHg;H?=zyxbdae9>|*Gj zDxAC+a*c{@c$>I7uZ#0D<>55iDCB^gVGjN`zRJGA7@`+dFjK&DBLKRbu$fc?!eNc1}xuAp`Kt2tBrebEyf=uV_t#Yy7sP99nZm(I%2G% z1wi-yUs;1;2h3Sc)|GQ2Q!%@y7Q#6fMa6w>Q)xBndzzhEQ(bt=E?jM&8u1$15h z<(9xFZoz!t2>i-rs`oRQ(?#P-OB@GC1;)z)Wk>=ds0aJQlb2K$QLpfrYUTL^)CmTexr zlDNRzEk#DMHX}hb(sh+x0H*O-49-%~?svRMAxMLUq@5K_`uKOEP{7v&c3iWW#=A|~ zpqlP&$K?fMU^qO;O?kxbRhU{K+>0~7`15SEluv;SR08~gX@r%kSWOIcwnq%S6%7w_ zF=|h-!??PHm<`%>V#M?cT@q}>Pj!z=Y-g91c9mklQOIuXH zrGO@tmGq)xVi;>f975zzvngP7u~UdCP^%vA_(^#&aD_{KJXmc0d#HmMt_oKFg{ERF zV!)Dw8w4>7aUC*LmT#C5In$Apky-XR97fNGi<|5bZKn4`kMX;lyj;!vj1@Q&j<7*r z%t!Ixl4Tf8)}nDzCDTf8tCMbOe%iqgC@r@DYhvLHGRzMaM&j@DrRQ4m+xS&}9-sHm z2V3b`S~MBs3JSnXIk{iSoG%)T0-VR6<|=S+(JR3R|KDa9qAZ3(FWH?jm6}Q$laIob zwNI!0unuI6K2wEe=hY#>UvXKO+dyK^c0fw@pFeBMW5H3qAj>#wr-C-SgGTJL&Q-mpJ4n}K%dG{va6W_YK zOHTgLD%-2nLFa*N-}4jow5U0_Z2_7kT&&983?+1FycvkfD4>`>HfiaW@MnY{!<*kp zVnzwJot!)zO7?OSOT=Xq01uE5+ptoY7I4DCsQ=Dfy~rTFqxs1x8qt7Q+u3#+p+p*R zxSDnql~zHeQ?0jACmHz-&*zkaf1F{3Ju|#xHurF$g8$ybPAZ5DM(s|MoW%{hI0+QD z_Oy`2zLiphmCI=Mw?O+XPW~^f#uL2fqXy41N{=5>0q?3aOA@D z{nmeeqEE$D+E!Y(E3`r%HMDR_ZH$-tb9Z<-f{}j*+&i;uz`JvTeoBK{t!fhLS0o0% zO-J`;rr^x5Zwh!Xg>2ppu^Wldl5DbU{0-Jd@N}|6@wWfFw8p0dN3D=?=0z?wo>a8M z(wc{x%ZTdIIBx4AruL$OAJVlwJd(xR;P!R;EM1z#A8!_lUk)3y0hfsQVf)lQC&tPS}f*$f7D8*jF{d zAV~VFp;(f-tHH!q5j`wFlqVisEtx$yJHczU^E>GGjiM5m?nM$e|2%u8d)9%p46=}T z%7t0|+d7f&-dZQ`U7MY_1N7ltrZl}yO$-mp;Zuw32qiwXc}z6=EePD-o`*cLO~H1G zYK|Qzk6fG9IXh(KCA>6uD^DMkoI^Eto0W)7&%F#CnKZc~r0--AMJ5SuW=+qBXfgjzba{IzEXXvqFUs%Lk!LRxh1|{A_?>T8*RvxvW?Qc>owk4e3&%Z zaf_vS8&ZS7kZ(`iTdgJ{R6zs0rxKNvTN^SV@5*`7OtbIfb)~I*ql{+MJ!n#d^>}F! z2%M-VKUM!|Wuf~HV5^sXb2u9vT3V@ia=ncfxyYNNy`zLp%TJw&j3v*nHq z+)B4gm>nlxG{fasF-4Z*r*kI@xIsfFY0i`6jUTmALeM(bpmTIshP%}k=iRDBHdwx9 z`!o=H|7?6+F&0jc$r4j7JBOk3S~Xf|!qvFjDXzJf+aS5}ssaXoaIj_Dk9U{Yd|=5Q zea3UQ+izU9dHwR!ic-g-AK z9}{9{7(Qhx&a&pIdF@4GDACeR=bdJU?bmYWvr*+N=HGw67A+Hds z8BCB>$IYsX!(*hd#ZK>#P!>^oYsv_{A=>_Fi5TSu z0p}z^sY6Bm=Yl8O=T$T{W(QDV%^jgPV)=k7H3lTd+u8~T#U z3jQf*EmZn@su2vk{SEmg9%3!%+C{$oUvgN=#k`cDc`=LTfLqtA#~RA{(>^{WflOTE zpoKB>A1-cNc5%DH=UvvWt~0sMZ}xlh3wNpdXN48-RU}g-Rm#}~&9R~djGAqf>4<2& zK9&t6AcWTQ{T36b0|z2dqC?~l;ZK3eRgpAv6F zJfb+ITZ#j__RVteJiv-R*z11#>3BI^c{*DikgiJ_QXW!jOE8YMzktSdxut?W+l6G+ zW5P|*+56IDWudydvC@aq_HhX?=P?2azj&z^6JOpFN87-$H_J$dmSUuN>>fuP>VE~< zk`HxOCg$CXcT8>o>?BLot!*t%E07!by`QX>{Sy8fz36HXysxHzKtWzZqPSLnLdciWtuEy&^%sD*J=e<5_fVR4YC|4_mFs>`4OYttd_ zmbEuDKz&Js>uTGp1~byN$!U%*BjFUt_zi;z;``a-UtUDGak3!;|J?Wc7ze~re+qY4S0IF3lK5voyFtXBDxO-x60#b@4#{8Nu;T8wkZ`+YBWaMBlz@Dtp=d)f>8gh#sp@W!6@8xRHW_Z4?^a&q!^UTmD7 z%~~MRK(P)i#o0YVRZ>gaB&rv=fBj+>7_>gjO_YdewP=tplX-rUh9*fVk41+ohvjVj zW0Wc(EZI&kQkek2uc;F0%@KPVN+6aXKB3Z!y|0jwGgE~^XHYM*M>4C~B{hA_=@gU? zP;xcqDVo{`5R1oT20ECJnV`#IGzG??;-6C>?&0gP93QZ~vi`-*(Kkw3y*_kZv4VG1 zK54tszs%W}ZkW=bNtZlc)n`7uVh<(;nkZ_NBS~PN35K>A7GOL64eB%uILV2v5W{zC z{vGuCJR8yC{&>%QNne;V!way^yL_l3AyB^XwoIh`R3AXwiz_Cce$629)~HVs_+bz% z!BqOkpXt|cVszSLnxb-Bz4o~z<;Lv!zVlRPAkg;nc-VDUjVYk%@biIJ{yOutHmBd? z@qRiIFQDn~`*Yb58-Lix`6&h=-b-D3{+_mktCvy6MY`{__8{VOnp(RtD@Wbt@nQqh zaHdi(B3F#4*C6iwR|7Vg(ZQ#4TgS~>hk<e)rA zYNt&$JdyOu=L7h=vRJy+mCbBNUbY;=Es;i;qSGB$JW#`>u6%hL^h>l0Y#w)sMeL4m zEM7tcs}kgnG-lIB@u)$}m;ZF*97W|`&RYg^IiB^x((B?N?1H&pM~v~Gd82+@v6R!z z^D_WyIIPcpKI-Wh%s@1*Lq=uxlG4muo~6S$uMXXOIvRl!9MveIsR0J2_0{>xhd&?x z^)u#^y*-?rXH@7jy_zsz;;HxHYQCe!@28IYGLBsr5mk9iopniw_8IyfE;YI5@+q?A zSLOk7RA9npeqkXV^Ei>8K5rsm3;u(EFnVW%z zvp_MoolhC%sl9T`t+=}G_tE*`G?~4^Ki*Mbhf#ORpP11%Z@V~0wmIYWsNrPLmw*GO zUMvtO`j3b(0<0M&ANhk7{;=GGaME1M^ARHj#*0_o)8QVYV+y|6pu%Lh@X9$}thV2B zV{fem@)v zT+fG6!U{6@)DcuNR<9Gqic2LNQg--e))?QzC1!~Ilis)^$Td`D?R>`-i{xcZ*e7)o zQa%GZ0uCCD7VjrN+*p6Fj4gfTIx@fJ@feG#CWO<49HKy}z7?!NWKg;z`VuZG1&lw& zPJgw5K#NHWqYyMqxguAbF8N0T4W-RQOpPIcg zvO-wjgwo{7*`)4F9o~u9!G7N@iK+j?Qc9juV$W48CV={c2En{aI(&9Yz*fFR*iCE}WwuswA^961qIIewO9lpEB`v6|`^klc@o;zSA} zd~0$sO9cTaQ=riO9=a?80Scp1=5#OLVeC`vv}x$xr9`-ms@4#b&Yif}6jzz#=tHI| zO~)L}()LD4U$+oafz1x(E<0cM?W2Dn4f=c@v{83#Q+!5rQ ziw;$l^{9QsUyUx3q)dloobsQ~#0=(Oz~_^TXW6?No!+o*aL$VW=zcUxwJ%M8ZG_gc z6vR5q}B*@`Ny3y@+el6J#Hr6RdX1g_0#Si zUDTRf2^e%2y0wz#CN?FBmC2~dQXMw1F=jhKauM))BP&^L3HI4@TMr}u_eujq*T2E6 zn?39EiJ8D=2U)C>$X@)k6BbX?EPt zifS>cum?aJW|U?z%Ctua&z<4LN6KbMmK5`bk162{k0|wm#z5>xr#@MJvJW84=f?Wo zW7Vj?9CxFaE8u3hhQ4l z9s)3rtKGd2?K-#Fq9Ntx<(DQq=sobSilgu=rc@TAn9{}`ko)Xyyf5wCiVgNI>VAf~ zb2U15S-*F>?DqgT2H!n$3P;6B9q?{qZZjWg{ekLPtYN`7!|jx6+X#m7N3SwQlPwA7R{-vQ8b^*3d|k($A8^TQ&IDO=16xEb_C? z1-XtTsXNau7567I3NYr|s~e;!I^_-mbw~xsO_k2UA~CP>^QlrJ46}@@tYY`{HEw<~ zO%O1TyAVUWiGB4Vae-c$@IBd`PP-?SE3t!~-J(>7i}5sst47!1ip^O>v$|a+4ITyh z$-Om|Tq5)KxwI8xn%R31mIQ?IP$AG|g%ylj=~F!k`u;72V95 z+Y>IRk7YwpvdcVnRqIApBU#o$912QoR~+0Z&fB~LKYhxs5|13OCe^k?*=POfC~==n zD(qL9BfRS~pm$!n*97`MlPwmZ*M!!+-v#cLH3V5s#cHw0%1H&~H)@ULbl<&E_V$|v z>+8?@x4zY?qKcvi`LNXo!;c9=M{X=%_A|GFz@WzfE~6ImUe4mI&iQ?2ML=1cN2avQ_J^tX>u` zB5EQ_D*+RH+O1(jr;}w{wH%<}qIN>KOIZuk6{5aQtixVMBtK76{wb>I*mvy_ZcZA{OerxtJ z4{Iaclpq|e{jsCN>8IryaGAcxA6B`sPKkzt3BQGkT1GB-a-2Z0f`&T4C*I?53Iec_ zcMtVj3r$EpJr1MFSFdAS-fqJy^9NO%ZTA=8Q!X9WrEM9K8%NVh^wlACC2o?sYmsf6 z0&HHocQJT$M8yCyXXMA}NF@ z!kRETFrlOW6lmT^g^1{df8k!`D*R`y#fd32oQ|-cQ(DajJ$~h~-*+blMei-6vphhb zQ$&RLYG_=AwniHt3qQf{B9yk=hO$K>JXr14eatz^WhtE=o}}uUNkx~Bb&{D1#-JEI zS#;{4e2T@x^5~M;92O4a1 z`7ePaOsDm|#11B%)E2B-R_sW+E7Q6bh*IuOaPjur+M9BTi(zQd9mt_pRVu4bGG^K644WskVBHk;hY{^4pX_sZjYBLy-Rl%VBg$q&xd>CI#zw`Mn|W-+_Q>lt#o zNG6_uV8>C>lrG*^m4J7HbN;X8gNdnHG5X(yg))|>ZhfHwZ@cA;-JDGwR?F={+@HBSGbfFv3AiF%<=W1U3#+s<1-U~^X3%^F>=h}(fW$8ERWLHxNI&eO!(K52+V_VvzU*_#zShGAO%Euh6|TQXPK#!Bv- zH{p!_M+Y6;&wh3iqh`D%mQ7Hd5Ax;j8S^la8>dX&RT6L4@<$uFqf6?_%bvaRN6S-x z6V*6^#FK598QbUN+ZcXKyWriAy%f?R&Mlqe7{TrV0MVXIR6!~`Kh_tc%zIMv;rYo6 zX8-1vw872}XTb>>hWA0Ckku%FoyDZmV6Y(nMIsEPwxk~`JVw} zkF*s~f1h+Axj%r1EX&y{a%&yGZZ&VvSqhvf)mGGem1Lf`TjGp+o^p2;S7smUFO5Ep zWE5y9;FQRusaU(BmUN=?<~PGTHTh&aq_N_u_uoII-Y%IW(HsWk@0zo~nE4(eR3Uyn)Mpwpnq`1bA_3MLmh*?T{Uy@z z{ltl8kJ0Nf(BnlG%_9)fH2CT|&*!;I-p*m_(_XdXL-8ZrF#j@Geu{P6Z;j{{c{H7- zuO7)JZ|cf2IxMfZe^cyBYRmE+{r&?MfWn#c~E^Vitc2e=`zmCw<7)${MMDP>X@Kh z=IM6{x9Jsb)y~^#od4Ux=+uG~gScDYMDOs;60;BAS`r2lnHNz76t&FPCfI3=|Ko(+ z_?vY1L}P3-8S{@Dvr(D~tt$nh%d38_o1l7}kbzKrDBG5L7P z8un=?tWz(?2-9MH1ltYw(ybxvKjMElJY&r8b-RkK{{XJ|Kg$1tUoyOFmz;GM%(@=< zbuEaS25?tZz>Zq~Ew!OLdj0<>|2O_uWdFZ#|6ld6LGNL`*2A_<%1(o`b*rS624M?5 z{D!^qe_&nhmc9J{fMAEbCEh%J6Qr%^h8XB6Fb@&^m~N`f4ZBw_Y|iv8fmt#G!-$DG z+fT;TcQd}}2Ni=)AfB8SAj}=oBB3nl0@a3JRgPV_CbMkI+jFTDc36EmzVK28#v*>A z24D(vcM?Hq86Zoo4FudEB9kljFI-Qq7+u`6k}eZTlQB^0L> z3)d4$)lu|2sIW^4)>kc3S8z|ti;g>>!bS>9X+?-}y>@_Bfk3NVO1>{>-PRy0VT6Pk zlIjKn>z}yTE091|i9I%k{;jnsS-{4ap_OA`8waw=s1P_eX*#uK0Hd+LzPb1NHY@5Y z11oQHTpmib$kHYlQ3@40bfQf=0H^Q! z2etlW_BFX3{t`$0_QVrj9K(}9d@URyq|5h<#zU_iRa2ZWrWz{z2)=8N2Si}1@s0>Q@O?Ek8{Qw%2FqAd4BVLx4(4-n zH`evB2#$f$|NOx@^FY6eyUY)PYQLWvi}#WS8h-l*01NM*RPAPC?g_N*c62yd(s*;pyO{y(MJt3kk>7l7>3m`i@nGfP_O9t9cn= zIh#iVg`p*B>;K6x#80mRmV?~IIO+T;##mdgVt_=|4kn2%OJ@=o^ZtIi2)+D0C|{d1 zc(H4QV;~W{o8y~5E`9*)GK|jAps`Ll#(KwspPeDcY37OzilKS&pH<$$p^~FqP|h7V zr9eDfO7rR56Tx{DXmkQ4LbiqC!(1bSPwAV|jSG`v%!fvD;3ZY`11?!S!n8IkK~S9n znk%;)s<3wps$#v!J0SwVtHLv6QJ;Y^b4<_#h6+eLN;8lhxnQ5!kdoTwlv- ztHFSKayO7Ax6f2#id?WYANF|%%OtHwp%@af#|5r8e5 z^A7Y`gRU_S%RqmB2r`m54a7<2!dAVwpmk#xQ$*AZ3i($kjlAWa3wJ$%} z;vs=%-JMp|RUuHxp8S)=&bV*c+M_@iT#n_-xl{a@O%4qyfBvXAWydB(DHz8J(%ygy zUfZU8XIEd2)AQ}O!Qx1GtbsXn;Q{0b$qs&mb%5nHkZ8VVh79S#HE|e-EDHsUB=>5^ZLBEIZBu^k|bJ0 z*toz8qTnXjXsFc^>Tqs`!s)GqGIL56LR}_nwe}oVpZ2QxT0Rj1h9uXADF@V}RhMA( zydb=fYHml0pGe+|CD&h`89zJUK?E%;$~0acJC3?66b4Oh!>Tiu!0Qz7%!_Yo3U1yiUB^~B5ELB?r zfk`@=L5+*TcT??6AjKK@fWy-cO!->h)rS6fNdtx3pa(?^VFqIyN{nCOpWZMsj79TW z(i{aDO7vqDov5pg>7&50kt5CM9;$K&3Sa1iJ#ygLC)9+Q|)oiv&f z0SA%Fc5aP9!h!N@*+FX4{f}LW9w@};wC91Y$Da9L*0Wn!;#^2gHr1_Q_o`Gd`E&0tRBb5^k&xOMU#c$!DF){7 z4MW?Gyo7u&{vOTPl-{pl#(i+!6huXgnz1=qNgeKkA?ZJRyjFXsz|tzOOi;w9+;029 z<}QRuZu32@fU!cW)jIo2_V4KOjVRcT-6Y=wz5twZ9kHU!F`~Uw>)@u=GXSu3kFg+-BZ7hloLpH3xqENA7OKy{mn(Q-J$B)(J?i*B^*w7+k8k~ zkBk#M4h6y-It;J4X97l)k6e5MhY`FS6W|_4v$r>ECGA9NU7dDd!v(la?6a>0dd7MYVv*C3c+b#U!p8)@Q*RiYuL68v}%T9F&koQokt$jkf(YenRd7a4sy4pYrQcD1vqS{*p0*t1`@Hf^4)l`OAntO zzIQMq4==p4wf+;`|B@4K_h(H9UR};~O=n587Hgyi;v7GA*Jkg(x^-H)Jz=eXHq_RT z-xFX$tIABeKFOc@2FD&yE<$$U+ACe1cg9WAterEzdmga+9G{kQAIEKDZgHZK1-OAM zbir3+d!Kc$9}=eYnV1^AWaWH>r)xS_pVi22TT)_Fy-k;6D1+*7<;M@Ge_(9BuGzb0 znxv zjIWXn_Zu0G@8lxz!4%(&J4%F!hWW@b!-@5`TS)*Y{u}w-oI6T{gX2fFoGWf3te5hC z#&}VK=L9na#l2aySsu9jRAh%-m`ySw4ID}0LvFW#3LM8JlU8F*NM-wI*JLLTsU1reptT>GFNTx1I)%rZdCa0+2S?!9HPHWJx0g%{ zH9}#)P`3th+jrc?;CzpNYO8%jB58q3um9p}0iO8Q6TyER$z7d@pjt($(WvyvP7$5i9+nEks#R^fR-PY@uY>q+e~A$y zwwf)tnsZl0C%XFlZE_N9lg@>@CUP&-ms|tM_dT(RShe9upO)zg2PG?tKpTyACwgOv ziFfb*_7ZIhrCxIJljYPQS$R!Dcv?E*BuD`+tY_I1R7>s_du0DZ1~iZu=Fq~1aVO=X z=LBWEVaI&HtoimM1n0K{C$f@SykGV`0tH26ROi&l(A1xJ2mJE}1F8+EB)bTlmO zAvJ?`lWi8QajGU{t=j&6WopI7*H;x+f5BnkQZSX!`xkuyoT! zXsL67m@%~1shyIl`S4IgF&QfG!*pbzjzS(`(3!mdOZ1bc(4vn2a z+vA9=8DzK{_YA+N7DzHqRnv*a>Di<(M*SmK{a?1|N8OXO)@aoDJwT!zVHw38+nJSa zLNiRyz4h7P|Lo!!>4f{JL^Y4S4LJv|1n%ts5OOa#g%e{ws_;s*Zw-NOMvs@TllZ#b zbxbMToDw*E;u8s!K&78NE?fzXp$k<3zDRmWqg1b; zJR+~6Pw+F6y#tq}Orc*a*6lA1Mbu=lh$jux34>yAB(|hYI4iR1odc6IH_g zHElG?KFGcXtP|)p5Ew7;El)q{9i3>;OQ~9lmFCd}+V5%>{IfBLpwM#FQ2A<;Q?q2s z3ml-PbcklvTN6HZt9Uaqz6?(&9d({?IJ85_x2aODS6#0!XcQa>Jw$R21O?iqk~x$R zJaU#f`~M264^{B4X-P^9qMyXoG#)&oqK9)8zpp-ci|Xp?me;oiMD{AX!Qo@{8*YDeoBL!b`{k|uK>!bPT#;xlI zu)1#Ov#qQ@j$n-g(hR8bN_Pp(UVV>Th4Q+T-C(id_~ zEP2uvH7tdm$L(MRORi@#SNa~?5KF*8V6-I?K;9}zga@WF2(RE?uq2$o4LYzSm|w>d zzZ;ggFV@A8+rSa{yuubMIPx4j3}GR1Lmb84FmTdg8l*$kh9jf_Y}pe$G-(F{I@~Ib zq;#EikrXzN#DG~uiz@PjEwNx>Q!KeG^pyj8&JWr-Ndl*GpXBL1a1sbSrvpuX7;F_w zcsl%=Sd!fK>saDGcWt(UB`L52s;2c#u;hfpNVqKV8FSmP3m%MZR z?P7^Z$M@D{CSf~ut>iG)BJmJQ@RN&;v6O_d6S_VCwG~UQ?Snie4R8;Lk^s1+{GP;}Ibc#afppys@#KR{8zxdE$x$CcwraC);G3Gc8BM;HUz#0?*m|+{DX_&@|ii2UOl+%VLWLFV5WD)=V zc*3)VyUYjh=eLR`;fCJ&rg)v17}(pa5qdCK0(Uo*o9dX9DUpIy)ohB!_0uaYmu!ODL1pt3jc_&?&L?UwBD2Y0-6#765{GQ|; z%%RI&qUJ=~wDSDmuJIvo{)2`kPDqlrCnVoEJ`^c`$ebbbxfiz~30XDtUC+HAB!MRG zQc8idT^zA>4F@h84z0j5l*3gNg*G^Sq1;s3c_8Wlnj&>0w+%%|0-mcK^4}jtz`=f( zM)J2Db`bvBaaOSIWFt5P$Y{Zm3yKP`M9%ppSPF12tt%84OtxYPK8oiU-wz}CuHPAU z5PZAR&gXLNK&#V`cHj&-Eh@?LY{q>iH${@;hRWed;CYeXh9n>Y8Td|kPr{CMI<$Lle2<1`&6PUN&24k%Dvo%g*5>yC*|e-?b~ zaLZu_zII~CYoutwoAtvMEP=ZznG^C&c_~d(<+4aJmbPLEsQ^Ib-%k(^07ph#E1Fz? z%h4gnTR(c*CXV5{o ze5*+Eco43ymRMaa1)Q}Q;Q;^a*yeLsZ2tG{WBxpmD57ZJcH@v5klu&lY9Jb9! zYgeRTOX@l!2|QnS6-7SqQK&-#n;GFkgi#oSr_~yAKoWr2-wzuFB5=EWSx;^`LFwq@}4Yk8FblpEOfSUtk^A|{vf-C7+CSBX} zF!i$`mclRuXeXQr7PMjs<`GJd-4`p#YbsKl?FR5X$6cG3+-5zB&v+Y_LX6xv;igzh zcq9}E6%p{Zw34O*>4o(n@6)3xpl;(AzkK7o~BW1a1O=Z54P( z1>jKM8mADtA)?%J-v<=d{^4?Mp9 z*1twTT!4MxuSOoY4{{LUv#E*4Jut*Q?)pKy@ql5%fU$cMd4by<5)Yj%D<;QX-*~9U zhcI^#tfpys5ZE3I-EdPIk2}m8rfle>O1AP7u&V^f-xHb~-{aRfwOjU=HmJjE8W0`} zSPMG|uryqshwi4)R-XD`<~c*bTN6ykuJC4fUm+sKe78H=x?Kw*Sl51VgJ#{aD_XXd z34z)4&~r8ow#G@Ecu5owRhZ{9xlB`@er6hK3g>* zV5-;-TF6bS9%AsA;)weR6Kz>Q5dYNBCcLMhI0*C3J{*iWTdwh9>+aGN&Egb+dmi{I zmOMLoW(;Kg=rf%Ar%Ow12y*|@q$iM&*lDhqvFEz*A;rwwZnS2)dNM| zqM1S;lM8{fY4|}(KaIm!1(9mgOe9rd{`bXA4A<6WK?GYa9IyzGT&ZhF2Wn*8-sOidfl^Mt$VFu7Rs?>YX) z>5J5hd62Rs;Hni#NCe=JiTjG>fnMsYB9d~;{Utqb{c3)%;nzy;tg0msXx(vS=x-W- zP_YmqN+e*a6-`J6@Y_A@s0A?}c5l~|A>VQv(6!Oj+!!OcaNSjr=efS^I?l$m56l@V zp*)dFziopVQGw?L^1dYAHEnHp$o=iw54x^-EmPcWmLa>o-_ni43%Jc4fAj5{9Ngu^ zA11!jnhYc|5b(!6nc@!b%&G^?V)5H!$!Uubd6ad`remf1hvPXO+1{FXle`h#!+q(X zI99|Us6kt}!tvaV_t|>Pb%hL=*S61=2!nwid_VR%?`+^I!3NqU-r!5&-SOsyO}sf0 z3sP4Y4!~`D@cYI)oq`1Hb)JM_Fl?)1A-Vt<$?&0d-u1j~*X@aQl~>?2yeb)ZEha|5 zIi`s=uYpZ{+%Kp=usUtNFVX(*$*kSwa89~c)kEI@Hb47qD33RfCX2G}^?ZN(7wmD|`*4?M+r1dKB5!cpit zp?Gj7DoEMDk$YDh$>6Se4d$>=uI4qs$-+8sXJh*>WTEdzsq{eLwIT^V>W9uf_1q2o z?y5@eY?YBX`q%6CMQ*g*JfK9#Z+Nk9aF8{X5-8Nt2o21UZ% zuE3Lg{hfl1;uRM17M(<=ymMH~rs6JQei#Q)?6wvd5u?g5yr(Yk*K~qlZu3NiyW+?X zUB13n3L0Hi(7A0vlV|&(xA{RoGK~U{r#x^{r%1D8YXU5)>LehmK?G@y%yH&dZ40?CnV1wTX zHn@0%{AvtVpLXtqY-6)Q4AW$o4wEoct!yN1&vChPKiR>6g`K5E&X(CoxUFrzu1Aw> z*x)_`mvXhDQlE70=pygUipU*5ih^OR+R;QxFVwtx523t(clLrXndPl&rFA2&L9@9y z=Wa^}czyD@qXTOjVJYyL7YtJsv$PdUo(&Yu*gbhl-F+ZTW7O>-iTm!_%p{w&MT~)V zZz+kOlayt!c^Wqua^bto1Fni%@dSELf;8Apez5%@%g*z|2etET8*Yi9UNcTBL;{nr zTH{2^6&S6amgq3w|1IHg-Qm@P+KIc3BF.txt, and requirements/ contains no vllm constraint. The patch newly depends on vLLM-provided `init_model_parallel_group(..., use_device_communicator=True)` (parallel_state.py) and `_vllm_fc.DPMetadata` (hunyuan_fused_moe.py), so the 0.24 compatibility assumption is only enforced by external pins (e.g. Docker). Consider an explicit vllm>=0.24 floor/pin or documented check so older vLLM installations fail fast. (evidence: [sweep] Dependency floor: pyproject declares dependencies dynamically and no vllm pin exists in requirements/; vllm version comes from external Docker pins (docker/Dockerfile.xpu v0.24.0). The new `use_device_communicator=True` kwarg and `_vllm_fc.DPMetadata` are 0.24 APIs, so an older vLLM floor is assumed rather than enforced in-repo — recorded as an assumption, not a posted comment.) + +`vllm_omni/diffusion/models/hunyuan_image3/hunyuan_fused_moe.py:38` [minor] — In `_set_forward_context_dp_metadata`, the line-38 early return for `dp_group is None` / `world_size <= 1` is reachable but untested: `TestSetForwardContextDPMetadata` only covers `world_size == 2` and the existing-metadata path. Please add a unit test for the None/single-rank return, or state explicitly that it is intentionally out of scope. (evidence: [sweep] hunyuan_fused_moe.py:31/34/38/47 new branches — read file; unavailable-context and existing-metadata early returns are correct, existing-metadata is unit-tested; single-rank/None-DP return is reachable but untested; None-gather failure path raises RuntimeError.) + +`vllm_omni/diffusion/models/hunyuan_image3/hunyuan_fused_moe.py:91` [minor] — Verified: `_set_forward_context_dp_metadata` is called unconditionally at hunyuan_fused_moe.py:91 from `_num_tokens_pre_hook`. In `initialize_model_parallel`, `vllm_parallel_state._DP = _DP` is set to the diffusion DP coordinator before the `if use_moe_parallel_mapping:` block and is only replaced with a vLLM GroupCoordinator inside that block. The helper guards only on `dp_group.world_size > 1` (hunyuan_fused_moe.py:38-46) and uses `dp_group.cpu_group`, which the diffusion GroupCoordinator also exposes. Therefore in a non-EP (`enable_expert_parallel=False`) DP>1 run, every FusedMoE forward performs a blocking CPU `all_gather_object` over the diffusion DP group and writes `ForwardContext.dp_metadata` that the non-EP path does not consume. Gate the call on the MoE-EP mapping (e.g. only when `enable_expert_parallel` is true, or `_vllm_ps._DP is not parallel_state._DP`) and add a non-EP DP>1 test. (evidence: Read vllm_omni/diffusion/distributed/parallel_state.py:800-816 (non-EP sets vllm_parallel_state._DP = _DP) and hunyuan_fused_moe.py:38-50 (guard checks only world_size and uses dp_group.cpu_group).) + +`vllm_omni/diffusion/distributed/parallel_state.py:526` [minor] — Verified: `init_vllm_model_parallel_group` (parallel_state.py:515-527) forwards `use_device_communicator=True` to vLLM and is used for `_PCP`/`_TP`/`_DP`/`_EP` whenever `use_moe_parallel_mapping` is true in `initialize_model_parallel`. `DiffusionWorker.init_device` calls `initialize_model_parallel` at diffusion_worker.py:306 without platform gating, so MoE-EP runs on AMD/NPU exercise this path. CI shows buildkite/vllm-omni-amd-ci and buildkite/vllm-omni-npu-ci FAILURE while buildkite/vllm-omni is SUCCESS. Please confirm whether the AMD/NPU failures are pre-existing on main or caused by this new coordinator path, and fix or explicitly acknowledge before merge. (evidence: gh_ci_read(4893) shows amd-ci and npu-ci FAILURE and vllm-omni SUCCESS; parallel_state.py:515-527 introduces the new call; diffusion_worker.py:306 is the shared call site.) + +`tests/diffusion/models/hunyuan_image3/test_hunyuan_fused_moe.py:?` [minor] — The added `TestSetForwardContextDPMetadata` tests call `_set_forward_context_dp_metadata(1024)` directly against a MagicMock `_DP` (test_hunyuan_fused_moe.py:74 and :89), so they verify only the helper body, not the production pre-hook wiring registered in `HunyuanFusedMoEDefault.__new__` (hunyuan_fused_moe.py:87-91). If `__new__` stopped registering `_num_tokens_pre_hook` or stopped invoking `_set_forward_context_dp_metadata`, these tests would still pass. Consider an integration-style test that constructs `HunyuanFusedMoEDefault` with a mocked upstream `FusedMoE`, fires the registered pre-hook, and asserts the metadata helper is invoked with `hidden_states.shape[0]`. (evidence: Read tests/diffusion/models/hunyuan_image3/test_hunyuan_fused_moe.py:53-93 — no test exercises HunyuanFusedMoEDefault.__new__'s hook registration; the helper is called directly at line 74.) + +**Verdict:** COMMENT diff --git a/eval/dataset/arms/copilot_v11_deep_val/pr4893.trace.json.gz b/eval/dataset/arms/copilot_v11_deep_val/pr4893.trace.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..ef4fe88f962a046fa7655f26432ff033a72b5f8b GIT binary patch literal 27194 zcmagFQ;;r9ur1iOZQI?uZQJ=es;_Oz}~#Pe0XE7@TgsSn&tk zy#$bXvk*;BB58iZWegP1yN!QGQNOrj%@)MC#?Fh=hyE$CGB`LbVf#1+EAERforC3l z0@>fnF+A8P=x5JBCj0nU5`2EW@BG{BI+|d`RX7dykVVufevBx^nPIpP@HenYC_Z<& zBgQ%M9is9Rm>fM~t&pRfIg&~`F@G~ii*L$iAJf4{+wd>U$!AD1YQ4J$ZW zL%l;Vg3 zL4cUO>>9@6gue%WV9W~VGut!&n&U7$EE$7mk5+6r9yT}vuGZuPs^0i)w!&4!`V{Vn zM!p>?7o9YmI5Zru!5w4ZRuBWqR6$?^co{y@s%*p3=T%7y1h5rP#PLU|8C0O^Mla>p#& zam{M5wY;k{!;5?XV1XyheNu8>aCRsQOlH%Qq41^L!%3Y0q8`7%`^ky*CyAxJ zB_~{nOS*#u@qWyl1iyh2_tQI4xzlmuu)_De%yz*P{%PpYGI}p=_|LeZ<$i~}v$Rk< z!#kh%{Zn|JzvuJE$bD5RmlVHjc)(l%CJZy&kzfo;wj@a~D|pyJoVedbF%W=}U?kT7 z#P1+Cfq)4zKd@)+R&qJ^kCC+&Qb0nNA`9FJ67wJlzXGFT6w`-qniVhrwNDBo=Y(h~ zfaU(czFM7#wq|ps?S1Pj%f4EPJAUr@Uz0^{?~M0q+umVWAd2mtpSOwc@6$*;quqk(Jv7!SKT*})f&rL>hBgRfiFUOa&d8==>Jche9_toF8*WZ?;$W(>X zRk~}jrY12fU*7oB)q7Z4oFT6y8JV754;wj3q7u3{N59-fJX=2f&}OUA5#CHjm~5Ne zQYmJOuh#bV2(!O$M+)axbaEEO?-JR9yWW0Ad_3>C!LM#b`l$C4PLYU=f@(t?iD;4%ZL{+}3?>v_Bn3e-Gf&pMX|MCLbIzdn z4Kl{O$tWjt!!&T+NlQ2quS>#n)$6;ZyxeDh{`Zw2;?tsut3pW90QLmgZ3OTe=RiRR4m*Av!?AM%&Up_Vhsc<6T?$Vbu!I*J75 ze^2155j90gK`UnV!Ldb-~O4r@aJr3=FU8C$88FxHzz95@qP}!{cD2#HGYDRXs0UU$GD}^bpD2yfExH-vD8X%t z8RnQvC7ptb_BYoXMlC(X7AfMj;R^D|6LQ>=7--bLy>0u!XJfHDux9CK+zNnDquCR% z+4()yDySPy`+gbZ(37b8RUeeNX2t>}ey|(hIDl^X`8b=SBGu%hcYU0=1j2P8p1k33 zO@Nmo#OtXy;2c=+%R(5DEfz6VLK?p@*5^r^Xd~Li1elQm6oA@o+JVNO@7sWh^kHzA zD_8P+;vkRCoEp=Qb{+pxz!B*HX`uxm<@+4QBp~irEBtkb=O6ut96>DxOE|ZLXYk4o zPJ{_w-O9CWELenn7UtZ}_Q5ofG)S}fo*m&*OKB;WFeqRqW+hNrQ<+U~ODfNP2a-uo z!qVTXg-A7`g5_jd8)BEgjM<`jTgLU%)lFHS`D4P$?P0?wa84XMhziw>DDd-^?CRuW zggkLl-xE*Y;@f-;g^E>LFAwRFyxTrdm8=5tIyPaGod7>isBj3?zG2gtuGc23uT@5^ zfk}HQH2H4%u#GcY$jjfhI|LM}Hi%Cl>Y-g^vhA0z9hwmWbAML7$$?{_h`=OS8XS+k ze*mIrW_9h)JGW1N0;6OBmsZN-As(&QL|a|^SC)P~ z_f-rtJ6FxXs}+P|%2u{3bq{J4(@r1Eca3tiG9S3Tk@Y}~z!#`AY#jm)_jt05tB3wO zWRO(gvS24e^l@CeXalR{(ua8p{*kLGHFB7I<*7N)sf^=pp zL<1VHM_BR=Gs4B=@y<-lR3c&FN~P%MQLf|s(1FH%6^HR|k()txMU-@jsn)Yzkxd<8 z<&Rn))YW9ld`#-)*s^$pl`+qEYHER!C5c;B{hFF)P*AVMf)a#)EOdEeLB#9jq5wK)JXm@khtM9iuXhQWB0S$61dzCT$$KQqPJiFuvK# z4`R1uCRf)24)QR|5egVhd4Go?TWCLsiT_tMyE=o=se`C^ff>=L`RGNapC2yB={p9mEQImNp=3_k z0E^c+!V;lbaZ_2?_#3bGRkiiyLu;}^Dkr07Q(##1Z=5M(ud*%ZctHrt2@{Gh5|(

8kF`adOY^I!y|a4HIW!8;|>QOE_lgaw&-vCEX1oF-+O zLxGe!e1Qz;Y7iP43qH#ki(+gB-#UV+D#BW;7S%%x9?YkbsGbXXb<+`+{MmM3L2rb8 zGJVDbCxVxS_mh6)1g`?NNurWs(ZQaMH512EtvH0rl3^prL^^{ELcX-aJ=`xx1frS}eo{Ibx^DQY2)Im8U~dp;38D69>X!Wa7)K^^f9HTR%Km1js^vC(j-A6?EF zz9p~sGLO-4HU^tbFuQBh-hxO4DXx(gxBnpI{5#cvp#tWf!}OX*Tr%qrE(EFG?*(^D z!ir z%$}JJFsnzW-c&9#FENg>O!mMvJl+=ug7;hox)CN_@ zy3f%KhLCT9g9S1ND$h66epi5k>;TyVws8Wv1$#@EFE%kvX4SvFWfbiA=^i$Pl4byQ z<-haK9bAX+2HdzrsEd0Mvuo`EHc;_*&m)|qD$f{eou|f`*-yyCxpYLT{@!4tBVS(& ziO&|zSn(~-8UN1V7-ukuhM$;5$iijpUr&?4J!@y1njUxQXm^7@lIUWJ*}?|%Z1sD+ zyIwmQ`Zp~X+V*;*ioVXa?DtOlw8!oyn%|5&nAj5PXNbcT?Q&t!=Xc z&_24zr_X2o9D$Bma(#*^Yt-`{=MUy?#}B3*lek|%Sc<%J3p8%j%Ov|i|5^}U-=2Fu z5rfiPGe&;XiVFaK3h^{`*2e9DH-8QBIGfS=tlfnheT0>_QW*aQ6!%s#z6Bg-n)(q# zlC4!g^b#cKOxgAJf}(6D)gAhnSQhwy-H&hSjHskBSN2`ILP{^pZW0BSZTYeQD8?GsqbQR*Pv)Ot45 z>gg;gQBrO6V!62Z!mP2)VnN%_c#^vST<`=?j|doNIDENbfDuRTADN2K5oKQ!e(K#8dKVF zh$h@-ZZQ$ng=I}&ph*61VJwk@Wv==4`3@{GJ-|K4`cBB-l-ojyPE{UqaK&L>!S>ej z^$4=I?&;VleYrt@hA$#V?M(`BKx6K!vZ3#N0!wdT+;BO{p3(RUE!I&pjxoN=vx_%t z#lul}P4^La*cU#&#wLui=VijFwT8|AJ-RNyUy+~^y5`0$Bo?pc9i_F^8d{#l<51Q2 z6k6T|)r4bmr~BDXw+qK}Y6-_2DI1rcOQrs9)iaa&BOf34$QlgI|HqTo*oWeV_L^~+ z0=fE0fW<7I>8t@q>B7sVo)D@SA9s9U>$AGdA`E71;1hG?$yc1CN<<%e9G|+Q={POd z0%=bYK>29~lD1%)RCbdh!=D~P_yJih0H{E%fZHFB{@C<&rUQ^ zN2}N+D8WrzR$iseSHEbQiEhxv+Dwox>I7@m{9}w=pRaGket|KIaNJKpLz+md8K4E= zZ{&K*I5*m}*8a@MFf~cL$7=YwQ?%EdBfxnRVZOi94~cG}2fwZohrATy**vF(OyVfA zy?j^BMOk^Xs5JGO1VF~8tNvx;-5>a)nB|FeH=v?8AErJ_feXi1#c#804%~;kz6KRY zu^=cbRRZ6I+5F!@RA+d>_S*ep6-XCio!RPe0a_y)c8XcrIRC;}fv@`7aKe!$577$g zAG^y0vOP6cu6_Ir6mnQB3D~Et>X580jLIs(J#ogB`C_JrZvYBF)jz*__XsoOm8tIL zkIT$tm%@UATnzUO^g4`fTkb_pV@=Y2dxRj4IVyb<~*r&A9?q= zBqu?&cO_Ojmr~_F5m6m)MoLz1WOya5edB zYk~EUTtd7~Rh40cd6ik%r-&B4mhA0wsikI~$>wKau)1H=Y~vdx(#|miFFB(Rd!77m zc*blsbu2Mhh_qFKU?3@69;a=5LIWgd`ji1sF_$_v8wC^u^Q}t)BLl?oJ2gZ=GU)QB z;IB|(iM_;lg?60^8DjHi@M;YjTE9KF1tZJ|ih-uI<~-zeC`uF!KX97c2dGokk#sEg zioA$(gd^LL=#;rpoS%ZsfRomqFH8hA8$9_)H0)y9)?plX>vJ>h!`bpSltVYmouq99 zv)5heV&~eQ8JJkUNUsgcE>nljTsw!M{zg)2uL2@?&b;!PO0?c8B}B5NDB}Z1e6rNw zTC!NEBACeF-#0s+SC6u!9qJk1+qO1u2azP;((WA|FHrqIZ);pW zU1dg;`%_yzCd|HFkbfyi-)Q!OvC(^zWDaT!30J`Dx6U`z7sNJmav<+=lpZwCeOrc) zJ}^tH5-DorP7>qI(X2}uXwL*R&!a-l3$S;L~vgdQwx zs=<0!)o0sP0Ln02LYod;w9CR4M;zXDlgf`Q@NGsg-_@+P3X*`;JND&M2W${-Nldd6 z;p{vPB+td@6}1a`CSiPa2p|%+E!Iod-k1=0V#E^)b#I_q)W^=Dd`)G{zAmbHe6XY| z?73ZXbDmNV4e;j>3UqTw3wVS-iK1Gvk#xfeHous9PcuadHP%QI%di{7PkiQgN8D6S z@{{yipNBdT!aMs820qaurs3y45K%wYC2vZhs!6RUboan-G950$oQ~)u4ixUw4{aX) z%s{7VQxD#&<$90IZwmw!)L182FL#qCcZUj-n_JjY+w_w@3wNDTCxF(w-(~*ziTeu( zC^?YFel+u#2$2^1iimWlG4+Nq+g-0kQCh_J+h(4K-d0gRsIQ;)cfb);slVVr;?bIu zN>-j-<}>L$(WgLgUG*DU9)W^E9UaseiCFg$NOl5xfOMKSrDx8>Hz-|gxbe*$#WS71 z`15?g9{86iEsB(+PrxSW7Y#S$9t4>C7}U4~8Fv%o?8LSIbUXw|579@NCl{7`%F5bC ztO|O?s>G{}z--bzc(F>;EqexZ>kIb#%% zX}3JreA@+2xyxktux_x@5Op_Zhb133a$A;jL8P%+8939(e{*vHpQ(|iZ(+y>XU+vo zuRdo(glF2hOpYh5?Kf!gQFpP84V(}jvzK?kxqyk2PIiY|xqUc%gIz!PRe->*>!o(r zfm5DA1*@5-h&Ok-?#0XbiBV6ZzfvW{>=AobN&;n0)MCukV#xJi%A>j#WkxW3=N;Q*bB6f~-HySkZK&)`0IPCKL5ZKWINTskp!rEa zHq29qdbg5i6pDIVDYs)eom?BtI>O-im*?cw8{*gl%^(=yT&bwq+YU^a;t}qmp z_eNHpMXrEA){Myf+W-$kTBx$*{18>!o*%I0IXmK0>FT#pNF$eCx$%0+fe?8z>Yd@Q zQc~%k119I}-(u=9&1Xu0)lP*`Q=vPkl%b>PUk0ekopQv!nKfEq(3esGFme|ygF#~k zi|0+!kstcje;Tu9EDIUt3} zqoXZl!dz}Jru*`U?^R4Z7JDAd9{$HMOR^p%E0;NeI7rX-30wl*14H)CMgRa)lqSm) z-bAtOm>r1`qqJ`tP@g03ns3&tR}b#YgYD0b``(U2lFIUV2bSIhfxDmK4F3+~i=ZMU`MTZyhl4C4Z-lAVaL zSuCNYk<9I&@L-Jsps$%1fGFl;WQcR^ck-f~*Vv64P>Kk&d?AHV_#!5M8+uB={e!=__++gC1qt4Lg}~ayXW(1m zxBmL$RRx0SS9SV{Emw7_Ya7*CFm&RbW$|T#rwf-XNhy!7wdYqge><*I!KYhRRyFXg z%hTSs6d{-onDuKas$)qp6EGf}WQ|(lHalnqGAh^3V{NT81Udjv%Qdr5v1JXR}@B3wDV^RnH`t4nT=imm^b$CU? zMR0g1Q7Uolt$J=@Sj+79k>_jdAs}@j&p|>gD1qQ1F)EKoN?oDyk*NZZ$Y?V3J%~sS28I0q;Yq(H|A!^t*%Ecu)L+9;n87T>6}PjHA|zF$O7r*EMjSJ3>2t2HVp(Wx9#gz55oD!j|Mf!AL*I z>CJx*`uIxPeEC%*u<+YM4$c@Kvv z@U@QIHeao7b3>y;2{UK^O}x&)0ue8gJlRsv7)&a}Lo2EFPm3OcO5;jw8ZmOy!xCVN z-c}G!uL=kn9p~Tv{+lijxIV45>_Qyln`?p}MEly2aDy|Lh>r({XsddZR)Fh>>`5&d zZAxo}I0=O|PE9jl+Srop#E3X7kOFVUH%(oKi=tdwH&xE~Z&Ne6;1z}76}rYgu`GQ; zccwu{!TlHNe3@5=ZY~Um4(n;JxLgyZtCALu-Ka^IQ}@Mem|0Mi)E1xG_fZ&Vg<(Q` z^?-E9TJM7!!J$a_W=6x+%tE)wV?p6_i%fjE(jH!qNUONWkp8MWhIdp< z#QEAv)&vv6Hs-qGia(>f`huTl>V#B{MU(vEpFlPr-v9(%X8|lYyjJEZR_Re@Z&#}F;}3f zng-oj|1!D^=~>2+uJpGCb1ha}_(F=ysd-@1Uh1&??kLKZF5`TF z?N`jgkI7lP5>6*6#s>RKyG+Qfi~^mdzjjRBQx81>8GO8>^8G&HY)o8RT85gX%?asR z=Zm%818HTC48k*qCcnSB*9R0cY_Wjl*x_|6n+mj)5uEZ3Vs9PRPutTd$ykV|-Mz7yr-<`uYqv-VI5AYS zuf0ORd*pk8!^bQ&ifW0;wB4;6bx;NL91v~_Fa@5>D88#x#u91ANKVPkRE{Q?Mp4<& zi$|BJpa%d_xrmcY0z1&{5TT|H671UYm?*y=}K7%WTap z$ zbOg+|)AQHglw2(7giwK^vjG!l4#3zKK9wU4>lejxLbtdh!{PY4nx~kNH%ZB5bi$f*Ex> zew*ldmC298Q?re{dxADt{pQ=jHj-JjB3?)fCb#pUumZ^T2{uqZ?q>ngU$1CKySLCnI$u}zvO}lE0!KYj6g_f^wbeUZM=7Q zAq=hUBbr+7%=ZN;c9@Cz<=G@ncFj8{)_m}}P0$Kj{U=gB?MW5K7sMKOd98>2HQ1Tx z%GnBuG9gd}%W%mp%!ny@k!xq?#6)eksQ8Bpus`t)KT_lV6eXLuSQJ6A+8|#+!_~a_P)Bt-r z<_mWpIEVPvoF=)|Y1vhuT2H-V?~x1rOpm58n%0J-5J@ zOo29qdG;h-l^wn&->WQd|H8^VG5}e6?`fmOpH|VSv)y~GfdCLV7T>~~VEU)+U{iN{ zZ!6=ap-DafM;USm5pH|j`~((PVV4dy;Oc}b|s8;EXrTEiAP>17YHHr)U`C9 z&N`fOJYsi&W>qL+U)KAOyOPw`)ZJHc^BBrByP^gcb8XmIoNq1bRIeJxe{qFWGFoGD zQKJB>TBj#enQ*x_xk5`Wi;h&=PjAV{5^nCL>>-Ux$uAc(ija4bJ)hu-Tq&*LrWmcD z?wP(oYx|#GPF&NxUCll7ytK-Uw4Y9+^ZGq6&={QKU*y!QPM>g6P0DEO(u>|1bx5zY z9`>5+#f@=QHoC{ViYQ?%_p>0ljI5+j6D6 z%Se!yn?@PEo-%xNsr}Pag4-U}f@xnad41AC%&1QdOrI4~0jiZ+aBsO@<68Q&n(xsH z6sk@DS1F6wQ7wb8$Y|ex=j_;3)P^JTTN*Um;oUBb*Pr$V+8 zNo!G6=KK;iO8SUNGdWr7kmd~}?ej^R2Hb`Y%_)U z)iAZJn!~FlTV`V_ZegG&^m5<{HR}{|?}i|tkK7^tOEVME-}GH6W~>{^uPSzNCESAk z>Q7$zHNw)MpbKMh2LsuaI;jh711rtTD|un*Nt3z;@)Zq;L?^3E??r~A!oAZL@}sri z=hClQNc97`+n2e2kCu+hw{wN@FW68!BS`%$qzH*~w&}vZuBKuaPVKtATAnWYHoln#UAmP@A_>j?5P6oE5r`;Qes&|$T<+TD^48Gv zm}jWVA6{jnKtTEG%24QmirsN%QGzE7HLZ|zP=k3R!6k~$A6B3BV{rQi>CGm>OI`Iy z5&Xp7mz(jAM5;G1>NmDjd}}7S6|o%y@YHRK-wdRy1d~>lpk}18*fJP6AiQ~^WM-dW0N#KAE4|L8b=gXt#wJkgasY|nceBc#vT?s4M8y4ikh z+sxVhk}QrcoxCaM_7EJko-lu#;@rWpRAJ+<{p#!=8h7DI(EPs6rEc(BBapV26UH$C zUB{)l@3KoqO%6LBJ<5X1HMzOw+izr`h=Ik>mU!Nic`xo9&FPkURj>%P*NAY5IA?<% zC8(hK=FsQa&DZT+B#%aRNQB9?lt6u%I$*!%x8-DZ=Y3PVc+Wej^##s)bcWX_0gI#; zDBM-CuwO-3+61-fMaK`~$%bjB`bo_+j5UE@vrM@W9&1(zB_ZD#@2C}C&%<>l8NFRc zN3I^9hY;f2`cbj@_0!2Wsx$YH&L8Y)J0GZ3qp6F-aJq>?z3zS*iMx<~37duwZxH{H z`*b^{U*l_znbXtUI3)~&)+PEl{P9)o2Si6%cZV%R?wj-(<3Ez0^f>rWas_K%OL$=7 z>`Sz_9@@FD2`G8I(6$(Zii_5L&{w#sQYhQ*441g}H#o5l_5phv8jEEo?&I$(u45jx>hQSRt!Z4`H(pY}Sipt- zr|R1apfFTkLs0r%yFC{(HLrRx-XR|#Zb+-`EWdaOsseMi1nUQr@Pt38=_CK6PkW^x zFt+_U&T(TgyiXQ#j4LsJxnNT)P9En>TYMJxL-C5s`Y=R6{8pL{_wKxy0zzEBANYAc zCb|ASSyP|Wf3YPoWYRU;pkx{-+wfqyHJ*{IE>T{-(3dw~S22vAWU{;l9xHrc=&w@H zD_li`DN(!ma{~jZHm>QY;nwPGid1P08DH}})n!lzqB2~nDOL*5PUg1a^SwWc&hxL2 z`~7~u7ZAL71hpaE(Y;3f6m$Rk;Zy8|VqG>1$M*PV9$ui3^zcm#`?*+8pSQIX_37&} z$Vau_RYCpH5~I@-C>cUjCDg&dPk_(=edy7g)}gP<^47Ojo!N>+w57)X{U0Oa@BPK` z`TgS$?C@P4)KENIQ1CqN>%xFi@gqMw@s0H|q0j&7X3>fcJ~dL>>X7>Obdf55AJ+09 z8T0bK9v)hJ6R+rPQ!ZDlY)-;H5OwpiNOKVbfMg@--^9faa4=&AY8Wr{SlilH*$9VD;naAXR(1}^ zfp?dyM;UB1MOSJYHC#m$XLXzP9R)9(YjxEpQ6b&f6-iu0zaw<77ZogJ!&BdGZsi2P zPrxnU$NLMmc>ESU$imAZNjnYHm<0Wm0-j^UI`)Hox}2167XGKPV5qiCTZ*V$5n9yS zaIamAUI0~q2d$SRZ#e)TL5P=$YK(9y5`G9TwBjhPB30poV;!5}NsTXcN|P{tibXq* zIjk>7*eOVAj#?nSD)JOcT`J3$2faHt0O@hlz?Ac3Kvt=Lp&2>KP{+jy> zi$PSa08uT#l#NE#;kUT)=Sr&_(W!&$`i)_UaR`UK3t_(-8V6FWx&(H4CHt$Y| zX$S2B6CZZQFiK@VodgT-4Ufdna#1}9LUF$bmSgEfwCL1&8>%^^*I1n0b7jrYDR6ee zOSOlvm#(rB2GRDy;L}U*v%SFnuB*4ei+8H^S{xpPS1L4yA|srUt&vx#rE@6LvOv

IyJUJybz8RwZ|eC=MmgC?Kv&9cBk$YV%%~+0gHUy zt($1+F++e*5#sd(JdHgn_$YQ264J;-yb9@d+W9R_{rAnj(evi-z)D7lqO|%@&uJG* z^}l{E<3gThhab1w&A7wZ#9C2b8e3YHvcp*Ug1$OiC3KQ?z1{^^A}x?D?zE^qC%HNXh%(SkNZ=lqf?374{q6!%`wDE@8}ATU6e}U z0a;a9yc>SCk&`LQg{&5Hgky3`q5ON%qUA!P@}B?dwzFa-USho4Hj^*i2T;Wa=cz43 z3V%7^`GdYBuoXvkjCZ>%`4;dl8qQHeQ{Mz+SrlIa7!yhjO~Bzo5|s#>NNlSsOTyzY zbvD#2Olw`_BVJ@}S3j?7j1AbnCoEjr4wl9#tGZpX8CK@J_|XKL)_3^ z*XmMrJM1!go8?t5TDqGF=yqc_L-`E~(cU(Z`=VH@2$lH&ase$A;pOw=5XxLn=y|}` zqOYnr#VXeNCu6C!-G#y$+Uhx%0PMS&fQtP>T=9(lKpLmO2sQA{Dw1Sfhf1qVCCmtl zbhlZ**HREa2c*^vzXmE}d>rQKr{ zhp{zIy^&X?|CTW<;`#x6`zgP&4I)Olvr7~Aq48Px`NvDTMOPUpNi2wD>LGPRqlKta zB|USFMKZ$TRcM^5wqhtXdS&a+l}IL@=30mx78Ei5N;E}}BLders0O+Z3_^zcxbEA} z*RT;m&F12Lb7|d$v{0EuY3@?dkVI#C@_R~3-r{+YJ1y*WfIzF(XsRA2j)t+HRn*9;cWg|x@{+s%TU%;Ak*znS>6n#sPYl37oyn_i9$>&SR;x9 z7<02wekZehU3)i{{OIUaCbgF6-=HW&i@D)TP|!2P+#!D^Mgz<$tGh%84+YNJD9b_a z5GWe*!)jlxPp1IluzZ8xxxq@i75enm)vo1BrhVTG9jgx7Pe|@S%@!M_Tr>@MTM~k> zsyHjjz4pXdTTnHG7!jr(hQHZ} znV7jCyAc)C=1RmYaUM2aOND8BS)CS@P)KoDn8!f|qs~1^&&qP%KX<((S&VT2X1t0I z-@qI52@ipULPBo1V*Im&8dc+9|7$bFXij;>|<2!-*HON+RI(N&|~IgMVqu6g*5ctQ{}3U8IfRB0M~cG zk-s1{gJRSzaGW9}M-ZeSpVH1APSiTg@XxMLRRlnc8^E^-gfYe#SAEkN)^3n`mEW_3 zxp39v`T-HrJtnGycP@K3zC(JSR@g#Hzb!kN z7_@gRU1LXOOm;3EPStFspLNOdn7OOfKv%a1p`7tuBykIMb6}t9=16uDkZxq06JF3_ zDlJ8&)C1HDMUEabrRuyU#L}#Gj6PIr_8+EHyL`qrsA3t7WqZyQSF|#9!nPsLM-e86 z}7o2^y7+pKP&?p96Z)M0Y55EOd$q-tnqF(1LI z%BmA+ThW_TwHu1!;Hb8pR}iP?Z>1?vtNT>j>*Jut`9Qx@46)g$tg+H~y((r=mYW*Qj*$szwM*xa zsW3$h4Z2nV9l+?Uiowe!Mln|(5fm#p%|6Ia!zkK(Sixln&=VVq`kvW;AZe?&nfa(T zb8M+V;&Pd`DL+c^PIzRJwTq_ut(~pta==@MUR&vU-j`iaL5+O$TEWFa97rn|RUa4X z!pLzI`HRxCYgv|PSQl!v8ro4|n|>%4Yw=pazRs`bxmdM(1uU=Hw``cNWRqnM!;?q6 zOhN?)o+Cu?@9oJBDQ+)eW(La_z3~yurfq2u2wO>oa1}?rVktG!HI3&hVWg$0z#q6; zU~7+AY;iYxR= z!ykTN&2=hO7OpSDyIPk0h`$tq82MLfy#`(B>y&@7IN|MQTd`;k$vlgwe3^rRgx=A8 zE9P+CJY{_r?kRqaD{T>zfVZB7xp!7E;bW*w^~{B@q9kjyozr7i^C7eH*3swA8xIvGhj->mrz zdO23w@||4!V{nvd- zo<;GmObsZ&#}eu~h0lh$@|_-S8t($*CSy*w{LiVDw}C1_T)tZ8IZ}OK1WEIMg5Lmz>o8NQjG1pJym8ap`MG(5K-KN3*IWkc9CgS z8lHD!4zmfiLtJ&$FxxX2boNeR&JyzI7jjEV3?cAm^ag7RTl{%JB z?Dawa>E04r(rvqKWY#uB!^bLG>1DJH5EfP%W!97^anu(E#m=FGt39@%-B5k2e5{9U zqGihhbVlH6W8s&d7(3QxfFLqK>6}nyv|Ygo(~YMB2`0uy&v)pIA6tQCe860R3|=iZ z^sWD3h2vbUo_jI!)Npmr&c6qYs&7~dt3A}e@Wk`P`TR{dpn{UA`c)Pq(x+k2cdmh_ zoG59$O(BFv4#ER2kvTUGtMR5+`IHnR&(phhyBD5H57dt_KtEkI;vYz9d4R9P zQphr@pd)tW@PuBevBr)Y_fmZUI?<~7qa5R7-N|IN+p5RhhgZgyr%fMFWc*zwo>lCJ z*>d-!;!G-QmXd*@-@?bXRr3}G4K$D?4JsuPL&yJ=V71b6+1#jm-)@L>oru<>m>`y} zu)M08J8sR_K&!Vt=BipfodEOw4BT?do<2=43ex6~;hxHY3Z^X|DKEvX#+87Doi9?= zfQDsiML%j;@W&G7L=ZrkiZ$Zpwwq--^7*ii>`WRbE4W1Qg^YS&S8B-Tr7m-Vv#dPR z`~9MZcOtRp_sRad!yUGkBPV$ItBdK@;1@pCqZj`(#G{A${^ZIaQq8;Ar#-lM-}^Zd z$bcx^)+c=ge#5j4QeJplVbwR!H^os1K9aS*l^#63NmBPW`DUFl_Y{t7FWfe@~$(1zyS|?sO*C#zc&kx2wloP}# z*}E1^S+TQ3*Z2q6ZQSu;xl7ko99X!A;^2{PEYB{VHjg?UO7ki}YoRX|@%EO8Kq8lz zUFu)UkSJ3~z()y&FrnL7k+G3(RFNbVJC-nIcp!YFxqeM42=M*ByV9rhtB&kcl<_>f zl;z$mFXec3swlW$oXhili!3iKnH?`8^R(`hnskp-Co8lnxqNR6#bAQXNRHJ?*Da%-2Q|A3&ga+(DzJy(&o1cT?;#Hv@^BDN)2(* z9!oM>nO?u+czx5}>ZTLDE_BR9#euSv|62MtYP*f!wn&fI8evh~1aRxybN)2x5M_2> zEAioI5+#S?SDZK4G`bsFl6U)k2{F}ZN9?yWeKhEW7OYk9#C39;%RsR61kVw&p<>mU zldq+rNmupV6{yR~v$}+K)-MliX#_@(tZ|*5e~cs;Z|?Ch|Gy15TPIsvCw?y=Q6{*9+cV4)>u3GYq{-reeE6`M%G)B zKoD=IlBW1u%Z=wyZPr9|f-|4+EHJV80HQ~T$df=P)C_{VRC-~r-N1aLJ%Vj3ls~hZ z(9vBO7~XjTDz_xUU|(fp_%~d@sC=hEadcjBPeDvODje{tU@*I%@SP~GqA->4eAWc$ zxAE0`1c9yTyxffha^_lM{TMRirgug-3U{>p;rcS!qtQ#)I>J1V{N`c}{gCr{6}OPN zK}-SB$83X;6eHa~=$zwXz?goYUz&1}K5+d$aX|Opb^2dV+g{%f@6L*uN^;Ivh`btA zsHL)oOvJNv;MM)c^m0p3NQ(U=s%ZpELEeN5%-t4?A+FFmdsGG^6>ez zjgQx>m_|eCK3AA{bm0VPSKhVVyobgJx}OP~(ShNjANN!~P5}`xpm_xc} z+F`P{fKycQz(GcQS(^MAbXK@L*8d5gDPYz*0TlsE09Uecxf=3%zo)~@e`}m{?lqT# zme;@>ld5IR&+)hZ`Q#$)|K-WxC$$v zt|>qZ4_0wRLIPbBQG*#KX^+x*d`@R1s)s{5#UJSEQe+!lgMB@DX?=qWQwPKQjJv`C z%`uu4hnBIuT^WSx+QH)u4xw9D?n$48L1XF6_HJS67#*WPUtc<2U0OLx@)tk*Pa z^W%TKviA7_5Zj|}#R$*xyY-?+N{%KI5?5ViZ|Nh2k;u$$~sL=h@kLZ6q zzL}@w(K`Zlc9sLfA;uR?vjM3$@Qq`oQ1}jroS%Ul&QucDa2qPJD>Q3)q`+Hr;ZrM;$IAwA8xRvDJXDw;xZP%F)s6osQAr2)Nzxz#iVC zsJSL{;3uI&`cpRUm?`^QgmkN;nwx>yCP+VRNC*y(k=FAG} zn&2b1Cw5)NUon@Hlr>r}k_HGJb&bRs+=HSC=^H)FgE{col;+Es>wm6sM!@hK)aI<9 zqcc)%(u>I~D~WGO(;*VU?`B244=x_9k=zG80hA|A5WnqB7sYu^4?^B^Am6fW$kwQn>4%I=fUo`q0T!6hR zwL9E`%i55*_2}s!j`fucX1 zE90Y5F-x7bH-xpI&C^qeLc)cn--~j9@2qF!6}{95bAqg`B_CqNH();id&sN zuwqb57BgL3dpsFN6CT#8Dpm=qX8Y`)y=z`8%_PKG#H$yQf_Og(^y{^+O87_Pl$WD!p*v2thZdo9N)Y9 z<=XWb8+xxzmN^cyL*}@gJ%qUm*jUG7>?=Scy;>o>(ET#`f?lEYczE{!y;(-l8cX$T z-}4>mihH-zEz-NzTyH4-@By$D=Gva)x!mUuVX~ghJ>fg9ygyuFok<67AbsIJgdGl$ z?*<^i@7HAOYIhE*<^gy8;2zJso5^u$GcO2O_z)Ne4{T2Y znmh=tl!taG9Kk*R>kyZ;ULl#_U%VznOZ9+Vd3U~4lNxX>ItpnEFL2!8?$NOxioVvo z;iL4}y_$C)W5FglrdM3Q1m6xlU|Qk5-T6+rj8pd+^LX$OcE6iqUB~9a4PD>AXX{ub z(T1z=T`BLL@7U$Z^tI^kg67NuKfDLZ=!Uu-&YZhkuv?*A9bI7FxZ*u-zS_5i6n=O+ z2)p#^W#9D5n|kG$s zAEYe=(|4tC-D_>3b>XupW-6n-KhB1IRpLn4jQ26Fwx=^ki`(0NX-f}gA3iV@PkMIf zhLUsfaMWp>vy=`Cf_pjVZY1$ghX?~E<-klO; z>Ouxi@Br9M0Bk03J=c3chdx1Qa~b&3eF#D)bZkFh4tLxKw8J5@T_K!MJOC7iy#?jn z4IM{3gvkoqcOCy89ACGDUR*zW*LFk*pTxb798uxcwd~M!b#cbI9R^(9j=8+#<7tLB zHeTy;8x|(rdvH=y?X5{PF8cX&v3x?c@o5yTF?1dAa9AO8F%vD2VaV?#%By7IeoOZ7 znh*(rc?OL64+$n<$Vo5c+>;M!(#&Q7*hCJ0kgOl3@4}8V5kA!EgOd9IVo{lo{z8Y} zr+vPD&YsO(>3ZQqP{DlP_WeNc;C^y+%!yx{eS@hPI$q%416%WtkWRF`Fl6ButZss> zAxbPx@wlOM0{5X#-wiPW^Fhv@Gn<30&fR;R zSTvr0;`A4m3CRmI)YfONoV#`qa8Jkw7a6$r5+y@JA@uK)@pdilNzazjae}~mfZ0gu z>sI&(R-NYxhY9|bt=j{A6jpt&u$x@P}o?M%nctTlM4fz z`|w5j4|V#2(e^*s=}X33=6_vmLRBm9L+=3{EeIxU$MZrjcvy4xBnx;Dd{qc0SZMPg z4B7oWjlLFaoY`RrY|Vd2V4G%BD#SBrUSAt#+_#16gOMw~7TrMzVRJ{i>{@l{PY7hY zZ9um#*tH$bJXrDF<3daQ5v$a_9PT9$yS0w5l|DWSU8EisJJ!5EMGwC;2|o?JGUHxLwab{VaiK?Xd^W+B#JVShkGU$cB+ zs~3EL~5l}jM~&H-MdJW zY;n<2NgU4t1GZ9FRMi1**__71!V{dAg)s+Nj`^Kb5Bm?QkQvPv_};xdO_QS5vS8iX znFi`qXWuu4O9!~kjq0W5-v(o8eA)iaE1z8J$9nSB&EJ3Rzg65a0s&AVk0S-;vixfe z52eR#5TQaxl{V3EJo(sygP=u*(%TXTUB{9490ylw6W7tu4Oh{y8<^XJ`C6n|=fUpR zuM|&ME@`kODXm2{t52+hslF51OiIAmrfVNNC=C89olORpSvpPt#B;?y#tP&1R^YiZ zXrNCEcIBjZ<)rU*sq}t{-0RhqCj91M5cr&bp%r9(7TPWkeeXdA;cT3kL(2toKX?FT zG=0~x9f36x`9qj2vn6x!{O=)5)&s3=jY=!`Vy?a&a3{PE6*KaJ?M{Yu)#`~J#+=8`vW2SU)7RB_~L$D8Ra3Ub^+QfUs)!7h<;*E z-ixkpf9|?NFxwm%@cUnWi(_A4(EXX~vxhKQj?Ei*==)z*P&CK38l1VF0Fjs66`pW}!T5cP5>4D+(aKbKYaZZikOMjE#%Z(y=|DYJ) zYXgECH`IUV`rI6`;izkikL$Rw6joP{yjwFKle=DE3s7DHZO@lMba)Sj2Pd=1bV`r2 zRcsDCcNLf6r}Sc|_X5ADaJ%II&MHX!9fkeP&gd(zy!lmWk8Dn$nVojBpL! zlZ`SO9SMup86^o_;I--N0pkfgfGmDGn_NtA>o$2c#_9nJ3OU;%8a0_kqxk^O0n=PX zn=-ZhOEb58M@gAU5U)Yw-c9*W>Y^wdicezBA8twhcQKoWkht?Z%6hjc3j^ zJKt8Avny6}!^qS1Kp5DZ3+ZeELg{d}ZP8#i@SXb%LT3ep-qsabTj8}9R03v)U@qyb z__ct-r7yRoE6EtY&p=@-K-nyEdo?2YX3+6%#zLVTdr7Kdb5#gK6{YO71@zK-ia{9& zd~HDQWFDd1lWlY0U{dSeq0z-KJ6lX>Tah$Do<^{saw1v&AsT1P(~9!=d@_TbDJJ8^ zT<4YNlR=SZu!y;iN~WJbltId?(+D^ZkxCZ%1qODgm#V#4%m#EjC{#AiYZPVU#VoDd z)n+QV3pkO*cm-RFX&QHKklDRjO2Fdc0)x5Bi`H~DNfvRMSo_j4Ab4yg>2!YnWCYW& zK4g(KTa44A5}yNRG6Kq%_#?voWhqu0^o^LBBn*2#W=MmulL$8Q-Hdw z`@0%UbZp1FAs(s{%r@+81Ujn5wgxIviRNWg#ETY2>CDso$ok7ae|hhZyzfU?F+uCqgAa@iFdS-tcL1X*`o^GMh z(F5)fglYffuRjl7K0o>8Uu{CAKEc@oW%-;o}F zj~0N`NM+M3F9`S~fU*y%p-mdc8X?A?0}8_8t^Fc3KVzkdJSV&8VDB@Ds>z2mkIGux zfUtnvgCT+Dd~p#G=c|I0J_xG-x*4av;(Ri<7Sl7JPN0SpbJK}Uc34xGf_Q}uGKfg`0uTvuqg2P+85Et)Y`Uh{C=rOd6IZizBb6Not{%Uy z-?&B!R{eHZ%zSBFyj08N`goq^<(-WemDJRruO<2;+%@_H*@=@y4j?G<|g8i9{c>6beV`S8c^Zt_f@o9k?_BH zd#h7f_w`0`PhE!lh}})Xn^s*c~>L zwRs?ot00=sX9O|$kX}sZT77D#6aB-WPE@#gC|fR+VdT0t!YHNLm#@TjKw;pc(rWl5 zy+MINMbBzfDW;L(w?QIP8K_`=yF(vx7qq-ELrTO2@7S?x_;PJKjbv*&DS4Cu*LyN1{8UvZG`%=`i*Rg z^1A2Ii&2Xir_D#OG!g0CBNA?UORtJ%#)nljG80N%0Vrw%IO)gC29LjC^?QRZD$Ee%yAss71)s&EX{l;P{L|ayaP%K^KTDI(iLJ;QgWAoaoDJ1Y&5b(PJk|$&rkeDZK6-Yv|P5GV6miV0rF-?61NSxb( zOO@HS4Ul3#PMmZ&O1NwV5(|KmWGMW*0a6%>F6F_$T|!b$;Y~oq1KVkeq=1VRtrpv` zU^yRZV73i}oX{CcA(DtYZX1LU13;~~BQjzb``)QH(4O9+NaBod3MJ{;K@&Bbl*f8{XlDbhVyAvprq_`V6fp*XdC+XiVn1n}7BR7OnQzjwj@mh#U zNbDHPJ-)3HE=O2MJruIdR#e35v+!;d2W%`9bP4&zz3L>9s zt;uZ(De^cEq996q#@j%IC=kHj-w`267Px|ke>BoasVw*n#80ZS8$nQv<& zi8LOi5g1@j+6p1S0MPs$_imI1=64!Nj5)UpBam^-+uW87mfS&s3|IU?04LCpL#)BN zEhVJ^s^Bt6`LLDQg*Iq*VQ@!;Bv_}FwCoqUA(T9KV}Zn$NXd76*#;sXeT!^srx5Id zA2`DxbONspL`15g4DW`NT#q|l0!cXHR(*HeVH*ROGwMx(%W zlTi#NLMyvN9Pr)HyCV>ZP7?`~&RZoT*9)CZjU-?(O*>W~S8D^2OG)9|IvYkJl95b? zPSk400;+&q3c|Z_uDXHjNfVPGJn|@1eU7X0wtf@jwiis)w<)_2(EKGmiP>v|x2R#k62(e^`SeQiZY+zXN+5K-c`(h=eScf?&Oc7d|Y+D+Ko z^{M)*)agcqB$p(ThvleOweKLX0RZascq_kmAh>iwH%Y=)AO*}0f-t-ziR8(UcM+1j z?YIWc`AsP)L@jRRaDrj%NeEQ7Es#bmb`$BQ{xEKJG=$7{@!a~3PA3AtQ*D50-LCr0 z`KJ1}&sM@3oUn!8F|0K2`*LenLpb7m6opPWl&vm14@k)m{5v{B@_gB;fOGejPbe6d zn}W&nmcs3X&%@PlJH}kR5|t@`W9+z+u})H;lY62?a}sLE3h7PNFZs9-en3nnZVa!!1iD z9kRBOmgg@~lfeC2@v#DY?9c&=bX)yl#DU+9{4f$;&;}&mxBbBJ#oZ7Spoi~uJBNbX zb+Wo!MC&hkSsh0>tIi2U@&u-rcG&T*nR83lnqblV(x#8lxW@@~Ks0zGvzy<3KoySa;~ zY1q216syVu=dgv+RZ&-4g*haVAB>nxeYfpk2{AxAj_3P#1fuTL7bcxs22sE_KUrFm zOCDIdY!yhZjg6JIRvuX7ry&c5G7xQX4jimB&cq%0!nQJKfLsHf4BV87<{jxS?RHVR#(0KHu3oBs^saPqCD0 z;)_;(hq=vIviokJPujn4d5=ZPmm>tg!`B zT-=dS>vU(7sW5`u3~SvuHPdCw0phK3t>}*6fyctFVFZx1yp||y6E%V$o9D)F7b>vZ{A(y*%q>lT% zo9l6YyN0dj1slsIzNC!<=V=L?6x_Fk%eN1lOxZ}Lk>os1+M-(#3&PNa*W3})RTm-ZTN;5=5OgG zyre*YbZFst!lNw|TN}0`*O5^g$T(_qKp>@Z$M=IfQuD$qIG!q1b&Ekn-sZurOSRR& zxNIXM-ws@<5^uITP>)Ap>W%_03H??W0htQN^X>{nqB~RbmW7h%-?X+G_J>^!Yz3vk zhE4Id4s3;u4nR8?2A*K8AOaMSq3hq(@sieD+v^70wtL9iyd>0em^j1cDyt{8LIRwa zC&jj;6eq3=DjtR)x04dEQ;+%buDs%*(_JTpd$;T$a=e>5#XZLptH(=11~v=4VC%zL zF-%4<%!sGF%|%35z=gP@Xnx>$-Bmc?Ew>#f?ujj&MD9{Bo@`5D5*Q+kd1hNj>+6YSFfa~8;wD?s76?o3=cC>IejerQ2i-2M! z_m=wZJ_~IZNY&ZeH7@b8!-lLidJ=I!0B>b?=FICXDS#=fw+kgunYIY}T8e=H#k)m` z@C8ULkWPHtSQR&L+{6*VC>pjgJi-G3;|F)tEW!0UrBisj#fUb~+?3vG-L;m$JRe(; zIKT?GF7nQq6v*NwYx6_+F0&;AILbSMQP`Oi$k;6xjM}(1p6f1Gc$cBmB9pKQGZWs{ zXSUo;oxqC(7T9Zrk#7fJ-G+Bn^_Yv!a=LDCt1yz%*|_Y3vmD~;2jObf2OkNJ3CG`- z*$v&$_me1QiD(r{)QMZ%Re^WFzD~oI`}}rgk{4{=7{*(2y0Vs{556aW8M&dit#%iW zKy&k*kr&3Y6-KBy2=|WiE`;Bm03F<}+Ti)_=4ob2W|D*D@>W3vWF&>Rbs+%}#v+OQ zBuPeXVGll_3Kt%icf{*-7+ZUh$=f9(;j>MTX(e9{n{-3j!s{?5OiIt$){;$!GEKaQ z4V_^-kPrz%?~V>HTo!bvFWqvLNl=nEuhuG-VANKZ+V?S^GTgeCMIgL1kYUP$cAWuH z!2kc7yS5cOZYBCJ{(wfRD%IsqzN*U{?!{P=hH-1dEUuX>VYc5|JlIV+3u+4CM}B*zEKU zQ5Zg)#jCdfU&QD%4JcrZP{0OvxJW9zHnH2&O=F&|)EcZgkn*?z?rm<07|1j1!%_oZ z>*@4tP)AxbwU$=JK06W{K!~iH)rMnUJua9gTbdC06#$Ai)xe)BE5XGjyVKJSySXlZ zj2ctNQnpq6HF|-9e~i57vOrU!Y`X(yH!&VRQ&z$lFRs{~-YnBvCjkY3_b9oeE9BnL z{T#)}BhLZDxHju`B=aaC*|g4*jv zlo6et2~SWJ?Z{7RU$j#$jK`ki^pg%r;Uy%a(5kCyk|8A1n)6~t{u!T)Hf$SA7g`eN=>mB1ZWYT~A(h(8T>SCH9Gf`#Jzz{X$vc;%`*{D)E*|Vkr$y$Z-YZLP{jh@xio7ob zi`@j1lJmF{!aC25PCA4lEX@+qP4>h|^^k`{ubv6V&_Tw(8{Owd98t;-r7o`BGvS7F zUSd(GU8Dt6xd{F=-WBT!9Sp8`v$DR5fcyC9JUB0=sQk;s@caaz-oTQR$ookR~|ZW>X3f)Uip zj`$)*OmLo9rIKbc-p?BO8KwcukEb{NUFK3sQp_;%0Y>PlH|x8zBd$0o)2rQgJ3DG5 zRNIkxyumGulEr?uFTr)?3DOi<4lv@Pa3JpuBMR3N6m{6qj`$`SY6~MC%8-8dkn-do zyWryPQt|;trMXM^?l7t+E9pgy(rJveCXq^-tmW(2eUWrezpk8T)Qn+GN6(Q*dv^_k zeG@xq$9m~vUk9h_v9;=^221<$kpOzQoRaj$3?ZzVWSTV2p$SqNFa)}3V=9 z8#TSk`~auNBRoW?(xw=s|}>w=G_2Hv{`XDTTD@ z0Qbp32Meat=+3i+QgNwNe$7oCsN>}stj8npRg#X=7}XMIq{1yrk^$9v-Q6k#aT+7e z(khcbZa|SR=ev4@$HQQx*G71$)o}k2r6E_OQDqoxU%4y0(-!-=1$d1x5sxXgHf|qw zWd8Yr?=EzRhS$~{AcoT&LQXZkfJva;Y-O2@8A=_ro50$lL%jryIW=*L>e(0|-X%|> z;aDB@;t;PT>G-@14t2YoF~vrNw~+=>cZoHwW%MDBnc8KCLIm$Q*wK=?c*Qc`A|nwQYvQc6G4>kh=3ly>7PgP>z7ZTssWz*jJ`XFV5Q0DIWN+` zco1*-^KWl|4C}{V$4@{1{^gfn#<&0O&Z7Lthe{)$Q8@HtfIu AutoWeightsLoader and already applies the quant-config cache-scale mapper before dispatching to this delegated language_model loader. Standard mappers are idempotent so this is not a correctness break. Drop the block and classify this as a delegated loader in tests, or document the specific direct-call path that requires it. (evidence: Read mammoth_moda2.py lines 476-500 and 810; confirmed MammothModa2ARForConditionalGeneration inherits Qwen2_5_VLForConditionalGeneration.load_weights which uses AutoWeightsLoader; read installed utils.py:407-416.) + +`tests/model_executor/models/test_kv_cache_scale_mapper.py:112` [minor] — In `test_auto_weights_loader_delegated_loaders_accept_mapped_cache_scale`, both parametrizations call `HunyuanModel.load_weights` / `Qwen2Model.load_weights` directly with the already-remapped `_MAPPED_SCALE_NAME` (line 112), so this only proves the inner loaders accept a mapped name. The PR's hunyuan_image3.py and qwen2_old.py changes delete the old `get_cache_scale` branch and rely on the outer `AutoWeightsLoader` (hunyuan_image3.py:2178, qwen2_old.py:443-447) to apply `get_cache_scale_mapper()` before delegation, so a regression in that mapping step would not be caught. Add a sibling case that drives the unmapped `_SOURCE_SCALE_NAME` through the outer loader class and asserts the mapped scale is loaded. (evidence: Read the new test file (delegated test lines 99-115) and installed vLLM AutoWeightsLoader.load_weights, which applies the quant-config mapper before `_load_module`.) + +`tests/model_executor/models/test_kv_cache_scale_mapper.py:12` [minor] — tests/model_executor/models/test_kv_cache_scale_mapper.py:14 sets pytestmark = [pytest.mark.core_model, pytest.mark.cpu], so the new tests are collected by the AMD merge step at .buildkite/test-amd-merge.yml:32 (`pytest -sv tests/model_executor -m 'core_model and cpu'`). CI currently shows buildkite/vllm-omni-amd-ci and buildkite/vllm-omni-npu-ci FAILURE while buildkite/vllm-omni passes. Confirm the AMD/NPU failures are pre-existing on main and not introduced by this diff (or fix the introduced break) before merging. (evidence: gate_report: FAILING CHECKS buildkite/vllm-omni-amd-ci and buildkite/vllm-omni-npu-ci; .buildkite/test-amd-merge.yml:32 runs tests/model_executor with `-m 'core_model and cpu'`; pyproject.toml:236 registers `core_model`.) + +`vllm_omni/model_executor/models/mammoth_moda2/mammoth_moda2.py:486` [minor] — The changed paths remove the legacy get_cache_scale() KV-cache-scale handling from HunyuanImage3 AR, MammothModa2 AR, MiMo-Audio, and Qwen2.5-Omni qwen2_old. The only added validation (tests/model_executor/models/test_kv_cache_scale_mapper.py) is marked pytest.mark.cpu and uses fake models with a fake quant config/mapper, so it does not exercise a real FP8/compressed-tensors checkpoint load. Please add or point to a real FP8/compressed-tensors checkpoint load covering these loaders before merge. (evidence: [sweep] blast radius: changed paths affect quantized checkpoints for HunyuanImage3 AR, MammothModa2 AR, MiMo-Audio, and Qwen2.5-Omni old Qwen2; PR validation is CPU fakes only, no real FP8/compressed-tensors checkpoint load shown.) + +`vllm_omni/model_executor/models/mammoth_moda2/mammoth_moda2.py:487` [minor] — The new load_weights path calls self.quant_config.get_cache_scale_mapper() (line 487). pyproject declares dependencies dynamically and setup.py does not pin vLLM, so the minimum supported vLLM is not visible there. Please confirm the declared minimum vLLM version includes QuantizationConfig.get_cache_scale_mapper(), or add an explicit floor. (evidence: [sweep] dependency floor for get_cache_scale_mapper not fully verified against pyproject lower bound; PR env pins vLLM 0.24.0 and the API landed in v0.23.0rc2 per the description.) + +`tests/model_executor/models/test_kv_cache_scale_mapper.py:81` [minor] — The guard at line 81 (`assert ".get_cache_scale(" not in source`) does not cover the removed qwen2_old.py pattern `get_cache_scale = getattr(self.quant_config, "get_cache_scale", None)` + `get_cache_scale(name)`, since that text contains no `.get_cache_scale(`. The qwen2_old behavior test would also not catch its reintroduction because _QuantConfigWithoutOldCacheScale lacks get_cache_scale, so the old alias returns None and falls through. Tighten the assertion to require that `get_cache_scale` appears only as part of `get_cache_scale_mapper` rather than relying on the behavior test. (evidence: qwen2_old.py removed lines in the diff show the old `getattr(self.quant_config, "get_cache_scale", None)` pattern; test line 81 checks only the literal `.get_cache_scale(` substring.) + +`vllm_omni/model_executor/models/mammoth_moda2/mammoth_moda2.py:?` [nit] — In MammothModa2Qwen2ForCausalLM.load_weights the new call is at line 487 (line 486 is the guard `if self.quant_config is not None:`). ComponentQuantizationConfig (vllm_omni/quantization/component_config.py:75) indeed does not override get_cache_scale_mapper; if MammothModa2 can be constructed with ComponentQuantizationConfig, confirm the inherited QuantizationConfig.get_cache_scale_mapper returns the expected FP8 mapper or add an override/guard. (evidence: component_config.py does not override get_cache_scale_mapper.) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v12_hybrid_val/pr4810.trace.json.gz b/eval/dataset/arms/copilot_v12_hybrid_val/pr4810.trace.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..8c16724ef6999b4a42ae490d8cdc41d49b65ba48 GIT binary patch literal 36777 zcmV)2K+L}%iwFP!00002|Lnc#cH2mnAp9z5Wu0R(Qv|UuQn{wxwzJx&(rLHsteWrG zJ|qZ$5=tb%1W3uMOwXBL^8jhFlV|fVhO~JxN-O8w}Z(n znPdk?(r;f5rGqFA!q52QAYH|H^}}xm;YAoPN9S-Wf9%Z5Mq!+V^T{lPYs+*M!c}+$ ze&9a?P0=jHQT2gp=#HTbLc_DI=>a^HExdRXd-IS+KbXztbafdn4nDyjOD{X8??+j- z!m;DkYzEiy^Emh-9i`z#6kg(6^CZAW4njY{p9?Sb=J^~KNw$Q)r`{~X$t_a&VK}OS z6Ac#$zIOp%A0+cQI^Y-6)ecMkK(JWbx{29G450@TI)l28)+ZU=f zI=h;rQ82pjX87d9^UtTzY!(Kk35Btr1W|m7uSW4Kio=7_)lnAxE6hfx|6+^5?jK3I zW$G&J(=tqB8qcyU*Zz`U8!gY$Fgr_TcuzAeT9@V7Xznd&s6YMZo7XS#n}#`cf{Cvg zMqqh{NniidKi-`D?cLjF&*=$ms(H2>s-b5YhGA>?#PF9aOtayietGxu1fNlDOLdKj zHg(Nl;wlC`bC%4*;*qJMtA=klTA)pA!>|r;vZ)WZ*k1Ta8ic{fThccHaR!PrQ1$n! zab!A2s!RBhtWrPZ2syc0(w~lPnI_ysY}$-?L%20srk)?l|IA>AX~;7VmIvRP=$7IJ zmZnb)$JBIV>Q9u3W5N2krmpF}F?D^*3*1oiR0n=rDvTOvKHQy5G*}k^sF;zdPT zR%>Tu_%_VPp1R(|302#f1d8L@?iA*t_;#pTj;k52?di7Vgj%RriVDL{rmjAjno}dJ z&PX*}quYc4rPIjIXiBv;3IfFnOeb*s!1V#>i5ps`?|Y%;`mSwR6Hn2M0OsXS;SV*m zCxI0zK1|;;jEPa5k)xSr$2D4{$tfUo+nkh%?ZT%W%ho5hYU;3w{?ySz&$L2KH#No5 z%&De&)5&Byof@8NDwBzWYiby-XWF$@(rmrswgHW5nT`%eQ=jS{Y@6v#rmAJTfukCL zC;G%yCyL`tRKuBsy5XCFYlq=9RAE}Mh*MvIlUAJ$oJ-g3HXlKWN8Tw=YMIf;Q4me1 zEwfY&)iPC6b1Z-2`@S{R9B-nlw(f+8iA&%YT&vi;MUZH6JDLA zYpEUg@#L4EM<*}ezBzd>&x7ied@C?KO@-s;d$3W8>r73}(x;var_Qnz#hv%hk=m^c~AkRaoU3ZmEiE+ui2$;?=Xi{`}_M`&ZB3H6}GNOgL(uV+6ioyPo0c zmSJfMyr_rDbmE6oH4NZH1QSiQr?xlIG`MfrPG}fTsKXKbMBHu`5nbb)0QlUTu`y&t zL7VhxPrYUM zF&;?Ye~*8S7r`y=VcXRm&G_48k)=l}Wg`QP42J*^MlcrsZ5e|@gaW+lTd6-JD39}hYWj4!b;53{k7x476x38q$ z-1}#e_7CaCYi~YJmS->w@V%upOS}MHpL}{^&p&u zxM;zEzX`xgxLT1vI9+*ZAi+8;GqD|>zg)qb05*2781*trLCC->R{*FN&5Q5gkoaWy z8D5v3y?!zL`Ry;bmoqO85j@zu4CdkE9!8lI!@53)C-BSg1EBVnz(ppjC9cvWge8K{ z;(m#5$kNZzaxeqAa+Yrs{BZ&EVnBL<^m7u2Y}$v?9ArV9ZTNWsbMTf4?3Q@%PNvGn&lMg^oXa>P2w&g>Vg=eGibLg*Nf!NP%i1$Y4 z7iGjAu|<&=S4Xxpa5Or<_%SA94S?G&*u+s3mf$`AoME*gJv$kKU@6cRMA{IJG~O}+ zrxE?742Xn;L}G+QVwfKnF`=_ndb0Mv(wV~fX`1eAJX^S+fPye`G=aTa zhA`p$j7UHY-kwO`8IIyega)Wm&~H~PyNuMWyqLw*+ZS}>{Atc0uF;bIC%g5E?);df zfRaIkxarOQ8pb$V_*=%M99@Rd=^5b1Dg%sq2A}%}yYLgO^C9d^1@bx2C2uB8V6pr& zgr6|Cz?x*WB%(sw3;=#S*apI1FT+?Hnd}kv1NIz2GYowI%#EP?FT8@p|`L@{Ac@wCYw4{941;UPm6ivU~$ z8_=X#MxZG600bcdr7_PcEO_wX=t?le3fRJB1gQQilA4jvhzw4TC`4AdF4q_eKX<)8 z<5Rd%NbDopP$E;DRt6F1$iT>nKqtTgC;ZdpX)(mjF2itf{g%!d*BQZ2PpkgI#Q;e} z`tSeqe+6@Tjn|)@tV3BrO*6KYfGL0fsT+Ft8Y|tW?+SpCyCjuaY{W7~d7>CTCaqg* znfNKJIWR3$p(VqyB^0bAZ@T3Ch8>s2+zN`!cZp~u2$iLiwFL&wyem|ke=RYo;dJ3} zGB?n4jSy+#WuZVIxTU)00B+|rnqV9-0fkKiAW4xloQ9u=*=h33Pj6t$E z=F$sWDxafC`3QqFz$6krIk<4T9STSDWaO=aC@K9pf|qEORKpyADxlChA#;C*w}1NS zbvML}GR|#@!~k0W0izcu>oB8F>xL(QAKV()afmW2%9JI_wLQ8Bz7I@xguB5w|Bbml ze*d=It;-mjiA|>wHtcnm_t$c;>dUv1O>n5;DuazmV3~&?e+(B_%d;fbWmSWrv*lm` zj11n)h8-Bvj)oJEU(9weW*fz-lBcin1x6Ehv>s#J$v=@|EE9I6q`=GbU?)p6Vj}#IQNU4og53|uB3gvV zR7cy<2_So?;V6n1tK|qlMOvApSv#?*uNE2|h%?Do9xTqMGL3}KL{nhYj85)J-7Q!J zUsoDnlxwSOVHa1V91eb!ZpDh8fg}J*%#Uv!IzedS4THflj1d`_*2!!|h^bOGPtHRL zsBc9)1bsAD=Lf{{QGldln9d~-SMVgsz{L^OXbyvHc?ELH-~K+JiAi)Z<2vSy>E}{K z*pYf9DuJwX>LJT!k^(c6%gK0UQa_36-i2#;OoK4MSm z6?zrD8Fv_-Bt1Yj2XFAZXn~*${oJ5ZHK+9|ID$9>_|ZqyfYnI-C=0nd4U;Zy;UP(= z|6K7so_gnDMTy=4;;-Alj|bHgw~E*I?r%F~pW}6ih z!ppMNe1Ry=4O8z)CCepS5WywRjijC;YepZIl&gC1B-(JY6>(^0FH1xC9}l=8D&EOD zy^ifiM|Cww;A6P6Xe3FapsGi6QGOex_DPFrgVBHqi0EEruNv+`R;+RxyBW^6%`wsn_JnI zMj9_S2B2<@69-)x&C;+9NPhf+qTq_KoqbQIjP^TXmf#3ZSXtv_bSo>!Qy3?!(=%L^ z%9N40;pstZ6t{`jHv*7u2B!&an}-rcZb&#!7>i8(05i~E+$*APB*!v-d771WIcS>w z(Z~?S7g3tTI5xlu8wI3(5Pk-U6K8_Eg~!Z?)fTo$w9SGLd03hRZe9kzWAA9_ zfTlvU14)Lg;FC+_zH8`#Q33sP;0Nhr%fBtzb0tI?xGYc+DstLX&zvY^XSm98tI-edFQUk^b5|hNY8Mv6_>S ztztQ!&AdsHq8~<>(Q_)AaW^0`4|H#!i3rFaB011kwy`DLntIV}1-oAM82Mg|V@yys zfU`FT1p}B+n#{$g{QG15s~R!;uY&|%iJ z7#EQzt=G?vbjKn*Z6|!l2XS(tKt;yvCva5Gfld*p=K=t&SO8or#H(j85?9IC&QjPI z-oY>~b|(%$FGn7U#dzaccFt_!bXswd+L+|Wc4t6` z3)ioI$lVcbPRQ(#+wug(3ds?OP$C{w$fP%eB}|zXfc%m?&@p#`B7K69!a1|fvSTKQ zL&b~6{eGimlFYIZ($qpG6q25h*+wWxG=6M+vVL>jzc(HJY(zw3glBQ{INiS>{)Md^ zWuce)XZULlyOv&+F3f;UP@OWtMOqnkT;KjjIsEGj851fjo{F5rx2oUwl{d7&EnuoWwA@)BHw@i4%o`k z_uqevf0v$vpm2&Y1iwosVYZqrrQhN5U@)Nn!zY-;Hso}h6P({AOX5!WUe*U@FiS4! zP3av>5g0uV7$@SHf(gWlf{!U(nn@Sa2w)S!B7K?DT%sID+|81aCCOu*a!fIksHkMMdEV zKgsz4KL1`=KQN>(lg2Fy<B7NdB#1v}n6IAb;#ujFY;c>UmPIBF}GM7$geJu6pxB+QlXMEg^xQqpLLE<;w zlE94Fml`Lzhg;RJJld0Gxj$x_Ve^G??d-z}blm8|NGnW+-C2@q^kwN?QjYZ{O0~R< z{7}R_69HmCNG#%nd_7^as}k5d6b13KD|x#SULIZ6peOaki!CUDNskC~9_ zJR7ZLLF6wpVHHQI4E;2@a4t(Pn9%?YiD;#uH&c}3ILK@0^PyC>3=}5W!YsfHw`g3? zaC&)ckIarcP?b1aWjuTz!Ai(B76zlq6=k@20c=C+q-5y*z^g4nS{?xF%g>w*z25lZ(W*Y4)x1&$Q) zjIkM#a+W<9#TTGqM5kzrq$TBPa)yI>Z&`>3X^Go#K#}JxQKXf$w@~L!w4FPLx1-F& zQ^(hT6-WUYlQ6T8Y}@GOX^DO_9XRykX8ijG8biF7We%W0^(+Y(qIx|I=w=D#mlnj zG!||MT5tRzSk>1!tW`+Xc_w;%K`99!UHLo>DkvAMzFsvoFQ$blrTOK8T;8m4w;NIFLI8Cu=Z z1w>F({6M+qsOqA;$V$LgPR5OmU*b?`-42Dm4O^Sf@d8D}6|*`B)rmX)^Dz;JxrP98c%HJCd(b#r z$aDHY1z;M=QrJFeF0urxkVOc_ioz&T@3I`I#wLL{38W;TR6r1Kfq&&6;;;eziTut8 z+Mo($1Z`vDmLg`H@i~FKl@cvuGNf2)cG7dXr&x;PA`dZ0BRu5%9VVQ~iZIMHLM6vl z>1k(_(&&KV$~I&&H~-d-Gh<>kWd0HLy;{s+mMf>iK3vAi>mmnAXvZnqn4+sk%-{+r z`ww7iXhQ?)m*$DdLQ(e5qQVtI*vCz_jp(B#rL1Bx7HPU#uE=0F5d4hPJT$!tv%WBb zw<2d5@7e;@(T=c9XhaW0*EK$jcHMC+vGh7;iNGwg6Y(@}gj+S&LJh0lp+TjQoDFg* zR%AgLu;RrjQ6C;YR?BJzInYZRpU#MmJz6fk1RnkTFUjG%YrdASlz6^b_Szm`0jme}1j0l~Z1a_tUi*EBPI zL$o8aE=M@rqjh4J5M5E4;Nnw>(;+4{1RpMSk@=jDdMurXAq&gOg{wU9HRAgEFb{7K z43rskK_R$83PzURYst|L1fgcU*mfAQAxK5whtmN+&izVY?&#-mv*fEvmc2$Gi$R*Pp z7mfu#!t#L9CD{(p4n)OpVls`5caW7c-i?LfMA&X~D<0AtsaJ|5!i&tO#u$W%Yy5ao|^K-n8o?JJsaeDhaHZ%X12-Z6j_=rT=c`lwtk@<~CtAbg~|_ zI!%CfH60B{!r)9Y&yAfb+^51ME%Hk=Jt@V$kDG3haj6s&5kesXWk8fcR%4R^T8UFk z&-u^3u=?U>g&Q~bjtch*ED5s}3-vc)DGqH1SKwO_ImbcEGhkSlGzk$m^RsLe3s0&( zMmizg980u(MRd%uF#ItiAFL?8g5fhbPRHEeqeL{>5)GOjHutx;Gg4MYDkDSs1``ipr9DxqguaLEFNs_PEj?$Urt^gw>@)68A3C7?2z3) zB-jy2qUE9V;`tj11%P+IJb(W3-MiiL7FF%QTyDc#HpX(%s`c2^DBA3(L#P(>&-B;F z3NT|5h>x`w$3=P*P^VO?E?dP(i0XYVUf`j@-6vwkP5`}6C#Qt;MJ5rK&^Er5+9V0CFxl)f2Awwr`HZ_s zP*N4{%qU}WVc{7*l&(Uaa~{R$KpDMzkCHlU$LPmbKfQc6mXbLcc9uEUB@ePN_0xD5 z-Px!fiAi@P!Ix5-M8Nm7DY^@{3%s2%*_3PjFliRY7q>alci{m>-u$Gtuum!pkGa{3 zDpSrlbtNQQ%vKdcJfZgSbqR2#n2ctM9mV^}XH$-fhtgUs?H@~<^CMJkpt^kj>8!S$ za?0cRnyy8)=Av$Jnj5+ozm3^~ij-z8uA3*^3c!`!QjExxJf@IQx(&7n)LGYABZ@Rm zqE;=I6QujXX2FyRA$c&Q6ugU{)z1LMBE)=vBtGh3aUhT0xLq6_QwnNaQuf>R>xLx) zb%KZ!1ay90VaZ}ynis+7uhZ}=9xh&$wovr9w=&o1N}-2yy&DU~ zX-Cqy;$3-)BppLqdQ4G1M10FK;{E*^I*{pMd|N~xA8&Ln*7zha(>3n3w#DADB&K(x z8!CxxTZv;#fRR2P$Svhfm^97P$=c`w_eU_5mf;M(1Beh3@-X$Ik|S8;3SkK0VG-UM zG!9q^CmM3lVY9pnrfs9-*dGh6J#Gxi1jI4V!|*8fniU@A)eYhV&TsuCkS?BqbOVdJ z3}Yrsv2`PlZ!D@i7X^yaP$(fnKH!d~AXzSnL}Et`T&mn6z~M(|$3^-S3-l7<|GcgP z5wz85rGyE{KZv}c@Mwb|CG|oXhz#H1ro@X_=B1n;dMI6_R0U)mouhEIazJ-MR9ShA zB(ESXCe_XD$!moOykJy<6DN||v$N!qoz+>fO55TDcLg68CP2wiMN%Y}-;qS2yKvTq z1en3a&}GM`+vs+>ovjY{PAm}bMLZ@C3}gb-?!qzO3xmM8nABMM0e&u4W42L9WvJA7 zkrnyu^$Q-qhDHXdmm8q7i&hKtx{^<%+iWZbc%*A38=x?YtWge8!&p*@jk+B}M2B(( z@mfnplC~K!P`N28vBa9eKxuh+}NOTeC?X@e3-VaV(RF+4Fi3mQkwBd(=C)} zVEu3h%2>4HGir9^z()HS{lvN(QD38j-#+5|luq;U;ON80SdxCj|B--xMIR~lO8jI+ zI`G=Z1Jnb7o{3w@^rPs_Brb^$v*+p4r5E_QW;aE7lig&-9ZOp!UjV1rA}y}$VK2fu zW=XQRK4Ffkv9KyQjbcx2iE~s;EzFosATsAp6Vf7>Abyh^y1`$%=gswQlQ`F>Pp$VL z)YIV_o~veFs%YLpRT?UpVLV?{I0$>CEH!lS)-%~iJzJA{)|aK15s}ECQxaJ#i1jqG zY!Ui+NY~_(@kk&v>Byy1a(Tv&&%69}(TXCdRGTcS-y9d!g;==|kpxq$RXLhxk|yUU zMrN2v1V%zyT zF8Q&fP39Up`Hj|FE_)Trkgl6l3c3nd;j}xI}%&+f<_9GBFQsMvE=4C&N=5#~Z?x z$1Op<>x@3N(B_u-=tLuG|u2s}E zOt2@;N1?(Q;zke2C}WTKSO16=qIoJ+uS-AB^>I0dx;-kHTaogR=xZaZy)?bz;b7Hx z?PDHfIRz~u6S0C6dx^f1GHva%T|+W0-QnKRh3axIV{j}0$+-mzk<+%K~RDH%NZ zJIN#q(+h^1tU6FEG0b70=RroTo1~;*k5SjGORcc8zzgxiY&%9;1Gd?L6qX?XY3xNzCbk`C!GzQ*Uw9 z%quI!dsiLJ>qKS&jPO5GmkQjuP1?Wm?brfY0lSn#Vc8v9$toh&b=*=x!{Hu-HQ~YX zEkc2>ucIOi`{yx+3zb58hn0xlQrvG#Xm6>u(z?nivUF0!{(SBD-;JVmt)_B54IxkS z=^Pg_c*aV+a7l<6@ixjbs`_mWIeDkp+n`_3`xhBR96HZ9!6#h67$0!J%km#99?^%1CV(u2$9O#D0H&~v7 zm9_CuLFpLQ$nE09!H1;~Lcvgk;vJXc8WDX1f(1pI#&(jeT%-=dmQsRNsZ2DGM&CG`2X&D0Iv$pcDM7= zE7`@pTlj7usg_jKRc0m*Rh7=s3?V872oFD2W0w3drcjzBE?O-u(uj(cHGGD%(2G}~ z9#iyllZTh&yN!&zM{;kU1y*wVaxGM2?p=wFkBtP29vwOUdP7sE^(Q_v;PzjlZd_!^ z14V>wIIIOiQ_n6Y*yP_4+m`QnwNDp|E2S+j_WLaAl)2&SBn)ti@phV(kWFe2YE%Fz@Hdt%QTZRLN_7$x^O#@zQ@Z`+zUaWm3jD zx{sdBMHn2BWIkt(*TSwu{sS-Og*ow&!UyT)o}ZGVvWNww%)XBYUk)h1YMEY*M3@!P zXz5@H!XE{1z$H?~zkGqKpg^aQpYgim%U_G<{Ulq`u$(^(7>wM)aw%;V zeZNbL=U^92!!)pL3HJ);cQnnlEX@8`WjGO4Rvkmrv@bMkq&jJ3bggY{TWK9xF&#S} zSdkUYR$Y~VWM>sf32k`sijv=>COO3?Y}a&QqIf$R0Vb|kEp~-3%LsUA_?f2f!lU9U z3L23&6YJYLbyYJd$`v%ld1vQl(s=rDPj+Pido1CE*;2!GZw)In?K1=Rx2>*uv95W zpwKLu6gU)weuNaTxB@4MT^d~!7vUOAX>?g!0ofGvCH@=dQLHWY2gQjM5A(*+;!v{P zCyW9ipQZ)s0h%uINiMw%nA`%yV7?42I+6bk{2rr5%eCL=2M(Ee({On;ny+Td2pvdi zu~cxrs7%)?9%zbYDUPZSR6}8MuE`ji19YM69LGl?9((H{ROc3 zXNWMegD3Blt%4?NIt$wbiHEJOMZakBal(xK(JuBQO2HeWkaQB zXpZ^DrU}V@Y-dU!6|?z`W4@RzieIaRXpjJJp%nri837|><(v40BaF&<|LXNi>HW#e zms0NxiR}~!km>uHru$Am??XY|Udz1|NiO2qeaBL@VrM43cmtQF#M$5z)M}CDg-eTX zj+WtP3LK|l27^=3wY}lZd-YCk{_e*Z*)Eww@Npn7ie(rdW1?Y_Qv4^&t}|6sYcequ z>tj3^45VQYUJO~gydQq};dZn5_kWiL%Aq3tpdPA*BmMio!=NwC~P`d>oO+1D#17#Gq-@;FEYk9Rmft1~#I^rQo(*S?S zKE&@ab*K1Ed0gy?o8>sLcSk2L-@X~0ym|8;-o=UaM$|5DH0sOTTGi{zEH)?mpyOM0 zI~slzE;F6rwiOgfz34dB=Asr`jkKZ6);8m;Eg{GT5o_M?3Zmctpj&Vq;a}5%yKaPX zg)D)(@c${PP#T?tfFWoQc#dfQ4j*D(p(745HjEyPuo%f`#QKh4Xf^r3_!Iaa*@}2i zmZ#E>C>*fQh#^==MeU8l(r@lSKFW{Ihb4&C;Rj&UB4zPCb#ednNxVigx&HC>^L!7Y zMCm5d0hT|@%X)z3ET&a@4F9`|!Wnm1^Df7|j|c6Wg~?d8P3Vg+)OUmQ5~h&+gSoWc z%z|QPZm=hL4#WDL?)9r%wZ@T{f9HCf!T-2#ur!G}us?d|Aw>|Z1*@9qest~ntY3yo zagA!J!}_C?Spm;5B#LACtw^=#u}l97(}YrI5A)FZ(P`=hPYLa7i-{-_jPh)aY|{~7 zrhri%W`872K#J#h+7ao?^R1#j8nNnMXzo{g%5bqT{ZDP%SdJCx6&u_C$~(JaEz$Ck zE0qu$r(SG+jmP$CM|&BCqf5Z=Q8=H30it=vcJ{FLV7!_mg=KB4M<`onPt~@E&cmzG zg*RJ;>mFKRxu}`f{7lQq#E8I@1^iG~O>rt2CZK!rLoz{m=hLBtnNy|*6Yat^N>ap;*r&6yZXALDW7ZkD3@&K}IFO2;J(e3HSgVl06aDY$nLAvLr zcUWRJWRGGDSG=YW<5ESP0rtBnB>>o&A=~0Fe<B{xHU6eKQ)qPD!6^i&_4lZX@d#p#RrRhu3WBe z_Yo;3ABwr>V%I#0fud1^%Ojyvyrye!@ftzXe!5;XDyHPeiv>DY4p{wcu~?t#@#2j{ zpCWf5tq8b43G731h#^sY`k{dOQ@;rb2+Y-@8y^p%Y_wR6xJ9{O;dnM?-h2{x()Xhy zsm}C^RfJhpm$Y{smjjuCk8;C`AjLFxRt2XRl2aiu@>A)z<^W&#vuqed4fGhXH(?2q zEG*4|(Iq+XmR{q-J0oN}B3f15vhu5q9BfK?s6Ky73RNhpa#e3_omGbz_BFF>5-H}F z8u@U7lX*duP|s^qA8tmIZb569+l)=GN^c6s8Cu4wHEPkLJ`k}?_|{#gH#N=XdJ@w{ zrwS_HDnox8YSgfabfHF-m^f~2WcvLdL$xaQ-rJhOq{*#^5O&_c4X3^zxSr<7vT9Ez zUZ5zdLFQoFG?}^A+rj6Uo%r_Spt)Gnv_sX9e!zdxVobV;h!`5JVv!w%8WyT-J$459 zuME{ssjbu|N0$~&kNmD54`A}Bs>xh$IY0rMXj5%KqFf2K$w5Psggv0YRb|iGxA4F3 zB#e_r|MdAqx%eEjaHDSp?cQN9SdyNWN#&x&E#nDG2>lB>7HO0DY(KTU!98(2;-H`u zGy>AN$^%H*U`IXl5mj#VTZU!_m_NrUN&fqTa_wxy94ha%eW=*aL5+caSI;=*rG?4H zgDwYV=KJmZ@Z#{-FQW4kahk?R6`aUp4>n~^$aQ4SMy!5xZf~vJo>Co00C`k4Y4RW0 znFnYeQP_L7^!#&SqsoL~p$*dkpG%_%#gm_ltq9*{yL|`xagE`h6Fo~;RZX=5U-4yG zn}(KZ`P0Dhb01hYqIccLx{c{QlL7FVR^a?qY9K{!-X&U}|U)a(8VIR0Ol?!qZ z&3k>678NPPj?Z6>+f)vV>;*w-1Fq#xe<;D1+Wsr1w;ey&;C<~vS~odwIWO=mLxH8% zJ#Q+@hGVPdbmBUpwJk5$_^FPZVB;9b2UOJr24G7+us@8i%n5cu{r3!@g#r9H4Blx2@7fBFbj*NN;s9=N5E={pOWee zt&eW2bgT`{=|HmAhxG_m!*&#F>YA!%%Cf75-ZaoW$KIB`Z1_$$-m+l;WGvX66cr5m zSNY0pg`4M3S-d92#xVPQ>4(}s{CX-$j`};Qr@al2`vV34I<2ome+{MI)Af5#*YB`i za6mENc-TaU^mpIYiqC$Gzk_QZ{$q9eWAyox^b0KV-~KMO#O1afvL9*IH6<1y|!EmAG%|H+?eSXL_h^e<0%$P$&U zh?6`~ENnLB;RB0rP!TMi9>5ZZ#k_01+e#HjRy>s#{ee{*FOFC$c6(UQ6<%Lz=o0gw zdDq{`tXSCRw9gqDhS?1A)fq_Nf6oJ12IL8mo}GOE{ZWo_EGaKuVO9P$B`;VFFdR%O zRe~19pBtU1))h2tu7SjhhM~o~)R}2fG#aON3+2-YTNej2wh41t09YxT$KbnJD_b_J zmb!oMt7Wixv~OPBiYG0mJlhZ*!Z0q1m2AXP>LV3gik2iJNT$X1ZK*n~H4|MZdgq8f|{k zPQn=l8&M&SBk63pTx3VXA!rL{tBLFmj|64S_{iVk}@Il$ZY&D06XM zvEV2sw3hPMMHU)uWQC8c4JBN+vI3TY^mkTzGH8uyVBrkw3!2r&XvAQNVrF`{c^V&7N;?x*U-3Lyz^pkgJT}=xP8T$)4v3?N{F`%Q56iyH zsj8>PYF|tLpvpC@{tQejuP_9h7RxO$h6(@w!kbazFJ zoY$HWh(n3eK1>Yc|9=xueE?8aXPqYU? zW12`C-g!z=*0WbXy?k-=xg8zdEb#Zkj_NXfx+6f`ETZCW&wjIW^p4McgE@bfGrwh2 z@P}K)We>^`)`LdO^)BJuF0k;!Eh0Auo5D2jGDz!!J_lVQJU^`=@RH42)g4M2t2YCG zskU_KQP+5?g$`WW&ZIR2zNCm@u`>`bPNlNgl`Z9x?B$&UM_*n<+1YAnYi?)+zNYI_ z!?K*I9-5{eT9zA3RL9o>F*@Ufz^}1b4bylyAN+GP2N^mFmS;~t9;mvdd_0iO=&ZnR zQ+MH~H^aXPo#9uO`tYkE6H5R6zyH4v{tUZ^1)n~WDFVxe%&~%8G^fILQZHMPZpuqc zi4<}&Mo+?7aw%F9XP^;esCdPpsG>)Pn~9;=hN=2DvT@)5B8Od~;x?l6-4m1xp0L_B z*m2_gVo;Kzc^#MmFIK~0`-xS5E7htyqFH@9lAc$^Op;oKnB6L@GP}+HgOwJAq!Pv# zQJTb5_z6Dv=OA2S{o234pImnM%X_B3ELK#c>(AH>@h>3BoRKU|$<-qm1gg3#mkkXX|iI-ru5VMcT(@O zV~s2Ww)wBmpJSn-R=F77?eUjYRR#fhpswhr`oGv@kf zqTrms-X-xHP?WIt4QPqO+3X@$c_fgo2ez!qx`g#A3~6dQ6U7SrP@THIYTAnJn^Q+| zrh%epAs$6wFH;+_`1xyK0qdtFZs@YE$|`{IQ#A3{uE&vdI)nQIn2CwI+Ho`AUcE<8 zM90k(Pq$72t6ac}D?h;E1&Z%X)G4SADE(u3H!8{1m@^N%d;ZIdXRMeWoICcU_!K_% z^Y9s{qHmtR-tY|WfvIb%+8y|}?1;Q9AF%6JfZl1}1+4m^!XoHchAN5=rPp36Nrr^A zOb#WW3z~kU*nQ~(A~r+LPg2h^29t;iOCovf=igX>>Wz|f{To1UR-Xr!>>$o-#^4m$ zQsfWVu^FJhZ$P-PB`VM+45hzN4P;zms0UjILnn&?ZcZ=j zZ+J!55a$mihWqfPAV52Ultur-j(O|Ek71#S6KvX^H4Hvb_EVG)?fP={|D&s z(i^NZwsFo99HiwE*779?cwS7C-#lsYp9Zw}P0SkPu$QQ&ar>KaM%|C`-@`aaQt8!; z>xKszb5#+bxU&ai=|>>#p|_`8e#g=r%{G0{u{<|4Jv~sgiEe4OtE&D1uQ)_$viRcg zESZPHKmGFV<;hTv;%S)9y=5AG9{9;3nkCEOWs;s_8K10FVsVigj;b6`6)-L~&?K<3 z5D;TlyjN9t6+{)Q6rrGoP5QWp3U(PAzTvh0vDCe?5VoTg1(d1A2TlYH`t8cZ<1*Dp{0`tn6%-~YI2AABiS(5fOtv8;dk$D5PCy#sRja`*}{{54_t zbB^HQg*O{w-VrdAp_k6HAxHhug{qCtt|n;|V4;B_c9k76)E}-dM>ah9q@%pB`*Y}b1M^7!UHdbA`5f-}3MkrcQKUyr$g~%ZOrSy90ep~;s@yYtlb^qRU z__GmdTq9AAZT&dizaah%5`UD19;_DoW*4IH z-3KZSjJ@NT+m2Q60tpB$<302y^ebiB&{m(V(y8Z%MIE;#9K?|O(%Ri3L@>H_qHXCB zR*shv4R3hi&`i3%Rqd0kr7sHkxO#d2)X5c9nj;X#^%$*jrQ>~{fO#dF&iYrP;(y+F zw%N>JL>+57 z$BM&7tVcnuiF20s-;u(Y{H;k9X{c=h6^QrpGP(6_;I*t!n>>`l3#z;>S`}SUlmzpk zVK%H(j@QUadmJPbBf%jkbxtUSBeCOErAV+F!&R0JY0rkBuMU~X3RUP~05A5?}tVp0a@5Bzkm>trS3K01VbG<%BS z7rnA&H$SF5gdmSr$?2_kiYUC~CE9ajgdOFje+n{N6dva;npd=oECDIYKe`3?Rfwa%{_s>0(h-x(*CD!+~j+NPxMmrAc7&}dN{O_dHy^A67>{5xc0~1YsE$034)U6*0->H|yBkhG zelgp@n5=s0upGlaL_t;%_pkGX>B8CFdLUmWZ7919>NVc2OgOLkUR{@BR@GhJ*pGz_ zH&@&rB?B|0Di@e~8t!!;^1$`-1iK%Q^?Y&TQUgak4M!~N2|&fzt|ZOciA{aA z(9}eBB>BpN#bNZ3TKF(^`9mIO_URU33br$FwA_=;1d0JEpcLmter~dTGFEIz&rXII z8_L6MiDDnqEfhivp>vll@viNGMm^d9JE(g9tQ_hH!d68tt;(b%hLR{!Ud-Eajw4OA z9a_?g6xOMCG}A8EtCa$B9t~dfoTnhdqA5sO{oESBwM^y6F!X_8FyK09$?b3y6%(Ii zMa@4JYg{z8YDE2tme$m7H(4~Gs>;UKFrgzuu?L1~5hTruD-}@i)S>vwOMsDRSxKP9 z*%4jqq!RS{EKq|o6u4EOrYuAml?vn9LbjUlPnV~~;Lk3@aB=;PZWdII&=KUPq z9llF>bpD!iwHX7~O8`jZ2AZxBB2~qXBh@tra66~aZ5BONjOaA{JS4;3XI_~oKkg$%gk>GlaT{4NUze%9ue1ez8BZFc|$dlJfEiDwn8X}d*yLcl!)SZ32 zu3`!AvbmWq!9(d>@JVj&JZ^E$P#O>4DW0H*U1x>liPm_p>U`-}0cxXc){w|s1tyXr zvSqrWAf>jclyWP_8*0Ifd51!P3mx`hF6IzJtjxUBvYSZg63*c zsWXx2x+)TFS*O=ndj9H2x~^v{lhs=LwMy&FyQMv;Ht=fm3`?w5^OiCq-@L+>6;1@vgHdQ35}m zl1*JZ$v6M=oA<1pQ*duV*XCoJC$??dwr$(CZQHhO=S2VH#I{eIoFtR?`=(~5rsi_0 zo~!P?_eIz4-m6!y=eNTg`ter7Gx%SWV1**_dD0h#$cu>?G{=vQwIu0ka&!q0)FPEb zI;`R&NMFTO#s@+|&33*!9XF_(<2Svb;>hS1V$@$USjV7!J(G)G`ZMa9#g0hA_w4bV znR3{W&>-Teyb5aT2S}zt_+Ff|fT_WNg)@c;m=hVpbWby<%W7w#_KqM4Ncs|f9U5dZfq^-md!O?P%l&lo@lDxc4f1V$e6FDNAB* z=nmS?hA}1Bbs5fPnQ{^;b!{-A+DbAO0g}B{Gft^$rS_}m=1aiyE`SH79v_{3=fb_X zmk=*o>*S|#g$0d4@z$hqMOE?qydb%O)0?6 zC_hG#`eB5{=OH;o$^gfDk)1`MHqCcNtC!ic+&MdzMM(|wx4>OO&@HtO!CJ~6MEG{luvPIRSho>RwKp+CM+sd!~iakDvgYTdRr z9SIg38}@!B@Mh<(xQ$|WmBZMlm<}8fSRDuy%s@0%oRsvOBxBH}CRRit-f>I(B3#iK z6JvK!6gyvfPv$OQ5uUKBl3^j|kyuBGI^)&kv@GL9AvS&)H|5;$H+4Q1C}m@Z?ZCg1QScVe}ina?$z9mA+)&GB=`(MC_y z+*4;B;;$x2~KXt9dDcMZBoDCdk3?u7E zSzW)C=a@deGFg<+UMuu`f`=j&+znJ<9L2jU#vv|+*Q>n}|au&f1m8YVad;bzkI4Lsm;+?_@-FML$4WKa4 zN;f(!ns*A_ePM~u13KH-=i2fS%?Wrn;_^~@3>KE87Sm-bDiNdedt4ECWb-Hyv7|kR z8jl<5@m|g#>;=~v) z_VBicZe0V;k}uZ+Z--wcsb?KJ-_Z}X@hDslwe8{O;BBug198#He-@39<`DpKE^$1@ z3DD|sE5=Wf`EZAnIR2c*G}gZD?of|45=c8yHB1f3sg61f3CnT<Vuc zGT&F)Sti*gX;{cr0_))7h$+BSdAPp=Ts$HliY)^(*^>CjQt&P(CdBuMl`Ja`Ymrkp zl`pUCwaU3}nfM^ri@~P<+x#~xP@2^tWpE2z{JeRtoL+oAX!NE{)+fZ&W2cDD786I- zff!1QuR|EIhQFvZktB!OEXE&L?mB7YK_zl2NAQveeHd)2T@d}7&?wYHyu~4=H+{^^ zp~Twe^)yNI{eB%*OC}Xoz1YNBR_LPCS(2sUzi?$%k6MbO5Sa8mnHHocw}WG{2NRc= z%2d+cmZ<2qzn`lg}|yBxqa8C^?)NVCYa2mnT8(&Ry9bi;;D5 zHS@K)40A4S^4QPo<7*qGp+Wo9IHh112goEm7&LONSATYT0s7ekYT*!7lXeB|{pKVI~SFckVw zCT|E~0x4$MelA*S%hHG385rLN(t;%Ilw?*pCm7XQ))UTfNbo#l7x9lwWxD4kl;6|@ zM_H33@n~paADcr^-~*(1b;kQ7L6?KFLWlytqGB76v5gQK%@b}6PiTC|VwyBTjspT( z$nhXk&xC>YJj7NNr&56=5EIu1v~{e1ZT@)PsZqKn*2WzC1TdHfjl}R90Qi5a-Ju$ah-oM}sie+Uv^)jq*meWca#IaWGqQgL(!Nh}!?zUY4Z5w4mz2p{M(87ge1juH9(}qtfP+b^lPIhxm66wmt4&;}3 zpVExmDk{-wH$S&+7h*_ZegTLZ^rmdX$)=}-vYOv-WJNKY#3T1f+c=~nG?IxQCOH<= zq9##QO*!Vso|N6`SBw7Sl4~Sa!d0jS(U-wHo59$jtRQWK!(PP%Nd(d zh?UmvCtoo#P>w7^5@RFqynIbf>)A@71--N_joSFpuxUSA(~ysb1kvdu{T%IXr1cOT zcBe9E#@QZIBQ-JZeaL=ND-`QMS(72v>L9f{dTw!I_}pc*J96;arBzylf3|{y_JhwM z9kB&n&OWP73Jcb#XCkVo3z(kgQK~Bk8_|w8gPA@hmthJ^8~KPX z0Y8_IQzCe0ouaL6ogg3bU@eyFk`kGjAioE8_=_;^M^zAo!I%nT(V*fsE3$FtIPW(6 zIQqh(ynYfyuvie~k8LKmSlz7>_js@`1&y|A%cQG`2|4Cs>Xi`MoLB%$%oy%ABnQhX zI=a!EBjetfzukk2Fqi&N*pLqicimZsZ9A&gCr#rvob^`6qD7tN@!@T4-CPwm)zwe1 z4jfGo3)5k~6I;Dc)OW;=p~di3r2^=5mfScnm|YjU+OmXo6+!fOw(Vq)#YjJ&n1& zDAtWJ%abmNXJ^7$#rvg!X~?KTB{rms*Toj(gR7s#QYXC2dO3S1Et4;e*Q8ETYTnw} zDZ~MOH}K+0W(bGQR}Y=SO7>}Ib7fG1Z+4V20D(FGsaF;Hk7}C#I6K%-oMV8rK_S^fKEXfn~Bz3d|KbWm|dcPQQhqm8mwed~M zs)e9^^_jsUxrushL3J{vTj4msj>0T1ioAyn!p&h~%Z<=lJR!*9{kv{rBeAP2@xp?x zB&*R2dhU~PJz~@Qg~VbHvz0z2m+RQVpSBCO;2}HAt1G$lWiW{Cn=5JleD>x*XJ}OE z!<(HHf*-cWp9jMt511z9luD-NuJqwFAe-6l7fD3ckOrOz597)(8M~QzeJPblE;nMs z=J}xg%_Vlgspz8ZGjnRHyebKn@<^X@FKLW5v#&tgeq2fqmOnOm@*GBXr?Ay%y|SNx zTbCB|BAknOOX%j%n|HIt(G)>umzcQMr~oPsD# z&V3vDSMp{}Z&m}|fGZ{pyN1d@%>39AZ@@L3Za415MbCuPM}L8JiXPbx;c2h znv~;Sh$kb!I&S=I;>ssc`;CS%J~lnpwaq2f{ zcS$i)Nc5VVDD5gm+*R<;)LtLS?riYs>yQW7eeB;{>lR8FvZV>)hDm|>>vNTxMsjp$ z^0-z?Q)-iaaDDP&>=BGYGD+~n37IbJ*nDM%vhJiyVXnXi#SKI_%(W0aY8e3M3OXE; zItS4u+u~bb4E}OVqs5TERJ;+=zRP zRc9C{|8;ta4^EB~S6AF{r8F6tvrls+#UPYi1Q_s24m)Z0Q@6!D-a7dz1~g+~xmHiR zpIH3jwq{a5W8Nhxt?#d1%kvZ24rYU6a%s;7mdn}S{QZczDOOE{tk5{ZLmJ?`BHDZ7 z$hI<+SKSx}i_=&VGa(EGFa&==I_cAzAijQAZz*``3H72zP^lg0&ke_}$<1emKjR|gKAa5#R zHN6}2cQPAqf_Mr{M*@nxZOulZzWlJ71R7(141S#qBlCLaF%b(;sR=F+!Eg%NHdvEp;HOT1mj2Fk_W}n-^6j1Pi`giTs93}qX+w9 z(-O$W8p)FE(!_kxB?mw0(?~!x27!MzN4F4qHHImU{deLl0tfY`*QMHlY@ootk-lEA#Tp{Y)uQ-~ z|4xTqx_+dpj{bOvIl*1o&;d92{GuNkZy+iDJ=y>*`j>ZlMN*?X!R%*3LSYb1X)?V& zesaDG`d`=DXkq++{|$0{bRK4qvS) zOBIE0g2WR4VywK>fVhx7DK7H9;C2CWyr20Rf6FD}Vk%kY?M|O7JXbg2gA|L zijA#4)f{H+UFx~=fD|aQ3qM9Ati_^!zS%!F1Y4nbX?}5cjg$~am1xq zx>VJum_+ZbC%RDuZ9;eJSN@cCMrf@SLa|gvlOy5`&G*)9-(&k;+e=xAM$8_p{BRZ|0B<}Iohk0by^qH>fG$~hft|7FXZm$OvW|i25(;cK`R5ZVV>oIqEE?2KsA`G07eUv-y{AH*QfeFlGRl8gfJNFO9D# z{60APf6vFTIVeg*d!Q15q#-X2`U9yu1--shPxDB(74>FW{S8*#w~r5jy=K@4?|;w4 zibU=UqfJ0O9i>1h|7A%r>@X?>(w>!~)`~SOH-gsAFoVea5%X`Il{|D0ft6xk{wD&5Pn9yPeKq@s~b!^OS zh%86z$tYuv<~|1k62z_ahc^%b)1QYBRp1{ws7%PE@fI&F9D%s+L`+e^LI-6q?Cy9a zYcAD4KC^wjyMrJoL>@sksa8+Iw9`O?BO&7hHe7S8u-B6z1?({P!@KHh&Gh1$Y+{;h zql;*vI606)#6{W&h(>a$-&BNGyEKc&kZO*M7dE8AbH9Ggu=H~9%AD-qDGc=bcrmAz zcw(2kAWjaR1{Kg2Yc=CW-?t>+v`Zq7hhPt`XGM^tTNqizo9FbOYUbl)=i^guh<31U zs;Lb?SJCsN7im?G6Y7B| z1{UaHS?}4FE-AETXq06msn)qsTU~u!d>212W?C_+P6v9k-1c>kk>q<_xLI#=hQ7dt z6mE~mO?AWC#eC-32H_=Fb!+9vX{WGy`TPs1OTR5(*is+?Np2)4Bfnnw97XHzgGQG` zu)L^?5UrM)?pPD&62q_d;V*@$AscAekb>DX^8uiOg!~yY_lP$EaE1bulD45z$`fb2^$VFCAUYO zdhL2~>ff(3U4B?%qk*nJM^cKLsMZcSViinC0>r~R#Mnvj!3>OF=F)%9-~fhBSYW-! zk;H^==mh*}28^TN$&If-oTWoIgf-Q@awl$&w1ja(CK#mt$H4a;&;f_;@s2zX6#Teb z?j+EM_S@g5>^SQ`kqM3zB$z4;x!dwSn_fPBVgpM1*$-Eh=1+xIYoS{SuW$&Pl=S^a zno9Opq~Jch^t7RB&iK!Hd8skh~mz)uL2X$zp3^PXibs=yQwQy3i zfjz2`9|q46dw)H1;6c4B5TN%axvQV#f;AoVQ1%10#>+DS;xo~W#v(ZaMcuh&$ak&^ z!Q0eyoLD7x51;WnZprO^fU$9uAhtl7++O7I#RV@x-p0b9d1j;6+yH0PJ`< zcibJUq>+{Q%ly;c7?w~arrgtldI&S&ftxjJBYKk=oDSvaX^K?{jdeQR)z}{RdzWGR z@q66$<7_iL@HcG9J{*Sd04BT7dC(sHxif?+;Dc3y09yw&??RH&bSs<<0)#5v{3b&~ zz?61#IZnKd&WllVyDd$0)t?*v()Act50%m?P09`=8(FzkwlP*b3rxtQoT7(iRjs!R z4oe@AjS5pKx;F8W>;s$P_g(YC|VA%T5~K9p_$wH zT&LCUl0PrPlx>2^3v%e@9}x=e*_BIHC*jpQO0$$TA9GeX@n4;j>2=fTbv4yC{3sD1 zAaL1L#@q?RMS^=5@#O*#5g@=e=i%1;t`Q(Y7nRn-Zy zbS;bGX4cU*y!4%TVY;F1E)lOwZN>Q|(}Oy!FNhZOYrj=f^k&-i#=Ijf0SL|XI^po6 zDgmm5Io&Nd0rtyrs!wNX1Ga(oRyfgC|BwsZft?>{-sYwA=PxYiw47YUTnG@oyQ;6S zqwNH!1LeRIRz88dP!$O^hY}{pj5@I;)zj;_6QBazv|LzaIsg;k%_dzNBg|*DVE{HZ z5f^;$+U{cO^pDAZ)bwXVK)pu1!E_K&)2wVN9R!fI>D9Yo3o{O&HFfLV?~Jp_HbJd! zVoe0xvGwJA%*&}ZGxaBfMt-)AJk8~s@gl1xht5Qn(Gb*$Hr?HClY8u@V@h@aXMD`9YGR#m8=i4DM_8x<^Mxs~Uj}d%i>95O z+WBNn^fenuK85fY`2h%o%^uv3p64j7#Z}zl4rdo_z!M&jkC&4eAyaIkH_wMcCNisR zYwB*NO}}bbs|zj5t+z8pd?rN4Ms`1ZC^OFbyw55RaK2;{IMu9Md{{B^K)Xx&Qo#U~&ly)U&dGA8 zjL~WvGoj4ZnzMN40_9*yrMP<$#cPZkf{36aJ0PMmnH@>n9_6AnbH%hJWe zE7sTLRs1+<2O8$c&5Og}HTP5p(kfSBp3IbRaEL33aYtJ{_olvuLnmAAK)=H4?GWYa z>=F*EM38q{hMQwE#^!24c11yg}YdLg57_pi*ZDQH4M&Wt|4%JXS)VLhzU#P^rAWOVu=AbH*QURdCrf7X@FL^8Xtt zp((G;DwOJenTl_T#8`bMsKa?!h3?ye?IKt1q9--tUfnL+9VL>|A9Wqdrzu#bq#Otz~>4)WF#-p$ugxL)5t5GcYDZ8q(VWG3oQ2tY=R1DZ%<9E|g*L|Joq+!8a`WUM{d{h%t z$Zs?10|#6cHG}cgcn(A2DJNA!5Ip# z;gaXDg(bHZG`vo(2y>bPYx0ZyGW&i0fbuIb1X%D_yPZ|{o1K~ENUGZQ3f_~=(enxp zDto^z_w!+8O#@DP?Pj})K;dl6+@P`Pg9zf2S>VO$JavSx%@D$eOjzAQ)g}vdq z`88s}J6jv)s{Agi>Kx;Mn>3mKhAZxza$dvRxwFIO#i`r6lI%oltIup*lTKBUWsfy| zr_K?Md@t>&{Ot0XD^DcrGTSYw=yF&K9%=X!l(V(#auOb;G_t*PUDSRKbUgsnhQ{nT_GZ z(&QxE+@64K^NTVw)hjZypB-ryu&aqp$ZN`R5OF2vR%j)9bS&2aX%Rqy@76XRV^Q8L zK)7OUc|LG~HI&g-I|Mi&mOzc9(FdF_HR0|Qovva^c^@Y!9IBv{Ne7kbP68G7T<;>k zdstLu+R>0URbTYO{~r9)TT^74>sKh9PNxGj!CmstttYX~IqSV3mtut^Qrg5@Rh^7* zP-A7&s}O;T_~15Lhud#=>`>(aEeo0VgQg|&jFsV}_L@|6dr^p}am($t0ZT~ z9N%$X3a{D)H!96!Q%V-<)r41U>`ZRxeXGssG%_T3=P^u$5%75!qwUT2)~UAbiZ)xn z(Q9}a)rHL?Y(`g8Yx7Zkm}_J_(`&Jv@Y+iX3O5^pR@zJagV+rP9kQ%^BZ63^1A7Qm zQ*YT_vb#LYjA!HuC)URL5Hw=By=h#T7Y73N17_-}I$4lSy)Qxb?|svw#W38=HWEOY zsICiUuA$C0;ll{T(_%x@X_=D%X)IAfUi(l4rG{4!+LvkQ9yyt;kAq;lESemLO8W6| zJR|}+Z%-KBxyr4#qL2BLCTh2r@uARLQCHV;M%#716cRGXW0W+s@hI*DIJHTp1cZMn z>yxX2&TVAzl$&h_#kX)|cGc{FTTHx1IS4x7&9_)>ra#oN9LTAq3C(icv^E*hghLjm zUvbt~^HT;I#>9SD2I!s&3Kr_p7M%Zj0jqAt*k0eTtJcI{QF8lia0{cHq(7h9bdDsM)|7Z+o(Hc6TZBW}w%R_aO|H!PUMG5WDR0Q{A1Q>562-7o$4EeVZ zHjK_S@v#5|DrfoK{&G;j$yf!gebQAoD8oGc2%0*TetG@??`vp5P?GxVb_ge2h^=aD zCY=%A=#iPbU#OmQ|3fe=;h}yuD&&SJ?kLl1nr-X!YblNDA-TZlcw|720M!>ALGd_? zTaCwNm@?XQw}Y*~OkB%wfKvdFjPb4s6ydxOmWbVim=bKICTw^sM~$n*+wEfEvIpP9NCnfwcuLn~-A< zXkUqka!6a8+Mjy(T4Aw~9T7;BOJki)Mo~Q#G(aEKkdmHU-zFI3A0yVdf6?6qsJJQv z`FpMvNCx*4>aBJ(!yT|Y`u{3ZaN{}5c->fa*l{-((EE6FR?y_u>T-;$eqwlnptAMp ziTdpvWLK1YFH9vI4pr^MmZiaynxeD#dBK-8_9-130^0BG`yM^b;*y@`W6kKX!*V|c zgkm85LJK755LV#g*jpPpRW&sg08S>%%kW$1NT2}o#_w)URpi~7OeG?d0!cVY1uGbP z*zual1r8&_e^o_4f*=u~LHK{YPXT{VX(Qcx!O_DR9=jsG{>7aHx{?%c(W-53L6JvZ zCA+_>E|gu@;><}~L{RVuyR03vLIVlQS3!re;a1cEjbZPr5|zLvCX*!>nmXKTTnCN% z*_Pgk#{O*);`GWAmqASQ4Gl&adODYp_ZYzMq9egBx=9m^x{JWy)IC zV!k)Gi>%DfP7HMji%c)Mxpz#6;KHxgIVd`@EcfwYZ3XaQ-z+u?irC4pvZ_uXk#XBc zL}oWjjPenmL{>UPMV42s_qH!ygNcf9yhL_NjN&54(IGE7+io9ULpX#*#arJZJ0(PN z5ohZV7l*{lo(5(3j6$R|?^9Qo3;Y|m0;@efTjhf?8t{`a3S3P`*Kl$BG+GPZ5A5cN z+3Ee*w9aa`wi+L?zmQDq4llxvc6U%_F|J3JGFU8SVB0 z`WsG7qp`fv?()T`zPdK2y-0gQr2a(Akvsdwvi#w0e+7MR@#-VcH@lWAy>5M+&zPl~ z{<7WtV%6^;uA7MFPts5;K5z?=VbCkK2esG7J08gXlC#`$pMr~fpqFO7%?_;m>14T0 z);nM2g4YZTFHjB5_xH8mr9Ih1(n$?26l`rAUK;aZHrF_ajpb=X(;=1+PsnEzBeq*w zr+qTKzA3B?@CVXE=mRN^mY6$7O|zwX^SGO~HLQKxRzHx1H&Q_ROd39a z3AN8xR)Azeu)aT5#;OE&O&pNV)fJuqmdfLhA53AaZWxBz0YfnID1Uu}C|zv$tBV3> zLID~Hk{?V*V$I%l!5tOYzf$xn&^NH!M;g@rLy9!Pe^!!Y#}h!uDxR7y{zrtlYbEuw zLHxY+gb?qd_?dZF0^yDbI>!3XquHhuLZ`z);w@pwxY4LEHp^%w%-Jw@3Gi zg>n_B&atkFw1x`Qytu7){gh)d!c(fe)T;%Mn9^MqlU z@>@R3!#?A~o_!8i{=~%kx}vubk06X-5gl0Ie6NoW47_!Zjc`#$=$-UgKJIL|jVGD1Grdl93Gk`mw+~p48FihRIJ0A9hPTw33 z1cVCfhd&sD%jy$6`}Nu1VF(=d{^qen(BzTd_Ck_7pkmOx4G%k86ZRaAyu z!F9CkZ9Z;EG#b7_BS9eJ#bh+SkzXf}941igrr(W%-CmmmtWorG<`vPM6}z7O+y0~_ z$4fH7j!0jcBPkKC_=}jF6nOSGRG{eF%0b5c6lfPqCgMha@glxKO(F8tRlp6f04cWs zFzJ_>rRsYpQLfPMlW{#@Fsp$aFr@%r`lt4Ro`9F+P$O@1vQvQ|zn6wfi0~CV0=PzV zt~8BA>R?#SNLA1R4&|{7LcnLIrUWqK*&*>zh8WDAbOHhTe76LNE&o?9EkEHJCUlUo zm=LbLG$Ba#2UV)bsN*WYpo7%L->hNQ=lhO0h*8Zppn6v|mVoOudIe-LuZ8oVCg#2& zszi$h+?{?Crk5B#%kwNc`Lfo+uEdoH&bNq>J-_EQy13WC>s`5}G+ako=T&sCHb+^v ziJ15I_!qI4=mW}01CxAz_az0rqt6iPv}*SdH=X;UA4^u$xd-dnn`#A2=_0#6Z#2B*1nuC1<{gucT zUt?jy?w64-%`6`{_bFi+?QH|S%94%uu`BxvhXZ-N=NQa#4*b*D=10z#h~DkJ))G@G zMOQ#iXMV@+yZcipe!^`D9@a}nMwHgqC#C|1kjIPNZk|87d;;_9T-D|7w%@k_q)+TJ zBIeWi$(LQg$K~>mu`qz=^Hu`mW4YiQv0^s??va9_0uk=5BBrLt=0meWU?ZnD;P=HN z!|qE(gu%v!#(U1@@)O{EzW;r$J- zZ54ZcJtwcR=0UN1;^Ra9VY~do-+Ng=Z$FDs+VIbkptZmA-<{8rPPxW&Ks$lJ=kv?= zXHCHSiGhN__swH_|Hbk|&8>XLCvy_JKF{+%MK0f7^LppM(>&?SDI6XEM<=xbzX#8h zeBGAE&oMR`1KzU7BfWs9mx}Mlfv>mC?|XrPx9hd`6L*h}PmE3yG z&<0*#lHb?UnSFD2raA_$x$JQV}(0xrX@R`Q90RUu@y+2_J#N|)0 z?p6ECZgiL*UGMZ;iSPRc1ZX9MUyFD@L-KBKq_LuRe(u!YeD>VhUxj;Axay?1%9DT1 z>pS|qt1ShppnRPVvIbDodnm=csP2Y5WZ&IV*YhmbzxI9@3b*p7bZHO?;e`o8X)<(-Dq$6~VQ4NE{5 z=Q!C9_eA6{?JQ2fEtvEjo!m72Vfp%_AP}cnhVRg-7j_!33#s>2IQ(r#J28iUTsH89 zj49h_gq8!`a6U%PiaIiJXXQKe2}l7o$o;b|RxrOG6Oc=OwIt!gD0eyT(2qN7JTUFp z&`p(~w!6I@i_$Xim~nf9zo4kEfZuNAZkVK5yPn?4PA5$C;q85g9+P!c)BVfqhHA6%Y+u z(G~awN{2z{1qSeGyqU4|r$chET`U7&+U*SN!V|w)NPeLbR%&C;ePbP=PnDdbaI0O#XCT zDsSTWP^h#FyPN6Z?ZTs7IEc=dBujWpqa_(mIp-+)on$CmS1@@Ddx~tL2#_=p{YVP+ zhTP)F^R6_J7p4q9m;1MWMS67Dsd`0%{I(V)=~xoEa&)iig=QizK8p_9qckTIYAV2O z`MBgVqQwT225!Lm%!yA#%CRw4i#?Wwy?<#6cF4a@C!DMqMj`z7dA+DOH&NJlC`g&e zYnA?ha61t|snxgH9n zSCE1nqQw(hI59~XtkdGoYHr}JS_%mfZg+xE02O88icT9!YPu?KdgVxiyrhT<&_ivl z*Hf1)GDgR5ctDOuGXjta73!6l-jLb?$bzym2cu4>>5kE;AE!F zgpDS-xOcY#?KQYc+a^t_%Q639M^$D$Y>!1qqY+f1ZS%y(SsFL=L?xvmtrise$7Bx; z$55xu+tHB9%e6&KF5;pv7J4$mvL}?X3pA*Fx94Zqy@KO!8(E9{?)G7=5}R`$0#4m> zrHD3rbY0v)OpGxCxfbf}$kwxAW~U!WZGC{pEiU~mZ=;;eeC@Nr8>K@+zDhlxN=5!R zsVs9gj0(G#GVDM^fN}-lXUk+GU9UidkIEcW&`hlbw6ElmhKf~MdcEtO>LtvJ{HofUnl!|ST97_X z=?!{#5ngZ8%-qZq2%&JmM9u)V%7!(SX}F`57Ho@ozduAHB`dy zj+?e|*4!0gcw6xP`;dw7cS8vzRA-C9(jugNtB1lT3W?cv@*4x;vnnMS0Q+O%bz_Wn z9A{fUF(j0fzut>1G>_%Li3Y(!z(ooKqtg^^3JoDDnh!;24e-ayo-hYl0f_n9hMT-H z?nwPIK=7j+pKRtCqP5D?JT^1BzuLB*vu?7HgCVugjy!~Z2zZH z>Tajfs3y@deh8W?#l5jaYFX)*3<(+h)+o0l1<~e3qHPtCuZL{8n$h;3ri30J(gDC$ zn3nsbV>~Tive3u06{CY1Hs#X_D}DR0p1w2$K0F_ODx!j}h>db`BLk2|0yq?|1r@Jz zFRgZ8M1oS_R^BK%Nr@tSEIz!hEM|Qjb5>O6Gl|Xn;8QLSX%=BD*b$C@ z<$q&)dvFpMHC$Vl26t;(*eNx&qwiOx6OHgue|)A_z~(WPi~Z%|nb<`#P%a(m<3sjQ zs8fb^s>RDj+Ry4~X;hIyrK7xZ%=W9@2*PrsmYyucI%rhLOgo0j6+pG$50N03(GDYJ z&IbE@;apvsw#LcEkwL_}%FfQmriT%k0Kq1U@e!!cyS%T~LF#WoIuG0)Ewxo_wgfqF zWdj}zqi5|&d+pVj*rpU?$Fo~EITai@37N&44j5P8>H?ZHa+wKv%}OOL!(tX@hBD{J zkS5yMn~JDUAQ~s4AyrX_)sD~B`HR2m)VayCkKL}GA!@EH{J>9tm>H?;3>32;67$?v z35w{K1Ma0MwejYoNax2#)&EP^`$;zTC-23oz{fdRrX#hiI;*Nlzf{(KIYlT6%V+Ku zMf1@$b%k3bBWXW(P*su#D@$Zwuh-d?%gAh@EQ+D61@}WWK4yzsk>fF3c4n`YLp6V1 zc2;B&h~!_U=yxG=^chw&2%SJla)BK*Y`1gvnoFpMq zjKnNNEhLLK1fj8hnylJlPe{2jZ8U7_^h2XVE9s`nfs5R5U{`FXF}ezLW9#o4tP5Zy zP|rDX{N&dnaE_JfG(ec0MS(7s(jy?sYu9X(*Zq*|8}iTyJ+f3&^X3XksjOHBQPxw?_VU zFeUePcOxE%0*!RWlTcFhl+XiL&L{`gXwb;@X=O%^vy~YkB;I@3g3WB9d$*0_j+La{PsdGihx-DWq(@(FoI|- zuu|~}I?dY9bR%`kA(Sdted?gv1kZRCaXO%6xgL=!)-qZU(#X_OL9$Fg1^6~9 zDfBC!jl?CENCdBh9iphQI$4qHYYgBc{~E?TC9_SUnibofFLas4Z9EI>;pKFCYz1?Y zudzb}C@%$j^;-kEkR~*KTiM$a6lY~hrZE#TWUm><7iD%a)`dhD$;^6iQI+W+{WA)+ zxmr!?-0cZ~BC?(ER4D4y;?Ps4UFf3bY)+vn8^b9?h$Hg$d|lO*wpz;#BH)MNsr_B1 z@M!jNo-Fo+(%$hmVts+7$E+AVPeCk9k+xhMI=cvdXu(%qmVYQ>FF~1{U{!8so>z%X zPQQgoP*|^!TCR>TA=3BduU#YA?MSe}nK&FdK+g}|o(2J?Yo1)o3k{9J_frYI>DSe6S!<6c>&PX&g( zfsZ}e!6S>I=F2;IM0DXD4)tQr0JkY4)AE=&RBJ|n52BVN{3dFLdqK`BS7O&*!M3gy zf`}GRt+T-GRg}=lFihhp>*8(A%s4R1mnl`P>k2H5(_4BjGO}fpmnTo)0Bpl+sk&TJ z`!lP2?z~(AIxl!5B#6vasTRY;bHy8@Lsz^{A;jGJ=V7SI-C=m6^Rp)#gn4yMwGFCd zaJ^V!IOi@gs*z(0kiO}%@y71r_~XJ^@ZuKoVC z$oGQrWjv9MDk6JQha>89Hd4iUxb3h=jX^xc|CN1CzjTA$?t+^~rRnnmn ztluy-2Y#5}8okocBg*VvXSirf4;o4-Qvf>UJPI;$`wF}tdsc?5wIe-#n|qX%TAC$4 zyrS2Yfj*v5W`8`Tw;PhJ=fvE$G$kKH$RZP<_ zLkcdguxVKgv`orw> zn?Jx#Og|;4MeJbfz|t6}>)Vl6=amcWODuOrrwi$S94L;VSo$Lp&xP9yi=BZ>&}6DE znKV^(*7Vq$@zU4|c`E&jn)4}923i(HFb{38`ig}PWXDzod%#4D#`>p99Te8YIXY`I zl|k!>3Bbobx$m)W`WVDHMDD$krfhMJU8h3P;B`{T^?3gF6X1FV>Yg4|)-3jvpdeX6 zIJLPrJ4EL+X5I4Ld(24((H~J?TA42Zi84Vu<8%j^#H_%rce1-pcb;tty%LQ-A)s&N zRdd@~O!!DQ&?vDji>G2gT`8v-^VUZcitJWBYGGwnXSa=nT(s?Nxon=e>j!spRf}a4 z-CMD}D~ne-ERFpoGndnOxq5FuZ+=^Kg$xW9n5LOx(VOgj083dfse?kac#apDR0D)G zn37&n;{PGN`pAdo#97t$tb=&CuY_A<*+d<^i7D_^PjCDyDjXIsa3jv^m2|=49GtN4t8N%^r6%PWO zNd55v*NTZ;q7m}rL&&J`{ou7fFcg8?5`?*Br#R88r5IkLl7so+vd z^(vbk|K;qzrLQ(iD%d6_jI>Ga+D%Mq5zHIyE@#&FFmyYIW)}3h%8EY=SvXc1jgy{5W<(cZWrtPc7o1Zy%5K2YQJ13m=2U$jHI&(jr&peUOfQ`sp9!OydCSF&cOC0kLO za9Vy;WkBlW5Y)wesIS12Vb&UCCooDC9&;8_DOWE^-8;_(6^pL9ZxGd*MWP)0@KTB# zx%ov(y_BC!Tx<1Wwb299&tM$Ezi6FBlnGyO*AJ;m70y$uD6c{de*NoK;r+4omHP8! z4eCl>WV|{4O~yVc&Abb$dbZnj?;EA3A+5b)yrYby=y^@_V?M5fW%t9}9mn+{9@2cyi&QGB=&##vxN5!SVpiN{ zVrCNSZ&@FfAcrI|{pd&OYWCD?U!`K+TQFZNLb{;6sRngf4T)^$# z8YacUJU;Z{iP>;7lm*wc^SCV-s=ObQgEW|GOe$^qOnJC?#_6^aY{9kMr?pwscK>Ix zQ?}ervIi82GR(vtu%Wlo zgdHL2jNn&7uZn*OhwK%P`uhKm03HtE@mW@gZtu}0x1AX8X-^8+kt4v>eGny zRSA!3A6lQcK-Rqn%(h0gIv-s!l}W|8oR|W3`aBm$|s%ekRO zXm!QrS=fwn49Ej=vKln)yWk~N0i`DzsAivo4(SH?m=MH4d50L*o5OYI^;ocZrH?E~`N2bm zNSu8NmeP)c(wsO_pV}c~pJfB3w6w-KJgNprhL={SX@zz!)rQT=Bw@ z{sJa zK`pPdiF@5!oi>L{FyCXQhDkHpglagJ)d{mXL%{r!Z31XO{vO|QV>jRrn_H|6DpyjsX@fmSzip7_yA~Dxtr&yjBuN_j| zN3)tfE2wl4*ueThd2s`J@o&Z?;Au0#?qX}Lc+sWN6T(XEEQ`Y?Md?BSt(zv-m^WU7 zn3i4~2guhFMIQDZG_(KFcIQO1a-cIzzJ%oFnJPO*qT(Gtirx)_h-R4#$3zFasCZNl z#Cp${A?Tf3^VIq6Ir66{!mFL!G7(|_uOcB7=g1Li6BG$)ozSmZ+4YE~Y@m2BMdhEFN7V0Wym)MT6G?@)LC=vTi zeXyLl+U3!N-zX&S+zg%DL%9qCgr>p}dFM;@X+Y7sCg?P~#&*(Ivwr@dyb(ULA=QcO zOKqrrd!zrnZDl;_%<#HKt&r`IikZF|s5|ZqF?3lsmHM0{@Ig&rK0LF!+uMWf+!sNc z?HTH!F7a{JP1|H?#uKij#4vb}-4nk~eq(-fX8Pcj5IxtR z{H&b99QF|yI$OeXyKhz6fbL55i5&NA>KL`om2*dZ)wI6r7=3cjw7&WN?CN2(!5{aH82ddK+m}(`gAj5JO~4>0}H14 zkjCre#(vB|zgBEC6PtyrjEhntdGM8U99^khq=0?hoABE%g9oQu!4M8wg(k8N2myCB zisUf7dhXN5<;q)=QQCn;Ts&P9nwpr#G3g8(9Pa=Gb_e;cXJoWD)0ADu0ya^cGsZ9X zpp7yT4Ze8AR_%yG&xTMUugHj*=^!BZjEraw*ZnJH@<{*4_GE&%gpWyZ#s)@|{X(k;qt2sv*31e|4sTXP*buDp@cx4n3fr|&x zy*?+RIiWwlR*gWm7vlX)M6_IgHi{^w97Sm_Uq3Xi$Fx^#s8?hp(H)dQ@@}2LNatmE zd1bG}X42#kXCewN?~P&=D7f+>ywpusqYNz%Dqv$E0p%4}FyGqF%lG_d%o~Dk1kD0L6 zr={CLu(-4v1`k7Qo$p+`A|#3+hYd}yQ(JrSySgM#1@uL;Ney#6i!KS1kwv=_Lou`U z;0)RIY2;P2s39pTLQsF)hMCniOWjqGwvVPQ=qbU_RZfq@Py-WS@#2z!qe$9|idh*^ zWxh}k@<`sV=E()!u4)wG<^U-zg`O?*PA%kF8F5t6;$dkblMx8^d52{11>i6W(PxA- zvEuIt8)Wk?&HytZ83^V@CBuYdhZsqs78l#?jkq5Sct}=}AaTLrLCyc*WOc{mym9Z< z{s?qCXVKHnfA53jkX@{p#Fa*|N-~7n$EI^3E22;#M>;5}YR~Q0=R5ZeV_`F`)UutN zNHUm#EXp($D5iL2X+qOHB*&Wz6fj>rY`R2?1~8KnR{?0E z>MM(6m4{})=p(EEaFkT1I*6E(-7}O$yNU_jR-Z&VGkEBu%K&h8X7FMtt|BS5;s>c6 z*4aQd5ov35$A}2=nI1iseeK;;4BqIJW{Vb(vMC{j%3{!Bf>#DpN%4%sUvdeDz?FFw=SrC&+7?^y)EU~%=D_U^1P_Skmjg%Tos-}pOA!Bj z2a3#FzCzHDL3rk@n2CxRJp6#Y_-^R>Ht&}V=&SwZpn*Z}R^wS_zjLi3ZjNZh&(h%0 zIn7DQ5MCfI?r(4usi_<7JJ6hz67{d@)IV`vo{4nl9IH2wE8DecK8R+|wX}NRe5IPY znF-5~gNkDdvao>e?Gr zOi(%Z*(i%hL0oo=|lg&LO zn;MCa;tqXAmx%T?d9pb{6x?EqcWP=_%E+7*hjPk8Puz4W-1}_FBH7bu2GQXeModxm z4Ql~tt6!2iD+IE6pJizYoJF4OC-pCD5AQneEFA0ey$*^Mdz`wc&P$hg-XH!-F>%~n z{+mZk9LxhP-IC|Zhhlz}HdsjF3^+s`vYHet42jz1?Bb<~b9O%oA!l11o>wGXq*{Gk za!L_E0$>(N6&t*IaE&3*&dgd?Oy=}FO#4C3OObHraEUiFB`Yl`{K02HIG7@tnie;9 z5wmnOv=%hwKzOk=rMGCs16Z?KQ@>Z^d_EgWe&Xep+&HqK3J>SQ|i#Tcf zMCPpH!I4tTIHpwmNgP7OT*muKz0+*c0#YVm8QP{5#1`4Q($d9HGg55& z9x#g`MvNi?=T`=tXtsCMrUw?JEn?y@pQ4RGkV%Ut9swU>@f>XcGHr)eeGgeN8RAz9 zDE0}P>@U5Y&BO5c?Op$Fj0(Q9=d5n}A1?Yoz3DE7hpb}%?(yx%?|d=q$5up7nFMq? z5d@`=QPudHK0cDG`?qgrb?f;VJ*6eFAgxM0;Eex)2XWKCyS}-rhKsBE?VH>C%gg%a z$HiHsmpnZS$;H!l4D8U6-*yba_1t(D)G$l#KBUt%8(zMD;}7=FPyZi_u$Dg-VFUnu Cq4FdE literal 0 HcmV?d00001 diff --git a/eval/dataset/arms/copilot_v12_hybrid_val/pr4816.cost.json b/eval/dataset/arms/copilot_v12_hybrid_val/pr4816.cost.json new file mode 100644 index 000000000..a3b5705d6 --- /dev/null +++ b/eval/dataset/arms/copilot_v12_hybrid_val/pr4816.cost.json @@ -0,0 +1,113 @@ +{ + "wall_s": 857.7, + "split": "val", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v12_hybrid_val/runs/pr4816/run-20260813-143839-f8a305", + "recorded_at": "2026-08-13T14:52:53", + "attempts": [ + { + "invocation_id": "inv-ab69fa61", + "outcome": "run:run-20260813-143839-f8a305", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-143839-f8a305", + "llm_calls": 54, + "input_tokens": 515696, + "output_tokens": 93525, + "usd": 0.325478, + "source": "spans" + } + ], + "attempt_usd_total": 0.325478, + "llm_calls": 54, + "input_tokens": 515696, + "output_tokens": 93525, + "usd": 0.325478, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-143839-f8a305", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.3255, + "minutes": 14.24, + "ci_minutes": 0, + "input_tokens": 515696, + "output_tokens": 93525, + "tool_calls": 83, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.5539, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 54, + "by_role": { + "planner": { + "usd": 0.000515, + "input_tokens": 87, + "output_tokens": 400, + "calls": 1 + }, + "agent": { + "usd": 0.303846, + "input_tokens": 506940, + "output_tokens": 76091, + "calls": 51 + }, + "reducer": { + "usd": 0.021116, + "input_tokens": 8669, + "output_tokens": 17034, + "calls": 2 + } + }, + "cache_read_tokens": 3087488, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 3.24, + "gate": 1.66, + "review": 849.25, + "report": 0.07 + }, + "time_to_first_result_s": 854.19 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4816.trace.json.gz", + "trace_bytes": 17449 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v12_hybrid_val/pr4816.md b/eval/dataset/arms/copilot_v12_hybrid_val/pr4816.md new file mode 100644 index 000000000..1659537e2 --- /dev/null +++ b/eval/dataset/arms/copilot_v12_hybrid_val/pr4816.md @@ -0,0 +1,36 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4816, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head e1dac23ac02c) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 2 finding(s) below | +| Security | no finding reported | +| Docs / comments | no finding reported | +| Behavior / compatibility | 1 finding(s) below | +| Correctness | 1 finding(s) below | + +**Validated:** +- [sweep] Grepped repo for `openai_serving_tokenization`: 0 matches — no stale references to the old app-state key remain. +- [sweep] Grepped repo for `serving_tokenization`: 9 matches, all in the two changed files (api_server.py:754,1017,1157,1248,1286,1329; test_serving_speech.py:2533,2546,2615). +- [validated] Read api_server.py hunks in situ (751-754 pure-diffusion None; 1014-1022 ServingTokenization; 1154-1157, 1245-1248, 1283-1286, 1326-1329 getattr fallbacks) — all renamed consistently. +- [validated] Read test_serving_speech.py changed hunks (2530-2533, 2543-2546, 2612-2615) — they set state.serving_tokenization and exercise the 404 getattr branches. +- [sweep] Read `_create_speech_error_json_response` (api_server.py:369-379): it still calls `base(raw_request).create_error_response(...)` with no None guard. +- [sweep] Read serving_speech.py for_diffusion (416-433) and upload_voice (1214+): pure diffusion installs a real speech handler, and undecodable audio raises ValueError, so `_create_speech_error_json_response` is reachable with state.serving_tokenization=None. +- [sweep] Checked changed_symbol_consumers for omni_init_app_state/create_chat_completion/create_speech/create_speech_batch/create_audio_generate: no stale in-repo consumer of the old attr remains. +- [sweep] Checked pyproject.toml/requirements: vllm is not declared as a repo dependency, so no lower-bound compatibility check for the new upstream base/ServingTokenization API is possible from repo metadata. + +`vllm_omni/entrypoints/openai/api_server.py:754` [major] — In pure-diffusion mode `state.serving_tokenization = None` (api_server.py:754) while `state.openai_serving_speech` is still installed via `OmniOpenAIServingSpeech.for_diffusion`, so `Omnispeech()` is non-None and the handler-is-None branches don't run. `upload_voice`'s ValueError and both-`audio_sample`+`speaker_embedding` branches call `_create_speech_error_json_response` (api_server.py:376), which calls `base(raw_request).create_error_response(...)` with no None guard. Since upstream `base()` returns `request.app.state.serving_tokenization` (None here), this raises AttributeError and surfaces as 500 instead of the intended 400. Guard the `base()` result in `_create_speech_error_json_response` and add a pure-diffusion regression test for upload_voice validation errors. (evidence: Read api_server.py:369-379 and 744-756; read serving_speech.py:416-433 and 1214+; repo grep shows these are the only `serving_tokenization` references.) + +`tests/entrypoints/openai_api/test_serving_speech.py:?` [minor] — The diff updates the 404 tests to set `raw_request.app.state.serving_tokenization = fake_base` instead of `openai_serving_tokenization` (test_serving_speech.py:2533). Those tests only exercise the handler-is-None fallback in create_speech/create_speech_batch/create_audio_generate; the reported 500 originates in `_create_speech_error_json_response` -> `base(raw_request)`, and existing voice tests patch `api_server_module.base` through `_patch_api_server_base` (test_serving_speech.py:2339-2353), so no test fails on the real `base()` path. Add an unmocked regression test that initializes app state via the changed `omni_init_app_state` path (so the pre-fix state lacks `serving_tokenization`) and calls `upload_voice` with undecodable audio or both `audio_sample`+`speaker_embedding`, asserting a 400 OpenAI ErrorResponse rather than 500; also cover `serving_tokenization=None` on the pure-diffusion path. (evidence: Read test_serving_speech.py:2340-2351 (_patch_api_server_base mocks base) and 2514-2615 (api_server tests); read api_server.py:369-379.) + +`pyproject.toml:1` [minor] — vllm-omni's packaging metadata (pyproject.toml with dynamic deps, setup.py platform requirement routing, and requirements/*.txt) does not declare or pin vllm, so the rename to upstream serving_tokenization/ServingTokenization has no repo-metadata lower-bound guard. Consider adding a minimum vllm version pin or a CI compatibility check for the upstream version this change requires. (evidence: Checked pyproject.toml/requirements: vllm is not declared as a repo dependency, so no lower-bound compatibility check for the new upstream base/ServingTokenization API is possible from repo metadata.) + +`vllm_omni/entrypoints/openai/api_server.py:754` [minor] — Confirmed against gh_ci_read for PR 4816: buildkite/vllm-omni-amd-ci, buildkite/vllm-omni-intel-ci, and buildkite/vllm-omni-npu-ci are all FAILURE, while buildkite/vllm-omni and the GitHub checks pass. The lane logs are not readable with the tools available here, so this rename cannot be exonerated or blamed from CI state alone. Minor ask: confirm these three lanes are pre-existing on main or attach the failing logs before merge. (evidence: gate_report lists 3 failing Buildkite lanes (amd/intel/npu); cannot read lane logs from this environment, so attribution to this PR is recorded only as a minor ask.) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v12_hybrid_val/pr4816.trace.json.gz b/eval/dataset/arms/copilot_v12_hybrid_val/pr4816.trace.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..deb9f2c74e55af89f4ad42ae34f2ca8d545486c7 GIT binary patch literal 17449 zcmb5VLy#~`6ErxsZSxu1wr$(CZQHhO+qP}nGynU=#_mS!ZM*wWk=>WRR90kG;YUFL z{BHsM-S|YgZizPT*X>g{#g;@v6ym#2Ty-6>=VIQ;80u|xxV;VQLV;i*jiL+?ice$? zpIwVxqu-`ktGEDzh{uyrno?M#X9y$V#oX4u6;=82T*5#4IyjsDe4Z?GPKHK&M*HE! zlnHfzPSuHBYQmED$k|tUUNiplv{Wyz`sE*JnQ*V%|JM!Vw;u=R)_w*e?VOGh0KsK} z06E4J7#69(d<|PfB~pYB3#R7mQYb2he~>1j>|J)xvDg(0_HN7<$yc(W zpP#rHyi7lvWbGbMXF4Rnsw~!0)kE|Z_RkajZ$7KwE>B|`x|oDrX4zf*$YJkjH| z&XGfP#Dq8Bt5)obvEy1tgc$1*-p}XfNb0-Pd5Pd+OnY)Ts#r?zZ^*C5_v`L%m)lQ3 z?mk(}w>&+Po3do?A>wu}1PgtK@B91qM9Vw$0}t9HudE>JSx9V=DeWUAlP8fga6%o7 z`@ly}@(zNWEA}8&XbbP`NSL(8TyPI$_<~&^EaFW%x8r*nH5mWH8HXw;{$~sFXVNF{ z%O1JGg5}WVK-G>h*3jgy1^{9yJ`K?tzV~dz0qJS|s1g5By~F2y=b)ZkU>!afp#Uav z-Az09ecQm6s2!WWZSU^OmMDq2A9Tf(G^bz7PF2b*`QO2NgS{f3N-a#zunf-xF`3F< z#CQKw$66fsTEtHL&@VJ!L{Wa;XM{bsT17br@hL;YZZJT}n1=Rzawbz`AjyqX%HuF{ zLKc}WQA*&)iwFvrHi-`%oH9~bH61YGd%+Q8AAXelL;W03MJ5xFv*YS^(E@uPK(K9A z(iO>KjL-njMOcI*pk+nc=;Db_8{K|i!X;HMS-^7!I+rb!IM$SWXodWu)9^N}gWrsf zAl?dNPP|A03=jxRn)ayd9<1WT7tsrF8~#fX!TuoX&p0A4G&eshxx}ZyImOByS1unL zE*QR!9WN>Qtq6L5h0P4UmqcnQO%np%QqcgkMG;>N|YP4 zlOn%?#Pz6jut8;^Jk)6viHl)sCuP4fS)>ROg)&HjGU7B}q@r4fzdk=jfz$H|F=Us3 zsPzm45Q3AMW@G^gk*NA`+9lKpMA4U#iLkm10J(-U1BeA!as-{9Q4g?%#Y(L&#((=M zCZ$rs+rqamoa}O_-pJ>R=67{z@MgWJ-7>}zV=XW%NlL`3e#Vno@d_0+$()hKojZ&EtE5^G+;;lb54RAvAv@hUP^h=BQf!8O+-c;7{OCf^z|+ zSilcTNd$JVb$1AuPS*u&!M>GmH3o}{3iGT8@!CaM{Y0uA9N@B0KE(Ga2}=eBgNCKs z9}@j}g3T@uWz{^JaDBSW!W9!nmTMmKj6Dov?eC3}LY%%3tW|@)V31DEm(n&F%cuDl zXgxSdptw3(jt1413ElR{SzuhUO(-c;4_JSk z-39ve*2Nxh74&@BpTC?0UfzmU7J{N(0E`L}o>9MeXkx$Bhwvs#-lj@-gW)=2sUQZB zMI#cuaJ61GG7f$SOCdS6yq2f?3 z+yTqMbL2*a5C0hfuuM%K6w#xo{>o&?dJ%YazUZLK)bt(?BTCb}AG|XLlI`#EB>_Eo z&3uPP{?xxtZh0e`+++yuA;l3&HFuUBxiHIAT;z&Tv@KuiPi6$!D^U-&Gn@jGyRHNW z%66L;WeLDvfSL7Lo2gM45L(6`dw214C>V^x{#TX{+`WJv05ai)Y2Gz{cHgU+jlG4i z{_8@ViWSY!qAf&BeQvLNur6pu*BGM8Ia}|7t|$d|7?&kgdC8($Aaqd%V9I|k==rNw zS(F+@d7Q5>~~pQ}`WW(j*_l*quJAH?oNGKL2SKqM?HA2$yI3-ANG;3TW^<{p2)2*3@-o zd+em&C&pJ!vh{m-vex`*MpE9$%JJV^ORQM_VC6ANaMxw|901unP&7;1d@Z41d^^e6 zm?a}FyaQu9Xh}Y}#JW(pg+ko=rqCqtmKkF&X-&hqCBvSLG)P0}b6zUBLr~|@sshxDs#jnHi2y^G0uaL&x0C=wMOYPSUOj8ed1A;7CmIQ zwM~P0)l&tSNKbxOv`QM}IaC&wg>Y3~X~hA`74Xs~Tq1t#ZEGw!>;c$u47J#w%6^`V zY^)&|LWfzg=1l;ZR-WJeL|757bPE{Y6-GO1JVC?3ML09bvJB@i%Q-MPer#qrGw$jG z)j$5p==cXr)GwYG<8%)bDPxKQgg6#D@KK2B`@1EaKC@-b{j`XGhYq4e37>@*ILhcj zeXQehULEeXDw^Z<^qQdF(ZTBw@5q`$qUj=Q{hQ*yMWzg^*I#7#et{ws$7l5_8lKRa zQ1(xXCEiQM3Z6m&jNV{?C6}eB4b_1t=m=Dr8|rc_9kYd*l0w-y?X)svLb|?EKu{a{ zN;jEmt<8*aU;eRrEy^&uB5ec&Slt-IsUeqxOPLrT9dcW)ITixa_rwZjFl~~omVjaa z1RD8?9LoSVsDOrR@#zFr*1_xUl?-I~Tle;8dFsS0p(VPgU?^9_qtmi6h6EC@u}D1- zTDG()qeB}-^<>Lens)lbAE}r0-rm*!js;xqKH~P=^b!i6 zm$LHE5ys19em2R-N>+v_fxN!d~{R_g|CoU@eC;WlpFc7s)*8Z(cCeg(}uHuyK`F_EL|6Fg`@xk~3 zSD6&jK0egEWWA`OE)5KS!qU2N;(ns%xy#*-)S?xiD~W8r&m5NUJ+`w0$^yI+^;q;M zJvc|Z9@WHpk$}79g!u-q_N}1jh;TQR17M82O zE`$58`lb4kZbrto@PQv)*6j&_6U)3auw!VUJJR7syWn)h!HtG|{8({fTy5R^<3`&| zx$nb?8h`$2G7P~B)IHSGeft5^WT z3*@aYuLJ;e*=$0OI!24fs7J9B?5FQx_gG)4Z=koj=$ciW>0jhl%`t#km_^@vJM^W) z@aJQ0N}(LOqO1Hl=Ebmc3NW!|6goP|de^xi&(+NKY0ODEDS8R+M_#d)*f&E(gJW}v zZLP9^f2RN~3f)M+GR`|Y2-T>STm<7(P~tIXt3g9vx%)JE3BX#4Qv+!@C83MJHZ+JN zICrzRqInGCLS9YyS4ZG)c{0hHS6Nn}=PfXRbhp;RdP_SC(k(sjW^$!?T`v=D3*4b< z6mtd?DRubIZXI|lHszn$JI>~}vgrAz@dKff(UF10@Z6nIR43~Gr80_ri9VDyE4bA7 zfQ?I_xy&fm6iB=)1{PlMtN`Yh&=;Fo%;i~f1c>ISXrHnfpB2gJSGO=g{Wen_u;Avc zhCsPxxOe7GVed^!SCU8)C$VP1))~|_LOkoD;}1GUtGadxSHx1 zx#%58^}8>fCl%^1D&E%M&WMzhT^=J7C@4G5sWI}*G)xRh(k+a#qU<>pz2LRUV*Mo8 z2o$#n ziVl~ILsW#kr;2EKihe#9AN7OJ3fR6d7tNWAewIqHoq2q^FwgoA0kQRsTL5ilwNUrF zTkdF^_Di~CKXkidW4xiLC!aOVv`#LE8_cqp!lNDqkxh@8D&)$DFcRO z!A-)?lHAkdyUJ6iX}gJ8duDJ6h|}kUOjRGbDp!RxwwNE#CK;=)(P$zcGgkMikT9=5 zq6xGm@-@A@H@SjakLIfZ)qonrr9GP798jo!RfV&(>ceL?B;USE5YM+RzU>l7Vkle8lL`$Xe%Xr_)a07%67;OQs+y;)~0*pR(shJ?!K(qBl8_Q4J;;&!F2dh!<~` zGp?RFIo+1aKfFMsau+y(L~$@wLAY8yq2q?ZZriZfytrcM_681lDo z%DLJI)kRZTiRj&8%weCtwA)scRwzm_o$h)p4f8bR z?z`XFiCs!msA-FyV^xim%72$Cx~nD%)5-1e;WnCdLzc9MyN zytd^P0p+V~1m>Lk4GPBu`pmgjV815{3cfeh8rRgizZaj9eR*M-S6` zD(w7Pt)FjX_9_i;Dy|U?*)v?!h=B33q3_p%{$yLUA>{vjEUbf#l&nSOm|4M0pBhj8kWn|*6LXk{K>|=L=h~5v2Cnp zUb#s>o3`rxWZDa13UFw%1KIaD1Nmp0bw~vaasZ4tU4qfL-Odu;>90=KHBJ7cH&>p( z^z)5OW;>iq@YX!{93~!S(ZuD7#%IV5vl{RAgeA=GL*isy{piKs zBWh!>F#6&Q)wlR%X>HD+6#QC?Ed83t+pbF*b}V`*qGc8!C5!d_u?K-C(cq@!#mV^r z@-w|(;`p&IZo)i>gBQz7#sZA{vrjI&XHyw^myeN_Lugqb4i!AV{9>eRa{grYA+#t3 zi)e+}as5Tm2S4a!N;PI)=%q~QcR_nt+`X$MNv7|U^kiYF47f~5tBkz1ES*}Y z&sSnEj!-AU$SE?^Lg3D7;&Fw6@>o|4Vd9RDt^*$tTu~skn@J)S6LisZzBJ=)KUixG zP?(1tJE;LE|Azh*uG)g-6D0@Kyuv5m5(n*?UT0R{ zgl#^^IuStjoHG+r%)m+4w;rC0jo8^a(q^-*wnWnxI6uB^JT=sAgpFi-9&HyFE0R@uV7PN+>PJuR#FQe#Dt4Z_7YtKV zrW+iE-fW9ZtZqT*rP zauHG2#T79^S!!VndD@!RrW;>4I3(m+Z;fNU|figAs#mmLsg%77>q^R0y=a0ST*qJSWBTCja#xO_=lbf z2%r-qp5JW`cHRE;yAnrN^Y)))+Z;=`AgV|mVJORjesl?tAR@Lg#bml-q5U6Ka;cWp zx!M8*Q3R@~@AZv#F9U|=93Z!w(||b`JVHy@qzUSj#R3g+5*nEl|S>%)JdQK;mI(Vo~bkOeo4=>bLncw`j|@ zpxCtZCbe)N;@$Knz8k}So3;n=6y~` zM93+cdl%+O-Cxk|>p>TduRYA)S4uVYrmer=&RWCOkX=@~zee?nzv&I6KyA?u)N*~e zn;9>~YsWa^9R)jno>?Yq>2#PvhF*hHPD)rN>R=O|GLq-2ypZ0~^?AlB{YGUxaBoFc z`?AVXKa=svMi+m>B;wZ^t@vIc(&r8@0+f}de{XRocCvrZf3|wfS*)rf_g11SeG-p;M zwiYCoBw0gZVil@WDY%BVdoJ>!Ti)isrvSMM4@k!}oXh4W!~NCS5b{6=v*3q%0j^~U zv>U-XR*U%4v+=P3cFsQXmWr#i)bKPrJ}cODI(#35e@}Pfs3V>kmH3tPTx58WyNCl7 zt8}Qn%TYqD0g7q0h5*iuYG7RzTwSc;VIN7{JNz&VnR*6&9}33gvHWqf*c;S_M*VL1 zOMQ8KB&OtOKwP7*ytL8a+)jIH-PStjz04B;n?|ryL4;gV>+VrqL$h%Bek{G&>GpkI z_}qo9&Gp^=yg&YIsnPAe{CAU<@L5vTGR>?(&`8j{xrdcWKMo>I0732dYz*)2?aBK z*TK%8fWzhg<_O?Iz+Uoy+ve*s8Y^+}Sur4QidA1%1CL$<8-ct*7(LIOU^KC~Z_$_P zeDHo%64oPvv4F7?EZ&8SFPBqVK8qd5cL1D*uCI+LrQ}vQ%#Q zfFyR#fBWV5s%C<(?juUS?`ZpUj(nLSxC~k>QHryZt3p|cTUI@qagHXH;q%Y@&XtXm z56Q*$feGW4KFRsAbSzq)@BBRo!-q7=jBs+^AOW0B*edq+X-qE_{;xqIfCO!IhcbB*683msDY`OyttwNC^B_g zYwcr7H5fB8G2f2d2B6w zICSwh#F_(?64}9O=eQN5-qkTRinAnVR{Ai7PO+%mIOi{D#ley0vKZu}jy2x;2=fyx zH~HSTe?~XCE2RHlid==eV*$m=Vz%irHStoYLxSFgAS-Ce*wvHIhdw`f;T#OV0b<%a zl?ql-oyMO-QK-&$+&O_f*WgF(leYJohsvp+t2rjxcQMHe0SvGe9N(@H%c`J4r5GY< z3KJZGz!;5{U*23!H+NnZW^OdTdd}eGx7-h{E%Wsg0l2>Qss96eK*&=TKCe0V-8bej zHkk=GF&oX!kNt9YP0|7R^YC|%r|MP{p9`|f&IteM(!`B8&yrquMC|~r#yWt`V>(nh zV1(y@@O}-jzSab;lmT#@X9@fQ7ZQlZM~gdu6mvC=hGDv zQfc&8EAXF@*`fhsJtzIkYfue^W!JQ6y}WdSYmyQe^l zth9t6_f_c=|AYN_#!J;_;BGqV&@32NlPa8v;+28U_}`xv-PFxp{p?@W+R2CA{@|<` z-;G!9qXv@jY+sHBn9B{10lvM(=5bwAN}UZ)H_&_gtwqdByi?kHdFYymO&FL)UzH2K z0n;u`W5i{8EzoDf3(GX-WN335%JRoQz&h28Rf26Doo0cni9k4@PO+c&2#%#xrA+G0vE(A{CI%_3 z#%c+)fOzreE~l~R`Q(uz26yD7N$C#ErQmR7+kX`*t%cqRdO896Bi)pN{SFKzB^gTO zCW>eyLM}WH(>Q;hU&?Bo7?lwXtrUt%WG)v?*DD(fp()Zg(k)K_M$kQnjD@TvE#CDc zQ0nKw3jcTrmG{YfGr<<6GKk$b_A?9p-wkWToYrPTD+J5qn&xnqMmn`x#Va5aWH(xq z8(*9~+iq2(Gt3}A8F~l6pBLkF2ywJe+u2*9hlY?0_Ajm0!>w+l*)EeU7e6u~U0R4s z*;B$+cb>GyxLii@hCryc<;g{{Kd9ANTUiA8o=(#>UkI3;T>RK)eZT{5G z7elb`^E;gng5MTOs=jagY?Eryls(jWIwcW)mAsr0%_y;*3AH}TrEH3uIOEo{qax{7 zHJxT+TbQ?0Q;DZJJ&>w4tG`E6L#$b!7!zI>&y5Q>9Ic4wMzl&3$SGY*6Ur#r3%yd7 z8sF9=AkJR$IUyH<@rB3>RztR@WCi~x*yaSIoxas=NR;=&!nY^MZ!zMzlNWp@;t5tc zX5JMjee7_iw8a%(sRjl_kI}h!Gz~A`1;8(F*r~*qCfFG3pqp4A{u2x@fsdAtuuGHn z7Fa$Xtf5-MFf|kA`U_1b?jF4xFuuF%~s6daBrhQ z-dkS&{Eu-SIOfed%O{)_!Y&eD9ko?lLd4e0vlA73=ZhUqYqyMl$<-H8hrJVgSgBY@ zNmMHZsQwu)f}u7^rkk}q2AJ?U>Bt|=t_aI?MQMn|l%)_CszRo4XNPLrQS4d?%B#fG zrg{r3c@hV_H4b=e^i@^In5ka4aQsDX71*S;u9>h@0z@*2 znT6@M+YVeC41!}1rhE$vHBhgyimc`sv(vf1y}BuceWlj{sizw&%_q;Qso4mtI!CMo zFq1EhL}^7HSb;-&-mcQAzdK)yr8QQOA_L*AmUyC?PwP@F8oo4NbMVEhG@=qC+TSrS zsjR_gl?2d8y28-HQI#&QD{$9aMc9g$#DYD)YZRGgsJ2XG2}62vf>sPpWmt2s+ua<6!H7L{GhFZCQ`RuR$nDDv&O zu+6I7=rF8Of^8|@p$gptw-lR1>?7HyUvr;cczXr*@3AY-_0=Qhif+G1Q55Wk~!mdnY?d<%T>X(CZa zUa-f^3t%cBu5o6_uTtbl0K`wF0@ZmsVyC$4wDDKj_G!35U{dmvBU5~2j^8M}rD>O) ziOk)jAtaH#vs9U^Q1#IFm`c3wwaASYklg%Vr-wqq>)D)(i`L>x)ks(+6H#@xI2)|; z*pho-)nO^ArXYJekTEtyZl%1Rc9ByuoTictL#xg5*7lLv)O}SjY|1KO)Q(}=ol$8A z7FjeJ6H`y!BozAvf8CJub&G%J@{ScU>)5?&RmH6WyX7$C0ikGng+07=a^+c)>zaWj zh+edMNI3>g_N4$U*15`k38gf?GVIWmWC8(Nk(gE|;siW(TBbOSDzV@y zp*3l|fVvTD#}I2<<)c9C=hmaM+R1?vTWFULW=W>j^w1{jubrw>@zkV2ZB&z5{eInY ztG_Y(&r36S&w2sMcT0$~tJ$k=+YJ8re=RgDWi_*Q16=FvuRUtr)p%Z7S!pZ|o?qD! zrN^P~(Cmt8CLIg!=jk_(XkyiYF4Q<*wNL9#r|cWHD4L7`=$k-1L!`CFGG3UzpI1I9 zKP(%Ou}!_jssFsQpSf|)nS?QNh@)yDhuW-kmM6y@F!!_Sv4W^6|7pIX7o+E_(9SgtCSf*Iv zY_4_1+^qd0LXxjO!v!ZAU$Fc01ncbN2&Z%y$yJbs8~qiW#OnXT4-a}i_|r4OiO+ar zAMr+B5)J&sYIup&IOhM5G4JtJ4&y7FMi;mZ&M@m7A-pHTxR3wAM!f%@zrcA+hxxw| z?)&_Aba zsq0q6W&TkYYtziz)Glh{%(HCNY8&sGQg^YDY>S(}Co%g%p$46KVlzK)Ge4A##xr;IQ>B2d zw7%Cb1}_z`bv8s!x{<2H0le>ThE0`bzFg#waNZq)84oBXjQ{BWPsqnL|8Ah(>E0mH zu~?EYpClCmpfA3Zbk>@8mUkZf(#{km1XOSqWU;a1R$bcp3i0_^agXHTbkKI-Oz(en z(w-8AfiD^bM>p7bu5~;;(zYNF!xXZobaN4c9w#aHfGAj~pu9h>&HQ`#$bw^+M}9vhcAB>-+;_sn@^a=t3M zGxaU$@^}B#m1fuF$_k?L@egR+_jdS$0{b}0mcv(pcx3@&&7HYz|G&!Cr*eiU0tk)M zAX!ol0<}FNuQcFU@icrS#nJXlcVGPWB1LwLr<=w)b}T}s>rg~LppyI2q6<8Xgqps^ z8YbHy)wNQ7zvwhy%Dw_|H^0gW(5e#*8+!d4wpNjP1wEz>xg;m%-7Vbj%;CuQgR9p!BVdqz zHd1Z{xzC$KJ;8z=J4pHYGuqjj7uT2=wj8z@!^S5A`V3?q{GD;62V0l`WS-BhAA*z! z;3hFth&w3ZS~vhPf0-QHd8&gExet%#iEc=S4d!P_loaZ-l*D$IsT%f!mY&Tgrk$)C z0tbNsNvFR@0%S3x^vdHm4eLe(z24EigaA1Tq`Y%-Oe2KW=@I3R#6z{V%!x*z-&`YV zS-`c4%zvS|y;?#`SyfU&^RY*Y1!EbH@Ff-j%15kIR3u;t8$ShDz5+UE)xF>w_uRP} zoqj1`Mz5GPUb-X_{Uxeo6#!lT>>n;r+kPu6iLLkDzd}sZyvBiiSGm6$9x0yrDWphY zM59hjGvs4D^FCd)CTB=*A-bg2CSc85bHVTGQW%v?g)(?6KzAKy=1i1fS%=RYbM^d% zKy`r2)nH!DNSMHC8+WS1VYZEXtaEl?<;7@aVv)M>G9#s1~Y z!#%O#S}X7?qL$;(wKhF_xBmNKyVklf|rG)o_OBy z_;0Q#PD5Ca7LUGj7ld?{bkghXfkxl@x}rjO`(kl;n{W}|AMq$mUgp;{wc?!8w$p0e z?9O&0k9z9^mgv*JcF%M|oz^m&jBDw>y*SQLA>e>mK`K!1y+TNIra9-rFi`cJaP@gY zkbJ$IA1C0i=fMqM@d8tE*#W@np?H4l-QlLUQV#LNSI7{hlgxdP@>$voAn++5n}Nyd z16oZGz~XFF0``B8j#LTh0Z!c?5jXh@>Uq}OviW&?OF~-7ard6^bHD?b#$3HezwdLK zMplU@tI{FNdT%%asDOVl)JBW%#6SGiT7gJ~v?TXZ4vLKavWqwn*@>R*jxBQ&_op`ne*6r^B zxY4#>Z1TXR!G49(t<8s|bLiLoFifWkx%9?z`$41joQl!8cUeMvaoKB~PlWUgu~VVQ ztk)h&p4`>ONc!OMp7v}!4930x6lmj^w8rOs-eH_CF16X>A0H;c2Y9eY&jyCd`s%6~ zjtjoR*C0}(jBD@jl6?PEfS%H4fYdl=c>t$5kzdm(8b@Ng!H$q3J^n*ORB`qiuf=z0}7_@*4q+B{zLJKWHNK*6x`HK#-uB~Pwq z@Sa9TjUcOsHL3n_n$Z=Ys6>^b2d zHV|d^qA=bUw~sB^IVmnolOpf(u;+byd6YWD0G>dGw(#*W<5V%L12|Ygo?WOICX7I;uGsXzj&u@vLi7~_&si25m zdGKIIU`Kgi&j^Q~#+VC~Ute49=6qRW7|d)j>p>3QpG zeWR)U!!_jq&4t#iq8dJOYJcZVC2~4Z>)AXZ>TPYcV7P^IbZ_Ar|J+%)IcCI%%4)Dp zkWm(EYiEGAq7FFS8L=fkqBI0NU2y(VNEpGc3E@hK>XMKM`YsKL210M&O*IZC6})A_ zYyf41>J=2VGJ&s6YHNX02$5bdm>TR~7T1wGW+ybdO$7*Iqg*=L6`a=aS9eDbr-2Q= zg$eGh@9Ad|<3lQWnsBE4(=g0gN)0Z>=6VwC{QD?SzX8G4h`;#W+6=0D<;aVl39)5+KSQ1xT~X@O$E)-3`z z01XZZ+zQfP>=6>KJXdgjsxFmtp{08@w~lS|mA42h|JbY$!i!)~TNA9F4U1_5OL1Mwu4{^={kX18UJg~lHW^i5 z7XvORrsA<&G%8t)IW0vL$Y751Q)#*Lh0%6)kR`DvYtGFxK*br`Iyli?qX`vU{V`zGy4W`5Iv z`hTOz1jxtFisq9bcE&dgIx;Lv9Hoakt_NCiHpfD%?A~VTGUvQ2_{qS1xOxCSP@SW{9mHT7y`?aP2MjALQ(vM=2Fu#jjyp z8^=24_^mO)6Q2fQ1r4Z^&DEp>TVMwK9_^j5bWZ?`^$T2?Z^xAc_%w91th8k`f#)Me z5$llQG*@~Iqz;Asr_eATGsJvss*?wk$oeIO7;#QsMJv$+ZgrDv3192Lp8w0<0&5i; zm1b!MV@tbI@7efJ5SfdG!YURJ+*R@@U@LDeRF0%?#9)Bcg1{{AC0TP6S!s8rbj`?4 z$TNJ~nmKx~2xFx=`@Oj}#jv(^K~5p;Tt%rrtyUDF;J$e= zynVhB0mTMi0&P>WPt>f?Pm$?UG^cf*G^qS2@fipfBrC#Pt{7dIxYtM!4t!dyfB^k- zGw$r3Q6tO_SF*SWcG!h|J)MbNzb;#&c7z)9!Zi0{)6wp0ZF|~YyKVgwo9M%LED;a( z)B1AM3I;S?oKwvgs~@9DCz!Bi9#VRjFhJi_-~U_sYEWFsqMH!A+NYFgV=AUr5_;Ih z*`icdG??x(gv(%=4d3j|qO)Pn=cueYscNmZaQ2AHNOa$Z2T`pnAU%=dIuC%xd3bFP zsJr8YP0^!CLQ4))u0G0ag7{oX88msV}^f zW{gnJvw%o#3mNZD;3_Fr`^sYrM7-ykNq{uxUSRE8?L zZZi(6UXPE_;g2T;DRG+|?>*218*mKf1T-HbvBwZ!+*Bd{$dfhWz%{B#g4E*S;QcSP9yb1v9?=KCS?Bh%hJWys zuR#ARK^$k?)(Eayg?w3*kXOkp28eK zJ@y-Mvi1Sz)I07|b`Frl`|-_A_#TJTH=JB$v%183LTFISl`SL*`LP9#m~CHP^XBWa zj=1NoZ!;IVO%N$Rl&h3DwKE$?0zhE~aKhBqIYg96R{(h{h{8W$)KY*Vd&uy5O2qv1 zZRBunvkY;^A>5$xEoDG-YdBGAh)rj%ORC{-8=GSBQ4DQ2#Z|i(762%BIN%WjInAJB zDGl5wH>-)jO#)Wdq(BcpU{+A`hNOB4B#aYltppT$P^_TGVrR`8l4U#&`#+wD;(47* zWJ6_CGdKN3-Ygk^l96m-q}Y3FH$c(#Jn{ZGo0;PIheF{HfpO#wNh$Hl+_O6|B0x+p zsg0Z|Hz`To`ij;5vuC?KE0w;A9i-Ow4b_$>ZZvQrL`K^eNQJn6&k=W!qJ(;-eqmjt z#I&p{y`))AkPohD2(IctT&eQV|gyt`Q1nQv^{sM6ApGAj0% zo3_BxglCN+pkt$IgJ!dvP=cITNx;rrLJ}VQa&T^2J^R8{1@|Mm`NvQ86{q?T-eLNKw02pR2fN(q z-t=1NE#Nx|(H;k_tBFwE?RQn;A~z4tmP+K!d{24sWdP8v)G^byD3XgVx0O8g{$Eir za&LIRL|dyg#YCy4K4c~sO=t)gt2VmhZz7V06$L1gW@|8WTgucPu{SIQAf*CQ)~as` zNrWD2t&4c{Xb0kKa5=@O(&=XZxO?;B755`c8Z`EHS`x=PwzCgdTEIg~N;Ug~YnIx_ zj$2O?!7YXUeFY%d_Mu9P*BI>Ijs>BE8Fr9sU#t7vv#CBDhk#nGubkKRG1>5xj{KDb`hKU8ebQbbl4d&gD4is4O6g{p4m!0d*Pvz?y1Lz z-bnQ=%R-mzAQ$0&eqa3NlI!dWO+2!$Jz>E;)OQXre?%JyjzJnQ+}un_E6sh}X14EG zf+D-lzlgBs+XX)bfip>OE!}5m*wR*AH}X^B|9JWZ$K&i>GGtMZ6Qpe|CHMFH=aE@V zXm#T04>5f(y%VP*@H%) zP$p4tAJRuo&@ru*={@ac&E8??)Kp=y(}|SaiAG>h#xGCO23Pr}!v@P!um97EL$sr? z2)~}Io_qgOf~dvq*?8rie`M>sUML}*Z_i=4V7qt&mvLt+C9^2iJuIVvFx5KVnd;-a zKgB~D(WKvjKe*R%fnuATd^TfNVZUeb6c0VWJJ4vLW%l$H5&fy}$x*37xfZ@3Hk+I5 zIH!}lJDGzUX4?@za7wA?#3_vNPQxHzokk&4UE1Cfvn6Jy)OXifdU`RNQQKDrqJ|*H zX;r@S@vlb*II>McuKuWJ!kOE;E%)>V0xVG+EQ*q9Ic*-=qn_q2Dv4Rq7ss7aN;3on zLSXj2cbNx1$4hGIa|f5p0N755oyJq&XYNJo8?loQAKh@vI(!|SNJeSltPxdq`5^hr zQO>lsiF;$k`Dg3@1eyb7{Mm_U)2y&O6FpW~>!C3Y!p$(DB%| z%rHv(P6TS8!=tzba)=qsVa}dBNn&}fkqCob!VS!k;O=k(;%e#|z{`Dq5p2}J5mdIV zbNONgW^|l*JeF;Cu#;Bgy*S5;_XFzbBl$B@n*~m{mo+=b9Dxs zc9nq(+YkM4Yk!1FBOww$aYbu|BHfnI<#Inhb0OTN$wqv>($^iqB$?R!pi?t9LUMNOwevq`*5Y+9Cgukb_Mz}ou#Q3~p@2)nibY!@BEO#-xQY?DQ zsijEZ`Qi2sBG4KnOFia>t)vv#A?OXke)I;;Rs=PeyIWra^W>&}JTWJf1a|dnAtq`+ z;;^j~SXcsrMz9=l+3inxdQuJcf$&LcA5&ej<$ zek#Mb?*O$%tvnCgwizLCufXK%1{J$=-nl^#_?w0*#GEdUZPi+QL%Owx6FOTH6Z2(1 z=)1mXw`!?hkkNpCuUgSdn|1UWCF7fDD`8rE@qoW>;2^?q!{ z?fbB*LqxAp)V}H1U4j>&viOb4J8NtMk31lG5s4sZa~0CCC~CJ>`#-|}zWjeqJf=$H G+yMXwnD^-b literal 0 HcmV?d00001 diff --git a/eval/dataset/arms/copilot_v12_hybrid_val/pr4825.cost.json b/eval/dataset/arms/copilot_v12_hybrid_val/pr4825.cost.json new file mode 100644 index 000000000..1c72c686b --- /dev/null +++ b/eval/dataset/arms/copilot_v12_hybrid_val/pr4825.cost.json @@ -0,0 +1,107 @@ +{ + "wall_s": 859.9, + "split": "val", + "rc": 0, + "run_dir": "/data/zhoutaichang/copilot/InferMatrixCopilot/eval/dataset/arms/copilot_v12_hybrid_val/runs/pr4825/run-20260813-143838-2c0697", + "recorded_at": "2026-08-13T14:52:56", + "attempts": [ + { + "invocation_id": "inv-104fe864", + "outcome": "run:run-20260813-143838-2c0697", + "rc": 0, + "pre_run": null + }, + { + "run_dir": "run-20260813-143838-2c0697", + "llm_calls": 115, + "input_tokens": 735264, + "output_tokens": 154650, + "usd": 0.546098, + "source": "spans" + } + ], + "attempt_usd_total": 0.546098, + "llm_calls": 115, + "input_tokens": 735264, + "output_tokens": 154650, + "usd": 0.546098, + "source": "spans", + "metrics": { + "schema": 1, + "run": "run-20260813-143838-2c0697", + "task_kind": "pr_review", + "status": "done", + "quality": { + "q": null, + "weight_coverage": 0.0, + "partial": true, + "abstained": false, + "components": { + "recall_w": null, + "precision_v": null, + "useful": null, + "calib": null, + "decision": null + } + }, + "cost": { + "usd": 0.5461, + "minutes": 14.3, + "ci_minutes": 0, + "input_tokens": 735264, + "output_tokens": 154650, + "tool_calls": 192, + "usd_ref": 1.0, + "min_ref": 10.0, + "cost_index": 1.6478, + "cost_partial": false, + "unpriced_calls": 0, + "source": "spans", + "llm_calls": 115, + "by_role": { + "agent": { + "usd": 0.526845, + "input_tokens": 724016, + "output_tokens": 139939, + "calls": 113 + }, + "reducer": { + "usd": 0.019254, + "input_tokens": 11248, + "output_tokens": 14711, + "calls": 2 + } + }, + "cache_read_tokens": 6572672, + "cache_creation_tokens": 0 + }, + "timings": { + "steps": { + "fetch": 3.28, + "gate": 1.61, + "review": 852.88, + "report": 0.07 + }, + "time_to_first_result_s": 857.81 + }, + "risk": { + "incidents": { + "catastrophic": 0, + "severe": 0, + "moderate": 0, + "minor": 0 + }, + "safety_multiplier": 1.0 + }, + "catq": null, + "tus": null, + "signals": { + "escalations": 0, + "pushes": 0, + "posted": 0, + "steps_completed": 4 + } + }, + "trace": "pr4825.trace.json.gz", + "trace_bytes": 28905 +} \ No newline at end of file diff --git a/eval/dataset/arms/copilot_v12_hybrid_val/pr4825.md b/eval/dataset/arms/copilot_v12_hybrid_val/pr4825.md new file mode 100644 index 000000000..3df210f48 --- /dev/null +++ b/eval/dataset/arms/copilot_v12_hybrid_val/pr4825.md @@ -0,0 +1,40 @@ +# Run report + +- task: {'kind': 'pr_review', 'mode': 'eco', 'repo': 'vllm-omni', 'pr': 4825, 'issue': None, 'report_only': False, 'post': False, 'params': {}} + +- checkout: PR-TIME TREE (head ac3a2ce5a9b9) — the tree matches the diff exactly; repo-wide greps DO reflect PR-time state + +## review_text + +**Scan:** +| Category | Result | +|---|---| +| Tests / verification | 4 finding(s) below | +| Security | no finding reported | +| Docs / comments | 2 finding(s) below | +| Behavior / compatibility | no finding reported | +| Correctness | no finding reported | + +**Validated:** +- [sweep] Read manager.py hunk in situ (offset 300, lines 350-430): default_components tuple now includes `unet`; loop guards `hasattr`/`isinstance(nn.Module)` unchanged. +- [sweep] Read test hunk in situ (offset 520, lines 575-620): test_lora_manager_discovers_unet_component builds synthetic `pipeline.unet.down_block.proj` and asserts replacement; peft_helper has only `r`, no target_modules. +- [sweep] Branch `if isinstance(layer, _FakeLinearBase):` (test line 587): both arms reachable — container modules (`unet`, `down_block`) return `layer` unchanged, `proj` returns `_DummyBaseLayerWithLoRA`; not dead/redundant. +- [validated] tests/diffusion/lora/test_lora_manager.py:617-619: new test asserts `unet.down_block.proj` in manager._lora_modules and that the tree module is actually replaced (not just recorded). +- [validated] vllm_omni/diffusion/models/sdxl/pipeline_sdxl.py:46,110: SDXL pipeline declares `_dit_modules=["unet"]` and assigns `self.unet = SDXLUNet2DConditionModel`. +- [sweep] Blast radius: grep `self.unet =` across vllm_omni shows only sdxl/pipeline_sdxl.py top-level; soulx_singer rmvpe.py:236 `self.unet` is nested inside a preprocess submodule, not a top-level pipeline attr; flux2_klein/ovis_image hits are docstrings. +- [sweep] `_lora_components` consumers: only sensenova_u1 pipeline_sensenova_u1.py:543 sets it (`["language_model"]`); adding `unet` to defaults does not conflict with the opt-in list. +- [sweep] DiffusionLoRAManager consumers: diffusion_worker.py:342, ltx2 pipelines; none depend on the exact default_components tuple, so no stale caller. + +`vllm_omni/diffusion/models/sdxl/sdxl_unet.py:?` [major] — Confirmed: adding `unet` to manager.py default_components means SDXL adapters are now scanned, but SDXLUNet2DConditionModel.load_weights (sdxl_unet.py:835-845) remaps diffusers `.ff.net.0.proj.` -> `.ff.geglu.proj.` and `.ff.net.2.` -> `.ff.out_proj.` while DiffusionLoRAManager._replace_layers_with_lora (manager.py:340-418) matches raw full module names and _get_lora_weights (manager.py:487-510) does no ff remap; the packed fallback only derives from SDXL's attn1.to_qkv stacked mapping. SDXL LoRAs targeting ff.net.0.proj/ff.net.2 therefore silently skip those FF layers. Add the same normalization (or explicit reject/log) plus a round-trip FF-target test. (evidence: Read vllm_omni/diffusion/lora/manager.py (target matching uses the tree module name with no ff remap) and vllm_omni/diffusion/models/sdxl/sdxl_unet.py:835-848 (load_weights remaps ff.net.0.proj and ff.net.2).) + +`vllm_omni/diffusion/lora/manager.py:380` [minor] — This diff adds "unet" to default_components (manager.py:374-380) so SDXL's denoiser is scanned, but there is still no `to_out.0` -> `to_out` normalization anywhere in vllm_omni/diffusion/lora/. SDXL stores attention outputs as nn.ModuleList (sdxl_unet.py:151,175), so live paths are `...attn1.to_out.0` / `...attn2.to_out.0`, while common SDXL adapters target `to_out.0`. Since `_expected_lora_modules` is derived from vLLM get_supported_lora_modules before replacement (manager.py:85-88) and activation falls back to the last name segment (`0`), those weights can be silently skipped. Please state why the attempted normalization was reverted and confirm with a real SDXL adapter target-name CPU test that `to_out`/`ff` weights actually load and activate. (evidence: Read manager.py:85-88,374-380; sdxl_unet.py:151,175,243,256; pr_context commit timeline 632efb42 and ac3a2ce5.) + +`vllm_omni/diffusion/lora/manager.py:?` [minor] — Anchor verified: manager.py:380 adds "unet" to default_components, and tests/diffusion/lora/test_lora_manager.py:19 marks the file core_model+cpu, so the new test_lora_manager_discovers_unet_component runs in AMD's tests/diffusion -m 'core_model and cpu' lane (.buildkite/test-amd-merge.yml:21/44). gh_ci_read(4825) confirms buildkite/vllm-omni-amd-ci and buildkite/vllm-omni-npu-ci are FAILURE while buildkite/vllm-omni is SUCCESS. Correction: .buildkite/test-npu-ready.yaml only runs a TTS -m 'core_model' test, not the diffusion core_model+cpu lane, so the NPU failure cannot come from this new test; still confirm whether the AMD and NPU failures are pre-existing on main and link both failing job logs before merge. (evidence: gh_ci_read(4825) returned FAILURE for buildkite/vllm-omni-amd-ci and buildkite/vllm-omni-npu-ci and SUCCESS for buildkite/vllm-omni; .buildkite/test-merge.yml and test-amd-merge.yml run tests/diffusion with -m 'core_model and cpu'.) + +`vllm_omni/diffusion/lora/manager.py:380` [minor] — Confirmed: manager.py:380 adds "unet" to the shared default_components tuple that _replace_layers_with_lora iterates for every pipeline, while manager.py:370 frames _lora_components as the opt-in channel for pipeline-specific components. Grep shows StableDiffusionXLPipeline (pipeline_sdxl.py:110) is the only pipeline with a top-level self.unet, and SenseNovaU1Pipeline already models pipeline-scoped components via _lora_components (pipeline_sensenova_u1.py:543). Prefer StableDiffusionXLPipeline._lora_components = ['unet'] over widening the shared default, or document 'unet' as a universal denoiser name. (evidence: Read manager.py:368-382; grep self.unet across vllm_omni; grep _lora_components across repo; read diffusion_worker.py:342.) + +`tests/diffusion/lora/test_lora_manager.py:600` [minor] — The new test verifies that _replace_layers_with_lora scans the 'unet' component, but it does not exercise the real SDXL replacement path: peft_helper has no target_modules (manager.py:351 returns True for every module), and the only replaceable layer is a dummy _FakeLinearBase. SDXLUNet2DConditionModel's actual QKVParallelLinear/MergedColumnParallelLinear/RowParallelLinear layers (sdxl_unet.py:129-177, routed via utils.py:58) have no committed test (grep tests/ for SDXL returns nothing). Consider a test using real SDXL layer classes and a target_modules list, or commit lora_sdxl_test.py as an integration test; run pytest -m 'core_model and cpu' tests/diffusion/lora/test_lora_manager.py. (evidence: tests/diffusion/lora/test_lora_manager.py:580-619; vllm_omni/diffusion/lora/manager.py:351-355; vllm_omni/diffusion/lora/utils.py:58-78; vllm_omni/diffusion/models/sdxl/sdxl_unet.py:121-177; grep tests/ for SDXL symbols returned no matches.) + +`vllm_omni/diffusion/lora/manager.py:85` [nit] — The raw review notes that vLLM `get_supported_lora_modules` returns leaf-name suffixes, so an SDXL `to_out` module stored as `nn.ModuleList` contributes suffix `0` to `_expected_lora_modules`; the downstream vLLM checkpoint-filtering effect cannot be verified in-repo. Since `check_unexpected_modules` raises when a checkpoint module suffix is not in `expected_lora_modules` (manager.py:85-88), please confirm that `to_out.0` checkpoint weights are not rejected or dropped by this leaf-suffix behavior, or scope a follow-up to verify the real SDXL checkpoint path. (evidence: vLLM get_supported_lora_modules returns leaf suffixes, so SDXL `to_out` (nn.ModuleList -> leaf `to_out.0`) contributes suffix `0` to `_expected_lora_modules`; downstream vLLM checkpoint filtering effect cannot be verified in-repo, so this stays out of review_comments.) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v12_hybrid_val/pr4825.trace.json.gz b/eval/dataset/arms/copilot_v12_hybrid_val/pr4825.trace.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..8185d31d776e1d67571e97cd537f1504245f0d89 GIT binary patch literal 28905 zcmV({K+?Y-iwFP!00002|LlG1cH2gp;HzMjJ}b!)L7-5$Nt@Y89HrYQH?Qq<&pDP4 z2?C&m7D+GwC|i?w&pyPy%l_X-*(cfW`>F~T5|Btri7w5a>9z=hKvjMB`)_-rNjl2+ zjjLq+l#N_SuwbTugcplf_xCq@_0I$#Bi<17BO6fpTGzF zr^gKD7@leMOm2CW*JGjK`0gIul+T0NU=~bcdivgEGNr3UJm32Oe-uG}N$*ece1XrN zEhZDVj<3gu&$B@mUnTKJ{M9s#@Rq$eO!4D9$bxCP#(A0-@b@^F_G_i9%MdGeoeJ~;nRaTwzJV~u&X z?-87eIGfS)Y}0rB&*im2agoLOMLNOnnC;NH6c>YOFsDcT%x9 z0&dgW|MHKY-+ce}_17=y4mM_i>&IpsIGl5r;T`>7@;J-;fBEI@=^MPqbRE;@BR2N! zXyhB5?zu>(adpesuuLAp(MN3La_;QmVzUswGU(XDG>hVB5ES%IK%Ac8^-SxX$&YPz zY&(P>=^_hbiIAh~g8sBE*TKU;Y`Pio`ta4D$bvA||Cs}X>5&%!91op@Z+YXuH%9Tu zuskms8_Z^G%meP&LF_Rv4kCx~5qC#nY?`6rMci|^X&GMNI*yGSs#e8)$8pzNRp*KT zVLTW9_le3N1p3(!*`?DcA_zy$e5Xd;ag+Rx5u#o_+(htn7f|A zoxrk3%nzF@atzb=*IUv&OV0tJ+tvh_A333K`!N8`wEWl~Mb6mt{2&^KCTFo}8@6eg zrZWnoz!;gn!vSJ3AhT}=rnPiR%(d188}O*M<%}(N9Iz;Mm>rLRAY79LmSF*o*ztJe z0rp3uQEZq1Hy;+u0w9af^+%!2Mottpm%~hMn(M7capFO64xCyS{BaN^<8jL>{m6^L zDE5Mp0XXI`!!%$mmXF9Y9=moNM}{4nBgYEN(6yXkJci@r<9K9qXJoPFDqYVs)?4J8 zUw#_AIeq=}n|Jy&YA(sM4TJk59t3bO$R+@8z7H7Ygb_1h->`h{#nw20w*}@H4%oFu z0Q}GjA{WrHxg_5;)`a7?ufG24Pd~qX_v+=_l{NW+6O6pjkDbu-;}Gc0=9W1cMPoP~ zlgDmojzgaZe#A$HAHmZ{BR?GbF+kn5?D5{`4_jypnXS1I6;syRS z5W?-ni(~D(uV4M}>ZiYIFaLJ>^82^i0Xu|u?2i`7B)UwBxL?Y|UNDV%VX}XuwSGIB zFYwzB+W-4s|67YvZI%{V7Wabed=YCOFM>jQ`AQ2Wu%GB!6R*!tw0R0rN`o(piJR6)Vn*pjhO%*nANJ*U#nU#dI2E*LeLM{QC8q{al-830~8}i(qyhYe}vJ5sFW3 zc(%8g#l_j)P%Bbx7{%jYF#*w?PUqu5)f;}f^=5!W^EoUS)__{kM4O~Rq>UB@0VbJ=T}%VO5ICMAEesYpo>QJoU=tdsPDK(- z;PiqyXb~DbY?6dQk)*Q|+(R;hJ&YznUi7jcN)|b6XcmIHA-4T7f#bu+Ay|QkGFXW` za`>1oCJ`(FkRO*McbU$MUQ&BF5O2}LlNp1yB_1WwWEOvf{gDuWF8N7^FN{c)r()g>LJGC$&C|u zC_NAn3|~DCk_qnm>mR@E|MdD7Z4v;g=-Rgn0SDT0wcQ?`7CZ=+3p}6R0#C=^G@t$v zk!%{nK}1J%7ru(rWEAvrPqPFLEy%SfO9cw$2psqZwnu2G3;L6QA;NsW|33d1$MX-` z8wBb!n87+SeSWQ7EM}L8Z+TKIw1ad!&H*Nt0a5~0*j%%0qsJ`+xXvm~Y6WZ;0G@t? zCxnwl1osaSsKXOD&vdSxFMuZC8rUB}QDi^|NsfpW04^TPX8Ml=?86~of}q4mNB-4V zD;-_yDs5J-(su10$VONt-aeS5Sul{B1i_lq$>#%HTD5$Q*koQq%Hcej$M`@U@6@C8 z<7@!tR(L6>SYSsE^bcFFF&JQz@wgaV#FIH-)Jv9%=`47`Q^s_FUNQC4c;bGWh#l%@lernQ|A_s;uy z^l8#p=rh1ixEbyonWk|pIH>v-&MKS$%f&N|k`gh#dap#_hZ570b3FMxo{R}2wXf(& zzx)&z?Aw0iW&lF3L~-aBSN2` zdVY0I$TCi}@nrFd4K62fGV7;TNj?A#0(j^}QV7u>r6H(c$mkkJFcip8S0I2WKqmo4 za4a{9iuf$O3I+>Pt2SJ_iu(dR67GbT0oy$o0+4`h17ACj2LyTq*WpR2#)!Rx3M08C zu;~;ajeA5+55hr8JVZ(%Q-PlIqOtaG6?P$k{V0&TzWNgT1|QSxQqYaX88E}*6DxO; zBV`i0&0|2PbVl!tKLPG{^713pvVdm+CjvqVfkfcx+Y>Vb+#k+S(Jw&ly9E4wFFxt` zQ3pDcr+B0_uFn@h1IWipCXmcPgd@mt4|o$QJt*9h$pm#2Ks->_0NF;@AX*!zdG*5& zKazNdpB!ZI2sjw%eQ|h+Y6Mb%*nr|4F`_%L*5Noyr?4Y9gaO@7x-f_^`hc?J)0|$Z z2pD)}kf5^j_h7P!PqPeE1<*Y}3DK(GD;|_D#^dA@5sA?EhUMexObobd?KnNV=Xt2e z&KCu-k61~1xOIT4yO8y5e2Vs1Jqr(LZ8+l1ozGkp$t1twCV`^;7M$(+#Z4@=>o=fq z0u^P~Yd#kLhtz@F?j1V z$p_~#91-dKc?SqeeN_Tr89HRmcYTTef(H27Zjet1+V4Q8LgyWMm?d-AA;=YAzJqeD zu!|3Mz6GZS23{<9m;8p#W`LwXDuiHbLPV@Deze}NL{PwL1qasFV2s)(T=8sAuBn$_ zZL+SZwqMCUl4Bt9_RsdVed6))24>Hl-cG*`qAN61qWTjSQLwM|dg#6rYTVLYOV4j# z!ymKb`+YMgd<`63%<6~>O8a6;*0{90(T44jx+z>{C!}5h9~K-o2ZbyS7eY-25N=gh`+Djki{Zhb3dNG`RnPoq^Ff0EmVV*V?tOjpo);s1MHs}s@;OnHL{yP zPo?K7*Tu_MhdZy}H)wB?nRK>Sy7=Iv zz_bFL3|18PgE4wrgvuTx{{@VIkMl`_Zv$I4MoT46r_#+tZZ9%MbKw#rOK3iyTub z-hT_$d4xW`QkM5G(~IjsEF6t8v3mKyTob|@ATOOG@Kosx$hC%lQ0uGD?1UB;CV1ds znorSpG_H1o&Z2Yj{*ciPwlFES5n3G5lO(`FN-w${^2Ho|xl4LwY1YH}k9+8}lo}~| z3g&Sp)oj9fIBR%MWoI1{^3_~!*GWu%6rs`%jiETNHvF0f*Ql){M#i5&VMCiKMFSpP zi}21Qlh~ET!Y));R#Hc{L{D|>(?)4@y^IzomRyS(I=JCiBViUxk_4#Jk6`O)RH8wHM+rwDeDVY-LU-k*g`4LW8R(DV z05lS?fb%u#!Y61gAp}AECA(TuFmfOer0ZDwzyGH)V733Mk$XZrrJuoJzk{FcYnVTv z9ni|?&kMq6O@xgHs22>>>4(564v7uS(GwIM5&O7S-fOfF7w8ft?k;Xowl3`iS1k|{ zHwQ>rLr9<*^mTV1ERs5Wh2HmA`$nBf6G!n32#jbjuCVYE^c%pX8rsd%JWob|RCO{S zmi$t1bm8HfEv6%6BZQFxTjEGK56GhVgR}fP{&VvGDO4oF~ z(>IS?tn}cMvo1w93j8VKJQo4Ew*hUYfxuG`rS**WScA2J3!@%e`KY z{)>0KBbBn>44VC}amD`+0RKnvB>hOg)ZPM9&XNKj2;5i62oe1Sf}?xB1-TDTES1gQ zHP-R8Z{myKDghoqcf%1RqhxZ+(_hx&tev-=0eJqd{Eu`NJ?*K)vsTvn+~QgF4L;j= z7H#s$$Fr7q@u=fj4ZQl-jAzx4;Hky4>T7>w@hlW5k1!+YY$g(l%QWrZ_PR3#>1%jJ zM6@gu&ye}1*&v@K^Lb2pY+CIeUANCknvEYAK@@IPIuC%1X+yI9!g4WfJx3e>Lj>PJxqTu7xtm?C zr@yZ7QQ?gmuCKBWg~vMADAb1#7V@^z>4|?L#*mP0Aj8snP}#*4hv5ynK44I#*$)NT zN!Q!3sn!okasv2fcy{RZ%R-{)I_8pG%NGXCC ze~-udd3-)u)N*CvyMaV@?etTD@*dR{kfr1y($IfK9zZziLvc2BpAkV!=HiNtORrIL zkEE%PhJX%mm2BIW^i5JgC>eqW2z~*|*#bUhq$?0qC3C5AUMWi@6G@^+4K9(z!gERb zntXUYYEttr{Ru=~CuQ2)TTpkuRwA&ES6-rC-MsW&97o^DbZ5{Gk{N030Dpk@>mWlN zYa-C{2+aG*Vme#C`X>F@yo55KA!JLzU71bj+ow$QwM{SEGtl&Eu$Z-ftr(5D9-{&C zbrVJd4CZW4Zv=t#vpqTnK*i%u4Q&b}uY3}K)hcHj39*N)qQ*IDGQCYbMkfc5mC(g% zN}q`CG!+uOSE?Q=Hit$zs&s_NsN+hxE@+Mz3(=F%*+7t?YaDD=ooWFpy(^2wFUpwO zK1Ru8P7I+HrmK@E?vVgSae|QpI1zGv3xA1cxjm12$Dn^T4^=9a4Hly#(^2=a+AWIf z=y(brFYxhgri(t1hcOU`F~B1v{OJ#ny$~iT^#K?y9M6=DkOiCvW&Z2h*R$(d!1Lop z3aU(#JC#wx?pgxvV7VZH&J+&tGQQ3aMRFXmBGMiOD;Ho9QbO04d9S!svfpF)IWDbK zhO$_b5l2wwQd+$T@E|gj+K+gXjGi88B6Nm)FIcHukTD=!DFtuxiYM;l?qKnN%)}S> zfe(jZIH6D$8X}xyB-lib9GuwrlHHKf*Ntk5eu+=%XDsvSV$Z-YH*mZjfC&c#H^*b{)Cxs-QfPb9j+Km>05B|_Ov@HCIxTXo zljW4POG-ne%9^?Zx31+av6us3^#ekWI3GxCQ4Z2~*+OKM;GtbhY>_Chp%DYPGh9(K zF`q!!Uajhlg<|`pqU9n^S~*?dMaN2S9F}G^X_NV4B=ote8mi_qsB{K=_Z8YsG7|;} z6{+flFxYd!D^@8LfD0s^E^_1*7(e+2a~_dlyumx);f)6yW=euxWE{#;D?m8ZLTj3P zS+s=~DM1%ZUy7(+#e?$8^8gKyw?J7qGA^zz)8V0+BDP_5^}Z>3yOsyY$=2UWn2X)_tGEmI2V9<(o+kAxKu zHD-E3&WrL}3MgJSN*4c_jzj{BbSjD^{^0~&p~C^9!C{5l3ZRH3+oJ`3Eur4^q!X<( zeO=g_D^Np>S*BdU6g%=rxAkZ7HR#gXj4rqF{g>gMT48W&2tY*2i9Uv#*+`(|6v*e| zyrx}LhgC;NI)c5?t};t0aF#l?UHpnDBdy61If~3E%2*%l%wbQ!N1Z}xUqu^SonB<=q26}n+QkATBJxgc3GOQ!9!4uV;Bo=`m%feL2o9Z&*2Hc^}f4)TlaWaa(?AH~D2Lm9}S> z_iFPsxN0E60+qDD$QfuGJ*s1jlo=A0$?6J!N~S0r?J^W_q77?3LWXzNr=(s`M$}d{ z6Z8OyEM(+Dr1@kb%a+uUgSwJ(sEM*i`O{J_>DNj*OVvfGCFo5@k76Tv8E9g*5Bxnr zXMdSp8

8pa)7Ju;g?iwYo}aR+$(w@AXfhq8eBzv~U-wfR!QvbD~v=zG%E`S5JF4 zv%$HY-0aP@lN;01U~@|mu7kraI-`J8*>U7xlL*{l%}24)r9?7k?9L{OmlsW|{{^4r;7c7~Ys`S9RySVj_B z^9hDa&Su!y_@HLn&H@ zKJ5&-+{)1?S-aAz^A!`3bq*N9> zzI?U0TW*p5%Nbn;?!+j zDY%NDsP=O|7t!#XOhIkfZFCG(J4xh!7@!8DP9$?h*Om8&yyU~~P5|<=Xyg&A!p;tj z2>92{eI<@fUu0W^7tP9>YIkeHQRv~=K;Jh$%~~wDZS&!i!jl%=j{LfN9?kPR=Yy@JF5T#zztyWCz4j;Ct>9g~ zrGEC?dVu`(X!U`;K&96ssyaj7Xr79=Vl6{7m`u~0%Ed>pxhaWj%64fec}L<}NyDYl zq`F`VNkZR4hd zUiuNVivX=`5iBcTMcoJb;*mX4*XBWz+~55ANR;vxqL5py9PbtT z9H$ekO)S%c=4k$9vp<-UHW0X67*DdeO=zHW;+0FVDshhssp$(=N!$o@$R57A0L+@) zG0)LhkDgZDJhP+_-IGy^qAS0h%}~<8El~nyJ6Z?t@sg(fCjP;1lQT-=t}cj@a=d!B zcNxcX5R%oG>V@KKBH_mOlJ3OUD-|tgQl6gui&eF>l(S%I64tszU6n04D9m6AKHl;c zQQMMlU)%C8T|3gC8rZs^m~SbGtkL!Q*1pRmJjUW6#O|9%cmj{R5b2xbzMvc#(dS6? zw;_k1Vx6!sv98ZY?X1G=U=<)(8+%H}XQauu?D8#20jiRa{Xce~j)YQ$A;31EZ<-|0AFia54>!prU-5#hkxhR0ba5Gp!v(<6D=tWf-I6X&vS{0s zam36jZTxv8jpISlmN(28Z|CVR(^;6s)WE(u%RnjnBAo))nuN#gvkht;%a+F$R3y|r zwE)xI0C=RrS9?MQN`ZYbt9q>6%m=M|;{VG`zsc}rhIUH`O8;5 zVx#U02OCF64^^gxtVF*tP2x->N}N@x5@(n!akhQB#Er#y%`=ku5w$r|il>q+DV@&4 zb(8OgWk&gisTMcnPc$Oed=MO|J@r8wuH+XC;b2?x zAW&(Qx^QPQL@gd9IxW_dA2v;VkPB?jb~yQylIFThVAvBLb7}KQMszHDghz4INUkxJU}+vl(;ZNoxK}m^PD+O%xd|!Pv(-s>E9dp^NkUdVxOehGGb_I5DXkCZ4vY^n!HDV|x%Hm4_K8xy@X9>2ZEW?dTnJSIATIejP z&BU9h@W=}SMA^G85H&G1&q%c;{O7}w2w}-itV(d*-ZO43S+<*GxtnCUn`F7xBugO5 z&tlqpkzEf|gjo)mFF?595~FGWyqNQziP5{*A+h+ix<3>n=5hA$AWLPBMQW`mZw8AO znHP3Z-%DomMKLJS%b4c+x{MnpmO+R}YW0O_`M!n8D2p7|Vdyqxes z_0guuxYzb%?$StU^z{D*TCJ&}cCqqP#NC!?l^a2Q`) z!%{)e14Uy3^n^}Heo4zpW+7#LlsFFD5lHAF3FR?}6esyNFlBT<$&X~J(FdAl9gO4R zdN2icnV=g!tBwk2>*jgfdn?nhlXIY4v8Wj2*hDllkT0I6xu8_~u*fe;48#c#ISI%~ zEN;J7KW zRnn{wr|Z&ZdKg%}Hq$-tvv_I@_&FG(uN2;fE!PJ*-Qwz=Nw?4nOR#1sNecm&+J>sN z{3h{UtrdNwi@9pz3cr$_F9#q!i3Ze#W6(W%?_T|Qs=a%2da4~G13sqLI|sHc z^z-$`U5(bi_iRR-NOL`VY)2kvrmO2Nvtm9nt-!Z{nHUb|XR}_fx5*mLW?z2!7we>45{3_=7gf^5DghFq(j@(=5Eu%k-Kf92Id1dewP;cr2A) znR|Y=M|!K?Y{RVodU-`LRnZl$wIjHE&KKI>QLzw7EYcG|nL{f*!DR%3(4(b4cynE~ z2AE}?-i<3g(mGnI!-KyR|9_Y4O8B|381Vnf9fPLU-5LoMx?DLIM0ZT#+||WX zU(?zT@ISIhsAIL_DTlS+$ouJfwG{;jQe+KWucCZAU@J*?CZhH$1$y z@cMgdm(TXp+RM~evS7;_Cc$(RVKhI*0b`)y9$}K~F zeX? zp!jfbwinJ9XM2Z@`_N@8ZySxlhAK?wsDT`x2BcX>szq1+mDvKTjmqVkTkVb^*1KBO z-c?GFrGphRqk&e-zi5NUjiTnudlofc-Xdz+7I+iOZ0IKIbVnSmbC=Rbyis5oBiE13 zv97amIOf*Ki7j(UA6e^(LK9i*R@6Z($2)ShFYzDKEp?D)VBkQ&VnEGfu{`b&bJ0d% zG@-JgoNWZaZ>4iYc%9JjuSGv8P=sOFpcXBdtF@W96eV)BC#fYs&oy%WqW-t`)6eftkLy0SdOpQGmGwSl zOcgVuMc5U`kkp>@51HXzCIyM~5ViKB+U&<*Mvj<%nea=wF^#8*#yrUtjv1p7m-NI` z`_DK_rPfkUgu}-8Rkpv(=++youJtUN(tP=7Z~B@JdgTUj>?@Zg5LUmC1#h+Mhcz?; z(_%RSPuU0u=TbmZ8bit1;d#qQCpath9f5ypj~V}Wz(-&W0eQ^|vhzV*fvoGM<`|siaL%SceS5q;-bgA)6 zbC^~cZPMOPf>Dgh6qQqf67}Ey3veNfhtOb9pxGX%)>Nw2>;2mqC}XdGIQ{zVsrKEg zAAZnY{iMD7+bOQ-+tar%-@JPL?$ysfX@5I?bE@O5_+MP`B18JXPw(WU$z&>%9=PtNdure3#bN~OnD#M7&=avfF_E5TcK4YNHZix{{>oDh zmyc9Ts>3V8;Sm2JEeW*rc|q(ctLW@(?==`=S&)c~QSD+~j zYuT{iV?v<^>o)!JuaHvppG>< zfNHnRO$t{V#gs?L5HXt6Jd33)@5V{|38xTRZeY0)>&0Uy>T!3(d%hjVJ%_Qe6L@YA z+u_P7ym;}=>32WBIo00&^39L0-o1PE(_fKQP_LBtFMs~&-J4h6{DN2YY1B%LlTck%=ZBHYV-|DU8IAln2+c9~q9A0(p7eVUFjohE z*uO}pasMyByghx>*OM8L6!5q#`P2*3c``{0>ZFNATwcq{o@ZDhzDI#586d}Jk%t=& zs%-R2hk0NDi@<|JWmb&_Vg_uPt{t9e>pQryXj^&ln|}0KsxdA_b~(^q{_XVT_iwcW zc8IfByHn6{tZR4v`Thq{GM&ou{~p*}3smTX0#m1hiRhmoUZ0tUQGLR824mw){H=KHs=zkYeze}xGCBO&-p ziQn;6FzH7aL5qt%4j%1G%pY8tY;bWs%93b+_h1TtUtoT}J0{kD^UF_zH>aC^TA~oAACS^1)Wa&za*APYDh0(FaBRmkjcfL( z3>H=O+US>QzGe0CxhNZxNriH%yzJ#F2ul=Ymx)D<`SS-=mHwC7ci$b-iOZrmeDT`? z7DB_eRL)laGQGGCYL&0TlrDWwRCP~XJ%lY?tKw@8V(FJ-SVX~+sLL*m&uTN6+H+2> zUK);!?gcRgs7O`b53QBT@w=<>%gk6D6GDwYrzD2`P9z$7C4yeN97yka&z;PUI1s7#gR+`m3gouF}GOKVUen~Ol7x=M$cZEA|4cFG9C6mx&U)o zZocf!Tb>}*-L*||5E-Y6LpA#bVWIC^q~%!qzyGHS5NZEa6XnIHsw)+`Aoev(3BlB1 z8Z`xf%4s+v(TI5VK%ILC+(Fn)q9xQ3v5ae#VkoLb6ICrrx2TqJWwWVrZ`cT`LPVex z(J{U2RHgMHl8JBYJ|+7hH<|Vc<{+cxQsLV?&9Q;5XwTKFqWauP_WXoleLLWr8opIG62R%27DRrw62cvR& zWC9eqn2=@A8q_nyRouSlPqSX9sEzvI+}g{o>!$8`16{wkX-YuXZv>rZ*K0m@c~;g9 z_hZ^VwWzrL)=~(mFle>>m9ON}=GW4|2g`Rl5vtO!izJJMOGSt)z4%n%SAY52&p)2_ z#TZ~>2K|l^z@5;o`p~xBa?2zi{7L7KkhQ@tTH}E(K3fL~aLRdgiuh@p--t0n48=-% z`|qpC^VL(TYq$TtKE=L#K<(nKo}c;H8S^>RpO%ivKseeAmK0CtB;+uVFSkYAL>{Z}6K8djM*~^)jPm#h{Rw zw&%ibscZ(VbCk?*geaQPgCXndV%5n8ODPC2NA;+StjFiFtg6n)rWl$v9@Sm}6NwJ%U7-Z5IbD``aL{Y#stK65f@ zOBrcN6It2C#shb_b7Y!^i2YYN4N)9U0<;Wa1oQ zHt`B8Ry0wuax65c`csB0lT>7%48q&ityH@GNn`SlmQAnb$Z?LD)Yx}AK~*n0*)pe#`h z0?YE&ETBOif5*lJr`Ws!Yo;)MT>W10An7-drZFJFu)W{liMHH8ka7_g)XBB0M6lChRqb~bC-zRUa5FPX_HHLrt*c<+#WZ2BFnuk zMH`3h(L!Cdl@9Z8VP zYQ>4f4U}XSz8sLCNrWEPB`;vfML{t1YeVbsdFi4|S;@tEF*&d7Iul^Z4nP;%pirXJ zNS3gW+{7wPSx_y~3(JSsm5}*$zj9WLV8vL-D)?#TG!a_Xsa=(g-qA#`pWy09sIHr|uyEH`bd$P5dvjVY-n;yS z60^p@Ek?j0D%US{6)4r*Zsk>NHJyo~(>m(Z?I+5>#G*FQuxoAW<3wZ))RYFojuycz zT4a)Uzv`lmzN;0%+?Y07N=(%+yAs3i>w*`BEu9xO#;ZxuPCvx4X@MeiUk&;X1q&Sp z(G}uERR4Yv1w$1W5fc%+7sY)fSz14;YAa`Rqe^^smcRIFD2sGjNKMxA)U{ON2&#K| z=eSM4U6Irbcg>WxtfrZF>b637$*4@VN?im($mGFbEEbzDDZjN1gZP{USeFR=No24Jj zC2MWbCE-%*nS~lNpI33Esu5#zv7Mzt!>a8QI@WOm2+XRVAFLba&>~*o*m_tl{@o)j zRrJTd2qHC6ePbQI@?CaR4}lpSATLveohkLCq5ad#MbdQHsRxbjUOhjyWMJCx?o}5g zC9roJX09$KOTtt{SHQL!$KiE25zPY1Fw1o_U--CE(atsezT1Lblo?5x?JWQ|9(%vz<1b8Mx=#<2uw#?4!F+T$f8(@RdJjd?Eve@9h_4u%)@8RL9^jm*?<7-LX5wV(~+xa zXBqpF-R-LBqJR$96wt@)WDy4*s62=j@@~-zi4Q6ls>(PN&A24XQz|3I0$0@#%$CF@ ztI%$D;VT(d8me!#ym`Vm=?B)n)LQ+(wAnJjp4LW5^rClVRW-BHmXU;U`J4vQlBC}+ zW$6Bj>q|O=I=eKMez;a26}`E=!R3ioiS%RJH~O~E2-j;;|JXB(zGtu{Pu5A(t(J%> zzT`}6dd(WXGGRNC+eBF?sz+iqnMEdss4ieFXVvC?+CNt9Z3Nhgq~tfZ&Pmq}x+bC5 zddO9^ETk~~@a{ht^2Hv!?7Utub+8<%f}{4VBkiVO)UoS$hte4av?Dr%SC701RyDg) zgdw}9MwG%HQ7w47u|=ME%egK)B=;5OwG983Wp8!NYU%KKBIms-vssOfs?tmvjZ)R{ z7STjgYAdKniZZ;)3$&H5PYvU2A*Rz=S>7=V@5Eded`lEx2&BKxjtb(5`J%TV^q;K3}5=(J05d8)Rm8QMFq=oNO zSFUP5pl;K#)i0U0Kp0$pLf!gN4NqzXn`IwtM8eY1E=$nGz=k-E#A!AsNXf#{21SS! zuIj_1nt3RgM2+@aW1pD%g5L@y4Hd+wEY2p-*SK37OBJm=l|%%YNz0>F#HHO)F)5tN zQi@*XFO###+p@oK9Iwm(6V1?S(~yz!dU9rkYA?2WAmwDpBA#m2F+O6x7=eC^T5lg- zU8G?;=~LSsF$7DwRPgL5rvnwwYSI>s^PmopvfN1eXfG*)5fqwqk;4V7KkZ4qs5!8s z4BAmz31_sU^gT)ZsEMuHkH5-=}IQfp(%8kLcc6#rXy;E&@t{K*ALiUFxTGcY>IPCtQTr z759^E6Wg;PHj~HY%2mpw+(A{{T3>rZo@yT{Wf05~90GP-ZcH{#5Z@xMx56XP2>a?i z43Q(2yS+g&ZslCwQdE|HW@TZc)z}k$;?jLAda=u~S8bk~wdW+4(?(@JFCBNTRl{25 zGQ`!BbDpH`vEqQobHHw(OQ|O-2`3_Hq$SV75;A{DXt&SAZfWg&BHz<1ofe`6vk3Mp zBvbJm)sA_YE0Jnkr6Eu&|5~QPK3rK)@VYB8L3ml!oDedRC_ka$YF#mxDs8{SebK!^RnH?%-Px2DL}K)pzG^hKBD$qM$BH1-0px9dCn zh;>~gS=7V!h}ZPEK6SBDNfBixsfHoZtT!Um6vd>J?Gi^z>cq=ety+Z4+y-TK8udYC zARdan$}wZRl2}4yl7~M(r85iP85(Iq*>9b0_Wib^xl(>Ih4u|*ZViUHHwUNArmaDZ%$0? z3P#f@kmFO3wDNkX$*m$5R>*$l62nxFSe1t;*g{d;bBzUbDi^m69H+MlowqE?xg`Z+ zowoExM0J&f0f4m;X(`03#?pB_4Mf8i%;1-h7Zq14V<>Zxd_MywG7vqtG28y@T+Azy zbtgr-H1@Ky z`lzzmaOq{qXkte?!7PEY_JT^kgylh6@m^WrSnBAQhWp#zKu&#L%#^FQYQs8^nSs$j zj=S|N>xMJ(!k8LCcX;adfJB%0-1H7hNt^{A1%@d(x{~!n`)T zuW_1vbzrq&i;dy0)d)p#Fv}AQ8)RA^LMTTNg4TO}s1lxPLk7S5uBTeYD+;Mku3pz9 z1ASIJrPMpi&}%)(m&*H8Pjq$EA~9qm)+}doY6lHWmMGh-XC&XgJlw-u2g~17EYlbm zkA3vJ2FADcx@wyQU#U9kwhrTU+d0&POmvmTV$Ra#A=K>w%V2gbMgjou>IA+-YnmLL zw~dv~wPp;jeJC^uxCsyxsl|emtvzU3jTc;Rs5GoNLA|;&MU!G|NZe6vlJpJ1%+-X= zmeJDn8P8;HlMzqsMEXI=x)GJH@ws@)(7-vdg60)^hcp&?WhQjT=;~@n_?knkrEh;l zg-{kHEwG$BE#{DIIN`dTy)2*Dk;L%j89V}Eju^L&nC+ZsRVTF-9ktA@6mJ&^lD86n zORb<3`PRyWd(lpv7{Er#gLu*^&*x!k#?=IYd( zeP4$2#fo2P&B`7%@e=H*{d~pI*fj`htZ)vmMGT*dHCJ~>Mbz0ncC_?}3I^MjF`&An z2(_}dVoW}Z>LCXlO1iT5n-dK{@Ka)t3^3*J0HD zlcHX{rF>;j96+-z5?I`VdvFL2K^NB`AwY2VAd9=ZJ7jUU5Zo=eYzRvTE{ogZ?)viG zd+WZc_xDwuA5*7#=EqcZS9Q;v^DcL)4a**A%1F-P{c`?if@iX4txaU!fQw z*}*15!lokAmvX34Bw6eIrFCehJrmRS{&K6ZKEol)FW3Vpw5qs}&Tow7 zd&Gk;S@!JiuQD|;TZMDh7sGLF&R-#1#<}Bx^Twgpf%&1)FGil5bN^RGQKB9EyzBjY z?L)1;H`R7jBO~X^ivSLZH?=e7gNIcPS4sm)eUaZkxl6vT0Q!@W1(n*PJY&ao#sg&S z>?)%rraZ6BtZS#YDw;M=PR);vO7}nN36c64!8-@h`oByC{s8cf5%2(q{Xg=FqdpCZ z(2BW`vYs1-xeyeT-y)a2j1MQ)L8t$PHJ6*VzZ;wENRc67ygiTiQ8a-s3a1(KOF7f8 z=VMiZ6>D|?p*lJ516+G_3A@dIX|ZZMAgbzJfqu%rk;n9NI{Z}aEw$se>+7k^vDkiz zdFvP_nP&#p<6Wv*3-XSH#9Y1s-H{pdc52SBR2Yi8Ey;Goy|-~fZ~ZhA_4 zo_Y3lm0`x{WRKJ9%4a8WF6PXH!RZPo)HmazL;3zGOjFz`H8#k&v!XBs+fZ>NR;f$5 zce%_&yyI$2S};Th;0E!T->&Pk$tO|C^3OKQc%9Z@Noppq6p)AIdq% zJOl7NXHB{e>Wr@EYai@&o?EJ3mae51{ulqx{di@c)#46tB70YmjCP2_9$_*0xSA(6i?BGF_L!=50!o2 ze`_u8d^vJu^Asiu(*J~_L4$7m*FDJSvm}ma%F#y-D^R!e;j_{{ao_hh^Lk@OJ?fRA z8o3u$s^HmLYtZA*tu5?jZ7*-ZXMyNi?sb~kpx)@vfSP$^ISukMWv>tUfjUHPX)CL6 zzgzJyx*dJOA*T}lj!Nj2y@-%_V#0hcrSgE{Nvwf77SzZ&wJ$|t$;hNKshN!VC!r~Czf(Funn@bO8HJx2FkbdH5&^J-rj^94-H!so--!3`hM(~oUcHA6KINJiH5^#(|`rqKI_1^}aM`(`) zy**gsdx}MsMUSR(}^hxfkr+}vV5gq1tlFi4p8*W`yBhj#@ zR9BW@#H6V!g1YgheDBIV%hWJkDQh+YT4EoSN;hW$ezj<%xwCl(o~%BP>t$ZF#D7(6 zAkHOos*A-qp?8E7SWEaq9cL==&Me2g4s)>&6O7RUb&*|{oJaN1FK+PRR zOHyb6MZ&wrgyB%#MOHqKIJfDGVR@28gx(hSU9&+2Goso7Hm*%!NlcmZJ6UNY;a;gw zGUCc4i)2LP?K|`OFF**_-hl=uqfwY!z_a`#k|5I)7(m=a#m~-LsIn7phHA07v@wdR zhj~nKduE%Em(yvn8UW9J6%*s+TEjd3LlXO_S(M z*a&zhTJptpjrzL`ruzaSB=G)6vrja&Hg5QSkA(pfoB$3-(&Ld05oA6~H2++hVS3i* zQR2yxx`CiTJHjKC7QGvOtU7hcFk~M)9LKrJxBVW)YG#g4fF)1K+bf4_fO+Cgi$-dC z;Q=iPd$}SB;pYy7{pqpPCyf}fi`&hR+M=MNAlknSaK3fip^8u+UIfhC3oP3==>Ej& z*NET0N9a==6bvKeeLYY@gI4`C{eemNI}Nu_AzOuVSV-9EiTq1FDXz>jDOUH$?Sa#i z@QHV@DCennueR45?i)2M$Qa2YSjE!5`jIV@-IO`avgf8G$sldwhJ~Xx#u0oX(gVl9 zp_rgc4ICQVjiDYs2PSPPQ&Bh0fN$Hxd?JyHyl{R8vr)%x+$n@k{lH>>EAzwen@q-u z#RYi-^SXDR9o`FRI~5uqQUSbk7fxP0$Yg`kDFF|$G@tg)w*H8o%h(}@@gY$Aao@$(MWb|I76EnEb<$N3vSa0UfH>2rLx*qt zIS6h#7lZxfMfg5u=u%R+i-P2vdnBf_YgzROeS57K0WLK=nZ9~pmI_nL>5vVNu`@!p zs;n?scrr~>_%b6h9gHoDAn0A|N62Eycf|9!>?w6S+HU+Kly#5Qw_l}yhJx2Htr1qM zD;99Oznoi#R&$b(Z4Qx)9%1agL;R8o)mQ-gbpF~V1%DL5VDjULKBfS}vjogMD+S7; ze}t@;c{GIZ?79PKTiR-8myLAx>K4C~WiTyxV_T5>HDaw?8V!2iae&gJyLU(YlO9Dm zo42HsQZP~Qh25JDRdthx-KY7n##YU)ajIzOwdZR>JW{eLI?dc5Ub%33osKet9*E(f zKhW<-o#95H+oYi&``Y#0Rz#SqXB|3J*dTpL4ba(QW)agkp`IQRiA;!^(mya`F-9YH!nJGb-FB+*LBfBHrG30cYx_s(mCz{d5{(^!q6Cf`x*~=(4-@B3xxF8Tjt&a}^beG?}t*53mY5P0pbJtYICg*6>Y`+KHt~Kja1I-e{Ywaa` zMQST(>!Pmeh2zNb%_LB8G1%v}pl#Lge&?^Ety_<)d_;K7Qo)($M}`NS04sE%tm{%z zGWMlbtO5z9qDhU;P9`+q%B>`->*?fix!&K-eow-!=DuOWfHp4J6#`5FEP6*|lsf36?+q=Q? zygQr)PPOPkpJ0CR#vSJU>5$p5n6D2>ixWYIO#TLK>|<8Tfpm^4H25pLph2H0-++IO z^f!PnLdlML+>nRohHm0kms;QKAXYiwSOe&gwR(0+3>Ji?e(V^QJ~0z8ji5Z{KK5)#s6AbPO!baTtN0o zSv`(j%bv#yElEIVs<2|_aPC7a7^Y7m0-XZk(6Ap)>TOMJv~&7rRvF~?0|$Ne{j0yx zJ1Ya+;tWQ%YU*|EadF%nZVb0E+C1J~&QQMkM7EjQcOsj*6@(Mn^5u!50$Pq(*};dA zW2z@PYT5%{$78KnIbDDFmNT||sDrE@U3ttR8Xnb0v^61i-^3DbI|aUzfo-4g$?V5d zOLy(5sP4!vkD1x!U2r8$!MBQeWW^}E4;q9wTOwflMw><|lQzo1t)!fWd6Ai$u7|nR z#F%!ic>x+dYpseV^tr?AS__F3Zp_;GrVxR61*2gN?2}6F%Fb3C>U*XiS+x-AOM%0t z*xvzjoV8u*4t#LnfvmDaKu$uKb3jE%k{D(?TW_Gap0l|{t(jO+I^(XW$ZeJ_usk~- zMsge~HUvCbWfGXZpq}x)6k8GSU-QY`@j7H*YoUD6<*$X!*^`w!3e{SO6DN4(p}0XB zRZoB$lt{}Frv{NW5_Y%R!iykU1%TF+&rX%cea}Lt#hqFqbaf#9&F=l`RyFHS``8~= z!9IJ!N$k>wJMK=IC?K<|u`A)*mt}piIG&Y@_Kk&!f!~pCGi1LSAUeNUJEqWPpfR6; z`x6NS&g^5e!h6z5@96CU6X?;l>i@rNZgx4_@-%tCRLr%==L_eofzbss~>q6)2H(bV-*TFUtwM{2+w(0+TIdtFJ z`1g@PKGam0qb#(Zbb$|>;L}Hwja2KRoJ~g+7C*MKsU#^+#2&WWT3R<@Nk1QKyZGN@ zDr*`YO|Ane;4Zh;V&$R>yt6@{IJa}!;_?n(t{O8u2H3&yJpqBp-q+6A7UNS2)BVi+ zSA}pNCR_vvY zqC~zCqn-$Tz<00VIXzNAZu?&w0$#5_Bd(FB<4P9dS6D90sqYVK(hsRyqXcO7&|nNK zg23O({4_4co8?Mmf&ukrpg%yVPW~i?Ys~U<5ss21e}Kd4x78kVNA)o>{y=-*TATE4 zTfnG_%CE<*>>1Qyx{7|SIPUmMf2Zf3BaG^12NMr$>U9KueaEfO6f}f4?$3UYTwSLN zyMx)M5M<=~@c~K6(j%~;0;%;uj~Q}PP^2$I8~e(Lyvjr^i2MY4Jx^xVqzY|N6=`+- z{C7HF-m?4^z7T95Lq3nx^{eqUS{#Yiw9#4Yu08SOe);N!VT&~5-)PkkdEutJ`iLv& z@g6cANSqW3JUP^I0`m2aO3zU6ntN0UeXdun69fp+z69r z!h9W@^mWH=>|jxbyFGMt#8xGaR}-lWFAeM~#u7i#mY;D9cK4SaG(<%@JWHk_HM!}B ztPpGVchn2MTW%EDdRX{_!tb~Hjq80<6wbToV%I@C`->1W42fVJgoSC_SV-q+Rfv#> zqcC-va6d&tU3{adAr|-j`*M@Mk(F0Om+uOgbv=pJ=M^8RKH+#mw{NzZ^m(_Jyz zVyhfc+2wb|(z0$cvhLWpd|{`F(fLlNXUsXwl!t}I)jvwF}bMl|<;ZZk{-oNTmhZ_r2H!E!( zJ0U9YluuDBj%q^ZbG&3*fB2TtD%~g(b{m*sa)~#lqPba5`|0@`#YVe1;LD*ohkIzi z02Pl@>6?Yj(e8=`0K|<{fNC@UntkeU$P;0j629mFX~MqA>|Cp9`kJ;Fzj(vkB$j7&H+DOu4n6_(a->hY;W7TO2~08+4ZV=oA1obXzQ^RugWGF;O*r9FU-GF$aO;slc(ohJYze z-g2Of0_ln-k3{>v~V(bvC`aG*xKis%f8d(x^tbcR#>igw$FP2PPBPM8aw=<$s?ioY(5==+)f2) zq?*{7kggg$4mMhUn46Btbaxu^SO(mJmU}cz+CbC%1!K$B4e#pjj4p+f!;`jB{C9;X zT%O_QpJ9@clJX`+XNQzIKg z1_M(!?YEg`quOS+%bH2a67?Y=PI@9XV0y7~dnx=JP&Q0j{po5_deShYxUKA_lL=G{ zU?6FNQ%)}i&pBgUwtjZa+!42C(esL&5)fnh_gClmFw&|I@j{h6)Olh#o2Rpo188kZ z_P)=x)%LTSLVLTwZq8z@11kA&aZA`9g8GL(?xhE*zv35CiwgfHhIqF4Dl_6b#2a^p zfAG=DOSb(}IKQBLN#90vsGEz~V#{P08^f^PkjoBn{xv-+{h=beEX*I=$Aocj@bC~z zZPmhw%+MBtX$URXO_uZe-K1xUT+EuYPfaiuJ!l({cB~ z$KrrZnr1JOhMR}+K%AtHXZ_idjvS*cw6O@VmW@SfU!{qgNc|MMK^AI2J$CpFFmGcl zQa|6$srNaF#@ELMI|+sD(_2c(N))~79W(Bn{(}+Q^5^89dwt~Q+&AyWou*n&&9ZiW zd{9R%BO;M1n!{%#OWyxqJ4VG!Yj0arhO#_>+wqoM?nZ#?LD@e$Mgwf+}Enyqp)g@c=e_b{g>g&XtHr~|;##JD0wxin9wic>R(N4+7N#~SBfT2U2Xe8_2 ziOt*yL^OX`0R%ZjOTb^T44GeD@5)hebAF+6GPg%rqWY1lB5~(6A+#ITM)-leZQQ1< z9g!!n@(7>0Ov%9GRuST#totnsohw7W-aTU!Z^&WW!(+WYtDkphBzo^(jt}ch@^4Iv zYOd%PmAJ|(36VK&N(#{~+V=p9gi-F3f?@B|usLfL^a!xesMlqEOt7!@p}tVl#EmTN ziEXfrj~Fp|;_1!W)5Y}q`2JzR+Ry0#qqX;|bdxXQMeTO#q>kFrf zPQ3uk!ndW(KD@}es7UmNinVwh(U$wdviQR&c;pVoqQ5MzJ!rs820GLy0ZjPm#=}jL zMCp-zlOc?_@^5nEP)Nt$)Hiyc{hq?3XLjN0MMyV-P9RPrl)!@%(WtQdl1i~>oN?5q z@a~o}vVZ8KgsYu5;i=IsJFhY$Nr(d{jX#CT5!!Qw@HTsU=0nYo32aFVU?mz;6OAI8 zUv#DbHTceG+DaMPs`&|3Ggq- zzW7lY*A(-6<66B`0-Ee%BwVxl{)lh0zj+B5f0@z@8vwH;JL~MDZ=WlY@F6UwM8c6W zBaif@j%5c`ce1@TDD%GFX$NyrOFSh-0;uDN=*6&-C8DPbZ0yEqWP6!^kRl;Ib!_7I zqL*2J>;58P%pqO-H3E-Vt!US3RME|%bqH%feMeq>_@`bGTgtvofkav(0iHZE+wxOS zEciXaiXfyTRSwYS+p`(Kr6iP-#|x9#+&8S`KX#|ua87H#r#N{mRZsfZJWAhP2|;+#zCz=cvq1L#Du$&X6BtYQA&r(~ zNLD_vl-Xqq!E&0G`DH6x#7PqMt+t033NN_GPdxkS!!HCaL4+oo#1i}@Xo>{JsD(N* z{+1_YVt|c1@+D%K3<`6~aDe~xr6-mpf&m{?_miyLezAC$jWz`5V z2E0_Z%CtTfk@%S0{l)dzip0|!v(Nxqkw&a8OvQ33n}DEjOMUQ`BN}rQ*WjpUfToxB zMmV2T$9Ol#hNWmcKx%!++OlO}wD1S^ZOGRce4@6H(FSm#n-|nKd!zIgo@-c$+ke82(bY1AAK(_`zAapQ{LCcT=uv^lFmd)o zob^T$sbhmL$RLYQsOLE1WZ~vno7M$^;8N^TDtM^=R34DbezN*LncYz+M>BP@lmP

F$7s?f_bkIXOZh>dwiq$t3S8P$TO!wjN4&g|8`D}FBPTE zZMHEW=bl4fNOHQ(FatDpZiuG@rP8bI|0zIg1{XrcSqXASJ&v8SBd#wVnvt zj$Rv;3=6+mzrr?Ef;~`;Z2WRJPeQ2$d3?w1v1-3TMrYD-*K+*liuyd2YGIJ0uYA+! z70oyJRecE2wcH7I(n<{sA`IVHAP3;^8Q6}F-1H~ja*I&IS6oM%PxK(jBaV(r$|I>A zmU^6u(E=>}uffEvjOlbROra*E-S6?qj4X%>JFU->=A%gVm6%GVJ1g(g`+RqEMDvb2AR@KG4^@!eO__yEJWih4OCH~;B_mk7!Kd(6vUFSM=d_r6ucNl^^~hXm1H(69 zh5<*db1o;2nnyxmOIB@HEqU449%~pV*6k5ng@v)#qq!Xb5NXnm%6-w(cPZ z-U3$|fU(gIBwxrINJ|sP6>9J?OTU=B0Z>bJBNHAWrLYWjNJwAoAIP({H(9cT5U+t( z*I}Leoz!arrubnzV*yD9>~>lwKodwWHBc2!}Q-`UcJeD35O>{lZN#R2RYVx+OA~2VQ9}nYLn(%y^ z2blCPG(%XlMSaSVb{pQ`N{FoY%RAZ`Dh=LeKOEMqoX}`4O?*KF@)x$~`l|)!Hm2f~ zfA0&^W<6BRJDBe(EcNNNxrl z6eHqp11_c!X3K0T!KvIo?<7CH8}sODMx8gk#bC5JKd35oG$0Ig~Jx$cT^3bq;z1 zZyZig&lkRWVIaD;)2-&;pH_E3zldV(L_YqPwyGu z5jf|FrEi)FYgd$6o;}V|QbHO8Cgd8xm;IAk%}7j+``-(NVN=4EkM;s3-rNx)D~=HjnA6jy4X)tYtn*dw!g)_HQfiFoc%< z3UdP>+j&QSI^l$X`89s9rJpie`}i|4KnQtqhkI~YV3`CGX$C0y`vIwhNH!QL-tT0v zol=t2lSznstk*>3LKbNtvVkF}vTyRTdKL>1o(ce`_Hs1EZT@P(dVev3Z?8pJ?;2)+ zOZeScx2?$1_qJMybY$d2)Jx9g395JF6Dg<55}eS6d}rJ4Uhp)^)1`?XH*gI^o=7fwV3C=c4a(R6~Ss ze;wbAylDBwEEYM=Go>0Tc!BO5VHYG2euYSU?x2}$%ul7UM2NU=FD^4@^CO#17V{bIG!C&bOh(ABGpMPQNW?LA&4A zA%}Yl+q{fy^Q;?6B>mL>Q0bDp&GR!WfU#3x8gfZslpK_ldt+ z8@8k4#EG;8rEu*S9ps=s4ePc)U37kZ#e?hRxw9|JeuYS}u$QcgV3ZeYrO1rYfLVyN zXupXiI#-)-)XRhaTX&SpN(Zr57ijVETChFR21G65z^p3AH4G6;QPT>ERs?nCvDVm2_P(&VTH{G3GFCT$sF zw_of9=YUO*_Ha(x6*qOht(Ot`DW$2r2?LR0UDQ1YHD7?+;gdPvO*M~2&uKHlsB{6| z$kR^%fuf_AUFk;7?FwY$504%m5L6m&_GX$Gv7=^yFp((CTu?xI&@m#l(hV?eNR6tL zzmjEnBs>z(CVMV8F6{z-t}?9*4(QU@vf)8C@_K0{BBG{=ppQ#{Xj`^Qqko-+G(!O) zsZK(98l$au0r0wiX=?G2#V3EK?K6ROIX7%k1mb;Up#D|Aq+veW1yKFz^>y6H-ifLM z`yzMbTk= zWeCN(?I72PAFQ#|--xo1V5zTo*u6PL;#P22FGS#*A>GvunihKQ~R;(Jn|BuyU$HLvMS9P{4N;CX*d?g09* z8T=~Baf&inmD^=Jk5r|;*S$Tu`Ka~rJ2Zl(Yj-ltoZ;2__AJvU^qaY%2MA1|R2)ys`#p^T zo|Xwd0o|e-=TmQWIs;-t`D3N6pz#r1vlK$%!zLw#6wR?!69%SGAj|rwZEsSGNDm(*Ch5gSvajtDna5(f5N^;l+o!{v-C%u4CH zxX^Vw$JSYS_iHIWNO^a>x#(6G8&LG^xkg#v#W=LO_7t4*i`UuaOw4!_l4UrpV@8=c4&~o0EDODcS#W$coFZ4I%DeWxs2ZiCS{5QsZ zXZ?JEJXE`XQW6D}5G#f?_+wEY4;nz<$hNu56&@KoVp?+-BCEtbtT!i zzDC7T6EevfO?rf^^>#lm^J|W({%B=y6o7_*&PDIq%7Z47nZetKtUHPb^I{+7S(fpx z{7Z#r%voq^8pL-{Ie&-Df6)HQsKh!_`SqLoSlvU$<-y5!GmzG4{s|jzv{iL@MHttK zEx$f(@ZqG}&22@$9IG`!$&0_X^jty{=g$`I)sh*`r=u5~eq2u_w|YM<*Dj4Rj3`Wg zOIX?+i59kRjuNTZQVl}5UzL&i!Z=g7NVyRr<8I!TbB|9nR^91NB?Ef@`hu)lBA*hQ zOgBGZ+{YxI!&d1(>BgN{a-oNI-YjE(`=b>Z%~%awYA%9DMG;i_>&J9j^xqDS!~s+R z8xT|T4o-Rtko%W>@+T5n#rDOdqs|-6Sv9t|mVi$>uH5(>A8*%{mw7m?Mej z!7PIgLluX(vH)D@h%fQm5`ph2A2Gl7E#$oy{;vGeEjG2bToKSH9hRy`b;ql9clJy3aWKtTkf;HTErJ{7=VW zT|6Ht;{{e3-p{_n6etGvoPHInUsyuAxaZfVNy|tM`qzMsUOxhkXt3$*)6J-+_T+RWsK}JMSHIn zcKaQh7ZsXRC~994Wdb+%CslwEL(je=%8eg56UCFk<$^>`THpyC@`s-9J0r;zo-IwK zEqrn@3358=?GEY(8E(;pdicTu@Mkm^U#t~u^R;1TP8wDvK?xN}k{SkpSj+mokq}#n zGg;C{%auz@Sa!^GWYdZL^rw?*72uCo9sc`2RyR&J<)kcz{Dkwd9g=}~`=r-G!j5ef z4yoQkzMSIL#k`oA ze4nCPRZ_V2CuzP=iU^dZl+je`y6$E{0$R{qrGi%aqgiLf#K3qPm$1m(DJky)Y;HWz zT37Si_|-Xv zLtwhElu#*)mWL|(xtnNaV~1RLevJL^cXfX)b;t^Nv!LIK$7=dKsCD-)0z-8a*Y0mI z?>9}H@P-vTy5TWG9b9w$!*a>@OW(v$dBl|;0_@vL*xjJEju^o|qICWJU-pd|WP3g= zhnO}V8uic2s1$WK%BsaBVNcF!ivZxZGb&`!z=>C2@sL08o zRil!Xp5iZ;D}r~`BOI+BuzBx6s`5&m`*34xqAJ6saFCvLm)ACaj*@HgZ4y<}=s92o zo~d9@h#n!dEsYLb$B!nmMKazsLx=Di{4^q^C*5-4?feRYG~60U*xB{N?)iMf`d-H+ z!f%Z@H>s}fo`>|Uw9`h_ggq_JspiDO-WaybsssW3SoaD=4wc!dcUGaq;iVQBVfN^j z(U6`3l-?tyWsa#0Tw}u$2(fR0 zyK~K4dFQm~U7>=m*~DOspft*edW^#)3@Xf7$>sgn^y!LRat5ndcc~PTF$!F{j*pS4 zJUND}ssj2_)Co@AIVY4MmL(ZFn(9v$_CySMK2koz!u0A6R;o!A6SAlc zk;a)Zp^M#8TOstr8Azf((lKCWR*{MNaQN?nU1T# DiffusionOutput` (vllm_omni/diffusion/models/z_image/pipeline_z_image.py:413) with no **kwargs, so this raises TypeError before reaching `raw = outputs.output`. Additionally, the `_ZPipelineSamplingParams` shim (pipeline_ming_imagegen.py:40-52) lacks `latents`, `cfg_normalize`, `extra_args`, and `output_type`, which ZImagePipeline.forward reads directly (pipeline_z_image.py:464,466,467,471); removing the kwargs alone is insufficient. Align the call with the req-only contract and populate the shim, then re-run the serve+curl imagegen test. (evidence: Read pipeline_z_image.py:413 (forward(self, req) only) and pipeline_ming_imagegen.py:40-52 (_ZPipelineSamplingParams fields); grep confirms a single `def forward` in pipeline_z_image.py.) + +`vllm_omni/engine/orchestrator.py:?` [major] — Removing `already_submitted and` correctly makes the one-element unwrap apply to initial submits too; without it, Ming's thinker2imagegen result `[{"prompt": "", "extra": extra}]` reaches StagePool.submit_initial as a list, which raises ValueError on diffusion stages. Please add a regression test that drives `_forward_to_next_stage` with `next_pool.stage_type == 'diffusion'`, `already_submitted=False`, and a `[{...}]` processor result, asserting `submit_initial` receives a dict. (evidence: Read vllm_omni/engine/stage_pool.py:932-1010 and vllm_omni/model_executor/stage_input_processors/ming_flash_omni.py:346-433; grep confirmed no existing test exercises the diffusion list-prompt unwrap.) + +`vllm_omni/diffusion/models/ming_flash_omni/pipeline_ming_imagegen.py:475` [minor] — Confirmed: `return outputs` at pipeline_ming_imagegen.py:475 changes MingImagePipeline.forward to return the parent DiffusionOutput instead of a rebuilt DiffusionOutput(output=raw), and this path has no automated coverage. grep tests/ finds no `pipeline_ming_imagegen` references, and `MingImagePipeline` appears only in tests/test_config_factory.py as registry/model_arch assertions; e2e ming tests use ci/ming_flash_omni_thinker_only.yaml and ci/ming_flash_omni.yaml, not the ming_flash_omni_image deploy. Please add a CPU/mock unit test stubbing ZImagePipeline.forward and asserting the same DiffusionOutput (including stage_durations) is returned. (evidence: grep tests/ for `pipeline_ming_imagegen` and `MingImagePipeline`; read tests/e2e/offline_inference/test_ming_flash_omni_expansion.py and tests/e2e/online_serving/test_ming_flash_omni_expansion.py.) + +`vllm_omni/engine/orchestrator.py:1290` [minor] — Confirmed: orchestrator.py:1290 now performs the singleton unwrap regardless of `already_submitted`, so the change affects every diffusion stage whose `custom_process_input_func` returns a single-element list on initial submit (hunyuan_image3, glm_image, mammoth_moda2, ming_flash_omni, etc.), while the PR only changes Ming's pipeline. Please confirm the other bridge functions return single-dict lists, or add coverage for those paths. (evidence: [sweep] blast radius: grep for `custom_process_input_func` shows the singleton unwrap now applies repo-wide on initial submit to every diffusion stage with a bridge func (hunyuan_image3, glm_image, mammoth_moda2, ming_flash_omni, etc.), but the PR validates only Ming.) + +`vllm_omni/engine/orchestrator.py:1293` [minor] — The else branch (`custom_process_input_func is None`) assigns `diffusion_prompt = req_state.prompt` with no list unwrap, while `StagePool.submit_initial`/`submit_update` raise ValueError for diffusion list prompts. Confirm no diffusion-stage config reaches this branch with `req_state.prompt` still a list. (evidence: [sweep] orchestrator.py:1293 else branch `diffusion_prompt = req_state.prompt` is not covered by the unwrap; if `req_state.prompt` is a list for a diffusion stage the submit paths would still reject it (not verified that any config reaches this branch with a list).) + +`vllm_omni/diffusion/models/ming_flash_omni/pipeline_ming_imagegen.py:449` [minor] — The head has failing Buildkite lanes (`vllm-omni-amd-ci` and `vllm-omni-npu-ci`). The raw CI attribution to main vs this PR is unavailable; can you confirm attribution and hold the PR if these lanes are failing because of this diff? (evidence: [sweep] CI: gh_ci_read shows buildkite/vllm-omni-amd-ci and buildkite/vllm-omni-npu-ci FAILURE; attribution to main cannot be determined from the available evidence.) + +`vllm_omni/engine/orchestrator.py:?` [nit] — The unwrap at orchestrator.py:1290-1291 now applies to both submit_initial and submit_update for len(diffusion_prompt) == 1. When a diffusion-stage custom_process_input_func returns len > 1, the list still reaches StagePool.submit_initial/submit_update and raises the generic ValueError('Diffusion list-prompt batch requests are no longer supported...') without request id or bridge context; empty lists are already handled above with a terminal error. Consider a logger.warning or an explicit RuntimeError naming next_client.custom_process_input_func and req_id before submission so a multi-request bridge result is distinguishable from a malformed single request. (evidence: Read vllm_omni/engine/orchestrator.py:1268-1298 and vllm_omni/engine/stage_pool.py:932-1010; the only current guard is the empty-list terminal error, so len>1 reaches the StagePool ValueError with no request/bridge context.) + +`vllm_omni/diffusion/models/ming_flash_omni/pipeline_ming_imagegen.py:?` [nit] — Returning the parent `outputs` object is safe today: ZImagePipeline.forward is annotated and returns `DiffusionOutput(output=image, stage_durations=stage_durations)`, and `DiffusionOutput` has no `images` field, so the removed fallbacks were unreachable. The `raw = outputs.output` + `isinstance(raw, torch.Tensor)` check still fires before the return, so a non-tensor `.output` is caught; the only non-self-enforcing part is that the returned object itself is a `DiffusionOutput`. As defensive polish, consider asserting `isinstance(outputs, DiffusionOutput)` before `return outputs` to make the pipeline's advertised 'one DiffusionOutput with .output tensor' invariant explicit. (evidence: Read vllm_omni/diffusion/models/z_image/pipeline_z_image.py:413-774 (forward returns DiffusionOutput(output=image, stage_durations=...)); read vllm_omni/diffusion/data.py:1180 (DiffusionOutput has no `images` field).) + +**Verdict:** FOLLOW-UP REQUIRED (post-merge) diff --git a/eval/dataset/arms/copilot_v12_hybrid_val/pr4837.trace.json.gz b/eval/dataset/arms/copilot_v12_hybrid_val/pr4837.trace.json.gz new file mode 100644 index 0000000000000000000000000000000000000000..ecf946f65c906f84fab11cbb56ac7518502d7b2f GIT binary patch literal 28683 zcmb@NLvSZR*RNySwrz7_+jcUs?TKw;V%xTDJO9}BH*dXlH}`(a+x1j0PW9tAa%83h8Y_$#Xv zsR;io!s1FIJ<698)0q>oCC`w+sHe69D2RwEfo-wC)SNs)^gvN<_wDmSsa5bCAYrjn zOQ_IYXp>&lyGJOIw2ma``*5N3E`Tr3d$U)0-*7>7Q#8nj@A5(I!;v$k{EB85D0nLo zg^Up&e~*FAcVt8>j}{R`+D1hvF*w77Qv^7Z`1b#y$0{v`fYU`T8x4{dDM;O*(;^z}3;Zf3fGudGDps+RZE< z28>~D_ub)SZ?D(glZrF=MZM%%a)}brA4RxY%b(=Zq^_^9Jbixeh*d>&Mf#4a zQnQ6+(RAw?iC`B$sW=B3W8{WEt|>_U$P8{MgBnA4-h7#4Rtpg&fSdp>dXyF_f4O7q zzZS~apIFlrj^{`X-mL7s5Sd7Pa6=`V$hGhWt|duS@?ZYPKt>N^7f`XI*`h+;MnysC zPo@eIbl1wrj>RJ4hv`E`IIf+yT3a%vnDR2==~S6>Lan{0`Bq}4KkqQ)1PZI`+wR>>gCBVBEwzHfM6p$aU^h3(a4E!Lb6%GE&pQOjPNh3byJ}m zT}8L)yGtF8pTZz@Sjqj0*Hl>#CRZe&pexC?#-6uBD`v)&j#5ox2n028#FB!qTLb}; zHH|NkJ>Q^aZU!XAOR&>Xno*N?>ub1S8_H?iYtAQMfW^b!io$HbFK}IoOChRn$EZ%2 zjF&AyYv>hd)*>_s)kJK8Q(lW|^p*Qg67ou|T(DOui-pe%-Dtmf1tv-*TMnBDISwaG zNDZ-LN|6BdHc9V;al;RC3A_CzJB+GaqBFgqzdu$GbkP^^{&(H0-$dp3*(I-W3tgC? zcH_wq-Ql5CxGv}>)B*8ey64G@U{vh+fs!QKDasOj&_KD(iVfTO!9GE$pL26%_B1RlijM~;M*MdGuflBJ#W#zl*UUrHPsRjIyzO31} z*%gM(6c)~GSZsz#`93|)he