H3 Ref2VA lossless follow-up: validate audio coverage and scorer identity

evidence/h3-ref2va-lossless-293/README.md

@@ -0,0 +1,94 @@
+# MiniMax-H3 Ref2VA lossless follow-up
+
+Status: **INCONCLUSIVE**. No new lossless performance gain is qualified.
+
+The requester requires lossless acceleration, no audio-quality degradation, and
+both 5-second and 15-second outputs. The accepted scope is recorded in
+[the campaign update](https://github.com/InferMatrixLab/InferRSI/issues/293#issuecomment-6573035907).
+The old Cache-DiT result is lossy and cannot satisfy this request. The historical
+`h3-ref2va-ascend-293` bundle is preserved byte for byte.
+
+## Current delivery
+
+The new validator rejects two equally truncated audio tracks, checks audio
+timestamps and decoded sample coverage, and authenticates the actual bytes used
+for both LPIPS networks. Weight loading is explicit and offline, with strict
+parameter loading; the complete scorer state was compared against LPIPS 0.1.4's
+previous loader. The synthetic regression is not model-quality evidence.
+
+Old perceptual gates remain diagnostic. A media-threshold PASS does not become a
+lossless PASS: this validator returns INCONCLUSIVE for unequal decoded outputs
+that pass those thresholds. It records audio and video equality separately.
+Control repetitions are themselves non-bitwise-identical; any tolerance-based
+adjudication needs a separately frozen matched repeatability protocol. This
+bundle does not certify subjective sound quality, meaning, or lip synchronization.
+
+## Experiment plan
+
+- Native BF16, dense attention, 50 denoising steps, same references/seeds and
+  resolution. Disable approximate block caches, reference-KV reuse, quantization
+  and sparse attention in every arm.
+- First screen exact weight placement: AllGather, rank-local streaming, and a
+  16-layer resident prefix. The retained [screening plan](plan/screening.json)
+  fixes one source for all three arms. This is an existing-configuration search.
+- Screen 5 seconds first with two warmups and one measured request per server,
+  inside one eight-card reservation. A pilot is not a repeated E2E qualification.
+- After a candidate passes quality, run fresh A/B/B/A servers at **both 5 and
+  15 seconds**, two warmups and three measured requests per duration/server.
+  Freeze the selected final source/configuration and quality protocol first.
+- Report HTTP upload-to-full-MP4 E2E, independent server aggregates, completion,
+  sampled HBM and actual worker exit codes. Run stage profiling separately from
+  unprofiled timings. Do not estimate p95/p99 from the small sample set.
+- Compare the final combination directly against the original dense baseline;
+  keep the matched diagnostic baseline shutdown budget explicit. Do not add
+  separate optimization percentages or claim a global optimum.
+
+## Screening update
+
+The [first screening attempt](results/screening-v1.json) completed one 5-second
+baseline measurement at **400.943 seconds HTTP E2E**, after two warmups. All eight
+workers exited with code 0. This single baseline is not a speedup measurement.
+The rank-local arm failed before inference because the harness combined a unified
+offload policy with a legacy offload flag; the third arm was not run.
+
+The replacement harness passed [actual shell argument assembly](results/argv-preflight-v2.json)
+and the [real configuration resolver](results/offload-resolution-v2.json) on CPU.
+The latter also shows that rank-local with zero resident layers selects the
+`layer_wise` backend, while 16 resident layers selects `distributed_layer_wise`.
+These are complete configuration candidates, not a pure collective comparison.
+The corrected campaign must rerun the baseline and both candidates together.
+
+[Candidate provenance](plan/candidate-sources-v2.json) distinguishes existing
+configuration mechanisms from the unqualified H3 NPU adaptation of QKV packing.
+No candidate gain, 15-second result, or audio-quality acceptance is claimed.
+Original frozen validator and screening files remain unchanged; these additions
+have their own hash manifest.
+
+## Sources and attribution
+
+| Item | Source | Current evidence |
+| --- | --- | --- |
+| Exact weight placement | [vLLM-Omni #5929](https://github.com/vllm-project/vllm-omni/pull/5929), component-selective DLO; current control source in the plan | Existing implementation; 16-layer choice is a campaign hypothesis, no gain claimed |
+| HTTP workload and video/audio metrics | Prior campaign's frozen vLLM-Omni accuracy/quantization-test extracts | Metrics retained, old permissive audio gate is not a no-degradation certificate |
+| Audio coverage repair | [review counterexample](https://github.com/InferMatrixLab/InferRSI/pull/294#discussion_r4432729023) | Reproduced with synthetic equal truncated tracks; new validator rejects |
+| Authenticated scorer | [review finding](https://github.com/InferMatrixLab/InferRSI/pull/294#discussion_r4432729025), installed LPIPS 0.1.4 `LPIPS` and `alexnet` loaders | Explicit verified-byte loads; complete state equality checked on CPU |
+| No approximation as an acceptance constraint | Requester's follow-up, linked above | Source/configuration audit plus new output validation still required for each promoted optimization |
+
+## Reproduction and limits
+
+The optional CPU scorer needs NumPy, PyTorch, torchvision, LPIPS 0.1.4 and FFmpeg.
+Supply the two already provisioned weight files explicitly; the hashes are in
+`validator/weights.json`. The validator does not download weights.
+
+```bash
+python validator/compare.py --baseline <BASELINE_MP4> --candidate <CANDIDATE_MP4> \
+  --frames 124 --alexnet-weights <ALEXNET_PTH> --lpips-weights <LPIPS_PTH> \
+  --output <RESULT_JSON>
+```
+
+Use 362 frames for the existing 15-second request contract. The 2048-sample upper
+padding allowance is two AAC frames, not permission to truncate the expected
+audio duration. The process returns 0 for exact decoded parity, 1 for a failed
+media gate, and 2 for nonidentical outputs requiring further adjudication;
+execution errors do not establish a campaign verdict. Model files, media and
+host scripts/logs remain external. No completed paired 5/15-second E2E qualification is claimed.

evidence/h3-ref2va-lossless-293/patch/README.md

@@ -0,0 +1,11 @@
+# Engine delivery
+
+All accepted engine changes remain consolidated in
+[vLLM-Omni #9698](https://github.com/vllm-project/vllm-omni/pull/9698).
+No new engine patch is accepted by this preparation slice.
+
+Initial configuration screening uses runtime
+`098b51f6b96dd160a0959e74e8822422527ff06d`, which is runtime-identical to the
+documentation-only PR head `922f1e9f1cb85b9f2b66d86e1eb500e6872b2dcb`.
+The original dense baseline for final comparison remains
+`c7f56f0155953c7d4a77276a2f4fcbd844ab0da2`.

evidence/h3-ref2va-lossless-293/plan/candidate-sources-v2.json

@@ -0,0 +1,48 @@
+{
+  "acceptance": "No lossy cache, no quantization, no omitted diffusion steps; complete audio and no demonstrated quality regression. No candidate has qualified yet.",
+  "candidates": [
+    {
+      "name": "rank-local layer weight transfers",
+      "source": "https://github.com/vllm-project/vllm-omni/pull/5929",
+      "source_revision": "ea3cf9989",
+      "kind": "existing configuration, not new engine code",
+      "hypothesis": "Replicated host reads may avoid all-gather overhead",
+      "screening": "ranklocal0 versus allgather on the same reserved devices",
+      "status": "pending",
+      "gain": null,
+      "qualification_note": "Unified ranklocal0 policy selects layer_wise backend; ranklocal16 selects distributed_layer_wise. Report complete config gains, not isolated collective gains."
+    },
+    {
+      "name": "bounded resident DiT layers",
+      "source": "https://github.com/vllm-project/vllm-omni/pull/5929",
+      "kind": "existing configuration, not new engine code",
+      "hypothesis": "Keep 16 layers resident to avoid repeated weight transfer if memory permits",
+      "screening": "ranklocal16 versus ranklocal0; combined versus allgather",
+      "status": "pending",
+      "gain": null,
+      "implementation_source": "https://github.com/vllm-project/vllm-omni/pull/5764",
+      "implementation_revision": "1c2a81f6d84aea4fff53bd2f894c2a287c237245",
+      "source_revision": "ea3cf99894f379c7732e6d62aa464a6f4285fbd9"
+    },
+    {
+      "name": "combined strict QKV collective",
+      "source": "https://github.com/vllm-project/vllm-omni/pull/4150",
+      "source_revision": "5c0cdb7916bff945ba6b6a3c645a17e6973e5990",
+      "kind": "local NPU H3 adaptation of upstream proposal, reuses existing 5D collective",
+      "hypothesis": "One collective instead of three may reduce launch overhead, but packing and noncontiguous consumers can erase benefit",
+      "status": "10 CPU dispatch cases pass; real collective and NPU validation pending",
+      "gain": null
+    }
+  ],
+  "not_selected": [
+    {
+      "source": "https://github.com/vllm-project/vllm-omni/pull/9353",
+      "reason": "Author reports eager regression; no basis for assuming benefit on current eager H3 workload."
+    },
+    {
+      "source": "https://github.com/vllm-project/vllm-omni/pull/9510",
+      "reason": "Additional SHMEM NPU implementation/dependencies; draft is not evidence of qualified H3 lossless benefit."
+    }
+  ],
+  "reporting": "Report baseline, individual and combined E2E directly. Do not add component percentages. Synthetic collective speed is never reported as model E2E gain."
+}

evidence/h3-ref2va-lossless-293/plan/integrity-screening-v2.json

@@ -0,0 +1,6 @@
+{
+  "results/screening-v1.json": "0031fefa77b7992a3076d9310ed882e9b32798cd3ed9d22a974b9d6765e2c2c5",
+  "plan/candidate-sources-v2.json": "5089168edb63f5322ec698b8e9d038348c7eed4c4ca7fbcd5a6263be41988a2c",
+  "results/offload-resolution-v2.json": "a09a0eacecf5846c4bd0fabafc98aa6655631b469f56f63876b920f5837ecd40",
+  "results/argv-preflight-v2.json": "e166678cf7ef218962866b7db6279be4a1fe7609a22eea674e58b58fa55442f2"
+}

evidence/h3-ref2va-lossless-293/plan/integrity.json

@@ -0,0 +1,11 @@
+{
+  "validator/compare.py": "af3d7e05763e4b80aaaceecf771a76d94d810a6b134c14ec1a8537a4a5fde6f0",
+  "validator/integrity.py": "3ddaadcaf04b7e8962b5397454d52d5f26c0c7c8933505bb83f3c12e1fdcae10",
+  "validator/smoke.py": "54299a636b6f59b337b66706469e17c6fbfa7412adf918f1e626a3a84e6543a2",
+  "validator/upstream_metrics.py": "d76f3c66300e4a396fedb92da7a24e436f6241102c65032eb1eb16593cdabf71",
+  "validator/gates.json": "aa4330075b6da7ab268e6ef092c42b0e037681695b54b238b09692c21d58bb9a",
+  "validator/sources.json": "9ad2c43f3943be23944e5b5fcd1677d66e3f4a1b66c32eba3bf50dfb4921ad01",
+  "validator/weights.json": "605e463a0b64244974f5475dead6b3bdc9cc4899930801e161cdf48d8c057171",
+  "plan/screening.json": "689fb482e8d2a1648f7a3efead9028916da0d2237870e64f8beca95119a7b1d2",
+  "results/validator-regression.json": "7545936f2f1835c0eed427e822b8007caaddf1076e14249bdcb301d71518974d"
+}

evidence/h3-ref2va-lossless-293/plan/screening.json

@@ -0,0 +1,53 @@
+{
+  "status": "frozen_screening_not_acceptance",
+  "issue": "https://github.com/InferMatrixLab/InferRSI/issues/293#issuecomment-6573035907",
+  "source_revision": "098b51f6b96dd160a0959e74e8822422527ff06d",
+  "image_id": "sha256:7fc6c73bd19a6ef4ee8d6048009972234796a1fc1bd3067168a425343cc476f7",
+  "duration_s": 5,
+  "steps": 50,
+  "seed": 0,
+  "warmups": 2,
+  "measured_requests_per_server": 1,
+  "profiles": {
+    "allgather": {
+      "quality": "lossless",
+      "serve_args": [
+        "--diffusion-attention-backend",
+        "FLASH_ATTN"
+      ]
+    },
+    "ranklocal0": {
+      "quality": "lossless",
+      "serve_args": [
+        "--diffusion-attention-backend",
+        "FLASH_ATTN",
+        "--diffusion-offload-config",
+        "{\"mode\": \"layer\", \"components\": [\"dit\"], \"layer_options\": {\"dit\": {\"weight_transfer\": \"rank-local\", \"resident_layers\": 0}}}"
+      ]
+    },
+    "ranklocal16": {
+      "quality": "lossless",
+      "serve_args": [
+        "--diffusion-attention-backend",
+        "FLASH_ATTN",
+        "--diffusion-offload-config",
+        "{\"mode\": \"layer\", \"components\": [\"dit\"], \"layer_options\": {\"dit\": {\"weight_transfer\": \"rank-local\", \"resident_layers\": 16}}}"
+      ]
+    }
+  },
+  "source": "vLLM-Omni PR5929 component-selective DLO, ea3cf9989; campaign-authored 16-layer hypothesis",
+  "stop_policy": "Stop on first failed arm. No final acceptance from this screen. Final AB/BA at 5s and 15s required.",
+  "reservation": "one 8-card reservation, 6h hard, 315m soft, 100m per arm; 1200s HTTP and execution deadlines",
+  "quality": "No approximate cache, quantization, sparse attention, step reduction or reference-KV reuse. New coverage/authenticated-weight validator; nonidentical media remains INCONCLUSIVE pending matched repeatability adjudication.",
+  "script_sha256": {
+    "run_host.sh": "805c2b2cdf22b9ac85745c9755bf4b1bfa96f13a00e31125e1173d321affa30f",
+    "run_inner.sh": "d3a7ecd7108474a79e17783206e9b13fd85c8a6673d16751ed25cc9a435c9cf0",
+    "run_client.py": "7a3e697077524c9729ebdae78a533feb55777718b0db72ac3735e5164444bcde",
+    "run_heldout.py": "489e3048120fa34a1c8d089053f45f1d4cda4fd6fdf1652457d079302f77eded",
+    "allgather.json": "5b4ffd4364a6324811b2ad3a24eb623b9d9eb85093da2e025008f37267d4eaab",
+    "ranklocal0.json": "a2d01ea417740660f9794823f9d934662046b9546a4859322dae173383d16baf",
+    "ranklocal16.json": "4ec8b91f3f7b1714cae759e39e1507e92afb81a46203ce7a3f2b5282d30c79f5",
+    "campaign.sh": "2baf792b332a95fa55f90cd5c64ece0db565b7548ccb3e898f6d941209458929",
+    "launch.sh": "980943b33d94fa844d443d4811d234aa27064f05d3ec04dd6374c3401e18ef45"
+  }
+}

evidence/h3-ref2va-lossless-293/results/argv-preflight-v2.json

@@ -0,0 +1,65 @@
+{
+  "pass": true,
+  "scope": "actual shell argv assembly, CPU only; not model startup qualification",
+  "arms": {
+    "allgather": [
+      "--num-gpus",
+      "8",
+      "--usp",
+      "8",
+      "--ring",
+      "1",
+      "--text-encoder-tp-size",
+      "8",
+      "--vae-parallel-mode",
+      "tile",
+      "--vae-use-tiling",
+      "--vae-patch-parallel-size",
+      "8",
+      "--enable-distributed-layerwise-offload",
+      "--disable-log-stats",
+      "--diffusion-attention-backend",
+      "FLASH_ATTN"
+    ],
+    "ranklocal0": [
+      "--num-gpus",
+      "8",
+      "--usp",
+      "8",
+      "--ring",
+      "1",
+      "--text-encoder-tp-size",
+      "8",
+      "--vae-parallel-mode",
+      "tile",
+      "--vae-use-tiling",
+      "--vae-patch-parallel-size",
+      "8",
+      "--disable-log-stats",
+      "--diffusion-attention-backend",
+      "FLASH_ATTN",
+      "--diffusion-offload-config",
+      "{\"mode\": \"layer\", \"components\": [\"dit\"], \"layer_options\": {\"dit\": {\"weight_transfer\": \"rank-local\", \"resident_layers\": 0}}}"
+    ],
+    "ranklocal16": [
+      "--num-gpus",
+      "8",
+      "--usp",
+      "8",
+      "--ring",
+      "1",
+      "--text-encoder-tp-size",
+      "8",
+      "--vae-parallel-mode",
+      "tile",
+      "--vae-use-tiling",
+      "--vae-patch-parallel-size",
+      "8",
+      "--disable-log-stats",
+      "--diffusion-attention-backend",
+      "FLASH_ATTN",
+      "--diffusion-offload-config",
+      "{\"mode\": \"layer\", \"components\": [\"dit\"], \"layer_options\": {\"dit\": {\"weight_transfer\": \"rank-local\", \"resident_layers\": 16}}}"
+    ]
+  }
+}

evidence/h3-ref2va-lossless-293/results/offload-resolution-v2.json

@@ -0,0 +1,43 @@
+{
+  "source": "vllm_omni/diffusion/offloader/config.py at 098b51f6b96dd160a0959e74e8822422527ff06d",
+  "scope": "real pure config resolver, not engine startup",
+  "results": {
+    "allgather": {
+      "strategy": "distributed_layer_wise",
+      "components": [
+        "dit"
+      ],
+      "transfers": {
+        "dit": "allgather",
+        "text_encoder": "rank-local"
+      },
+      "resident_layers": 0,
+      "pin_memory": true
+    },
+    "ranklocal0": {
+      "strategy": "layer_wise",
+      "components": [
+        "dit"
+      ],
+      "transfers": {
+        "dit": "rank-local",
+        "text_encoder": "rank-local"
+      },
+      "resident_layers": 0,
+      "pin_memory": true
+    },
+    "ranklocal16": {
+      "strategy": "distributed_layer_wise",
+      "components": [
+        "dit"
+      ],
+      "transfers": {
+        "dit": "rank-local",
+        "text_encoder": "rank-local"
+      },
+      "resident_layers": 16,
+      "pin_memory": true
+    }
+  },
+  "note": "ranklocal0 and ranklocal16 select different backend strategies automatically; treat as complete configuration candidates, not isolated communication microbenchmarks"
+}