pgw#1141 (§4.31 + §4.32): a boot-adopted cell materializes, arms and SERVES — the parity gate MOVES to the mint

changelog.d/pgw1141.md

@@ -0,0 +1 @@
+- **pgw#1141 (DESIGN-RULINGS §4.31 + §4.32): a boot-adopted cell materializes, arms and SERVES — the warmup proof barrier and the adopt-side quality gate are both deleted.** Measured identically on two real pods (RTX 4000 Ada, 0.106.0): `boot_adopt=hit` -> materialize -> `cell_numerics cos=1.00000 ret=1.0000 rel_l2=0.0000` on 3/3 axes -> and then the setup warmup scored that same artifact `unexercised`, folded it into `unproven`, wrote `function_proofs[id]=set()` and unwrapped it; `functions=()` made `target_applicability_incomplete` and `armed_target_unresolved` inevitable and the pod served eager for life, publishing nothing. The SELF-MINT arm was healthy on the same wheel, card and release — a mint's warmup DRIVES its own capture and therefore dispatches, while an adopt arms *before* setup and by construction cannot. **§4.31 (Paul):** *"skip the warmup / arm check, so we can serve right away … try to serve, if an error is encountered, and it's the cause of the cell, de-arm the cell, and serve eager instead. If our cell is correct this adds zero cost."* An AOTI `.pt2` is ahead-of-time machine code for this exact sm x toolchain: the first call is full speed, and the warm pass never made a cell faster, it only checked it. **§4.32** then moved the quality question to where the defect is: every failure that gate ever caught (a baked `conv_out.bias`, timestep dtype scars) was an AUTHOR defect in endpoint code or config, and re-measuring on every adopter taxed the fleet forever for one author's one-time mistake. So on the exported lane the warm ledger gates nothing (`_install_compile_targets` permits the whole advertised contract; `function_proofs` survives as telemetry), and **adoption is materialize -> arm -> serve with no quality gate at all**. The pgw#868 probe now runs on exactly ONE path — `fleet_cells.adopt_delegated_mint` -> `provision.arm_aot(verify_numerics=True)`, the pod that just compiled the bytes, before `publish_self_mint` can ship them — and it is **strict: identical or refuse, no DEGRADED-publish band**, because an adopter runs no gate that could re-check what ships. Safety without re-measurement is CONSTRUCTION, not checkpoint identity (a `ck1` key is graph x envelope x sm x toolchain and carries no checkpoint hash — one cell serves every checkpoint of the architecture): the cell is compiled CODE and weights flow through it as data, so a mint-time parity proof proves the FUNCTION; a weight value baked into the artifact is fenced fail-closed by the constant-folding fence (0.100.0); and a checkpoint that changes the COMPUTATION hashes to a different graph, hence a different key, hence no match. **Failure handling carries the weight now:** a cell-attributable failure answers that request eager, de-arms sticky and types the cause; a **CUDA OOM is not the cell's fault** (a sibling load, a rotation) so it serves eager and leaves the cell armed rather than condemning a correct artifact on the first busy moment; and the sticky de-arm now reaches the target install, so a revoked artifact can no longer advertise `aot_cell` on a pipeline whose every call runs eager. The **dynamo lane keeps its cache-hit ledger** — a dynamo arm that does not serve its cell RECOMPILES (correct output, silently slower, no exception to catch and no numerics gate on that lane), so it is the only detector in existence there. The warm PLAN's per-class cost is deliberately left for its own change: collapsing it on an exported adopt (sdxl: 18 full generates per handler — 9 aspect buckets x 2 guidance classes — reduced to 2, against a pgw#654/ie#546 canary that priced the machinery at ~30-minute first boots) also removes pgw#844's boot-time coverage census and the dynamo lane's per-class ledger, and neither should be a rider on a P0 arm fix. Nothing about the arm depends on it. Accepted residual, on the record: an artifact pathological enough to corrupt the CUDA context is not recoverable in-request. `tests/test_adopted_cell_warm_proof_pgw1141.py` drives the real `provision.arm_aot` (real packed artifact, real gate, real ladder), the real `ensure_setup`, and the real serving wrapper — nothing stubbed, red on master in both directions, including an adoption that arms a cell the mint gate would have refused.

src/gen_worker/aot_serve.py

@@ -105,6 +105,7 @@
     sku_slug,
 )
 from .models import lora_lifted
+from .models.memory import is_cuda_oom
 
 logger = logging.getLogger(__name__)
 
@@ -2512,6 +2513,26 @@ def aot_forward(*args: Any, **kwargs: Any) -> Any:
             _revoke(state, f"constants unbound: {exc}")
             return original(*args, **eager_kwargs)
         except Exception as exc:  # noqa: BLE001 — ANY artifact problem => eager
+            if is_cuda_oom(exc):
+                # pgw#1141: ATTRIBUTION. The serve-first doctrine makes the
+                # first real request the proof, so what that request blames
+                # decides whether a good cell survives — and allocator
+                # exhaustion is a fact about the CARD at this instant (a
+                # sibling load, a concurrent rotation), not about the artifact.
+                # Condemning the cell for it would retire a correct one on the
+                # first busy moment and re-mint it on the replacement pod.
+                # Serve THIS request eager, stay armed, say so.
+                logger.warning(
+                    "aot-serve: %s hit CUDA OOM (%s); serving this request "
+                    "eager, artifact stays armed — allocator pressure is not "
+                    "the cell's fault", label, exc)
+                activity_mod.emit_event(
+                    "aot_serve_oom",
+                    f"family={meta.get('family')} target={label}: "
+                    f"{type(exc).__name__}: {exc}",
+                    phase="cuda_oom",
+                )
+                return original(*args, **eager_kwargs)
             state["failed"] = True
             detail = (
                 f"AOTI artifact {label} failed: "