Skip to content

pgw#1141 (REOPENED by POD PROOF #4): a boot-adopted cell is on the EXPORTED lane — read it off the OBJECT, not off a registry nobody fed - #669

Merged
PaulFidika merged 1 commit into
masterfrom
1141b-target-applicability
Aug 12, 2026
Merged

pgw#1141 (REOPENED by POD PROOF #4): a boot-adopted cell is on the EXPORTED lane — read it off the OBJECT, not off a registry nobody fed#669
PaulFidika merged 1 commit into
masterfrom
1141b-target-applicability

Conversation

@PaulFidika

Copy link
Copy Markdown
Contributor

What the pod said, on the wheel that was supposed to fix this

gen-worker 0.111.0 (which carries pgw#1141's 0dbf68e5), pod 5od13p840v1bgb, RTX A4500, first try — POD PROOF #4 in pgw#1108:

seq  3  boot_adopt           hit                              key=ck1-329a6fbe…  10 291 ms
seq 12  cell_numerics        armed_undispatched               "It STAYS ARMED and serves…"
seq 13  serve_eager_posture  target_applicability_incomplete  functions=() owned_slots=['pipeline']
seq 14  serve_eager_posture  armed_target_unresolved          armed=False targets_resolve=True
seq 15  serve_degrade        armed_target_unresolved
seq 17  self_mint_skipped    boot_ended_uncompiled

seq 12 is 0.111.0's own new sentence, so §4.31's warmup-barrier deletion is genuinely live on a pod. And 677 ms later the object was unwrapped anyway. PR #658's claim that this chain was "structurally unreachable" on the exported lane was false on the shipped wheel. The disarm moved one branch over; it did not disappear.

The locus — neither of the two recorded candidates, and it explains both

The tracker recorded two candidates: (1) permitted_names empty because exported_arm is False (active_selection None), or (2) compatible_names empty from the alias loop. It is candidate 1's OUTCOME by a third mechanism, and the mechanism is upstream of the install entirely.

aot_serve.is_aot_ref answers out of _KNOWN_AOT_KEYS, a process-global set fed by note_aot_key. pgw#1033 wrote the feeding rule as a convention"whoever reads a cell_key off an aot-inductor envelope registers it" — and it had exactly two feeders:

feeder route
fleet_cells.arm_from_local_store (fleet_cells.py:2044) this machine's own cell store
fleet_cells.adopt_delegated_mint (fleet_cells.py:2181) this pod's own mint

Both self-produced. fleet_cells.arm_ordered — which is every hub Plan arm and every §4.27 boot-adopt — fed it nothing. So on the pod:

  1. is_aot_ref(adopted_ref)False, on the very pod that was serving those bytes;
  2. _proves_by_fx(ref) → True → the artifact went into proof_before, the DYNAMO lane's cache-hit ledger — which an AOTI artifact can never move (it performs no FX lookup);
  3. aot_proof_beforeempty → §4.31's if id(pipe) in aot_proof_before: keep the arm branch could not fire for the one object it exists for;
  4. calls <= 0unexerciseddynamo_unexercised (also keyed on aot_proof_before) → unprovenfunction_proofs[id] = set(), aot_serve.unwrap, compile_cache.unwrap, active_compile_artifacts.pop. This is seq 12's armed_undispatched row — emitted with the dynamo reason, which is why the leg read as "pgw#1141 works";
  5. at the install, active_selection is now None → exported_arm False → permitted_names = function_proofs[id] = set()functions=() → seq 13, 14, 15.

Every observed field matches: functions=(), bindings non-empty and valid, owned_slots=['pipeline'], armed=False (compile_cache marker gone), targets_resolve=True (structural).

Why pgw#1141's tests were green on the wheel that fails on a pod: their fleet-policy stand-ins call aot_serve.note_aot_key(key) by handtest_adopted_cell_warm_proof_pgw1141.py::_fake_adopt_arm and test_aot_boot_proof_gap_pgw735.py::_fake_arm. That single line is the one thing no production arm route did. The tests entered one gate east of the bug, twice.

The cross-wheel confound in the tracker (the adopted cell was minted by 0.103.0) is eliminated: the SDK version is not an input to any branch here, and the repro is same-wheel throughout.

The fix

1. Root, structural — register at the wrap. aot_serve.load_and_wrap calls note_aot_key(meta["cell_key"]) at the setattr(pipeline, _MARKER_ATTR, …) that makes the cell live. That is the one seam every arm route passes and the moment the fact becomes true, so a future arm route inherits the registration instead of having to remember it. The two existing feeders stay (they register before a wrap, which serving_mode reads).

2. The reader — ask the object. New aot_serve.holds_exported_cell(pipeline) (marker present, armed or revoked) and executor._exported_arm(pipeline, ref). All five lane readings now go through it: _proves_by_fx, aot_proof_before, _install_compile_targets's exported_arm, dynamo_unexercised, and §4.31's own keep branch. holds_exported_cell is deliberately not is_armed — a revoked cell must still read as exported so the install refuses to advertise it, while an object carrying no cell reads as an ordinary dynamo/eager one. With this, the dynamo lane's disarm authority cannot be exercised over an object carrying a live cell even if no registry ever learned its key — that is the property the third row asserts, with the registry deliberately emptied after the wrap.

Reader sweep — every site that can unwrap / de-arm / refuse to serve an armed cell

# site trigger classification
1 executor.py:7725-7729 aot_serve.unwrap + compile_cache.unwrap in the unproven fold object was exercised and measurably did not serve its own graph legitimate (evidence) — and it was the pod's actual exit, reached only because the object was mis-sorted onto the dynamo lane. dynamo_unexercised now uses _exported_arm, so an exported cell can no longer enter this fold at all.
2 executor.py:7853 the unexercised loop no warm dispatch landed WAS the structural asymmetry (gated on aot_proof_before membership, i.e. on the registry). Now gated on _exported_arm(pipe) — the lane, not a snapshot.
3 executor.py:7873 compile_cache.unwrap (dynamo unexercised, non-mandatory) an unexercised dynamo object serving = silent recompile, no downstream detector legitimate (§4.31 keeps this on purpose). Now unreachable for an exported object by construction.
4 executor.py:4738-4756 _install_compile_targets exported_arm + the COMPILED_DEGRADED continue the artifact revoked itself before a guard was bound legitimate (evidence) — but it was unreachable pre-fix, because exported_arm was False, so a revoked adopted cell fell through to the same functions=() path and reported the wrong reason. The negative test pins that it now reports compiled_degraded.
5 executor.py:4610 _eager_only_reason early return topology degree > 1 (pgw#775) legitimate (operator/topology) — no cell is fetched, no scope opens. Untouched.
6 executor.py:9598 self_mint_skipped insufficient_vram unwrap the mint capture does not fit beside the serving working set legitimate (evidence) — scoped to pending_self_mints, an adopted cell has none.
7 executor.py:9717 abandon_background_mint unwrap a mint that will not happen; give the card back legitimate (evidence) — mint pipes only.
8 provision.py:460 aot_serve.unwrap on numerics_refused §4.32 mint-time parity gate, verify_numerics=True legitimate (evidence) — ONE caller, adopt_delegated_mint, the pod that just compiled the bytes. Adoption passes False.
9 local_cells.py:278,285 cc.unwrap a cozy-local mint failed / re-adopting the just-saved cell legitimate (evidence / rearm) — not on the fleet adopt path.
10 compile_cache.py:458,476,507 serve_degrade emitters a real exception at compile/serve time (graph break, declared-range refusal, permanent degrade) legitimate (evidence) — these are confessions of something that raised.
11 executor.py:4584 serve_degrade from _assert_armed_targets_installed armed-at-setup object owns no installed target downstream reporter, not an authority — it describes the wreckage of #2. Correct to keep; it is the invariant that made the pod diagnosable.
12 serving_mode.classify_mode:133 is_aot_ref(ref) classifying a served request's mode latent third reader, already object-backstopped (if pipeline is not None and aot_serve.is_armed(pipeline): return MODE_AOT_CELL). Pre-fix it would still have misreported aot_cell as jit_cell on any call site that passes no pipeline. Fix 1 closes it at the source; no change needed here.
13 arm_ordered's serve_posture.eager_only() short-circuit operator order (§4.32 item 4) legitimate (operator) — asserted intact by a negative row.
14 aot_serve.wrap_module in-request guard revocation a cell-attributable serve failure legitimate (evidence) — §4.31's replacement detector, untouched.

Verdict: no fourth structural asymmetry. Rows 2 and 4 were the surviving instances; row 12 is the same class caught before it could cost anything and is closed by fix 1.

RED evidence

tests/test_adopted_arm_lane_pgw1141b.py — 8 rows, 8/8 fail on origin/master's src/, 8/8 pass here. On master the log is the pod, verbatim:

compile target omitted for generate: immutable object applicability is incomplete
  (functions=() bindings=(('pipeline','acme/micro-probe',…)) owned_slots=['pipeline'])
generate: 1 ARMED compile object(s) own NO installed target — this boot compiled
  graphs nothing can dispatch to (AdoptedPipeline armed=False targets_resolve=True degrade=-)

What runs for real: ensure_setup_injection_kwargs_enable_compiled with a real _ArmOrder(adopt=…) → real fleet_cells.arm_ordered → the real receipt gate against a real RSA-signed receipt from a real HTTP HubStubprovision.arm_aotload_and_wrap on a real packed cell whose ck1 key is genuinely restatable from its own recorded facts (cell_key.from_exported_artifact_metadata, the same recomputation admission runs) → the real boot warmup → the real proof pass → the real _install_compile_targets and _assert_armed_targets_installed.

Three seams, all WEST of the locus, all named in the module docstring: Executor._boot_adopt (its derive+resolve half is driven end to end for real by test_boot_adopt_observability_pgw1116.py against examples/micro-diffusion and a real hub), provision.load_slot (the weights read), and pgw#868's _load_package (the AOTI .so — the one piece deferred to a pod). Nothing about applicability, the lane split, the proof pass or the target install is stubbed.

Negatives asserted: a revoked cell installs no target and names compiled_degraded; the operator's eager-only order still suppresses the arm (serving_tiers()["generate"] == "eager"); holds_exported_cell stays True across a revoke and False after unwrap.

Runbook fix (side item)

examples/micro-diffusion/README.md §4-§7 was unexecutable and burned pod-leg time. POST /v1/admin/compile-cells no longer exists — §4.28 retired the mint-only pod class; a serving pod self-mints in the background — so the runbook now buys a serving worker (POST /v1/admin/releases/<id>/workers?count=1&compute_class=gpu) and drives one request. size is an enum ("256x256"), not a {"width":…,"height":…} object, which is a 400. §7 now names the four typed rows that decide GREEN, and names target_applicability_incomplete / armed_target_unresolved as this issue recurring.

Not in this PR

Local verification

tests/test_adopted_arm_lane_pgw1141b.py + 34 neighbouring files (the whole adopt / arm / proof / install / receipt / eager-posture surface, incl. pgw#1116's fence, pgw#1141, pgw#735, pgw#1093, pgw#1033, pgw#1122, pgw#824, pgw#1078, pgw#844, pgw#672, pgw#1142): 261 passed. mypy src/gen_worker: clean, 269 files. ruff: clean. Whole-tree is CI's job.

…PORTED lane — read it off the OBJECT, not off a registry nobody fed

0.111.0 carried §4.31's warmup-barrier deletion and a real pod proved that half
works — then served eager anyway, first try: boot_adopt=hit -> cell_numerics=
armed_undispatched ("It STAYS ARMED and serves") -> 677 ms later functions=() ->
target_applicability_incomplete -> armed_target_unresolved -> serve_degrade.

THE LOCUS. `aot_serve.is_aot_ref` answers out of `_KNOWN_AOT_KEYS`, fed by
`note_aot_key`, whose rule (pgw#1033) was a CONVENTION. Its only two feeders
were `arm_from_local_store` and `adopt_delegated_mint` — both SELF-PRODUCED
routes. The ORDERED arm (`fleet_cells.arm_ordered`: every hub Plan and every
§4.27 boot-adopt) fed it nothing, so an adopted cell wrapped a live pipeline
while its own ref answered "not an AOT cell", and every reader asking "is this
the exported lane?" sorted it onto the DYNAMO lane — whose cache-hit ledger an
AOTI artifact can never move. aot_proof_before came out EMPTY, §4.31's
keep-the-arm branch could not fire for the one object it exists for, and the
cell was scored calls=0, folded into `unproven`, given function_proofs=set()
and unwrapped. functions=() made the rest inevitable.

THE FIX, in two parts.
1. `load_and_wrap` registers the key AT THE WRAP — the one seam every arm route
   passes — so a new route inherits it instead of remembering it.
2. The executor's lane readers go through `executor._exported_arm`, which asks
   the OBJECT (`aot_serve.holds_exported_cell`, deliberately distinct from
   `is_armed` so a REVOKED cell still reads as exported). The disarm authority
   can no longer be exercised over an object carrying a live cell.

Every honest de-arm keeps its teeth and each is asserted: revoked artifact ->
compiled_degraded, no target; operator eager-only still suppresses the arm;
measured evidence still disarms a dynamo object; the mint-time parity gate is
untouched.

RED, cardless, off-pod: tests/test_adopted_arm_lane_pgw1141b.py drives the real
ensure_setup -> _enable_compiled with a real _ArmOrder(adopt=) -> real
arm_ordered -> the real receipt gate against a real signed receipt from a real
HTTP hub -> provision.arm_aot -> load_and_wrap on a real packed cell whose ck1
key is restatable from its own recorded facts -> the real warmup, proof pass
and target install. On origin/master it reproduces the pod verbatim. Seams are
_boot_adopt (pgw#1116 drives its half for real), provision.load_slot, and
pgw#868's _load_package.

Also: examples/micro-diffusion/README.md §4-§7 was unexecutable —
POST /v1/admin/compile-cells no longer exists (§4.28) and `size` is an enum.
@PaulFidika
PaulFidika merged commit f3ab710 into master Aug 12, 2026
2 checks passed
@PaulFidika
PaulFidika deleted the 1141b-target-applicability branch August 12, 2026 00:05
PaulFidika added a commit that referenced this pull request Aug 12, 2026
…-arm fix in a WHEEL (#671) (#671)

Batched per docs/releasing.md rule 1 — cut when a pod run needs one. One does,
and a second consumer rides:

- pgw#1108 POD PROOF ATTEMPT #5 needs `f3ab710e` (pgw#1141b: an adopted cell is
  read off the OBJECT via `aot_serve.holds_exported_cell`, and `note_aot_key`
  now registers at the wrap site). The probe image resolves gen-worker from
  pypi.org/simple at build time, so without a published wheel attempt #5
  re-measures the exact boot 0.111.0 already failed.
- `246c0584` (pgw#1149: `Compile.speed_metric` / `min_speedup` and the open
  mint-blocker ids on the manifest's `compile` block) rides — its consumers are
  th#1811's enforcement flip and the ie#664 fleet train, both wheel-gated.

`v0.111.0..HEAD` is THREE commits, enumerated not assumed: `f3ab710e`,
`246c0584` and `cdb29129` (pgw#1108's 0.111.0 probe pin, no fragment). Both
must-rides confirmed by `git merge-base --is-ancestor`. The two `changelog.d/`
fragments on master are assembled into the 0.112.0 section; `pgw1141.md` is a
NEW file (deleted by the 0.111.0 cut at `5d8ceeaa`, re-added by `f3ab710e`), so
nothing is double-assembled.

pgw#1150 (PR #669's sibling, the author-CI harness) is NOT in this cut: PR #670
is BLOCKED with `tests` FAILURE at dispatch time and cannot merge.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant