pgw#1141 (REOPENED by POD PROOF #4): a boot-adopted cell is on the EXPORTED lane — read it off the OBJECT, not off a registry nobody fed - #669
Merged
Conversation
…PORTED lane — read it off the OBJECT, not off a registry nobody fed
0.111.0 carried §4.31's warmup-barrier deletion and a real pod proved that half
works — then served eager anyway, first try: boot_adopt=hit -> cell_numerics=
armed_undispatched ("It STAYS ARMED and serves") -> 677 ms later functions=() ->
target_applicability_incomplete -> armed_target_unresolved -> serve_degrade.
THE LOCUS. `aot_serve.is_aot_ref` answers out of `_KNOWN_AOT_KEYS`, fed by
`note_aot_key`, whose rule (pgw#1033) was a CONVENTION. Its only two feeders
were `arm_from_local_store` and `adopt_delegated_mint` — both SELF-PRODUCED
routes. The ORDERED arm (`fleet_cells.arm_ordered`: every hub Plan and every
§4.27 boot-adopt) fed it nothing, so an adopted cell wrapped a live pipeline
while its own ref answered "not an AOT cell", and every reader asking "is this
the exported lane?" sorted it onto the DYNAMO lane — whose cache-hit ledger an
AOTI artifact can never move. aot_proof_before came out EMPTY, §4.31's
keep-the-arm branch could not fire for the one object it exists for, and the
cell was scored calls=0, folded into `unproven`, given function_proofs=set()
and unwrapped. functions=() made the rest inevitable.
THE FIX, in two parts.
1. `load_and_wrap` registers the key AT THE WRAP — the one seam every arm route
passes — so a new route inherits it instead of remembering it.
2. The executor's lane readers go through `executor._exported_arm`, which asks
the OBJECT (`aot_serve.holds_exported_cell`, deliberately distinct from
`is_armed` so a REVOKED cell still reads as exported). The disarm authority
can no longer be exercised over an object carrying a live cell.
Every honest de-arm keeps its teeth and each is asserted: revoked artifact ->
compiled_degraded, no target; operator eager-only still suppresses the arm;
measured evidence still disarms a dynamo object; the mint-time parity gate is
untouched.
RED, cardless, off-pod: tests/test_adopted_arm_lane_pgw1141b.py drives the real
ensure_setup -> _enable_compiled with a real _ArmOrder(adopt=) -> real
arm_ordered -> the real receipt gate against a real signed receipt from a real
HTTP hub -> provision.arm_aot -> load_and_wrap on a real packed cell whose ck1
key is restatable from its own recorded facts -> the real warmup, proof pass
and target install. On origin/master it reproduces the pod verbatim. Seams are
_boot_adopt (pgw#1116 drives its half for real), provision.load_slot, and
pgw#868's _load_package.
Also: examples/micro-diffusion/README.md §4-§7 was unexecutable —
POST /v1/admin/compile-cells no longer exists (§4.28) and `size` is an enum.
PaulFidika
added a commit
that referenced
this pull request
Aug 12, 2026
…-arm fix in a WHEEL (#671) (#671) Batched per docs/releasing.md rule 1 — cut when a pod run needs one. One does, and a second consumer rides: - pgw#1108 POD PROOF ATTEMPT #5 needs `f3ab710e` (pgw#1141b: an adopted cell is read off the OBJECT via `aot_serve.holds_exported_cell`, and `note_aot_key` now registers at the wrap site). The probe image resolves gen-worker from pypi.org/simple at build time, so without a published wheel attempt #5 re-measures the exact boot 0.111.0 already failed. - `246c0584` (pgw#1149: `Compile.speed_metric` / `min_speedup` and the open mint-blocker ids on the manifest's `compile` block) rides — its consumers are th#1811's enforcement flip and the ie#664 fleet train, both wheel-gated. `v0.111.0..HEAD` is THREE commits, enumerated not assumed: `f3ab710e`, `246c0584` and `cdb29129` (pgw#1108's 0.111.0 probe pin, no fragment). Both must-rides confirmed by `git merge-base --is-ancestor`. The two `changelog.d/` fragments on master are assembled into the 0.112.0 section; `pgw1141.md` is a NEW file (deleted by the 0.111.0 cut at `5d8ceeaa`, re-added by `f3ab710e`), so nothing is double-assembled. pgw#1150 (PR #669's sibling, the author-CI harness) is NOT in this cut: PR #670 is BLOCKED with `tests` FAILURE at dispatch time and cannot merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What the pod said, on the wheel that was supposed to fix this
gen-worker 0.111.0 (which carries pgw#1141's
0dbf68e5), pod5od13p840v1bgb, RTX A4500, first try — POD PROOF #4 in pgw#1108:seq 12 is 0.111.0's own new sentence, so §4.31's warmup-barrier deletion is genuinely live on a pod. And 677 ms later the object was unwrapped anyway. PR #658's claim that this chain was "structurally unreachable" on the exported lane was false on the shipped wheel. The disarm moved one branch over; it did not disappear.
The locus — neither of the two recorded candidates, and it explains both
The tracker recorded two candidates: (1)
permitted_namesempty becauseexported_armis False (active_selectionNone), or (2)compatible_namesempty from the alias loop. It is candidate 1's OUTCOME by a third mechanism, and the mechanism is upstream of the install entirely.aot_serve.is_aot_refanswers out of_KNOWN_AOT_KEYS, a process-global set fed bynote_aot_key. pgw#1033 wrote the feeding rule as a convention — "whoever reads acell_keyoff an aot-inductor envelope registers it" — and it had exactly two feeders:fleet_cells.arm_from_local_store(fleet_cells.py:2044)fleet_cells.adopt_delegated_mint(fleet_cells.py:2181)Both self-produced.
fleet_cells.arm_ordered— which is every hub Plan arm and every §4.27 boot-adopt — fed it nothing. So on the pod:is_aot_ref(adopted_ref)→ False, on the very pod that was serving those bytes;_proves_by_fx(ref)→ True → the artifact went intoproof_before, the DYNAMO lane's cache-hit ledger — which an AOTI artifact can never move (it performs no FX lookup);aot_proof_before→ empty → §4.31'sif id(pipe) in aot_proof_before: keep the armbranch could not fire for the one object it exists for;calls <= 0→unexercised→dynamo_unexercised(also keyed onaot_proof_before) →unproven→function_proofs[id] = set(),aot_serve.unwrap,compile_cache.unwrap,active_compile_artifacts.pop. This is seq 12'sarmed_undispatchedrow — emitted with the dynamo reason, which is why the leg read as "pgw#1141 works";active_selectionis now None →exported_armFalse →permitted_names = function_proofs[id]=set()→functions=()→ seq 13, 14, 15.Every observed field matches:
functions=(),bindingsnon-empty and valid,owned_slots=['pipeline'],armed=False(compile_cache marker gone),targets_resolve=True(structural).Why pgw#1141's tests were green on the wheel that fails on a pod: their fleet-policy stand-ins call
aot_serve.note_aot_key(key)by hand —test_adopted_cell_warm_proof_pgw1141.py::_fake_adopt_armandtest_aot_boot_proof_gap_pgw735.py::_fake_arm. That single line is the one thing no production arm route did. The tests entered one gate east of the bug, twice.The cross-wheel confound in the tracker (the adopted cell was minted by 0.103.0) is eliminated: the SDK version is not an input to any branch here, and the repro is same-wheel throughout.
The fix
1. Root, structural — register at the wrap.
aot_serve.load_and_wrapcallsnote_aot_key(meta["cell_key"])at thesetattr(pipeline, _MARKER_ATTR, …)that makes the cell live. That is the one seam every arm route passes and the moment the fact becomes true, so a future arm route inherits the registration instead of having to remember it. The two existing feeders stay (they register before a wrap, whichserving_modereads).2. The reader — ask the object. New
aot_serve.holds_exported_cell(pipeline)(marker present, armed or revoked) andexecutor._exported_arm(pipeline, ref). All five lane readings now go through it:_proves_by_fx,aot_proof_before,_install_compile_targets'sexported_arm,dynamo_unexercised, and §4.31's own keep branch.holds_exported_cellis deliberately notis_armed— a revoked cell must still read as exported so the install refuses to advertise it, while an object carrying no cell reads as an ordinary dynamo/eager one. With this, the dynamo lane's disarm authority cannot be exercised over an object carrying a live cell even if no registry ever learned its key — that is the property the third row asserts, with the registry deliberately emptied after the wrap.Reader sweep — every site that can unwrap / de-arm / refuse to serve an armed cell
executor.py:7725-7729aot_serve.unwrap+compile_cache.unwrapin theunprovenfolddynamo_unexercisednow uses_exported_arm, so an exported cell can no longer enter this fold at all.executor.py:7853theunexercisedloopaot_proof_beforemembership, i.e. on the registry). Now gated on_exported_arm(pipe)— the lane, not a snapshot.executor.py:7873compile_cache.unwrap(dynamo unexercised, non-mandatory)executor.py:4738-4756_install_compile_targetsexported_arm+ theCOMPILED_DEGRADEDcontinueexported_armwas False, so a revoked adopted cell fell through to the samefunctions=()path and reported the wrong reason. The negative test pins that it now reportscompiled_degraded.executor.py:4610_eager_only_reasonearly returndegree > 1(pgw#775)executor.py:9598self_mint_skipped insufficient_vramunwrappending_self_mints, an adopted cell has none.executor.py:9717abandon_background_mintunwrapprovision.py:460aot_serve.unwraponnumerics_refusedverify_numerics=Trueadopt_delegated_mint, the pod that just compiled the bytes. Adoption passes False.local_cells.py:278,285cc.unwrapcompile_cache.py:458,476,507serve_degradeemittersexecutor.py:4584serve_degradefrom_assert_armed_targets_installedserving_mode.classify_mode:133is_aot_ref(ref)if pipeline is not None and aot_serve.is_armed(pipeline): return MODE_AOT_CELL). Pre-fix it would still have misreportedaot_cellasjit_cellon any call site that passes no pipeline. Fix 1 closes it at the source; no change needed here.arm_ordered'sserve_posture.eager_only()short-circuitaot_serve.wrap_modulein-request guard revocationVerdict: no fourth structural asymmetry. Rows 2 and 4 were the surviving instances; row 12 is the same class caught before it could cost anything and is closed by fix 1.
RED evidence
tests/test_adopted_arm_lane_pgw1141b.py— 8 rows, 8/8 fail onorigin/master'ssrc/, 8/8 pass here. On master the log is the pod, verbatim:What runs for real:
ensure_setup→_injection_kwargs→_enable_compiledwith a real_ArmOrder(adopt=…)→ realfleet_cells.arm_ordered→ the real receipt gate against a real RSA-signed receipt from a real HTTPHubStub→provision.arm_aot→load_and_wrapon a real packed cell whoseck1key is genuinely restatable from its own recorded facts (cell_key.from_exported_artifact_metadata, the same recomputation admission runs) → the real boot warmup → the real proof pass → the real_install_compile_targetsand_assert_armed_targets_installed.Three seams, all WEST of the locus, all named in the module docstring:
Executor._boot_adopt(its derive+resolve half is driven end to end for real bytest_boot_adopt_observability_pgw1116.pyagainstexamples/micro-diffusionand a real hub),provision.load_slot(the weights read), and pgw#868's_load_package(the AOTI.so— the one piece deferred to a pod). Nothing about applicability, the lane split, the proof pass or the target install is stubbed.Negatives asserted: a revoked cell installs no target and names
compiled_degraded; the operator's eager-only order still suppresses the arm (serving_tiers()["generate"] == "eager");holds_exported_cellstays True across a revoke and False afterunwrap.Runbook fix (side item)
examples/micro-diffusion/README.md§4-§7 was unexecutable and burned pod-leg time.POST /v1/admin/compile-cellsno longer exists — §4.28 retired the mint-only pod class; a serving pod self-mints in the background — so the runbook now buys a serving worker (POST /v1/admin/releases/<id>/workers?count=1&compute_class=gpu) and drives one request.sizeis an enum ("256x256"), not a{"width":…,"height":…}object, which is a400. §7 now names the four typed rows that decide GREEN, and namestarget_applicability_incomplete/armed_target_unresolvedas this issue recurring.Not in this PR
Local verification
tests/test_adopted_arm_lane_pgw1141b.py+ 34 neighbouring files (the whole adopt / arm / proof / install / receipt / eager-posture surface, incl. pgw#1116's fence, pgw#1141, pgw#735, pgw#1093, pgw#1033, pgw#1122, pgw#824, pgw#1078, pgw#844, pgw#672, pgw#1142): 261 passed.mypy src/gen_worker: clean, 269 files.ruff: clean. Whole-tree is CI's job.