Skip to content

Fix WIRE-385 post-mutation operator roster publication - #603

Open
huangminghuang wants to merge 26 commits into
masterfrom
fix/wire-385-stale-operator-rosters
Open

huangminghuang wants to merge 26 commits into
masterfrom
fix/wire-385-stale-operator-rosters

Conversation

@huangminghuang

@huangminghuang huangminghuang commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Publish authoritative operator rosters and select new tail members after epoch-close slash, termination and withdrawal mutations finish.
  • Restrict finishadv to self-originated inline calls for the current epoch, preserving accrual/payout ordering and refund action depth.
  • Keep ordinary schedule rotation, FIFO envelope packing and missed-delivery accounting.
  • Refresh the epoch ABI/WASM and align the Solana deposit-IDL test fixture with the existing program interface.

Why

advance read operator state before its queued inline mutations ran. An epoch could terminate an operator yet still announce it as ACTIVE or select it for the new tail. The self-inline continuation reads the resulting registry state before scheduling and publication, within the same atomic transaction.

This fixes stale publication and selection; an already unusable authorized majority can still stop delivery. WIRE-392 is required before deployment to decide recovery/resumption and duty-frequency policy. Automatic recovery remains undecided.

Validation

  • sysio_epoch_tests, sysio_epoch_flushwtdw_tests and sysio_msgch_chain_tests: 45 JIT cases / 1,837 assertions passed, covering mutation ordering, fresh rosters, slashing, termination, withdrawal, rotation and direct-call rejection.
  • Contract/native test builds passed; tracked epoch ABI/WASM match the rebuilt outputs.
  • Seven focused ROA bootstrap/accounting tests passed after merging the current base; the upstream ROA ABI/WASM also match the build.
  • Coordinated E2E: 15/15 flows passed. Verified remote checkouts: SYSIO 67fe61c2, Ethereum 3f23632a, Libraries 99c5c14a and Tools d50fc784, with unchanged Solana cbf72188 and CDT a00bd718. Tools installed SDK 1.0.93 directly from the candidate Libraries checkout. Post-remit rotation on both outposts and eligible collateral withdrawal passed; optional flow-swap-epoch-stress is excluded.
  • The known upstream SDK strict-null error (Builtins.ts:89, TS2722) is waived as a prerequisite for this E2E run only; it remains unresolved.

Companion PRs

  • Ethereum #198: enforces immediate revocation and usable next-group admission; deploy its matching library and inbound implementation together before the SYSIO producer change.
  • Libraries #82: generates the continuation interface from the SYSIO ABI; candidate E2E links this checkout directly. For published packages, release Libraries before Tools.
  • Tools #97: verifies post-termination rotation and eligible collateral withdrawal against the coordinated contract and SDK candidates.

Change-Id: I2fe1bac5e738da09019a96deae976fe727638b4f
@huangminghuang
huangminghuang requested review from a team and heifner September 7, 2026 19:08
@heifner

heifner commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Review — WNS-34 / WIRE-385

Verdict: the mechanism is correct and the change is close to minimal. One gap, on the Solana side, not in this PR.

Mechanism verified

apply_context::exec() runs for (ordinal : _inline_actions) trx_context.execute_action(ordinal, recurse_depth + 1) — each sibling's entire subtree completes before the next starts, so slashop/termcheck have mutated sysio.opreg before finishadv reads it.

Depth is fine: chkcons(0) → advance(1) → finishadv(2) → queueout/buildenv/accrueepoch/rcrdbatch(3), and all four send zero inline actions, so 3 is terminal against the limit of 4. Keeping flushwtdw/drainfwq/payepoch as siblings at depth 2 is the right call — theirs are the deep subtrees. chkcons is never sent inline.

On minimality

Nothing here looks cuttable:

  • The continuation is unavoidable. CertiK's cheaper suggestion (exclude the local to_slash list) covers slashes only — terminations are decided inside opreg::termcheck, and flushwtdw can drop collateral below minimum; neither is predictable from advance. And the slide both reads opreg and writes epoch state, so it can't move into msgch::queueout instead.
  • The withhold tightening isn't scope creep. Pruning newly creates short groups, and a published short group lowers the outpost quorum denominator — ETH's (groupSize + 1) / 2 on a group of 2 is 1, i.e. one operator settles alone.
  • Future-seat repair can't be dropped either; without it the tightened withhold wedges the window permanently on the first unrepaired hole.

Worth recording: the primary WNS-34 path never withholds at all. A non-canonical-delivery slash targets an expiring-group member, which is erased wholesale — no hole, complete window, fresh roster published. Holes only arise from out-of-band removals (chalg::slash, opreg::terminate, withdrawal-driven eligibility loss).

Gap: the Solana outpost has the defect #198 fixes on Ethereum

wire-ethereum#198 exists because "applying a newer OPERATORS status did not remove the address from cached batch-operator groups, and admission trusted that cached membership." That is true verbatim of Solana, and there is no Solana companion for WIRE-385.

  • liqsol-core/src/instructions/opp/epoch_in.rs:75-81 — Phase B admits on group membership only; the file has zero occurrences of status / operators / OperatorMapping.
  • liqsol-core/src/instructions/opp/inbound.rs:470 — handle_operators assigns registry.operators and never touches registry.groups.

So whenever this PR withholds an incomplete window, Solana retains its previous window and keeps admitting the removed operator's epoch_in deliveries, where ETH (post-#198) clears the seat. Narrower than the headline finding — it needs an out-of-band removal plus a pool too small to repair — but it is exactly the residual case #198 was written for, and Solana is uncovered.

The green e2e doesn't clear it: run 34154585838 left BRANCH_WIRE_SOLANA empty, so it floated to the manifest default and ran the unfixed program, and no flow exercises a removed operator still delivering.

Nits

  1. sysio.epoch.cpp:832 still prints sysio.epoch::advance: from inside finishadv; the sibling withhold message at :995 was updated.
  2. The comment above drainfwq (:661-670) reads as two stacked leftovers from the move, and "It needs the new epoch index" is confusing when the action takes std::make_tuple(). The point worth stating is that state_tbl.set at :652 now runs before these sends so they observe the new index.

Non-blocking: "emissions retain their … inline depth" in the description is loose — accrueepoch/rcrdbatch moved depth 2 → 3; only payepoch held depth. Safe (verified neither sends inlines), but worth correcting for anyone auditing depth headroom.

Checks run

  • sysio CI 34154430730 green on the exact head c412b58e86 (gcc/asan/ubsan/asserton/ubuntu24 — covers contracts_unit_test).
  • e2e 34154585838 pinned sysio c412b58e86 + ethereum 10993cad32, 14/14.
  • finishadv_rejects_direct_calls covers both the auth and the get_sender() guard; confirmed get_sender() returns the creator action's receiver, so the self-inline check is genuine.

…operator-rosters

Change-Id: I2d08a6a1a29481a7539da287fab3ac5382f992fb
Change-Id: I439be98d6210bfa418560a680c3436d68f9dc266
Change-Id: I1eb6737adbf418e80a81783ff327a7b267e9e302
@huangminghuang

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review. I addressed the two SYSIO nits in follow-up commit 902073f:

  • the short-tail log now identifies sysio.epoch::finishadv;
  • the deferred-maintenance comment now explains that state_tbl.set makes the new epoch visible before drainfwq, and the PR description records the actual depth change for accrueepoch/rcrdbatch.

I also traced the Solana concern against the exact revisions. The cited membership-only behavior predates SOL-377. Commit b8fae4ee added the live ACTIVE + batch-role roster check, and the current origin/next head eaebfc6 additionally scopes the seat to active_group_index. Even the earlier e2e revision 1c3f3dd already contains b8fae4ee, so no WIRE-385 Solana source delta is needed.

The Ethereum companion is now at 17392d6. It tombstones revoked or reassigned seats without shrinking quorum, rejects duplicate signer addresses, and requires a strict majority. Since the platform is pre-launch, I removed the no-op UUPS migration/reinitializer and retained the corrected contracts for fresh deployment.

Validation on exact heads 4cf466f / 17392d6 / eaebfc6:

  • all 14 local canonical flow runners and heartbeat monitors pass, including flow-batch-operator-slashing with epoch-freeze monitoring;
  • Ethereum Build and Test and artifact workflows pass;
  • SYSIO CI and the remote 14-flow platform run are still in progress.

Re-review is still requested from @heifner.

@heifner

heifner commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Re-review — WNS-34 / WIRE-385

My Solana finding was wrong — retracted. I read it off a local wire-solana checkout (af373e68) that predates b8fae4ee. origin/next (eaebfc63) contains it, and so does 1c3f3dd7, so the earlier e2e also ran the fixed program. check_epoch_in_signer (mod.rs:514) requires membership in groups[active_group_index] + a registry row + ACTIVE, and assert_signer_may_deliver → key_is_eligible requires every row for the key to be eligible. No Solana companion needed. Also refining my own claim: "the primary path never withholds" holds only while the surviving roster can still fill a full tail — your single-group test correctly asserts a withhold there.

Both nits are closed in 902073f, and CI is green on the exact head (34513732666).

One outstanding item: a withheld window permanently stalls the outposts on a multi-group schedule

The Solana program already names this failure as a rollout hazard (mod.rs:491-498): "groups[active_group_index] still holds the group that served epoch N-1, so epoch N's scheduled operator is refused NotActiveOperator and the roster that would authorise it can never land — it is inside the envelope being refused. That is a permanent stall whose only recovery is a redeploy." The withhold makes that state reachable at runtime.

  1. sysio.epoch.cpp:982 — active_group_index = current_batch_op_group + 1 = 1: every envelope announces the group serving next epoch. That lookahead is the precondition SOL-378 scoped admission to.
  2. :741-855 — the slide and state_tbl.set run unconditionally, before the publish gate at :991. Duty rotates whether or not the window ships.
  3. outpost_opp_job.cpp:52,63,95 — delivery is gated on is_elected (front group only) and pinned to read_pending_outbound(chain, current_epoch). No catch-up.

Envelope N is delivered by G_N, and the outpost learns "admit G_N" only from envelope N-1. Withhold once at N and the outpost still admits G_N when envelope N+1 arrives from G_{N+1} — disjoint by construction → refused. next_epoch_index never advances, so a later complete window fails Phase A with NonSequentialEpoch: repairing the roster does not recover it. ETH absorbs one extra epoch (_resolveChunkPosition scans all groups), and sysio.msgch.cpp:1476 requires all outposts, so the depot freezes too.

Trigger: ACTIVE batch ops below batch_operator_minimum_active on a window of ≥2 groups. Normal removals hit an expiring-group member and leave no hole, so it needs the roster genuinely below minimum — one slash or termcheck termination does it when the roster sits exactly at minimum, which is how the harness provisions and a plausible 21-operator launch shape.

Not covered: contract tests can't observe outpost admission; flow-batch-operator-slashing runs batch_op_groups = 1 (index falls back to 0, same group every epoch); flow-batch-operator-termination provisions a 10th operator, so ACTIVE returns to 9, the tail fills and nothing is withheld.

Refusing to publish a short group is right — master's alternative shrinks the quorum denominator. But the trade is fail-open → fail-closed-and-unrecoverable, and the description ("withholding … until the roster is repaired", "Current-group vacancies require roster recovery") promises a recovery the mechanism can't deliver.

Suggested direction: don't let duty advance past the group the outposts were last told about — commit the slide only when the previous window was published. The first withheld epoch is safe (that duty was already announced); freezing from the second keeps a full-size group serving, denominator intact, channel live until the roster is repaired. Failing that, say plainly in the deployment notes that a below-minimum roster on a multi-group window is terminal for the outposts.

Otherwise clean

emission_amount types line up; the reordered flushwtdw/drainfwq only read current_epoch_index (set at :652 before the sends) and queue no attestations; require_fresh_roster asserts the property CertiK actually raised.

…operator-rosters

Change-Id: I2d74cc7bccb284c3330e7845cde273ec6a858644

# Conflicts:
#	contracts/sysio.epoch/sysio.epoch.wasm
Change-Id: I19e4974e0f48b42dec305712c3d6eb3332d8c552
…operator-rosters

Change-Id: I545e7f1e2b830bdfeaed8d9b00f84d854c77bd53
…operator-rosters

Change-Id: Ie8265a67b31f01661ff8445996ece22bf6e652f5
@huangminghuang

Copy link
Copy Markdown
Contributor Author

Addressed in 22bf9d1 and now published on the updated PR head 5d6f8e0.

The held schedule is now an explicit publication checkpoint rather than a terminal state. While a window is incomplete, advance() continues publishing the authoritative OPERATORS roster, protects the group currently on duty, and retries future-seat repair from healthy standbys. It withholds BATCH_OPERATOR_GROUPS until the window is complete; once replacements are available, it publishes the repaired window and resumes normal rotation. The single-group case repairs an ineligible seat in place without shifting healthy positions.

The companion Solana change permits an inactive identity only in a group strictly before active_group_index, preserving a historical quorum denominator without authorizing an inactive active/future signer. The expanded tools scenario holds an incomplete window for two epochs while both outposts progress, provisions replacements, then proves each replacement serves its exact duty epoch on Ethereum and Solana.

Regression coverage for withheld-window recovery, announced-successor protection, single-group repair, historical placeholders, and pruning passes in both complete 735-case SYSIO suites. Companion PRs: Solana #522, libraries #82, tools #97, and Ethereum #198.

Change-Id: Icf26c469260273eae00807ee342de882412fa8af
Change-Id: Ib910bb73aa5fcb157dfababb4f257d13cab13b3e
Change-Id: I64773f671fdecc91a8b9ef7597547e8a4490ff8d
…operator-rosters

Change-Id: I05581c1d8f42690f41b49b364420c4ab81ce65c5
@heifner

heifner commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Re-review — scope and simplification

The next_batch_op_groups double-buffer is the right encoding, and publish-then-activate closes the stall cleanly. Notes below are about volume, not mechanism.

Split out of this PR

  1. contracts/tests/sysio.roa_tests.cpp (noop_wasm → sysio_token_wasm) — unrelated to WIRE-385, and master is green at this PR's own merge-base (927368eb) with noop_wasm() still in place. Either it fixes a real breakage — which belongs on master — or it's a local-build workaround.
  2. tests/fixtures/solana-idl-opp-outpost-stub.json — a deposit-IX account-list refresh (operator_registry loses writable, new collateral_position). That tracks a wire-solana deposit change; this PR touches epoch_in/groups. Split unless it's needed to keep the IDL-compat test green against the exact SOL revision this lands with — if so, name that commit.

(The six extra rebuilt .wasm artifacts are legitimate: epoch_state gained a field and msgch/opreg/uwrit read that table.)

Simplifications

  1. expired + retained_groups + the nested find_if (~8 lines, 2 locals) can become pool.front(). The exclusion is pure duty-spacing and provably never blocks a completable window: with ACTIVE >= window, vacancies = |pool| + (window - ACTIVE) <= |pool|, and the tail alone contributes operators_per_epoch vacancies, so |pool| > operators_per_epoch = |expired| — a non-expired candidate always exists.
  2. cfg.batch_op_groups > 1 ? 1 : 0 is computed three times (cursor assignment, next_group_index, the expired guard). One named local — it is the serving index — keeps the three from drifting.
  3. Restore the deleted lookahead rationale ("the roster that would authorise the delivery is inside the envelope being refused"). That is still the cross-repo contract Solana's check_epoch_in_signer depends on and what this redesign now protects; four lines replaced ~40.

Two things I looked at and would not cut: the name{} sentinel is load-bearing (at batch_op_groups == 1 the loop repairs the serving group in place, so replacements must keep their index), and dropping the old %0/empty-next guard is correct now that publication implies a complete window.

One gap: the new tests cannot see the production recovery path

advance_to_next_epoch() (:844) pushes advance directly under MSGCH_ACCOUNT authority, bypassing chkcons. In production the advance that publishes the repaired window only fires once all outposts have consensus, which requires the frozen duty group to deliver — and msgch::deliver gates on live ACTIVE status.

advance_preserves_ineligible_announced_successor_position is the concrete case: after the hold, duty_member() is BATCHOP_C, who was just terminated, and the fixture runs operators_per_epoch = 1 — so the frozen duty group is one non-ACTIVE operator, nobody can deliver, and the repaired window never publishes. The test passes only because it forces the advance. chkcons being unauthenticated doesn't help; the gate it can't pass is outpost consensus. Benign at operators_per_epoch >= 3 (a majority usually survives); it bites when the whole duty group goes, which at group size 1 is a single slash.

What advances the epoch out of a hold whose duty group can no longer deliver — and can a test assert it without forcing advance?

Cross-repo note

Retaining inactive placeholders in candidate group 0 is what forces Solana #522 to permit an inactive identity in groups before active_group_index — a deliberate relaxation of the SOL-377/378 gate. Defensible (late deliveries must no-op rather than revert), but the justification currently lives only in the companion PR; worth a sentence here.

Change-Id: I7d6033d5e3eb607065429fe2b567c97634771652
@huangminghuang

Copy link
Copy Markdown
Contributor Author

Addressed in SYSIO follow-up 2ca8d89, with the deterministic flow correction in Tools 8d53c83.

Scope and simplification

  • Restored the ROA test to its original noop_wasm() fixture; there is no ROA diff left in the PR. The substitution was a workaround for an incomplete local build, while the full build generates the fixture correctly.
  • Kept the Solana IDL fixture because solana_idl_stub_consistency_test must match companion Solana head 5c11c8f. collateral_position came from 46d110d, and operator_registry became read-only in 084bbe5. This keeps the packet-sizing fixture aligned with the actual companion IDL; WIRE-385 does not change deposit behavior.
  • Removed expired, retained_groups, and the nested find_if; vacancies now consume pool.front() deterministically.
  • Replaced the repeated one-group/multi-group expression with serving_group_index and restored the lookahead rationale above publication.

Production recovery path

The recovery regressions now use the production sequence after genesis: deliver → chkcons → advance. A healthy held duty group delivers through repeated holds, a newly registered standby completes the next publication, and the repaired window activates only on the following advance.

For the complete-duty-loss case, the direct answer is: nothing advances it through normal consensus. deliver still requires an ACTIVE authorized quorum, and permissionless chkcons cannot manufacture that quorum. A separate regression terminates every signer in the held group, registers ACTIVE standbys, calls chkcons repeatedly, and asserts that the epoch remains blocked. The PR description now states this limit explicitly: automatic recovery requires a healthy authorized quorum and does not recover total held-signer loss.

Historical seats and Solana

Candidate group 0 preserves the previous serving group’s exact positions, including inactive historical placeholders, so the quorum denominator and the duty that authorized the envelope do not change underneath late deliveries. Solana #522 permits an inactive identity only in a group strictly before active_group_index; the active and future groups still require live ACTIVE batch operators.

The Tools flow also now makes both replacements necessary for quorum by stopping an original peer in each replacement group before checking their accepted signatures. This avoids accepting a late third signature as proof. The final coordinated validation passed all 15 local flows and monitors and all 15 remote flows, with 2,734 successful steps and all 228 recovery steps.

@heifner

heifner commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Re-review — head 2ca8d89c45

All three simplifications landed, the lookahead rationale is back, the ROA fixture is reverted, and the IDL refresh is justified with its companion commits. Recovery tests now drive deliver → chkcons → advance instead of forcing the advance, and chkcons_cannot_recover_without_a_live_held_group_signer pins the limitation explicitly. CI green across five Linux variants plus macOS; 15 flows local and remote.

Re-verified on the final code: published groups stay mutually disjoint (resident includes group-0 placeholders, so the pool can't re-seat them), publish_schedule fails closed when the cursor is past the window, genesis works because the outpost skips Phase B while registry_initialized is false, and in both steady state and hold the depot's duty equals the outpost's admitted group.

Credit where due: Tools #97 covers the hold path better than I expected — it drives a withheld window, then runSlashHeldGroupMember removes a member of the currently held group, then provisions replacements and asserts the repaired window publishes and duty rotates. That is exactly the scenario I was worried about, and it passes.

Two things below.

1. termcheck runs at the rotation rate while duty is held

  1. sysio.epoch.cpp:500-504 — the miss ladder runs every advance past genesis against batch_op_groups[current_batch_op_group].
  2. On a hold that cursor and window don't move (finishadv:714-719), so the same group is the expiring group every epoch.
  3. :634-646 — each advance sends recorddel + termcheck for every member of it.
  4. sysio.opreg.cpp:234-237 — terminate_window_ms is validated as spanning "terminate_max_consecutive_misses + 1 duty rotations", sized from batch_op_groups. A frozen duty is on the ladder every epoch, so the consecutive rail fills batch_op_groupsx faster than the value setconfig accepted.

It self-limits: evalcons takes group_size = eligible.size() (the live count), so at 2 live the majority is 2, a miss stalls instead of advancing, and no further termination is recorded. The ladder can strip a held group to 2 live, not to zero — reaching zero needs an outside slash or manual terminate, which is how the test gets there.

Still the wrong direction: each termination permanently removes an ACTIVE operator (with bond remit) from a roster already below minimum, raising the number of standbys needed to escape the hold and ending with zero fault tolerance.

Suggested fix: gate termcheck on activate_schedule. A group serving only because the schedule is frozen is not missing an assignment it was rotated into. Keep recorddel for the audit trail.

Severity is bounded by topology, which leads to the second point.

2. Every e2e flow runs zero slack, so the production path is the untested one

All 15 flows configure seated == roster:

Flow count shape seated spares
10 flows on CLI defaults, plus node-owner-nft and yield-distribution 3 1 x 3 3 0
flow-batch-operator-slashing 3 3 x 1 3 0
flow-batch-operator-termination 9 3 x 3 9 0 (+1 doomed, +2 recovery mid-run)
flow-emissions-soak 21 7 x 3 21 0

With seated == roster the schedule has no slack at all: the fill pool each epoch is exactly the group that just rotated out, and it exactly fills the new tail. Any removal leaves the tail one short every epoch after, so the window holds until the roster is whole again — and "removal" includes the two events this PR is about, a non-canonical-delivery slash and a miss-ladder termination.

The cluster path assumes the opposite. create-wire-platform-cluster.yaml:417-458 caps the roster at 26 and names the canonical explicit topology as {batchOperatorCount: 26, operatorsPerEpoch: 7, batchOpGroups: 3} → 21 active, i.e. 5 standbys; the harness schema only rejects total > count, so slack is legal whenever the shape is authored. At 26/21 a slash is absorbed by a standby, the candidate completes, and nothing is ever withheld — no hold, no pinned duty, and finding 1 never fires.

So the coverage runs backwards: the flows exercise the rare path (hold, then recovery) while the common one is only passed through, never asserted. flow-batch-operator-termination does traverse an absorbed removal — its doomed operator is an extra 10th against 9 seats, so terminating it returns the roster to exactly 9 and the window still completes — but nothing asserts that, and with the roster back at the seat count the repair pool is always the group that just rotated out. A genuine standby seated ahead of a recycled member is never exercised, and that is the only path a 26/21 cluster takes. Concrete request below.

Also worth confirming what the deployed clusters actually pass: the real values live in cluster-build-options.json behind CLUSTER_CONFIG_URL, not in the repo. If dev or prod runs 21/21 rather than 26/21, the first slash the CertiK path ever produces holds that network, and finding 1 is on the likely path rather than a fallback.

3. Requested coverage: a removal absorbed by a standby

Suggested shape — extend flow-batch-operator-termination, don't add a flow:

  1. Give it a standing spare. BatchOperatorCount: 9 → 11, keeping OperatorsPerEpoch: 3 / BatchOperatorGroups: 3. That is 9 seats with 2 continuous standbys — the 26/21 shape in miniature. The explicit-shape path already permits it (total <= count; the {3,9,15,21} lattice only binds the fully-derived path), and schbatchgps seats 9 and leaves 2 in the pool.

  2. Assert the absorbed removal at the existing Terminate / RemitBonds boundary, before anything starves the roster:

    • a new window is published on the advance following the termination — epochstate.next_batch_op_groups is non-empty, and both outposts seat it (their active_group_index and group membership move forward in lock-step);
    • current_batch_op_group advances on the next activation, i.e. duty rotates and never freezes;
    • the terminated operator appears in no published group, and one of the standbys occupies the vacated seat;
    • both outpost epoch cursors keep advancing — no hold, no stale window.
  3. Knock-on: StarveScheduleWindow is tuned to "slashing one seated operator makes the next tail one seat short", which holds only at 9 ACTIVE / 9 seats. With 11 it needs to remove count - seats + 1 (= 3). Deriving that from the constants rather than hardcoding keeps the two halves of the flow independent.

Ordered this way, one run covers the degradation ladder in the order a real network experiences it: healthy with spares → removal absorbed silently → spares exhausted → hold → recovery. The PR97 phases are unchanged apart from the starve count.

Why not a new flow: a scenario fixes its topology once at bootstrap, so covering both "with spares" and "below minimum" needs either two flows or one flow that walks from the first state into the second. A new flow means another full cluster bootstrap in the gate plus duplicated provisioning, ad-hoc daemon, and checkpoint machinery that #97 already built. If you would rather not perturb a freshly tuned scenario, the alternative is a small dedicated flow at {count: 11, operatorsPerEpoch: 3, batchOpGroups: 3} doing only step 2 — same assertions, more wall-clock.

To be clear this is a coverage request, not a defect: the absorbed path is the simple case and advance_repairs_future_group_before_it_becomes_current covers the repair mechanics at unit level. What is missing is an end-to-end assertion that a production-shaped cluster rides through an operator loss with no visible schedule disruption on either outpost.

Minor

  • The PR says total held-signer loss has no recovery. advance accepts has_auth(MSGCH_ACCOUNT) || has_auth(get_self()), so sysio.epoch's own authority can force one advance and let finishadv publish a repaired window. Worth confirming the outposts accept that envelope given their frozen mirror — if they do, the runbook is a governance action rather than "no recovery".
  • contracts/test_contracts/sendinline/sendinline.wasm is the one artifact not explained by the epoch_state layout change: identical total and per-section sizes, only the import order swapped (send_inline <-> sysio_assert_code), which renumbers indices. It's a unittest fixture that can't include sysio.epoch.hpp — link-order churn, worth a git checkout.

…operator-rosters

Change-Id: I73a7e9d7387a93951cc529870d1bcc02e37df2f1
Change-Id: I887a051908fae75b95e4e97394cbaf1d81d67005
Change-Id: I4f087057908d9eb7c5e3624540e7f0066aaf13fc
@huangminghuang huangminghuang changed the title Fix WIRE-385 stale outbound operator rosters Fix WIRE-385 roster publication and held-duty accounting Sep 15, 2026
@huangminghuang

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed re-review. I addressed the four points in follow-up commits across SYSIO, Tools, and SDK.

  1. SYSIO still calls recorddel for each real outpost observation, but calls termcheck only when the expiring epoch had a complete published successor. A separate heldepochs marker keeps those audit rows out of both the immediate and later consecutive and percentage checks. The focused regression records four misses over two held epochs, then verifies a later check after publication resumes leaves the operator ACTIVE.

  2. The existing Tools termination flow now has 11 bootstrapped ACTIVE operators for nine seats after the doomed non-bootstrapped operator terminates. It checks a complete successor on the depot, an absorbed seat filled by a standing operator, live active and future groups on Ethereum and Solana, group identity changing on activation, and sequential epoch cursors on both outposts. Two separately reported standing-spare slashes precede the seated loss that starts the existing freeze/recovery path.

  3. sysio.epoch authority can call advance, but that does not bypass the normal chkcons signer gate or give either outpost an authorized held-group delivery after every held signer is ineligible. The PR states this limitation explicitly. I have not verified the private deployed cluster-build-options.json topology; the flow proves the explicit 11/9 spare shape without assuming dev or prod values.

  4. sendinline.cpp and its CMake inputs are unchanged. The current ON/ON CDT build regenerates the fixture with the import-order change you identified, and all tracked artifacts are required to match the generated bytes. Restoring only the upstream WASM would break that parity, so I kept the regenerated fixture and explained it in the PR description.

Exact-head native checks passed, and all 15 local flow runners, heartbeat monitors, and reports completed successfully. The termination-recovery report passed 248 steps, including the absorbed-removal and later hold/recovery assertions. I do not claim a remote full-platform result here.

Change-Id: Ifd9e44dc1087e9e6f68afe2fb87c8baf46d1c9fb
Change-Id: I52892664d52b635c86d0f8fb40b0d8cab7021dda
Change-Id: I406eeeadd32a75db1c60458a26d952a40a2b4fed
@huangminghuang
huangminghuang marked this pull request as draft September 22, 2026 16:30
Change-Id: I99509d0383e3f9eefed8b297f1f59704effdf65f
Change-Id: I87ef44d4d9eea670e995afb2fd5ba5ae66be5ede
Change-Id: I37064cb952eb7ec9af71f5c603a54245c1f08bfc
@huangminghuang huangminghuang changed the title Fix WIRE-385 roster publication and held-duty accounting Fix WIRE-385 post-mutation operator roster publication Sep 22, 2026
@huangminghuang
huangminghuang marked this pull request as ready for review September 23, 2026 10:37
@heifner

heifner commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Re-review — head 83b662d127

The reduced change is correct and matches the WIRE-385 agreed scope. Nothing blocking: one coverage request on Tools #97 and two nits.

Verified

  • Run order: slashop → recorddel/termcheck → flushwtdw → drainfwq → finishadv (roster + tail) → payepoch. The deepest action is at depth 3, under finishadv; queueout/buildenv/accrueepoch/rcrdbatch send no inline actions, here or on current master.
  • The only other reorder is drainfwq ahead of the roster queueouts. It queues nothing outbound, so packing is unchanged. The actions before finishadv now see the new epoch index with the old window, and none of them reads the window.
  • Making BATCHOP bootstrap-exempt in advance_ships_lookahead_batch_operator_group is required. It never delivers, so the percent rail (1/1 > 99%) terminates it at its first miss. On master the test passed only because the terminated operator stayed seated.
  • CI is green on the head. E2E 35807790317 passed 15/15 on the exact four heads plus Solana cbf72188.

Coverage — Tools #97

The absorbed-removal assertions from my 09-14 ask were dropped with the recovery scenarios in f3237b0. They aren't recovery. They cover the ordinary rotation this PR changes, and they're the one e2e check that shows its Solana effect:

  • At cbf72188, handle_batch_operator_groups refuses the whole window if any seated member is ineligible (inbound.rs:2270). active_group_index only moves through set_groups, and check_epoch_in_signer admits only groups[active_group_index] (mod.rs:649).
  • On master, flow-batch-operator-termination re-seats newop in the tail of the advance that terminates it. The next roster marks it TERMINATED, Solana refuses that window and keeps admitting the previous group, and so the envelope after that is refused. The remits land first, so the flow ends before the stall.
  • On this PR, newop never re-enters the window.

Ask: run the flow a few epochs past RemitBonds, and assert that both outposts keep advancing, that newop is in no published group, and that a standing operator took its seat. By the code above, that fails on master and passes here.

Nits

  • sysio.epoch.cpp:794 and :978 print sysio.epoch::advance: from finishadv. This was my 09-09 nit, fixed in 902073f and undone by the scope reset.
  • Comments in the moved block still name advance (:908, :915, :929, :972, :1007). :972 and :1007 describe returning early from the wrong function. The comment above advance_withholds_batch_operator_groups_when_next_group_is_empty has the same problem.

@huangminghuang
huangminghuang marked this pull request as draft September 24, 2026 15:35
…operator-rosters

Change-Id: Ie693257cc60afd26b53e52aea2041de956eb2b7c
Change-Id: I367666bb68d649234e30caede0ca3d1f79e6ea86
Change-Id: I1f198c1d4db6961958b2d2b787bb95d0df71ece6
@huangminghuang

Copy link
Copy Markdown
Contributor Author

Addressed the follow-up in the current candidate:

  • Corrected the finishadv log prefixes and moved function/test comments in fce632110e; rebuilt the tracked epoch artifact in dce1c13b8f.
  • Tools #97, commit 59e9cf6f, now continues after RemitBonds. It decodes the published group attestations for both chains, checks that each full group contains distinct standing operators and excludes the terminated account, and requires both outpost cursors to advance through a complete additional rotation. This extends the existing termination flow without introducing recovery behavior.
  • Upstream Ethereum net_plugin cleanup #205 restores the roster-seeding API used by the existing bootstrap. Removed the redundant six-file bootstrap refactor; the Tools PR now contains only the termination coverage and collateral-withdrawal fixture correction.

Focused validation passed: 24 SYSIO JIT tests (1,481 assertions), 107 Ethereum outpost tests plus seven deployment tests, 13 SDK client tests, and 32 bootstrap tests. The updated termination flow builds and passes changed-file lint; all 48 tracked SYSIO artifacts match the native build.

The new coordinated E2E run is validating the full standard suite against the four pinned companion heads, including the extended termination flow. All four PRs remain draft pending remote validation. Automatic recovery and duty-frequency policy remain in WIRE-392.

@huangminghuang
huangminghuang marked this pull request as ready for review September 24, 2026 20:36
heifner
heifner previously approved these changes Sep 25, 2026

@heifner heifner left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review — head dce1c13b8f

Everything from my last round is in. Approving.

  • Both prints now say sysio.epoch::finishadv:, and the moved comments name finishadv.
  • Tools #97 runs the termination flow a full rotation past RemitBonds. E2E 36044729108 passed 15/15 on the exact heads (sysio dce1c13b8f, ETH 7ee482f5, SDK c3c21fc0, Tools 082d21a1, Solana cbf72188), and post-remit-rotation passed after waiting 482s for both outposts to advance.
  • The earlier red run (36024314509) failed only on the flow's unresolved ctx.config.operatorsPerEpoch (null on the derived path). Its epoch-7 envelope already showed the terminated operator reported TERMINATED and its seat filled by a standing operator. 082d21a1 fixed the assertion.
  • CI is green on the head. Master since the merge base is query-engine work only.

Change-Id: If2e0112676a084c956e862e0d17ea3897a620e8a
@huangminghuang
huangminghuang marked this pull request as ready for review September 25, 2026 22:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants