Fix WIRE-385 post-mutation operator roster publication - #603
huangminghuang wants to merge 26 commits into
Conversation
Change-Id: I2fe1bac5e738da09019a96deae976fe727638b4f
Review — WNS-34 / WIRE-385Verdict: the mechanism is correct and the change is close to minimal. One gap, on the Solana side, not in this PR. Mechanism verified
Depth is fine: On minimalityNothing here looks cuttable:
Worth recording: the primary WNS-34 path never withholds at all. A non-canonical-delivery slash targets an expiring-group member, which is erased wholesale — no hole, complete window, fresh roster published. Holes only arise from out-of-band removals ( Gap: the Solana outpost has the defect #198 fixes on Ethereumwire-ethereum#198 exists because "applying a newer
So whenever this PR withholds an incomplete window, Solana retains its previous window and keeps admitting the removed operator's The green e2e doesn't clear it: run Nits
Non-blocking: "emissions retain their … inline depth" in the description is loose — Checks run
|
…operator-rosters Change-Id: I2d08a6a1a29481a7539da287fab3ac5382f992fb
Change-Id: I439be98d6210bfa418560a680c3436d68f9dc266
Change-Id: I1eb6737adbf418e80a81783ff327a7b267e9e302
|
Thanks for the detailed review. I addressed the two SYSIO nits in follow-up commit
I also traced the Solana concern against the exact revisions. The cited membership-only behavior predates SOL-377. Commit The Ethereum companion is now at Validation on exact heads
Re-review is still requested from @heifner. |
Re-review — WNS-34 / WIRE-385My Solana finding was wrong — retracted. I read it off a local wire-solana checkout ( Both nits are closed in One outstanding item: a withheld window permanently stalls the outposts on a multi-group scheduleThe Solana program already names this failure as a rollout hazard (mod.rs:491-498): "
Envelope N is delivered by G_N, and the outpost learns "admit G_N" only from envelope N-1. Withhold once at N and the outpost still admits G_N when envelope N+1 arrives from G_{N+1} — disjoint by construction → refused. Trigger: ACTIVE batch ops below Not covered: contract tests can't observe outpost admission; Refusing to publish a short group is right — master's alternative shrinks the quorum denominator. But the trade is fail-open → fail-closed-and-unrecoverable, and the description ("withholding … until the roster is repaired", "Current-group vacancies require roster recovery") promises a recovery the mechanism can't deliver. Suggested direction: don't let duty advance past the group the outposts were last told about — commit the slide only when the previous window was published. The first withheld epoch is safe (that duty was already announced); freezing from the second keeps a full-size group serving, denominator intact, channel live until the roster is repaired. Failing that, say plainly in the deployment notes that a below-minimum roster on a multi-group window is terminal for the outposts. Otherwise clean
|
…operator-rosters Change-Id: I2d74cc7bccb284c3330e7845cde273ec6a858644 # Conflicts: # contracts/sysio.epoch/sysio.epoch.wasm
Change-Id: I19e4974e0f48b42dec305712c3d6eb3332d8c552
…operator-rosters Change-Id: I545e7f1e2b830bdfeaed8d9b00f84d854c77bd53
…operator-rosters Change-Id: Ie8265a67b31f01661ff8445996ece22bf6e652f5
|
Addressed in The held schedule is now an explicit publication checkpoint rather than a terminal state. While a window is incomplete, The companion Solana change permits an inactive identity only in a group strictly before Regression coverage for withheld-window recovery, announced-successor protection, single-group repair, historical placeholders, and pruning passes in both complete 735-case SYSIO suites. Companion PRs: Solana #522, libraries #82, tools #97, and Ethereum #198. |
Change-Id: Icf26c469260273eae00807ee342de882412fa8af
Change-Id: Ib910bb73aa5fcb157dfababb4f257d13cab13b3e
Change-Id: I64773f671fdecc91a8b9ef7597547e8a4490ff8d
…operator-rosters Change-Id: I05581c1d8f42690f41b49b364420c4ab81ce65c5
Re-review — scope and simplificationThe Split out of this PR
(The six extra rebuilt Simplifications
Two things I looked at and would not cut: the One gap: the new tests cannot see the production recovery path
What advances the epoch out of a hold whose duty group can no longer deliver — and can a test assert it without forcing Cross-repo noteRetaining inactive placeholders in candidate group 0 is what forces Solana #522 to permit an inactive identity in groups before |
Change-Id: I7d6033d5e3eb607065429fe2b567c97634771652
|
Addressed in SYSIO follow-up Scope and simplification
Production recovery pathThe recovery regressions now use the production sequence after genesis: For the complete-duty-loss case, the direct answer is: nothing advances it through normal consensus. Historical seats and SolanaCandidate group 0 preserves the previous serving group’s exact positions, including inactive historical placeholders, so the quorum denominator and the duty that authorized the envelope do not change underneath late deliveries. Solana #522 permits an inactive identity only in a group strictly before The Tools flow also now makes both replacements necessary for quorum by stopping an original peer in each replacement group before checking their accepted signatures. This avoids accepting a late third signature as proof. The final coordinated validation passed all 15 local flows and monitors and all 15 remote flows, with 2,734 successful steps and all 228 recovery steps. |
Re-review — head
|
| Flow | count | shape | seated | spares |
|---|---|---|---|---|
| 10 flows on CLI defaults, plus node-owner-nft and yield-distribution | 3 | 1 x 3 | 3 | 0 |
| flow-batch-operator-slashing | 3 | 3 x 1 | 3 | 0 |
| flow-batch-operator-termination | 9 | 3 x 3 | 9 | 0 (+1 doomed, +2 recovery mid-run) |
| flow-emissions-soak | 21 | 7 x 3 | 21 | 0 |
With seated == roster the schedule has no slack at all: the fill pool each epoch is exactly the group that just rotated out, and it exactly fills the new tail. Any removal leaves the tail one short every epoch after, so the window holds until the roster is whole again — and "removal" includes the two events this PR is about, a non-canonical-delivery slash and a miss-ladder termination.
The cluster path assumes the opposite. create-wire-platform-cluster.yaml:417-458 caps the roster at 26 and names the canonical explicit topology as {batchOperatorCount: 26, operatorsPerEpoch: 7, batchOpGroups: 3} → 21 active, i.e. 5 standbys; the harness schema only rejects total > count, so slack is legal whenever the shape is authored. At 26/21 a slash is absorbed by a standby, the candidate completes, and nothing is ever withheld — no hold, no pinned duty, and finding 1 never fires.
So the coverage runs backwards: the flows exercise the rare path (hold, then recovery) while the common one is only passed through, never asserted. flow-batch-operator-termination does traverse an absorbed removal — its doomed operator is an extra 10th against 9 seats, so terminating it returns the roster to exactly 9 and the window still completes — but nothing asserts that, and with the roster back at the seat count the repair pool is always the group that just rotated out. A genuine standby seated ahead of a recycled member is never exercised, and that is the only path a 26/21 cluster takes. Concrete request below.
Also worth confirming what the deployed clusters actually pass: the real values live in cluster-build-options.json behind CLUSTER_CONFIG_URL, not in the repo. If dev or prod runs 21/21 rather than 26/21, the first slash the CertiK path ever produces holds that network, and finding 1 is on the likely path rather than a fallback.
3. Requested coverage: a removal absorbed by a standby
Suggested shape — extend flow-batch-operator-termination, don't add a flow:
-
Give it a standing spare.
BatchOperatorCount: 9 → 11, keepingOperatorsPerEpoch: 3/BatchOperatorGroups: 3. That is 9 seats with 2 continuous standbys — the 26/21 shape in miniature. The explicit-shape path already permits it (total <= count; the{3,9,15,21}lattice only binds the fully-derived path), andschbatchgpsseats 9 and leaves 2 in the pool. -
Assert the absorbed removal at the existing
Terminate/RemitBondsboundary, before anything starves the roster:- a new window is published on the advance following the termination —
epochstate.next_batch_op_groupsis non-empty, and both outposts seat it (theiractive_group_indexand group membership move forward in lock-step); current_batch_op_groupadvances on the next activation, i.e. duty rotates and never freezes;- the terminated operator appears in no published group, and one of the standbys occupies the vacated seat;
- both outpost epoch cursors keep advancing — no hold, no stale window.
- a new window is published on the advance following the termination —
-
Knock-on:
StarveScheduleWindowis tuned to "slashing one seated operator makes the next tail one seat short", which holds only at 9 ACTIVE / 9 seats. With 11 it needs to removecount - seats + 1(= 3). Deriving that from the constants rather than hardcoding keeps the two halves of the flow independent.
Ordered this way, one run covers the degradation ladder in the order a real network experiences it: healthy with spares → removal absorbed silently → spares exhausted → hold → recovery. The PR97 phases are unchanged apart from the starve count.
Why not a new flow: a scenario fixes its topology once at bootstrap, so covering both "with spares" and "below minimum" needs either two flows or one flow that walks from the first state into the second. A new flow means another full cluster bootstrap in the gate plus duplicated provisioning, ad-hoc daemon, and checkpoint machinery that #97 already built. If you would rather not perturb a freshly tuned scenario, the alternative is a small dedicated flow at {count: 11, operatorsPerEpoch: 3, batchOpGroups: 3} doing only step 2 — same assertions, more wall-clock.
To be clear this is a coverage request, not a defect: the absorbed path is the simple case and advance_repairs_future_group_before_it_becomes_current covers the repair mechanics at unit level. What is missing is an end-to-end assertion that a production-shaped cluster rides through an operator loss with no visible schedule disruption on either outpost.
Minor
- The PR says total held-signer loss has no recovery.
advanceacceptshas_auth(MSGCH_ACCOUNT) || has_auth(get_self()), sosysio.epoch's own authority can force one advance and letfinishadvpublish a repaired window. Worth confirming the outposts accept that envelope given their frozen mirror — if they do, the runbook is a governance action rather than "no recovery". contracts/test_contracts/sendinline/sendinline.wasmis the one artifact not explained by theepoch_statelayout change: identical total and per-section sizes, only the import order swapped (send_inline<->sysio_assert_code), which renumbers indices. It's a unittest fixture that can't includesysio.epoch.hpp— link-order churn, worth agit checkout.
…operator-rosters Change-Id: I73a7e9d7387a93951cc529870d1bcc02e37df2f1
Change-Id: I887a051908fae75b95e4e97394cbaf1d81d67005
Change-Id: I4f087057908d9eb7c5e3624540e7f0066aaf13fc
|
Thanks for the detailed re-review. I addressed the four points in follow-up commits across SYSIO, Tools, and SDK.
Exact-head native checks passed, and all 15 local flow runners, heartbeat monitors, and reports completed successfully. The termination-recovery report passed 248 steps, including the absorbed-removal and later hold/recovery assertions. I do not claim a remote full-platform result here. |
Change-Id: Ifd9e44dc1087e9e6f68afe2fb87c8baf46d1c9fb
Change-Id: I52892664d52b635c86d0f8fb40b0d8cab7021dda
Change-Id: I406eeeadd32a75db1c60458a26d952a40a2b4fed
Change-Id: I99509d0383e3f9eefed8b297f1f59704effdf65f
Change-Id: I87ef44d4d9eea670e995afb2fd5ba5ae66be5ede
Change-Id: I37064cb952eb7ec9af71f5c603a54245c1f08bfc
Re-review — head
|
…operator-rosters Change-Id: Ie693257cc60afd26b53e52aea2041de956eb2b7c
Change-Id: I367666bb68d649234e30caede0ca3d1f79e6ea86
Change-Id: I1f198c1d4db6961958b2d2b787bb95d0df71ece6
|
Addressed the follow-up in the current candidate:
Focused validation passed: 24 SYSIO JIT tests (1,481 assertions), 107 Ethereum outpost tests plus seven deployment tests, 13 SDK client tests, and 32 bootstrap tests. The updated termination flow builds and passes changed-file lint; all 48 tracked SYSIO artifacts match the native build. The new coordinated E2E run is validating the full standard suite against the four pinned companion heads, including the extended termination flow. All four PRs remain draft pending remote validation. Automatic recovery and duty-frequency policy remain in WIRE-392. |
heifner
left a comment
There was a problem hiding this comment.
Re-review — head dce1c13b8f
Everything from my last round is in. Approving.
- Both prints now say
sysio.epoch::finishadv:, and the moved comments namefinishadv. - Tools #97 runs the termination flow a full rotation past
RemitBonds. E2E36044729108passed 15/15 on the exact heads (sysiodce1c13b8f, ETH7ee482f5, SDKc3c21fc0, Tools082d21a1, Solanacbf72188), andpost-remit-rotationpassed after waiting 482s for both outposts to advance. - The earlier red run (
36024314509) failed only on the flow's unresolvedctx.config.operatorsPerEpoch(null on the derived path). Its epoch-7 envelope already showed the terminated operator reported TERMINATED and its seat filled by a standing operator.082d21a1fixed the assertion. - CI is green on the head. Master since the merge base is query-engine work only.
Change-Id: If2e0112676a084c956e862e0d17ea3897a620e8a
Summary
finishadvto self-originated inline calls for the current epoch, preserving accrual/payout ordering and refund action depth.Why
advanceread operator state before its queued inline mutations ran. An epoch could terminate an operator yet still announce it as ACTIVE or select it for the new tail. The self-inline continuation reads the resulting registry state before scheduling and publication, within the same atomic transaction.This fixes stale publication and selection; an already unusable authorized majority can still stop delivery. WIRE-392 is required before deployment to decide recovery/resumption and duty-frequency policy. Automatic recovery remains undecided.
Validation
sysio_epoch_tests,sysio_epoch_flushwtdw_testsandsysio_msgch_chain_tests: 45 JIT cases / 1,837 assertions passed, covering mutation ordering, fresh rosters, slashing, termination, withdrawal, rotation and direct-call rejection.67fe61c2, Ethereum3f23632a, Libraries99c5c14aand Toolsd50fc784, with unchanged Solanacbf72188and CDTa00bd718. Tools installed SDK1.0.93directly from the candidate Libraries checkout. Post-remit rotation on both outposts and eligible collateral withdrawal passed; optionalflow-swap-epoch-stressis excluded.Builtins.ts:89, TS2722) is waived as a prerequisite for this E2E run only; it remains unresolved.Companion PRs