fix(cluster): async qvotec marker submission — LMON liveness (spec-2.29a, BUG-C1 false-DEAD)#25
Open
sqlrush wants to merge 9 commits into
Open
fix(cluster): async qvotec marker submission — LMON liveness (spec-2.29a, BUG-C1 false-DEAD)#25sqlrush wants to merge 9 commits into
sqlrush wants to merge 9 commits into
Conversation
added 9 commits
July 8, 2026 15:53
…I3 recheck, P1-1/P1-2 hard assertions, t/363 rename Review r1 dispositions (0 P0 / 3 P1 / 2 P2, all fixed): - P1: join Phase-1 PREPARE TIMEOUT/non-ACK now drains best-effort and still publishes the staged JOIN_PENDING (the pre-async code ignored PREPARE results; aborting reverted the joiner to DEAD and the next tick re-detected it and re-bumped the epoch once per deadline period — the exact bump storm the staged record forbids). The revert helper is removed; t/363 gains a bounded epoch-advance assertion across the 95s hold window. - P1: clean-leave staged-ACK handoff re-runs the CL-I3 dead_gen-aware version_coherent check before apply (the COMMITTING marker wait now spans ticks; the guarded CAS alone misses a dead_gen-only move). - P1: unit hard assertions — fail-stop fence stage bumps the epoch exactly once while the marker is PENDING and publishes only on ACK; node-remove re-entry while PENDING reports no false contest and no re-bump. - P2: node-remove FENCE_ARMING skips REMOVING-marker/stripe-retire pre-work while the staged fence marker is in flight (new cluster_reconfig_node_removed_fence_stage_pending predicate). - P2: PG-style banners (Author/IDENTIFICATION/NOTES) on the new files. - TAP renamed t/358 -> t/363 (358-362 reserved by parallel lanes); nightly shard split accordingly. Unit fixture gains MyBackendType stub (B_INVALID) and controllable async-marker stubs. Local gates (cassert build): cluster_unit 158 binaries, t/363 all legs, smoke+touched-baseline TAP 13 files/327 tests, cluster_regress 13, PG 219/219, clang-format 0 violations, scn-cmp + no-clog-overlay clean. cppcheck reports one finding in test_cluster_pi_shadow.c:210 — byte-identical to the base commit, local cppcheck 2.20 version drift, not introduced by this branch. Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
…y baseline + async pre-bump WAIT_EPOCH wedge The ship-gate nightly on the marker-async head surfaced two lane-introduced regressions that the fast-gate matrix does not cover: 1. Dump-category baseline miss: reconfig telemetry added a pg_cluster_state category (55 -> 56), but three cross-node TAPs that assert the total (t/204/205/206) were not updated — only the locally-run L405 subset was. Bump them to 56 with the spec-2.29a note. 2. Async pre-bump WAIT_EPOCH wedge (behavior regression): the async fence / join-Phase-1 / node-remove marker staging bumps the membership epoch at stage-entry but only publishes the reconfig event once the voting-disk marker ACKs — now several LMON ticks later instead of the pre-async single-tick spin. In that window the coordinator's OWN GRD recovery IDLE tick re-captured the already-bumped epoch as its WAIT_EPOCH baseline, so the P0 accept that followed the publish read old == cur and froze the affected shards forever — the spec-4.6a section-0 shape, here triggered by the coordinator on itself with no IC piggyback (reproduced deterministically as t/293 fail-stop remaster + t/326 node-leave shard reopen). Fix: while any pre-bump stage is live, the GRD IDLE tick holds its last stable pre-reconfig baseline instead of re-capturing (cluster_reconfig_has_pending_prebump_stage guard). This restores the pre-async bump->publish atomicity as seen by GRD without touching the epoch bump timing or any user-visible reconfig contract. Adds the cluster_grd_recovery_event_old_epoch accessor + a baseline-hold unit test. Local gates (cassert): unit 158 binaries (incl. baseline-hold test), t/293 + t/325 + t/326 family all green (were red), t/204/205/206 + observability TAPs, PG regress 219/219, clang-format/headers/scn-cmp clean. Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
…sion ②b, others-dead) The spec-2.29a marker-async change turned the clean-leave COMMITTING marker into an async, multi-tick wait. In a slow environment (nightly Linux) the leaving node finishes its drain and stops heart-beating inside that wait window, so CSSD marks it SUSPECTED->DEAD and bumps the global dead_generation — and the coherence gate, which compared that SCALAR dead_generation, wrongly read it as a third-party death intruding mid-drain and escalated an otherwise healthy clean leave (t/310 dormant-member legs, t/331 clean_leave x idle). apply_clean_leave_as_coordinator bumps + publishes atomically, so this is NOT the baseline-hold wedge (②a); it is the coherence predicate counting the leaving node's OWN expected DEAD. Fix: compare an others-dead bitmap (the dead set EXCLUDING the leaving node) instead of the scalar dead_generation, at all four coherence sites (CL_COHERENT macro, drive_drain commit-point, staged-ACK re-check, non-staged pre-check). cl_state snapshots the others-dead set at bind; the pure predicate does a bitmap compare (fail-closed on a missing view) and is unit-tested against the reflexive-case matrix. Every third-party incoherence still escalates (epoch move OR a non-leaving node entering the others-dead set); only the leaving node's own expected transition is tolerated. 8.A / CL-I3 protection is unchanged — argued in spec-2.29a §②b with a case matrix. Local gates (cassert): policy unit reflexive-case matrix, cluster_unit 158 binaries, PG regress 219/219, clang-format/headers/scn-cmp clean. NOTE: t/310/331 are timing-sensitive — the false escalation only reproduces in a slow environment, so the fix is verified by nightly, not locally (a local run passes the assertions). A separate pre-existing cassert teardown SIGABRT flake in t/310 (isolated: reproduces without this change) is unrelated. Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
…e-bump shmem bit Two review-r2 items on top of the ②b others-dead coherence fix: P2-1 — the leaving node's barrier-tick own-commit latch (cl_leaving_barrier_tick) inferred 'my leave committed' from epoch>baseline AND the others-dead bitmap unchanged. The bitmap is not monotone: a third-party node that false-fail- stopped (bumping the epoch) then recovered leaves CSSD hysteresis DEAD->ALIVE, so the bitmap rebounds to its bound value while the scalar dead_generation only advances. If the leaver's first epoch>baseline observation lands after that rebound it would mis-latch a leave the survivor coordinator actually refused, suppress the barrier-deadline escalation, and hang forever in BARRIER_WAIT. Restore the scalar conjunct (epoch>baseline AND others_dead==bound AND dead_gen==baseline) via the pure, unit-tested cluster_clean_leave_own_commit_ latched predicate. This does not reintroduce the ②b false positive: the leaver's OWN alive->DEAD never bumps ITS OWN dead_generation (a node does not observe itself dead), so on the leaver side the scalar stays at baseline through its drain. Survivor-side coherence keeps the bitmap-only check. Adds the rebound negative-leg unit (others_dead unchanged + dead_gen advanced -> escalate). t/274 — a backend-context coordinator (the fail-stop inject test) advances the epoch then SYNCHRONOUSLY waits for the fence marker before publishing (MyBackendType != B_LMON path). During that seconds-long wait the epoch is bumped but unpublished, and a concurrent LMON GRD IDLE tick re-captures the post-bump epoch as its WAIT_EPOCH baseline -> old==cur -> the thread replay slot never reaches DONE -> HG#1b/HG#5 done-poll timeout. The ②a baseline-hold guard only reads LMON-process-local staged flags, invisible to a backend's pre-bump window. Publish it in shmem: ReconfigShmem.prebump_sync_active set before the bump, cleared at every return; the guard reads the shmem bit OR the local stage. 8.A (durable-marker-before-publish) unchanged. Reproduced the wedge locally at 87f7930 (HG#1b slot-DONE poll times out forever), then verified the fix: t/274 goes 18/18 green (exit 2 -> 0). This also fixes the long-standing main flake (the wedge is deterministic once LMON is freed from the sync-marker park; it was merely probabilistic before) — the t/274 known-flake / L66 registration should be revised. Local gates (cassert): cluster_unit 158 binaries (incl. own-commit-latched + version-coherent matrices), t/274 18/18, PG regress 219/219, clang-format/ headers/scn-cmp clean. Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
The leaving node's barrier-tick own-commit latch must be backed by
direct evidence (the durable COMMITTED marker confirmation for THIS
leave attempt) and must be immune to third-party transient false-DEAD
flap noise: the scalar dead_generation is monotone, so a flap during
the leave window advances it forever and the r2 P2-1 three-conjunct
inference then refuses a healthy committed leave until the barrier
deadline escalates it (nightly t/331 C1/C4 false-escalation, run
28948167577). The reworked U3b matrix pins:
(a) evidence, no noise -> latch
(d) evidence + flap noise -> still latch (RED on current code)
(c) no evidence, any noise state -> never latch (deadline escalation
stays armed; the r2 P2-1 refused-leave mis-latch wedge stays shut)
Known-red TDD commit: leg (d) fails on the current three-conjunct
predicate; the follow-up commit flips the latch to marker evidence.
Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
…not inference Nightly t/331 C1 (clean_leave x idle) / C4 (leave_remove x idle) regressed at the r2 P2-1 head (run 28948167577; parent green): the leaver's barrier-tick own-commit latch inferred its commit from epoch>baseline + others-dead bitmap unchanged + scalar dead_generation unchanged. The scalar is monotone, so a third-party transient SUSPECTED->DEAD->ALIVE flap on the leaver's local CSSD view during the leave window advances it forever; the latch then refuses a leave the coordinator ACTUALLY committed and the immediate-escalate arm (or the barrier deadline) aborts it: phase ends ABORTED_ESCALATE -> IDLE, never 'committed', breaking the C1/C4 drains+commits assertions. Reverting to bitmap-only would re-open the r2 P2-1 wedge (rebound mis-latch of a REFUSED leave suppresses escalation forever), so neither inference is usable. Replace inference with direct evidence: latch <=> the durable COMMITTED marker for THIS leave attempt is confirmed. The evidence already reaches the leaver with zero extra IO: the coordinator sends the nonce-bound LEAVE_COMMITTED only after its qvotec ACKs the COMMITTED marker majority-durable (there is no runtime voting-disk read path on the leaver, and none is needed - the LMON tick budget this spec protects is untouched). cl_committed_handler now routes through a pure identity gate (self-addressed + currently leaving + per-attempt nonce + committed epoch past the bound baseline; fail-closed on any mismatch, so a stale confirmation - and through it a stale COMMITTED marker from a previous leave of the same node - can never false-latch) and records the attested committed epoch E before publishing the evidence flag; the barrier tick consumes E instead of re-reading its possibly-stale local epoch view. The latch and the P1-V0.7 exit gate collapse into one step (the evidence IS the durable-truth-source confirmation). Escalation semantics are unchanged: no evidence by the barrier deadline -> the existing cl_escalate path, which is exactly what bounds a refused leave (it never gets a COMMITTED marker). Third-party flaps no longer matter: the coherence observations are now contract inputs the predicate must ignore (pinned by the U3b unit matrix, red-first in the parent commit) and feed only a flap-noise LOG at the latch. Survivor-side coherence sites (drive_drain / staged-ACK / pre-check) are untouched. Unit: U3b matrix flipped green; new U3c pins the evidence identity gate (match / stale-nonce / misrouted / not-leaving / epoch-not-advanced). Local gates: t/331 x4 all green (6/6), t/310 24/24, t/363 18/18, t/274 18/18 (prebump shmem-bit stays green), cluster_unit 158 binaries, cluster_regress 13/13, PG core 219/219, clang-format-18 + comment-headers clean. Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Reconfig-coordinator voting-disk marker submissions (join PREPARE/COMMITTED, fail-stop fence, clean-leave, node-remove) previously spin-waited inside the LMON main-loop tick (2ms pg_usleep poll, deadlines up to 17s, serially chained). Under cold-formation voting-disk contention the parks chained into 150-205s LMON relay blackouts; peers' transport-liveness (3s) severed the mesh and CSSD deadband (60s) declared live nodes DEAD — the false-DEAD storm feeding the reconfig wedge.
This makes every marker submission an async cross-tick FSM (
cluster_marker_async.h) while keeping the fail-closed ordering byte-identical: nothing publishes until the marker ACKs from a voting-disk majority. Pre-bump paths (fail-stop fence / node-remove / join Phase-1) stage the pending publish record and bump the epoch exactly once. QVOTEC wakes LMON on completion via a lifecycle-managed reverse latch. LMON gains per-iteration timing observability (cluster.lmon_slow_iteration_warn_ms, dump keys) and acluster-qvotec-marker-service-holdinjection point.Tests
Gates (local, cassert build)
cluster_unit 158 binaries · t/363 all legs ×2 · smoke+touched-baseline TAP 13 files/327 tests · cluster_regress 13/13 · PG regress 219/219 · clang-format 0 violations · scn-cmp/no-clog-overlay clean. cppcheck: 1 finding in a byte-identical-to-base file (local 2.20 version drift, not from this branch).
Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md