fix(cluster): GRD recovery liveness — same-episode remaster retry + proof-carrying WAIT_EPOCH + dead-node PCM cleanup (spec-4.6a, BUG-B/C2)#26
Open
sqlrush wants to merge 6 commits into
Conversation
added 2 commits
July 8, 2026 18:24
… WAIT_CLUSTER watchdog, event-scoped WAIT_EPOCH escape, dead-node PCM cleanup A fail-stop reconfig could wedge a settled cluster forever on three stacked liveness holes: a BLOCKED HW remaster worker was never relaunched within the same episode (BGW_NEVER_RESTART + launched-latch), WAIT_CLUSTER had no watchdog while its cluster/HW gates were held, and WAIT_EPOCH's strict progress gate wedged permanently when an IC-piggybacked epoch bump landed before the local reconfig event (baseline re-captured as the post-bump value, cur == old forever). - HW remaster: per-dead-node result/attempts/next_attempt shmem state; the FSM relaunches BLOCKED workers with exponential backoff (cap 60s) up to cluster.hw_remaster_retry_max_attempts (SIGHUP-raisable); abnormal worker exit marks BLOCKED via before_shmem_exit; the three silent BLOCKED returns now LOG their cause. wal_threads_dir-unset is detected up front as BLOCKED_STRUCTURAL: never retried, WARNING-once with the config hint, and (level 2) a multi-node shared_catalog=on boot without it now fails fast at startup. - WAIT_CLUSTER: observational watchdog on the rebuild-timeout cadence — WARNING with the missing-survivor list, gate states and per-dead-node retry state, plus a counter; it never unfreezes anything. - WAIT_EPOCH: three proof-carrying layers — abort-to-idle on a new event id, durable observed-epoch adoption, and an event-scoped coordinator witness (REDECLARE_DONE carries the event id; stale-event DONEs are dropped at accounting; the equal-epoch escape additionally requires the coordinator's DONE to have been accounted after the accept snapshot). No timeout-only advance exists; without proof the shards stay frozen. - Dead-node PCM cleanup: the GRD dead-sweep clears the dead node's x_holder/s_holders/pi_holders/master_holder and pending-X residue (LOG summary + pcm/dead_cleanup_entries counter), and a local-master S state with no local residency can now upgrade S->X through the standard invalidate/ACK path instead of failing closed forever — post-kill DDL recovers instead of timing out on cluster locks. - Serve-gate scope alignment: the merged-materialization check in gcs_block phase_for_tag now uses the same thread-recovery scope policy as the unfreeze gate, removing the permanent-RECOVERING mismatch outside the gate's applicable scope (in-scope behavior unchanged). - Observability: hw retry/exhausted + grd cluster_gate_timeout / wait_epoch_escape + pcm dead_cleanup_entries dump keys; S4-reject and HW fail-closed diagnostics. Tests: unit relaunch-decision table (8 rows) + event-scoped DONE rejection/ witness advance + PCM dead-cleanup forms incl. pending-X; t/293 gains the same-episode self-heal and retry-exhaustion legs (unfreeze-extend positive + watchdog WARNING greps); t/337 gains the level-2 fail-fast negative leg; new t/362 runs a 4-node shared_catalog formation, kill -9, and asserts remaster convergence, post-kill DDL, cleanup counters and the 0-wedge invariant end to end. Local gates (cassert): unit 158 binaries, t/293 + t/337 + t/362 + smoke subset, cluster_regress 13/13, PG regress 219/219. Spec: spec-4.6a-grd-recovery-liveness.md
t/358-361 are reserved by parallel lanes (spec-7.2 / S-xid), t/363 by the S-dead lane; occupancy verified against origin branches. The 4-node formation + kill -9 + convergence run gets its own shard for wall-clock isolation. Spec: spec-4.6a-grd-recovery-liveness.md
sqlrush
force-pushed
the
stage7-p0-reconfig-liveness
branch
from
e8fc3b9 to
e7ca433
Compare
added 4 commits
July 8, 2026 18:25
…adow unit Same inline suppression + reason as the other cluster_unit main() signatures (test_cluster_grd.c precedent); a newer local cppcheck flags it, the CI baseline version does not.
…hash) The P6 all-done gate and the WAIT_EPOCH coordinator witness keyed the cross-node DONE on event_id, but event_id folds the sender-local cssd_dead_generation, which drifts with each survivor's private flap observation history and never converges across nodes: any flap asymmetry made the survivors compute different ids for the same episode, drop each other's DONEs, and wedge P6 forever — nondeterministically reintroducing the permanent-freeze shape this branch exists to remove. The quorum-accepted dead SET is what actually converges, so the DONE payload now carries a hash over the dead bitmap alone (same kernel as the event_id hash, riding the same request-id field pair — no envelope change), stamped per episode at P0 accept. Accounting, the P6 gate and the witness compare against the stamp; event_id stays a purely local accept-dedup scope. A late DONE from a previous episode still cannot back a new one: a coordinator re-election changes the dead set (hash mismatch) and a same-node re-death rides a higher epoch through the interposed JOIN bump (epoch conjunct). Pre-accept frames now mismatch the previous stamp and are dropped; senders re-announce every tick, so accounting always lands after the accept snapshot — closing the first-event pre-accept window as a side effect. Spec: spec-4.6a-grd-recovery-liveness.md (Amendment v1.2 R2/R4)
Revert the scope short-circuit that returned NORMAL for a dead static master's blocks wherever online thread recovery cannot run (the default configuration and every >2-node deployment): it skipped both the is_materialized cold-block door and the redo-coverage lost-write door, so a committed write only the dead node saw could be silently read stale from shared storage. No other guard sits on that read path — the GRD freeze ends with the episode, the HW gate covers only extend high-water marks, and the page-SCN checks ride the ship path. Out of scope the posture is now an explicit bounded retryable error (53R9L, hint updated to name the way out: restart the failed node, or enable online thread recovery in a supported scope) for every tag whose static master is dead — including never-written blocks, which the cold-block door cannot distinguish from not-yet-replayed ones — and the door reopens the moment the failed node stops being DEAD. In-scope behavior is byte-identical. The now-orphaned scope helper is removed. t/362 asserts the honest two-outcome contract on fresh connections (success or explicit SQLSTATE — 53R9L or the TT-unknown visibility door — never a hang), rides pre-warmed sessions for convergence observability, and pins the D12 cleanup counter to a positive delta; t/293's post-kill extend leg follows the same posture. Spec: spec-4.6a-grd-recovery-liveness.md (Amendment v1.2 R1/R3)
…grade cleanup - Order the worker's terminal-result store after its backoff-deadline store (pg_memory_barrier) so the relaunch decider never pairs a fresh BLOCKED with a stale deadline and skips one backoff wait. - A structural cause discovered only by the WORKER (SIGHUP race) now leaves the retry deadline unset so the FSM's next tick takes the MARK_STRUCTURAL branch and emits the once-per-episode operator WARNING with the configuration hint. - Wrap the local S->X upgrade in PG_TRY and release the temporary S claim when the ACK wait throws (cancel via CHECK_FOR_INTERRUPTS), not only on the false return; document why the completed upgrade hard-resets the local S refcount (the X grant subsumes all local S declarations). Linker-only exception stubs for the units that link cluster_pcm_lock.o. Spec: spec-4.6a-grd-recovery-liveness.md (Amendment v1.2 R5/R6/R7/R9)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A fail-stop reconfig could wedge a settled cluster permanently (observed 56min+, zero self-heal) on three stacked liveness holes:
BGW_NEVER_RESTART+ launched-latch) — silent BLOCKED causes included an unconfiguredcluster.wal_threads_dir.WAIT_CLUSTERhad no watchdog while its cluster/HW gates were held — 8.A fail-closed, but invisible.WAIT_EPOCH's strict progress gate wedged forever when an IC-piggybacked epoch bump landed before the local reconfig event (baseline re-captured post-bump →cur == oldpermanently).Fixes: bounded same-episode HW remaster retry (backoff + SIGHUP-raisable cap, BLOCKED_STRUCTURAL never retried + WARNING-once + multi-node shared_catalog startup fail-fast), observational WAIT_CLUSTER watchdog (never unfreezes), three-layer proof-carrying WAIT_EPOCH escape (event-scoped REDECLARE_DONE witness — stale-event DONEs dropped at accounting; no timeout-only advance exists), dead-node PCM holder/pending-X cleanup on the dead-sweep (post-kill DDL recovers instead of timing out), serve-gate scope alignment, and full observability (retry/exhausted/watchdog/escape/cleanup counters).
Tests
Gates (local, cassert build)
cluster_unit 158 binaries · t/293 + t/337 + t/362 + smoke subset · cluster_regress 13/13 · PG regress 219/219 · clang-format 0 violations · headers/scn-cmp/no-clog-overlay/cppcheck clean.
Spec: spec-4.6a-grd-recovery-liveness.md