spec-7.4 D0+D1: commit-path census + event-driven durable-frontier freshness#39
Merged
Conversation
added 7 commits
July 11, 2026 12:52
Five new commit-decomposition xnode_profile buckets (undo precommit flush / ITL stamp / TT stamp / commit-record WAL flush / quorum read), instrumented at the xact.c commit sequence and gated by the existing cluster.xnode_profile GUC (zero cost while off). Buckets are APPENDED at the enum tail: bucket indexes address a shmem array, so the enum is append-only -- an insertion renumbers later buckets and any stale .o keeps writing old indexes (probe-verified failure mode). New REPORT-ONLY census runner scripts/perf/run_74_commit_census.pl: commit component table (pair, fsync on/off), single-node serialization ceiling ladder with crash detection + wait-event attribution, BOC staleness differencing (idle/busy), hwm publication-channel audit, and a t/328-methodology version-comparison leg. Baselines synced: unit U1 23->28 buckets, U5 51->61 dump keys, t/334 key-count 61, 5.59 runner bucket table. Spec: spec-7.4-commit-scn-propagation-freshness.md (D0)
…ayload v1 receive side
Contiguous durable frontier replaces any atomic-max watermark notion:
every own-origin commit SCN <= the published frontier is durable on
WAL. Pending registry registers at commit-SCN allocation (reservation
loop now runs for every commit advance, extending the INV-ADG8
register-before-observable invariant to the frontier), fills in the
commit-record LSN at insert, and discharges on three paths: sync
commit after END_CRIT_SECTION, walwriter behind XLogBackgroundFlush
(cluster_scn_boc_tick), abort cleanup (lsn-filled entries are real
commits and only detach). Registry overflow freezes the frontier
stickily instead of publishing an unsafe claim.
Receive side: BOC payload_length contract widens to {0, 8} (v1 = 8B
origin_durable_safe_scn, explicit LE). Handler validates length,
decodability, and SCN node bits == env.source_node_id (fail-closed
counted drops), then updates a per-(origin, epoch) seqlock cache;
same-epoch regressions are counted rejects, epoch changes rebuild.
Send-side payload attach + event wakeup + GUC arrive with the next
deliverable slice; 0-byte frames stay valid pulse-only traffic.
Unit: test_cluster_scn_frontier (20 tests) covers the out-of-order
flush P0 heart, walwriter horizon discharge, abort guard, sticky
overflow freeze, monotonic publish, LE codec, and handler validation
matrix.
Spec: spec-7.4-commit-scn-propagation-freshness.md
Spec: spec-2.9-scn-broadcast-service-activation.md
…e + payload attach) Event half of the durable-frontier channel. Producer: any discharge that STRICTLY advances the published frontier signals after releasing the SCN lwlock -- publish-before-signal order is inherent -- via exchange-set of a dirty flag, waking LMON only on the 0->1 transition (natural batching: N commits in one window collapse to one drain). Consumer: the LMON BOC drain exchange-clears dirty BEFORE snapshotting the frontier so a racing publish re-arms the wakeup, then fans out one BOC frame carrying payload v1; either a sweep beat or a commit event justifies a fanout, and the zero-peer short-circuit now runs before either gate so neither is consumed with nobody to send to. New GUC cluster.boc_event_publish (bool, on, SIGHUP): off restores the pre-D1 wire bytes (0-length BOC frames) and stops event wakeups; the walwriter sweep cadence remains the fallback either way. Injection point cluster-boc-event-publish suppresses the producer signal to prove the sweep fallback. Unit: 7 new tests (27 total in test_cluster_scn_frontier) pin the wake-on-advance / dedup single-wake / no-advance-no-wake / off / injection-suppressed producer matrix and the drain payload attach (on = 8B LE frontier, off = 0B even with a stale dirty flag). Spec: spec-7.4-commit-scn-propagation-freshness.md Spec: spec-2.9-scn-broadcast-service-activation.md
…ty rows pg_cluster_debug scn section gains the durable-frontier registry stats (safe scn as full 64-bit hex, pending count, frozen flag, overflow + regression counters), the four BOC payload v1 receive-side counters (accept / bad-length / node-mismatch / regression drops), and one sparse row pair (frontier hex + epoch) per remote origin with a cached claim. These back the TAP legs and keep every new counter SQL-visible. Spec: spec-7.4-commit-scn-propagation-freshness.md
The durable-frontier / BOC payload observability rows added 9 keys to the scn dump section; sparse per-origin rows never appear single-node so the count stays deterministic. Spec: spec-7.4-commit-scn-propagation-freshness.md
…8 + boc-event-publish injection point Adds t/387: node0 commits, node1 observes node0's published contiguous durable frontier through the BOC payload v1 channel. Legs L1 staleness (event path), L2 sweep fallback, L3 commit-unperturbed under a publish fault, L4 batching, L5 off byte-equivalence, L6 out-of-order (frontier must not pass a stalled commit), L7 idle async walwriter discharge, L8 seqlock reader liveness across an LMON kill/crash-restart. The commit-event publish path referenced cluster-boc-event-publish but the point was never in the registry, so suppression was a no-op. Register it and switch the probe from should_skip (one-shot SKIP dispatch, never armed here) to an armed-state peek (cluster_cr_injection_armed): the probe sits on the post-commit-durable path, so an armed ERROR/CRASH dispatch must never fire, and L2 needs sustained suppression while armed, not a single skip. Registry grows 164 -> 165; sweep the count assertions in t/015 (+ name list), t/017/018/020/021/022/023/024/030 accordingly. Unit stub updated for the armed-peek + armed_count gate. Spec: spec-7.4-commit-scn-propagation-freshness.md (D1)
…est_cluster_scn link test_cluster_scn links cluster_scn.o standalone. The D1 boc-event-publish probe switched cluster_scn.c from cluster_injection_should_skip to the armed-state peek cluster_cr_injection_armed, so every binary linking cluster_scn.o now needs that symbol; only test_cluster_scn_frontier had the stub. A local incremental build reused a stale test_cluster_scn.o and masked the missing symbol -- a clean CI build (and a clean local rebuild, all 169 binaries) exposed the link failure. Add the stub and correct cluster_injection_armed_count to int to match cluster_inject.h. Spec: spec-7.4-commit-scn-propagation-freshness.md (D1)
Owner
Author
fast-gate green (PR-level, rule 20.A)
Local surfaces: cluster_unit 169/169 binaries (clean rebuild) · cluster_regress 13/13 · PG core 219/219 (enable) · t/387 80/80 ×2 · affected cluster_tap count-sweep subset green · clang-format 18 + comment-header + scn-cmp/clog/ges gates clean. Fix history: the first PR run (29145244841) red on the cluster_unit link step — Note: nightly (ship-level) is a separate gate for the tag/ship step, not this PR. |
sqlrush
pushed a commit
that referenced
this pull request
Jul 11, 2026
…use-#1 integration lane Main advanced past the PR #40 base (b015369) with spec-7.4 D0/D1/D4 (commit-scn freshness + durable frontier + BOC observability, PRs #39/#41), so the 5.22 integration branch no longer merged cleanly. Bring current main in and reconcile: - injection registry: both lanes added points (5.22: authority-block0-prove + 2 horizon + this branch's cluster-undo-authority-scan; 7.4: cluster-boc-event -publish), so the merged total is 169. Resolved the count assertions in t/015/017/018/020/021/022/023/024/030 to 169 (M5 derived 336 -> 338) and merged cluster-boc-event-publish into the t/015 sorted name list. - src/test/cluster_unit/Makefile SIMPLE_TESTS: union of both lanes' custom-rule test exclusions (5.22 undo tests + 7.4's test_cluster_scn_frontier). Verified on the merged tree: registry band t/015-030 + t/068 pass, 7.4's t/387 durable-frontier passes, both root-cause-#1 fixes (t/359_mxid, t/360) still pass, and cluster_unit is 175/175 clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Spec-7.4 D0 (commit-path census substrate) + D1 (event-driven durable-frontier propagation) for the cluster commit/SCN freshness path.
D0 — commit-path census (report-only observability):
cluster_xnode_profilecommit-component probes around the precommit undo flush, ITL/TT stamp, WAL flush and quorum read, plus a standalone census runner (scripts/perf/run_74_commit_census.pl) that decomposes commit latency and measures propagation staleness. Trend datapoints only, never a gate.D1 — durable-frontier freshness:
durable_safe_scncontiguous durable frontier: a pending-commit registry (register at SCN allocation under the SCN lwlock, LSN back-fill after the commit record, three-way discharge — sync-commit / walwriter async flush horizon / abort), with overflow-freeze and strictly-monotone publish. Replaces a naiveatomic_max, which could falsely claim an out-of-order-flushed SCN durable.cluster.boc_event_publish=off).cluster.boc_event_publish(bool, SIGHUP).cluster-boc-event-publish(armed-state peek) suppresses the commit-side event to exercise the sweep fallback and the commit-unperturbed path.Test plan
test_cluster_scn_frontier(27 cases) — frontier out-of-order truth table, abort discharge, overflow freeze + re-anchor, monotone publish, payload LE roundtrip / bad-length / node-mismatch fail-closed, dirty-exchange (no lost wakeup), per-(origin,epoch) epoch invalidation. Fullcluster_unitsuite green.Spec: spec-7.4-commit-scn-propagation-freshness.md (D0+D1)