spec-7.6 6.3b: DRM affinity collection substrate + admission hooks#38
Open
sqlrush wants to merge 4 commits into
Open
spec-7.6 6.3b: DRM affinity collection substrate + admission hooks#38sqlrush wants to merge 4 commits into
sqlrush wants to merge 4 commits into
Conversation
added 4 commits
July 11, 2026 15:20
…ssion hooks
Spec: spec-7.6-drm-hot-resource-detection-remaster.md (wave 6.3b)
- D1 cluster_drm_affinity.{c,h}: per-shard x per-node access matrix (full
4096-slot, ~2.3MiB) + atomic dirty bitmap + per-backend sampling ring +
candidate collect + LMON sample-rate reconcile. Lock-free, pure statistics
(no GRD/GES linkage; nothing mutates master/lock state).
- D2 admission hooks (current-master authoritative): remote path in
cluster_ges.c (dedup MISS boundary, env->source_node_id), local-master fast
path in cluster_lock_acquire.c S3 (self). cluster_grd_lookup_master and its
_gen variant left byte-unchanged.
- cluster_lmon.c tick drains the remote-sample ring.
- 3 GUCs: cluster.drm_enabled (off) / drm_affinity_sample_rate (16) /
drm_min_access_count (1000).
- New shmem region "pgrac cluster drm affinity"; cluster.shmem_max_regions
80 -> 96 (registry was full at 80).
- pg_cluster_state category drm_affinity (counters + sample_epoch).
- Tests: cluster_unit test_cluster_drm_affinity (8) + 2-node cluster_tap
t/370 (current master records both self and peer access). Baselines synced:
t/017 (57 categories), t/020 (registry regions + max_regions 96),
test_cluster_shmem (boot value 96); link stubs for debug/ges/lmon/lock_acquire.
Default off: byte-identical to Stage 5 on the lock admission path.
…affinity Spec: spec-7.6-drm-hot-resource-detection-remaster.md (wave 6.3b) fast-gate caught a missed sibling baseline: t/030_acceptance.pl O2 asserts the full pg_cluster_state category set (56 -> 57) independently of t/017_debug.pl. Insert drm_affinity (sorts between dl and gcs) + bump the count text, matching the t/017 update.
…dows + anti-thrash
Wire the DRM decision half on top of the 6.3b affinity collection substrate.
The LMON aux process periodically scans candidate shards, rolls each shard's
tumbling observation window, and runs a pure predicate that proposes migrating
a hot shard's master to its dominant accessing node. This wave PROPOSES only
(report-only observability); the live remaster executor is a later wave.
- cluster_drm_decision.{c,h}: pure predicate cluster_drm_evaluate_shard with a
named skip reason per gate (below-min-access / ratio-low / not-sustained /
already-master / target-not-member / pinned / cooldown / rate-limit /
no-net-benefit) and a binary-cost net-benefit model + adaptive cooldown.
- cluster_drm_affinity: tumbling-window primitives (open/due/judge/reset)
counting consecutive hot windows with a remaster-identity + overdue streak
break, plus a per-reason scan observability surface (INV-DRM9, pure on slot).
- cluster_drm_scan.{c,h}: the LMON scan driver bridging collection + decision +
GRD master/generation + cluster epoch + live membership; self-gated to
cluster.drm_scan_interval_ms and a no-op when disabled.
- 8 new cluster.drm_* decision/anti-thrash GUCs (conservative defaults; the
whole feature stays off by default via cluster.drm_enabled).
- dump category drm_affinity extended with scans_run / scan_candidates /
scan_proposed + a per-reason verdict tally (append-only keys).
Tests: cluster_unit test_cluster_drm_decision (14) + test_cluster_drm_affinity
window/scan additions (15); cluster_tap t/371 2-node value leg proves sustained
skew yields real migration proposals, with a drm_manual_only suppression control.
Standalone link stubs added for cluster_debug / cluster_lmon.
Spec: spec-7.6-drm-hot-resource-detection-remaster.md
…ger) t/371-t/385 are reserved for the 7.5 lane in the Stage 7 ID ledger; the 6.3c DRM decision value leg must not squat on that block. Renames the 2-node test and its ClusterPair name (drm371 -> drm388). Nightly-only test; no fast-gate impact. Spec: spec-7.6-drm-hot-resource-detection-remaster.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Spec: spec-7.6-drm-hot-resource-detection-remaster.md (wave 6.3b — collection substrate).
Adds the DRM (Dynamic Resource Mastering) collection layer: sample first-logical lock requests on the shard's current master to build a per-shard × per-node access matrix. Default off — byte-identical to the prior behaviour on the lock admission path.
What's included
cluster_drm_affinity.{c,h}: full 4096-slot × per-node access matrix (~2.3 MiB) + atomic dirty-shard bitmap + per-backend sampling ring + candidate collect + LMON sample-rate reconcile. Lock-free, pure statistics (no GRD/GES linkage; nothing mutates master/lock state).cluster_ges.c(dedup MISS boundary,env->source_node_id), local-master fast path incluster_lock_acquire.cS3 (self).cluster_grd_lookup_masterand its_genvariant are byte-unchanged.cluster_lmon.ctick drains the remote-sample ring.cluster.drm_enabled(off) /drm_affinity_sample_rate(16) /drm_min_access_count(1000).pgrac cluster drm affinity;cluster.shmem_max_regions80 → 96 (registry was full at 80).pg_cluster_statecategorydrm_affinity(counters + sample_epoch).Test plan (local, macOS)
test_cluster_drm_affinity8/8,test_cluster_shmem12/12).t/017_debug.pl(dump category set) ✓t/020_shmem_registry.pl(region roster + max_regions) ✓t/370_drm_affinity_collection_2node.pl(new, 2-node): the current master records both its own (local) and its peer's (remote) access — measured node0 local 263 / remote 247, node1 local 255 / remote 239 ✓CI here (fast-gate + Linux PG219) is the remaining gate — not runnable on the local macOS harness.