diff --git a/changelog.d/11612-regex-scratch-not-gc-pressure.md b/changelog.d/11612-regex-scratch-not-gc-pressure.md new file mode 100644 index 0000000000..275eb59a86 --- /dev/null +++ b/changelog.d/11612-regex-scratch-not-gc-pressure.md @@ -0,0 +1 @@ +perf(regex): regex operation scratch is no longer reported to the collector as external side bytes (#11549). The owned search path's per-call match buffers, compile scratch, KMP tables and replacer argument slots are freed by the operation and bounded by its `MemoryBudget`, but each release fed `GC_EXTERNAL_SIDE_DRAINED_SINCE_FULL` and paced loops over >32-register programs (dotenv's `LINE`) into a full mark-sweep every few hundred calls. The lent per-thread scratch cell now grows its registers on demand up to 1024 (8 KiB per thread), so such programs stop building per-call buffers. Subject-proportional replace storage is still reported. Instructions: dotenv/parse −30.1%, moment/parse_format −13.8%, validator/batch −1.9%; peak RSS rises on moment/parse_format (+29%) because those loops now pace by the 16 MB nursery like other allocating programs — held pending the young-generation pacing half of #11549. diff --git a/changelog.d/11645-gc-nursery-pacing.md b/changelog.d/11645-gc-nursery-pacing.md new file mode 100644 index 0000000000..8d4ed73b71 --- /dev/null +++ b/changelog.d/11645-gc-nursery-pacing.md @@ -0,0 +1 @@ +- **gc: survival-aware nursery pacing (Part of #11549).** The scavenge nursery cap's influx ladder now starts at a quarter of the 16 MB base (4 MB). It climbs toward the unchanged 64 MB top only while survivor influx stays above 4% of the cap. A ladder move takes every step the one-step rule would take on the same reading, so survivor-heavy programs settle where they used to, two minors in. The survivor target, the promoted-cohort floor and the census seed point stay on the 16 MB base. Measured against main `7fa094cb4` with THP off: dotenv −30.5% instructions and −20.3% peak RSS; moment −20.6% instructions; validator −26.0% RSS; date-fns −30.0% RSS; jsonwebtoken −22.0% RSS; uuid −21.0% RSS; gc_ratchet 01 −25.5% RSS. **Owner-accepted trade, 2026-09-30, tracked in #11699** (re-measured on main `034b1ea38`, with #11676): a program that holds a large live structure and exits after 4–16 MB of allocation now runs a minor that main never ran (binary-trees n=3/6/10: +498/+433/+369% instructions, +60/+55/+49% RSS); gc_ratchet 02 is +19.8% instructions, 12_large_live_set +2.5% RSS, moment +13.8% RSS (an unexplained shift that #11699 tracks), qs parse +0.75% instructions, and the allocation loop +0.57%. Also classifies `GC_EXTERNAL_SIDE_*` as byte counters in `scripts/gc_runtime_root_holders.json`, because #11612's commits turned that gate red. diff --git a/crates/perry-runtime/src/gc/tenuring.rs b/crates/perry-runtime/src/gc/tenuring.rs index eb6b5f50af..bffb2f2431 100644 --- a/crates/perry-runtime/src/gc/tenuring.rs +++ b/crates/perry-runtime/src/gc/tenuring.rs @@ -24,8 +24,9 @@ //! S = min(4, 1 + desired / influx) //! ``` //! -//! `desired` is 1/16 of the scavenge nursery cap (1 MB at the default -//! 16 MB cap) — the same effective ratio HotSpot's defaults produce +//! `desired` is 1/16 of the effective scavenge nursery cap, never less than +//! 1/16 of the base cap (1 MB at the default 16 MB base; see +//! [`desired_survivor_bytes`]) — the same effective ratio HotSpot's defaults produce //! (SurvivorRatio=8, TargetSurvivorRatio=50% ⇒ Eden/16). //! //! The influx signal is deliberately *threshold-invariant*: live Eden bytes @@ -191,12 +192,49 @@ pub(super) const OCCUPANCY_MIN_SURVIVALS: u8 = 2; /// before it is raised (by one step). const RAISE_DEBOUNCE_CYCLES: u8 = 2; -/// Ceiling for the influx-driven nursery cap scale: 16 MB × 4 = 64 MB. -/// Bounds the young-gen RSS contribution on live-set-bound workloads while -/// still cutting their collection count 4× (each collection carries a fixed -/// root-scan/remembered-set/eligibility cost that dominates once the -/// adaptive threshold has eliminated the re-copying). -const NURSERY_CAP_SCALE_MAX: u8 = 4; +/// `NURSERY_CAP_SCALE` counts quarters of the base cap: this value is ×1. +const NURSERY_CAP_SCALE_UNIT: u8 = 4; + +/// Floor of the survival-driven nursery ladder: a quarter of the base cap, +/// 16 MB / 4 = 4 MB. Every thread starts here and returns here while its +/// minors find little alive (#11549). +/// +/// # Why the floor is below the base +/// +/// The base cap was the size every low-survival program ran its young +/// generation at, and on those programs a 16 MB Eden is 12 MB of garbage kept +/// resident between minors for nothing: a minor there copies almost nothing, +/// so its cost is its fixed cost, and #11634 cut that to ~100k instructions. +/// At that price collecting 4× as often is cheaper than it was to collect at +/// 16 MB before the cut, and it lowers peak RSS on every such program. +/// +/// Where survivors are a heavy share of each minor the opposite holds — +/// copying survivors, not the fixed cost, dominates, and a small Eden gives +/// them no time to die (qs stringify at a fixed 4 MB: 105 minors instead of +/// 24, +7% instructions). That is what the ladder's existing debounced +/// 4 %/1 % influx rule is for, and it now simply starts two steps lower. +/// [`retune_nursery_cap_scale`] takes all the steps a reading justifies in one +/// debounced move, so such a program leaves the floor after its second minor +/// and settles at the same size it settled at from 16 MB before. +/// +/// Only the effective nursery cap starts low. Everything else that is sized +/// from the base cap — the survivor target ([`desired_survivor_bytes`]), the +/// promoted-cohort floor, the allocation-census seed point, the JSON-leaf +/// routing gate — keeps the 16 MB base, because each of them measures a +/// LIFETIME or a volume in bytes allocated, which a smaller Eden must not +/// shorten. Measured before that decoupling (everything keyed on one 4 MB +/// base): validator's aging cohorts were promoted after 8 MB of allocation +/// instead of dying in the survivor space (+26% peak RSS), and a 5 MB startup +/// tree tripped a promoted-cohort full at 4 MB (+45% instructions). +const NURSERY_CAP_SCALE_MIN: u8 = 1; + +/// Ceiling of the ladder: 16 MB × 4 = 64 MB, unchanged. Bounds the young-gen +/// RSS contribution on live-set-bound workloads while still cutting their +/// collection count (each collection carries a fixed root-scan/remembered-set/ +/// eligibility cost that dominates once the adaptive threshold has eliminated +/// the re-copying). +const NURSERY_CAP_SCALE_MAX: u8 = 16; +const _: () = assert!(NURSERY_CAP_SCALE_MAX == 4 * NURSERY_CAP_SCALE_UNIT); crate::perry_thread_local! { /// Power-on threshold. This is `OCCUPANCY_MIN_SURVIVALS`, not the ceiling: @@ -217,9 +255,12 @@ crate::perry_thread_local! { /// Bytes the previous copying minor put into the to-survivor space — /// the denominator of this cycle's survival rate. static PREV_COPIED_BYTES: Cell = const { Cell::new(0) }; - /// Influx-driven multiplier (1, 2, or 4) applied to the scavenge nursery - /// cap. Power of two; grows/shrinks one step at a time, debounced. - static NURSERY_CAP_SCALE: super::TriggerInput = const { super::TriggerInput::new(1) }; + /// Influx-driven multiplier applied to the scavenge nursery cap, in + /// quarters of it (`NURSERY_CAP_SCALE_UNIT` is ×1): 1, 2, 4, 8 or 16, i.e. + /// ×¼ … ×4. Power of two; starts at the floor and grows/shrinks one step + /// at a time, debounced. + static NURSERY_CAP_SCALE: super::TriggerInput = + const { super::TriggerInput::new(NURSERY_CAP_SCALE_MIN) }; static CAP_GROW_STREAK: Cell = const { Cell::new(0) }; static CAP_SHRINK_STREAK: Cell = const { Cell::new(0) }; /// #7929: mean size of the objects the last copying minor moved. Seeded at @@ -323,11 +364,12 @@ fn tenuring_survivals_override() -> Option { /// survives; on live-set-bound workloads (tree/retain/deeplist shapes) that /// multiplies the per-collection fixed cost by an enormous collection /// count AND promotes objects that a larger Eden would have let die young. -/// The scale grows only while survivor influx stays a heavy fraction of -/// Eden, so the small-live-set workloads #7377 fixed never leave 16 MB. +/// The scale starts at a quarter of the base and grows only while survivor +/// influx stays a heavy fraction of Eden, so the small-live-set workloads +/// #7377 fixed never leave the 4 MB floor ([`NURSERY_CAP_SCALE_MIN`]). /// /// #7592: the influx-driven product is the floor, not the whole answer — it -/// is bounded by `base × NURSERY_CAP_SCALE_MAX` (64 MB), and any *constant* +/// is bounded by `base × 4` (64 MB), and any *constant* /// cap sets collection cadence independently of how much is live while each /// collection's fixed cost is O(old-gen) (#6181: full-region sweep walk, /// whole-heap remembered-set rebuild). Total young-GC work is then @@ -356,8 +398,14 @@ pub(super) fn scavenge_nursery_cap_effective_bytes() -> usize { /// [`nursery_cap_object_scale_permille`]. Named so the composition below reads /// as the two-term policy it is. pub(super) fn influx_driven_nursery_cap_bytes() -> usize { - let constant_band = gc_scavenge_nursery_cap_bytes() - .saturating_mul(NURSERY_CAP_SCALE.with(TriggerInput::get) as usize); + influx_driven_nursery_cap_bytes_at(NURSERY_CAP_SCALE.with(TriggerInput::get)) +} + +/// [`influx_driven_nursery_cap_bytes`] at a given ladder scale, so the retune +/// can price a candidate step before taking it. +fn influx_driven_nursery_cap_bytes_at(scale: u8) -> usize { + let constant_band = gc_scavenge_nursery_cap_bytes().saturating_mul(scale as usize) + / NURSERY_CAP_SCALE_UNIT as usize; // The multiply is done in u64 deliberately. `usize::saturating_mul` on an // ILP32 target (watchOS/visionOS are 32-bit) would saturate a 64 MB band // against a 1000-per-mille factor at `u32::MAX` and the following divide @@ -560,9 +608,25 @@ const TENURED_EDEN_DIVISOR: usize = 2; /// Target steady-state survivor occupancy: 1/16 of the effective nursery /// cap, so the tenuring dials track both the configured base and the -/// influx-driven scale. +/// influx-driven scale — but never less than 1/16 of the BASE cap. +/// +/// The floor is #11549's. Below the base the ladder shrinks Eden to save +/// resident garbage, and the survivor target must not shrink with it: it sets +/// how long a cohort may age before promotion, and a quarter-size target +/// would promote cohorts after a quarter of the allocation they had before. +/// validator/batch at a 4 MB Eden with the target following it promoted +/// 27.6 MB of cohorts that die in the survivor space at 16 MB (+26% peak RSS). +/// The survivor space this keeps is bounded by the target itself, 1 MB. pub(super) fn desired_survivor_bytes() -> usize { - scavenge_nursery_cap_effective_bytes() / 16 + scavenge_nursery_cap_effective_bytes().max(gc_scavenge_nursery_cap_bytes()) / 16 +} + +/// The effective influx-driven cap at the floor of the ladder, before the +/// object denomination: a quarter of the base (4 MB by default). +#[cfg(test)] +pub(super) fn nursery_cap_floor_bytes() -> usize { + gc_scavenge_nursery_cap_bytes() * NURSERY_CAP_SCALE_MIN as usize + / NURSERY_CAP_SCALE_UNIT as usize } pub(super) fn compute_target_survivals(eden_live_bytes: usize, desired_bytes: usize) -> u8 { @@ -704,36 +768,62 @@ pub(super) fn retune_after_scavenge( set_survivals(current, next, eden_live_bytes, "occupancy"); } -/// Grow the nursery cap one ×2 step (to at most ×4) when survivor influx -/// exceeds 4% of the current effective cap for two consecutive cycles — -/// objects are surviving because they aren't getting time to die, so a -/// bigger Eden both cuts the collection count and lets them die young. -/// Shrink one step when influx falls below 1% for two consecutive cycles. -/// The 4%/1% band is wide enough that the scale cannot oscillate on a -/// steady workload (growing halves the observed ratio, 4%/2 = 2% > 1%). +/// Retune the nursery cap scale from one copying minor's survivor influx. +/// +/// Grow when survivor influx exceeds 4% of the current effective cap for two +/// consecutive cycles — objects are surviving because they aren't getting +/// time to die, so a bigger Eden both cuts the collection count and lets them +/// die young. Shrink when influx falls below 1% for two consecutive cycles. +/// +/// A move takes, at once, every ×2 step that the one-step rule would take in a +/// row on the same reading (#11549): growth continues while the influx is +/// still above 4% of the next level's cap, shrinking while it is still below +/// 1% of it. So the scale lands exactly where one-step moves would have +/// settled — the steady-state sizes are unchanged — without paying two minors +/// per step to get there. With the floor two steps below the old base, +/// one-step climbing cost a survivor-heavy program up to eight extra minors, +/// each copying a cohort that a big enough Eden would have let die (measured +/// on the 12_large_live_set ratchet probe: 9 minors and +3% instructions +/// against main's 5). It cannot oscillate: a growth ends above 2% of the new +/// cap (the level below was above 4%), a shrink below 2% (the level above was +/// below 1%), and both are inside the 1%..4% dead band. The debounce is +/// unchanged: one heavy cycle — a program's startup cohort, say — moves +/// nothing. fn retune_nursery_cap_scale(eden_live_bytes: usize) { - let cap = scavenge_nursery_cap_effective_bytes(); let scale = NURSERY_CAP_SCALE.with(TriggerInput::get); + let old_reclaimable = old_gen_reclaimable_pressure_bytes(); + let cap_at = |scale: u8| { + scavenge_nursery_cap_from(influx_driven_nursery_cap_bytes_at(scale), old_reclaimable) + }; + let cap = cap_at(scale); if eden_live_bytes > cap / 25 { CAP_SHRINK_STREAK.with(|s| s.set(0)); if scale < NURSERY_CAP_SCALE_MAX { let streak = CAP_GROW_STREAK.with(|s| s.get()).saturating_add(1); if streak >= RAISE_DEBOUNCE_CYCLES { CAP_GROW_STREAK.with(|s| s.set(0)); - NURSERY_CAP_SCALE.with(|s| s.set(scale * 2)); - diag_cap_scale(scale, scale * 2, eden_live_bytes); + let mut next = scale * 2; + while next < NURSERY_CAP_SCALE_MAX && eden_live_bytes > cap_at(next) / 25 { + next *= 2; + } + NURSERY_CAP_SCALE.with(|s| s.set(next)); + diag_cap_scale(scale, next, eden_live_bytes); } else { CAP_GROW_STREAK.with(|s| s.set(streak)); } } } else if eden_live_bytes < cap / 100 { CAP_GROW_STREAK.with(|s| s.set(0)); - if scale > 1 { + if scale > NURSERY_CAP_SCALE_MIN { let streak = CAP_SHRINK_STREAK.with(|s| s.get()).saturating_add(1); if streak >= RAISE_DEBOUNCE_CYCLES { CAP_SHRINK_STREAK.with(|s| s.set(0)); - NURSERY_CAP_SCALE.with(|s| s.set(scale / 2)); - diag_cap_scale(scale, scale / 2, eden_live_bytes); + let mut next = scale / 2; + while next > NURSERY_CAP_SCALE_MIN && eden_live_bytes < cap_at(next) / 100 { + next /= 2; + } + NURSERY_CAP_SCALE.with(|s| s.set(next)); + diag_cap_scale(scale, next, eden_live_bytes); } else { CAP_SHRINK_STREAK.with(|s| s.set(streak)); } @@ -827,7 +917,10 @@ pub(super) fn seed_promote_lock_from_sweep(eden_live_bytes: usize, eden_dead_byt fn diag_cap_scale(from: u8, to: u8, eden_live_bytes: usize) { if crate::gc::gc_diag_enabled() { eprintln!( - "[gc-tenuring] nursery cap scale {from}x -> {to}x (eden_live_bytes={eden_live_bytes})" + "[gc-tenuring] nursery cap scale {from}/{unit} -> {to}/{unit} of base \ + (band {} B, eden_live_bytes={eden_live_bytes})", + influx_driven_nursery_cap_bytes_at(to), + unit = NURSERY_CAP_SCALE_UNIT ); } } @@ -856,7 +949,7 @@ pub(super) fn reset_for_test() { PROMOTE_LOCK.with(|l| l.set(false)); UNLOCK_STREAK.with(|s| s.set(0)); PREV_COPIED_BYTES.with(|c| c.set(0)); - NURSERY_CAP_SCALE.with(|s| s.set(1)); + NURSERY_CAP_SCALE.with(|s| s.set(NURSERY_CAP_SCALE_MIN)); CAP_GROW_STREAK.with(|s| s.set(0)); CAP_SHRINK_STREAK.with(|s| s.set(0)); MEAN_SURVIVING_OBJECT_BYTES.with(|s| s.set(NURSERY_CAP_REFERENCE_OBJECT_BYTES)); @@ -947,7 +1040,9 @@ mod tests { #[test] fn census_carries_forward_across_a_cycle_that_moved_nothing() { reset_for_test(); - let base = gc_scavenge_nursery_cap_bytes(); + // #11549: a fresh thread paces at the ladder's floor, a quarter of the + // base, and the object denomination scales that. + let base = nursery_cap_floor_bytes(); assert_eq!( influx_driven_nursery_cap_bytes(), base, @@ -1059,8 +1154,12 @@ mod tests { // Heavy influx: instant drop, no debounce. #9851 changed the FLOOR this // lands on (2, not 1 — the occupancy rule may not claim a lifetime), not // the asymmetry this test is named for: 4 -> 2 in one cycle is the same - // "drops immediately" property that 4 -> 1 was. - retune_after_scavenge(desired * 2, 0, 0); + // "drops immediately" property that 4 -> 1 was. Heavy relative to the + // desired size at the TOP of the cap ladder, which the warm-up above + // has already climbed (#11549: a ladder move takes all its steps at + // once), so the drop is to the floor whatever size the cap reached. + let heavy = desired * 16; + retune_after_scavenge(heavy, 0, 0); assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); // One quiet cycle: no rise yet (debounce). @@ -1071,7 +1170,7 @@ mod tests { assert_eq!(tenuring_survivals(), 3); // Heavy again: streak resets and threshold drops straight back. - retune_after_scavenge(desired * 2, 0, 0); + retune_after_scavenge(heavy, 0, 0); assert_eq!(tenuring_survivals(), OCCUPANCY_MIN_SURVIVALS); // Sustained quiet recovers to the ceiling two cycles per step. @@ -1100,7 +1199,8 @@ mod tests { } assert_eq!( scavenge_nursery_cap_effective_bytes(), - gc_scavenge_nursery_cap_bytes() * NURSERY_CAP_SCALE_MAX as usize, + gc_scavenge_nursery_cap_bytes() * NURSERY_CAP_SCALE_MAX as usize + / NURSERY_CAP_SCALE_UNIT as usize, "sustained heavy influx must also walk the cap to its ceiling" ); reset_for_test(); @@ -1324,31 +1424,89 @@ mod tests { #[test] fn cap_scale_grows_on_heavy_influx_and_shrinks_when_quiet() { reset_for_test(); + // #11549: a fresh thread starts at the floor, a quarter of the base. + let floor = nursery_cap_floor_bytes(); let base = gc_scavenge_nursery_cap_bytes(); - assert_eq!(scavenge_nursery_cap_effective_bytes(), base); + assert_eq!(floor, base / 4); + assert_eq!(scavenge_nursery_cap_effective_bytes(), floor); // Influx above 4% of the cap: one debounce cycle, then a ×2 step. - retune_after_scavenge(base / 15, 0, 0); - assert_eq!(scavenge_nursery_cap_effective_bytes(), base); - retune_after_scavenge(base / 15, 0, 0); - assert_eq!(scavenge_nursery_cap_effective_bytes(), base * 2); + retune_after_scavenge(floor / 15, 0, 0); + assert_eq!(scavenge_nursery_cap_effective_bytes(), floor); + retune_after_scavenge(floor / 15, 0, 0); + assert_eq!(scavenge_nursery_cap_effective_bytes(), floor * 2); // Growth halves the observed ratio into the dead band: stable. - retune_after_scavenge(base / 15, 0, 0); - retune_after_scavenge(base / 15, 0, 0); + retune_after_scavenge(floor / 15, 0, 0); + retune_after_scavenge(floor / 15, 0, 0); + assert_eq!(scavenge_nursery_cap_effective_bytes(), floor * 2); + // A heavier influx takes every step the one-step rule would take in a + // row, in ONE debounced move. 7% of the base is above 4% of 8 MB and + // of 16 MB but below 4% of 32 MB, so the move lands at 32 MB — where + // one-step moves would have settled — and not at the ceiling. + retune_after_scavenge(base * 7 / 100, 0, 0); + assert_eq!(scavenge_nursery_cap_effective_bytes(), floor * 2); + retune_after_scavenge(base * 7 / 100, 0, 0); assert_eq!(scavenge_nursery_cap_effective_bytes(), base * 2); - // Heavier influx reaches the ×4 ceiling and stops there. + // Heavier still reaches the ×4-of-base ceiling and stops there. for _ in 0..4 { retune_after_scavenge(base, 0, 0); } assert_eq!(scavenge_nursery_cap_effective_bytes(), base * 4); - // Quiet influx walks back one step at a time. + // A quiet reading walks back down in one debounced move, as far as + // the reading stays under 1% of each level: 0.6% of 64 MB is 1.2% of + // 32 MB, so this move is a single step. + let reading = base * 4 * 6 / 1000; + retune_after_scavenge(reading, 0, 0); + assert_eq!(scavenge_nursery_cap_effective_bytes(), base * 4); + retune_after_scavenge(reading, 0, 0); + assert_eq!(scavenge_nursery_cap_effective_bytes(), base * 2); + // A dead-quiet reading goes all the way to the floor, never below. for _ in 0..2 { retune_after_scavenge(0, 0, 0); } - assert_eq!(scavenge_nursery_cap_effective_bytes(), base * 2); - for _ in 0..2 { + assert_eq!(scavenge_nursery_cap_effective_bytes(), floor); + for _ in 0..4 { retune_after_scavenge(0, 0, 0); } - assert_eq!(scavenge_nursery_cap_effective_bytes(), base); + assert_eq!(scavenge_nursery_cap_effective_bytes(), floor); + reset_for_test(); + } + + /// #11549: a ladder move lands where one-step moves would settle, so a + /// steady influx reaches the same size it always did — and never + /// oscillates. Checked exhaustively over influxes from the floor's dead + /// band to far above the ceiling's. + #[test] + fn multi_step_moves_land_in_the_dead_band_and_hold() { + let base = gc_scavenge_nursery_cap_bytes(); + let floor = nursery_cap_floor_bytes(); + for permille_of_base in (0..=2000).step_by(7) { + reset_for_test(); + let influx = base * permille_of_base / 1000; + for _ in 0..4 { + retune_after_scavenge(influx, 0, 0); + } + let settled = scavenge_nursery_cap_effective_bytes(); + assert!(settled >= floor && settled <= base * 4); + // In the dead band, unless pinned at an end of the ladder. + assert!( + influx <= settled / 25 || settled == base * 4, + "influx {influx} still above 4% of the {settled} cap it settled at" + ); + assert!( + influx >= settled / 100 || settled == floor, + "influx {influx} still below 1% of the {settled} cap it settled at" + ); + // The smallest such level: one step down would be above 4%. + assert!( + settled == floor || influx > settled / 2 / 25, + "influx {influx} overshot: {settled} is bigger than the one-step rule settles at" + ); + // Fixed point: many more cycles of the same influx move nothing. + for _ in 0..8 { + retune_after_scavenge(influx, 0, 0); + assert_eq!(scavenge_nursery_cap_effective_bytes(), settled); + } + } reset_for_test(); } @@ -1579,7 +1737,7 @@ mod tests { old_gen_reclaimable_pressure_bytes(), ); assert_eq!(scavenge_nursery_cap_effective_bytes(), expected); - assert!(scavenge_nursery_cap_effective_bytes() >= gc_scavenge_nursery_cap_bytes()); + assert!(scavenge_nursery_cap_effective_bytes() >= nursery_cap_floor_bytes() / 2); reset_for_test(); } } diff --git a/crates/perry-runtime/src/gc/tests/copying/adaptive_tenuring.rs b/crates/perry-runtime/src/gc/tests/copying/adaptive_tenuring.rs index b8259420ac..42da1356a7 100644 --- a/crates/perry-runtime/src/gc/tests/copying/adaptive_tenuring.rs +++ b/crates/perry-runtime/src/gc/tests/copying/adaptive_tenuring.rs @@ -196,10 +196,13 @@ fn allocation_census_seeds_the_first_cap_before_any_minor() { seeded, crate::gc::tenuring::NURSERY_CAP_REFERENCE_OBJECT_BYTES ); - // ...and the first cap already reflects it — before any collection. + // ...and the first cap already reflects it — before any collection. The + // first cap is the ladder's floor (#11549), a quarter of the base. assert_eq!( crate::gc::tenuring::influx_driven_nursery_cap_bytes(), - base * crate::gc::tenuring::nursery_cap_object_scale_permille(seeded) / 1000 + crate::gc::tenuring::nursery_cap_floor_bytes() + * crate::gc::tenuring::nursery_cap_object_scale_permille(seeded) + / 1000 ); // One-shot: a different population allocated afterwards does not move diff --git a/crates/perry-runtime/src/gc/tests/runtime_roots.rs b/crates/perry-runtime/src/gc/tests/runtime_roots.rs index d7dbf94794..c3d9e6da74 100644 --- a/crates/perry-runtime/src/gc/tests/runtime_roots.rs +++ b/crates/perry-runtime/src/gc/tests/runtime_roots.rs @@ -47,6 +47,8 @@ mod perex_replace_direct; #[cfg(feature = "regex-engine")] mod perex_reuse; #[cfg(feature = "regex-engine")] +mod perex_scratch_pressure; +#[cfg(feature = "regex-engine")] mod perex_split; #[cfg(feature = "regex-engine")] mod perex_strings; diff --git a/crates/perry-runtime/src/gc/tests/runtime_roots/perex_execution.rs b/crates/perry-runtime/src/gc/tests/runtime_roots/perex_execution.rs index 572fa5b9f6..c2cdfac93b 100644 --- a/crates/perry-runtime/src/gc/tests/runtime_roots/perex_execution.rs +++ b/crates/perry-runtime/src/gc/tests/runtime_roots/perex_execution.rs @@ -41,20 +41,24 @@ fn compile<'s>( fn perex_host_buffers_account_overlap_failure_and_unwind() { let _guard = CopyingNurseryTestGuard::new(0); let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + // #11549: operation scratch is bounded by its budget and released by the + // operation, so the budget accounts the overlap and the collector's + // external-side reading never moves. let before = external_side_live_bytes(); let memory = MemoryBudget::new(128); { let first = Buffer::::new(&memory, 64).unwrap(); - assert_eq!(external_side_live_bytes(), before + memory.live_bytes()); + assert_eq!(memory.live_bytes(), 64); assert!(matches!( Buffer::::new(&memory, 65), Err(StorageError::Limit) )); let replacement = Buffer::::new(&memory, 64).unwrap(); assert_eq!(memory.peak_bytes(), 128); - assert_eq!(external_side_live_bytes(), before + 128); + assert_eq!(memory.live_bytes(), 128); + assert_eq!(external_side_live_bytes(), before); drop(first); - assert_eq!(external_side_live_bytes(), before + 64); + assert_eq!(memory.live_bytes(), 64); drop(replacement); } let unwind = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { diff --git a/crates/perry-runtime/src/gc/tests/runtime_roots/perex_scratch_pressure.rs b/crates/perry-runtime/src/gc/tests/runtime_roots/perex_scratch_pressure.rs new file mode 100644 index 0000000000..0589419798 --- /dev/null +++ b/crates/perry-runtime/src/gc/tests/runtime_roots/perex_scratch_pressure.rs @@ -0,0 +1,140 @@ +//! #11549: a regex operation's own scratch is not collector pressure. +//! +//! Operation scratch (`perex_memory::Buffer`, the owned search path's match +//! buffers, the lent cell) is freed by the operation, never by a collection. +//! Reporting it as external side bytes put every per-call buffer's release into +//! `GC_EXTERNAL_SIDE_DRAINED_SINCE_FULL`, which old-reclaim holds as pressure +//! until the next full: dotenv's `LINE` (42 registers, past the lent cell's old +//! fixed 32) ran a budgeted full mark-sweep every ~250 parses on phantom bytes. +use super::*; +use crate::gc::policy::{external_side_live_bytes, GC_EXTERNAL_SIDE_DRAINED_SINCE_FULL}; +use crate::regex::perex_api as api; +use crate::regex::perex_memory::{Buffer, MemoryBudget}; +use crate::regex::perex_runtime::{EngineError, OWNED_SEARCHES}; +use crate::regex::RegExpHeader; +use crate::string::StringHeader; + +fn text<'s>(scope: &'s RuntimeHandleScope, bytes: &[u8]) -> RuntimeHandle<'s> { + scope.root_string_ptr(crate::string::js_string_from_bytes( + bytes.as_ptr(), + bytes.len() as u32, + )) +} + +fn regex<'s>(scope: &'s RuntimeHandleScope, pattern: &str) -> RuntimeHandle<'s> { + let pattern = text(scope, pattern.as_bytes()); + let flags = text(scope, b""); + scope.root_raw_mut_ptr(pattern.with_const_ptr::(|pattern| { + flags.with_const_ptr::(|flags| crate::regex::js_regexp_new(pattern, flags)) + })) +} + +fn search( + re: &RuntimeHandle<'_>, + input: &RuntimeHandle<'_>, +) -> Result, EngineError> { + re.with_mut_ptr::(|re| { + input.with_const_ptr::(|input| { + api::execute(re, input, false, &mut || Ok(())) + .map(|found| found.map(|m| (m.full.start(), m.full.end()))) + }) + }) +} + +/// `groups` capturing groups of one `a` each: `(a)(a)...`. A program's register +/// count is at least twice its capture count (group zero included), so this +/// needs at least `2 * (groups + 1)` registers. +fn groups_pattern(groups: usize) -> String { + "(a)".repeat(groups) +} + +fn external_readings() -> (usize, usize) { + ( + external_side_live_bytes(), + GC_EXTERNAL_SIDE_DRAINED_SINCE_FULL.with(TriggerInput::get), + ) +} + +fn owned_searches() -> usize { + OWNED_SEARCHES.with(std::cell::Cell::get) +} + +#[test] +fn perex_operation_buffer_is_not_reported_as_external_side_bytes() { + let _guard = CopyingNurseryTestGuard::new(0); + let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + let before = external_readings(); + let memory = MemoryBudget::new(1 << 20); + { + let buffer = Buffer::::new(&memory, 4096).expect("fits the budget"); + assert_eq!(buffer.len(), 4096); + assert_eq!(memory.live_bytes(), 4096 * 8, "the budget still sees it"); + assert_eq!(external_readings(), before, "allocation noted nothing"); + } + assert_eq!(memory.live_bytes(), 0); + assert_eq!(memory.peak_bytes(), 4096 * 8); + assert_eq!( + external_readings(), + before, + "a released operation buffer is not drained external pressure (#11549)" + ); + // The budget remains the bound: past it, nothing is allocated. + assert!(Buffer::::new(&memory, (1 << 20) / 8 + 1).is_err()); +} + +#[test] +fn perex_search_past_32_registers_uses_the_lent_cell_and_notes_nothing() { + let _guard = CopyingNurseryTestGuard::new(0); + let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + super::perex_public::register_host_roots(); + let scope = RuntimeHandleScope::new(); + // 20 groups: 42+ registers, which the lent cell's old fixed 32 could not + // hold, so every call built and dropped owned match buffers. + let re = regex(&scope, &groups_pattern(20)); + let input = text(&scope, "a".repeat(25).as_bytes()); + // The first call may grow the thread's cell; every later one must not + // build anything. + assert_eq!(search(&re, &input).unwrap(), Some((0, 20))); + let before = external_readings(); + let owned = owned_searches(); + for _ in 0..64 { + assert_eq!(search(&re, &input).unwrap(), Some((0, 20))); + } + assert_eq!( + owned_searches(), + owned, + "a program within the lent bound searches on the lent cell" + ); + assert_eq!( + external_readings(), + before, + "and reports no scratch to the collector" + ); +} + +#[test] +fn perex_search_past_the_lent_bound_takes_the_owned_path_and_notes_nothing() { + let _guard = CopyingNurseryTestGuard::new(0); + let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + super::perex_public::register_host_roots(); + let scope = RuntimeHandleScope::new(); + // 600 groups: 1,202+ registers, past what the lent cell may retain. The + // cell must not grow to it; the operation builds, and frees, its own. + let re = regex(&scope, &groups_pattern(600)); + let input = text(&scope, "a".repeat(600).as_bytes()); + let before = external_readings(); + let owned = owned_searches(); + for _ in 0..4 { + assert_eq!(search(&re, &input).unwrap(), Some((0, 600))); + } + assert_eq!( + owned_searches(), + owned + 4, + "a program past the lent bound runs the owned path every call" + ); + assert_eq!( + external_readings(), + before, + "owned match buffers are released by the search, not drained into GC pressure" + ); +} diff --git a/crates/perry-runtime/src/gc/tests/triggers.rs b/crates/perry-runtime/src/gc/tests/triggers.rs index 8fd1fc2515..9ef3efee2f 100644 --- a/crates/perry-runtime/src/gc/tests/triggers.rs +++ b/crates/perry-runtime/src/gc/tests/triggers.rs @@ -753,7 +753,11 @@ fn test_effective_arena_trigger_respects_armed_values() { // the cap's own basis. let nursery_capped = gc_moving_loop_polls_enabled(); let ceiling = gc_trigger_absolute_ceiling_bytes(); - let nursery_cap = gc_scavenge_nursery_cap_bytes(); + // The cap the clamp applies is the EFFECTIVE one — the survival-driven + // ladder's current size (#11549: a fresh thread starts at a quarter of the + // base), not the configured base. + let nursery_cap = crate::gc::tenuring::scavenge_nursery_cap_effective_bytes(); + assert!(nursery_cap <= gc_scavenge_nursery_cap_bytes() * 4); let prev_trigger = GC_NEXT_TRIGGER_BYTES.with(|c| c.get()); let prev_armed = GC_TRIGGER_ARMED.with(|c| c.get()); diff --git a/crates/perry-runtime/src/regex/perex_api.rs b/crates/perry-runtime/src/regex/perex_api.rs index a3cc464fe2..e061698480 100644 --- a/crates/perry-runtime/src/regex/perex_api.rs +++ b/crates/perry-runtime/src/regex/perex_api.rs @@ -48,9 +48,7 @@ fn raise(error: EngineError) -> ! { if let EngineError::Abrupt(value) = error { crate::exception::js_throw(value); } - if let EngineError::Build(BuildError::Abrupt(bits)) - | EngineError::Storage(StorageError::Abrupt(bits)) = error - { + if let EngineError::Build(BuildError::Abrupt(bits)) = error { crate::exception::js_throw(f64::from_bits(bits)); } let type_error = matches!(error, EngineError::Type(_)); diff --git a/crates/perry-runtime/src/regex/perex_memory.rs b/crates/perry-runtime/src/regex/perex_memory.rs index 5a72bbedcc..c8afa4b7ad 100644 --- a/crates/perry-runtime/src/regex/perex_memory.rs +++ b/crates/perry-runtime/src/regex/perex_memory.rs @@ -1,6 +1,22 @@ -//! Operation-owned native scratch, charged to Perry's external-byte budget. -//! No GC pointer may be stored in these buffers. Allocation/accounting can -//! collect, so callers must release all program/subject views first. +//! Operation-owned native scratch, bounded by the operation's `MemoryBudget`. +//! No GC pointer may be stored in these buffers. +//! +//! This scratch is NOT reported to the collector as external side bytes +//! (#11549). Everything here is freed by the operation that allocated it, +//! when that operation returns or unwinds; no collection can ever reclaim a +//! byte of it, and no collection is needed for it to be released. Reporting it +//! told the old-reclaim pacing the opposite: every per-call buffer's release +//! landed in `GC_EXTERNAL_SIDE_DRAINED_SINCE_FULL`, which is held as pressure +//! until the next FULL collection, so a regex loop that took the owned search +//! path (a program with more registers than the lent cell holds) was paced by +//! phantom bytes into a budgeted full mark-sweep every few hundred calls. +//! +//! What bounds it instead is the budget: every buffer, inline charge and +//! reservation is checked against the operation's hard limit +//! (`perex_api::SCRATCH_BYTES`) before it exists, so one operation can never +//! hold more than that. Storage whose size follows the SUBJECT rather than +//! that limit (`perex_replace_direct::Spans`, `perex_replace_storage`'s +//! native piece records) is not scratch in this sense and is still reported. use std::cell::Cell; use std::ops::{Deref, DerefMut}; @@ -9,7 +25,6 @@ use std::ops::{Deref, DerefMut}; pub(crate) enum StorageError { Limit, Allocation, - Abrupt(u64), } /// One operation's hard scratch/result-metadata limit. Simultaneous old/new @@ -69,8 +84,11 @@ impl Drop for Charge<'_> { } } -/// Account a stable native allocation whose GC-bearing slots are separately -/// registered with the host's mutable root scanner before this can collect. +/// Charge a stable native allocation the caller owns, such as a replacer +/// call's argument slots, to the operation's limit. Its GC-bearing slots are +/// registered with the shadow stack separately; like every other buffer here +/// it is released by the operation, not by a collection, so the collector is +/// not told about it. pub(super) struct Reservation<'a> { budget: &'a MemoryBudget, bytes: usize, @@ -78,28 +96,19 @@ pub(super) struct Reservation<'a> { impl<'a> Reservation<'a> { pub(super) fn new(budget: &'a MemoryBudget, bytes: usize) -> Result { let live = budget.check(bytes)?; - let owned = Self { budget, bytes }; budget.live.set(live); budget.peak.set(budget.peak.get().max(live)); - if bytes != 0 { - crate::exception::catch_js_throw(|| crate::gc::gc_note_external_side_alloc(bytes)) - .map_err(|value| StorageError::Abrupt(value.to_bits()))?; - } - Ok(owned) + Ok(Self { budget, bytes }) } } impl Drop for Reservation<'_> { fn drop(&mut self) { self.budget.live.set(self.budget.live.get() - self.bytes); - if self.bytes != 0 { - crate::gc::gc_note_external_side_free(self.bytes); - } } } -/// Stable initialized native allocation. Its accounting owner is established -/// before notifying the collector, so a collecting/unwinding notification -/// cannot strand a buffer or leave its bytes charged. +/// Stable initialized native allocation, charged to the operation's limit for +/// as long as it lives. Creating one never collects. pub(crate) struct Buffer<'a, T: Copy + Default> { data: Vec, budget: &'a MemoryBudget, @@ -121,18 +130,13 @@ impl<'a, T: Copy + Default> Buffer<'a, T> { .ok_or(StorageError::Limit)?; let live = budget.check(bytes)?; data.resize(count, T::default()); - let owned = Self { + budget.live.set(live); + budget.peak.set(budget.peak.get().max(live)); + Ok(Self { data, budget, bytes, - }; - budget.live.set(live); - budget.peak.set(budget.peak.get().max(live)); - if bytes != 0 { - crate::exception::catch_js_throw(|| crate::gc::gc_note_external_side_alloc(bytes)) - .map_err(|value| StorageError::Abrupt(value.to_bits()))?; - } - Ok(owned) + }) } } @@ -152,8 +156,5 @@ impl DerefMut for Buffer<'_, T> { impl Drop for Buffer<'_, T> { fn drop(&mut self) { self.budget.live.set(self.budget.live.get() - self.bytes); - if self.bytes != 0 { - crate::gc::gc_note_external_side_free(self.bytes); - } } } diff --git a/crates/perry-runtime/src/regex/perex_runtime.rs b/crates/perry-runtime/src/regex/perex_runtime.rs index f2b8f9a6cf..01a529252b 100644 --- a/crates/perry-runtime/src/regex/perex_runtime.rs +++ b/crates/perry-runtime/src/regex/perex_runtime.rs @@ -201,10 +201,21 @@ impl std::ops::DerefMut for Slots<'_, T, N> { /// fewer: `/^[a-z]+_[0-9]+$/` needs 2. Frames and undo entries start empty and /// only grow through `rebuffer`, so they are never inline. const INLINE_REGISTERS: usize = 8; -/// Registers the lent cell holds. This array is allocated once per thread, not -/// per call, so it is sized for the programs a search may bring rather than -/// for what is cheap to move. -const LENT_REGISTERS: usize = 32; +/// The most registers the lent cell will grow to hold (#11549). The cell's +/// registers are allocated once per thread and kept, so a program up to this +/// size searches without building per-call buffers after its first call. +/// +/// This is the bound on what the cell retains for registers, and the reason +/// it needs no collector accounting: at most `LENT_REGISTERS * 8` bytes +/// (8 KiB) per thread, whatever programs run. dotenv's `LINE` needs 42 (the +/// old fixed 32 sent every one of its searches down the owned path); a +/// program past the bound (hundreds of capture groups) still takes the owned +/// path, whose buffers the operation's `MemoryBudget` bounds and the +/// operation frees. +/// +/// A register count is a property of the program, so the choice between the +/// two paths is made before any work, never by a failed attempt. +const LENT_REGISTERS: usize = 1024; /// Capture spans an `exec` result can have and still be read without /// allocating. const INLINE_CAPTURES: usize = 16; @@ -253,7 +264,9 @@ impl ScratchOwner for MatchBuffers<'_> { /// frames and undo entries are the engine's own opaque scratch, exactly as in /// the owned buffers this replaces (see this module's header). struct ScratchCell { - registers: [usize; LENT_REGISTERS], + /// Grown on demand to the largest register count a search on this thread + /// has needed, never past `LENT_REGISTERS`. + registers: Vec, frames: Vec, undo: Vec, } @@ -270,7 +283,7 @@ crate::perry_thread_local! { /// costing a `_tlv_get_addr` call — the opposite of what this change is for. static LENT_SCRATCH: std::cell::RefCell = const { std::cell::RefCell::new(ScratchCell { - registers: [0; LENT_REGISTERS], + registers: Vec::new(), frames: Vec::new(), undo: Vec::new(), }) @@ -306,6 +319,13 @@ crate::perry_thread_local! { const { std::cell::Cell::new(0) }; } +#[cfg(test)] +crate::perry_thread_local! { + /// Test-only: how many searches took the owned path (built per-call + /// `MatchBuffers`), so a test can assert which path a program took. + pub(crate) static OWNED_SEARCHES: std::cell::Cell = const { std::cell::Cell::new(0) }; +} + /// Run the pre-search poll on one call in `PRE_SEARCH_POLL_STRIDE`. #[inline] fn poll_on_stride(poll: &mut impl FnMut() -> Result<(), EngineError>) -> Result<(), EngineError> { @@ -398,6 +418,19 @@ fn find_near_lent<'mem, S: ImmutableSubject>( return Ok(Lent::Fallback); }; let cell = &mut *cell; + if cell.registers.len() < registers { + // Once per thread per new high-water mark; `find_near` has already + // checked `registers <= LENT_REGISTERS`. A search initializes the + // registers it reads, so the fill value is never observed. + if cell + .registers + .try_reserve_exact(registers - cell.registers.len()) + .is_err() + { + return Ok(Lent::Fallback); + } + cell.registers.resize(registers, 0); + } // Charged exactly like the owner it replaces: the operation's limit // sees the slots a search may use, whether or not they were allocated // for it. The thread keeps the memory; the operation only borrows it. @@ -548,6 +581,8 @@ pub(crate) fn find_near<'mem, S: ImmutableSubject>( } } + #[cfg(test)] + OWNED_SEARCHES.with(|n| n.set(n.get() + 1)); poll()?; let buffers = MatchBuffers::new(memory, size)?; // A failed run reports the work it left (perex 0.1.10), so the budget diff --git a/docs/src/internals/garbage-collector.md b/docs/src/internals/garbage-collector.md index 69006a2d45..90acdcec44 100644 --- a/docs/src/internals/garbage-collector.md +++ b/docs/src/internals/garbage-collector.md @@ -132,11 +132,22 @@ forwarding stub without queueing it. `PERRY_GC_SCAVENGE` is on by default and lets nursery pressure route to the direct minor. `PERRY_GC_SCAVENGE_NURSERY_MB` tunes its base high-water cap, 16 MiB by default -; -tenuring feedback may grow the effective cap by up to 4× - -on live-set-bound workloads, where a fixed cap would multiply the per-collection -fixed cost by an enormous collection count. Generated write barriers are also on +. +The effective cap is a survival-driven ladder over that base, counted in +quarters of it +: +each thread starts at a quarter of the base + +(4 MiB) and stays there while its minors find little alive, which keeps the +young generation's footprint small. While survivor influx stays above 4% of +the cap the ladder grows in debounced ×2 steps, up to 4× the base + +(64 MiB), and it shrinks back while influx stays below 1%. On survivor-heavy +and live-set-bound workloads copying survivors, not a minor's fixed cost, +dominates, and a bigger Eden gives them time to die. The survivor target, the +promoted-cohort floor and the allocation-census seed point stay keyed on the +base: they measure lifetimes in bytes allocated, which a smaller Eden must not +shorten. Generated write barriers are also on by default. Turning them off makes generational minors unsound, so the runtime deliberately falls back to full mark-sweep. diff --git a/scripts/gc_runtime_root_holders.json b/scripts/gc_runtime_root_holders.json index 6cc116cb91..89627241f8 100644 --- a/scripts/gc_runtime_root_holders.json +++ b/scripts/gc_runtime_root_holders.json @@ -445,6 +445,18 @@ "verdict": "not_a_gc_pointer", "why": "#9772: releasable block BYTES the last idle selection promised — a size, not an address. A `Cell` compared against what the collection actually released." }, + { + "file": "crates/perry-runtime/src/gc/policy.rs", + "name": "GC_EXTERNAL_SIDE_ALLOC_PENDING", + "verdict": "not_a_gc_pointer", + "why": "Byte counters for external side allocations (usize), not addresses: pending bytes noted since the last trigger check and the live external total. Nothing here can hold a GC pointer. They surfaced as rule T once regex operation scratch stopped reporting to them (#11549): the counters are no longer reached from a registered scanner's call graph, which is why the reachability walk used to cover them." + }, + { + "file": "crates/perry-runtime/src/gc/policy.rs", + "name": "GC_EXTERNAL_SIDE_LIVE_BYTES", + "verdict": "not_a_gc_pointer", + "why": "Byte counters for external side allocations (usize), not addresses: pending bytes noted since the last trigger check and the live external total. Nothing here can hold a GC pointer. They surfaced as rule T once regex operation scratch stopped reporting to them (#11549): the counters are no longer reached from a registered scanner's call graph, which is why the reachability walk used to cover them." + }, { "file": "crates/perry-runtime/src/gc/policy.rs", "name": "GC_EXTERNAL_SIDE_DRAINED_SINCE_FULL",