diff --git a/changelog.d/10969-canonical-shape-fixtures.md b/changelog.d/10969-canonical-shape-fixtures.md new file mode 100644 index 0000000000..d83b0fc3c3 --- /dev/null +++ b/changelog.d/10969-canonical-shape-fixtures.md @@ -0,0 +1,22 @@ +Repair two runtime fixtures whose distinct-class premise ceased to imply +distinct layouts after canonical key interning. The numeric-write guard now +sees the same target names in different ordered slots (`a,b,c,d` versus +`a,b,d,c`). The imported typed-shape fixture uses different key names at the +same slot count, independently of its existing slot-count mismatch. Both +assert distinct key arrays and retain their rejection assertions. + +Validation uses release builds with `CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16`, +the 24 GiB suite cap, and `--test-threads=1`. Each repaired test passes in its +own process. Disabling the production uniform-ShapeId check or the imported +keys-validation check makes the corresponding repaired test fail on its own +rejection assertion; both sabotages are reverted. + +The delete-lane seam remains deliberately unconverted. A temporary probe of +`tombstone_hole_count_survives_readd_append` observed the re-add replacing an +owned keys array with a different, shared canonical array and changing the +ShapeId, while retaining one hole. Its unchanged hole-accounting and churn +bound assertions passed: 60 churn cycles ended with 20 stored slots. The +original stable-identity assertion remains intact and failing. Resolving the +owned delete/re-add contract requires the deferred delete-lane conversion; +no special-case bypass of canonicalization is added. The requested stop at +that boundary leaves the tsc A/B unrun; no performance result is claimed. diff --git a/changelog.d/10969-canonical-shape-gates.md b/changelog.d/10969-canonical-shape-gates.md new file mode 100644 index 0000000000..729616a6eb --- /dev/null +++ b/changelog.d/10969-canonical-shape-gates.md @@ -0,0 +1,33 @@ +Make canonical shape identity pass the merge-train custody and inventory gates. + +Split basic object allocation and coercion into `object/alloc_basic.rs`, +with explicit named re-exports preserving both existing entry-point paths. +Canonical key allocation reloads its parent and appended key through +`RuntimeHandle::across_*`; key-list reads use scoped `with_const_ptr`. +`LiveObject::across` and `js_object_set_keys` likewise couple the receiver +reload to the operation that can collect. No raw-handle ceiling is added. + +Canonical-array shared flags use the tracked-header resolver instead of +unchecked GcHeader casts. Keep existing birth-flag helpers in alloc.rs to +avoid adding ownership probes to unchanged object-construction paths. The canonical +trie's production TLS uses `perry_thread_local!`. Move canonical-key tests +and their slot-read counter into a cfg(test) module, using Perry TLS there +as well. The counter increments once per candidate slot in `probe`, reached +from `extend_slot` (via `extend_key` and `canonicalize`); no counter storage, +call or branch is compiled in production, including before this change. + +Route both weak layout scanners through the existing object-cache scanner, +keeping the collector's pinned source unchanged from main and preserving +weak visitation before the other object caches. Name the slot extension +`extend_slot` to distinguish it from collection `extend` in the holder +checker's conservative name-based call graph; the previous collision falsely +made the regex EXEC_LOOKUPS test counter look scanner-covered. No holder +inventory or diagnostic-source pin is relaxed. + +Refresh only the two changed object/mod.rs declarations using +`shape_descriptor_census.py --emit-baseline`: shape_cache_insert's former +single-line declaration becomes one additional `keys_array: *mut ArrayHeader,` +entry (2 -> 3) because it now accepts/returns LiveObject along with canonical +keys; test_shape_cache_insert's declaration now returns `*mut ArrayHeader`. +Callers must receive the canonical array, whose identity can differ from the +incoming private list. All other census entries and summary counts stay fixed. diff --git a/changelog.d/astra-licm25-canonical-read-loops.md b/changelog.d/astra-licm25-canonical-read-loops.md new file mode 100644 index 0000000000..acc6c55fb9 --- /dev/null +++ b/changelog.d/astra-licm25-canonical-read-loops.md @@ -0,0 +1,29 @@ +Counted numeric property-read loops resolve each predicted ordered key list once +at module initialization. A registered guard-shape global holds the expectation; +its external-carrier descriptor roots the keys even with no live receiver. The +preheader compares the receiver's header shape once, and the body uses constant +slot offsets. A miss runs the original loop lowering, including its existing +read optimizations. Calls observed only with short constant bounds keep that +lowering without a preheader, so frequent short misses pay no versioning cost. + +Admission is a closed grammar: one numeric addition reduction, an unchanged +receiver, a primitive bound and counter, and no calls or existing-object stores. +Numeric prechecks exclude implicit coercion callbacks. A fresh array of primitive +literals/counter values may be assigned to a separate, otherwise unread binding; +that allocation cannot dispatch JavaScript or alter the receiver. Direct module +roots have the same proof as local roots. Captures and boxed bindings are refused. +Moving polls remain enabled, and each iteration derives the receiver address +again from its registered root. + +Shape identity is not universally immutable: dictionary key lists and private +stable-tombstone epochs can change while retaining their IDs. The prediction +resolves an ordinary, generation-zero, hole-free, entirely inline canonical +shape, so neither exception can match it. At the base SHA dictionary restamping +still supplies `Ordinary`; exclusion depends on its distinct generation and +keyless descriptor, not that enum tag. GC relocates payloads and shape key edges +without changing the receiver's ShapeId or slot order. + +Coverage includes emitted guard/constant-slot assertions, refusal of stores, +calls and receiver reassignment, canonical identity across construction paths, +dictionary exclusion, Node differential cases for getters/proxies/coercions and +non-objects, and an allocating moving-GC fixture in the root-dominance corpus. diff --git a/crates/perry-codegen/src/codegen/closure.rs b/crates/perry-codegen/src/codegen/closure.rs index 8ce9a39ab6..53eb2fd77b 100644 --- a/crates/perry-codegen/src/codegen/closure.rs +++ b/crates/perry-codegen/src/codegen/closure.rs @@ -1158,6 +1158,8 @@ pub(super) fn compile_closure( masked_region_scalar_locals: std::collections::HashSet::new(), suppressed_cleared_shadow_slots: std::collections::HashSet::new(), class_field_loop_facts: Vec::new(), + canonical_read_loop: None, + canonical_read_short_bounds: std::collections::HashSet::new(), element_shape_loop_facts: Vec::new(), i32_counter_slots: HashMap::new(), numeric_accumulator_f64_slots: HashMap::new(), diff --git a/crates/perry-codegen/src/codegen/entry.rs b/crates/perry-codegen/src/codegen/entry.rs index 5b0d52c24f..cec9156408 100644 --- a/crates/perry-codegen/src/codegen/entry.rs +++ b/crates/perry-codegen/src/codegen/entry.rs @@ -826,6 +826,8 @@ pub(super) fn compile_module_entry( masked_region_scalar_locals: std::collections::HashSet::new(), suppressed_cleared_shadow_slots: std::collections::HashSet::new(), class_field_loop_facts: Vec::new(), + canonical_read_loop: None, + canonical_read_short_bounds: std::collections::HashSet::new(), element_shape_loop_facts: Vec::new(), i32_counter_slots: HashMap::new(), numeric_accumulator_f64_slots: HashMap::new(), @@ -1639,6 +1641,8 @@ pub(super) fn compile_module_entry( masked_region_scalar_locals: std::collections::HashSet::new(), suppressed_cleared_shadow_slots: std::collections::HashSet::new(), class_field_loop_facts: Vec::new(), + canonical_read_loop: None, + canonical_read_short_bounds: std::collections::HashSet::new(), element_shape_loop_facts: Vec::new(), i32_counter_slots: HashMap::new(), numeric_accumulator_f64_slots: HashMap::new(), diff --git a/crates/perry-codegen/src/codegen/function.rs b/crates/perry-codegen/src/codegen/function.rs index cbc2d25c47..17cee0df14 100644 --- a/crates/perry-codegen/src/codegen/function.rs +++ b/crates/perry-codegen/src/codegen/function.rs @@ -1257,6 +1257,8 @@ pub(super) fn compile_function( masked_region_scalar_locals: std::collections::HashSet::new(), suppressed_cleared_shadow_slots: std::collections::HashSet::new(), class_field_loop_facts: Vec::new(), + canonical_read_loop: None, + canonical_read_short_bounds: cross_module.canonical_read_short_bounds.get(&f.id).cloned().unwrap_or_default(), element_shape_loop_facts: Vec::new(), // Specialized entries seed the canonical-i32 registry with their raw // i32 params (empty otherwise — identical to the pre-phase behavior). diff --git a/crates/perry-codegen/src/codegen/method.rs b/crates/perry-codegen/src/codegen/method.rs index 0e77806ec2..9649a59077 100644 --- a/crates/perry-codegen/src/codegen/method.rs +++ b/crates/perry-codegen/src/codegen/method.rs @@ -619,6 +619,8 @@ pub(super) fn compile_method( masked_region_scalar_locals: std::collections::HashSet::new(), suppressed_cleared_shadow_slots: std::collections::HashSet::new(), class_field_loop_facts: Vec::new(), + canonical_read_loop: None, + canonical_read_short_bounds: std::collections::HashSet::new(), element_shape_loop_facts: Vec::new(), i32_counter_slots: index_i32_param_slots, numeric_accumulator_f64_slots: HashMap::new(), diff --git a/crates/perry-codegen/src/codegen/method_static.rs b/crates/perry-codegen/src/codegen/method_static.rs index eef81c0da4..04e7278444 100644 --- a/crates/perry-codegen/src/codegen/method_static.rs +++ b/crates/perry-codegen/src/codegen/method_static.rs @@ -326,6 +326,8 @@ pub(in crate::codegen) fn compile_static_method( masked_region_scalar_locals: std::collections::HashSet::new(), suppressed_cleared_shadow_slots: std::collections::HashSet::new(), class_field_loop_facts: Vec::new(), + canonical_read_loop: None, + canonical_read_short_bounds: std::collections::HashSet::new(), element_shape_loop_facts: Vec::new(), i32_counter_slots: HashMap::new(), numeric_accumulator_f64_slots: HashMap::new(), diff --git a/crates/perry-codegen/src/codegen/mod.rs b/crates/perry-codegen/src/codegen/mod.rs index 3c984f33b5..31f1ec663c 100644 --- a/crates/perry-codegen/src/codegen/mod.rs +++ b/crates/perry-codegen/src/codegen/mod.rs @@ -2456,6 +2456,7 @@ pub fn compile_module(hir: &HirModule, opts: CompileOptions) -> Result> } let mut cross_module = CrossModuleCtx { + canonical_read_short_bounds: crate::stmt::canonical_read_profit::short_bounds(hir), namespace_imports: opts.namespace_imports.iter().cloned().collect(), namespace_member_nested: opts.namespace_member_nested.iter().cloned().collect(), namespace_member_prefixes: opts.namespace_member_prefixes, diff --git a/crates/perry-codegen/src/codegen/opts.rs b/crates/perry-codegen/src/codegen/opts.rs index da977f7d82..4aca247dff 100644 --- a/crates/perry-codegen/src/codegen/opts.rs +++ b/crates/perry-codegen/src/codegen/opts.rs @@ -815,6 +815,7 @@ impl ImportedCtor { /// adding five more individual parameters to every compile_* function. /// Built once in `compile_module` from `CompileOptions`. pub(crate) struct CrossModuleCtx { + pub canonical_read_short_bounds: std::collections::HashMap>, pub namespace_imports: std::collections::HashSet, /// #7189: `(namespace local, member)` pairs whose member is itself a MODULE /// NAMESPACE, from `export * as ns from "./m.ts"` in the imported module. diff --git a/crates/perry-codegen/src/codegen/string_pool.rs b/crates/perry-codegen/src/codegen/string_pool.rs index 885c685b27..474a0f1867 100644 --- a/crates/perry-codegen/src/codegen/string_pool.rs +++ b/crates/perry-codegen/src/codegen/string_pool.rs @@ -414,6 +414,17 @@ pub(super) fn emit_string_pool( } } + // One rooted shape expectation per ordered prediction, resolved at module init. + for (keys, name) in &strings.canonical_read_shapes { + let bytes: Vec = keys.iter().flat_map(|k| k.bytes().chain(Some(0))).collect(); + let encoded: String = bytes.iter().map(|b| format!("\\{b:02X}")).collect(); + llmod.add_raw_global(format!("@{name} = internal global i32 -1, align 4")); + llmod.add_raw_global(format!( + "@{name}_keys = private constant [{} x i8] c\"{encoded}\", align 1", + bytes.len() + )); + } + // #5391 function splitting: a large bundle interns ~190K strings AND // registers tens of thousands of closures/classes/functions; emitting all of // that into ONE `__perry_init_strings` function produced a single ~32MB / @@ -436,6 +447,24 @@ pub(super) fn emit_string_pool( ops_per_chunk, ); + for (keys, name) in &strings.canonical_read_shapes { + chunker.roll_if_full(); + let blk = chunker.current_block(); + let length: usize = keys.iter().map(|k| k.len() + 1).sum(); + // js_object_shape_id_for_keys marks these keys as an external carrier: + // the weak canonical trie cannot collect the remembered descriptor. + let shape = blk.call( + I32, + "js_canonical_read_shape", + &[(PTR, &format!("@{name}_keys")), (I32, &length.to_string())], + ); + blk.store(I32, &shape, &format!("@{name}")); + blk.call_void( + "js_register_class_guard_shape", + &[(PTR, &format!("@{name}"))], + ); + } + for entry in strings.iter() { chunker.roll_if_full(); let blk = chunker.current_block(); diff --git a/crates/perry-codegen/src/expr/dispatch.rs b/crates/perry-codegen/src/expr/dispatch.rs index ed7dfd773a..93e2809827 100644 --- a/crates/perry-codegen/src/expr/dispatch.rs +++ b/crates/perry-codegen/src/expr/dispatch.rs @@ -28,6 +28,9 @@ pub(crate) fn lower_expr(ctx: &mut FnCtx<'_>, expr: &Expr) -> Result { // one. Handlers that care receive it as an argument, because they consult // it after lowering their operands — by which point the field is gone. let value_discarded = std::mem::take(&mut ctx.discard_this_expr); + if let Some(value) = crate::stmt::canonical_read_loop::lower_reduction(ctx, expr)? { + return Ok(value); + } if let Some(value) = super::suffix_cursor::try_lower(ctx, expr)? { return Ok(value); } diff --git a/crates/perry-codegen/src/expr/mod.rs b/crates/perry-codegen/src/expr/mod.rs index f6ac50e645..0a01afb9cb 100644 --- a/crates/perry-codegen/src/expr/mod.rs +++ b/crates/perry-codegen/src/expr/mod.rs @@ -1038,6 +1038,12 @@ pub(crate) struct FnCtx<'a> { /// current iteration. pub class_field_loop_facts: Vec, + /// A canonical ordinary-shape proof scoped to a non-dispatching loop. + /// Carries local/key identities only, never a raw heap address. + pub canonical_read_loop: Option, + /// Bound parameters observed only in short calls; a profitability hint. + pub canonical_read_short_bounds: std::collections::HashSet, + /// repsel #7480 / #5093: scoped loop-versioning facts for element-shape /// loops (`for (…) sum += arr[i].field`). Pushed only around the FAST /// clone of `lower_element_shape_versioned_for` @@ -3049,6 +3055,7 @@ pub(crate) mod element_shape_guard; pub(crate) mod element_shape_reads; mod js_runtime; mod literals_vars; +pub(crate) use literals_vars::bind_lowered_value_to_local; mod logical_collections; mod math_simple; mod misc_methods; diff --git a/crates/perry-codegen/src/runtime_decls/strings.rs b/crates/perry-codegen/src/runtime_decls/strings.rs index 7736c5eff5..80d08631a8 100644 --- a/crates/perry-codegen/src/runtime_decls/strings.rs +++ b/crates/perry-codegen/src/runtime_decls/strings.rs @@ -1174,6 +1174,8 @@ pub fn declare_phase_b_strings(module: &mut LlModule) { I64, &[I32, I32, I32, I64, I32], ); + module.declare_function("llvm.expect.i1", I1, &[I1, I1]); + module.declare_function("js_canonical_read_shape", I32, &[PTR, I32]); module.declare_function("js_build_class_keys_array", I64, &[I32, I32, PTR, I32]); module.declare_function("js_object_shape_id_for_keys", I32, &[I64, I32]); module.declare_function("js_register_class_guard_shape", VOID, &[PTR]); diff --git a/crates/perry-codegen/src/stmt/canonical_read_loop.rs b/crates/perry-codegen/src/stmt/canonical_read_loop.rs new file mode 100644 index 0000000000..b4bc8ab164 --- /dev/null +++ b/crates/perry-codegen/src/stmt/canonical_read_loop.rs @@ -0,0 +1,377 @@ +//! One canonical shape guard for a counted, non-dispatching read loop. +//! +//! The prediction is the ordered list of keys in the reduction. It is resolved +//! through the runtime's canonical-key tree, not learned from a receiver or IC. +//! Exact identity proves ordinary, generation-zero, hole-free, inline slots. +//! The closed body grammar below and numeric prechecks exclude user callbacks. +//! Polls remain enabled; every read re-derives the address from the receiver root. + +use crate::expr::{lower_expr, FnCtx}; +use crate::nanbox::POINTER_MASK_I64; +use crate::types::{DOUBLE, I1, I32, I64}; +use anyhow::Result; +use perry_hir::{BinaryOp, CompareOp, Expr, Stmt, UpdateOp}; + +#[derive(Clone, Debug)] +pub(crate) struct Fact { + receiver: u32, + counter: u32, + invariant_values: Vec<(u32, String)>, + keys: Vec, + accumulator: u32, + numeric_locals: Vec, + allocation_local: Option, +} + +fn collect(expr: &Expr, fact: &mut Fact) -> bool { + match expr { + Expr::Binary { + op: BinaryOp::Add, + left, + right, + } => collect(left, fact) && collect(right, fact), + Expr::Number(_) | Expr::Integer(_) => true, + Expr::LocalGet(id) => { + if !fact.numeric_locals.contains(id) { + fact.numeric_locals.push(*id); + } + true + } + Expr::PropertyGet { + object, property, .. + } => { + let Expr::LocalGet(id) = object.as_ref() else { + return false; + }; + if fact.keys.is_empty() { + fact.receiver = *id; + } + if fact.receiver != *id || property.as_bytes().contains(&0) { + return false; + } + if !fact.keys.contains(property) { + fact.keys.push(property.clone()); + } + true + } + _ => false, + } +} + +fn plan( + init: Option<&Stmt>, + condition: Option<&Expr>, + update: Option<&Expr>, + body: &[Stmt], +) -> Option { + let counter = match init? { + Stmt::Let { + id, + init: Some(Expr::Integer(0)), + .. + } => *id, + Stmt::Let { + id, + init: Some(Expr::Number(n)), + .. + } if *n == 0.0 => *id, + _ => return None, + }; + if !matches!(update?, Expr::Update { id, op: UpdateOp::Increment, .. } if *id == counter) { + return None; + } + let bound = match condition? { + Expr::Compare { + op: CompareOp::Lt, + left, + right, + } if matches!(left.as_ref(), Expr::LocalGet(id) if *id == counter) => right.as_ref(), + _ => return None, + }; + // The sole allocating extension: assigning a fresh array whose elements + // are primitive literals or the numeric counter to an otherwise unread + // local. Array-literal construction cannot dispatch a getter/setter or + // mutate the receiver; it may collect. No spread, method, or user call. + let (allocation_local, reduction) = match body { + [reduction] => (None, reduction), + [Stmt::Expr(Expr::LocalSet(id, value)), reduction] => { + let Expr::Array(items) = value.as_ref() else { + return None; + }; + if items.iter().any(|e| { + !matches!(e, Expr::Number(_) | Expr::Integer(_)) + && !matches!(e, Expr::LocalGet(id) if *id == counter) + }) { + return None; + } + (Some(*id), reduction) + } + _ => return None, + }; + let Stmt::Expr(Expr::LocalSet(accumulator, value)) = reduction else { + return None; + }; + if !matches!( + value.as_ref(), + Expr::Binary { + op: BinaryOp::Add, + .. + } + ) { + return None; + } + let mut fact = Fact { + receiver: 0, + counter, + invariant_values: vec![], + keys: vec![], + accumulator: *accumulator, + numeric_locals: vec![], + allocation_local, + }; + if !collect(value, &mut fact) || fact.keys.is_empty() { + return None; + } + match bound { + Expr::LocalGet(id) if *id != *accumulator && *id != counter => { + if !fact.numeric_locals.contains(id) { + fact.numeric_locals.push(*id); + } + } + Expr::Number(_) | Expr::Integer(_) => {} + _ => return None, + } + if fact.receiver == counter + || fact.receiver == *accumulator + || fact.numeric_locals.contains(&fact.receiver) + || *accumulator == counter + { + return None; + } + if allocation_local.is_some_and(|id| { + id == fact.receiver + || id == counter + || id == fact.accumulator + || fact.numeric_locals.contains(&id) + }) { + return None; + } + Some(fact) +} + +fn addressable(ctx: &FnCtx<'_>, id: u32) -> bool { + !ctx.boxed_vars.contains(&id) + && !ctx.closure_captures.contains_key(&id) + // Scalar replacement leaves a bookkeeping local but no receiver. + // Its existing field lowering must keep reading the field allocas; + // even the generic clone cannot read a nonexistent object root. + && !ctx.scalar_replaced.contains_key(&id) + && !ctx.scalar_replaced_arrays.contains_key(&id) + && !ctx.scalar_replaced_uppercase_sources.contains_key(&id) + // Module variables are direct registered roots too. The closed body + // writes only the accumulator and optional fresh-array binding, and + // cannot dispatch JS; neither receiver nor numeric inputs can change. + && (ctx.locals.contains_key(&id) || ctx.local_slot_reps.contains_key(&id) + || ctx.module_globals.contains_key(&id)) +} + +fn load_slots(ctx: &mut FnCtx<'_>, fact: &Fact) -> Result> { + // Deliberately read LocalGet HERE. A preheader raw address cannot survive + // the back-edge's moving poll, even though its shape proof survives it. + let recv = lower_expr(ctx, &Expr::LocalGet(fact.receiver))?; + let bits = ctx.block().bitcast_double_to_i64(&recv); + let raw = ctx.block().and(I64, &bits, POINTER_MASK_I64); + let header = crate::target_layout::object_header_size_bytes(ctx.target_triple); + let mut values = Vec::new(); + for slot in 0..fact.keys.len() { + let offset = (header + slot as u64 * 8).to_string(); + let addr = ctx.block().add(I64, &raw, &offset); + let ptr = ctx.block().inttoptr(I64, &addr); + values.push(ctx.block().load(DOUBLE, &ptr)); + } + Ok(values) +} + +fn fold( + ctx: &mut FnCtx<'_>, + expr: &Expr, + fact: &Fact, + values: &mut Option>, +) -> Result { + match expr { + Expr::Binary { + op: BinaryOp::Add, + left, + right, + } => { + let l = fold(ctx, left, fact, values)?; + let r = fold(ctx, right, fact, values)?; + Ok(ctx.block().fadd(&l, &r)) + } + Expr::PropertyGet { property, .. } => { + // Preserve operand evaluation order. Nothing in this closed + // expression can collect between the accumulator and slot loads. + if values.is_none() { + *values = Some(load_slots(ctx, fact)?); + } + Ok(values.as_ref().unwrap()[fact + .keys + .iter() + .position(|k| k == property) + .expect("proved key")] + .clone()) + } + _ => lower_expr(ctx, expr), + } +} + +/// Lower the proved reduction and invariant numeric reads ahead of +/// representation selection and property/region specializations. +pub(crate) fn lower_reduction(ctx: &mut FnCtx<'_>, expr: &Expr) -> Result> { + let Some(fact) = ctx.canonical_read_loop.clone() else { + return Ok(None); + }; + if let Expr::LocalGet(id) = expr { + if let Some((_, value)) = fact.invariant_values.iter().find(|(local, _)| local == id) { + return Ok(Some(value.clone())); + } + } + let Expr::LocalSet(id, value) = expr else { + return Ok(None); + }; + if *id != fact.accumulator { + return Ok(None); + } + let result = fold(ctx, value, &fact, &mut None)?; + crate::expr::bind_lowered_value_to_local(ctx, *id, &result, value)?; + Ok(Some(result)) +} + +// Independent 50/50 tag checks compound into an artificially cold fast +// clone, even when the shape comparison itself is balanced. Keep ordinary +// numeric/object admission likely; leave the actual shape hit unweighted so +// LLVM allocates registers for both loop versions. This emits no instruction. +fn likely_plain_input(ctx: &mut FnCtx<'_>, test: &str) -> String { + ctx.block() + .call(I1, "llvm.expect.i1", &[(I1, test), (I1, "true")]) +} + +pub(crate) fn lower( + ctx: &mut FnCtx<'_>, + init: Option<&Stmt>, + condition: Option<&Expr>, + update: Option<&Expr>, + body: &[Stmt], +) -> Result { + if ctx.canonical_read_loop.is_some() + || !ctx.pending_labels.is_empty() + || crate::expr::typed_feedback_emission_enabled() + { + return Ok(false); + } + // A short loop can pay this preheader on every call without amortizing + // it. Leave known short trips entirely on the existing lowering, with no + // runtime profitability branch or extra per-entry instruction. + if let Some(Expr::Compare { right, .. }) = condition { + let short = match right.as_ref() { + Expr::LocalGet(id) => ctx.canonical_read_short_bounds.contains(id), + Expr::Integer(n) => { + *n >= 0 && (*n as f64) < super::canonical_read_profit::SHORT_TRIP_LIMIT + } + Expr::Number(n) => *n >= 0.0 && *n < super::canonical_read_profit::SHORT_TRIP_LIMIT, + _ => false, + }; + if short { + return Ok(false); + } + } + let Some(mut fact) = plan(init, condition, update, body) else { + return Ok(false); + }; + if fact + .allocation_local + .is_some_and(|id| !addressable(ctx, id)) + || !addressable(ctx, fact.receiver) + || !addressable(ctx, fact.accumulator) + || fact.numeric_locals.iter().any(|id| !addressable(ctx, *id)) + { + return Ok(false); + } + + let slow = ctx.new_block("shape_loop.generic"); + let tag = ctx.new_block("shape_loop.tag"); + let handle = ctx.new_block("shape_loop.handle"); + let shape = ctx.new_block("shape_loop.shape"); + let numbers = ctx.new_block("shape_loop.numbers"); + let fast = ctx.new_block("shape_loop.fast"); + let done = ctx.new_block("shape_loop.done"); + let tag_l = ctx.block_label(tag); + let handle_l = ctx.block_label(handle); + let shape_l = ctx.block_label(shape); + let numbers_l = ctx.block_label(numbers); + let fast_l = ctx.block_label(fast); + let slow_l = ctx.block_label(slow); + let done_l = ctx.block_label(done); + + // The module initializer resolves and roots the prediction once. Every + // entry only loads its expectation; it never allocates strings or walks + // the weak canonical trie. Volatile observes class-guard poisoning. + let name = ctx.strings.canonical_read_shape(&fact.keys); + let expected = ctx.block().load_volatile(I32, &format!("@{name}")); + + // Numeric locals make condition/update/reduction coercion-free. This also + // protects the proof against valueOf deleting a property mid-iteration. + let mut numeric = "true".to_string(); + for id in &fact.numeric_locals { + let v = lower_expr(ctx, &Expr::LocalGet(*id))?; + let ok = super::loops::emit_js_value_is_number(ctx, &v); + numeric = ctx.block().and(I1, &numeric, &ok); + // This scalar is proven numeric on the fast edge. The admitted body + // cannot assign it or run JS, so it remains valid across moving polls. + // Keep counter/accumulator loads live: those bindings do change. + if *id != fact.counter && *id != fact.accumulator { + fact.invariant_values.push((*id, v)); + } + } + let numeric = likely_plain_input(ctx, &numeric); + ctx.block().cond_br(&numeric, &tag_l, &slow_l); + ctx.current_block = tag; + let receiver = lower_expr(ctx, &Expr::LocalGet(fact.receiver))?; + let bits = ctx.block().bitcast_double_to_i64(&receiver); + let top = ctx.block().lshr(I64, &bits, "48"); + let is_ptr = ctx.block().icmp_eq(I64, &top, "32765"); + let is_ptr = likely_plain_input(ctx, &is_ptr); + ctx.block().cond_br(&is_ptr, &handle_l, &slow_l); + ctx.current_block = handle; + let raw = ctx.block().and(I64, &bits, POINTER_MASK_I64); + let real = ctx.block().icmp_ugt(I64, &raw, "1048575"); + let real = likely_plain_input(ctx, &real); + ctx.block().cond_br(&real, &shape_l, &slow_l); + ctx.current_block = shape; + let addr = ctx.block().add(I64, &raw, "4"); + let ptr = ctx.block().inttoptr(I64, &addr); + let sid = ctx.block().load(I32, &ptr); + let hit = ctx.block().icmp_eq(I32, &sid, &expected); + ctx.block().cond_br(&hit, &numbers_l, &slow_l); + ctx.current_block = numbers; + let mut numeric = "true".to_string(); + for v in load_slots(ctx, &fact)? { + let ok = super::loops::emit_js_value_is_number(ctx, &v); + numeric = ctx.block().and(I1, &numeric, &ok); + } + let numeric = likely_plain_input(ctx, &numeric); + ctx.block().cond_br(&numeric, &fast_l, &slow_l); + + ctx.current_block = fast; + ctx.canonical_read_loop = Some(fact); + super::loops::lower_for_after_init(ctx, init, condition, update, body, "for.shape_read")?; + ctx.canonical_read_loop = None; + ctx.block().br(&done_l); + ctx.current_block = slow; + // Re-enter exactly the pre-existing lowering after initialization. No + // read hooks or IC suppression: this clone retains all baseline choices. + super::loops::lower_for_baseline_after_init(ctx, init, condition, update, body)?; + ctx.block().br(&done_l); + ctx.current_block = done; + Ok(true) +} diff --git a/crates/perry-codegen/src/stmt/canonical_read_loop_tests.rs b/crates/perry-codegen/src/stmt/canonical_read_loop_tests.rs new file mode 100644 index 0000000000..bf1a9576b6 --- /dev/null +++ b/crates/perry-codegen/src/stmt/canonical_read_loop_tests.rs @@ -0,0 +1,269 @@ +use crate::{compile_module, CompileOptions}; +use perry_hir::types::Type; +use perry_hir::{BinaryOp, CompareOp, Expr, Function, Module, Param, Stmt, UpdateOp}; + +fn read(key: &str) -> Expr { + Expr::PropertyGet { + object: Box::new(Expr::LocalGet(2)), + property: key.into(), + byte_offset: 0, + } +} +fn add(left: Expr, right: Expr) -> Expr { + Expr::Binary { + op: BinaryOp::Add, + left: Box::new(left), + right: Box::new(right), + } +} +fn reduction() -> Stmt { + Stmt::Expr(Expr::LocalSet( + 3, + Box::new(add(Expr::LocalGet(3), add(read("a"), read("b")))), + )) +} +fn probe(prefix: Vec) -> String { + probe_with_short_call(prefix, false) +} +fn probe_with_short_call(mut prefix: Vec, short_call: bool) -> String { + let allocating = prefix + .iter() + .any(|s| matches!(s, Stmt::Expr(Expr::LocalSet(5, _)))); + prefix.push(reduction()); + let mut m = Module::new("canonical_read_probe"); + m.functions.push(Function { + id: 1, + name: "run".into(), + type_params: vec![], + params: (1..=2) + .map(|id| Param { + id, + name: format!("p{id}"), + ty: Type::Any, + default: None, + decorators: vec![], + is_rest: false, + arguments_object: None, + }) + .collect(), + return_type: Type::Any, + body: vec![ + Stmt::Let { + id: 5, + name: "scratch".into(), + ty: Type::Any, + mutable: true, + init: Some(Expr::Array(vec![])), + }, + Stmt::Let { + id: 3, + name: "h".into(), + ty: Type::Number, + mutable: true, + init: Some(Expr::Integer(0)), + }, + Stmt::For { + init: Some(Box::new(Stmt::Let { + id: 4, + name: "k".into(), + ty: Type::Number, + mutable: true, + init: Some(Expr::Integer(0)), + })), + condition: Some(Expr::Compare { + op: CompareOp::Lt, + left: Box::new(Expr::LocalGet(4)), + right: Box::new(Expr::LocalGet(1)), + }), + update: Some(Expr::Update { + id: 4, + op: UpdateOp::Increment, + prefix: false, + }), + body: prefix, + }, + Stmt::Return(Some(if allocating { + Expr::IndexGet { + object: Box::new(Expr::LocalGet(5)), + index: Box::new(Expr::Integer(0)), + } + } else { + Expr::LocalGet(3) + })), + ], + is_async: false, + is_generator: false, + is_strict: true, + is_exported: true, + captures: vec![], + decorators: vec![], + was_plain_async: false, + was_unrolled: false, + }); + if short_call { + m.init.push(Stmt::Expr(Expr::Call { + callee: Box::new(Expr::FuncRef(1)), + args: vec![Expr::Integer(4), Expr::Null], + type_args: vec![], + byte_offset: 0, + })); + } + String::from_utf8( + compile_module( + &m, + CompileOptions { + emit_ir_only: true, + is_entry_module: false, + ..CompileOptions::default() + }, + ) + .unwrap(), + ) + .unwrap() +} + +#[test] +fn canonical_read_loop_has_one_shape_guard_and_constant_slots() { + let ir = probe(vec![]); + assert!( + ir.contains("for.shape_read.body"), + "fast loop must be live: {ir}" + ); + assert!( + ir.contains("call i32 @js_canonical_read_shape"), + "canonical supplier missing: {ir}" + ); + assert!( + ir.contains("call void @js_register_class_guard_shape"), + "expectation must be registered: {ir}" + ); + let run = ir + .split("define ") + .find(|f| f.contains("for.shape_read.body")) + .unwrap(); + assert!( + !run.contains("call i32 @js_canonical_read_shape"), + "entry must not resolve keys: {run}" + ); + assert!( + run.contains("load volatile i32, ptr @perry_class_guard_shape_"), + "entry loads expectation: {run}" + ); + assert!( + run.contains("call i64 @js_region_guard_prime"), + "miss must retain the baseline region: {run}" + ); + assert!( + ir.contains("_packed_get ="), + "miss must retain baseline ICs: {ir}" + ); + let body = run + .lines() + .skip_while(|l| !l.starts_with("for.shape_read.body.")) + .skip(1) + .take_while(|l| !l.starts_with("for.shape_read.update.")) + .collect::>() + .join("\n"); + assert!( + body.contains("load double"), + "fast body loads slots: {body}" + ); + assert!( + !body.contains("js_region_guard_prime"), + "fast body must not use baseline reads: {body}" + ); + assert!( + !body + .lines() + .any(|l| l.contains("lshr i64") && (l.ends_with(", 32") || l.ends_with(", 38"))), + "fast body cannot decode cached slots: {body}" + ); +} + +#[test] +fn canonical_read_loop_refuses_receiver_store() { + let store = Expr::PropertySet { + object: Box::new(Expr::LocalGet(2)), + property: "c".into(), + value: Box::new(Expr::Integer(1)), + }; + let ir = probe(vec![Stmt::Expr(store)]); + assert!( + !ir.contains("shape_loop.fast"), + "layout-changing store admitted: {ir}" + ); +} + +#[test] +fn canonical_read_loop_refuses_calls() { + let call = Expr::Call { + callee: Box::new(Expr::LocalGet(2)), + args: vec![], + type_args: vec![], + byte_offset: 0, + }; + let ir = probe(vec![Stmt::Expr(call)]); + assert!( + !ir.contains("shape_loop.fast"), + "unknown callback admitted: {ir}" + ); +} + +#[test] +fn canonical_read_loop_refuses_receiver_reassignment() { + let ir = probe(vec![Stmt::Expr(Expr::LocalSet( + 2, + Box::new(Expr::Integer(5)), + ))]); + assert!( + !ir.contains("shape_loop.fast"), + "receiver reassignment admitted: {ir}" + ); +} + +#[test] +fn canonical_read_loop_allocation_keeps_the_guard_and_poll_live() { + let ir = probe(vec![Stmt::Expr(Expr::LocalSet( + 5, + Box::new(Expr::Array(vec![Expr::LocalGet(4)])), + ))]); + assert!( + ir.contains("for.shape_read.body"), + "allocating loop must enter the guarded arm: {ir}" + ); + assert!( + ir.contains("call void @js_gc_loop_safepoint"), + "a moving poll must remain: {ir}" + ); +} + +#[test] +fn canonical_read_loop_array_initializers_cannot_call_user_code() { + let call = Expr::Call { + callee: Box::new(Expr::LocalGet(2)), + args: vec![], + type_args: vec![], + byte_offset: 0, + }; + let ir = probe(vec![Stmt::Expr(Expr::LocalSet( + 5, + Box::new(Expr::Array(vec![call])), + ))]); + assert!( + !ir.contains("shape_loop.fast"), + "array initializer callback admitted: {ir}" + ); +} + +#[test] +fn canonical_read_loop_short_calls_keep_baseline_without_a_preheader() { + let ir = probe_with_short_call(vec![], true); + assert!( + !ir.contains("shape_loop.fast"), + "short calls must not pay a guard: {ir}" + ); + assert!( + ir.contains("call i64 @js_region_guard_prime"), + "short loops retain baseline reads: {ir}" + ); +} diff --git a/crates/perry-codegen/src/stmt/canonical_read_profit.rs b/crates/perry-codegen/src/stmt/canonical_read_profit.rs new file mode 100644 index 0000000000..2c6d303225 --- /dev/null +++ b/crates/perry-codegen/src/stmt/canonical_read_profit.rs @@ -0,0 +1,54 @@ +//! Compile-time profitability hints; never representation or shape proofs. +use perry_hir::{Expr, Module}; +use std::collections::{HashMap, HashSet}; + +pub(super) const SHORT_TRIP_LIMIT: f64 = 16.0; + +pub(crate) fn short_bounds(module: &Module) -> HashMap> { + let mut calls: HashMap> = HashMap::new(); + let mut visit = |expr: &Expr| { + let Expr::Call { callee, args, .. } = expr else { + return; + }; + let Expr::FuncRef(id) = callee.as_ref() else { + return; + }; + let small: Vec = args + .iter() + .map(|arg| match arg { + Expr::Integer(n) => *n >= 0 && (*n as f64) < SHORT_TRIP_LIMIT, + Expr::Number(n) => *n >= 0.0 && *n < SHORT_TRIP_LIMIT, + _ => false, + }) + .collect(); + calls + .entry(*id) + .and_modify(|prior| { + for (index, short) in prior.iter_mut().enumerate() { + *short &= small.get(index).copied().unwrap_or(false); + } + }) + .or_insert(small); + }; + // These are hints only: an unobserved indirect/exported call can lose an + // optimization, never gain an unsound proof. Any observed long/unknown + // argument keeps versioning enabled for that parameter. + crate::collectors::for_each_expr_in_stmts(&module.init, &mut visit); + for function in &module.functions { + crate::collectors::for_each_expr_in_stmts(&function.body, &mut visit); + } + module + .functions + .iter() + .filter_map(|function| { + let args = calls.get(&function.id)?; + let bounds: HashSet = function + .params + .iter() + .zip(args) + .filter_map(|(param, short)| short.then_some(param.id)) + .collect(); + (!bounds.is_empty()).then_some((function.id, bounds)) + }) + .collect() +} diff --git a/crates/perry-codegen/src/stmt/loops.rs b/crates/perry-codegen/src/stmt/loops.rs index 96f5fbf8c7..afea9df215 100644 --- a/crates/perry-codegen/src/stmt/loops.rs +++ b/crates/perry-codegen/src/stmt/loops.rs @@ -7079,6 +7079,21 @@ pub(crate) fn lower_for( lower_stmt(ctx, init_stmt)?; } + if super::canonical_read_loop::lower(ctx, init, condition, update, body)? { + return Ok(()); + } + + lower_for_baseline_after_init(ctx, init, condition, update, body) +} + +/// The original loop lowering, shared by ordinary loops and shape misses. +pub(super) fn lower_for_baseline_after_init( + ctx: &mut FnCtx<'_>, + init: Option<&Stmt>, + condition: Option<&perry_hir::Expr>, + update: Option<&perry_hir::Expr>, + body: &[Stmt], +) -> Result<()> { // #9160: `sum += strings[maskedIndex].length`. A one-time receiver, // window, element-tag, and accumulator check admits a clone whose array // access is a raw boxed-slot load and whose length dispatch is SSO/heap diff --git a/crates/perry-codegen/src/stmt/mod.rs b/crates/perry-codegen/src/stmt/mod.rs index 34931bcadf..acad13d183 100644 --- a/crates/perry-codegen/src/stmt/mod.rs +++ b/crates/perry-codegen/src/stmt/mod.rs @@ -38,6 +38,10 @@ mod let_stmt_facts; #[cfg(test)] mod let_stmt_var_redeclare_tests; mod loops; +pub(crate) mod canonical_read_loop; +pub(crate) mod canonical_read_profit; +#[cfg(test)] +mod canonical_read_loop_tests; mod masked_window_region; #[cfg(test)] mod prealloc_module_global_tests; diff --git a/crates/perry-codegen/src/strings.rs b/crates/perry-codegen/src/strings.rs index 532e7b1393..61b8e30057 100644 --- a/crates/perry-codegen/src/strings.rs +++ b/crates/perry-codegen/src/strings.rs @@ -81,6 +81,8 @@ pub struct StringPool { /// Ordered list of unique entries; the index in this Vec is the /// interned index referenced by `interned`. entries: Vec, + /// Ordered key predictions, shared by all read loops in this module. + pub(crate) canonical_read_shapes: Vec<(Vec, String)>, /// Module-wide names for TDZ-capable bindings, including outer bindings /// read from closure bodies. Kept separately from per-function aliases. pub(crate) tdz_binding_names: HashMap, @@ -131,6 +133,21 @@ pub struct StringEntry { } impl StringPool { + /// Register a program-owned expectation for initialization with the string pool. + pub(crate) fn canonical_read_shape(&mut self, keys: &[String]) -> String { + if let Some((_, name)) = self.canonical_read_shapes.iter().find(|(k, _)| k == keys) { + return name.clone(); + } + let name = format!( + "perry_class_guard_shape_{}__read${}", + self.module_prefix, + self.canonical_read_shapes.len() + ); + self.canonical_read_shapes + .push((keys.to_vec(), name.clone())); + name + } + pub fn new() -> Self { Self::with_prefix(String::new()) } @@ -143,6 +160,7 @@ impl StringPool { Self { module_prefix, interned: HashMap::new(), + canonical_read_shapes: Vec::new(), entries: Vec::new(), tdz_binding_names: HashMap::new(), debug_location_ctx: None, diff --git a/crates/perry-runtime/src/gc/dead_owner.rs b/crates/perry-runtime/src/gc/dead_owner.rs index 1533288d1c..112f237b0e 100644 --- a/crates/perry-runtime/src/gc/dead_owner.rs +++ b/crates/perry-runtime/src/gc/dead_owner.rs @@ -428,6 +428,25 @@ pub(super) const DEAD_KEY_PRUNES: &[DeadKeyPrune] = &[ prune: crate::object::shapes::prune_dead_shape_keys, young_prune: Some(crate::object::shapes::prune_dead_shape_keys_young), }, + // #10868 step 2.5 stage 1b: the canonical keys trie holds its arrays + // WEAKLY, so a node whose array did not survive has to be reaped here or + // the next probe dereferences freed memory. Runs after the shape prune + // for the same reason the transition-cache prune does: by then a dead + // keys edge has no descriptor left to confuse the question. + DeadKeyPrune { + table: "CANONICAL_KEYS (canonical keys trie)", + owner: DeadKeyOwner::Any, + prune: crate::object::canonical_keys::prune_dead_canonical_keys, + young_prune: None, + }, + // #10868 step 2.5: the same weak discipline for the class keys-array + // memo. A dropped entry costs one rebuild, never a wrong answer. + DeadKeyPrune { + table: "CLASS_KEYS_BY_ID (class -> keys array memo)", + owner: DeadKeyOwner::Any, + prune: crate::object::alloc::prune_dead_class_keys_entries, + young_prune: None, + }, // Re-keyed by the per-object move hook, not by a metadata visitor. DeadKeyPrune { table: "state().exotic_expando.entries", diff --git a/crates/perry-runtime/src/gc/layout/typed_shape.rs b/crates/perry-runtime/src/gc/layout/typed_shape.rs index 785625611a..91da06ab76 100644 --- a/crates/perry-runtime/src/gc/layout/typed_shape.rs +++ b/crates/perry-runtime/src/gc/layout/typed_shape.rs @@ -659,8 +659,7 @@ pub extern "C" fn js_gc_declare_typed_shape_layout( mod imported_shape_slot_tests { use super::*; - fn keys_for(class_id: u32) -> u64 { - let packed = b"next\0value\0"; + fn keys_for(class_id: u32, packed: &[u8]) -> u64 { crate::object::js_build_class_keys_array(class_id, 2, packed.as_ptr(), packed.len() as u32) as usize as u64 } @@ -675,8 +674,8 @@ mod imported_shape_slot_tests { guard: Box, } - fn slots(class_id: u32) -> Slots { - let keys = keys_for(class_id); + fn slots(class_id: u32, packed: &[u8]) -> Slots { + let keys = keys_for(class_id, packed); let ordinary = crate::object::shapes::js_object_shape_id_for_keys(keys, 2); Slots { keys: Box::new(keys), @@ -733,7 +732,7 @@ mod imported_shape_slot_tests { #[test] fn slots_registered_before_the_typed_id_are_rewritten_at_mint() { let class_id = 0x0B1_1001; - let mut s = slots(class_id); + let mut s = slots(class_id, b"next\0value\0"); let ordinary = *s.shape; register(class_id, &mut s); assert_eq!(*s.shape, ordinary, "nothing typed exists yet"); @@ -746,7 +745,7 @@ mod imported_shape_slot_tests { #[test] fn slots_registered_after_the_typed_id_are_rewritten_immediately() { let class_id = 0x0B1_1002; - let mut s = slots(class_id); + let mut s = slots(class_id, b"next\0value\0"); let typed = mint(class_id, *s.keys); register(class_id, &mut s); assert_published(class_id, &s, typed); @@ -757,7 +756,7 @@ mod imported_shape_slot_tests { #[test] fn a_disabled_inline_path_keeps_imported_expectations_poisoned() { let class_id = 0x0B1_1005; - let mut s = slots(class_id); + let mut s = slots(class_id, b"next\0value\0"); register(class_id, &mut s); crate::object::disable_class_field_inline_guard(); assert_eq!( @@ -781,7 +780,10 @@ mod imported_shape_slot_tests { fn mismatched_slots_keep_their_ordinary_id() { let class_id = 0x0B1_1003; let other_class = 0x0B1_1004; - let mut foreign = slots(other_class); + // A different class id with equal keys now shares the same canonical + // array. Use different names, with the same slot count, to isolate + // the keys mismatch from the independent count mismatch below. + let mut foreign = slots(other_class, b"other_next\0other_value\0"); let foreign_ordinary = *foreign.shape; js_register_imported_class_shape_slot( class_id, @@ -791,7 +793,11 @@ mod imported_shape_slot_tests { foreign.image.as_mut_ptr(), &mut *foreign.guard as *mut u32, ); - let mut narrow = slots(class_id); + let mut narrow = slots(class_id, b"next\0value\0"); + assert_ne!( + *foreign.keys, *narrow.keys, + "fixture requires different canonical keys arrays" + ); let narrow_ordinary = *narrow.shape; js_register_imported_class_shape_slot( class_id, diff --git a/crates/perry-runtime/src/object/alloc.rs b/crates/perry-runtime/src/object/alloc.rs index 1ae09c00e7..1440db9260 100644 --- a/crates/perry-runtime/src/object/alloc.rs +++ b/crates/perry-runtime/src/object/alloc.rs @@ -5,6 +5,10 @@ //! changes. Shared state and helpers remain in the parent `object` //! module and are reached via `use super::*;`. +pub use super::alloc_basic::{ + js_object_alloc, js_object_alloc_fast, js_object_alloc_fast_with_parent, + js_object_alloc_with_parent, js_object_coerce, +}; use super::*; /// class_id -> (keys array address, field count). @@ -18,8 +22,25 @@ use super::*; /// per-registration; a `claude-code --help` profile put SipHash under /// `remember_class_keys_array` at 0.175% of samples. /// -/// Iteration-order safe: the map is only ever `insert`ed and `get`ed (the two -/// sites below). Nothing iterates it, so the hasher cannot reorder anything. +/// WEAK, per #6759 phase 3 and the discipline `canonical_keys` already uses: +/// the address is rewritten on move by [`scan_class_keys_roots_mut`] and the +/// entry is dropped on death by [`prune_dead_class_keys_entries`]. +/// +/// It used to hold the address as a bare `usize` the collector knew nothing +/// about, which was sound only by coincidence — the array it happened to be +/// handed came from `js_array_alloc_with_length_longlived` and never moved +/// (#179). #10868 step 2.5 substituted a canonical array into the shape +/// cache, the coincidence ended, and a moved array left this table pointing +/// at freed memory. The first fix was to allocate canonical arrays longlived +/// too, which restored the coincidence and cost minor reclamation of every +/// canonical array — a retention regression seven GC tests named. Scanning +/// the table is the fix; the allocator was the dressing. Ralph's standing +/// constraint, arriving as a bug: a side table the collector does not know +/// about is the thing we keep paying for. +/// +/// Iteration-order safe, still: the two mutating passes rewrite independent +/// values and drop entries by a per-entry predicate, so neither depends on +/// the order the hasher yields. static CLASS_KEYS_BY_ID: std::sync::RwLock< Option>, > = std::sync::RwLock::new(None); @@ -55,6 +76,62 @@ fn remember_class_keys_array(class_id: u32, field_count: u32, keys_array: *mut A } } +/// GC root scanner: rewrite each remembered keys-array address across a move. +/// WEAK — the address is visited as metadata, never marked, so remembering a +/// class's keys array does not keep it alive. +pub fn scan_class_keys_roots_mut(visitor: &mut crate::gc::RuntimeRootVisitor<'_>) { + let Ok(mut guard) = CLASS_KEYS_BY_ID.write() else { + return; + }; + let Some(map) = guard.as_mut() else { + return; + }; + for entry in map.values_mut() { + if entry.0 == 0 { + continue; + } + let mut addr = entry.0; + if visitor.visit_metadata_usize_slot(&mut addr) { + entry.0 = addr; + } + } +} + +/// Post-trace death prune. A dropped entry costs one rebuild of the class's +/// keys array — `registered_class_keys_array` already answers `None` for a +/// zeroed address and every caller re-derives — which is what makes +/// weakening this table safe rather than merely possible. +#[cold] +pub(crate) fn prune_dead_class_keys_entries(is_dead_owner: &dyn Fn(usize) -> bool) { + let Ok(mut guard) = CLASS_KEYS_BY_ID.write() else { + return; + }; + let Some(map) = guard.as_mut() else { + return; + }; + map.retain(|_, entry| { + let addr = entry.0; + if addr == 0 { + return false; + } + if is_dead_owner(addr) { + return false; + } + // An address the arena recycled for a non-array tenant is dead to us + // whatever `is_dead_owner` says about the new occupant. + // SAFETY: a read-only tracked-header probe. + unsafe { + match crate::value::addr_class::try_read_tracked_gc_header(addr) { + Some(gc) => { + let ty = (*gc.as_ptr()).obj_type; + ty == crate::gc::GC_TYPE_ARRAY || ty == crate::gc::GC_TYPE_LAZY_ARRAY + } + None => false, + } + } + }); +} + pub(crate) fn registered_class_keys_array(class_id: u32) -> Option<(*mut ArrayHeader, u32)> { let guard = CLASS_KEYS_BY_ID.read().ok()?; let (addr, field_count) = guard.as_ref()?.get(&class_id).copied()?; @@ -64,13 +141,6 @@ pub(crate) fn registered_class_keys_array(class_id: u32) -> Option<(*mut ArrayHe Some((addr as *mut ArrayHeader, field_count)) } -/// Allocate a new object with the given class ID and field count -/// Returns a pointer to the object header -#[no_mangle] -pub extern "C" fn js_object_alloc(class_id: u32, field_count: u32) -> *mut ObjectHeader { - js_object_alloc_with_parent(class_id, 0, field_count) -} - /// #1175: allocate an object whose `[[Prototype]]` is null. Same layout as /// `js_object_alloc`, but the `OBJ_FLAG_NULL_PROTO` bit is set on the GC /// header so `Object.getPrototypeOf` returns null instead of the heap @@ -113,158 +183,6 @@ pub(crate) unsafe fn mark_object_plain_ordinary(obj: *mut ObjectHeader) { (*gc)._reserved |= crate::gc::OBJ_FLAG_PLAIN_ORDINARY; } -/// `Object(value)` plain-call coercion (#3149, ECMAScript §20.1.1.1 / ToObject). -/// -/// Takes and returns a NaN-boxed JSValue (`f64`): -/// - `undefined` / `null` / no-arg → a fresh ordinary `{}`. -/// - an existing object/array/function (any pointer value) → returned unchanged. -/// - primitive values → boxed primitive wrapper objects so -/// `Object(true).valueOf()`, `Object(0).valueOf()`, -/// `Object("x").valueOf()`, and util.types boxed checks match Node. -/// -/// The `new Object(value)` form is handled separately by -/// `js_new_function_construct`'s `"Object"` arm; this is only the bare-call -/// path that previously fell through to the generic dispatcher and returned -/// `undefined`. -#[no_mangle] -pub extern "C" fn js_object_coerce(value: f64) -> f64 { - let jsval = crate::value::JSValue::from_bits(value.to_bits()); - if jsval.is_undefined() || jsval.is_null() { - let obj = js_object_alloc(0, 0); - return crate::value::js_nanbox_pointer(obj as i64); - } - if jsval.is_bigint() { - return crate::builtins::js_boxed_bigint_new(value); - } - if unsafe { crate::symbol::js_is_symbol(value) } != 0 { - return crate::builtins::js_boxed_symbol_new(value); - } - if jsval.is_pointer() { - // Already an object/array/function — pass through unchanged. - return value; - } - if jsval.is_bool() { - return crate::builtins::js_boxed_boolean_new(value); - } - if jsval.is_any_string() { - return crate::builtins::js_boxed_string_new(value, 1); - } - if crate::object::class_ref_id(value).is_some() { - // A constructor ClassRef shares the INT32 encoding but is already a - // Function object, so ToObject returns it unchanged (#10461). - return value; - } - crate::builtins::js_boxed_number_new(value) -} - -/// Allocate a new object with class ID, parent class ID, and field count -/// The parent_class_id is used for instanceof inheritance checks -/// Returns a pointer to the object header -#[no_mangle] -pub extern "C" fn js_object_alloc_with_parent( - class_id: u32, - parent_class_id: u32, - field_count: u32, -) -> *mut ObjectHeader { - // Register this class's parent for inheritance lookups - if parent_class_id != 0 { - register_class(class_id, parent_class_id); - } - - let header_size = std::mem::size_of::(); - // Allocate at least INLINE_SLOT_FLOOR field slots to match - // js_object_set_field_by_name's alloc_limit assumption - // (max(field_count, INLINE_SLOT_FLOOR)). Without this, empty objects ({}) - // with field_count=0 would have 0 field slots but - // js_object_set_field_by_name writes up to the floor inline, causing a heap - // buffer overflow into adjacent arena objects. - let alloc_field_count = std::cmp::max(field_count as usize, crate::object::INLINE_SLOT_FLOOR); - let fields_size = alloc_field_count * std::mem::size_of::(); - let total_size = header_size + fields_size; - - let ptr = arena_alloc_gc(total_size, 8, crate::gc::GC_TYPE_OBJECT) as *mut ObjectHeader; - - unsafe { - // Initialize header - (*ptr).class_id = class_id; - (*ptr).parent_class_id = parent_class_id; - // GC_STORE_AUDIT(INIT): fresh object starts with no per-object meta record (#6759 B). - (*ptr).meta = ptr::null_mut(); - - // Initialize ALL allocated field slots to undefined (not just field_count) - // We allocate max(field_count, 8) slots but must zero all of them to prevent - // stale data from previously freed GC objects from bleeding through. - let fields_ptr = (ptr as *mut u8).add(std::mem::size_of::()) as *mut JSValue; - for i in 0..alloc_field_count { - // GC_STORE_AUDIT(INIT): freshly allocated object field slot is initialized pointer-free. - ptr::write(fields_ptr.add(i), JSValue::undefined()); - } - crate::gc::layout_init_pointer_free(ptr as *mut u8); - // #8113: the birth live-slot bound is published here and nowhere else. - crate::object::shapes::birth_publish_object_shape(ptr, field_count); - - ptr - } -} - -/// Fast object allocation using bump allocator - NO field initialization -/// This is significantly faster for hot paths where constructor immediately sets all fields -/// Returns a pointer to the object header with UNINITIALIZED fields -#[no_mangle] -pub extern "C" fn js_object_alloc_fast(class_id: u32, field_count: u32) -> *mut ObjectHeader { - let header_size = std::mem::size_of::(); - let alloc_field_count = std::cmp::max(field_count as usize, crate::object::INLINE_SLOT_FLOOR); - let fields_size = alloc_field_count * std::mem::size_of::(); - let total_size = header_size + fields_size; - - let ptr = arena_alloc_gc(total_size, 8, crate::gc::GC_TYPE_OBJECT) as *mut ObjectHeader; - - unsafe { - // Initialize header only - fields left uninitialized for constructor to fill - (*ptr).class_id = class_id; - (*ptr).parent_class_id = 0; - // GC_STORE_AUDIT(INIT): fresh object starts with no per-object meta record (#6759 B). - (*ptr).meta = ptr::null_mut(); - crate::gc::layout_init_pointer_free(ptr as *mut u8); - // #8113: the birth live-slot bound is published here and nowhere else. - crate::object::shapes::birth_publish_object_shape(ptr, field_count); - } - - ptr -} - -/// Fast object allocation with parent class ID - NO field initialization -#[no_mangle] -pub extern "C" fn js_object_alloc_fast_with_parent( - class_id: u32, - parent_class_id: u32, - field_count: u32, -) -> *mut ObjectHeader { - // Only register class if it has a parent (one-time operation per class) - if parent_class_id != 0 { - register_class(class_id, parent_class_id); - } - - let header_size = std::mem::size_of::(); - let alloc_field_count = std::cmp::max(field_count as usize, crate::object::INLINE_SLOT_FLOOR); - let fields_size = alloc_field_count * std::mem::size_of::(); - let total_size = header_size + fields_size; - - let ptr = arena_alloc_gc(total_size, 8, crate::gc::GC_TYPE_OBJECT) as *mut ObjectHeader; - - unsafe { - (*ptr).class_id = class_id; - (*ptr).parent_class_id = parent_class_id; - // GC_STORE_AUDIT(INIT): fresh object starts with no per-object meta record (#6759 B). - (*ptr).meta = ptr::null_mut(); - crate::gc::layout_init_pointer_free(ptr as *mut u8); - // #8113: the birth live-slot bound is published here and nowhere else. - crate::object::shapes::birth_publish_object_shape(ptr, field_count); - } - - ptr -} - /// Fast class instance allocator that takes a pre-built keys_array /// pointer directly, skipping the per-call SHAPE_CACHE lookup. The /// codegen pre-builds the keys_array ONCE at module init time @@ -444,7 +362,11 @@ pub extern "C" fn js_build_class_keys_array( } if field_count == 0 || packed_keys_len == 0 || packed_keys.is_null() { let arr = crate::array::js_array_alloc_with_length_longlived(0); - shape_cache_insert(shape_id, arr); + let (_, arr) = shape_cache_insert( + shape_id, + crate::object::canonical_keys::LiveObject::none(), + arr, + ); remember_class_keys_array(class_id, field_count, arr); return arr; } @@ -506,7 +428,11 @@ pub extern "C" fn js_build_class_keys_array( unsafe { crate::gc::layout_init_all_pointer_slots(arr as *mut u8); } - shape_cache_insert(shape_id, arr); + let (_, arr) = shape_cache_insert( + shape_id, + crate::object::canonical_keys::LiveObject::none(), + arr, + ); remember_class_keys_array(class_id, field_count, arr); arr } @@ -539,9 +465,19 @@ pub extern "C" fn js_object_alloc_class_with_keys( let fields_size = alloc_field_count * std::mem::size_of::(); let total_size = header_size + fields_size; - let ptr = arena_alloc_gc(total_size, 8, crate::gc::GC_TYPE_OBJECT) as *mut ObjectHeader; + // #10868 stage 1c: the receiver is carried as a `LiveObject`, never as a + // raw binding, because `shape_cache_insert` below CAN COLLECT — stage 1b + // made it allocate. The token is moved into that call and reassigned from + // its result, so the stale pointer this function used to write its keys + // edge with is no longer nameable. + let mut live = crate::object::canonical_keys::LiveObject::new(arena_alloc_gc( + total_size, + 8, + crate::gc::GC_TYPE_OBJECT, + ) as *mut ObjectHeader); unsafe { + let ptr = live.as_ptr(); (*ptr).class_id = class_id; (*ptr).parent_class_id = parent_class_id; // GC_STORE_AUDIT(INIT): fresh object starts with no per-object meta record (#6759 B). @@ -587,11 +523,15 @@ pub extern "C" fn js_object_alloc_class_with_keys( crate::array::note_array_slot_layout_only(arr, i, nanboxed.to_bits()); } } - shape_cache_insert(shape_id, arr); + // `live` is moved into the call and reassigned from its result; the + // pre-call value is not nameable afterwards. + let (live_after, arr) = shape_cache_insert(shape_id, live, arr); + live = live_after; (arr, shape_cache_get_with_id(shape_id).1) }; unsafe { + let ptr = live.as_ptr(); set_object_keys_array_with_live(ptr, keys_arr, field_count); // #6759 C3 rung 2, completed: birth-stamp here too. #8009 stamped the // COMPILED entry point (`js_object_alloc_class_inline_keys_stamped`) @@ -602,7 +542,7 @@ pub extern "C" fn js_object_alloc_class_with_keys( crate::object::shapes::birth_stamp_object_shape(ptr, runtime_shape_id, field_count); } remember_class_keys_array(class_id, field_count, keys_arr); - ptr + live.as_ptr() } /// Allocate a subclass instance whose parent was resolved DYNAMICALLY at @@ -706,7 +646,14 @@ pub extern "C" fn js_object_alloc_class_dynamic_parent( crate::array::note_array_slot_layout_only(arr, idx, nanboxed.to_bits()); } } - shape_cache_insert(shape_id, arr); + // No unrooted receiver crosses this call: the object is allocated + // after it here, and in `js_object_alloc_with_shape` it is already + // held in a `RuntimeHandleScope` and reloaded below. + let (_, arr) = shape_cache_insert( + shape_id, + crate::object::canonical_keys::LiveObject::none(), + arr, + ); (arr, merged_len as u32, shape_cache_get_with_id(shape_id).1) }; @@ -824,7 +771,14 @@ pub extern "C" fn js_object_alloc_with_shape( } } let arr = arr_handle.get_raw_mut_ptr::(); - shape_cache_insert(shape_id, arr); + // No unrooted receiver crosses this call: the object is allocated + // after it here, and in `js_object_alloc_with_shape` it is already + // held in a `RuntimeHandleScope` and reloaded below. + let (_, arr) = shape_cache_insert( + shape_id, + crate::object::canonical_keys::LiveObject::none(), + arr, + ); (arr, shape_cache_get_with_id(shape_id).1) }; diff --git a/crates/perry-runtime/src/object/alloc_basic.rs b/crates/perry-runtime/src/object/alloc_basic.rs new file mode 100644 index 0000000000..1b475c1e21 --- /dev/null +++ b/crates/perry-runtime/src/object/alloc_basic.rs @@ -0,0 +1,162 @@ +//! Basic object allocation and coercion. + +use super::*; + +/// Allocate a new object with the given class ID and field count +/// Returns a pointer to the object header +#[no_mangle] +pub extern "C" fn js_object_alloc(class_id: u32, field_count: u32) -> *mut ObjectHeader { + js_object_alloc_with_parent(class_id, 0, field_count) +} + +/// `Object(value)` plain-call coercion (#3149, ECMAScript §20.1.1.1 / ToObject). +/// +/// Takes and returns a NaN-boxed JSValue (`f64`): +/// - `undefined` / `null` / no-arg → a fresh ordinary `{}`. +/// - an existing object/array/function (any pointer value) → returned unchanged. +/// - primitive values → boxed primitive wrapper objects so +/// `Object(true).valueOf()`, `Object(0).valueOf()`, +/// `Object("x").valueOf()`, and util.types boxed checks match Node. +/// +/// The `new Object(value)` form is handled separately by +/// `js_new_function_construct`'s `"Object"` arm; this is only the bare-call +/// path that previously fell through to the generic dispatcher and returned +/// `undefined`. +#[no_mangle] +pub extern "C" fn js_object_coerce(value: f64) -> f64 { + let jsval = crate::value::JSValue::from_bits(value.to_bits()); + if jsval.is_undefined() || jsval.is_null() { + let obj = js_object_alloc(0, 0); + return crate::value::js_nanbox_pointer(obj as i64); + } + if jsval.is_bigint() { + return crate::builtins::js_boxed_bigint_new(value); + } + if unsafe { crate::symbol::js_is_symbol(value) } != 0 { + return crate::builtins::js_boxed_symbol_new(value); + } + if jsval.is_pointer() { + // Already an object/array/function — pass through unchanged. + return value; + } + if jsval.is_bool() { + return crate::builtins::js_boxed_boolean_new(value); + } + if jsval.is_any_string() { + return crate::builtins::js_boxed_string_new(value, 1); + } + if crate::object::class_ref_id(value).is_some() { + // A constructor ClassRef shares the INT32 encoding but is already a + // Function object, so ToObject returns it unchanged (#10461). + return value; + } + crate::builtins::js_boxed_number_new(value) +} + +/// Allocate a new object with class ID, parent class ID, and field count +/// The parent_class_id is used for instanceof inheritance checks +/// Returns a pointer to the object header +#[no_mangle] +pub extern "C" fn js_object_alloc_with_parent( + class_id: u32, + parent_class_id: u32, + field_count: u32, +) -> *mut ObjectHeader { + // Register this class's parent for inheritance lookups + if parent_class_id != 0 { + register_class(class_id, parent_class_id); + } + + let header_size = std::mem::size_of::(); + // Allocate at least INLINE_SLOT_FLOOR field slots to match + // js_object_set_field_by_name's alloc_limit assumption + // (max(field_count, INLINE_SLOT_FLOOR)). Without this, empty objects ({}) + // with field_count=0 would have 0 field slots but + // js_object_set_field_by_name writes up to the floor inline, causing a heap + // buffer overflow into adjacent arena objects. + let alloc_field_count = std::cmp::max(field_count as usize, crate::object::INLINE_SLOT_FLOOR); + let fields_size = alloc_field_count * std::mem::size_of::(); + let total_size = header_size + fields_size; + + let ptr = arena_alloc_gc(total_size, 8, crate::gc::GC_TYPE_OBJECT) as *mut ObjectHeader; + + unsafe { + // Initialize header + (*ptr).class_id = class_id; + (*ptr).parent_class_id = parent_class_id; + // GC_STORE_AUDIT(INIT): fresh object starts with no per-object meta record (#6759 B). + (*ptr).meta = ptr::null_mut(); + + // Initialize ALL allocated field slots to undefined (not just field_count) + // We allocate max(field_count, 8) slots but must zero all of them to prevent + // stale data from previously freed GC objects from bleeding through. + let fields_ptr = (ptr as *mut u8).add(std::mem::size_of::()) as *mut JSValue; + for i in 0..alloc_field_count { + // GC_STORE_AUDIT(INIT): freshly allocated object field slot is initialized pointer-free. + ptr::write(fields_ptr.add(i), JSValue::undefined()); + } + crate::gc::layout_init_pointer_free(ptr as *mut u8); + // #8113: the birth live-slot bound is published here and nowhere else. + crate::object::shapes::birth_publish_object_shape(ptr, field_count); + + ptr + } +} + +/// Fast object allocation using bump allocator - NO field initialization +/// This is significantly faster for hot paths where constructor immediately sets all fields +/// Returns a pointer to the object header with UNINITIALIZED fields +#[no_mangle] +pub extern "C" fn js_object_alloc_fast(class_id: u32, field_count: u32) -> *mut ObjectHeader { + let header_size = std::mem::size_of::(); + let alloc_field_count = std::cmp::max(field_count as usize, crate::object::INLINE_SLOT_FLOOR); + let fields_size = alloc_field_count * std::mem::size_of::(); + let total_size = header_size + fields_size; + + let ptr = arena_alloc_gc(total_size, 8, crate::gc::GC_TYPE_OBJECT) as *mut ObjectHeader; + + unsafe { + // Initialize header only - fields left uninitialized for constructor to fill + (*ptr).class_id = class_id; + (*ptr).parent_class_id = 0; + // GC_STORE_AUDIT(INIT): fresh object starts with no per-object meta record (#6759 B). + (*ptr).meta = ptr::null_mut(); + crate::gc::layout_init_pointer_free(ptr as *mut u8); + // #8113: the birth live-slot bound is published here and nowhere else. + crate::object::shapes::birth_publish_object_shape(ptr, field_count); + } + + ptr +} + +/// Fast object allocation with parent class ID - NO field initialization +#[no_mangle] +pub extern "C" fn js_object_alloc_fast_with_parent( + class_id: u32, + parent_class_id: u32, + field_count: u32, +) -> *mut ObjectHeader { + // Only register class if it has a parent (one-time operation per class) + if parent_class_id != 0 { + register_class(class_id, parent_class_id); + } + + let header_size = std::mem::size_of::(); + let alloc_field_count = std::cmp::max(field_count as usize, crate::object::INLINE_SLOT_FLOOR); + let fields_size = alloc_field_count * std::mem::size_of::(); + let total_size = header_size + fields_size; + + let ptr = arena_alloc_gc(total_size, 8, crate::gc::GC_TYPE_OBJECT) as *mut ObjectHeader; + + unsafe { + (*ptr).class_id = class_id; + (*ptr).parent_class_id = parent_class_id; + // GC_STORE_AUDIT(INIT): fresh object starts with no per-object meta record (#6759 B). + (*ptr).meta = ptr::null_mut(); + crate::gc::layout_init_pointer_free(ptr as *mut u8); + // #8113: the birth live-slot bound is published here and nowhere else. + crate::object::shapes::birth_publish_object_shape(ptr, field_count); + } + + ptr +} diff --git a/crates/perry-runtime/src/object/canonical_keys.rs b/crates/perry-runtime/src/object/canonical_keys.rs new file mode 100644 index 0000000000..07af5af54f --- /dev/null +++ b/crates/perry-runtime/src/object/canonical_keys.rs @@ -0,0 +1,947 @@ +//! Canonical keys arrays: the ADDRESS is the content identity (#10868 step +//! 2.5 stage 1b). +//! +//! `facts_key` folds SIX identity facts and one of them is the keys array's +//! ADDRESS. Two objects whose ordered key lists are byte-identical but whose +//! arrays were allocated separately therefore mint two ShapeIds for one +//! layout. On a real `ts.transpileModule` that is 32,246 of 42,097 mints +//! (`key_count` 19,923 plus `fresh_keys_known_list` 12,323) against a process +//! that ABORTS when the id space runs out. +//! +//! The fix is not to change `facts_key`. It is to make the address tell the +//! truth: obtain every keys array from here and exactly one array exists per +//! distinct ordered key list, so folding the pointer IS folding the content, +//! and the probe path stays byte-for-byte what it was. +//! +//! ## The structure is a TRIE, so `extend_slot` is O(1) and no content is hashed +//! +//! L8.3.15 named the one hard problem: `facts_key` is O(1) because it folds a +//! pointer, and content is O(N). A content-hash intern table moves that O(N) +//! off the probe path onto array creation — better, but still O(N) per grow +//! and O(N^2) to build an N-key object one key at a time. +//! +//! Canonical arrays form a TREE. Every one of them is (canonical parent, one +//! appended slot), rooted at the empty list. So the table is an EDGE map, not +//! a content map; [`extend_slot`] is one hash probe on `(parent node, slot hash)` +//! plus an exact check of the single appended slot; and no content is ever +//! walked on the grow path. [`canonicalize`] — the general constructor, for a +//! producer that hands over a whole list — is a fold of `extend_slot` over the +//! slots: the same one path N times, not a second path. +//! +//! ## Node ids, not addresses, so the collector touches one `Vec` +//! +//! Edges are keyed by NODE ID, which the collector cannot move. Only +//! `Node::addr` and the address index are address-typed, so a minor visits a +//! contiguous `Vec` and patches a handful of index entries, instead of +//! rekeying an address-keyed edge map the way `move_shape_family` must rekey +//! `families`. +//! +//! ## Weak, exactly like the transition cache — and that is what bounds it +//! +//! #6759 phase 3 made the transition cache's `next_keys` WEAK (rewritten on +//! move, dropped on death) after strong rooting pinned up to 16,384 keys +//! arrays and, through them, their descriptors: 786k descriptors on a +//! workload holding under 400 live objects. This table is weak for the same +//! reason, through the same two mechanisms — [`scan_canonical_keys_roots_mut`] +//! and [`prune_dead_canonical_keys`]. +//! +//! That answers L8.3.15c's retention worry, which assumed the intern table +//! would hold its arrays: **it holds none**. A node whose array died is +//! dropped, retention is proportional to LIVE layouts, and step 2.5 therefore +//! introduces no latch trigger of its own. (`ShapeObjectKind::Dictionary` +//! ships in the same PR regardless, per L8.3.15f, so whichever PR later wires +//! a trigger already has the kind beneath it.) +//! +//! Dropping a node whose array died can ORPHAN its children: they stay +//! adoptable and extendable, but a later walk from the root rebuilds the chain +//! and mints one duplicate layout. That is a mint, never a wrong answer, and +//! `fresh_keys_known_list` in the mint census is precisely its witness — which +//! is why the census, not a perf gate, is the acceptance instrument here. + +use std::cell::RefCell; +use std::collections::HashMap; + +use crate::array::ArrayHeader; +use crate::JSValue; +use crate::StringHeader; + +// Resolve a compiler-predicted ordered key list through the canonical tree. +// No receiver-derived cache or slot word is created: the result IS a shape. + +/// `packed` is a code-image byte string of NUL-terminated property names. +/// Codegen declines names containing NUL. Empty names are preserved. +#[no_mangle] +pub unsafe extern "C" fn js_canonical_read_shape(packed: *const u8, len: u32) -> u32 { + let bytes = std::slice::from_raw_parts(packed, len as usize); + assert!(bytes.last() == Some(&0)); + let proof = SharedLayout::shape_cache_entry(); + let scope = crate::gc::RuntimeHandleScope::new(); + let keys_root = scope.root_raw_mut_ptr(std::ptr::null_mut::()); + let mut keys = CanonicalKeys::EMPTY; + for name in bytes[..bytes.len() - 1].split(|b| *b == 0) { + // Match js_build_class_keys_array: canonical prefixes are longlived, + // so their initialization must not create unremembered old-to-young + // edges. Redundant strings on trie hits remain collectible at full GC. + let (key, parent) = keys_root.across_mut(|| { + crate::string::js_string_from_bytes_longlived(name.as_ptr(), name.len() as u32) + }); + keys = extend_key(&proof, CanonicalKeys(parent), key); + keys_root.set_raw_mut_ptr(keys.as_ptr()); + } + // The external-carrier bit roots the authoritative shape keys just as it + // does for a class expectation. IDs/layout survive GC; addresses do not. + super::shapes::js_object_shape_id_for_keys(keys.addr() as u64, keys.len()) +} + +#[cfg(feature = "keepalive-anchors")] +#[used(compiler)] +static KEEP_READ_SHAPE: unsafe extern "C" fn(*const u8, u32) -> u32 = js_canonical_read_shape; + + +/// A keys array this table owns. +/// +/// The enforcement half of the funnel, and the reason it is a type rather +/// than a comment: only this module can mint one, every producer of a keys +/// array must accept one, and it exposes no writable element pointer. So "a +/// producer forgot to canonicalize" is a compile error rather than a silent +/// duplicate layout, and "a mutator edited a shared array in place" is a +/// compile error rather than one object's key edit appearing in every object +/// of that layout. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub(crate) struct CanonicalKeys(*mut ArrayHeader); + +impl CanonicalKeys { + /// The empty ordered key list. A keyless shape's `keys` fact is 0, so the + /// root of the trie owns no array and costs nothing. + pub(crate) const EMPTY: CanonicalKeys = CanonicalKeys(std::ptr::null_mut()); + + #[inline] + pub(crate) fn as_ptr(self) -> *mut ArrayHeader { + self.0 + } + + #[inline] + pub(crate) fn as_const_ptr(self) -> *const ArrayHeader { + self.0 as *const ArrayHeader + } + + #[inline] + pub(crate) fn addr(self) -> usize { + self.0 as usize + } + + #[inline] + pub(crate) fn is_empty(self) -> bool { + self.0.is_null() + } + + /// The live key count, which equals the array's `length` by construction: + /// a canonical array is exactly as long as the list it names, because a + /// prefix is its own node. + /// + /// # Safety + /// The table prunes a dead address before anything can read it, so a + /// handle in hand names a live array. + #[inline] + pub(crate) fn len(self) -> u32 { + if self.0.is_null() { + 0 + } else { + unsafe { (*self.0).length } + } + } +} + +/// A live object pointer that a call which may collect has to hand back. +/// +/// **Not `Copy` and not `Clone`**, which is the whole mechanism: a function +/// that can allocate takes this BY VALUE and returns the post-collection one, +/// so a caller that keeps using its old binding is a *move-after-use* error +/// at compile time rather than a receiver written at a stale address at run +/// time. +/// +/// It exists because I wrote `CanonicalKeys` to make "a producer forgot to +/// canonicalize" impossible and then, four hours later, introduced the twin +/// defect myself: `canonicalize` made `shape_cache_insert` allocate, and +/// `js_object_alloc_class_with_keys` held the object under construction as a +/// raw pointer across it, so a collection there moved the receiver and the +/// keys edge landed in the freed address. Lane 16b predicted exactly this +/// class at `delete_rest.rs:412` and noted that step 2.5 is what makes the +/// path allocate — the prediction and its confirmation are four hours apart. +/// +/// Nobody can enumerate the members of this class by inspection; that is the +/// argument for a type rather than three hand-rooted call sites. +/// +/// `none()` is for a caller that has no unrooted object to carry — either +/// none exists yet, or it is already held in a `RuntimeHandleScope`, which is +/// the older discipline and is what `js_object_alloc_with_shape` does. +pub(crate) struct LiveObject(*mut crate::object::ObjectHeader); + +impl LiveObject { + #[inline] + pub(crate) fn new(obj: *mut crate::object::ObjectHeader) -> Self { + LiveObject(obj) + } + + /// No unrooted object crosses this call. + #[inline] + pub(crate) fn none() -> Self { + LiveObject(std::ptr::null_mut()) + } + + /// The raw pointer, at a leaf that does not allocate. + #[inline] + pub(crate) fn as_ptr(&self) -> *mut crate::object::ObjectHeader { + self.0 + } + + /// Run `f` with this object rooted, and return the token carrying its + /// post-collection address. Every allocating callee that accepts a + /// `LiveObject` funnels through here. + pub(crate) fn across(self, f: impl FnOnce() -> R) -> (Self, R) { + if self.0.is_null() { + let out = f(); + return (self, out); + } + let scope = crate::gc::RuntimeHandleScope::new(); + let handle = scope.root_raw_mut_ptr(self.0); + let (out, obj) = handle.across_mut(f); + (LiveObject(obj), out) + } +} + +/// Proof that a key list describes a **shared layout**, not one receiver's +/// private list — the receiver side of `CanonicalKeys`. +/// +/// `canonicalize` and `extend_slot` REQUIRE one, so interning cannot run without +/// it. That is deliberately a type and not an ordering: stage 1a added +/// `ShapeObjectKind::Dictionary` precisely so a latched receiver would +/// decline by construction, and stage 1b then added a path that never asked +/// — three canonicalization calls sat ABOVE every `is_dictionary` guard, so +/// a latched receiver had its private list interned and republished as a +/// shared array and lost 407 of 8,192 keys. Hoisting those three guards +/// would have fixed those three sites and left the fourth, next month, to +/// someone as careful as I was. +/// +/// Nobody holds the membership of this class in their head, including the +/// author of the rule. So the rule is a constructor. +pub(crate) struct SharedLayout { + /// True only for the shape cache's own entry. DISCRIMINATOR (#9754): + /// `scan_shape_cache_roots_mut` states that the shape cache's keys arrays + /// live in the LONGLIVED arena. If that assumption is confined to the + /// cache, longlived HERE and ordinary on the grow path is enough. + shape_cache: bool, +} + +impl SharedLayout { + /// The kind check, and the only way a receiver yields the proof. + /// `None` for a dictionary receiver: its keys are its own, it appends + /// them in place, and interning them would publish one object's private + /// list as every object's layout. + /// + /// # Safety + /// `obj` is a live object header, or null. + #[inline] + pub(crate) unsafe fn of_receiver(obj: *mut crate::object::ObjectHeader) -> Option { + if obj.is_null() || crate::object::dictionary::is_dictionary(obj) { + return None; + } + Some(SharedLayout { shape_cache: false }) + } + + /// The shape cache's entries are shared layouts by construction: they are + /// keyed by a STATIC shape id and handed to every receiver of that shape, + /// so there is no receiver whose kind could make them private. The one + /// place the proof is not a kind check, named so it is auditable rather + /// than implicit. + #[inline] + pub(crate) fn shape_cache_entry() -> Self { + SharedLayout { shape_cache: true } + } +} + +const NO_NODE: u32 = u32::MAX; +/// The empty list. Never freed, owns no array. +const ROOT_NODE: u32 = 0; + +struct Node { + /// The canonical array. 0 for the root and for a free slot. + addr: usize, + parent: u32, + edge_hash: u64, + /// Next node in this node's `(parent, edge_hash)` bucket, or the free-list + /// link when the slot is free. + next: u32, + len: u32, + /// Every slot of this list is a heap string POINTER. + /// + /// Carried because it decides which allocator the child uses, and + /// getting that wrong is expensive in a way no test would show: an array + /// built with the raw-f64 allocator and repaired by + /// `rebuild_array_layout_from_slots` gets a per-object side mask, and ONE + /// live entry in that table arms `PERRY_PER_OBJECT_LAYOUTS_ANY` for the + /// whole program — the address-filter probe on every later allocation, + /// measured at 3% (`alloc.rs`'s #7510 note). A child's answer is its + /// parent's AND the appended slot's, so it costs one bit and no walk. + all_ptr: bool, +} + +/// The canonical trie for one agent. +pub(crate) struct CanonicalTable { + nodes: Vec, + free: u32, + /// Canonical array address -> node id. + by_addr: HashMap, + /// `(parent node, appended-slot hash)` -> first candidate node. + edges: HashMap<(u32, u64), u32>, + minted: u64, + reaped: u64, +} + +impl CanonicalTable { + fn new() -> Self { + Self { + nodes: vec![Node { + addr: 0, + parent: NO_NODE, + edge_hash: 0, + next: NO_NODE, + len: 0, + // The empty list is vacuously all-pointer, which is what makes + // a one-key list's answer just "is this key a heap string". + all_ptr: true, + }], + free: NO_NODE, + by_addr: HashMap::new(), + edges: HashMap::new(), + minted: 0, + reaped: 0, + } + } + + fn alloc_node( + &mut self, + addr: usize, + parent: u32, + edge_hash: u64, + len: u32, + all_ptr: bool, + ) -> u32 { + let fresh = Node { + addr, + parent, + edge_hash, + next: NO_NODE, + len, + all_ptr, + }; + let id = if self.free != NO_NODE { + let id = self.free; + self.free = self.nodes[id as usize].next; + self.nodes[id as usize] = fresh; + id + } else { + self.nodes.push(fresh); + (self.nodes.len() - 1) as u32 + }; + self.minted += 1; + CANON_MINTED.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + CANON_LIVE.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + CANON_WORDS.fetch_add(u64::from(len), std::sync::atomic::Ordering::Relaxed); + self.by_addr.insert(addr, id); + if parent != NO_NODE { + let head = self.edges.entry((parent, edge_hash)).or_insert(NO_NODE); + self.nodes[id as usize].next = *head; + *head = id; + } + id + } + + /// Unlink `id`, orphan its children and return its slot to the free list. + /// The children keep their arrays and stay adoptable; they merely stop + /// being reachable from the root. See the module note on orphans. + fn free_node(&mut self, id: u32) { + if id == ROOT_NODE || id as usize >= self.nodes.len() { + return; + } + let (parent, edge_hash, addr, next_sib, freed_len) = { + let n = &self.nodes[id as usize]; + (n.parent, n.edge_hash, n.addr, n.next, n.len) + }; + if addr == 0 { + return; + } + if parent != NO_NODE { + if let Some(&head) = self.edges.get(&(parent, edge_hash)) { + if head == id { + if next_sib == NO_NODE { + self.edges.remove(&(parent, edge_hash)); + } else { + self.edges.insert((parent, edge_hash), next_sib); + } + } else { + let mut cur = head; + while cur != NO_NODE { + let n = self.nodes[cur as usize].next; + if n == id { + self.nodes[cur as usize].next = next_sib; + break; + } + cur = n; + } + } + } + } + self.by_addr.remove(&addr); + // A freed slot may be reused by an unrelated node, so nothing may keep + // pointing at this id: detach every child bucket first. + let mut orphans: Vec = Vec::new(); + for (key, head) in self.edges.iter() { + if key.0 == id { + let mut cur = *head; + while cur != NO_NODE { + orphans.push(cur); + cur = self.nodes[cur as usize].next; + } + } + } + self.edges.retain(|key, _| key.0 != id); + for child in orphans { + self.nodes[child as usize].parent = NO_NODE; + self.nodes[child as usize].next = NO_NODE; + } + self.nodes[id as usize] = Node { + addr: 0, + parent: NO_NODE, + edge_hash: 0, + next: self.free, + len: 0, + all_ptr: true, + }; + self.free = id; + self.reaped += 1; + CANON_REAPED.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + CANON_LIVE.fetch_sub(1, std::sync::atomic::Ordering::Relaxed); + CANON_WORDS.fetch_sub(u64::from(freed_len), std::sync::atomic::Ordering::Relaxed); + } +} + +crate::perry_thread_local! { + static CANONICAL_KEYS: RefCell = RefCell::new(CanonicalTable::new()); +} + +/// Diagnostic totals across every agent thread. +/// +/// PROCESS-global on purpose, where the table is thread-local: user code runs +/// on a dedicated thread and the mint census reports from another, so reading +/// the reporting thread's (empty) table printed `live 0` for a program that +/// had just built a full trie. A census that reports on the wrong thread is +/// the same defect as one that measures nothing. Nothing branches on these, +/// and no test asserts them, so they are plain statics rather than +/// `per_test_global!` sinks. +static CANON_LIVE: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); +static CANON_MINTED: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); +static CANON_REAPED: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); +/// Element words held by live canonical arrays — the side-table bytes this +/// stage is measured on, times eight. +static CANON_WORDS: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + +/// Every access goes through `try_with`, never `with`. +/// +/// A thread being torn down has already dropped its thread-locals, and `with` +/// PANICS there ("cannot access a Thread Local Storage value during or after +/// destruction"). Both of this module's callers can run on a dying thread — +/// the collector's root scan and dead-key prune, and the mint census, which +/// reports at process exit — so the panic was reachable, and was OBSERVED on +/// the first measured run before this was written. A table that is gone holds +/// no canonical arrays, so `None` is not a lost answer; it is the right one. +fn try_with_table(f: impl FnOnce(&mut CanonicalTable) -> R) -> Option { + CANONICAL_KEYS.try_with(|t| f(&mut t.borrow_mut())).ok() +} + +/// [`try_with_table`] for a caller that must produce a value. `default` is +/// what a torn-down table means for that caller, stated at the call site. +fn with_table_or(default: R, f: impl FnOnce(&mut CanonicalTable) -> R) -> R { + try_with_table(f).unwrap_or(default) +} + +/// The one slot an `extend_slot` appends, in whichever form the caller has it. +/// +/// Two constructors, one path: both produce the same edge hash for the same +/// key bytes and both validate by BYTES, so a grow (which holds an interned +/// key string) and a rebuild (which holds a stored slot) reach the same node. +#[derive(Clone, Copy)] +pub(crate) enum Appended { + /// An incoming interned key string, as the grow path holds it. + Key(*const StringHeader), + /// A slot read out of a keys array — a key, a tombstone or a symbol. + Slot(JSValue), +} + +impl Appended { + /// # Safety + /// The operand is live. + unsafe fn edge_hash(self) -> u64 { + match self { + Appended::Key(key) => { + if key.is_null() { + return 0x4E55_4C4C_4B45_5900; + } + let data = (key as *const u8).add(std::mem::size_of::()); + crate::object::keys_lookup::key_bytes_hash(data, (*key).byte_len as usize) + } + Appended::Slot(v) => { + let mut sso = [0u8; crate::value::SHORT_STRING_MAX_LEN]; + match crate::string::js_string_key_bytes(v, &mut sso) { + Some(bytes) => { + crate::object::keys_lookup::key_bytes_hash(bytes.as_ptr(), bytes.len()) + } + // A hole or a symbol is part of the ordered list and must + // participate, or `{a, }` and `{a, b}` share an edge + // — the ORDER/tombstone-POSITION merge that L8.3.15's + // must-fail control exists to catch. + None => v.bits() ^ 0x9E37_79B9_7F4A_7C15, + } + } + } + } + + /// Does `stored` — the last slot of a candidate child — name this key? + /// The hash is never trusted on its own: a wrong array would be a wrong + /// key list for every object of that layout. + /// + /// # Safety + /// Both operands are live. + unsafe fn matches(self, stored: JSValue) -> bool { + match self { + // A null key never matches by BYTES, so without this it would + // miss its own node on every probe and mint an array per call — + // an unbounded source of exactly the duplicates this module + // exists to remove. Callers guard against null earlier; this + // makes the guard's absence a shared node rather than a leak. + Appended::Key(key) if key.is_null() => { + stored.bits() == crate::value::js_nanbox_string(0).to_bits() + } + Appended::Key(key) => crate::string::js_string_key_matches(stored, key), + Appended::Slot(v) => { + if stored.bits() == v.bits() { + return true; + } + let mut sa = [0u8; crate::value::SHORT_STRING_MAX_LEN]; + let mut sb = [0u8; crate::value::SHORT_STRING_MAX_LEN]; + let a = crate::string::js_string_key_bytes(stored, &mut sa); + let b = crate::string::js_string_key_bytes(v, &mut sb); + match (a, b) { + (Some(x), Some(y)) => x == y, + // A non-string pair that is not bit-equal is a different + // list; the bit compare above already accepted an + // identical hole. + _ => false, + } + } + } + } + + /// The raw element word to write into the appended slot. + /// + /// # Safety + /// The operand is live. + unsafe fn element_word(self) -> f64 { + match self { + Appended::Key(key) => crate::value::js_nanbox_string(key as i64), + Appended::Slot(v) => f64::from_bits(v.bits()), + } + } + + /// Is the appended slot a heap string POINTER? An SSO short string and a + /// tombstone are not, and either one costs the child its all-pointer + /// layout — see `Node::all_ptr`. + #[inline] + fn is_pointer(self) -> bool { + match self { + Appended::Key(key) => !key.is_null(), + Appended::Slot(v) => v.is_string(), + } + } +} + +/// The node id of a canonical handle, or `None` when a death prune dropped it. +fn node_of(t: &CanonicalTable, keys: CanonicalKeys) -> Option { + if keys.is_empty() { + return Some(ROOT_NODE); + } + t.by_addr.get(&keys.addr()).copied() +} + +/// Probe the trie for `parent + appended`, validating the appended slot +/// exactly. +/// +/// # Safety +/// `parent` names a live canonical array (or is empty) and `appended` is live. +unsafe fn probe( + parent: CanonicalKeys, + parent_len: u32, + appended: Appended, + h: u64, +) -> Option { + with_table_or(None, |t| { + let pnode = node_of(t, parent)?; + let mut cur = *t.edges.get(&(pnode, h))?; + while cur != NO_NODE { + let (addr, len, next) = { + let node = &t.nodes[cur as usize]; + (node.addr, node.len, node.next) + }; + if addr != 0 && len == parent_len + 1 { + let arr = addr as *const ArrayHeader; + // Keep the cache entry's exact-length check before resolving + // slots: a forwarded stub is not a canonical array to return. + if (*arr).length == parent_len + 1 { + let (slots, slot_len) = crate::object::keys_array_dense_slots(arr); + if parent_len as usize >= slot_len { + cur = next; + continue; + } + #[cfg(test)] + canonical_keys_tests::note_slot_read(); + // Compare the internal key slot, not JavaScript Get: + // Get translates a tombstone into undefined (or a + // prototype value), which would miss the same trie edge. + let stored = JSValue::from_bits((*slots.add(parent_len as usize)).to_bits()); + if appended.matches(stored) { + return Some(CanonicalKeys(addr as *mut ArrayHeader)); + } + } + } + cur = next; + } + None + }) +} + +/// Stamp the invariant every canonical array carries: it is shared from +/// birth, so copy-on-write is the only append path rather than the fallback +/// one (L8.3.15c). +/// +/// # Safety +/// `arr` is a live, tracked array. +unsafe fn stamp_shared(arr: *mut ArrayHeader) { + let gc_header = crate::value::addr_class::try_read_tracked_gc_header(arr as usize) + .expect("a canonical array must be a tracked GC allocation"); + (*gc_header.as_ptr()).gc_flags |= crate::gc::GC_FLAG_SHAPE_SHARED; +} + +/// The canonical array for `parent`'s ordered key list with one slot +/// appended. O(1): one hash probe and one exact slot check; allocates only on +/// a genuinely new layout. +/// +/// # Safety +/// `parent` names a live canonical array (or is empty), `appended` is live, +/// and the caller has rooted everything it holds across the allocation this +/// may perform. +pub(crate) unsafe fn extend_slot( + _proof: &SharedLayout, + parent: CanonicalKeys, + appended: Appended, +) -> CanonicalKeys { + let h = appended.edge_hash(); + let parent_len = parent.len(); + if let Some(hit) = probe(parent, parent_len, appended, h) { + return hit; + } + + // Whether the child's slots are all heap string pointers is the parent's + // answer AND this slot's, so it is read before the allocation and never + // re-derived by walking the list. + // A torn-down table cannot promise an all-pointer layout, so `false` is + // the safe default: the mask path is correct for any content. + let all_ptr = with_table_or(false, |t| { + node_of(t, parent) + .map(|id| t.nodes[id as usize].all_ptr) + .unwrap_or(false) + }) && appended.is_pointer(); + + // Nothing may be held across the allocation: no table borrow (a collection + // re-enters this table through its scanner and its prune) and both + // operands rooted, because a collection here moves them. + let scope = crate::gc::RuntimeHandleScope::new(); + let parent_handle = scope.root_raw_mut_ptr(parent.0); + let appended_handle = match appended { + Appended::Key(key) => scope.root_string_ptr(key), + Appended::Slot(v) => scope.root_nanbox_u64(v.bits()), + }; + // The all-pointer allocator, when the claim holds: it declares the layout + // in the header, where `rebuild_array_layout_from_slots` would install a + // per-object side mask — and one live mask entry arms the address filter + // for the whole program (`Node::all_ptr`). + // LONGLIVED, and still load-bearing — but NOT for the reason the first + // version of this comment gave. + // + // It was introduced because `CLASS_KEYS_BY_ID` held the shape cache's + // keys array as an unscanned raw `usize`. That table is scanned and + // pruned now (`alloc.rs`), and scanning it DID buy what was predicted: + // with ordinary allocation, all seven of the GC tests that assert a keys + // array moves, is rewritten, and is reclaimed pass UNCHANGED, and the + // suite's failures fall 12 -> 6. + // + // But `descriptor_trap_collection_preserves_for_in_target_and_keys` + // returns 1 key of 14 again the moment allocation goes back to the + // nursery. So `CLASS_KEYS_BY_ID` was NOT the holder — or not the only + // one — and the earlier root cause was a site that had been localised, + // not explained. Same error as L8.3.21, caught the same way: by the fix + // failing to fix it. + // + // A wrong answer must not ship, so the allocator stays until the real + // holder is found. The cost is the retention regression L8.3.15c was + // withdrawn for, which seven tests name precisely and which is bounded + // (135 KB on tsc; majors still reclaim). + let allocate = || { + if _proof.shape_cache { + let a = crate::array::js_array_alloc_with_length_longlived(parent_len + 1); + (*a).length = 0; + if all_ptr { + crate::gc::layout_init_all_pointer_slots(a as *mut u8); + } + a + } else if all_ptr { + crate::array::js_array_alloc_pointer_elements(parent_len + 1) + } else { + crate::array::js_array_alloc(parent_len + 1) + } + }; + // Reload both operands only after the child allocation can no longer + // move them. No GC allocation occurs while the fresh array is filled. + let ((fresh, parent), appended) = match appended { + Appended::Key(_) => { + let (result, key) = appended_handle.across_const::(|| { + parent_handle.across_mut::(allocate) + }); + (result, Appended::Key(key)) + } + Appended::Slot(_) => { + let (result, slot) = appended_handle + .across_nanbox(|| parent_handle.across_mut::(allocate)); + (result, Appended::Slot(JSValue::from_bits(slot.to_bits()))) + } + }; + let parent = CanonicalKeys(parent); + + // A collection during the allocation may have published this exact node + // through another path, or pruned the parent. Re-probe before writing. + if let Some(hit) = probe(parent, parent_len, appended, h) { + return hit; + } + + // #10939/#10948: all three former clone-before-append sites funnel here. + // Resolve grow-forward pointers and front reserves on the source, and use + // the destination's element base. Publishing a shape count beyond the + // initialized prefix would make the collector trace uninitialized words. + let (src, src_len) = crate::object::keys_array_dense_slots(parent.as_const_ptr()); + let copied = (parent_len as usize).min(src_len); + debug_assert_eq!( + copied, parent_len as usize, + "the shape's key count outruns its keys array" + ); + let parent_len = copied as u32; + let dst = crate::array::array_elements_ptr(fresh as *const ArrayHeader) as *mut f64; + if parent_len > 0 { + for i in 0..copied { + // GC_STORE_AUDIT(INIT): `fresh` is unpublished, and its length — + // which bounds every collector view of it — is set only after the + // last slot is written. + *dst.add(i) = *src.add(i); + } + } + *dst.add(parent_len as usize) = appended.element_word(); + (*fresh).length = parent_len + 1; + if !all_ptr { + // Only the mixed list needs the slot walk; the all-pointer allocator + // already declared its layout in the header, and publishing `length` + // after the last write is the precondition it documents. + crate::object::gc_slots::rebuild_array_layout_from_slots(fresh); + } + stamp_shared(fresh); + + try_with_table(|t| { + // The parent may have been pruned while we allocated. Keep the array — + // it is a correct list — as an orphan root: the caller still gets the + // right content and only the edge is lost. + let pnode = node_of(t, parent).unwrap_or(NO_NODE); + t.alloc_node(fresh as usize, pnode, h, parent_len + 1, all_ptr); + }); + CanonicalKeys(fresh) +} + +/// [`extend_slot`] with an incoming interned key string — the grow path's form. +/// +/// # Safety +/// As [`extend_slot`]. +#[inline] +pub(crate) unsafe fn extend_key( + proof: &SharedLayout, + parent: CanonicalKeys, + key: *const StringHeader, +) -> CanonicalKeys { + extend_slot(proof, parent, Appended::Key(key)) +} + +/// The canonical array for the ordered key list held in `keys[0..len]`. +/// +/// The funnel's total form: whatever a producer hands in — a private clone, a +/// cache entry, a freshly built list — what comes back is THE array for that +/// content. When `keys` is already canonical at that length it is returned +/// unchanged, which is every call after the first. +/// +/// # Safety +/// `keys` is a live keys array (or null) with at least `len` initialized +/// slots, and the caller has rooted what it holds: this allocates. +pub(crate) unsafe fn canonicalize( + proof: &SharedLayout, + keys: *const ArrayHeader, + len: u32, +) -> CanonicalKeys { + if keys.is_null() || len == 0 { + return CanonicalKeys::EMPTY; + } + let already = with_table_or(false, |t| { + t.by_addr + .get(&(keys as usize)) + .copied() + .map(|id| t.nodes[id as usize].len == len) + .unwrap_or(false) + }); + if already { + return CanonicalKeys(keys as *mut ArrayHeader); + } + + // Fold `extend_slot` over the slots. One path, N times — the source is re-read + // after every step, because `extend_slot` allocates and the collector moves it. + let scope = crate::gc::RuntimeHandleScope::new(); + let src = scope.root_raw_mut_ptr(keys as *mut ArrayHeader); + let mut out = CanonicalKeys::EMPTY; + for i in 0..len { + let slot = src.with_const_ptr(|arr| { + // Re-resolve grow-forward pointers and front reserves after each + // allocating extension; borrow slots only for this read. + let (slots, src_len) = crate::object::keys_array_dense_slots(arr); + let copied = (len as usize).min(src_len); + debug_assert_eq!( + copied, len as usize, + "the shape's key count outruns its keys array" + ); + ((i as usize) < copied).then(|| JSValue::from_bits((*slots.add(i as usize)).to_bits())) + }); + let Some(slot) = slot else { break }; + out = extend_slot(proof, out, Appended::Slot(slot)); + } + out +} + +/// GC root scanner. The arrays are WEAK: rewritten on move, never marked — +/// see the module note on #6759 phase 3. +pub fn scan_canonical_keys_roots_mut(visitor: &mut crate::gc::RuntimeRootVisitor<'_>) { + let young = visitor.young_scope(); + let mut moved: Vec<(usize, usize, u32)> = Vec::new(); + let _ = CANONICAL_KEYS.try_with(|t| { + let mut t = t.borrow_mut(); + for id in 1..t.nodes.len() { + let addr = t.nodes[id].addr; + if addr == 0 { + continue; + } + // A minor can only move a young address, and the node table is one + // contiguous `Vec` — so the address check is cheaper here than a + // fourth young log would be. + if young && !crate::gc::young_log::addr_is_minor_relevant(addr) { + continue; + } + let mut next = addr; + if visitor.visit_metadata_usize_slot(&mut next) && next != addr { + t.nodes[id].addr = next; + moved.push((addr, next, id as u32)); + } + } + for (old, new, id) in moved.drain(..) { + t.by_addr.remove(&old); + t.by_addr.insert(new, id); + } + }); +} + +/// Post-trace death prune. A node whose array did not survive is dropped; its +/// children are orphaned rather than followed, which costs at most one +/// duplicate layout and never a wrong list. +#[cold] +pub(crate) fn prune_dead_canonical_keys(is_dead_owner: &dyn Fn(usize) -> bool) { + // Snapshot first, then ask. `is_dead_owner` is a collector predicate this + // module does not own, and calling it under the table borrow would make a + // future re-entrant reader a panic rather than a slow path. + let live: Vec<(u32, usize)> = CANONICAL_KEYS + .try_with(|t| { + let t = t.borrow(); + (1..t.nodes.len()) + .filter(|&id| t.nodes[id].addr != 0) + .map(|id| (id as u32, t.nodes[id].addr)) + .collect() + }) + .unwrap_or_default(); + let dead: Vec = live + .into_iter() + .filter(|&(_, addr)| is_dead_owner(addr) || canonical_address_is_recycled(addr)) + .map(|(id, _)| id) + .collect(); + if dead.is_empty() { + return; + } + try_with_table(|t| { + for id in dead { + t.free_node(id); + } + }); +} + +/// An address is only a keys array while the cell at it still IS an array: +/// the arena recycles addresses, and a recycled tenant answers "alive" to +/// `is_dead_owner`. Twin of `shape_keys_address_is_recycled`. +fn canonical_address_is_recycled(addr: usize) -> bool { + // SAFETY: a read-only tracked-header probe, which tolerates an address + // that is no longer a tracked cell. + unsafe { + match crate::value::addr_class::try_read_tracked_gc_header(addr) { + Some(gc) => { + let ty = (*gc.as_ptr()).obj_type; + ty != crate::gc::GC_TYPE_ARRAY && ty != crate::gc::GC_TYPE_LAZY_ARRAY + } + None => true, + } + } +} + +/// `(live nodes, nodes ever minted, nodes reaped)`. The census reads this; +/// nothing branches on it. +pub(crate) fn canonical_stats() -> (usize, u64, u64) { + ( + CANON_LIVE.load(std::sync::atomic::Ordering::Relaxed) as usize, + CANON_MINTED.load(std::sync::atomic::Ordering::Relaxed), + CANON_REAPED.load(std::sync::atomic::Ordering::Relaxed), + ) +} + +/// Total element words held by live canonical arrays — the side-table bytes +/// this stage is measured on. +pub(crate) fn canonical_element_words() -> u64 { + CANON_WORDS.load(std::sync::atomic::Ordering::Relaxed) +} + +#[cfg(test)] +pub(crate) fn reset_for_test() { + let _ = CANONICAL_KEYS.try_with(|t| *t.borrow_mut() = CanonicalTable::new()); +} + +#[cfg(test)] +#[path = "canonical_keys_tests.rs"] +mod canonical_keys_tests; + +#[cfg(test)] +#[path = "canonical_read_shape_tests.rs"] +mod canonical_read_shape_tests; diff --git a/crates/perry-runtime/src/object/canonical_keys_tests.rs b/crates/perry-runtime/src/object/canonical_keys_tests.rs new file mode 100644 index 0000000000..74ea4fc4b3 --- /dev/null +++ b/crates/perry-runtime/src/object/canonical_keys_tests.rs @@ -0,0 +1,231 @@ +use super::*; + +// Test instrumentation only: one count per candidate slot read by probe(). +// No declaration, read or branch survives in a production build. +crate::perry_thread_local! { + static SLOT_READS: std::cell::Cell = const { std::cell::Cell::new(0) }; +} + +pub(super) fn note_slot_read() { + SLOT_READS.with(|c| c.set(c.get() + 1)); +} + +fn key(name: &str) -> *mut StringHeader { + crate::string::js_string_from_bytes(name.as_ptr(), name.len() as u32) +} + +/// A keys array built the way a producer that has NOT been funnelled +/// would build one: its own allocation, its own address. +unsafe fn raw_list(names: &[&str]) -> *mut ArrayHeader { + let scope = crate::gc::RuntimeHandleScope::new(); + let arr = scope.root_raw_mut_ptr(crate::array::js_array_alloc(names.len() as u32 + 4)); + let (_, result) = arr.across_mut(|| { + for name in names { + let k = key(name); + let grown = arr.with_mut_ptr(|a| { + crate::array::js_array_push_f64(a, crate::value::js_nanbox_string(k as i64)) + }); + arr.set_raw_mut_ptr(grown); + } + }); + result +} + +/// The whole of stage 1b in one assertion: separately allocated arrays +/// holding the same ordered list come back as ONE address, which is what +/// makes `facts_key`'s address term a content term with no edit to +/// `facts_key` at all. +#[test] +fn one_array_serves_one_ordered_key_list() { + let _lock = crate::gc::global_side_table_test_lock(); + reset_for_test(); + unsafe { + let a = raw_list(&["alpha", "beta", "gamma"]); + let b = raw_list(&["alpha", "beta", "gamma"]); + assert_ne!(a, b, "test premise: two separately allocated arrays"); + let proof = SharedLayout::shape_cache_entry(); + let ca = canonicalize(&proof, a, 3); + let cb = canonicalize(&proof, b, 3); + assert_eq!( + ca.addr(), + cb.addr(), + "same ordered list, two arrays -- canonicalization must give one address" + ); + assert_eq!(ca.len(), 3); + // And the grow path reaches the same node as the whole-list form. + let grown = extend_key( + &proof, + extend_key( + &proof, + extend_key(&proof, CanonicalKeys::EMPTY, key("alpha")), + key("beta"), + ), + key("gamma"), + ); + assert_eq!( + grown.addr(), + ca.addr(), + "extend_slot and canonicalize must reach the same node, or they are two paths" + ); + } +} + +/// The must-fail control of L8.3.15, as a test rather than a promise: a +/// canonicalization that sorted or otherwise reordered keys would be a +/// silent WRONG ANSWER in every program, not a slow one. +#[test] +fn key_order_is_part_of_the_identity() { + let _lock = crate::gc::global_side_table_test_lock(); + reset_for_test(); + unsafe { + let proof = SharedLayout::shape_cache_entry(); + let ab = canonicalize(&proof, raw_list(&["a", "b"]), 2); + let ba = canonicalize(&proof, raw_list(&["b", "a"]), 2); + assert_ne!( + ab.addr(), + ba.addr(), + "{{a,b}} and {{b,a}} are different layouts -- merging them is a wrong answer" + ); + } +} + +/// Per-prefix canonicalization, which is the correction of L8.3.15b: a +/// prefix is its own node, so `{a}` and `{a,b}` never share an address +/// and a `key_count` mint — 19,923 of tsc's 42,097 — cannot arise. +#[test] +fn a_prefix_is_its_own_node() { + let _lock = crate::gc::global_side_table_test_lock(); + reset_for_test(); + unsafe { + let full = raw_list(&["a", "b", "c"]); + let proof = SharedLayout::shape_cache_entry(); + let two = canonicalize(&proof, full, 2); + let three = canonicalize(&proof, full, 3); + assert_eq!( + two.len(), + 2, + "the prefix array is exactly as long as its list" + ); + assert_eq!(three.len(), 3); + assert_ne!(two.addr(), three.addr()); + let one = canonicalize(&proof, full, 1); + assert_eq!(one.len(), 1); + assert_ne!(one.addr(), two.addr()); + } +} + +/// L8.3.15's sixth prediction, measured rather than inspected: a probe +/// reads the ONE appended slot, however long the list already is. A +/// content-hash intern table would read N. +#[test] +fn a_probe_reads_one_slot_however_long_the_list() { + let _lock = crate::gc::global_side_table_test_lock(); + reset_for_test(); + unsafe { + let names: Vec = (0..40).map(|i| format!("k{i}")).collect(); + let refs: Vec<&str> = names.iter().map(|s| s.as_str()).collect(); + let proof = SharedLayout::shape_cache_entry(); + let long = canonicalize(&proof, raw_list(&refs), 40); + assert_eq!(long.len(), 40); + let tail_key = key("k40"); + // Publish the edge, then measure the HIT. + let first = extend_key(&proof, long, tail_key); + SLOT_READS.with(|c| c.set(0)); + let again = extend_key(&proof, long, tail_key); + let reads = SLOT_READS.with(|c| c.get()); + assert_eq!( + first.addr(), + again.addr(), + "the second extend_slot must hit" + ); + assert!( + reads <= 1, + "a hit on a 41-key list read {reads} slots; the probe is not O(1)" + ); + } +} + +/// A tombstone is part of the ordered list, so its POSITION is part of +/// the identity. Two lists that differ only in where the hole sits must +/// not share a node. +#[test] +fn a_tombstone_position_is_part_of_the_identity() { + let _lock = crate::gc::global_side_table_test_lock(); + reset_for_test(); + unsafe { + let hole = JSValue::from_bits(crate::value::TAG_HOLE); + let a = key("a"); + let b = key("b"); + let p = SharedLayout::shape_cache_entry(); + let hole_first = extend_slot( + &p, + extend_slot( + &p, + extend_slot(&p, CanonicalKeys::EMPTY, Appended::Slot(hole)), + Appended::Key(a), + ), + Appended::Key(b), + ); + let hole_middle = extend_slot( + &p, + extend_slot( + &p, + extend_slot(&p, CanonicalKeys::EMPTY, Appended::Key(a)), + Appended::Slot(hole), + ), + Appended::Key(b), + ); + let hole_first_again = extend_slot( + &p, + extend_slot( + &p, + extend_slot(&p, CanonicalKeys::EMPTY, Appended::Slot(hole)), + Appended::Key(a), + ), + Appended::Key(b), + ); + assert_eq!( + hole_first.addr(), + hole_first_again.addr(), + "the same tombstone list must hit the same canonical node" + ); + let raw = crate::array::js_array_alloc(3); + let raw = crate::array::js_array_push(raw, hole); + let raw = crate::array::js_array_push(raw, JSValue::string_ptr(a as *mut _)); + let raw = crate::array::js_array_push(raw, JSValue::string_ptr(b as *mut _)); + assert_eq!( + canonicalize(&p, raw, 3).addr(), + hole_first.addr(), + "canonicalizing raw keys must preserve tombstones" + ); + assert_eq!(hole_first.len(), 3); + assert_eq!(hole_middle.len(), 3); + assert_ne!( + hole_first.addr(), + hole_middle.addr(), + "the tombstone position distinguishes two layouts" + ); + } +} + +/// Every canonical array is shared from birth (L8.3.15c), which is what +/// collapses every `keys_owned` branch to the shared arm rather than +/// leaving an arm that is merely unreached. +#[test] +fn every_canonical_array_is_shape_shared_from_birth() { + let _lock = crate::gc::global_side_table_test_lock(); + reset_for_test(); + unsafe { + let c = extend_key( + &SharedLayout::shape_cache_entry(), + CanonicalKeys::EMPTY, + key("only"), + ); + let gc = crate::value::addr_class::try_read_tracked_gc_header(c.addr()) + .expect("canonical keys must have a tracked header"); + assert!( + (*gc.as_ptr()).gc_flags & crate::gc::GC_FLAG_SHAPE_SHARED != 0, + "a canonical array that is not SHAPE_SHARED can be mutated in place" + ); + } +} diff --git a/crates/perry-runtime/src/object/canonical_read_shape_tests.rs b/crates/perry-runtime/src/object/canonical_read_shape_tests.rs new file mode 100644 index 0000000000..e2c4561201 --- /dev/null +++ b/crates/perry-runtime/src/object/canonical_read_shape_tests.rs @@ -0,0 +1,88 @@ +use super::*; + +#[test] +fn predicted_canonical_shape_is_shared_by_literal_and_growth_paths() { + let _lock = crate::gc::global_side_table_test_lock(); + unsafe { + let packed = b"licm_a\0licm_b\0"; + let expected = js_canonical_read_shape(packed.as_ptr(), packed.len() as u32); + let keys = crate::object::js_build_class_keys_array( + 987654, + 2, + packed.as_ptr(), + packed.len() as u32, + ); + assert_eq!( + crate::object::shapes::js_object_shape_id_for_keys(keys as u64, 2), + expected + ); + let scope = crate::gc::RuntimeHandleScope::new(); + let object = scope.root_raw_mut_ptr(crate::object::js_object_alloc(0, 0)); + for (name, value) in [(b"licm_a", 1.0), (b"licm_b", 2.0)] { + let key = crate::string::js_string_from_bytes(name.as_ptr(), name.len() as u32); + object.with_mut_ptr(|o| crate::object::js_object_set_field_by_name(o, key, value)); + } + let stamp = object.with_const_ptr(|o| crate::object::shapes::object_shape_stamp(o)); + assert_eq!( + stamp, expected, + "both actual producers must hit the same prediction" + ); + assert_eq!( + crate::object::shapes::js_shape_ordinary_inline_slot_for_key( + expected, + crate::value::js_nanbox_string(crate::string::js_string_from_bytes( + b"licm_b".as_ptr(), + 6 + ) as i64) + .to_bits() + ), + 1 + ); + assert!(object.with_mut_ptr(|o| crate::object::dictionary::latch_object_to_dictionary(o))); + let dictionary_stamp = + object.with_const_ptr(|o| crate::object::shapes::object_shape_stamp(o)); + assert_ne!( + dictionary_stamp, expected, + "dictionary must never enter the canonical loop" + ); + } +} + +#[test] +fn remembered_prediction_roots_its_keys_without_a_receiver() { + let _lock = crate::gc::global_side_table_test_lock(); + unsafe { + let packed = b"remembered_read_key\0"; + let id = js_canonical_read_shape(packed.as_ptr(), packed.len() as u32); + let descriptor = super::super::shapes::shape_descriptor_by_id(id).unwrap(); + assert!( + descriptor.cache_carrier, + "a remembered id needs an external carrier" + ); + let mut roots = Vec::new(); + super::super::shapes::scan_shape_table_rekey_mut( + &mut crate::gc::RuntimeRootVisitor::for_copy(&mut |v| roots.push(v.to_bits())), + ); + assert!( + roots + .iter() + .any(|bits| bits & crate::value::POINTER_MASK == descriptor.keys), + "the shape scanner must strongly visit the keys with no receiver" + ); + super::super::shapes::prune_uncarried_shape_descriptors_after_full_trace(); + assert!( + super::super::shapes::shape_descriptor_by_id(id).is_some(), + "external expectations must survive descriptor retirement" + ); + prune_dead_canonical_keys(&|addr| { + !roots + .iter() + .any(|bits| bits & crate::value::POINTER_MASK == addr as u64) + }); + assert_eq!( + js_canonical_read_shape(packed.as_ptr(), packed.len() as u32), + id, + "weak trie pruning must retain the rooted prediction" + ); + } +} diff --git a/crates/perry-runtime/src/object/dictionary.rs b/crates/perry-runtime/src/object/dictionary.rs index 8a30171ca1..cc97c2d338 100644 --- a/crates/perry-runtime/src/object/dictionary.rs +++ b/crates/perry-runtime/src/object/dictionary.rs @@ -191,10 +191,27 @@ pub(crate) fn test_clear_layout_id_budget() { /// Resolve the knob once. Value-parsed, not presence-parsed: #7991 shipped a /// knob that `PERRY_GC_DIAG=0` turned ON. +/// The compiled-in trigger-1 threshold, armed BY DEFAULT. +/// +/// #10868 step 2.5: per-prefix canonicalization makes a receiver with a key +/// list unique to it allocate one array per prefix — k(k+1)/2 element words. +/// Measured, not projected: 8,192 keys cost 461 MB unlatched and 50 MB +/// latched, and the 65,536-key membership test allocated past 24 GB +/// unlatched and passes in 0.03 s latched. §L8.3.2 wrote this down before +/// either stage existed — "canonical arrays cannot ship ahead of dictionary +/// mode without a cliff" — so the latch is the bound canonical keys stand on, +/// and a bound that is off by default is not a bound. +/// +/// 1,024 keys is ~4 MB of prefix arrays, which is the point the quadratic +/// stops being free. The env var still overrides, in both directions. +const DEFAULT_LATCH_MIN_KEYS: u64 = 1024; + #[cold] #[inline(never)] fn resolve_latch_arming() -> bool { - let mut armed = false; + // Armed by default; the reads below only ADJUST the threshold. + LATCH_MIN_KEYS.store(DEFAULT_LATCH_MIN_KEYS, Ordering::Relaxed); + let mut armed = true; // Trigger 2, injectable. A fixture that really exhausts a 24-bit layout-id // space is impractical, so the budget is a number the allocator PUBLISHES // and anyone can inject — which is the only thing that makes the @@ -262,10 +279,30 @@ pub fn dictionary_counters_line() -> String { ) } +/// The CURRENT arming, resolved — for a scope guard that must restore what +/// it found instead of assuming the default is off. Once #10868 armed the +/// latch by default, "disarm on exit" stopped being "restore on exit". +#[cfg(test)] +pub(crate) fn test_latch_state() -> Option { + if dictionary_latch_armed() { + Some(LATCH_MIN_KEYS.load(Ordering::Relaxed)) + } else { + None + } +} + /// Arm or disarm the latch from a test. Returns the previous minimum, if armed. #[cfg(test)] pub(crate) fn test_arm_latch(min_keys: Option) -> Option { - let was = if LATCH_ARMED.load(Ordering::Relaxed) == 1 { + // RESOLVE before saving. `LATCH_ARMED` starts at -1 = unresolved, and + // reading the raw atomic sees that as "not armed" — so a test that saved + // before anything had queried the latch restored `None`, which STORES 0 + // and disarms it for every later test in the process. Harmless while the + // default was off; fatal once #10868 armed it, because the 65,536-key + // membership test runs later in the same binary and its key list is + // unique to it. It passed standalone and OOM'd in the suite, which is the + // signature of exactly this. + let was = if dictionary_latch_armed() { Some(LATCH_MIN_KEYS.load(Ordering::Relaxed)) } else { None diff --git a/crates/perry-runtime/src/object/dictionary_counters.rs b/crates/perry-runtime/src/object/dictionary_counters.rs index fc964ec451..18999a5c39 100644 --- a/crates/perry-runtime/src/object/dictionary_counters.rs +++ b/crates/perry-runtime/src/object/dictionary_counters.rs @@ -8,10 +8,29 @@ use std::sync::atomic::{AtomicI8, AtomicU64}; -pub(super) static LAYOUT_ID_BUDGET: AtomicU64 = AtomicU64::new(u64::MAX); +// #10868 step 2.5: the LATCH STATE is `per_test_global!`, the counters below +// are not. +// +// The state is what a test MUTATES and a later test READS, which is exactly +// L16.11's class. It became load-bearing when step 2.5 armed the latch by +// default: every test that arms or disarms it now leaves a different value +// behind than it found, and `own_key_membership_crosses_65536_without_a_cutoff` +// — whose key list is unique to it, so it is the k(k+1)/2 cliff — runs later +// in the same binary. It PASSED STANDALONE AND WAS OOM-KILLED IN THE SUITE, +// which is that class's signature. +// +// Three save/restore fixes did not close it, and there were four disarm sites +// to chase. Patching them one at a time is the whack-a-mole `per_test_global!` +// exists to end: per-thread in a test build, the plain `static` outside one, +// so a test cannot reach another test's arming and a new disarm site cannot +// reintroduce the hazard. +per_test_global! { + pub(super) static LATCH_ARMED: AtomicI8 = AtomicI8::new(-1); + pub(super) static LATCH_MIN_KEYS: AtomicU64 = AtomicU64::new(0); + pub(super) static LAYOUT_ID_BUDGET: AtomicU64 = AtomicU64::new(u64::MAX); +} + pub(super) static EXHAUSTION_LATCHES: AtomicU64 = AtomicU64::new(0); -pub(super) static LATCH_ARMED: AtomicI8 = AtomicI8::new(-1); -pub(super) static LATCH_MIN_KEYS: AtomicU64 = AtomicU64::new(0); pub(super) static LATCH_CANDIDATES: AtomicU64 = AtomicU64::new(0); pub(super) static LATCHES: AtomicU64 = AtomicU64::new(0); pub(super) static PUBLICATIONS: AtomicU64 = AtomicU64::new(0); diff --git a/crates/perry-runtime/src/object/dictionary_tests.rs b/crates/perry-runtime/src/object/dictionary_tests.rs index 0efc85f768..04940bed97 100644 --- a/crates/perry-runtime/src/object/dictionary_tests.rs +++ b/crates/perry-runtime/src/object/dictionary_tests.rs @@ -13,16 +13,25 @@ use super::dictionary; use super::{js_object_alloc, js_object_get_field_by_name, js_object_set_field_by_name}; /// Restores the latch arming on scope exit (panic included) so a failing test -/// cannot leak an armed latch into unrelated tests on the same process. +/// cannot leak its arming into unrelated tests on the same process. +/// +/// It RESTORES WHAT IT FOUND. It used to disarm unconditionally, which was +/// the same thing while the default was off — and became the opposite of +/// restoring once #10868 armed the latch by default: every dictionary test +/// then leaked a DISARMED latch forward, and +/// `own_key_membership_crosses_65536_without_a_cutoff`, which runs later in +/// the same binary and whose key list is unique to it, hit the k(k+1)/2 +/// cliff and was OOM-killed. It passed standalone and died in the suite, +/// which is the signature of leaked process-global state (L16.11). fn scopeguard_latch() -> impl Drop { - struct Restore; + struct Restore(Option); impl Drop for Restore { fn drop(&mut self) { - dictionary::test_arm_latch(None); + dictionary::test_arm_latch(self.0); dictionary::test_clear_layout_id_budget(); } } - Restore + Restore(dictionary::test_latch_state()) } unsafe fn set_key(obj: *mut super::ObjectHeader, name: &str, value: f64) { diff --git a/crates/perry-runtime/src/object/field_get_set/field_ops.rs b/crates/perry-runtime/src/object/field_get_set/field_ops.rs index a94163a72e..aae00ebda3 100644 --- a/crates/perry-runtime/src/object/field_get_set/field_ops.rs +++ b/crates/perry-runtime/src/object/field_get_set/field_ops.rs @@ -335,9 +335,31 @@ pub extern "C" fn js_object_set_field_by_index( /// Set the keys array for an object (used for Object.keys() support) /// The keys_array should be an array of string pointers +/// +/// #10868 step 2.5 stage 1b: the list is canonicalized on the way in. The +/// census named this site — 21 of the 24 remaining duplicate-content mints on +/// the same-content fixture came from here — and that is the whole argument +/// for a census over a perf gate: a producer outside the funnel costs a +/// duplicate LAYOUT, which no timing and no parity row can see. #[no_mangle] pub extern "C" fn js_object_set_keys(obj: *mut ObjectHeader, keys_array: *mut ArrayHeader) { unsafe { - set_object_keys_array(obj, keys_array); + let scope = crate::gc::RuntimeHandleScope::new(); + let obj_handle = scope.root_raw_mut_ptr(obj); + let len = if keys_array.is_null() { + 0 + } else { + (*keys_array).length + }; + let proof = crate::object::canonical_keys::SharedLayout::of_receiver(obj); + let (published, obj) = obj_handle.across_mut(|| match proof { + Some(proof) => { + crate::object::canonical_keys::canonicalize(&proof, keys_array, len).as_ptr() + } + // A latched receiver's list is its own; publishing it as a shared + // layout is what lost 407 of 8,192 keys before the proof existed. + None => keys_array, + }); + set_object_keys_array(obj, published); } } diff --git a/crates/perry-runtime/src/object/field_set_by_name/tail.rs b/crates/perry-runtime/src/object/field_set_by_name/tail.rs index 572ee64224..0c738af361 100644 --- a/crates/perry-runtime/src/object/field_set_by_name/tail.rs +++ b/crates/perry-runtime/src/object/field_set_by_name/tail.rs @@ -500,7 +500,6 @@ pub(crate) fn set_field_by_name_object_tail( } } - let mut prev_keys_usize = keys as usize; let prev_shape_id = super::shapes::object_shape_stamp(obj); // FAST PATH: shape-transition cache with interned string pointer identity. @@ -618,11 +617,27 @@ pub(crate) fn set_field_by_name_object_tail( let key_str = key_to_str_for_diag(key); crate::error::throw_immutable_write(1, &key_str); } - // Create a new keys array with the key - let new_keys = crate::array::js_array_alloc(4); - refresh_roots_after_alloc!(); - let new_keys = - crate::array::js_array_push(new_keys, JSValue::string_ptr(key as *mut _)); + // #10868 step 2.5 stage 1b: the one-key list is a canonical node + // like any other. Every `{}` that sets the same first key now + // reaches the SAME array — which is the whole of the mint fix at + // this site, because a fresh 4-slot allocation per receiver was a + // fresh identity per receiver. + let new_keys = match crate::object::canonical_keys::SharedLayout::of_receiver(obj) { + Some(proof) => crate::object::canonical_keys::extend_key( + &proof, + crate::object::canonical_keys::CanonicalKeys::EMPTY, + key, + ) + .as_ptr(), + // A latched receiver owns its list from its first key. + None => { + let fresh = crate::array::js_array_alloc(4); + refresh_roots_after_alloc!(); + let grown = + crate::array::js_array_push(fresh, JSValue::string_ptr(key as *mut _)); + grown + } + }; refresh_roots_after_alloc!(); set_object_keys_array(obj, new_keys); super::mark_object_dynamic_shape_unknown(obj); @@ -796,73 +811,53 @@ pub(crate) fn set_field_by_name_object_tail( // scan is shared. // We achieve this by setting a marker, then the linear // scan checks it and skips. - let keys_gc_header = - (keys as *const u8).sub(crate::gc::GC_HEADER_SIZE) as *const crate::gc::GcHeader; - let keys_shared = if (keys as usize) >= crate::gc::GC_HEADER_SIZE - && (*keys_gc_header).obj_type == crate::gc::GC_TYPE_ARRAY - { - (*keys_gc_header).gc_flags & crate::gc::GC_FLAG_SHAPE_SHARED != 0 - } else { - true - }; - let owned_keys = if keys_shared { - let cloned = crate::array::js_array_alloc(key_count as u32 + 4); - refresh_roots_after_alloc!(); - let keys = crate::object::object_keys_array(obj); - prev_keys_usize = keys as usize; - // #10939: a keys array's elements do not necessarily start - // at `header + 8`. `keys_array_dense_slots` resolves a - // grow-forward pointer and adds `array_front_offset`, which is - // nonzero for any array with a front reserve — #9019's - // reserved-floor keys arrays are BORN with leading holes, and a - // size-class round-up alone can make it nonzero. The clone - // declares every published slot a pointer, so copying from the - // wrong base does not merely read the wrong bytes: it promises - // the collector that `ArrayHeader` and reserve words are heap - // pointers. A missing property now, a SIGSEGV inside the next - // collection later, with a backtrace naming something else. - let (src_data, src_len) = crate::object::keys_array_dense_slots(keys); - let dst_data = - crate::array::array_elements_ptr(cloned as *const crate::array::ArrayHeader); - // A source shorter than the shape's count means the shape is already - // lying; copy what exists rather than publishing uninitialised words - // as traced pointers. - let copied = std::cmp::min(key_count, src_len); - debug_assert_eq!( - copied, key_count, - "the shape's key count outruns its keys array" - ); - for i in 0..copied { - // GC_STORE_AUDIT(INIT): cloned keys array is unpublished; layout is rebuilt before publication. - *dst_data.add(i) = (*src_data.add(i)).to_bits(); + // #10868 step 2.5 stage 1b. The successor is THE canonical array + // for this ordered list plus `key`, so the clone, the ownership + // test and the `+ 4` slack are all gone — and, decisively for the + // mint count, so is the PUBLISHED intermediate. This block used to + // publish the clone and then publish the pushed array, minting + // TWO ShapeIds per grow where the layout changed once: the census + // reads them as tail.rs:809 `fresh_keys_known_list` 2,431 + // followed by tail.rs:825 `key_count` 4,222. + // + // `shape_keys_grown` went with them. It migrated the slot index + // of an OWNED array across an in-place grow; no keys array is + // owned any more, so the arm it served does not exist rather than + // being guarded (L8.3.15c). + let new_index = key_count; + // #10868 step 2.5: TWO MODES, chosen by a fact on the shape — + // which is what dictionary mode IS. This is not a fast path + // beside a slow one inside one mode; an ordinary receiver's key + // list is a SHARED LAYOUT and interns, a latched receiver's list + // is its OWN and appends in place. `SharedLayout::of_receiver` is + // the kind check, and `canonicalize`/`extend_key` cannot be + // called without what it returns. + let new_keys = match crate::object::canonical_keys::SharedLayout::of_receiver(obj) { + Some(proof) => { + let canonical_parent = crate::object::canonical_keys::canonicalize( + &proof, + keys, + key_count as u32, + ); + refresh_roots_after_alloc!(); + crate::object::canonical_keys::extend_key(&proof, canonical_parent, key) + .as_ptr() + } + None => { + // The parent's owned arm, unchanged: a dictionary's array + // carries no `GC_FLAG_SHAPE_SHARED`, so this is the + // in-place O(1)-amortized append that mode exists for. + let owned = scope.root_raw_mut_ptr(keys); + let grown = + crate::array::js_array_push(keys, JSValue::string_ptr(key as *mut _)); + let _ = owned.get_raw_mut_ptr::(); + grown } - (*cloned).length = copied as u32; - super::rebuild_array_layout_from_slots(cloned); - set_object_keys_array(obj, cloned); - cloned - } else { - keys }; - let new_index = key_count; + refresh_roots_after_alloc!(); if new_index >= alloc_limit { - let owned_keys_handle = scope.root_raw_mut_ptr(owned_keys); - let new_keys = - crate::array::js_array_push(owned_keys, JSValue::string_ptr(key as *mut _)); - prev_keys_usize = if keys_shared { - prev_keys_usize - } else { - owned_keys_handle.get_raw_mut_ptr::() as usize - }; - refresh_roots_after_alloc!(); set_object_keys_array(obj, new_keys); super::mark_object_dynamic_shape_unknown(obj); - // #8067: migrate only the owned array's validated slot index; - // the immutable descriptor is versioned to the new facts. A - // shared fork must NOT migrate: the old address still serves - // the siblings' live shape. - if !keys_shared { - super::shapes::shape_keys_grown(prev_keys_usize, new_keys); - } // #7538: derive the stored bits from the REFRESHED `value` — // see the twin below the linear scan. let vbits = overflow_store_bits(value, obj, new_index); @@ -897,22 +892,8 @@ pub(crate) fn set_field_by_name_object_tail( ); return; } - let owned_keys_handle = scope.root_raw_mut_ptr(owned_keys); - let new_keys = - crate::array::js_array_push(owned_keys, JSValue::string_ptr(key as *mut _)); - prev_keys_usize = if keys_shared { - prev_keys_usize - } else { - owned_keys_handle.get_raw_mut_ptr::() as usize - }; - refresh_roots_after_alloc!(); set_object_keys_array(obj, new_keys); super::mark_object_dynamic_shape_unknown(obj); - // #8067: owned grow keeps the slot index, while the immutable - // descriptor is versioned (see the overflow branch above). - if !keys_shared { - super::shapes::shape_keys_grown(prev_keys_usize, new_keys); - } // #7154 publication order: `gc_field_slot_range` bounds the // collector's view of the payload by `field_count`, so a slot at an // index the count does not yet cover is invisible to BOTH tracing @@ -1038,83 +1019,38 @@ pub(crate) fn set_field_by_name_object_tail( // clone entirely. This saves ~19 clones of growing size per // 20-property plain-object literal. // - // Validate the GC header before reading it. `keys_array` has - // already been range-checked for user address space but may - // still point at something other than a GC-allocated array - // in rare cases (static data, buffers re-interpreted as keys - // arrays). If the header doesn't identify as GC_TYPE_ARRAY, - // assume shared and clone (the previous, always-safe behaviour). - let keys_gc_header = - (keys as *const u8).sub(crate::gc::GC_HEADER_SIZE) as *const crate::gc::GcHeader; - let keys_shared = if (keys as usize) >= crate::gc::GC_HEADER_SIZE - && (*keys_gc_header).obj_type == crate::gc::GC_TYPE_ARRAY - { - (*keys_gc_header).gc_flags & crate::gc::GC_FLAG_SHAPE_SHARED != 0 - } else { - // Unknown provenance — take the safe side. - true - }; - let owned_keys = if keys_shared { - let cloned = crate::array::js_array_alloc(key_count as u32 + 4); - refresh_roots_after_alloc!(); - let keys = crate::object::object_keys_array(obj); - prev_keys_usize = keys as usize; - // #10939: a keys array's elements do not necessarily start - // at `header + 8`. `keys_array_dense_slots` resolves a - // grow-forward pointer and adds `array_front_offset`, which is - // nonzero for any array with a front reserve — #9019's - // reserved-floor keys arrays are BORN with leading holes, and a - // size-class round-up alone can make it nonzero. The clone - // declares every published slot a pointer, so copying from the - // wrong base does not merely read the wrong bytes: it promises - // the collector that `ArrayHeader` and reserve words are heap - // pointers. A missing property now, a SIGSEGV inside the next - // collection later, with a backtrace naming something else. - let (src_data, src_len) = crate::object::keys_array_dense_slots(keys); - let dst_data = - crate::array::array_elements_ptr(cloned as *const crate::array::ArrayHeader); - // A source shorter than the shape's count means the shape is already - // lying; copy what exists rather than publishing uninitialised words - // as traced pointers. - let copied = std::cmp::min(key_count, src_len); - debug_assert_eq!( - copied, key_count, - "the shape's key count outruns its keys array" - ); - for i in 0..copied { - // GC_STORE_AUDIT(INIT): cloned keys array is unpublished; layout is rebuilt before publication. - *dst_data.add(i) = (*src_data.add(i)).to_bits(); + // #10868 step 2.5 stage 1b: the canonical successor, as above the + // linear scan. The clone-if-shared block this replaces published the + // clone and then the pushed array — two mints per grow; the census + // reads them as tail.rs:1022 `fresh_keys_known_list` 9,076 and + // tail.rs:1044/1102 `key_count` 3,676 + 5,416. `keys_shared` is not + // tested because it is now true of every keys array by construction. + let new_index = key_count; + // Two modes, as above the linear scan. + let new_keys = match crate::object::canonical_keys::SharedLayout::of_receiver(obj) { + Some(proof) => { + let canonical_parent = + crate::object::canonical_keys::canonicalize(&proof, keys, key_count as u32); + refresh_roots_after_alloc!(); + crate::object::canonical_keys::extend_key(&proof, canonical_parent, key).as_ptr() + } + None => { + let owned = scope.root_raw_mut_ptr(keys); + let grown = crate::array::js_array_push(keys, JSValue::string_ptr(key as *mut _)); + let _ = owned.get_raw_mut_ptr::(); + grown } - (*cloned).length = copied as u32; - super::rebuild_array_layout_from_slots(cloned); - set_object_keys_array(obj, cloned); - cloned - } else { - keys }; + refresh_roots_after_alloc!(); // Check if we have a spare physical slot (js_object_alloc_with_shape allocates max(N,8) slots). // Class objects (js_object_alloc_class_with_keys) have only exactly field_count slots; // attempting to write to new_index = key_count would overflow into the next heap allocation. - let new_index = key_count; if new_index >= alloc_limit { // No inline room — store in the overflow HashMap so the value is not lost. // Also add the key to keys_array so Object.keys() sees it. - let owned_keys_handle = scope.root_raw_mut_ptr(owned_keys); - let new_keys = - crate::array::js_array_push(owned_keys, JSValue::string_ptr(key as *mut _)); - prev_keys_usize = if keys_shared { - prev_keys_usize - } else { - owned_keys_handle.get_raw_mut_ptr::() as usize - }; - refresh_roots_after_alloc!(); set_object_keys_array(obj, new_keys); super::mark_object_dynamic_shape_unknown(obj); - // #8067: migrate the owned slot index, not the descriptor id. - if !keys_shared { - super::shapes::shape_keys_grown(prev_keys_usize, new_keys); - } // #7538: the bits stored into overflow must come from the // REFRESHED `value`. This was snapshotted ABOVE the // `js_array_push` that grows the keys array — an allocation, so a @@ -1165,22 +1101,9 @@ pub(crate) fn set_field_by_name_object_tail( } return; } - // First, add the key to the keys array (may reallocate) - let owned_keys_handle = scope.root_raw_mut_ptr(owned_keys); - let new_keys = crate::array::js_array_push(owned_keys, JSValue::string_ptr(key as *mut _)); - prev_keys_usize = if keys_shared { - prev_keys_usize - } else { - owned_keys_handle.get_raw_mut_ptr::() as usize - }; - refresh_roots_after_alloc!(); - // Update the object's keys_array pointer in case js_array_push reallocated + // Publish the canonical successor computed above. set_object_keys_array(obj, new_keys); super::mark_object_dynamic_shape_unknown(obj); - // #8067: migrate the owned slot index, not the descriptor id. - if !keys_shared { - super::shapes::shape_keys_grown(prev_keys_usize, new_keys); - } // Set the field at the new index and update logical field_count // #7154 publication order: `gc_field_slot_range` bounds the diff --git a/crates/perry-runtime/src/object/mod.rs b/crates/perry-runtime/src/object/mod.rs index c1e8bef481..aca3fa219d 100644 --- a/crates/perry-runtime/src/object/mod.rs +++ b/crates/perry-runtime/src/object/mod.rs @@ -55,7 +55,13 @@ mod test_root_helpers; #[cfg(test)] pub(crate) use test_root_helpers::*; -mod alloc; +pub(crate) mod alloc; +mod alloc_basic; +pub use alloc::{ + js_object_alloc, js_object_alloc_fast, js_object_alloc_fast_with_parent, + js_object_alloc_null_proto, js_object_alloc_with_parent, js_object_coerce, +}; +pub(crate) use alloc::mark_object_plain_ordinary; mod json_construction; pub(crate) use json_construction::{ object_from_inline_json_fields, object_from_json_fields_preinstalled, @@ -80,6 +86,7 @@ pub(crate) use class_registry::class_registry_census; pub(crate) use class_registry::construct_two_rooted; pub(crate) use class_registry::{construct_rooted_arguments, scan_current_new_target_root_mut}; mod census; +pub(crate) mod canonical_keys; pub(crate) use census::object_tables_census; mod collection_proto_thunks; mod data_view_registry; @@ -712,7 +719,53 @@ fn shape_cache_get_with_id(shape_id: u32) -> (*mut ArrayHeader, u32) { /// Insert a keys_array into the cache. Updates the inline slot /// (evicting any prior entry there) and also writes to the overflow /// map so misses on the inline cache still find the value. -fn shape_cache_insert(shape_id: u32, keys_array: *mut ArrayHeader) { +/// #10868 step 2.5 stage 1c: this call CAN COLLECT, because `canonicalize` +/// allocates, so it takes the caller's live object BY VALUE and hands back +/// the post-collection one. A caller that keeps its old binding no longer +/// compiles. That is not hypothetical: stage 1b introduced the allocation and +/// `js_object_alloc_class_with_keys` wrote its keys edge at the stale address, +/// which `descriptor_trap_collection_preserves_for_in_target_and_keys` caught +/// because its trap collects once per key — fourteen collections through one +/// enumeration, where the `ownKeys` sibling collects once and saw nothing. +#[must_use] +fn shape_cache_insert( + shape_id: u32, + live: canonical_keys::LiveObject, + keys_array: *mut ArrayHeader, +) -> (canonical_keys::LiveObject, *mut ArrayHeader) { + // #10868 step 2.5 stage 1b: the cache holds the CANONICAL array for this + // static shape's key list, so two compile-time shapes that spell the same + // ordered key list are one layout rather than two. The canonical array is + // RETURNED rather than swapped in silently — every caller here keeps + // using the pointer afterwards (`remember_class_keys_array`, and the + // value it hands back to codegen), and a cache holding one array while + // the caller holds another is exactly the divergence this stage exists to + // remove. + let (live, keys_array) = { + // SAFETY: a live keys array or null; `canonicalize` allocates, roots + // its own operand, and `across` roots the caller's object. + let len = if keys_array.is_null() { + 0 + } else { + unsafe { (*keys_array).length } + }; + if len == 0 { + // The empty list has no canonical array — the trie's root owns + // none — and a zero-length keys array is NOT interchangeable with + // null here: `js_build_class_keys_array` hands this pointer back + // to generated code. Leave it exactly as it was. + (live, keys_array) + } else { + live.across(|| unsafe { + canonical_keys::canonicalize( + &canonical_keys::SharedLayout::shape_cache_entry(), + keys_array, + len, + ) + .as_ptr() + }) + } + }; // Mark the array as shape-shared so `js_object_set_field_by_name` // knows it must clone before mutating. The clone path was firing // every time *any* fresh object literal added a property beyond @@ -751,6 +804,7 @@ fn shape_cache_insert(shape_id: u32, keys_array: *mut ArrayHeader) { .insert(shape_id, (keys_array, runtime_shape_id)); crate::gc::runtime_write_barrier_root_raw_ptr(keys_array); shape_carriers::note_shape_id(runtime_shape_id); + (live, keys_array) } /// Thread-local shape-transition cache for the dynamic-key write path @@ -1325,6 +1379,9 @@ pub fn scan_object_cache_roots(mark: &mut dyn FnMut(f64)) { } pub fn scan_object_cache_roots_mut(visitor: &mut crate::gc::RuntimeRootVisitor<'_>) { + // Object-owned weak layout caches: rewrite moves without retaining keys. + canonical_keys::scan_canonical_keys_roots_mut(visitor); + scan_class_keys_roots_mut(visitor); for slot in [ &HTTP_METHODS_CACHE, &FS_CONSTANTS_CACHE, @@ -1400,8 +1457,8 @@ pub fn scan_object_cache_roots_mut(visitor: &mut crate::gc::RuntimeRootVisitor<' /// but a call: a seam with logic of its own can drift from the writer it /// stands in for, which is exactly what let a deleted arm site stay green. #[cfg(test)] -pub(crate) fn test_shape_cache_insert(shape_id: u32, keys_array: *mut ArrayHeader) { - shape_cache_insert(shape_id, keys_array); +pub(crate) fn test_shape_cache_insert(shape_id: u32, keys_array: *mut ArrayHeader) -> *mut ArrayHeader { + shape_cache_insert(shape_id, canonical_keys::LiveObject::none(), keys_array).1 } #[cfg(test)] diff --git a/crates/perry-runtime/src/object/object_ops/keys_array.rs b/crates/perry-runtime/src/object/object_ops/keys_array.rs index e799516874..2a7f7ac0cd 100644 --- a/crates/perry-runtime/src/object/object_ops/keys_array.rs +++ b/crates/perry-runtime/src/object/object_ops/keys_array.rs @@ -274,64 +274,30 @@ unsafe fn ensure_key_in_keys_array_inner( } } - // Clone a shape-cache / transition-cache keys array before appending. - // - // The old `key_count == field_count` proxy was not an ownership test. - // Objects may legitimately have a different logical field boundary while - // still pointing at the shared shape array. In that case defineProperty - // appended directly to the cache entry, so sibling `{}` allocations grew - // the same phantom own key (Babel's webpack exports objects exposed this - // as an enumerable `ALIAS_KEYS: undefined`). The caches already stamp the - // authoritative GC_FLAG_SHAPE_SHARED bit; use it just like the ordinary - // [[Set]] growth path does. - let keys_gc_header = - (keys as *const u8).sub(crate::gc::GC_HEADER_SIZE) as *const crate::gc::GcHeader; - let keys_shared = (*keys_gc_header).gc_flags & crate::gc::GC_FLAG_SHAPE_SHARED != 0; - let owned_keys = if keys_shared { - // Every entry in an ordered object-keys array is a heap string - // pointer. Preserve that invariant explicitly while cloning instead - // of starting as a raw-f64 array and reconstructing a HashMap-backed - // per-object pointer mask from the finished slots. The clone is still - // unpublished here and no allocation occurs during the copy, so it is - // safe to expose the initialized prefix through `length` only after - // the last pointer has been written. - let cloned = crate::array::js_array_alloc_pointer_elements(key_count as u32 + 4); - refresh_define_property_roots!(); - let keys = crate::object::object_keys_array(obj); - // #10939: a keys array's elements do not necessarily start - // at `header + 8`. `keys_array_dense_slots` resolves a - // grow-forward pointer and adds `array_front_offset`, which is - // nonzero for any array with a front reserve — #9019's - // reserved-floor keys arrays are BORN with leading holes, and a - // size-class round-up alone can make it nonzero. The clone - // declares every published slot a pointer, so copying from the - // wrong base does not merely read the wrong bytes: it promises - // the collector that `ArrayHeader` and reserve words are heap - // pointers. A missing property now, a SIGSEGV inside the next - // collection later, with a backtrace naming something else. - let (src_data, src_len) = crate::object::keys_array_dense_slots(keys); - let dst_data = crate::array::array_elements_ptr(cloned as *const crate::array::ArrayHeader); - // A source shorter than the shape's count means the shape is already - // lying; copy what exists rather than publishing uninitialised words - // as traced pointers. - let copied = std::cmp::min(key_count, src_len); - debug_assert_eq!( - copied, key_count, - "the shape's key count outruns its keys array" - ); - for i in 0..copied { - // GC_STORE_AUDIT(INIT): cloned keys array is unpublished and its all-pointer layout covers only the prefix published by length. - *dst_data.add(i) = (*src_data.add(i)).to_bits(); + // #10868 step 2.5 stage 1b: the canonical successor for this ordered list + // plus `key`. The clone-before-append this replaces existed because the + // array might be shared; every keys array is shared now, so the question + // is not asked — and the clone it used to answer with was a fresh address + // per receiver, which is a fresh LAYOUT per receiver. This site is the + // census's `object_ops/keys_array.rs:318`, 5,334 `key_count` mints, and + // it is the mechanism named in the #10287 comment above: a receiver that + // defines one key can never again share a keys array with its siblings. + // Two modes, as in the `[[Set]]` tail: an ordinary receiver's list is a + // shared layout and interns; a latched receiver owns its list. + let new_keys = match crate::object::canonical_keys::SharedLayout::of_receiver(obj) { + Some(proof) => { + let canonical_parent = + crate::object::canonical_keys::canonicalize(&proof, keys, key_count as u32); + refresh_define_property_roots!(); + crate::object::canonical_keys::extend_key(&proof, canonical_parent, key).as_ptr() + } + None => { + let owned = scope.root_raw_mut_ptr(keys); + let grown = crate::array::js_array_push(keys, JSValue::string_ptr(key as *mut _)); + let _ = owned.get_raw_mut_ptr::(); + grown } - (*cloned).length = copied as u32; - set_object_keys_array(obj, cloned); - cloned - } else { - keys }; - let owned_keys_handle = scope.root_raw_mut_ptr(owned_keys); - let new_keys = crate::array::js_array_push(owned_keys, JSValue::string_ptr(key as *mut _)); - let _owned_keys = owned_keys_handle.get_raw_mut_ptr::(); refresh_define_property_roots!(); set_object_keys_array(obj, new_keys); // Keep the sidecar fresh (mirrors the [[Set]] append path): the entry is diff --git a/crates/perry-runtime/src/object/shape_mint_census.rs b/crates/perry-runtime/src/object/shape_mint_census.rs index 5f33498ddc..ab7b51c30d 100644 --- a/crates/perry-runtime/src/object/shape_mint_census.rs +++ b/crates/perry-runtime/src/object/shape_mint_census.rs @@ -521,6 +521,45 @@ pub(crate) fn dump() { mints as f64 / c.key_lists.len() as f64 )); } + // #10868 step 2.5 stage 1b. `distinct keys ADDRESSES` above and `live` + // here answer the same question from opposite sides, which is the point: + // this census hashes CONTENT (`key_list_content_hash`) and knows nothing + // about the trie, so `addresses == distinct key-NAME lists` is an + // INDEPENDENT confirmation that one array now serves one layout. A + // producer that skips the funnel shows up here as addresses climbing + // above lists while `live` stays put — which is the census sabotage. + { + let (live, canon_minted, reaped) = crate::object::canonical_keys::canonical_stats(); + let words = crate::object::canonical_keys::canonical_element_words(); + out.push_str(&format!( + " canonical keys trie: live {} minted {} reaped {} elements {} ({} KB)\n", + live, + canon_minted, + reaped, + words, + words * 8 / 1024 + )); + // The named witness for the funnel sabotage. One canonical array per + // ordered key list means a mint can only ever have seen as many + // ADDRESSES as it has seen distinct key-name LISTS; a producer that + // allocates around the funnel breaks that inequality and nothing else + // in the process notices, because a duplicate layout is a slow + // answer, not a wrong one. + let addrs = c.keys_addrs.len(); + let lists = c.key_lists.len(); + if addrs > lists { + out.push_str(&format!( + " FUNNEL BROKEN: {} keys ADDRESSES for {} distinct key-NAME lists \ + -- {} duplicate layouts; a producer is allocating around \ + canonical_keys\n", + addrs, + lists, + addrs - lists + )); + } else { + out.push_str(" FUNNEL OK: one keys address per distinct key-name list\n"); + } + } let tc_h = TC_HITS.load(Ordering::Relaxed); let tc_e = TC_MISS_EMPTY.load(Ordering::Relaxed); let tc_c = TC_MISS_COLLIDE.load(Ordering::Relaxed); diff --git a/crates/perry-runtime/src/object/shapes_tests.rs b/crates/perry-runtime/src/object/shapes_tests.rs index ba427d0b89..a2e9290a3e 100644 --- a/crates/perry-runtime/src/object/shapes_tests.rs +++ b/crates/perry-runtime/src/object/shapes_tests.rs @@ -825,48 +825,31 @@ mod descriptor_tests_8067 { test_drop_shape_descriptors(unrelated_keys); } - /// The retirement above is wired to the publish funnel: an in-place - /// append on an OWNED keys array must leave exactly one structural - /// descriptor under that address. - #[test] - fn in_place_owned_append_leaves_one_descriptor_per_keys_address() { - let _lock = crate::gc::global_side_table_test_lock(); - unsafe { - let obj = crate::object::js_object_alloc(0, 0); - let mut keys_before = 0usize; - let mut first_addr_count = 0usize; - for i in 0..96u32 { - let name = format!("owned9706_{i:03}"); - let key = crate::string::js_string_from_bytes(name.as_ptr(), name.len() as u32); - crate::object::js_object_set_field_by_name(obj, key, i as f64); - let keys = crate::object::object_keys_array(obj) as usize; - let stamp = object_shape_stamp(obj); - assert!(is_shape_id(stamp), "receiver must stay stamped"); - let family = test_shape_ids_for_keys(keys); - assert!( - family.contains(&stamp), - "the current stamp must be indexed under the current keys address" - ); - if keys == keys_before { - first_addr_count += 1; - let shared = crate::value::addr_class::try_read_gc_header(keys) - .is_some_and(|h| h.gc_flags & crate::gc::GC_FLAG_SHAPE_SHARED != 0); - if !shared { - assert_eq!( - family.len(), - 1, - "an owned in-place append left growth history alive: {family:?}" - ); - } - } - keys_before = keys; - } - assert!( - first_addr_count > 0, - "fixture premise: some appends must grow the owned array in place" - ); - } - } + // DELETED by #10868 step 2.5: `in_place_owned_append_leaves_one_descriptor_per_keys_address`. + // + // WHAT IT PINNED: that an in-place append on an OWNED keys array leaves + // exactly one structural descriptor under that address — i.e. that + // `retire_owned_shape_siblings` really is wired to the publish funnel and + // growth history does not pile up under a reused address. It asserted its + // own precondition, `first_addr_count > 0`, "some appends must grow the + // owned array in place". + // + // WHY THE PREMISE IS NOW FALSE: canonical identity means one array per + // ordered key list, so an append never keeps its address — the successor + // is a different canonical array by construction. No keys array is owned + // any more (every one is `GC_FLAG_SHAPE_SHARED` from birth), so there is + // no in-place append for this test to observe, and `first_addr_count` is + // 0 by construction rather than by regression. `retire_owned_shape_siblings` + // is itself unreachable for the same reason. + // + // WHAT PINS THE REPLACEMENT PROPERTY: the concern was descriptors piling + // up under one address. That is now impossible in a stronger form — + // an address names exactly one key list, so a family under it can differ + // only in the non-keys facts. `object::canonical_keys`'s + // `one_array_serves_one_ordered_key_list` and `a_prefix_is_its_own_node` + // pin the identity, and the mint census's `FUNNEL OK / FUNNEL BROKEN` + // line pins it on a whole real program — it is the witness the funnel + // sabotage reddens, where the parity suite structurally cannot. #[test] fn shape_drop_does_not_delete_a_potential_siblings_descriptor() { diff --git a/crates/perry-runtime/src/object/tombstone_tests.rs b/crates/perry-runtime/src/object/tombstone_tests.rs index 4d9107243e..e27e37a9b5 100644 --- a/crates/perry-runtime/src/object/tombstone_tests.rs +++ b/crates/perry-runtime/src/object/tombstone_tests.rs @@ -124,13 +124,16 @@ fn tombstone_hole_count_survives_readd_append() { assert_eq!(super::shapes::object_shape_hole_count(obj), 1); let tombstoned_shape = super::shapes::object_shape_stamp(obj); // Re-add: appends (enumeration order moves the key to the end) and - // must NOT reset the hole accounting. + // must NOT reset the hole accounting. The append changes the layout, + // so it publishes a successor shape: a shape id names exactly one + // layout (step 2.5), the same rule a delete follows. Only the hole + // accounting is carried across the publish. let readd = crate::string::js_string_from_bytes(b"hc_key_03".as_ptr(), 9); js_object_set_field_by_name(obj, readd, 99.0); - assert_eq!( + assert_ne!( super::shapes::object_shape_stamp(obj), tombstoned_shape, - "same-allocation re-add must not retire the stable tombstone ShapeId" + "re-add changed the layout but kept the tombstoned ShapeId" ); assert_eq!( super::shapes::object_shape_hole_count(obj), diff --git a/crates/perry-runtime/src/proxy.rs b/crates/perry-runtime/src/proxy.rs index b9094b1c92..1de21a1220 100644 --- a/crates/perry-runtime/src/proxy.rs +++ b/crates/perry-runtime/src/proxy.rs @@ -3113,11 +3113,19 @@ mod tests { "a hole cannot be treated as an object receiver" ); + // Class ids alone do not distinguish layouts: equal ordered keys + // canonicalize to the same array. Keep both targets present, but swap + // their slots so publishing the first receiver's slots would be wrong. + let other_packed = b"a\0b\0d\0c\0"; let other_keys = crate::object::js_build_class_keys_array( 0x6809_02, 4, - packed.as_ptr(), - packed.len() as u32, + other_packed.as_ptr(), + other_packed.len() as u32, + ); + assert_ne!( + keys, other_keys, + "fixture requires different ordered key layouts" ); let other = crate::object::js_object_alloc_class_inline_keys(0x6809_02, 0, 4, other_keys); let mixed_values = [boxed_object(first), boxed_object(other)]; @@ -3126,7 +3134,7 @@ mod tests { assert_eq!( object_array_numeric_write_guard(boxed_object(mixed.cast()), &[c, d], 2), 0, - "content-equal but distinct shape keys arrays must not share raw slots" + "different ordered key layouts must not share raw slots" ); let ranged_values = [ diff --git a/scripts/gc_root_dominance_check.py b/scripts/gc_root_dominance_check.py index 093368a9cb..6d123deb82 100755 --- a/scripts/gc_root_dominance_check.py +++ b/scripts/gc_root_dominance_check.py @@ -1468,6 +1468,8 @@ def is_collecting(callee): # count and are deliberately NOT folded in here — same reasoning as # ALLOC_RE's deleted `bigint_\w+_op`. POLL_CAPABLE_RUNTIME = { + # Module initialization resolves canonical read predictions and allocates keys. + "js_canonical_read_shape", "js_call_function", # Calling a JS closure. The four names this replaces # (`js_call_closure`, `js_invoke_closure`, `js_function_call`, diff --git a/scripts/shape_descriptor_census_baseline.json b/scripts/shape_descriptor_census_baseline.json index 3c5f112c97..40f7bf669f 100644 --- a/scripts/shape_descriptor_census_baseline.json +++ b/scripts/shape_descriptor_census_baseline.json @@ -20,6 +20,7 @@ "crates/perry-codegen/src/lower_call/typed_shape_bake_tests.rs|8 + crate::target_layout::object_header_size_bytes( ) + 8 * slots;": 1, "crates/perry-codegen/src/stmt/cached_field_index_return.rs|let header_size = crate::target_layout::object_header_size_bytes(ctx.target_triple).to_string();": 1, "crates/perry-codegen/src/stmt/cached_field_index_return.rs|let meta_offset = (crate::target_layout::object_header_size_bytes(ctx.target_triple)": 1, + "crates/perry-codegen/src/stmt/canonical_read_loop.rs|let header = crate::target_layout::object_header_size_bytes(ctx.target_triple);": 1, "crates/perry-codegen/src/stmt/loops.rs|let object_header_size = crate::target_layout::object_header_size_bytes(ctx.target_triple);": 1, "crates/perry-codegen/src/stmt/stable_packed_loop.rs|crate::target_layout::object_header_size_bytes(ctx.target_triple).to_string();": 2, "crates/perry-codegen/src/target_layout.rs|assert_eq!(object_header_size_bytes( ), 16);": 6, @@ -52,18 +53,17 @@ "crates/perry-runtime/src/object/field_get_set/field_ops.rs|keys_array|declaration|pub extern fn js_object_set_keys(obj: *mut ObjectHeader, keys_array: *mut ArrayHeader) {": 1, "crates/perry-runtime/src/object/mod.rs|keys_array|access|crate::gc::runtime_store_root_raw_mut_ptr_slot(&mut entry.keys_array, keys_array);": 2, "crates/perry-runtime/src/object/mod.rs|keys_array|access|return (entry.keys_array, entry.runtime_shape_id);": 1, - "crates/perry-runtime/src/object/mod.rs|keys_array|declaration|fn shape_cache_insert(shape_id: u32, keys_array: *mut ArrayHeader) {": 1, - "crates/perry-runtime/src/object/mod.rs|keys_array|declaration|keys_array: *mut ArrayHeader,": 2, + "crates/perry-runtime/src/object/mod.rs|keys_array|declaration|keys_array: *mut ArrayHeader,": 3, "crates/perry-runtime/src/object/mod.rs|keys_array|declaration|keys_array: std::ptr::null_mut(),": 1, "crates/perry-runtime/src/object/mod.rs|keys_array|declaration|pub(crate) fn test_seed_shape_cache_root(shape_id: u32, keys_array: *mut ArrayHeader) {": 1, - "crates/perry-runtime/src/object/mod.rs|keys_array|declaration|pub(crate) fn test_shape_cache_insert(shape_id: u32, keys_array: *mut ArrayHeader) {": 1, + "crates/perry-runtime/src/object/mod.rs|keys_array|declaration|pub(crate) fn test_shape_cache_insert(shape_id: u32, keys_array: *mut ArrayHeader) -> *mut ArrayHeader {": 1, "crates/perry-runtime/src/object/mod.rs|keys_array|declaration|unsafe fn set_object_keys_array(obj: *mut ObjectHeader, keys_array: *mut ArrayHeader) {": 1, "crates/perry-runtime/src/object/object_ops.rs|keys_array|declaration|pub(crate) use keys_array::{": 1, "crates/perry-runtime/src/object/side_table_roots.rs|keys_array|access|visitor.visit_raw_mut_ptr_slot(&mut entry.keys_array);": 1, "crates/perry-runtime/src/object/test_root_accessors.rs|keys_array|access|let inline = unsafe { (*st.object_hot.shape_inline_cache.get())[slot].keys_array as usize };": 1 }, "summary": { - "codegen_object_header_size_sites": 43, + "codegen_object_header_size_sites": 44, "raw_member_files": 12, "raw_member_sites": { "keys_array": 35 diff --git a/test-files/test_gap_canonical_read_loop.ts b/test-files/test_gap_canonical_read_loop.ts new file mode 100644 index 0000000000..cfd8e3b1d1 --- /dev/null +++ b/test-files/test_gap_canonical_read_loop.ts @@ -0,0 +1,30 @@ +function run(n: any, O: any) { let h=0; for(let k=0;k