Skip to content

-mm=own phase 7b: a generator over a parameter's block borrows it; .map and .filter - #441

Merged
ASDAlexander77 merged 5 commits into
mainfrom
own-phase-7b
Oct 2, 2026
Merged

ASDAlexander77 merged 5 commits into
mainfrom
own-phase-7b

Conversation

@ASDAlexander77

@ASDAlexander77 ASDAlexander77 commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner

Phase 7b of -mm=own: generators that borrow. The plan is docs/superpowers/plans/2026-10-02-own-phase-7b.md and the results are in spec §19.

A generator over a parameter that holds a block now compiles: function* each(a: number[]) { for (const v of a) yield v; }. In Rust terms that's fn each<'a>(a: &'a [f64]) -> impl Iterator + 'a. The caller owns the generator's state object, but the state object borrows the argument and may not outlive it. .map and .filter, which MLIRGen builds as such a generator over the source array, compile too.

How it works

  • The signature pass decides which fields of a state object borrow. The release routine is built once per type, while the inference runs per function and in parallel, so the decision has to be module-level. A state type is unique to its generator: the type of its next names a per-site object_storage.
    • The .captured box borrows when every fill is a parameter's cell, or a copy read out of a parameter's field.
    • A state field borrows when everything stored into it, anywhere in the module, is a read of a borrowing cell or field (computed as a least fixpoint). The for...of copy of the array is such a field.
    • Outputs: the module attribute ts.own_borrowing_fields, and __own_result_bounded (the parameters whose blocks the state borrows) on the maker and on each call to it.
  • The lowering skips a borrowing field in the state object's release routine. A borrowing box gets tsrelcbc_, which frees each cell but none of what the cells or the by-value fields hold.
  • Inside the generator: a store into a borrowing field takes nothing. What is read out of a borrowing cell, field or box field is the caller's, and may only be borrowed: yielding it is an error, because it would give the caller's value a second owner and rc retains nothing there for any other check to see.
  • At the caller, the generator is owned like any fresh value, and bounded:
    • no use of it may come after an end of a bounding argument (its release, its assignment, or the overwrite of the field it was read from);
    • it may not be stored, returned or captured;
    • for .map, the bound is the closure box: it ends where the closures on it are released, and where what it holds ends.
  • .map and .filter:
    • The caller's box holding a copy of the array borrows it when nothing in the box is fresh. So two .maps of one array no longer move it twice.
    • A callback that captures is a fresh closure made for that box. In that case the box owns its copies and the array moves in.
    • rc reads the array twice for one copy and retains the first read. The second read is now taken as the same value.

For review

  • calledOnlyDirectly is a second closed-world test, which disagrees with findOpen by design. findOpen marks .map's maker (..iter) open, because a ts.SymbolRef names it in a ts.CreateBoundFunction. The bounded fact is pinned anyway. The argument is that the maker is private, it is the only ts.New of its state type, and every closure over it is dead: only retained and released, with the real call made directly through the box.
  • New coupling: the lowering now reads a pass output (ts.own_borrowing_fields) to pick a release routine. It's own-only: borrowingFieldsOf is empty under rc, gc and none, and whenever the attribute is absent.
  • The twin-read rule is general, not generator-only: it changes takerOf and retainBefore for every move out of a slot. The evidence that it's safe is that the corpus didn't move (364 before and after it, none lost). Its barrier stops at any op given the place, and at any call.

Verification

  • Release suite: 3389 of 3390 pass. The one failure, test-compile-gc-defaultlib-collector, is the same stale installed default library as on -mm=own phase 5a: a closure that does not escape borrows what it captures #436, -mm=own phase 5b: a closure that escapes owns what it captures; a call in a try body is a call #437 and -mm=own phase 7a: generators and async; an interface is a view of its object #439. All 388 own tests pass.

  • WSL/GCC: builds, and 387 of 387 own tests pass (Windows has one more, the Windows-only shared-library check).

  • Corpus: 359 → 364 of 591, none lost: 00funcs_generic_iterator, 01disposable, 02disposable (an object literal whose methods capture a parameter is the same shape), 00filter, 00class_iterator_super. Each passes under test-runner -mm=own, AOT and JIT.

  • Memory: own_generator_borrow measures flat, the same as rc, at both sizes:

    iterations gc rc none own
    20000 6.1 4.3 1441.5 4.3
    200000 6.1 4.3 14376.5 4.3

    The ten-times run caught a leak in this PR's first form of the copies rule (a borrowed capturing callback, which nothing then freed). It's fixed in 12a7bd2.

  • Teeth: each new rule was undone with a temporary switch:

    • the owning box routine for a borrowing box breaks own_generator_borrow, AOT (0xC0000374) and JIT;
    • releasing a borrowing field does the same;
    • skipping the bounded check makes the three bounded negatives compile;
    • skipping the borrowed-read check makes the yield negative compile.

Tests

  • Positive: own_generator_borrow: an array, a class and a string parameter; a generator given up early; two generators over one argument; an argument read out of a field; map; filter; and a capturing filter as the array's last use.
  • Negatives: own_err_generator_outlives_arg, own_err_generator_bounded_escapes, own_err_generator_yields_borrow, own_err_generator_map_outlives, own_err_generator_place_overwritten.
  • Removed: own_err_generator_param_object, which now compiles.

Found, not fixed (every model)

a.map(f).map(g) fails to compile with "the condition has no value": #442. Without the default library, the built-in .map returns a generator, and the real error is lost in a condition.

Known limits (spec §19.5)

  • A yielded borrow is an error: the result record type is shared between generators, so a fact per type can't say that a record borrows.
  • A capturing .map/.filter callback moves the array into the closure's box.
  • A bounded generator can't be returned past its argument, even when that argument is the returning function's own parameter.

🤖 Generated with Claude Code

ASDAlexander77 and others added 5 commits October 2, 2026 14:46
`function* each(a: number[])` keeps `a`'s cell in a box in its state object's `.captured` field,
and its `for...of` copies `a` into a state field. Under 5b that was "its value is the caller's".
Now the state object borrows, Rust's `-> impl Iterator + 'a`:

- The signature pass finds each generator state type (unique per site: its `next` names a
  per-site `object_storage`) whose closed maker fills its box with parameters' cells only, and
  decides which fields borrow: the box, and each field everything stored into is a read of a
  borrowing cell or field (a least fixpoint). It writes them as the module attribute
  `ts.own_borrowing_fields`, and pins `__own_result_bounded` (the parameters holding blocks) on the
  maker and its calls.
- The lowering skips a borrowing field in the state object's release routine, and gives a
  borrowing box the new routine `tsrelcbc_`, which frees each cell but not what it holds.
- The inference lets a parameter's heap cell move into a borrowing box. In `next`, a store into a
  borrowing field takes nothing (rc's retain goes, its release on overwrite goes), and what is read
  out of a borrowing cell or field may only be borrowed: yielding it would give the caller's value
  a second owner, and rc retains nothing there for anything else to notice
  (`own_err_generator_yields_borrow`).
- At a call with a bounded result the generator is owned as usual, but no use of it - through its
  locals and the `next` it is called through - may come after an end of a bounding argument, and
  it may not be stored, returned or captured (`own_err_generator_outlives_arg`,
  `own_err_generator_bounded_escapes`). Its releases are not uses: the routine gives back nothing it
  borrows.

`own_generator_borrow` covers an array, a class and a string parameter, a generator given up early
and two over one argument. `own_err_generator_param_object` compiles now and is gone. Corpus: 359
-> 362 of 591 (`00funcs_generic_iterator`, `01disposable`, `02disposable`; an object literal whose
methods capture a parameter is the same shape), each run under test-runner -mm=own, AOT and JIT,
with gc's output.

Plan: docs/superpowers/plans/2026-10-02-own-phase-7b.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MLIRGen builds `a.map(f)` as a closure whose box holds a copy of `a` and `f`, called at once
through the box: its body is the generator's maker, and it copies its box's fields by value into
the generator's own box.

- The signature pass takes a box copied from a parameter's fields as borrowing, bounded by that
  parameter, and a maker whose symbol is used only by direct calls and closures nothing but
  retains and releases as closed. In `next`, a by-value field of a borrowing box is a borrowing
  read too.
- The inference: a store into a borrowing state box is a borrow; a read of a borrowing box's field
  is the caller's (no place read: only the maker fills the box, so no call overwrites it - before,
  a `filter` callback called in a loop "dropped" the field it was read from); a closure box that
  holds copies only, does not escape, and is given to such a maker borrows its copies, so two
  `.map`s of one array no longer move it twice.
- At the caller, a bounding box ends where the closures on it are released and where what it was
  filled from ends (`own_err_generator_map_outlives`). The box is a `ref`, so it is checked before
  the test that skips arguments holding no block.
- rc reads a place twice for one boxed copy and retains the first read: the second is the same
  value, both where a taker is looked for and where rc's retain for a box field is.

`own_generator_borrow` gains `map` and `filter`. Corpus: 362 -> 364 of 591 (`00filter`,
`00class_iterator_super`), none lost, each run under test-runner -mm=own, AOT and JIT.

Found, not fixed (every model): `a.map(f).map(g)` fails to compile ("the condition has no
value").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r of its own

The previous commit let a copies-only closure box given to `.map`/`.filter`'s maker borrow
everything it holds. A callback that captures (`a.filter((x) => x > n)`) is a closure made for that
box, which rc hands its reference: borrowed, it was nobody's, a leak of 31 bytes a call that
measure.ps1 showed (rc 4.3 MB, own 10.5 MB over 200000 calls). The box now borrows only when no
copy it holds is fresh; otherwise it owns its copies as in phase 4, and the array moves into it.

`own_generator_borrow` puts its capturing `filter` last, after the array's other uses, and makes
its string in a helper (a temporary in front of a `yield` leaks under rc and own alike, spec 18.5).
It measures flat: 4.3 MB at 20000 and at 200000 iterations, as rc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Results, teeth, measurements and limits of phase 7b. Section 18.5's await item is #440 now: an
await of a function that returns nothing never awaits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ead's barrier

The bounded check's branch for an argument read out of a place had no test. `each(h.items)` now
runs in `own_generator_borrow`, and `own_err_generator_place_overwritten` assigns `h.items` while
the generator is used.

twinReadAfter stopped at a store into the place or a call; it now stops at any op given the place
(a push writes through it too). MLIRGen emits no such op there today; the rule runs on every move
out of a slot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant