[rjb32]

Unwind a list built per row without building the lists of a whole chunk.

UNWIND range(1, 65536) AS i UNWIND [x IN range(i, i + 999) | x * 2] AS j RETURN count(*)    2322 MB -> 106 MB
UNWIND range(1, 65536) AS i UNWIND reverse(range(i, i + 999)) AS j RETURN count(*)          2316 MB -> 104 MB
UNWIND range(1, 65536) AS i UNWIND range(i, i + 999)[1..] AS j RETURN count(*)              1212 MB -> 106 MB
UNWIND range(1, 65536) AS i UNWIND tail(range(i, i + 999)) AS j RETURN count(*)             1211 MB -> 106 MB
UNWIND range(1, 65536) AS i UNWIND split(<string of 1000 fields>, ',') AS j RETURN count(*)  1995 MB -> 104 MB
UNWIND range(1, 65536) AS i UNWIND range(i, i + 499) + range(i, i + 499) AS j ...           2323 MB -> 104 MB
UNWIND range(1, 65536) AS i UNWIND reverse([x IN range(i, i + 999) | x * 2]) AS j ...       3429 MB -> 112 MB
UNWIND range(1, 65536) AS i UNWIND COLLECT { UNWIND range(1, 1000) AS k RETURN k + i } ...  4801 MB -> 112 MB
UNWIND range(1, 65536) AS i UNWIND reduce(acc = [], k IN range(1, 100) | acc + [k + i]) ... 705 MB -> 116 MB
UNWIND range(1, 65536) AS i WITH range(i, i + 999) AS l UNWIND l AS j RETURN count(*)      1212 MB -> 104 MB

test_query_ir_unwind_comprehension            21 tests
test_query_ir_unwind_range_window             33 tests
test_query_ir_unwind_list_producer_codegen     9 tests
test_query_ir_unwind_computed                 17 tests
test_query_ir_unwind_computed_codegen          6 tests
test_query_ir_unwind_row_count                 7 tests
test_query_ir_unwind_row_count_codegen         6 tests

[rjb32]

Note

AI-generated summary.

Summary

An UNWIND over a list built per row laid out the lists of the whole 64Ki-row chunk before the first row was unwound. b69d60f fixed this for a bare range(). This PR fixes it for every list in three commits:

• d547734 builds no list at all for a comprehension, or for the reverse, a slice or the tail of a range.
• 408bae5 computes any other list a few input rows at a time.
• c287b92 stops a count(*) from keeping a carried list alive.

d547734: build no list

 UNWIND [x IN xs WHERE p | f] AS y
-  db.unwind(db.list_comprehension(xs) { db.filter(p); yield f })
+  db.unwind(xs)  db.filter(p, {x, rows in flight...})  f          # unnest_unwind_comprehension, first pass

 UNWIND reverse(range(a, b, s)[i..j]) AS x
-  db.unwind(db.reverse(db.list_slice(db.range(a, b, s), i, j)))
+  db.unwind_range(a, b, s, from i, to j, {...}) reversed           # fuse_unwind_range

unnest_unwind_comprehension leaves the comprehension in place in three cases:

• something else reads its list;
• the projection is a constant, which would stand for one row instead of one per element;
• the body uses its row tag for anything other than its WHERE.

The range cursor (NLRangeSpan::keepWindow) reads the slice bounds as positions in the range, with runListSlice's rules: negative bounds count from the end, bounds are clamped, and a null bound gives no rows. So ranges of up to 2^64 integers slice without int64 arithmetic.

408bae5: compute the list a few rows at a time

A new last pass, sink_unwind_source, moves into a region of a new db.unwind_computed the ops that compute the unwound list from the row. An op is moved only when it computes one value per row (the ConstantThroughOperands ops, the row-wise ops, and the COLLECT, COUNT, EXISTS and pattern subqueries) and nothing outside reads it. The region reads the rows through one block argument per carried column.

%i = db.unwind_range(%c1, %c65536, %c1, {}) : ...
%j, %iOut = db.unwind_computed({%i}) ({
^bb0(%iRow: !db.column<!storage.nullable<i64>>):
  %end = db.add %iRow, %c999 : ...
  %xs = db.range(%iRow, %end) : ...
  %ys = db.list_comprehension(%xs, {}) ({ ... }) : ...
  %r = db.reverse(%ys) : ...
  db.unwind_yield %r : !db.column<none>
}) : ...

The pass leaves an unwind alone when its source is a constant or a stored property read, because no list is built for those. lowerUnwindComputed lowers the region in place, as a comprehension body is lowered. runUnwindComputedLoop then runs it over runs of input rows:

run = 1 row
while input rows are left and the limit is not reached
  cut the carried inputs to the run              # the existing gather functions
  run the region                                 -> the run's lists
  unwind them with NLElementCursor, one chunk per step, carried columns gathered from the whole input
  next run = chunkSize * run / elements of this run, at least 1 row

A run holds about one chunk of elements, so peak memory is one chunk plus the longest single list. For short lists the run grows back to the whole chunk.

c287b92: read a row count off the cheapest column

Codegen anchors every read that needs only a row count on whatever column is in scope. Three ops read a column that way: db.count ... rows, a count_rows aggregate in db.group_aggregate, and db.broadcast_constant's driver. In WITH range(i, i + 999) AS l UNWIND l AS j RETURN count(*) the count read the carried list l, which kept l alive and gave the range a second reader.

 %j, %l2 = db.unwind(%l, {%l})
-%n = db.count(%l2) rows
+%n = db.count(%j) rows                                          # anchor_row_counts

A new pass, anchor_row_counts, moves each such read onto another result of the same op. All results of an op with a carry set hold the same rows. It ranks the candidates in three steps:

1. a column something else already reads;
2. otherwise the cheapest column to carry: fixed-width values first, then untyped columns, then strings, then lists, maps and other values that own data;
3. on a tie, a column the op computes over a carried copy.

The pass runs just before the first trim_unread_columns, which then drops l, and unnest_unwind_comprehension now runs just after that trim.

Evidence

• Before: peak RSS, one query per server capped at 8 GB, each query under UNWIND range(1, 65536) AS i. The first block was measured on main 840ea2a, the other two on d547734.
  After: the same queries on c287b92.

UNWIND [x IN range(i, i + 999) | x * 2] AS j RETURN count(*)          2322 MB -> 106 MB
UNWIND reverse(range(i, i + 999)) AS j RETURN count(*)                2316 MB -> 104 MB
UNWIND range(i, i + 999)[1..] AS j RETURN count(*)                    1212 MB -> 106 MB
UNWIND tail(range(i, i + 999)) AS j RETURN count(*)                   1211 MB -> 106 MB

UNWIND split(<per-row string of 1000 fields>, ',')                    1995 MB -> 104 MB    1198 ms -> 600 ms
UNWIND range(i, i + 499) + range(i, i + 499)                          2323 MB -> 104 MB    1204 ms -> 465 ms
UNWIND range(i, i + 999) + i                                          2320 MB -> 105 MB    1212 ms -> 468 ms
UNWIND reverse([x IN range(i, i + 999) | x * 2])                      3429 MB -> 112 MB    2400 ms -> 1273 ms
UNWIND [x IN range(i, i + 999) | x * 2][1..]                          2325 MB -> 111 MB    1889 ms -> 1113 ms
UNWIND COLLECT { UNWIND range(1, 1000) AS k RETURN k + i }            4801 MB -> 112 MB    3456 ms -> 1082 ms
UNWIND reduce(acc = [], k IN range(1, 100) | acc + [k + i])            705 MB -> 116 MB    9670 ms -> 6742 ms

WITH range(i, i + 999) AS l UNWIND l AS j RETURN count(*)             1212 MB -> 104 MB     697 ms -> 40 ms

• Before: test_query_ir_unwind_list_producer_codegen failed 6 of 8. UnwindRangeWindowTest failed its 3 full-int64 cases, which build a list past the 100000-integer limit. Before c287b92, test_query_ir_unwind_row_count_codegen failed 4 of 6.
  After: test_query_ir_unwind_comprehension 21/21, test_query_ir_unwind_range_window 33/33, test_query_ir_unwind_list_producer_codegen 9/9, test_query_ir_unwind_computed 17/17, test_query_ir_unwind_computed_codegen 6/6, test_query_ir_unwind_row_count 7/7, test_query_ir_unwind_row_count_codegen 6/6. ctest: all 708 test_query* and the other 172 pass.

Merge Danger

Door: two-way

The rewrites are db passes and the op changes are additive. Reverting a commit restores the lists it stopped building.

Blast Radius: queries

Every query now runs three more passes. Every UNWIND whose list is computed from the row now goes through db.unwind_computed and its run loop. Every count(*), count_rows aggregate and db.broadcast_constant driver may now read a different column of the same rows. Still open: a CASE whose branches are lists is rejected (db.case requires scalar or entity branches), with or without this PR.