Skip to content

test(workbook): assert recalc work, and add the missing chain fixture - #967

Merged
hhimanshu merged 1 commit into
mainfrom
feat/965-recalc-work-assertions-chain-bench
Aug 29, 2026
Merged

test(workbook): assert recalc work, and add the missing chain fixture#967
hhimanshu merged 1 commit into
mainfrom
feat/965-recalc-work-assertions-chain-bench

Conversation

@hhimanshu

@hhimanshu hhimanshu commented Aug 29, 2026

Copy link
Copy Markdown
Member

closes #965

The workbook benchmark suite had two gaps. This PR closes both, in that order: the assertions first (they change no CI gate), then the missing fixture.

1. Work is now asserted, not only timed

New file crates/workbook/tests/recalc_work_tests.rs. It rebuilds the benchmark file's fixture shapes at small sizes and asserts, as exact counts, how much work a recalc of each one does — the same instrument depgraph_range_index_tests.rs already uses for cells-examined-per-range-reference.

Two numbers per shape, because they are not the same number:

  • the dirty closure, from Workbook::recalc_incremental_measured — the formula cells an incremental recalc actually recomputed. This is the work.
  • the changes emitted, Vec<Change>::len() — recalc's write-back skips a cell whose recomputed value equals its stored one (if old == new { continue; }), so this counts formulas whose value moved. It equals the closure only because a never-recalculated workbook stores Value::Empty in every formula cell, which is why a first full recalc's change count is exactly the fixture's formula count.

Every expectation is derived from the fixture's construction and written as a literal with the arithmetic in a comment, never as an expression recomputing what the code under test computes.

shape formulas (full recalc) edit dirty closure
independent(50) 50 = 50 rows x 1 A1 1
row_totals(20) 20 = 20 rows x 1 A1 1
block_subtotals(200) 6 = r in {1,21,41,61,81,101} A1 1
block_subtotals(200) A101 5
tall_sparse(30) 30 = 30 rows x 1 A1 1
multi_sheet(4,10) 40 = 4 tabs x 10 x 1 S0!A1 1
multi_sheet_cross(4,10) 40 = 4 tabs x 10 x 1 S0!A1 4 (one per tab)
chain(50) 50 = A2..A51 A1 50 (whole chain)
chain(50) B51 1
chain(5000) 5000 A1 / B5001 5000 / 1

Every one of these passed on the first run: no shape dirties more than its construction implies, so there is no over-dirtying to report.

What the counts did reveal

incremental_recalc/block_subtotals_edit_root is measuring the cheapest edit its fixture admits, not the fan-out its builder describes. The subtotal windows are A{r}:A{r+99} at r = 1, 21, 41, …, so an interior source row sits inside five of them — but row 1 sits inside exactly one, because no window starts above row 1. The benchmark edits A1. The test therefore pins both: 1 for the root edit, and 5 for an interior edit at A101, which is the number that actually exercises overlapping range-precedent invalidation. Nothing is wrong with the benchmark; it just measures less than its name suggests, and now that is written down.

2. A chain fixture, and benchmarks for it

build_chain(n): A1 a literal, A2 = =A1+1, A3 = =A2+1, … through A{n+1} — one linear chain, n formulas deep. Every previously existing fixture is exactly one level deep: a formula reads a literal, and nothing reads that formula. multi_sheet_cross looks like a counter-example and is not — its formulas read a literal on the first tab, so it is depth 1 with high fan-out. Propagation through a dependency chain had no fixture at all.

New benchmarks:

  • full_recalc/chain/{1000,5000} — the same formula counts as independent/{1000,5000}, arranged as one chain instead of N unrelated pairs, so the pair isolates what graph topology costs.
  • incremental_recalc/chain_edit_root/{1000,5000} — editing A1 dirties the whole chain: worst-case propagation.
  • incremental_recalc/chain_edit_leaf/{1000,5000} — dirties exactly 1: best case. Together they bracket propagation cost, and the ratio between them is what should grow with n.

What the chain benchmarks measure (recorded numbers)

Best of 5, Apple M1 Max, ns/iter:

benchmark 1000 5000
full_recalc/chain 3,822,395 21,046,666
full_recalc/independent (existing, same formula count) 4,189,383 22,797,180
incremental_recalc/chain_edit_root 3,719,666 20,419,766
incremental_recalc/chain_edit_leaf 1,017,376 5,201,330

Two things fall straight out of the bracket, neither of which any existing benchmark could show:

  1. A chain costs about what N independent formulas cost. full_recalc/chain/5000 is ~8% faster than full_recalc/independent/5000 despite being 5,000 levels deep instead of 1. Depth is not, on its own, expensive — which is worth having recorded, because it is the assumption a future ordering change would silently break.

  2. The best case is not cheap, and it scales with the workbook rather than the edit. chain_edit_leaf dirties exactly 1 formula at both sizes, yet it costs 1.02ms at n=1000 and 5.20ms at n=5000 — 5.1x more work for a dirty set that did not grow. That is the incremental path's per-formula-cell sweeps (the volatile sweep over graph.formula_cells(), snapshot_formula_values, and spill-occupancy seeding), all O(formula cells) by construction and documented as such in recalc.rs. So this is a known design characteristic rather than a defect — but until now nothing measured it, because every other incremental benchmark varies the dirty set and the workbook size together. chain_edit_leaf holds the dirty set at 1 while the workbook grows, which isolates that floor and now gates it.

The root/leaf ratio is 3.7x at n=1000 and 3.9x at n=5000. If a future change lets that ratio collapse, propagation has stopped being proportional to the dirty set.

The one design decision worth reviewing

The chain carries one extra literal: the last link is A{n+1} = =A{n}+B{n+1}, with B{n+1} read by nothing else.

Without it there is no cheap edit to measure. In a pure chain the only non-formula cell is A1, and editing it dirties everything; writing a literal over a chain cell instead destroys a formula node, and any formula write invalidates the dependency-graph cache (see the graph_cache module docs). chain_edit_leaf would then be timing a cold graph rebuild while chain_edit_root timed a warm walk — the two would not bracket anything, and the mismatch would be invisible in the numbers. Editing B{n+1} is a literal-over-literal write in a table-free workbook, so it is structure-preserving: same fixture, same warm graph, only the distance travelled differs.

Deep chains

No recursion or stack limit. A 5,000-link chain recalculates in the test suite without incident, and it is the shape that would find one: evaluation order is Kahn's algorithm (iterative), the dirty-closure walk is a queue, and a formula reads its precedents' stored values rather than recursing into them, so no stage of a recalc recurses with the chain's depth. The 5,000-deep case is asserted in the test file so that stays true.

Baselines

The perf gate treats a benchmark with no baseline entry as a failure (UNGATED), so the six new benchmarks needed recorded entries. They were recorded to the method baselines.json documents, on the machine its recorded_on field names — Apple M1 Max (10 core), macOS 14.4, rustc 1.94.1, release profile — best of 5 full bench runs, normalised to calibration/hash_alloc measured across the same five runs. That is the count the file's own recorded_on and note fields specify; the best-of-3 recordings the note mentions are flagged there as exceptions, so they are not the method to copy.

Only the six new entries were added. Every pre-existing number is byte-identical; nothing in this PR moves them, so any future drift stays attributable.

Explicitly not in scope

template.clone() stays inside the timed closures. Moving it to iter_batched is the right eventual fix, but it re-records every existing baseline and belongs in its own PR so the change in numbers is attributable to it.

Verification

  • cargo test -p truecalc-workbook — 445 passed (52 suites)
  • cargo clippy --workspace --exclude truecalc-python -- -D warnings — clean
  • cargo bench -p truecalc-workbook --bench workbook_perf -- --test — every benchmark, new ones included, runs once successfully

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

The benchmark suite had two gaps: no fixture was ever more than one level
deep, and no benchmark ever checked how much work a recalc actually did.

Add `crates/workbook/tests/recalc_work_tests.rs`, which rebuilds the
benchmark shapes at small sizes and asserts recalculation work as exact
counts — the dirty closure from `recalc_incremental_measured`, and the
changes emitted. Every expectation is a literal derived from the fixture's
construction with the arithmetic in a comment, never an expression
recomputing what the code under test computes. Over-dirtying previously
showed up only as a slightly slower benchmark, indistinguishable from
noise; it is now a hard failure naming an exact number.

The counts show `incremental_recalc/block_subtotals_edit_root` measures the
cheapest edit its fixture admits, not the fan-out its builder describes:
the windows are `A{r}:A{r+99}` at r = 1, 21, 41, …, so row 1 is covered by
exactly one of them. Both numbers are now pinned — 1 for the root edit, 5
for an interior edit at A101.

Add `build_chain(n)` plus `full_recalc/chain`,
`incremental_recalc/chain_edit_root` and `incremental_recalc/chain_edit_leaf`
at n = 1000 and 5000. Every previously existing fixture is depth 1;
`multi_sheet_cross` only looks like a counter-example, since its formulas
read a literal on the first tab. The chain carries one tail literal so a
cheap edit is expressible: in a pure chain the only non-formula cell is A1,
and writing a literal over a chain cell would destroy a formula node and
invalidate the graph cache, making the leaf case time a cold rebuild
against the root case's warm walk.

Deep chains hit no recursion or stack limit — evaluation order is Kahn's
algorithm, the closure walk is a queue, and a formula reads stored values
rather than recursing — so a 5000-link chain is asserted to keep it that
way.

Record baselines for the six new benchmarks per the method baselines.json
documents: same machine as its `recorded_on` field, best of 5 full bench
runs on one unmodified binary. No pre-existing entry moved.

closes #965

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAt4Keq1m4fEyHPmPjHSJ7
@hhimanshu hhimanshu self-assigned this Aug 29, 2026
@github-actions

github-actions Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Test Coverage by Category

Category Unit Tests Google Sheets Conformance Property Cases Total
Array 42 552/552 ✓ 1,000 (2×500) 1,594
Database 35 182/182 ✓ 3,500 (7×500) 3,717
Date 373 418/418 ✓ 2,500 (5×500) 3,291
Engineering 245 886/888 ⚠ 5,500 (11×500) 6,633
Filter 11 81/81 ✓ 4,500 (9×500) 4,592
Financial 149 1,208/1,208 ✓ 2,000 (4×500) 3,357
Info 0 256/256 ✓ 4,500 (9×500) 4,756
Logical 121 267/267 ✓ 3,500 (7×500) 3,888
Lookup 69 393/393 ✓ 1,000 (2×500) 1,462
Math 545 2,006/2,006 ✓ 8,000 (16×500) 10,551
Operator 87 251/251 ✓ 7,500 (15×500) 7,838
Parser 83 93/93 ✓ 4,000 (8×500) 4,176
Query 37 37
Statistical 529 3,191/3,191 ✓ 5,000 (10×500) 8,720
Text 327 803/804 ⚠ 4,000 (8×500) 5,131
Timezone 47 47
Volatile 0 3,500 (7×500) 3,500
Web 29 59/59 ✓ 6,000 (12×500) 6,088
Total 2,997 10,646/10,649 66,000 (132×500) ~79,646

✓ = 100% passing · ⚠ = known deviation · The ~79,646 total counts formula evaluations (each conformance row and each property case = 1). GitHub Checks reports 4,034 Rust test functions: 2,997 unit + 159 property functions (shown as cases above) + 878 conformance/integration.

@hhimanshu
hhimanshu merged commit 5995b1c into main Aug 29, 2026
9 checks passed
@hhimanshu
hhimanshu deleted the feat/965-recalc-work-assertions-chain-bench branch August 29, 2026 02:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: benchmark suite measures no dependency chain, and asserts no work

1 participant