test(workbook): assert recalc work, and add the missing chain fixture - #967
Merged
Merged
Conversation
The benchmark suite had two gaps: no fixture was ever more than one level
deep, and no benchmark ever checked how much work a recalc actually did.
Add `crates/workbook/tests/recalc_work_tests.rs`, which rebuilds the
benchmark shapes at small sizes and asserts recalculation work as exact
counts — the dirty closure from `recalc_incremental_measured`, and the
changes emitted. Every expectation is a literal derived from the fixture's
construction with the arithmetic in a comment, never an expression
recomputing what the code under test computes. Over-dirtying previously
showed up only as a slightly slower benchmark, indistinguishable from
noise; it is now a hard failure naming an exact number.
The counts show `incremental_recalc/block_subtotals_edit_root` measures the
cheapest edit its fixture admits, not the fan-out its builder describes:
the windows are `A{r}:A{r+99}` at r = 1, 21, 41, …, so row 1 is covered by
exactly one of them. Both numbers are now pinned — 1 for the root edit, 5
for an interior edit at A101.
Add `build_chain(n)` plus `full_recalc/chain`,
`incremental_recalc/chain_edit_root` and `incremental_recalc/chain_edit_leaf`
at n = 1000 and 5000. Every previously existing fixture is depth 1;
`multi_sheet_cross` only looks like a counter-example, since its formulas
read a literal on the first tab. The chain carries one tail literal so a
cheap edit is expressible: in a pure chain the only non-formula cell is A1,
and writing a literal over a chain cell would destroy a formula node and
invalidate the graph cache, making the leaf case time a cold rebuild
against the root case's warm walk.
Deep chains hit no recursion or stack limit — evaluation order is Kahn's
algorithm, the closure walk is a queue, and a formula reads stored values
rather than recursing — so a 5000-link chain is asserted to keep it that
way.
Record baselines for the six new benchmarks per the method baselines.json
documents: same machine as its `recorded_on` field, best of 5 full bench
runs on one unmodified binary. No pre-existing entry moved.
closes #965
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAt4Keq1m4fEyHPmPjHSJ7
Contributor
Test Coverage by Category
✓ = 100% passing · ⚠ = known deviation · The ~79,646 total counts formula evaluations (each conformance row and each property case = 1). GitHub Checks reports 4,034 Rust test functions: 2,997 unit + 159 property functions (shown as cases above) + 878 conformance/integration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #965
The workbook benchmark suite had two gaps. This PR closes both, in that order: the assertions first (they change no CI gate), then the missing fixture.
1. Work is now asserted, not only timed
New file
crates/workbook/tests/recalc_work_tests.rs. It rebuilds the benchmark file's fixture shapes at small sizes and asserts, as exact counts, how much work a recalc of each one does — the same instrumentdepgraph_range_index_tests.rsalready uses for cells-examined-per-range-reference.Two numbers per shape, because they are not the same number:
Workbook::recalc_incremental_measured— the formula cells an incremental recalc actually recomputed. This is the work.Vec<Change>::len()— recalc's write-back skips a cell whose recomputed value equals its stored one (if old == new { continue; }), so this counts formulas whose value moved. It equals the closure only because a never-recalculated workbook storesValue::Emptyin every formula cell, which is why a first full recalc's change count is exactly the fixture's formula count.Every expectation is derived from the fixture's construction and written as a literal with the arithmetic in a comment, never as an expression recomputing what the code under test computes.
independent(50)A1row_totals(20)A1block_subtotals(200)A1block_subtotals(200)A101tall_sparse(30)A1multi_sheet(4,10)S0!A1multi_sheet_cross(4,10)S0!A1chain(50)A1chain(50)B51chain(5000)A1/B5001Every one of these passed on the first run: no shape dirties more than its construction implies, so there is no over-dirtying to report.
What the counts did reveal
incremental_recalc/block_subtotals_edit_rootis measuring the cheapest edit its fixture admits, not the fan-out its builder describes. The subtotal windows areA{r}:A{r+99}atr = 1, 21, 41, …, so an interior source row sits inside five of them — but row 1 sits inside exactly one, because no window starts above row 1. The benchmark editsA1. The test therefore pins both: 1 for the root edit, and 5 for an interior edit atA101, which is the number that actually exercises overlapping range-precedent invalidation. Nothing is wrong with the benchmark; it just measures less than its name suggests, and now that is written down.2. A chain fixture, and benchmarks for it
build_chain(n):A1a literal,A2 = =A1+1,A3 = =A2+1, … throughA{n+1}— one linear chain,nformulas deep. Every previously existing fixture is exactly one level deep: a formula reads a literal, and nothing reads that formula.multi_sheet_crosslooks like a counter-example and is not — its formulas read a literal on the first tab, so it is depth 1 with high fan-out. Propagation through a dependency chain had no fixture at all.New benchmarks:
full_recalc/chain/{1000,5000}— the same formula counts asindependent/{1000,5000}, arranged as one chain instead of N unrelated pairs, so the pair isolates what graph topology costs.incremental_recalc/chain_edit_root/{1000,5000}— editingA1dirties the whole chain: worst-case propagation.incremental_recalc/chain_edit_leaf/{1000,5000}— dirties exactly 1: best case. Together they bracket propagation cost, and the ratio between them is what should grow withn.What the chain benchmarks measure (recorded numbers)
Best of 5, Apple M1 Max, ns/iter:
full_recalc/chainfull_recalc/independent(existing, same formula count)incremental_recalc/chain_edit_rootincremental_recalc/chain_edit_leafTwo things fall straight out of the bracket, neither of which any existing benchmark could show:
A chain costs about what N independent formulas cost.
full_recalc/chain/5000is ~8% faster thanfull_recalc/independent/5000despite being 5,000 levels deep instead of 1. Depth is not, on its own, expensive — which is worth having recorded, because it is the assumption a future ordering change would silently break.The best case is not cheap, and it scales with the workbook rather than the edit.
chain_edit_leafdirties exactly 1 formula at both sizes, yet it costs 1.02ms at n=1000 and 5.20ms at n=5000 — 5.1x more work for a dirty set that did not grow. That is the incremental path's per-formula-cell sweeps (the volatile sweep overgraph.formula_cells(),snapshot_formula_values, and spill-occupancy seeding), allO(formula cells)by construction and documented as such inrecalc.rs. So this is a known design characteristic rather than a defect — but until now nothing measured it, because every other incremental benchmark varies the dirty set and the workbook size together.chain_edit_leafholds the dirty set at 1 while the workbook grows, which isolates that floor and now gates it.The root/leaf ratio is 3.7x at n=1000 and 3.9x at n=5000. If a future change lets that ratio collapse, propagation has stopped being proportional to the dirty set.
The one design decision worth reviewing
The chain carries one extra literal: the last link is
A{n+1} = =A{n}+B{n+1}, withB{n+1}read by nothing else.Without it there is no cheap edit to measure. In a pure chain the only non-formula cell is
A1, and editing it dirties everything; writing a literal over a chain cell instead destroys a formula node, and any formula write invalidates the dependency-graph cache (see thegraph_cachemodule docs).chain_edit_leafwould then be timing a cold graph rebuild whilechain_edit_roottimed a warm walk — the two would not bracket anything, and the mismatch would be invisible in the numbers. EditingB{n+1}is a literal-over-literal write in a table-free workbook, so it is structure-preserving: same fixture, same warm graph, only the distance travelled differs.Deep chains
No recursion or stack limit. A 5,000-link chain recalculates in the test suite without incident, and it is the shape that would find one: evaluation order is Kahn's algorithm (iterative), the dirty-closure walk is a queue, and a formula reads its precedents' stored values rather than recursing into them, so no stage of a recalc recurses with the chain's depth. The 5,000-deep case is asserted in the test file so that stays true.
Baselines
The perf gate treats a benchmark with no baseline entry as a failure (
UNGATED), so the six new benchmarks needed recorded entries. They were recorded to the methodbaselines.jsondocuments, on the machine itsrecorded_onfield names — Apple M1 Max (10 core), macOS 14.4, rustc 1.94.1, release profile — best of 5 full bench runs, normalised tocalibration/hash_allocmeasured across the same five runs. That is the count the file's ownrecorded_onandnotefields specify; the best-of-3 recordings the note mentions are flagged there as exceptions, so they are not the method to copy.Only the six new entries were added. Every pre-existing number is byte-identical; nothing in this PR moves them, so any future drift stays attributable.
Explicitly not in scope
template.clone()stays inside the timed closures. Moving it toiter_batchedis the right eventual fix, but it re-records every existing baseline and belongs in its own PR so the change in numbers is attributable to it.Verification
cargo test -p truecalc-workbook— 445 passed (52 suites)cargo clippy --workspace --exclude truecalc-python -- -D warnings— cleancargo bench -p truecalc-workbook --bench workbook_perf -- --test— every benchmark, new ones included, runs once successfullyNeed help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.