perf(ownership): compute the ownership summaries with fewer solves and rounds by SizzleUnrlsd · Pull Request #187 · CoreTrace/coretrace-stack-analyzer

SizzleUnrlsd · 2026-10-07T19:16:48Z

Part 2 of #184: the cost of the ownership summaries. It builds on part 1 (#186), which makes the resource model load in every run, compiles every command in its own directory, resolves relative inputs from the start directory, moves the compile IR cache to schema v3, and stops when a requested model cannot be read. CI only runs for pull requests into main, so this PR targets main and also carries #186's ten commits. It is the PR planned for delivery: merging it also delivers #186.

Problem

With the resource model loaded, the self-analysis spends most of its time in computeOwnershipSummaries:

computeSummary solved each function once per parameter (and pointee path) and per entry state;
the cross-function fixed point recomputed every summary in every round, even when nothing it consulted had changed.

One 207-function module cost about 90 s per round.

Changes

A, computeSummary: one solve per entry state, with every parameter in that state. This relies on resources being independent of each other. Fresh resources come from the all-Owned solve; a function without parameters takes a single solve. On the work-test function: 4 solves and 16 block visits, against 25 and 100 before.
B, computeSummaryFixpoint: keeps a summary while every summary it consulted is unchanged. The order, the 16-round budget and the global incomplete marking when the budget runs out are unchanged.
- Every internal callee is read through consult.
- sameSummary compares the same fields as the previous comparison.
Counters: SummaryWork, SummaryFixpointWork and OwnershipResult::blockVisits only count work.

Validation

Unit tests (ownership_engine_unit_tests):
- A gives the same transformers as one solve per parameter, on aliases, pointee paths, loops, exceptional exits, the smallest converging budget, and fresh returns and out-arguments.
- The fixed point gives the same summaries as recomputing everything, on independent functions, chains, recursion, mutual recursion, a change of the incomplete status alone, an input that changes later in a round, and an exhausted budget.
- Both work tests fail on the previous algorithm.
Differential checks, at every summary computation (throwaway builds; same facts and external summaries):
- A against the previous computation: fixtures 2 060, Lua 81 483 (stopped at 50 min) and self-analysis 88 458 summaries, 0 mismatches.
- A+B against A: 230 fixed points and 15 005 final summaries, 0 mismatches. Rounds are equal (1 037), with 27 055 computations against 90 720.
Incomplete summaries: 21 final summaries of 6 large functions are incomplete, with and without this change. All come from the per-solve iteration limit, none from the round budget.
Final diagnostics against part 1 alone (previous algorithm, model loaded on both sides, same corpus at 112ffcf, before the review fixes of fix(input): load models and caches the same way whatever the job count #186, with the same input layer on both sides):
- fixtures, 419 units alone, default and z3 passes: 838 runs, 1 275 diagnostics on each side, 0 differences;
- self-analysis, the 4 CI chunks with the CI arguments: 70 diagnostics on each side (17, 18, 26, 9 per chunk), 0 differences;
- Lua 5.4.8 together: the run with the previous algorithm never finished. It was still going after 8 h 54 min, and stopped with the session that launched it. In its place, A+B was compared with A (397b28b) on the same 34 files, with the same options and the model loaded:
  - 511 diagnostics on each side, 0 differences;
  - the max stacks of all 1 096 functions are identical;
  - A took 3 677 s, A+B 2 412 s, run at the same time.
  Together with A's summary-level equivalence to the previous algorithm on Lua (81 483 summaries, 0 mismatches), this closes the chain.
On main c1e60b8 (with fix(analysis): resolve calls through constructor and destructor aliases #190), head ecf7f07 (also checked on 4f5d3a8 before fix(input): load models and caches the same way whatever the job count #186's last two fixes):
- run_test.py passes 2383/2383, both without and with CI's compile IR cache option, and both unit-test binaries pass (macOS arm64);
- every fixture, analyzed alone in the default and z3 configurations, gives the same diagnostics as main: 423 units for the host, and for x86_64-unknown-linux-gnu the 274 that compile without a Linux sysroot.

CI time of the Self-analysis step (Linux, 15-minute limit)

Run	Commit	Resource summary cache	Time
this PR, rebased on `c1e60b8`	`ecf7f07` (and `4f5d3a8`)	restored, but written under schema v4: no entry readable by v5, so cold	2 min 48 s
this PR, before the rebase	`8f74e11`	restored	1 min 20 s
measurement, `workflow_dispatch` on the temporary branch `184-cold-ci-measure`	`8f74e11` plus a workflow-only commit (`03e27ab`)	new directory under `RUNNER_TEMP`, absent before the step	2 min 17 s
#186 alone, previous summaries, rebased	`7962221` (`e2e9f87`)	as for this PR: cold	11 min 21 s (8 min 31 s)
#186 alone, previous summaries	`5951161`	restored	10 min 53 s

The measurement run used the same arguments as this step, plus --analyzer-arg=--resource-summary-cache-dir=$RUNNER_TEMP/resource-summary-cache-cold-<run>:

the step checked that the directory did not exist before the analysis;
156 cache files were written to it during the run;
the restored cache (125 files) was left unused.

The compile IR cache is not used by this step. Before the rebase, both runs of this PR found 0 errors, 9 warnings and 63 infos; after it, 0 errors, 8 warnings and 63 infos. The warning that goes is alert 251, which #190 removes.

Code scanning

New alerts, all dismissed as false positives:
- 256, ConstParameterNotModified.Pointer on computeSummary's SummaryWork* work, which the lambda solveCounted modifies (False positive: ConstParameterNotModified.Pointer on a pointee modified in a lambda #188);
- 257 to 260, UninitializedLocalRead on ParamTransformer normal{} and exceptional{}, value-initialized, in the test's reference loop (False positive: UninitializedLocalRead on a value-initialized local #189).
Alerts of main that this PR removes: 158 to 161, the same UninitializedLocalRead false positive on the previous computeSummary loop. That loop moved into the unit test as the reference, which moved the alerts.
No longer present: alerts 254 and 255 (see fix(input): load models and caches the same way whatever the job count #186), and alert 251, which fix(analysis): resolve calls through constructor and destructor aliases #190 removes.
Already on main: every other alert listed here.

Closes #184

…tive model When jobs are automatic, the default, files compile in parallel and each one changes the working directory of the whole process to its compile command's directory (#184). The changes interleave: a file can compile from another file's directory, and the analyzer can end in one of them, where a relative resource model or cache directory no longer resolves. check_resource_model_across_compile_directories generates 8 files, each in a directory of its own with the one header it includes through a relative -I, and a resource model, given by a relative path, that alone defines the leak every file has. It requires: - with automatic jobs, --jobs=1 and --jobs=4, three runs each with empty caches: the model loads, the 8 leaks and no other are reported, and the compile IR and resource caches are under the start directory; - the relative model and cache paths anchored to the start directory (--print-effective-config); - no compile with a changed working directory, under automatic jobs, for compile commands that need none (--timing). On main, automatic jobs fail every run ('unit6.h' file not found, no output), --jobs=4 writes caches under the compile directories, the paths stay relative, and automatic jobs change the directory for every file. Refs #184

…y change A compile that needs its compile command's directory changes the working directory of the whole process (#184). Three things let that leak into the rest of the run: - InputPipeline chose its path with config.jobs > 1, but jobs is 0 when automatic, the default: parallel loads then changed the directory for every file. It now uses resolveConfiguredJobs, moved out of AnalyzerApp into StackUsageAnalyzer for both, so parallel runs try a compile from the current directory first, as with an explicit job count. - The change was made before taking the compile lock, and the separate lock of the fallback covered fallbacks only. Two threads could save and restore the directory in an interleaved order, compile in a directory not their own, and leave the process in one. The compile lock is now taken before the change and held until it is undone; the fallback lock goes. - The resource, escape and buffer models, the compile IR and resource summary caches, the --dump-ir output and the compile database were opened through paths relative to whatever the directory was at that time. They are now anchored to the directory the analyzer starts from, once the arguments are parsed. A compile from its command's directory still changes the directory of the whole process while it runs; removing that needs compilerlib to compile against a working directory of its own. Refs #184

…mand's Review of #184: a compile command's relative -I resolves against its directory, but the analyzer first compiled from the current directory and only used the command's directory when that compile failed. When both directories hold the header, the start directory's one was used, with no error: on main with --jobs>1, and with automatic jobs since the fix for the working-directory race. check_resource_model_across_compile_directories now puts, in the start directory, a header of the same name for every file, one that drops the leak. Each mode must still report the 8 leaks. On 3b2260e, automatic jobs and --jobs=4 report none; --jobs=1 reports all 8. The check no longer requires automatic jobs to compile commands with absolute paths without changing the working directory: that was the behavior that picks the wrong header. Refs #184

A compile command whose directory differs from the current one was first compiled from the current directory, and from its own only when that failed. Relative paths then resolved against the wrong directory without any error when both held the file: another header, another IR. This "optimistic fast path" (2289afb) predates the lock that serializes every compile (967be74), so it no longer ran anything in parallel. A command with a directory is now always compiled there, under the compile lock, whatever the job count. The retry and its log suffix go, and resolveConfiguredJobs returns to AnalyzerApp, its only user again. Refs #184

Review of #184: each analysis loads its model itself, logs a failure to stderr and goes on without it. A resource, escape or buffer model given on the command line but missing ends with code 0 and no finding, and run_code_analysis.py reports "CI gate passed". check_unreadable_model_fails gives each of the three options a missing file and a directory: the analyzer must exit with a nonzero code naming the path. run_code_analysis.py given a missing resource model must fail. On 274c1c1 all six runs exit with 0 and the CI script passes. A model that can be read but is malformed keeps its current handling (check_escape_model_rejects_unsupported_brackets); it is a separate question. Refs #184

The resource, escape and buffer models are read by the analyses that use them, which log a failure and run without the model. A missing model then passed for a run that found nothing, and run_code_analysis.py for a passing CI gate. Before any input is loaded, every model given on the command line or in the config file must now be a readable file; otherwise the run stops with "Cannot read <kind> model file: <path>" and exit code 1, which the CI script already turns into a failure. This check covers missing and unreadable files only. A model that can be read but holds malformed rules is still reported on stderr and skipped by the analysis: a separate question, noted in the README. Refs #184

…irectories Review of #187: a compile command with a directory of its own is now always compiled there, and the change of directory holds for the whole process. Meanwhile, another thread resolves its own relative input path against that directory, to find its compile command and to compile it: with 24 files given by relative paths, --jobs=4 fails with "No compile command found for: src/unit03.c", where --jobs=1 and main succeed. check_resource_model_across_compile_directories now runs every mode with the files given by absolute paths, then by paths relative to the start directory, each run with empty caches of its own. Each must report the 8 leaks and name each file as it was given (#124). On e2e9f87, 5 of the 6 parallel runs with relative paths fail; --jobs=1 and absolute paths pass. Refs #184

…tarted in A compile command with a directory of its own is compiled from there, and the change of directory holds for the whole process while it lasts. loadModuleForAnalysis resolved a relative input against the current directory: to find its compile command, to compile it, for its compile IR cache key and to read it. Another thread could then resolve it from a compile command's directory, and with --jobs=4 an input given as src/unit03.c had no compile command. The directory the analyzer starts from is now recorded with the other anchored paths, as AnalysisConfig::inputBaseDir, before any input is loaded. Every access to an input goes through its path resolved from there. The path given is kept for messages and results (#124), and a file without a compile command is still compiled under the name given: no other thread changes the directory while a compile holds the lock. Refs #184

…compiled again Review of #187: the compile IR cache keeps the IR that the previous pipeline built from the current directory, with a header of the same name as the compile command's own, under schema compile-ir-cache-v2. A run that now compiles from the command's directory reused it: 7 where a run with an empty cache gives 42. check_compile_ir_cache_rebuilds_previous_schema rewrites the entry of the current key as the previous pipeline left it: the previous schema, and the IR built with the start directory's header, with the sources and dependencies left valid. The next run must compile again, give 42 and rewrite the entry, and the run after it must reuse that entry. An entry stored under a previous key is never looked up again; the schema in the key covers it, not this check. On c5911fb the stale entry is reused: 7. Refs #184

A compile command with a directory of its own is now always compiled from there, so an unchanged command can give other IR than before: the IR of a file whose header of the same name also sits in the start directory. The compile IR cache kept the previous IR under the same key and accepted it, and the fix had no effect on a cache filled before it. The schema becomes compile-ir-cache-v3, in the cache key and in the metadata of each entry: previous entries are no longer looked up, and an entry whose metadata names the previous schema is compiled again. Refs #184

…e per parameter computeSummary solves the whole function 4 * (parameters + pointee paths) + 1 times (#184): 101 solves for a function with 25, which makes the self-analysis run for minutes once the resource model loads. - A reference, written with the public solve(), builds the transformers as computeSummary does today: one solve per parameter and entry state, the other parameters Owned, and one with every parameter Owned for the fresh resources. computeSummary must give the same transformers, and be incomplete exactly when the reference is, with aliases, pointee paths, a loop, exceptional exits, and at the smallest budget the reference converges with and just below it. - Fresh resources beside several parameters: a returned one and one handed through a pointee, guaranteed then conditional. - Work: SummaryWork counts the solves and block visits of a summary. Six parameters and pointees take four solves and at most four times the block visits of one solve; a function without parameters, one. Refs #184

SummaryWork, an optional out-parameter of computeSummary, counts the solves a summary takes and the blocks their worklists process; solve() reports its own block visits. They only observe: no summary depends on them. With the summary computed as before, the test of a function with six parameters and pointees reports 25 solves and 100 block visits, where one solve visits 4 blocks. Refs #184

…y state computeSummary solved the whole function once per parameter and entry state, the other parameters Owned, and once more with all of them Owned for the fresh resources: 4 * (parameters + pointee paths) + 1 solves, 101 for a function with 25 (#184). It now solves once per entry state, with every parameter in that state. Each parameter has a resource of its own; an event acts on a resource from that resource's own state, what a location holds never depends on any state, and paths join resource by resource. Each parameter therefore ends as in a solve of its own with the others Owned. The solve with every parameter Owned is the one the fresh resources were read off; without parameters it remains the only one. Four solves instead of 4 * P + 1; the test of six parameters and pointees goes from 25 solves and 100 block visits to 4 and 16. Refs #184

…es changed computeOwnershipSummaries recomputes every function of the module each round, up to 16, although a round may change a single summary (#184): the self-analysis computes 88 458 summaries for 53 files. A reference written in the test runs that fixed point: every function recomputed in index order each round until a round changes none, every summary incomplete past the budget. computeSummaryFixpoint must give the same summaries with no more computations: independent functions, a chain against the index order, recursion, mutual recursion, a change of the incomplete status alone, a consulted summary that changes later in the same round while the computations that consulted it changed nothing, and an exhausted budget, which takes exactly 16 rounds. Independent functions are computed once each, and the chain fewer times than every function each round. SummaryFixpointWork counts the computations, the reused summaries and the rounds; sameSummary compares summaries, incompleteness included. Refs #184

…ryFixpoint The fixed point of computeOwnershipSummaries moves into the ownership engine, as computeSummaryFixpoint, which knows nothing of LLVM. It runs the same rounds in the same order, starts from the same empty summaries, stops after the first round that changes nothing, and past 16 rounds marks every summary incomplete. The summaries of the module's functions are now read through consult; those of other modules, which do not change here, are still read directly. sameSummary, the comparison of summaries, moves into the engine with the same fields; ownershipSummaryEquals keeps its interface and calls it. Every function is still recomputed each round: the work tests report 10 computations for 5 independent functions and 42 for the chain. Refs #184

computeSummaryFixpoint recomputed every function each round. A summary is now kept when none of the summaries its last computation consulted has changed since: computing it again would give the same summary, as the computation depends on nothing else that changes. Each summary has a version raised when it changes, its incompleteness included; each computation records the version of every summary it consults, its own when recursive. The rounds, their order, the stop after a round that changes nothing and the incomplete marking after 16 rounds are unchanged. The work tests go from 10 to 5 computations for 5 independent functions and from 42 to 21 for the chain. Refs #184

SizzleUnrlsd added the bug Something isn't working label Oct 7, 2026

SizzleUnrlsd self-assigned this Oct 7, 2026

github-advanced-security Bot found potential problems Oct 7, 2026

View reviewed changes

This was referenced Oct 7, 2026

False positive: ConstParameterNotModified.Pointer on a pointee modified in a lambda #188

Open

False positive: UninitializedLocalRead on a value-initialized local #189

Open

SizzleUnrlsd force-pushed the 184-ownership-summary-cost branch from 28b5e40 to 8f74e11 Compare October 7, 2026 21:52

This was referenced Oct 7, 2026

fix(input): load models and caches the same way whatever the job count #186

Merged

fix(analysis): resolve calls through constructor and destructor aliases #190

Merged

SizzleUnrlsd added 6 commits October 8, 2026 04:11

SizzleUnrlsd force-pushed the 184-ownership-summary-cost branch from 8f74e11 to 4f5d3a8 Compare October 8, 2026 02:21

This was referenced Oct 8, 2026

CI: the resource summary cache is never saved again, so every self-analysis runs cold #193

Closed

ci: save the resource summary cache after every self-analysis #194

Merged

SizzleUnrlsd added 10 commits October 8, 2026 18:16

SizzleUnrlsd force-pushed the 184-ownership-summary-cost branch from 4f5d3a8 to ecf7f07 Compare October 8, 2026 16:30

SizzleUnrlsd marked this pull request as ready for review October 8, 2026 17:12

SizzleUnrlsd merged commit 1598979 into main Oct 8, 2026
15 of 16 checks passed

SizzleUnrlsd deleted the 184-ownership-summary-cost branch October 8, 2026 17:13

This was referenced Oct 15, 2026

Release v0.23.0: CoreTrace needs the ownership summary speed-up of #187 #227

Closed

build(deps): update coretrace-stack-analyzer to v0.23.1 CoreTrace/coretrace#194

Merged

Navigation Menu

Repository navigation

perf(ownership): compute the ownership summaries with fewer solves and rounds - #187

perf(ownership): compute the ownership summaries with fewer solves and rounds#187
SizzleUnrlsd merged 16 commits into
mainfrom
184-ownership-summary-cost

SizzleUnrlsd commented Oct 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

SizzleUnrlsd commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

Validation

CI time of the Self-analysis step (Linux, 15-minute limit)

Code scanning

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SizzleUnrlsd commented Oct 7, 2026 •

edited

Loading