Repository navigation
fix(input): load models and caches the same way whatever the job count - #186
Merged
Merged
Conversation
…tive model When jobs are automatic, the default, files compile in parallel and each one changes the working directory of the whole process to its compile command's directory (#184). The changes interleave: a file can compile from another file's directory, and the analyzer can end in one of them, where a relative resource model or cache directory no longer resolves. check_resource_model_across_compile_directories generates 8 files, each in a directory of its own with the one header it includes through a relative -I, and a resource model, given by a relative path, that alone defines the leak every file has. It requires: - with automatic jobs, --jobs=1 and --jobs=4, three runs each with empty caches: the model loads, the 8 leaks and no other are reported, and the compile IR and resource caches are under the start directory; - the relative model and cache paths anchored to the start directory (--print-effective-config); - no compile with a changed working directory, under automatic jobs, for compile commands that need none (--timing). On main, automatic jobs fail every run ('unit6.h' file not found, no output), --jobs=4 writes caches under the compile directories, the paths stay relative, and automatic jobs change the directory for every file. Refs #184
…y change A compile that needs its compile command's directory changes the working directory of the whole process (#184). Three things let that leak into the rest of the run: - InputPipeline chose its path with config.jobs > 1, but jobs is 0 when automatic, the default: parallel loads then changed the directory for every file. It now uses resolveConfiguredJobs, moved out of AnalyzerApp into StackUsageAnalyzer for both, so parallel runs try a compile from the current directory first, as with an explicit job count. - The change was made before taking the compile lock, and the separate lock of the fallback covered fallbacks only. Two threads could save and restore the directory in an interleaved order, compile in a directory not their own, and leave the process in one. The compile lock is now taken before the change and held until it is undone; the fallback lock goes. - The resource, escape and buffer models, the compile IR and resource summary caches, the --dump-ir output and the compile database were opened through paths relative to whatever the directory was at that time. They are now anchored to the directory the analyzer starts from, once the arguments are parsed. A compile from its command's directory still changes the directory of the whole process while it runs; removing that needs compilerlib to compile against a working directory of its own. Refs #184
…mand's Review of #184: a compile command's relative -I resolves against its directory, but the analyzer first compiled from the current directory and only used the command's directory when that compile failed. When both directories hold the header, the start directory's one was used, with no error: on main with --jobs>1, and with automatic jobs since the fix for the working-directory race. check_resource_model_across_compile_directories now puts, in the start directory, a header of the same name for every file, one that drops the leak. Each mode must still report the 8 leaks. On 3b2260e, automatic jobs and --jobs=4 report none; --jobs=1 reports all 8. The check no longer requires automatic jobs to compile commands with absolute paths without changing the working directory: that was the behavior that picks the wrong header. Refs #184
A compile command whose directory differs from the current one was first compiled from the current directory, and from its own only when that failed. Relative paths then resolved against the wrong directory without any error when both held the file: another header, another IR. This "optimistic fast path" (2289afb) predates the lock that serializes every compile (967be74), so it no longer ran anything in parallel. A command with a directory is now always compiled there, under the compile lock, whatever the job count. The retry and its log suffix go, and resolveConfiguredJobs returns to AnalyzerApp, its only user again. Refs #184
Review of #184: each analysis loads its model itself, logs a failure to stderr and goes on without it. A resource, escape or buffer model given on the command line but missing ends with code 0 and no finding, and run_code_analysis.py reports "CI gate passed". check_unreadable_model_fails gives each of the three options a missing file and a directory: the analyzer must exit with a nonzero code naming the path. run_code_analysis.py given a missing resource model must fail. On 274c1c1 all six runs exit with 0 and the CI script passes. A model that can be read but is malformed keeps its current handling (check_escape_model_rejects_unsupported_brackets); it is a separate question. Refs #184
The resource, escape and buffer models are read by the analyses that use them, which log a failure and run without the model. A missing model then passed for a run that found nothing, and run_code_analysis.py for a passing CI gate. Before any input is loaded, every model given on the command line or in the config file must now be a readable file; otherwise the run stops with "Cannot read <kind> model file: <path>" and exit code 1, which the CI script already turns into a failure. This check covers missing and unreadable files only. A model that can be read but holds malformed rules is still reported on stderr and skipped by the analysis: a separate question, noted in the README. Refs #184
SizzleUnrlsd
force-pushed
the
184-deterministic-working-directory
branch
from
October 8, 2026 02:20
5951161 to
e2e9f87
Compare
…irectories Review of #187: a compile command with a directory of its own is now always compiled there, and the change of directory holds for the whole process. Meanwhile, another thread resolves its own relative input path against that directory, to find its compile command and to compile it: with 24 files given by relative paths, --jobs=4 fails with "No compile command found for: src/unit03.c", where --jobs=1 and main succeed. check_resource_model_across_compile_directories now runs every mode with the files given by absolute paths, then by paths relative to the start directory, each run with empty caches of its own. Each must report the 8 leaks and name each file as it was given (#124). On e2e9f87, 5 of the 6 parallel runs with relative paths fail; --jobs=1 and absolute paths pass. Refs #184
…tarted in A compile command with a directory of its own is compiled from there, and the change of directory holds for the whole process while it lasts. loadModuleForAnalysis resolved a relative input against the current directory: to find its compile command, to compile it, for its compile IR cache key and to read it. Another thread could then resolve it from a compile command's directory, and with --jobs=4 an input given as src/unit03.c had no compile command. The directory the analyzer starts from is now recorded with the other anchored paths, as AnalysisConfig::inputBaseDir, before any input is loaded. Every access to an input goes through its path resolved from there. The path given is kept for messages and results (#124), and a file without a compile command is still compiled under the name given: no other thread changes the directory while a compile holds the lock. Refs #184
…compiled again Review of #187: the compile IR cache keeps the IR that the previous pipeline built from the current directory, with a header of the same name as the compile command's own, under schema compile-ir-cache-v2. A run that now compiles from the command's directory reused it: 7 where a run with an empty cache gives 42. check_compile_ir_cache_rebuilds_previous_schema rewrites the entry of the current key as the previous pipeline left it: the previous schema, and the IR built with the start directory's header, with the sources and dependencies left valid. The next run must compile again, give 42 and rewrite the entry, and the run after it must reuse that entry. An entry stored under a previous key is never looked up again; the schema in the key covers it, not this check. On c5911fb the stale entry is reused: 7. Refs #184
A compile command with a directory of its own is now always compiled from there, so an unchanged command can give other IR than before: the IR of a file whose header of the same name also sits in the start directory. The compile IR cache kept the previous IR under the same key and accepted it, and the fix had no effect on a cache filled before it. The schema becomes compile-ir-cache-v3, in the cache key and in the metadata of each entry: previous entries are no longer looked up, and an entry whose metadata names the previous schema is compiled again. Refs #184
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 1 of #184: the working-directory race, and four defects found while reviewing the fix. Part 2 (the cost of the ownership summaries) is #187, which carries these commits and is the PR planned for delivery.
Problem
A compile that needs its compile command's directory changes the working directory of the whole process (
ScopedCurrentPath). With automatic jobs, the default, save/restore pairs interleaved: a file could compile from another file's directory, and the process could stay in one of them. Relative paths opened later, the resource model and the caches among them, then failed. In the CI self-analysis, the resource model is passed relatively. Locally, on main f065508, every finished run of a chunk printed "cannot open model file" 14 to 27 times. The effect on the CI runs themselves is not established.The reviews of the fix found four more defects:
--jobs>1, and with automatic jobs after the first fix.run_code_analysis.pythen reports "CI gate passed".--jobs=4failed with "No compile command found for: src/unit03.c", wheremainsucceeded.Changes
--dump-irand the compile database are anchored there once the arguments are parsed.AnalysisConfig::inputBaseDir, with the other anchored paths, before any input loads. Every access to an input goes through its path resolved from there: compile command lookup, compile, cache keys, reading. The path given is kept for messages and results (fix(report): spell every result's file as the input was given #124).Cannot read <kind> model file: <path>and exit code 1, and the CI script fails.This check covers missing and unreadable files only. A model that can be read but holds malformed rules is still reported on stderr and skipped by the analysis. That is a separate question, noted in the README. A compile from its command's directory still changes the directory of the whole process while it runs. Removing that requires compilerlib to compile against a working directory of its own: clang's
-working-directorywould not avoid it, as compilerlib gives the driver the real file system, whose working directory is the process's.Tests
check_resource_model_across_compile_directories. It uses 8 files, each in its own compile directory, with the header it includes through a relative-I. A model given relatively alone defines each file's leak, and the start directory holds, for every file, a header of the same name that drops the leak. With automatic jobs,--jobs=1and--jobs=4, with the files given by absolute and then by relative paths, three runs each with empty caches of their own, the check requires the 8 leaks and nothing else, each file named as it was given. It also requires caches under the start directory and anchored paths in--print-effective-config.--jobs=4reports no leak.--jobs=4report no leak.check_compile_ir_cache_rebuilds_previous_schema. The start directory and the compile command's directory hold a header of the same name, giving 7 and 42. After a run with an empty cache (42), the check rewrites the entry as the previous pipeline left it: the previous schema and 7, with sources and dependencies valid. The next run must compile again, give 42 and rewrite the entry, and the run after it must reuse that entry. It covers the schema in the metadata; an entry under a previous key is never looked up. On c5911fb the stale entry is reused (7).check_unreadable_model_fails. A missing file and a directory, for each of the three model options, must give a nonzero code that names the path.run_code_analysis.pywith a missing model must fail. On 274c1c1, all six runs exit with 0 and the script passes.Locally (macOS arm64, on
mainc1e60b8, head 7962221),run_test.pypasses 2383/2383, both without and with CI's compile IR cache option, and both unit-test binaries pass.CI
The Self-analysis step on this PR's head (7962221) took 11 min 21 s against its 15-minute limit (8 min 31 s on e2e9f87). The ownership summaries are still the previous ones; #187 brings this down. Since #190 moved the resource summary cache to schema v5, the cache CI restores holds no entry this run can read, so the run is cold.
Code scanning
AnalyzerApp.cppis no longer changed by the first fix.main.Refs #184