Skip to content

graph: refuse an output under a file or onto a directory at parse time - #497

Open
typeless wants to merge 2 commits into
mainfrom
fix/495-output-under-source-file
Open

typeless wants to merge 2 commits into
mainfrom
fix/495-output-under-source-file

Conversation

@typeless

Copy link
Copy Markdown
Owner

Closes #495.

Problem

Whether putup refused an output under a non-directory depended on rule order, and an output could claim a path another output had made a directory. Both ran the command and failed in the shell, where tup refuses at parse time.

Change

  • Output or group under a source file (in-tree builds). Before binding an output or creating a group, the builder stats each component of the directory that no node holds yet, and binds a regular file as a File node. graph: refuse unresolvable and non-directory parents #494's parent check in add_file_node then refuses it, whichever rule comes first. Variant builds skip this: real tup accepts both an output and a group under a source file there.
  • Output onto a directory. After binding, an output whose node is not a file, a generated file or a ghost is refused with tup's validate_output message. It fires under an unsatisfied guard too, like REQ-OUTPUT-INSIDE-HIERARCHY.
  • Swallowed group error. The output-group site discarded get_or_create_group_node's error, so a group under a file was refused only when some consumer resolved the same group.
  • Dangling read in ensure_file_node (separate commit). The three alias branches read the id through a find pointer after inserting into the same SortedPairVec, which shifts entries. The new parse-time stat reached it: d/f/x.o over a source file d/f was built at d/x.o. A sweep of find-then-mutate pointer sites in src/ found no other.

Checked against real tup

Case tup putup
in-tree: output sub/x.o, group sub/<g> under source file sub refuses refuses
variant: same accepts accepts
gen/x then output gen refuses refuses

Not covered (recorded in spec/requirements/output-paths.ears.md)

  • An output onto a source directory or a group's directory is not refused at parse time. tup refuses both; in-tree putup refuses the source directory later, as a file the build does not own.
  • A group under an earlier rule's generated output is accepted (already accepted before this change).
  • In a variant build, an input sub named after an output sub/x.o binds to the output's directory node, so the command fails in the shell. tup accepts it.

Verification

  • Red-then-green for every new case. Removing the group-site stat turns its case red again.
  • make check: 182598 assertions in 961 test cases, 32 e2e shards.
  • Reviewed by a three-lens workflow with a different-model regression reviewer and verifiers. Its findings produced the dangling-read fix, the variant scope and the spec wording.

🤖 Generated with Claude Code

typeless and others added 2 commits September 29, 2026 23:01
ensure_file_node's alias branches found a node through
path_to_node.find, inserted the alias into the same SortedPairVec, and
then read the id back through the pointer find had returned. The insert
shifts every entry with a larger key, so when the alias key sorted below
the target, the pointer read the neighbouring entry.

The in-tree branch reaches it when a build path is interned before its
source twin. #495's parse-time stat does exactly that: for an output
d/f/x.o over a source file d/f inside an unbound directory d, the parser
interns build d/f first, the stat binds source d/f as a File, and the
alias for build d/f then returned the directory d. The output landed at
d/x.o, a path no rule named, and the build succeeded.

All three alias branches now copy the id before inserting. A sweep of
every pointer taken from a container's find and read after a mutation of
that container in src/ found no other site: the rest return before
mutating or never read the pointer again.

Verified red-then-green: the new unit test got node 2 (d) where the
file is node 3.

Ref: #495

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
putup refused an output under the source file sub only when a rule had
already read sub. Named first, the output minted a Directory node for
sub, the later input bound to it, and the command failed in the shell
("Directory nonexistent"). And an output could claim a path an earlier
output had made a directory (gen/x, then gen), failing in the shell with
"Is a directory". Upstream refuses both at parse time: the first in
find_dir_tupid_dt_pg, which knows the file from its scan, the second in
validate_output.

In an in-tree build, before binding an output or creating a group,
putup now stats each component of the directory that no node holds yet,
outermost first, and binds a regular file as a File node, so
add_file_node's parent check (#494) refuses it. The graph stays free of
disk access; the stat sits in the builder next to resolve_input_node,
which already types source paths from the disk. Variant builds skip it:
real tup accepts both an output and a group under a source file there,
because both land in the variant tree. Outputs into directories that do
not exist yet are unaffected.

After binding, an output whose node is not a file, a generated file or
a ghost is refused with upstream's message, naming the type. It fires
under an unsatisfied guard too, as REQ-OUTPUT-INSIDE-HIERARCHY does.

The output-group site discarded get_or_create_group_node's error, so a
group under a file was refused only when some consumer happened to
resolve the same group. It now returns the error.

Not covered, recorded in the requirements: an output onto a source
directory or a group's directory is not refused at parse time (tup
refuses it; in-tree putup refuses the source directory later as a file
the build does not own), and a group under an earlier rule's generated
output is accepted. In a variant build, an input sub named after an
output sub/x.o still binds to the output's directory node.

Verified red-then-green: the new cases ran the command and failed in
the shell, or built, before the change; removing the group-site stat
turns its case red again. Real tup was run on the variant and group
cases to settle the in-tree-only scope. The new stats run at parse
time, once per directory component with no node yet (a Tupfile's own
directory already has one); the stat-call metric counts only change
detection (cmd_build.cpp), so it does not see them.

Ref: #495

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

PR metrics

Performance (gcc example, Linux)

Workload Instructions CPU time Page faults D1 miss LL miss Wall Peak RSS
parse 1760 M (+0.2%) 0.4 s 15.2 k 0.6% 0% 0.42 s 35.1 MB (-0.2MB)
dry-run 2435 M (-1.9%) 0.49 s 16.8 k 0.6% 0% 0.497 s 41.4 MB (-0.1MB)

Deterministic signals: instructions (cachegrind-simulated instruction reads — exact across runs, no PMU needed), page faults, peak RSS, and the cachegrind D1/LL miss rates. CPU time is user+sys from time(1).

Internal statistics (gcc example, up-to-date dry run)

Metric Value
Tupfiles parsed 24
Commands 3545
Commands scheduled 0
Files checked 5834
Files changed 0
Files in index 6188
Graph edges 384927
Index size (bytes) 7869819
Implicit deps 344126
Hash computations 0 (-100.0%)
Hashes skipped (stat cache) 5833 (+3.5%)
Stat calls 5885
Parse time (ms) 370.8
Total time (ms) 465.4
Runner CPU AMD EPYC 9V45 96-Core Processor

Counters from putup -n --stat on the fully-built gcc example (up-to-date dry run): deterministic work measures — a jump in commands scheduled, hash computations, or stat calls is a real behavior change, not noise. Timings are the minimum over repeated runs, compared only against a baseline from the same CPU model; the counters are the regression signal.
Timing deltas suppressed: baseline ran on different hardware (Intel(R) Xeon(R) 6973P-C).

Binary size (Linux)

Binary .text .data .bss File
putup 604.6 KB (+0.7%) 2.4 KB (+4.3%) 98.8 KB 716.4 KB (+0.7%)

Code churn (whole codebase, last 30d)

Files Lines written Still present Churned Churn rate
28 2315 2080 235 10.2%

Of the lines written across the codebase in the last 30 days, how many are already gone — work that was written and then discarded or rewritten inside the same window. This is the state of the tree including this PR, not a measure of the PR itself. Only code we write is counted: tests, examples, vendored and generated files, CI plumbing and prose are excluded. 1560 lines were deleted in the window in total, most of them older than it.

Where the churn is
File Lines written then discarded
src/parser/eval.cpp 61
src/graph/dag.cpp 45
src/index/entry.cpp 39
include/pup/core/token_list.hpp 23
src/graph/builder.cpp 16
src/index/reader.cpp 8
include/pup/core/instruction.hpp 7
src/core/instruction.cpp 7
src/cli/cmd_build.cpp 6
include/pup/parser/eval.hpp 5

Test coverage (lines)

Overall Median file Min file Max file
88.9% 96.5% (-0.4pp) 14.7% include/pup/parser/token.hpp 100.0% include/pup/core/arena.hpp

105 files · 17953/20197 lines covered

Deltas vs main@2987923bf.

Updated for 0dc8b53

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refusing an output under a non-directory depends on rule order, because putup learns a path's type only when a rule names it

1 participant