Skip to content

Scan only a command whose invocations are all compiles - #400

Merged
typeless merged 2 commits into
mainfrom
fix/356-invocation-scan-criterion
Aug 12, 2026
Merged

Scan only a command whose invocations are all compiles#400
typeless merged 2 commits into
mainfrom
fix/356-invocation-scan-criterion

Conversation

@typeless

Copy link
Copy Markdown
Owner

Summary

Two commits, split because two files carry both changes:

  1. Split a scan command into invocations before reading it — a scan is built
    by walking a command's words, but two of the four walks that do it decided
    what a word meant from its position in the whole list rather than from the
    invocation it belongs to: both matchers searched for the compile flag across
    every word, and both builders folded source words in from wherever they
    appeared. This splits the command into invocations in dep_words.cpp,
    beside the separator notion the scanners already share, and re-expresses
    all four call sites over it. Along the way, is_command_separator is
    renamed to is_flag_barrier — it never meant "separates commands", it
    means "the scan reads no flags past this word", which is why redirections
    belong in it and why it is the wrong set to divide invocations on. No
    behaviour change.

  2. Scan only a command whose invocations are all compiles — a scan runs
    from the rule's directory with the rest of the command stripped, so a word
    taken from a later invocation is read in a state that invocation never had.
    A chained command like gcc -c a.c -o a.o && cd sub && gcc -c b.c -o b.o
    could emit a scan that reads a same-named file in the wrong directory,
    producing headers stale against the rule and never rebuilding on the one it
    actually reads. The fold wasn't limited to compiles either — a trailing
    && rm junk.c contributed junk.c, and a later invocation's -c could turn
    a link into a false compile match. Now a scan may carry a word only from an
    invocation putup can reproduce, and it can reproduce an invocation only
    when it and every invocation before it is a compile it recognizes.

RED was quoted first: seven tests, plus one earned by pair-partner review
during the closing pass. The whole suite passes in one process: 434 ~[e2e]
cases plus 395 [e2e] cases. A Sonnet pair-partner review caught a real
blocker — a regression on trailing separators — which was fixed before this
branch was considered done.

Closes #356

Test plan

  • RED quoted before production code changed
  • Whole suite passes in one process (~[e2e] + [e2e])
  • make format-check, make spec-check clean
  • Pair-partner (Sonnet) review, one blocker regression on trailing
    separators caught and fixed
  • Measured cost in tree: none — no example rule is compiler-first with a
    separator

A scan is built by walking a command's words, and two of the four walks that do
it decided what a word meant from its position in the whole list rather than in
the invocation it belongs to. Both matchers searched for the compile flag across
every word; both builders folded source words in from wherever they appeared.
The criterion the next commit applies has to be asked once, of each invocation,
which needs the invocations to exist as a thing the code can name.

Split them in dep_words.cpp, beside the separator notion the scanners already
share, and re-express all four sites over it. Invocation boundaries are the
generic part and live in the primitive; which invocations count as compiles is
each scanner's own question and stays there.

Two separators, not one: a redirection stops a scan reading further flags but
hands its target to the same program, so it divides no invocation. An operator
with nothing after it likewise begins none.

While the two notions are being separated, `is_command_separator` is renamed to
`is_flag_barrier`. It never meant "separates commands": it means "the scan reads
no flags past this word", which is why redirections belong in it and why it is
the wrong set to divide invocations on. The old name is what let a design
premise be written from a correct reading of the code, and a name that has to be
corrected by its own comment is the encoding paying for the mistake twice.

No behaviour change.
@github-actions

github-actions Bot commented Aug 12, 2026

Copy link
Copy Markdown

PR metrics

Performance (gcc example, Linux)

Workload Instructions CPU time Page faults D1 miss LL miss Wall Peak RSS
parse 1768 M (+0.3%) 0.65 s 14.7 k (+1.4%) 0.7% 0% 0.65 s 33.2 MB (+1.0MB)
dry-run 2210 M (+0.3%) 0.7 s 15.6 k (+0.6%) 0.8% 0% 0.724 s 36.5 MB (+0.4MB)

Deterministic signals: instructions (cachegrind-simulated instruction reads — exact across runs, no PMU needed), page faults, peak RSS, and the cachegrind D1/LL miss rates. CPU time is user+sys from time(1).

Internal statistics (gcc example, up-to-date dry run)

Metric Value
Tupfiles parsed 24
Commands 3545
Commands scheduled 0
Files checked 5834
Files changed 0
Files in index 6102
Graph edges 381415
Index size (bytes) 7683630
Implicit deps 344126
Hash computations 0
Hashes skipped (stat cache) 5833
Stat calls 5885
Parse time (ms) 564.7 (-0.6%)
Total time (ms) 698.8 (-0.6%)
Runner CPU AMD EPYC 7763 64-Core Processor

Counters from putup -n --stat on the fully-built gcc example (up-to-date dry run): deterministic work measures — a jump in commands scheduled, hash computations, or stat calls is a real behavior change, not noise. Timings are the minimum over repeated runs, compared only against a baseline from the same CPU model; the counters are the regression signal.

Binary size (Linux)

Binary .text .data .bss File
putup 564 KB (+0.3%) 2.3 KB 98.7 KB 670.9 KB (+0.7%)

Code churn (whole codebase, last 30d)

Files Lines written Still present Churned Churn rate
140 11234 9575 1659 14.8%

Of the lines written across the codebase in the last 30 days, how many are already gone — work that was written and then discarded or rewritten inside the same window. This is the state of the tree including this PR, not a measure of the PR itself. Only code we write is counted: tests, examples, vendored and generated files, CI plumbing and prose are excluded. 5065 lines were deleted in the window in total, most of them older than it.

Where the churn is
File Lines written then discarded
src/cli/cmd_build.cpp 653
src/graph/scanners/clang_cl.cpp 155
src/cli/context.cpp 82
src/graph/builder.cpp 72
src/core/layout.cpp 69
src/core/new_delete.cpp 63
src/graph/scanners/dep_words.cpp 63
src/platform/file_io-posix.cpp 42
include/pup/index/format.hpp 35
src/cli/cmd_clean.cpp 35

Test coverage (lines)

Overall Median file Min file Max file
88% 96.2% (-0.2pp) 14.7% include/pup/parser/token.hpp 100.0% include/pup/core/arena.hpp

102 files · 16512/18758 lines covered

Deltas vs main@bb2c7877f.

Updated for 12bfb89

A scan runs from the rule's directory with the rest of the command stripped, so
a word it takes from a later invocation is a word it reads in a state that
invocation never had. `gcc -c a.c -o a.o && cd sub && gcc -c b.c -o b.o` emitted
`gcc -M a.c b.c`: with no b.c here the build failed outright, and with a
same-named b.c here it succeeded and recorded that file's headers instead --
stale against the header the rule reads, and rebuilding on one it does not. The
fold was not even limited to compiles: `&& rm junk.c` contributed junk.c, and
the matcher would call a link a compile on a later invocation's -c.

So: a scan may carry a word only from an invocation putup can reproduce, and it
can reproduce an invocation only when it and every invocation before it is a
compile it recognizes. Invocation 0 is the old front gate -- the same rule with
the word "first" deleted -- and the matcher and the builder now ask it through
one call, which is what keeps the unscanned-rule report and the scan agreeing
about a rule.

Refusal needs no new channel: with no scan node, the report #352 installed names
the object under `parse` and counts the rule under `build`. That is the standing
constraint here -- reporting is binary per rule, so no scan decision may create
a state it cannot express.

Measured cost in tree: none. No example rule is compiler-first with a separator.

Closes #356.
@typeless
typeless force-pushed the fix/356-invocation-scan-criterion branch from d8a2fe5 to 12bfb89 Compare August 12, 2026 07:55
@typeless
typeless merged commit 0e79076 into main Aug 12, 2026
13 checks passed
@typeless
typeless deleted the fix/356-invocation-scan-criterion branch August 12, 2026 13:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dep scan folds source words across a mid-command cd, breaking its own build

1 participant