Skip to content

Scan a compile behind a prefix that changes nothing it reads - #403

Merged
typeless merged 1 commit into
mainfrom
wip/352-scan-transparent-prefix
Aug 13, 2026
Merged

Scan a compile behind a prefix that changes nothing it reads#403
typeless merged 1 commit into
mainfrom
wip/352-scan-transparent-prefix

Conversation

@typeless

Copy link
Copy Markdown
Owner

putup forfeited a command's whole dependency scan unless every leading invocation was a compile it recognized — a harmless prefix (echo building %f &&, FOO=1, a leading ;) cost the rule all of its header dependencies.

Recognition now follows one rule: what precedes a compile is either reproduced in the scan or proven inert, and never silently dropped.

  • Leading NAME=VALUE words are absorbed by the scanner's compiler index and kept verbatim in the scan text, so the scan preprocesses under the environment the compile did — for every variable, CPATH included, with no per-name judgment.
  • An invocation that runs nothing, or runs only echo/true/: with all-plain words, is scan-transparent: the prefix walk steps over it. The allowlist is deliberately tiny and grows only on evidence; the unscanned-object reporter continues to measure the lag.
  • Everything else — cd, export, standalone assignments, redirections, shell-special words — still ends the scannable region.

Both new fences live in one shared predicate (hides_shell_syntax): words are split on whitespace, so FOO=1; is an assignment the shell ends and x>f.h a redirection it never stood alone as. The word-shape and transparency tests are shared in dep_words.cpp, so the gcc and clang-cl scanners cannot drift on either. The prefix walk truncates at the last compile, keeping "non-empty prefix ⇔ at least one scan" true by construction.

Spec: dep-scan.ears.md gains REQ-SCAN-KEEPS-ASSIGNMENT-PREFIX and REQ-SCAN-TRANSPARENT-INVOCATION, and the region-ending requirement is reworded to "neither a recognized compile nor scan-transparent". DESIGN.md records the reproduce-or-prove-inert law.

Verification: four unit RED runs and one E2E RED run quoted before implementation; full suite green (170231 assertions, 835 cases); spec-check, format, tidy, iwyu clean. The gcc BSP example's unscanned count is unchanged at 103 — measured, not assumed: those are shell-loop rules, which stay correctly opaque; the widening's payoff is pinned by the new tests and the E2E fixture.

Closes #352

🤖 Generated with Claude Code

https://claude.ai/code/session_01AjcxNmyFfoTJYSZKKgjrCp

putup forfeited a command's whole dependency scan unless every leading
invocation was a compile it recognized, so `echo building f.c && gcc -c
f.c`, `FOO=1 gcc -c f.c` and a leading `;` each cost the rule all of its
header dependencies -- the rule built, and every header it read went
unrecorded until someone touched a source file.

Recognition now follows one rule: what precedes a compile is either
reproduced in the scan or proven inert, and never silently dropped.
Leading NAME=VALUE words are absorbed by the scanner's compiler index and
kept verbatim in front of the compiler, so the scan preprocesses under the
environment the compile did -- keeping them is what makes it sound, since
a dropped CPATH would resolve the header search against a path the compile
never read. An invocation that runs nothing, or runs only echo/true/:,
changes neither directory nor environment nor any file the compile reads,
so the walk steps over it. Everything else still ends the scannable
region.

Both fences live in one predicate because words are split on whitespace:
`FOO=1;` is an assignment the shell ends and `x>f.h` a redirection it
never stood alone as, and neither is a word putup ever classified. The
word-shape test and the transparency test are shared in dep_words.cpp, so
the gcc and clang-cl scanners cannot drift on either.

The prefix walk ends at the last compile in the run, which keeps a
non-empty prefix and a non-empty scan set the same answer -- the unscanned
report consumes the production predicate, so a matcher that claimed a
command of pure echoes would report it covered and scan nothing.

The in-tree gcc BSP example is unchanged at 103 unscanned objects: those
are shell-loop rules, which stay opaque.

Closes #352

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjcxNmyFfoTJYSZKKgjrCp
@github-actions

Copy link
Copy Markdown

PR metrics

Performance (gcc example, Linux)

Workload Instructions CPU time Page faults D1 miss LL miss Wall Peak RSS
parse 1698 M 0.63 s 14.9 k 0.7% (+0.2pp) 0.1% (+0.1pp) 0.65 s 34 MB
dry-run 2367 M 0.74 s 16.3 k 0.8% (+0.2pp) 0% 0.753 s 39.1 MB (+0.1MB)

Deterministic signals: instructions (cachegrind-simulated instruction reads — exact across runs, no PMU needed), page faults, peak RSS, and the cachegrind D1/LL miss rates. CPU time is user+sys from time(1).

Internal statistics (gcc example, up-to-date dry run)

Metric Value
Tupfiles parsed 24
Commands 3545
Commands scheduled 0
Files checked 5834
Files changed 0
Files in index 6102
Graph edges 381415
Index size (bytes) 7683630
Implicit deps 344126
Hash computations 197
Hashes skipped (stat cache) 5636
Stat calls 5885
Parse time (ms) 574.9
Total time (ms) 744.9
Runner CPU AMD EPYC 9V74 80-Core Processor

Counters from putup -n --stat on the fully-built gcc example (up-to-date dry run): deterministic work measures — a jump in commands scheduled, hash computations, or stat calls is a real behavior change, not noise. Timings are the minimum over repeated runs, compared only against a baseline from the same CPU model; the counters are the regression signal.
Timing deltas suppressed: baseline ran on different hardware (Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz).

Binary size (Linux)

Binary .text .data .bss File
putup 565.4 KB (+0.2%) 2.3 KB 98.7 KB 671 KB (+0.1%)

Code churn (whole codebase, last 30d)

Files Lines written Still present Churned Churn rate
141 11750 9818 1932 16.4%

Of the lines written across the codebase in the last 30 days, how many are already gone — work that was written and then discarded or rewritten inside the same window. This is the state of the tree including this PR, not a measure of the PR itself. Only code we write is counted: tests, examples, vendored and generated files, CI plumbing and prose are excluded. 5442 lines were deleted in the window in total, most of them older than it.

Where the churn is
File Lines written then discarded
src/cli/cmd_build.cpp 656
src/graph/scanners/clang_cl.cpp 295
src/graph/scanners/gcc.cpp 113
src/cli/context.cpp 82
src/graph/builder.cpp 72
src/core/layout.cpp 69
src/graph/scanners/dep_words.cpp 64
src/core/new_delete.cpp 63
src/platform/file_io-posix.cpp 42
src/cli/strict_checks.cpp 36

Test coverage (lines)

Overall Median file Min file Max file
88.1% 96.4% (+0.1pp) 14.7% include/pup/parser/token.hpp 100.0% include/pup/core/arena.hpp

102 files · 16614/18854 lines covered

Deltas vs main@60832faff.

Updated for 0e57a90

@typeless
typeless merged commit b8d6498 into main Aug 13, 2026
13 checks passed
@typeless
typeless deleted the wip/352-scan-transparent-prefix branch August 13, 2026 07:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dep scan forfeits a whole command when a harmless prefix precedes the compiler

1 participant