Scan a compile behind a prefix that changes nothing it reads

DESIGN.md

@@ -899,8 +899,12 @@ staleness story.
 
 **What a scan may cover.** A scan runs from the rule's directory with the rest of the command
 stripped, so it may carry a word only from the invocation it reproduces, and it can reproduce an
-invocation only when it and every invocation before it is a compile the scanner recognizes — one
-scan per invocation of that leading prefix. Refusal is decidable from the token stream; anything
+invocation only when it and every invocation before it is a compile the scanner recognizes or an
+invocation it proves inert — one scan per compile of that leading prefix. **Reproduce or prove
+inert — never silently drop**: what precedes a compile is kept in the scan (the wrapper, the
+compile's own `NAME=VALUE` words) or shown to change nothing the compile reads (an invocation that
+runs nothing, or only announces); a word dropped because it looked harmless is how a scan comes to
+preprocess a tree the compile never saw. Refusal is decidable from the token stream; anything
 finer requires modeling the shell. A redirection ends what the scan may read from an invocation
 but divides no invocation — only a control operator starts a new one. Three consequences bind
 every future change here. A scan travels with the object it covers, because only the scanner's

include/pup/graph/scanners/dep_words.hpp

@@ -62,6 +62,20 @@ auto is_flag_barrier(std::string_view word) -> bool;
 [[nodiscard]]
 auto is_invocation_separator(std::string_view word) -> bool;
 
+/// True for a leading `NAME=VALUE` word the scan keeps in front of the compiler, so the scan
+/// preprocesses under the environment the compile did. A value carrying shell syntax is not one:
+/// `FOO=$(id)` would be evaluated a second time, and `FOO=1;` ends the assignment where putup's
+/// whitespace split did not.
+[[nodiscard]]
+auto is_env_assignment_word(std::string_view word) -> bool;
+
+/// True for an invocation a scan may step over: one that runs nothing, or one running a program
+/// that only writes to its output stream, no word of which carries shell syntax putup's whitespace
+/// split left unresolved. Such an invocation changes neither the directory nor the environment nor
+/// any file a later compile reads, so it need not be reproduced.
+[[nodiscard]]
+auto is_scan_transparent(std::span<std::string_view const> invocation) -> bool;
+
 /// The invocations a command's control operators divide its words into, in order. An invocation is
 /// empty where two operators meet or where the command begins with one; a word list with no
 /// operator is one invocation, and a trailing operator adds none. Which of them a scan may draw

spec/requirements/dep-scan.ears.md

@@ -16,10 +16,12 @@ accesses instead — so every requirement here is `putup-only` and no citation i
 Recognition is deliberately narrow. putup runs a scan from the rule's directory without the
 rest of the command, so it can only scan a command whose compile it can reproduce in that
 state; a prefix that changes the working directory or the environment makes the reproduction
-false rather than incomplete. A recognized compiler wrapper is the one exception, because it
-changes neither and the scan keeps it. Declining is therefore correct — but declining in
-silence leaves a rule whose headers are never recorded, which is why the last requirement here
-exists.
+false rather than incomplete. What precedes a compile is therefore either reproduced or proven
+inert, and never silently dropped: a recognized compiler wrapper and the compile's own leading
+environment assignments are reproduced, because the scan keeps them; an invocation that runs
+nothing, or only announces, is inert. Anything else is opaque. Declining is therefore correct —
+but declining in silence leaves a rule whose headers are never recorded, which is why the last
+requirement here exists.
 
 The unit that narrowness is measured in is the invocation: a command's control operators divide it
 into invocations, each gets its own scan, and a scan may draw a word only from the invocation it
@@ -49,11 +51,38 @@ Which commands a scan is generated for.
 - discharge: test "A command whose first invocation is not a compile is scanned nowhere"
 - discharge: test "ClangClScanner scans the prefix before an invocation that is not a compile"
 
-Where a command runs an invocation that is not a compile putup recognizes, whether a loop, a
-directory change, an environment assignment, a link or any other program, putup shall generate no
-dependency scan for that invocation or for any that follows it, because the scan runs from the
-rule's directory with the rest of the command stripped and past such an invocation would
-preprocess in a state the compile never had.
+Where a command runs an invocation that is neither a compile putup recognizes nor one it proves
+inert, whether a loop, a directory change, a standalone environment assignment, a link or any
+other program, putup shall generate no dependency scan for that invocation or for any that
+follows it, because the scan runs from the rule's directory with the rest of the command
+stripped and past such an invocation would preprocess in a state the compile never had.
+
+### REQ-SCAN-KEEPS-ASSIGNMENT-PREFIX
+
+- conformance: putup-only
+- discharge: test "GccScanner keeps a leading environment assignment in the scan"
+- discharge: test "ClangClScanner reads the same prefix its sibling does"
+
+Where a compile's own invocation begins with `NAME=VALUE` words carrying no shell syntax putup's
+word split left unresolved, putup shall keep those words in front of the compiler in the scan it builds, because the scan then preprocesses
+under the environment the compile did, including a variable such as `CPATH` that moves the header
+search and that a dropped word would resolve against a path the compile never read; an assignment
+standing as its own invocation scopes to the rest of the command line rather than to one
+invocation, so it is opaque instead.
+
+### REQ-SCAN-TRANSPARENT-INVOCATION
+
+- conformance: putup-only
+- discharge: test "GccScanner scans past an invocation that changes nothing"
+- discharge: test "ClangClScanner reads the same prefix its sibling does"
+- discharge: test "A separator with nothing after it begins no invocation"
+
+Where an invocation before a compile runs nothing, or runs only a program that writes to its
+output stream — `echo`, `true`, `:` — with no word that redirects and no shell syntax putup's word
+split left unresolved, putup shall generate the scans for the compiles after it as if it were
+absent, because such an invocation changes neither the directory, nor the environment, nor any
+file the compile reads; where it redirects, the file it writes may be the very header the compile
+includes, so it is opaque even though its program is one of these.
 
 ### REQ-SCAN-PER-INVOCATION
 

src/graph/scanners/clang_cl.cpp

@@ -116,15 +116,19 @@ auto normalize_flag_path_into(Buf& out, std::string_view flag) -> void
     out += flag;
 }
 
-/// Where the driver stands in one invocation -- first, or behind one recognized wrapper.
+/// Where the driver stands in one invocation -- first, or behind leading environment assignments
+/// and one recognized wrapper. The words before it are the scan's to keep.
 auto driver_index(std::span<std::string_view const> invocation) -> std::optional<std::size_t>
 {
-    if (invocation.empty()) {
+    auto idx = std::size_t { 0 };
+    while (idx < invocation.size() && is_env_assignment_word(invocation[idx])) {
+        ++idx;
+    }
+    if (idx >= invocation.size()) {
         return std::nullopt;
     }
-    auto idx = std::size_t { 0 };
-    if (is_compiler_wrapper(program_basename(invocation[0])) && invocation.size() > 1) {
-        idx = 1;
+    if (is_compiler_wrapper(program_basename(invocation[idx])) && idx + 1 < invocation.size()) {
+        ++idx;
     }
     if (!is_clang_cl_name(program_basename(invocation[idx]))) {
         return std::nullopt;
@@ -141,15 +145,22 @@ auto is_recognized_compile(std::span<std::string_view const> invocation) -> bool
     return std::ranges::any_of(invocation.subspan(*idx + 1), is_compile_flag);
 }
 
-/// The leading invocations a scan can reproduce from the rule's directory: one that is not a
-/// compile may change the directory or the environment, so it and everything after it are out of
-/// reach (#356).
-auto compile_prefix(std::span<std::span<std::string_view const> const> invocations)
+/// The leading invocations a scan can draw from, ending at the last compile in them: an
+/// invocation that is neither a compile nor scan-transparent may change the directory or the
+/// environment, so it and everything after it are out of reach (#356, #352). Ending at the last
+/// compile is what keeps a non-empty prefix and a non-empty scan set the same answer.
+auto scannable_prefix(std::span<std::span<std::string_view const> const> invocations)
     -> std::span<std::span<std::string_view const> const>
 {
     auto length = std::size_t { 0 };
-    while (length < invocations.size() && is_recognized_compile(invocations[length])) {
-        ++length;
+    for (auto i = std::size_t { 0 }; i < invocations.size(); ++i) {
+        if (is_recognized_compile(invocations[i])) {
+            length = i + 1;
+            continue;
+        }
+        if (!is_scan_transparent(invocations[i])) {
+            break;
+        }
     }
     return invocations.first(length);
 }
@@ -159,12 +170,12 @@ auto compile_prefix(std::span<std::span<std::string_view const> const> invocatio
 auto matches_clang_cl_compile(std::string_view command) -> bool
 {
     auto tokens = tokenize_command(global_pool().intern(command));
-    return !compile_prefix(tokens.invocations()).empty();
+    return !scannable_prefix(tokens.invocations()).empty();
 }
 
 auto ClangClScanner::matches(CommandInfo const& /*cmd*/, CommandTokens const& tokens) const -> bool
 {
-    return !compile_prefix(tokens.invocations()).empty();
+    return !scannable_prefix(tokens.invocations()).empty();
 }
 
 auto ClangClScanner::has_dep_flags(CommandTokens const& tokens) const -> bool
@@ -182,7 +193,7 @@ auto ClangClScanner::build_dep_scans(CommandInfo const& /*cmd*/, CommandTokens c
     auto& pool = global_pool();
     auto scans = Vec<DepScan> {};
 
-    for (auto invocation : compile_prefix(tokens.invocations())) {
+    for (auto invocation : scannable_prefix(tokens.invocations())) {
         auto driver_idx = driver_index(invocation);
         if (!driver_idx) {
             continue;

src/graph/scanners/dep_words.cpp

@@ -69,6 +69,28 @@ auto quote_windows_into(Buf& out, std::string_view s) -> void
     out += '"';
 }
 
+auto is_name_start(char c) -> bool
+{
+    return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || c == '_';
+}
+
+auto is_name_char(char c) -> bool
+{
+    return is_name_start(c) || (c >= '0' && c <= '9');
+}
+
+auto is_transparent_program(std::string_view name) -> bool
+{
+    return name == "echo" || name == "true" || name == ":";
+}
+
+// Words are split on whitespace, so one word can carry shell syntax that was never classified as
+// such -- `FOO=1;` is an assignment the shell ends, `x>f.h` a redirection it never stood alone as.
+auto hides_shell_syntax(std::string_view word) -> bool
+{
+    return has_shell_special(word) || word.find_first_of(";&|<>\n") != std::string_view::npos;
+}
+
 } // namespace
 
 auto is_compiler_wrapper(std::string_view name) -> bool
@@ -132,6 +154,26 @@ auto is_invocation_separator(std::string_view word) -> bool
     return word == "&&" || word == "||" || word == ";" || word == "|" || word == "&";
 }
 
+auto is_env_assignment_word(std::string_view word) -> bool
+{
+    auto eq = word.find('=');
+    if (eq == std::string_view::npos || eq == 0 || !is_name_start(word[0])) {
+        return false;
+    }
+    return std::ranges::all_of(word.substr(0, eq), is_name_char) && !hides_shell_syntax(word);
+}
+
+auto is_scan_transparent(std::span<std::string_view const> invocation) -> bool
+{
+    if (invocation.empty()) {
+        return true;
+    }
+    if (!is_transparent_program(program_basename(invocation[0]))) {
+        return false;
+    }
+    return std::ranges::none_of(invocation, hides_shell_syntax);
+}
+
 auto split_invocations(std::span<std::string_view const> words) -> Vec<std::span<std::string_view const>>
 {
     auto result = Vec<std::span<std::string_view const>> {};
@@ -143,7 +185,7 @@ auto split_invocations(std::span<std::string_view const> words) -> Vec<std::span
         }
     }
     // An operator with nothing after it ends the last invocation rather than beginning another; one
-    // with nothing before it does begin an empty one, which is a command no scan can reproduce.
+    // with nothing before it does begin an empty one, which each scanner classifies for itself.
     if (start < words.size() || result.empty()) {
         result.push_back(words.subspan(start));
     }

src/graph/scanners/gcc.cpp

@@ -105,15 +105,19 @@ auto is_scan_hazard(std::string_view flag) -> bool
     return is_blank_word(flag) || has_shell_special(flag) || leads_any(hazard_flags, flag);
 }
 
-/// Where the compiler stands in one invocation -- first, or behind one recognized wrapper.
+/// Where the compiler stands in one invocation -- first, or behind leading environment
+/// assignments and one recognized wrapper. The words before it are the scan's to keep.
 auto compiler_index(std::span<std::string_view const> invocation) -> std::optional<std::size_t>
 {
-    if (invocation.empty()) {
+    auto idx = std::size_t { 0 };
+    while (idx < invocation.size() && is_env_assignment_word(invocation[idx])) {
+        ++idx;
+    }
+    if (idx >= invocation.size()) {
         return std::nullopt;
     }
-    auto idx = std::size_t { 0 };
-    if (is_compiler_wrapper(program_basename(invocation[0])) && invocation.size() > 1) {
-        idx = 1;
+    if (is_compiler_wrapper(program_basename(invocation[idx])) && idx + 1 < invocation.size()) {
+        ++idx;
     }
     if (!is_compiler_name(program_basename(invocation[idx]))) {
         return std::nullopt;
@@ -130,15 +134,22 @@ auto is_recognized_compile(std::span<std::string_view const> invocation) -> bool
     return std::ranges::any_of(invocation.subspan(*idx + 1), [](auto w) { return w == "-c"; });
 }
 
-/// The leading invocations a scan can reproduce from the rule's directory: one that is not a
-/// compile may change the directory or the environment, so it and everything after it are out of
-/// reach (#356).
-auto compile_prefix(std::span<std::span<std::string_view const> const> invocations)
+/// The leading invocations a scan can draw from, ending at the last compile in them: an
+/// invocation that is neither a compile nor scan-transparent may change the directory or the
+/// environment, so it and everything after it are out of reach (#356, #352). Ending at the last
+/// compile is what keeps a non-empty prefix and a non-empty scan set the same answer.
+auto scannable_prefix(std::span<std::span<std::string_view const> const> invocations)
     -> std::span<std::span<std::string_view const> const>
 {
     auto length = std::size_t { 0 };
-    while (length < invocations.size() && is_recognized_compile(invocations[length])) {
-        ++length;
+    for (auto i = std::size_t { 0 }; i < invocations.size(); ++i) {
+        if (is_recognized_compile(invocations[i])) {
+            length = i + 1;
+            continue;
+        }
+        if (!is_scan_transparent(invocations[i])) {
+            break;
+        }
     }
     return invocations.first(length);
 }
@@ -148,12 +159,12 @@ auto compile_prefix(std::span<std::span<std::string_view const> const> invocatio
 auto matches_gcc_compile(std::string_view command) -> bool
 {
     auto tokens = tokenize_command(global_pool().intern(command));
-    return !compile_prefix(tokens.invocations()).empty();
+    return !scannable_prefix(tokens.invocations()).empty();
 }
 
 auto GccScanner::matches(CommandInfo const& /*cmd*/, CommandTokens const& tokens) const -> bool
 {
-    return !compile_prefix(tokens.invocations()).empty();
+    return !scannable_prefix(tokens.invocations()).empty();
 }
 
 auto GccScanner::has_dep_flags(CommandTokens const& tokens) const -> bool
@@ -185,7 +196,7 @@ auto GccScanner::build_dep_scans(CommandInfo const& /*cmd*/, CommandTokens const
     auto& pool = global_pool();
     auto scans = Vec<DepScan> {};
 
-    for (auto invocation : compile_prefix(tokens.invocations())) {
+    for (auto invocation : scannable_prefix(tokens.invocations())) {
         auto compiler_idx = compiler_index(invocation);
         if (!compiler_idx) {
             continue;

test/e2e/fixtures/unscanned_compile/Tupfile.fixture

@@ -1,6 +1,8 @@
 # scanned: an ordinary compile
 : plain.c |> gcc -c %f -o %o |> %B.o
-# unscanned: the compile is not the command's first invocation
-: hidden.c |> echo building %f && gcc -c %f -o %o |> %B.o
+# scanned: the compile follows an invocation that changes nothing it reads
+: shown.c |> echo building %f && gcc -c %f -o %o |> %B.o
+# unscanned: the compile follows an invocation putup cannot reproduce
+: hidden.c |> cd . && gcc -c %f -o %o |> %B.o
 # not a finding: the compile writes its own depfile
 : owndep.c |> gcc -MD -MF %o.d -c %f -o %o |> %B.o

test/e2e/fixtures/unscanned_compile/shown.c

@@ -0,0 +1,2 @@
+#include "shown.h"
+int shown(void){return SHOWN_VALUE;}

test/e2e/fixtures/unscanned_compile/shown.h

@@ -0,0 +1 @@
+#define SHOWN_VALUE 1