Scan only a command whose invocations are all compiles

DESIGN.md

@@ -890,6 +890,17 @@ class DepScannerRegistry {
 
 The `GccScanner` implementation handles GCC, Clang, and compatible compilers.
 
+**What a scan may cover.** A scan runs from the rule's directory with the rest of the command
+stripped, so it may carry a word only from an invocation it can reproduce, and it can reproduce an
+invocation only when it and every invocation before it is a compile the scanner recognizes.
+Refusal is decidable from the token stream; anything finer requires modeling the shell. A
+redirection ends what the scan may read from an invocation but divides no invocation — only a
+control operator starts a new one. Two
+consequences bind every future change here. Per rule, reporting is binary — scanned, or reported
+unscanned — so no encoding may create a scan state the reporter cannot express; widen the reporter
+first. And `matches` and `build_dep_command` answer through one criterion, because the unscanned
+report consumes the production predicate rather than a re-derivation of it.
+
 ### Generation Flow
 
 ```

include/pup/graph/scanners/dep_words.hpp

@@ -4,6 +4,7 @@
 #pragma once
 
 #include "pup/core/buf.hpp"
+#include "pup/core/vec.hpp"
 
 #include <span>
 #include <string_view>
@@ -50,10 +51,23 @@ auto append_separate_arg_into(Buf& out, std::string_view word, SeparateArg kind)
 [[nodiscard]]
 auto is_blank_word(std::string_view word) -> bool;
 
-/// True for the shell words that end one command and begin another. A scan carries the flags of
-/// the invocation it was built for, so it takes none from what follows one of these.
+/// True for the shell words after which a scan may read no further flags -- a control operator, or
+/// a redirection whose target belongs to the program rather than to the scan.
 [[nodiscard]]
-auto is_command_separator(std::string_view word) -> bool;
+auto is_flag_barrier(std::string_view word) -> bool;
+
+/// True for the shell words that end one program's invocation and begin another's. Narrower than
+/// `is_flag_barrier`: a redirection stops a scan from reading further flags but hands its
+/// target to the same program, so it divides no invocation.
+[[nodiscard]]
+auto is_invocation_separator(std::string_view word) -> bool;
+
+/// The invocations a command's control operators divide its words into, in order. An invocation is
+/// empty where two operators meet or where the command begins with one; a word list with no
+/// operator is one invocation, and a trailing operator adds none. Which of them a scan may draw
+/// from is each scanner's question, but where they begin and end is not, so it is answered here.
+[[nodiscard]]
+auto split_invocations(std::span<std::string_view const> words) -> Vec<std::span<std::string_view const>>;
 
 /// A flag the scan command carries, and what its argument is.
 struct ArgFlag {

spec/requirements/dep-scan.ears.md

@@ -21,11 +21,10 @@ changes neither and the scan keeps it. Declining is therefore correct — but de
 silence leaves a rule whose headers are never recorded, which is why the last requirement here
 exists.
 
-That narrowness is enforced at the *front* of a command only. Once a scan is generated, source
-words are folded in from every later invocation without asking what ran between them, so a
-command whose second invocation changes directory can still contribute a source to a scan that
-will not find it — issue #356. `REQ-SCAN-FLAG-SOURCE` below records that folding as it is
-today, not as it should be.
+The unit that narrowness is measured in is the invocation: a command's control operators divide it
+into invocations, and a scan may draw a word only from one it can reproduce. A redirection is not
+such a divider — it hands its target to the same program — though the scan still carries no flag
+from beyond it.
 
 Two classes escape the report. A compile-and-link command with no `-c` produces an executable
 rather than an object file, so the output-shaped trigger cannot see it (the `HOSTCC` generator
@@ -39,29 +38,36 @@ unrelated word spelling one silences the report for that rule.
 
 Which commands a scan is generated for.
 
-### REQ-SCAN-UNREPRODUCIBLE-PREFIX
+### REQ-SCAN-UNREPRODUCIBLE-INVOCATION
 
 - conformance: putup-only
 - discharge: test "GccScanner rejects compound shell commands"
 - discharge: test "GccScanner compiler wrapper handling"
+- discharge: test "GccScanner refuses a command whose later invocation changes directory"
+- discharge: test "GccScanner refuses a command whose later invocation is not a compile"
+- discharge: test "matches_gcc_compile refuses a link whose later invocation compiles"
+- discharge: test "ClangClScanner refuses a later invocation that is not a compile"
 
-Where any word other than a recognized compiler wrapper precedes a command's compiler, such as
-a loop, a directory change or an environment assignment, putup shall generate no dependency
-scan for that command, because the scan runs without that word and would preprocess in a state
-the compile never had.
+Where a command runs any invocation that is not a compile putup recognizes, whether a loop, a
+directory change, an environment assignment, a link or any other program, putup shall generate no
+dependency scan for that command, because the scan runs from the rule's directory with the rest of
+the command stripped and would preprocess in a state the compile never had.
 
 ### REQ-SCAN-FLAG-SOURCE
 
 - conformance: putup-only
 - discharge: test "GccScanner takes flags from the invocation it scans, sources from all of them"
 
-Where a command runs more than one invocation, putup shall build the scan from the flags of the
-first invocation only, and from the source-file words of every invocation, whether or not the
-later ones run a compiler.
+Where a command runs more than one invocation and every one of them is a compile putup recognizes,
+putup shall build one scan carrying the source-file words of all of them; that scan takes its flags
+from the first invocation alone, which issue #355 records as a limitation rather than a behaviour a
+later invocation may rely on.
 
 ## Group: reporting
 
-What putup says about a rule it did not scan.
+What putup says about a rule it did not scan. Per rule the reporting is binary — scanned, or
+reported unscanned — so no scan decision may create a state this group cannot express, such as a
+rule scanned in part; widening what a scan may cover means widening this group first.
 
 ### REQ-SCAN-REPORT-UNSCANNED
 

src/graph/scanners/clang_cl.cpp

@@ -117,31 +117,56 @@ auto normalize_flag_path_into(Buf& out, std::string_view flag) -> void
     out += flag;
 }
 
-} // namespace
-
-auto matches_clang_cl_compile(std::string_view command) -> bool
+/// Where the driver stands in one invocation -- first, or behind one recognized wrapper.
+auto driver_index(std::span<std::string_view const> invocation) -> std::optional<std::size_t>
 {
-    auto word_ids = core::tokenize_shell_command(command);
-    if (word_ids.empty()) {
-        return false;
+    if (invocation.empty()) {
+        return std::nullopt;
     }
+    auto idx = std::size_t { 0 };
+    if (is_compiler_wrapper(program_basename(invocation[0])) && invocation.size() > 1) {
+        idx = 1;
+    }
+    if (!is_clang_cl_name(program_basename(invocation[idx]))) {
+        return std::nullopt;
+    }
+    return idx;
+}
 
+/// A scan reproduces one invocation from the rule's directory, so it may carry a word only from a
+/// command whose invocations are all compiles it recognizes (#356).
+auto every_invocation_is_a_compile(std::span<std::span<std::string_view const> const> invocations) -> bool
+{
+    return std::ranges::all_of(invocations, [](auto invocation) {
+        auto idx = driver_index(invocation);
+        if (!idx) {
+            return false;
+        }
+        return std::ranges::any_of(invocation.subspan(*idx + 1), is_compile_flag);
+    });
+}
+
+auto command_words(std::string_view command) -> Vec<std::string_view>
+{
     auto& pool = global_pool();
-    auto driver_idx = std::size_t { 0 };
-    if (is_compiler_wrapper(program_basename(pool.get(word_ids[0]))) && word_ids.size() > 1) {
-        driver_idx = 1;
+    auto words = Vec<std::string_view> {};
+    for (auto id : core::tokenize_shell_command(command)) {
+        words.push_back(pool.get(id));
     }
+    return words;
+}
+
+} // namespace
 
-    if (!is_clang_cl_name(program_basename(pool.get(word_ids[driver_idx])))) {
+auto matches_clang_cl_compile(std::string_view command) -> bool
+{
+    auto words = command_words(command);
+    if (words.empty()) {
         return false;
     }
 
-    for (auto i = driver_idx + 1; i < word_ids.size(); ++i) {
-        if (is_compile_flag(pool.get(word_ids[i]))) {
-            return true;
-        }
-    }
-    return false;
+    auto invocations = split_invocations(std::span { words.data(), words.size() });
+    return every_invocation_is_a_compile(std::span { invocations.data(), invocations.size() });
 }
 
 auto ClangClScanner::matches(CommandInfo const& cmd) const -> bool
@@ -166,49 +191,48 @@ auto ClangClScanner::has_dep_flags(std::string_view cmd) const -> bool
 auto ClangClScanner::build_dep_command(CommandInfo const& cmd) const -> std::optional<StringId>
 {
     auto& pool = global_pool();
-    auto word_ids = core::tokenize_shell_command(pool.get(cmd.command));
-    if (word_ids.empty()) {
+    auto words = command_words(pool.get(cmd.command));
+    if (words.empty()) {
         return std::nullopt;
     }
 
-    auto words = Vec<std::string_view> {};
-    words.reserve(word_ids.size());
-    for (auto id : word_ids) {
-        words.push_back(pool.get(id));
-    }
-
-    auto driver_idx = std::size_t { 0 };
-    if (is_compiler_wrapper(program_basename(words[0])) && words.size() > 1) {
-        driver_idx = 1;
+    auto invocations = split_invocations(std::span { words.data(), words.size() });
+    if (!every_invocation_is_a_compile(std::span { invocations.data(), invocations.size() })) {
+        return std::nullopt;
     }
 
-    if (!is_clang_cl_name(program_basename(words[driver_idx]))) {
+    auto const first = invocations[0];
+    auto driver_idx = driver_index(first);
+    if (!driver_idx) {
         return std::nullopt;
     }
 
     auto dep_cmd = Buf {};
-    for (auto i = std::size_t { 0 }; i <= driver_idx; ++i) {
+    for (auto i = std::size_t { 0 }; i <= *driver_idx; ++i) {
         if (i > 0) {
             dep_cmd += ' ';
         }
-        dep_cmd += words[i];
+        dep_cmd += first[i];
     }
 
     dep_cmd += " /clang:-M";
 
     auto pending = std::optional<SeparateArg> {};
-    auto later_invocation = false;
+    auto linker_tail = false;
+    auto redirected = false;
     auto source_files = Vec<std::string_view> {};
-    for (auto i = driver_idx + 1; i < words.size(); ++i) {
-        auto w = words[i];
+    for (auto i = *driver_idx + 1; i < first.size(); ++i) {
+        auto w = first[i];
 
-        if (is_command_separator(w)) {
-            later_invocation = true;
+        // A redirection hands its target to this same invocation, so the words after it are not
+        // flags the scan may carry.
+        if (is_flag_barrier(w)) {
+            redirected = true;
             pending.reset();
             continue;
         }
 
-        if (later_invocation) {
+        if (redirected) {
             if (is_source_file(w)) {
                 source_files.push_back(w);
             }
@@ -230,8 +254,10 @@ auto ClangClScanner::build_dep_command(CommandInfo const& cmd) const -> std::opt
             continue;
         }
 
-        // /link hands everything after it to the linker, source words included.
+        // /link hands everything after it to the linker, source words included -- and the rest of
+        // the command with it, so no later invocation contributes one either.
         if (w == "/link") {
+            linker_tail = true;
             break;
         }
 
@@ -251,6 +277,14 @@ auto ClangClScanner::build_dep_command(CommandInfo const& cmd) const -> std::opt
         pending = separate_arg(w);
     }
 
+    for (auto later = std::size_t { 1 }; !linker_tail && later < invocations.size(); ++later) {
+        for (auto w : invocations[later]) {
+            if (is_source_file(w)) {
+                source_files.push_back(w);
+            }
+        }
+    }
+
     if (source_files.empty()) {
         return std::nullopt;
     }

src/graph/scanners/dep_words.cpp

@@ -117,16 +117,39 @@ auto is_blank_word(std::string_view word) -> bool
     return word.find_first_not_of(" \t\n\r") == std::string_view::npos;
 }
 
-auto is_command_separator(std::string_view word) -> bool
+auto is_flag_barrier(std::string_view word) -> bool
 {
-    if (word == "&&" || word == "||" || word == ";" || word == "|" || word == "&") {
+    if (is_invocation_separator(word)) {
         return true;
     }
     // A redirection may name its file descriptor first: `>log`, `1>log`, `2>&1`, `3<x`.
     auto rest = word.substr(std::min(word.find_first_not_of("0123456789"), word.size()));
     return rest.starts_with(">") || rest.starts_with("<");
 }
 
+auto is_invocation_separator(std::string_view word) -> bool
+{
+    return word == "&&" || word == "||" || word == ";" || word == "|" || word == "&";
+}
+
+auto split_invocations(std::span<std::string_view const> words) -> Vec<std::span<std::string_view const>>
+{
+    auto result = Vec<std::span<std::string_view const>> {};
+    auto start = std::size_t { 0 };
+    for (auto i = std::size_t { 0 }; i < words.size(); ++i) {
+        if (is_invocation_separator(words[i])) {
+            result.push_back(words.subspan(start, i - start));
+            start = i + 1;
+        }
+    }
+    // An operator with nothing after it ends the last invocation rather than beginning another; one
+    // with nothing before it does begin an empty one, which is a command no scan can reproduce.
+    if (start < words.size() || result.empty()) {
+        result.push_back(words.subspan(start));
+    }
+    return result;
+}
+
 auto find_joined_flag(std::span<ArgFlag const> table, std::string_view word) -> ArgFlag const*
 {
     for (auto const& flag : table) {