Scan a command per invocation and report a rule per object

DESIGN.md

@@ -877,9 +877,9 @@ Scanners detect compiler commands and generate dependency extraction commands:
 
 ```cpp
 class DepScanner {
-    virtual auto matches(CommandInfo const&) const -> bool = 0;
-    virtual auto has_dep_flags(std::string_view) const -> bool = 0;
-    virtual auto build_dep_command(CommandInfo const&) -> std::optional<StringId> = 0;
+    virtual auto matches(CommandInfo const&, CommandTokens const&) const -> bool = 0;
+    virtual auto has_dep_flags(CommandTokens const&) const -> bool = 0;
+    virtual auto build_dep_scans(CommandInfo const&, CommandTokens const&) -> Vec<DepScan> = 0;
 };
 
 class DepScannerRegistry {
@@ -890,16 +890,25 @@ class DepScannerRegistry {
 
 The `GccScanner` implementation handles GCC, Clang, and compatible compilers.
 
+**Derived values share down the stack; only truths cross the build boundary.** `CommandTokens` is
+the command's words and invocation split, derived once per command by `tokenize_command` and read by
+every scanner. It is a stack value passed `const&`, never a member and never on a node: its views
+cannot outlive the frame, so it cannot become a second source of truth about a command. That is the
+same class as `PathCache` and the opposite of anything persisted, which needs an owner and a
+staleness story.
+
 **What a scan may cover.** A scan runs from the rule's directory with the rest of the command
-stripped, so it may carry a word only from an invocation it can reproduce, and it can reproduce an
-invocation only when it and every invocation before it is a compile the scanner recognizes.
-Refusal is decidable from the token stream; anything finer requires modeling the shell. A
-redirection ends what the scan may read from an invocation but divides no invocation — only a
-control operator starts a new one. Two
-consequences bind every future change here. Per rule, reporting is binary — scanned, or reported
-unscanned — so no encoding may create a scan state the reporter cannot express; widen the reporter
-first. And `matches` and `build_dep_command` answer through one criterion, because the unscanned
-report consumes the production predicate rather than a re-derivation of it.
+stripped, so it may carry a word only from the invocation it reproduces, and it can reproduce an
+invocation only when it and every invocation before it is a compile the scanner recognizes — one
+scan per invocation of that leading prefix. Refusal is decidable from the token stream; anything
+finer requires modeling the shell. A redirection ends what the scan may read from an invocation
+but divides no invocation — only a control operator starts a new one. Three consequences bind
+every future change here. A scan travels with the object it covers, because only the scanner's
+parse of the invocation can say which one it is: anything a consumer must attribute rides inside
+`DepScan`, never beside it. Per object, reporting is binary — covered, or reported unscanned — so
+no encoding may create a scan state the reporter cannot express; widen the reporter first. And
+`matches` and `build_dep_scans` answer through one criterion, because the unscanned report
+consumes the production predicate rather than a re-derivation of it.
 
 ### Generation Flow
 

docs/reference.md

@@ -1979,10 +1979,14 @@ the scan itself are left out:
   substitution a second time, on results the compile never saw. A `-D` whose value is
   a command substitution is therefore invisible to the scan.
 
-A rule that runs several compiles in one command (`gcc -c a.c … && gcc -c b.c …`) is
-scanned once: the scan takes its flags from the first invocation and its sources from
-all of them, so every source is covered but a flag that only the later invocation
-carries is not. A redirection (`> log`, `2>&1`) ends the invocation the same way.
+A rule that runs several compiles in one command (`gcc -c a.c … && gcc -c b.c …`) gets
+one scan per compile, each carrying that invocation's own flags and sources. Scanning
+stops at the first invocation putup cannot reproduce from the rule's directory — a
+directory change, an environment assignment, a link, any other program — because past
+it the scan would preprocess in a state the compile never had; that invocation and
+every one after it go unscanned, and `parse` names each object they leave uncovered.
+A redirection (`> log`, `2>&1`) divides no invocation, though the scan carries no flag
+from beyond it.
 
 Flags whose argument putup recognizes (GNU driver): `-I`, `-isystem`, `-iquote`,
 `-include`, `-isysroot`, `--sysroot`, `-D`, `-U`

include/pup/graph/dep_scanner.hpp

@@ -8,7 +8,7 @@
 #include "pup/graph/rule_pattern.hpp"
 
 #include <memory>
-#include <optional>
+#include <span>
 #include <string_view>
 
 namespace pup::graph {
@@ -23,6 +23,56 @@ struct DepSpec {
     DepOutputMode output_mode = DepOutputMode::Stdout;
 };
 
+/// One scan and the object file whose compile it reproduces. The object travels with the scan
+/// because only the scanner's own parse of the invocation can say which one it is.
+struct DepScan {
+    StringId command = StringId::Empty;
+    StringId object = StringId::Empty;
+};
+
+/// A command's words and the invocations they divide into, derived once and read by every scanner.
+/// Obtainable only from `tokenize_command`, so the words can never disagree with the text they came
+/// from; its views die with the frame that built it, which is why it is never stored anywhere.
+class CommandTokens final {
+public:
+    friend auto tokenize_command(StringId text) -> CommandTokens;
+
+    // Copying would leave the copy's invocation spans pointing into the source's word buffer, so
+    // the value moves out of its constructor and is passed by reference from there on.
+    CommandTokens(CommandTokens const&) = delete;
+    auto operator=(CommandTokens const&) -> CommandTokens& = delete;
+    CommandTokens(CommandTokens&&) = default;
+    auto operator=(CommandTokens&&) -> CommandTokens& = default;
+    ~CommandTokens() = default;
+
+    [[nodiscard]]
+    auto text() const -> StringId
+    {
+        return text_;
+    }
+    [[nodiscard]]
+    auto words() const -> std::span<std::string_view const>
+    {
+        return { words_.data(), words_.size() };
+    }
+    [[nodiscard]]
+    auto invocations() const -> std::span<std::span<std::string_view const> const>
+    {
+        return { invocations_.data(), invocations_.size() };
+    }
+
+private:
+    CommandTokens() = default;
+
+    StringId text_ = StringId::Empty;
+    Vec<std::string_view> words_;
+    Vec<std::span<std::string_view const>> invocations_;
+};
+
+/// The one way to obtain a CommandTokens.
+[[nodiscard]]
+auto tokenize_command(StringId text) -> CommandTokens;
+
 /// Abstract interface for dependency scanners.
 /// Implementations detect specific tools (compilers, assemblers, linkers)
 /// and generate commands to extract their implicit dependencies.
@@ -38,18 +88,19 @@ class DepScanner {
 
     /// Check if this scanner applies to the given command
     [[nodiscard]]
-    virtual auto matches(CommandInfo const& cmd) const -> bool = 0;
+    virtual auto matches(CommandInfo const& cmd, CommandTokens const& tokens) const -> bool = 0;
 
     /// Check if command already has dependency generation enabled
     [[nodiscard]]
-    virtual auto has_dep_flags(std::string_view cmd) const -> bool = 0;
+    virtual auto has_dep_flags(CommandTokens const& tokens) const -> bool = 0;
 
-    /// Build a command to extract dependencies from the given command.
-    /// Returns nullopt if deps shouldn't be extracted (e.g., already has flags).
+    /// Build the scans that extract dependencies from the given command, one per compile
+    /// invocation it can reproduce. Empty means the command carries no such invocation.
     [[nodiscard]]
-    virtual auto build_dep_command(
-        CommandInfo const& cmd
-    ) const -> std::optional<StringId> = 0;
+    virtual auto build_dep_scans(
+        CommandInfo const& cmd,
+        CommandTokens const& tokens
+    ) const -> Vec<DepScan> = 0;
 
     /// Get the dependency extraction specification
     [[nodiscard]]
@@ -75,17 +126,17 @@ class DepScannerRegistry final {
 
     /// Find a scanner that matches the command (nullptr if none)
     [[nodiscard]]
-    auto find_match(CommandInfo const& cmd) const -> DepScanner const*;
+    auto find_match(CommandInfo const& cmd, CommandTokens const& tokens) const -> DepScanner const*;
 
     /// Generate rules for a command using matching scanners
     [[nodiscard]]
-    auto match_and_generate(CommandInfo const& cmd) const
+    auto match_and_generate(CommandInfo const& cmd, CommandTokens const& tokens) const
         -> Vec<GeneratedRule>;
 
     /// Whether any scanner recognizes the command as writing its own depfile, which the
     /// build reads back from beside the object whether or not a scan was generated.
     [[nodiscard]]
-    auto reports_own_deps(std::string_view cmd) const -> bool;
+    auto reports_own_deps(CommandTokens const& tokens) const -> bool;
 
     [[nodiscard]]
     auto empty() const -> bool

include/pup/graph/rule_pattern.hpp

@@ -9,7 +9,6 @@
 #include "pup/core/types.hpp"
 #include "pup/core/vec.hpp"
 
-#include <optional>
 #include <string_view>
 
 namespace pup::graph {
@@ -52,16 +51,16 @@ struct GeneratedRule {
     StringId display = StringId::Empty;
     Vec<GeneratedOutput> outputs;
     OutputAction action = OutputAction::Normal;
-    NodeId parent_command = INVALID_NODE_ID; ///< For InjectImplicitDeps
+    NodeId parent_command = INVALID_NODE_ID;   ///< For InjectImplicitDeps
+    StringId covered_object = StringId::Empty; ///< The parent's object this rule records headers for
 };
 
 /// Pattern that generates additional rules when matched
 struct RulePattern {
     bool (*matches)(std::string_view command);
 
-    /// Generate a rule from a matched command
-    /// Returns nullopt if pattern matches but rule shouldn't be generated
-    Function<std::optional<GeneratedRule>(CommandInfo const&)> generate;
+    /// Generate the rules for a matched command; empty if the pattern matches but generates none
+    Function<Vec<GeneratedRule>(CommandInfo const&)> generate;
 };
 
 /// Registry for rule patterns

include/pup/graph/scanners/clang_cl.hpp

@@ -3,7 +3,6 @@
 
 #pragma once
 
-#include "pup/core/string_id.hpp"
 #include "pup/graph/dep_scanner.hpp"
 #include "pup/graph/scanners/dep_words.hpp"
 
@@ -17,12 +16,12 @@ namespace pup::graph::scanners {
 class ClangClScanner final : public DepScanner {
 public:
     [[nodiscard]]
-    auto matches(CommandInfo const& cmd) const -> bool override;
+    auto matches(CommandInfo const& cmd, CommandTokens const& tokens) const -> bool override;
     [[nodiscard]]
-    auto has_dep_flags(std::string_view cmd) const -> bool override;
+    auto has_dep_flags(CommandTokens const& tokens) const -> bool override;
     [[nodiscard]]
-    auto build_dep_command(CommandInfo const& cmd) const
-        -> std::optional<StringId> override;
+    auto build_dep_scans(CommandInfo const& cmd, CommandTokens const& tokens) const
+        -> Vec<DepScan> override;
     [[nodiscard]]
     auto dep_spec() const -> DepSpec override;
     [[nodiscard]]

include/pup/graph/scanners/gcc.hpp

@@ -3,7 +3,6 @@
 
 #pragma once
 
-#include "pup/core/string_id.hpp"
 #include "pup/graph/dep_scanner.hpp"
 #include "pup/graph/scanners/dep_words.hpp"
 
@@ -17,12 +16,12 @@ namespace pup::graph::scanners {
 class GccScanner final : public DepScanner {
 public:
     [[nodiscard]]
-    auto matches(CommandInfo const& cmd) const -> bool override;
+    auto matches(CommandInfo const& cmd, CommandTokens const& tokens) const -> bool override;
     [[nodiscard]]
-    auto has_dep_flags(std::string_view cmd) const -> bool override;
+    auto has_dep_flags(CommandTokens const& tokens) const -> bool override;
     [[nodiscard]]
-    auto build_dep_command(CommandInfo const& cmd) const
-        -> std::optional<StringId> override;
+    auto build_dep_scans(CommandInfo const& cmd, CommandTokens const& tokens) const
+        -> Vec<DepScan> override;
     [[nodiscard]]
     auto dep_spec() const -> DepSpec override;
     [[nodiscard]]

spec/requirements/dep-scan.ears.md

@@ -22,9 +22,9 @@ silence leaves a rule whose headers are never recorded, which is why the last re
 exists.
 
 The unit that narrowness is measured in is the invocation: a command's control operators divide it
-into invocations, and a scan may draw a word only from one it can reproduce. A redirection is not
-such a divider — it hands its target to the same program — though the scan still carries no flag
-from beyond it.
+into invocations, each gets its own scan, and a scan may draw a word only from the invocation it
+reproduces. A redirection is not such a divider — it hands its target to the same program — though
+the scan still carries no flag from beyond it.
 
 Two classes escape the report. A compile-and-link command with no `-c` produces an executable
 rather than an object file, so the output-shaped trigger cannot see it (the `HOSTCC` generator
@@ -43,37 +43,43 @@ Which commands a scan is generated for.
 - conformance: putup-only
 - discharge: test "GccScanner rejects compound shell commands"
 - discharge: test "GccScanner compiler wrapper handling"
-- discharge: test "GccScanner refuses a command whose later invocation changes directory"
-- discharge: test "GccScanner refuses a command whose later invocation is not a compile"
-- discharge: test "matches_gcc_compile refuses a link whose later invocation compiles"
-- discharge: test "ClangClScanner refuses a later invocation that is not a compile"
+- discharge: test "GccScanner scans the prefix before a directory change"
+- discharge: test "GccScanner scans the prefix before an invocation that is not a compile"
+- discharge: test "matches_gcc_compile refuses a command whose first invocation is not a compile"
+- discharge: test "A command whose first invocation is not a compile is scanned nowhere"
+- discharge: test "ClangClScanner scans the prefix before an invocation that is not a compile"
 
-Where a command runs any invocation that is not a compile putup recognizes, whether a loop, a
+Where a command runs an invocation that is not a compile putup recognizes, whether a loop, a
 directory change, an environment assignment, a link or any other program, putup shall generate no
-dependency scan for that command, because the scan runs from the rule's directory with the rest of
-the command stripped and would preprocess in a state the compile never had.
+dependency scan for that invocation or for any that follows it, because the scan runs from the
+rule's directory with the rest of the command stripped and past such an invocation would
+preprocess in a state the compile never had.
 
-### REQ-SCAN-FLAG-SOURCE
+### REQ-SCAN-PER-INVOCATION
 
 - conformance: putup-only
-- discharge: test "GccScanner takes flags from the invocation it scans, sources from all of them"
+- discharge: test "GccScanner scans each compile of an all-compile command with its own flags"
 
-Where a command runs more than one invocation and every one of them is a compile putup recognizes,
-putup shall build one scan carrying the source-file words of all of them; that scan takes its flags
-from the first invocation alone, which issue #355 records as a limitation rather than a behaviour a
-later invocation may rely on.
+Where a command's leading invocations are compiles putup recognizes, putup shall build one scan per
+such invocation, each carrying that invocation's own flags and its own source-file words, because
+each one preprocesses a different translation unit and the object it writes is covered only by a
+scan derived from it.
 
 ## Group: reporting
 
-What putup says about a rule it did not scan. Per rule the reporting is binary — scanned, or
-reported unscanned — so no scan decision may create a state this group cannot express, such as a
-rule scanned in part; widening what a scan may cover means widening this group first.
+What putup says about an object it did not scan. The unit is the object, not the rule: per object
+the reporting is binary — covered by a scan derived from the compile that writes it, or reported
+unscanned — so a rule scanned in part is not a state this group must express, only what a reader
+sees when some of a rule's objects are named. The report's sentences carry that unit too: each
+speaks about the object it names, not about the command that declares it.
 
 ### REQ-SCAN-REPORT-UNSCANNED
 
 - conformance: putup-only
 - discharge: test "Scenario: A compile-shaped rule with no dependency scan is reported"
+- discharge: test "Scenario: An object no scanned invocation writes is reported beside its scanned sibling"
 
-When a rule's declared outputs include an object file, no scan is generated for its command,
-and its command carries no depfile flag anywhere in its text, putup shall name that object and
-the rule's Tupfile under `parse`, and report how many such rules exist under a build.
+When a rule declares an object file that no generated scan covers — every object it declares,
+where no scan at all is generated — and the rule's command carries no depfile flag anywhere in its
+text, putup shall name that object and the rule's Tupfile under `parse`, and report how many such
+objects exist under a build.

src/cli/cmd_build.cpp

@@ -2202,13 +2202,13 @@ auto build_single_variant(
     auto num_commands = std::size_t { pup::graph::nodes_of_type(bs.graph, pup::NodeType::Command).size() };
 
     // One line, not one per rule: a warning that fires on every rule of a green build teaches
-    // everyone to scroll past warnings. The per-rule findings live in `parse`.
+    // everyone to scroll past warnings. The per-object findings live in `parse`.
     if (auto unscanned = check_unscanned_compiles(bs.graph, bs.path_cache); !unscanned.empty()) {
         vprint(
             variant_name,
-            "{} rule{} an object file with no dependency scan; run 'putup parse' for the list.\n",
+            "{} object file{} no dependency scan; run 'putup parse' for the list.\n",
             unscanned.size(),
-            unscanned.size() == 1 ? " produces" : "s produce"
+            unscanned.size() == 1 ? " has" : "s have"
         );
     }