Optional tree provider for family merges, cross-checked against git

CHANGELOG.md

@@ -7,6 +7,15 @@ project intends to follow [Semantic Versioning](https://semver.org/) once it rea
 
 ## [Unreleased]
 
+### Added
+
+- `[merge] tree_command`: an optional tree provider for the family merges. For every
+  merge step the train builds, the command proposes a tree; git's own merge tree is always
+  computed and the proposed tree is used only when identical. A different tree, a failure,
+  a timeout or unreadable output keeps git's tree and records why. Each step and merge
+  row carries `tree_source`; the receipt and the round summary carry the totals. Unset,
+  nothing changes. See `docs/TREE_PROVIDER.md`.
+
 ## [0.2.0] - 2026-09-25
 
 ### Changed

README.md

@@ -193,6 +193,7 @@ own production train (see [docs/CASE_STUDY.md](docs/CASE_STUDY.md)):
 | One train per repository: a file lock; a second owner exits 3 without reading anything | `owner_lock` | |
 | Every git call names its clone (`git -C`) and every `gh` call names its repository | `gh_argv`, a test over the source | |
 | Branches are updated by refspec push, never forced, never checked out | `RealGit` | |
+| A configured tree provider's tree is used only when it is git's own merge tree for that step; anything else keeps git's tree | `Train.merge_step` | |
 
 The theorems are about a model of the merge step, not about git. The code never relies
 on the model being right: it reads every landed tree and compares it with the gated one,
@@ -249,6 +250,8 @@ Unknown keys are refused.
 | `ui.pr_comments` | `true` | one living comment per pull request (see [docs/PR_SURFACE.md](docs/PR_SURFACE.md)) |
 | `ui.status_checks` | `true` | a `svrf` commit status on each candidate head |
 | `ui.dashboard_url` | `""` | optional: linked from the status as `target_url` |
+| `merge.tree_command` | `""` | optional alternative merge engine for the family merges, cross-checked against git's own merge tree on every step ([docs/TREE_PROVIDER.md](docs/TREE_PROVIDER.md)) |
+| `merge.tree_timeout_seconds` | `120` | per merge step; a timeout keeps git's tree |
 
 Gate commands see `SVRF_BASE`, `SVRF_COMMIT`, `SVRF_LABEL` and `SVRF_CHANGED_FILES` (a
 file listing the changed paths), so a gate can build only what changed.

docs/TREE_PROVIDER.md

@@ -0,0 +1,70 @@
+# A tree provider for the family merges
+
+Every commit SVRF lands is built from one merge step: a pull request's head merged with
+the base (or with the fold of the earlier pull requests in its family), union-merge paths
+resolved, committed with plumbing. Git builds those trees. If you have another merge
+engine you want to run on real traffic, `[merge] tree_command` lets the train ask it for
+each step's tree and cross-check the answer against git's, without ever trusting it.
+
+```toml
+[merge]
+tree_command = "my-merge-engine --print-tree"   # "" (the default): git alone
+tree_timeout_seconds = 120                      # per step; a timeout falls back to git
+```
+
+Unset, nothing changes: no command runs and the receipts are exactly as before.
+
+## What the command is given
+
+For each merge step the train builds — while folding a family for its gate, and again
+while preparing each pull request's branch for its merge — the command is run with
+`bash -c` in the train's clone, with:
+
+| Variable | Value |
+| --- | --- |
+| `SVRF_MERGE_OURS` | the pull request's head (git's "ours") |
+| `SVRF_MERGE_THEIRS` | the base, or the fold so far (git's "theirs") |
+| `SVRF_MERGE_BASES` | their merge bases (`git merge-base --all`), space separated |
+| `SVRF_CLONE` | the clone; also the working directory |
+
+It prints the tree id of the merge on the first line of its output and exits 0. Only the
+id is read: a step that uses the provider's answer commits the tree git itself wrote,
+which is the same tree.
+
+Steps git does not merge into a new commit are not offered: a head already in the base,
+a base already in the head, a conflict, or a failed read. The pairwise conflict read, the
+admission check, repairs and re-lands never run the command.
+
+## How the answer is used
+
+Git's own merge tree is always computed, exactly as without a provider. Then:
+
+| The command | The step uses | Recorded |
+| --- | --- | --- |
+| printed git's tree | that tree | `tree_source: "provider"` |
+| printed a different tree | git's tree | `tree_source: "git"`, `tree_provider: "PROVIDER_MISMATCH"`, `proposed_tree` |
+| exited non-zero | git's tree | `PROVIDER_EXIT:<code>:<last stderr line>` |
+| ran past `tree_timeout_seconds` | git's tree | `PROVIDER_TIMEOUT` (its whole process group is killed) |
+| printed no tree id | git's tree | `PROVIDER_OUTPUT_INVALID` |
+| could not be started, or the merge bases could not be read | git's tree | `PROVIDER_UNAVAILABLE:<error>`, `MERGE_BASES_UNREAD` |
+
+The provider can never put a tree on the base that git did not produce, and nothing it
+does can hold a pull request, fail a gate or stop a landing. Every landed tree is still
+compared with the gated tree after its merge, as always.
+
+## Where it shows up
+
+Each step of a family in the round receipt, and each merge row, carries `tree_source`
+(and the reason when git's tree was used). The receipt has the totals:
+
+```json
+"tree_provider": {"consulted": 4, "provider": 3, "git": 1, "reasons": {"PROVIDER_MISMATCH": 1}}
+```
+
+and `svrf run --once` prints the same totals for the round (also kept in the state's
+`last_tick`). A reason's counted part is its first two fields, so exit codes are counted
+separately and stderr text is not.
+
+The command runs once per merge step: twice per landed pull request (once in the gated
+fold, once when its branch is prepared), plus once per step of a replanned bisection
+half. Keep it well inside the timeout; the gate and the landing wait for it.

src/svrf/app.py

@@ -15,6 +15,7 @@
 from .git import RealGit
 from .github import RealGitHub
 from .globs import PathSet
+from .tree_provider import TreeCommand
 
 
 def ensure_clone(config: Config) -> None:
@@ -56,6 +57,14 @@ def build(config: Config, *, github=None, dry_run: bool = False, clock=None, sle
         extra["clock"] = clock
     if sleep is not None:
         extra["sleep"] = sleep
+    train_options = {"jobs": config.train.jobs, "family_size": config.train.family_size,
+                     "memory": memory, "rate_floor": config.train.rate_floor,
+                     "max_rounds": config.train.max_rounds, "comment": config.train.comment,
+                     "pr_comments": config.ui.pr_comments, "status_checks": config.ui.status_checks,
+                     "dashboard_url": config.ui.dashboard_url}
+    if config.merge.tree_command:
+        train_options["tree_provider"] = TreeCommand(config.merge.tree_command, config.clone,
+                                                     timeout=config.merge.tree_timeout_seconds)
     if config.demand_driver:
         try:
             module, factory = config.demand_driver.split(":", 1)
@@ -76,9 +85,5 @@ def build(config: Config, *, github=None, dry_run: bool = False, clock=None, sle
                   # but a command supplies the same class of refusal.
                   reland=config.history.reland, kind=kind,
                   history_verdict=lambda b, h: history.verdict(git, b, h, kind, prefixes),
-                  train_options={"jobs": config.train.jobs, "family_size": config.train.family_size,
-                                 "memory": memory, "rate_floor": config.train.rate_floor,
-                                 "max_rounds": config.train.max_rounds, "comment": config.train.comment,
-                                 "pr_comments": config.ui.pr_comments, "status_checks": config.ui.status_checks,
-                                 "dashboard_url": config.ui.dashboard_url},
+                  train_options=train_options,
                   **extra)

src/svrf/config.py

@@ -49,6 +49,12 @@ class UiConfig:
     dashboard_url: str = ""
 
 
+@dataclass
+class MergeConfig:
+    tree_command: str = ""        # "": git alone builds every merge step
+    tree_timeout_seconds: float = 120
+
+
 @dataclass
 class HistoryConfig:
     order: str = "off"            # "off" or "tests-first"
@@ -78,6 +84,7 @@ class Config:
     train: TrainConfig = field(default_factory=TrainConfig)
     history: HistoryConfig = field(default_factory=HistoryConfig)
     ui: UiConfig = field(default_factory=UiConfig)
+    merge: MergeConfig = field(default_factory=MergeConfig)
 
     # ---- derived paths
     @property
@@ -101,7 +108,8 @@ def union_paths(self) -> PathSet:
         return PathSet(self.union_merge)
 
 
-_SECTIONS = {"gate": GateConfig, "train": TrainConfig, "history": HistoryConfig, "ui": UiConfig}
+_SECTIONS = {"gate": GateConfig, "train": TrainConfig, "history": HistoryConfig, "ui": UiConfig,
+             "merge": MergeConfig}
 _TOP = {"repo", "base", "clone", "state_dir", "remote", "git_name", "git_email", "hold_label", "admission_command", "demand_driver"}
 
 
@@ -157,6 +165,11 @@ def from_dict(value: dict, *, root: Path | None = None) -> Config:
         raise ConfigError("history.order must be \"off\" or \"tests-first\"")
     if config.train.family_size < 1 or config.train.jobs < 1:
         raise ConfigError("train.family_size and train.jobs must be at least 1")
+    timeout = config.merge.tree_timeout_seconds
+    if isinstance(timeout, bool) or not isinstance(timeout, (int, float)) or timeout <= 0:
+        raise ConfigError("[merge] tree_timeout_seconds must be a positive number of seconds")
+    if not isinstance(config.merge.tree_command, str):
+        raise ConfigError("[merge] tree_command must be a string")
     if not config.gate.commands:
         raise ConfigError("gate.commands must name at least one command")
     return config

src/svrf/daemon.py

@@ -120,7 +120,7 @@ def save(self, state: dict, summary: dict) -> None:
             return
         state["last_tick"] = {k: summary[k] for k in (
             "tick", "at", "receipt", "receipts", "stopped", "reason", "retry_at",
-            "admitted", "held", "merged", "skipped", "retry_later", "reasons",
+            "admitted", "held", "merged", "skipped", "retry_later", "reasons", "tree_provider",
         ) if k in summary}
         state["last_tick"]["admission_sources"] = {
             str(n): {"head": inputs["row"].get("headRefOid"), "base": inputs["admission_base"]}
@@ -319,6 +319,12 @@ def land(self, admitted: list[int], rows: list[dict], by_number: dict, state: di
         summary["merged"].extend(m["number"] for m in receipt["merges"] if m.get("identity"))
         summary["api_calls"] = receipt.get("api_calls")
         summary["gates"] = summary.get("gates", 0) + len([g for g in receipt["gates"] if "reused" not in g])
+        if "tree_provider" in receipt:
+            total = summary.setdefault("tree_provider", {"consulted": 0, "provider": 0, "git": 0, "reasons": {}})
+            for key in ("consulted", "provider", "git"):
+                total[key] += receipt["tree_provider"][key]
+            for reason, count in receipt["tree_provider"]["reasons"].items():
+                total["reasons"][reason] = total["reasons"].get(reason, 0) + count
         repaired: set[int] = set()
         for hold in receipt["holds"]:
             n = int(hold["number"])

src/svrf/train.py

@@ -49,6 +49,7 @@
 from . import pr_surface
 from .errors import RateLimited, ReadFailed
 from .rules import choose_families, chunk
+from .tree_provider import reason_class
 
 SCHEMA = "svrf.receipt/1"
 
@@ -66,7 +67,7 @@ def __init__(self, git, github, gate, *, receipts: Path, jobs: int = 1, family_s
                  poll_seconds: float = 5, poll_tries: int = 36, max_rounds: int = 8, dry_run: bool = False,
                  comment: bool = True, is_union: Callable[[str], bool] = lambda p: False,
                  pr_comments: bool = True, status_checks: bool = True, dashboard_url: str = "",
-                 hold_label: str = "train:hold"):
+                 hold_label: str = "train:hold", tree_provider=None):
         self.git, self.gh, self.gate = git, github, gate
         self.jobs, self.family_size, self.memory = max(1, jobs), max(1, family_size), memory
         self.clock, self.sleep = clock, sleep
@@ -75,6 +76,7 @@ def __init__(self, git, github, gate, *, receipts: Path, jobs: int = 1, family_s
         self.max_rounds, self.dry_run, self.comment = max_rounds, dry_run, comment
         self.is_union = is_union
         self.hold_label = hold_label
+        self.tree_provider = tree_provider
         self.pr_comments, self.status_checks, self.dashboard_url = pr_comments, status_checks, dashboard_url
         self.lock = threading.RLock()
         self.rows: dict[int, dict] = {}
@@ -97,6 +99,8 @@ def __init__(self, git, github, gate, *, receipts: Path, jobs: int = 1, family_s
             "requested": [], "prs": {}, "pairs": None, "out": {}, "rounds": [], "families": [], "gates": [],
             "merges": [], "holds": [], "retry_later": [], "pending": [], "alerts": [], "rate_waits": [],
             "api_calls": {}, "surface": dict(self._surface_calls), "stopped": False}
+        if tree_provider is not None:
+            self.receipt["tree_provider"] = {"consulted": 0, "provider": 0, "git": 0, "reasons": {}}
 
     # ---- receipt
 
@@ -229,6 +233,35 @@ def rate_guard(self) -> None:
             self._wait_until(float(value.get("reset", self.clock() + 300)),
                              f"RATE_FLOOR:{kind}:{value.get('remaining')}<{self.rate_floor}")
 
+    # ---- the merge step
+
+    def merge_step(self, acc: str, head: str, message: str):
+        """The union step, as always. With a tree provider, a step that git merged into a
+        new commit is also offered to the provider; its tree is used only when it is
+        git's own tree. Returns the step and, with a provider, what to record about it."""
+        step = self.git.union_step(acc, head, message)
+        if self.tree_provider is None or step.status != "CLEAN" or step.commit in (acc, head):
+            return step, {}
+        try:
+            proposed, reason = self.tree_provider.propose(head, acc)
+        except Exception as error:  # a provider failure is never a landing failure
+            proposed, reason = None, f"PROVIDER_FAILED:{type(error).__name__}"
+        if reason is None and proposed != step.tree:
+            reason = "PROVIDER_MISMATCH"
+        record = {"tree_source": "git" if reason else "provider"}
+        if reason:
+            record["tree_provider"] = reason
+            if proposed is not None:
+                record["proposed_tree"] = proposed
+        with self.lock:
+            counts = self.receipt["tree_provider"]
+            counts["consulted"] += 1
+            counts[record["tree_source"]] += 1
+            if reason:
+                key = reason_class(reason)
+                counts["reasons"][key] = counts["reasons"].get(key, 0) + 1
+        return step, record
+
     # ---- planning and gating
 
     def plan(self, numbers: list[int], base: str, parent: str | None = None) -> dict | None:
@@ -237,15 +270,15 @@ def plan(self, numbers: list[int], base: str, parent: str | None = None) -> dict
         acc, steps = base, []
         for n in numbers:
             row = self.rows[n]
-            step = self.git.union_step(acc, row["head_sha"], f"train preview #{n}")
+            step, record = self.merge_step(acc, row["head_sha"], f"train preview #{n}")
             if step.status == "CONFLICT":
                 self.hold(n, "CONFLICT", paths=step.conflicts)
             elif step.status != "CLEAN":
                 self.retry_later(n, step.reason or "UNION_STEP_FAILED")
             elif step.commit == acc:
                 self.retry_later(n, "ALREADY_MERGED")
             else:
-                steps.append({"number": n, "commit": step.commit, "tree": step.tree})
+                steps.append({"number": n, "commit": step.commit, "tree": step.tree, **record})
                 acc = step.commit
         if not steps:
             return None
@@ -341,7 +374,7 @@ def land_family(self, family: dict, expected: str, gate_seconds: float = 0.0) ->
                                observed=pull.get("head_sha"))
                     self._stop_family(family, index, "REQUEUED")
                     return False, rest
-                prepared = self.git.union_step(current, row["head_sha"], f"Merge base into {row['head_ref']}")
+                prepared, record = self.merge_step(current, row["head_sha"], f"Merge base into {row['head_ref']}")
                 if prepared.status != "CLEAN" or prepared.tree != planned["tree"]:
                     self.alert("PREPARE_DIVERGED", family=family["id"], number=n, status=prepared.status,
                                planned=planned["tree"], prepared=prepared.tree)
@@ -366,14 +399,14 @@ def land_family(self, family: dict, expected: str, gate_seconds: float = 0.0) ->
                 self.retry_later(n, f"LANDED_TREE_UNREAD:{unread}")
                 self._add("merges", {"number": n, "family": family["id"], "head": prepared.commit, "merge": merged,
                                      "parents": [], "gated_tree": planned["tree"], "observed_tree": None,
-                                     "identity": None, "at": self.clock()})
+                                     "identity": None, "at": self.clock(), **record})
                 family["status"] = "LANDED_TREE_UNREAD"
                 self._save()
                 return False, [s["number"] for s in steps[index + 1:]]
             identity = observed == planned["tree"]
             self._add("merges", {"number": n, "family": family["id"], "head": prepared.commit, "merge": merged,
                                  "parents": parents, "gated_tree": planned["tree"], "observed_tree": observed,
-                                 "identity": identity, "at": self.clock()})
+                                 "identity": identity, "at": self.clock(), **record})
             if not identity:
                 self.alert("TREE_MISMATCH", family=family["id"], number=n, planned=planned["tree"],
                            observed=observed, merge=merged)