diff --git a/doc/rfc/index.md b/doc/rfc/index.md
index 02f012002..2f6d3f163 100644
--- a/doc/rfc/index.md
+++ b/doc/rfc/index.md
@@ -23,6 +23,9 @@ Design documents and technical proposals, grouped by scope. Shared/cross-cutting
- [Gateway Status and List APIs](submitqueue/status-list-api.md) - Gateway-owned request context, materialized current status, sqid or change-URI status lookup, and queue admission listing
- [Speculation](submitqueue/speculation.md) - Why SubmitQueue speculates, the path/tree model, and the two pluggable seams: speculation-tree enumeration and path selection
- [Outcome Scorer](submitqueue/outcome-scorer.md) - How likely a batch is to reach Succeeded: `Score(ctx, batch, paths)` as a logit-linear model — a base content price plus YAML weights on path and batch evidence (`pathPassed`, `pathFailed`, `landing`, `cancelling`)
+- [Code-Aware Outcome Scoring](submitqueue/code-aware-scoring.md) - What a model-backed base scorer sees: the change's diff (implemented as `changediff` + `jev.diff`), then proposed tiers of full touched files, reverse references and batches ahead, selected from the orchestrator's local repository copy rather than sending the repository
+- [Replay Benchmark](submitqueue/replay-benchmark.md) - Benchmarking scorers on real GitHub merge-queue history: `tool/replaybench` fetches attempts and rebuilds them as sandbox commits in short epochs, the `replay` build runner answers with recorded outcomes and durations, and the `oracle` scorer reads the same record as the ceiling
+- [Synthetic Scorer Corpus](submitqueue/synthetic-corpus.md) - Grading scorers on generated Go changes with build-verified labels: `tool/jevcorpus` mutates a seed repository into safe and unsafe changes balanced across size tiers, plants needles before and past the diff budget, writes a replay-format dataset, and grades `jev` against a size baseline offline
- [Best-First Speculation Path Generation](submitqueue/speculation-generator-best-first.md) - The default Generator: per-head lazy streams of flip subsets merged best-first across heads, log-probability ranking, and the strict snapshot contract
- [Modular Queue Wiring](submitqueue/modular-queue-wiring.md) - `pipeline.Construct` for the SubmitQueue orchestrator (`Deps` + `Stages` in `submitqueue/orchestrator/pipeline.go`, host in `service/submitqueue/orchestrator/server`). Stovepipe, and the gateway and Runway servers, remain hand-wired
diff --git a/doc/rfc/submitqueue/code-aware-scoring.md b/doc/rfc/submitqueue/code-aware-scoring.md
new file mode 100644
index 000000000..92cc793a9
--- /dev/null
+++ b/doc/rfc/submitqueue/code-aware-scoring.md
@@ -0,0 +1,96 @@
+# Code-Aware Outcome Scoring
+
+How a model-backed base scorer (`jev`) sees enough of the code to tell a batch that will fail from one that will pass, without sending the whole repository on every call.
+
+**Status:** Tiers 1–3 are implemented: `submitqueue/extension/changediff` (git) returns each change's diff and, optionally, its touched files in full and references found in the tree; the `jev.diff` scorer option sends them (`files`, `references`). Tier 4 is proposed.
+
+See [outcome-scorer.md](outcome-scorer.md) for where a base price sits under the evidence layer, and [speculation.md](speculation.md) for how that price ranks speculation paths.
+
+## Problem
+
+A base scorer is only worth its latency and cost if it separates failing batches from passing ones. Given file names and line counts alone, a decision model has nothing a size `heuristic` does not: in local runs it priced every batch at roughly the same confident value and, on a serialized queue with real failures, spent more builds than a flat 0.5 because it bet heavily on optimistic paths that one early failure invalidated.
+
+Given each change's diff, the same model separated blatant failures (a compile error, a test asserting the wrong value) from correct changes almost perfectly, and the queue reached the minimum possible number of builds at a constrained build budget. The question this document answers is how far beyond the diff the model needs to see, and how to supply that without the costs of sending the repository.
+
+## Why not the whole repository
+
+- **Size.** A repository is megabytes to gigabytes. A System One model is built for a small structured state and an answer in about 100ms; a repository per call either does not fit or turns a fast call into a slow, expensive one.
+- **No reusable server-side context.** The System One request is `state` plus `questions`. There is no upload-once-and-reference mechanism, and whether the backend caches repeated prompt prefixes is not exposed through the proxy.
+- **Staleness.** The target moves with every land, so any snapshot is keyed by target commit and must be rebuilt as the queue advances.
+- **Egress.** Sending the codebase off-network is a larger decision than sending diffs, which already requires sign-off.
+
+The repository is already cached on our side: the orchestrator keeps a local copy for the git change provider, and the git differ reads from it. The design is to *select* from that copy per batch rather than to ship it.
+
+## Context tiers
+
+Each tier is an addition to the scorer state, bounded by a byte budget, computed once per batch and covered by the existing per-batch price cache.
+
+1. **The change's diff** (implemented). The unified diff of each change, measured from the same baseline as its numstat, capped at 64KB per batch and cut at a line boundary with a truncation flag. Catches failures visible in the changed lines.
+2. **Touched files in full** (implemented, `files`). The post-change contents of each touched file, not just the hunks. Catches failures that depend on code adjacent to the change in the same file, such as a new call to a function whose existing definition takes different arguments.
+3. **References** (implemented, `references`). For identifiers the diff removes, renames or changes the signature of, `git grep` the local copy at the change's head for remaining references outside the diff and include those snippets. Catches the cross-file break: a signature change whose callers, untouched by the change, no longer compile. The reverse direction is included too: the definitions of functions the change calls but does not define, so a call into existing code is not mistaken for a call to nothing. Both follow Go scoping — a bare call resolves only in the caller's package directory, and a use from another package is qualified — because an unscoped search returns same-named symbols from unrelated packages, which in testing crowded out the real caller and made existing callees look unreachable.
+4. **Batches ahead.** The diffs of the batches this batch is being built on top of. Catches interaction failures: a change correct on its own that calls a function a batch ahead of it renames.
+
+Tier 4 is not a property of the batch alone, and that is a contract change rather than an addition. The scorer today prices "this batch succeeds", independent of path, and the generator multiplies those prices assuming independence. An interaction failure is only likely *given* a particular dependency succeeding. Supporting it means pricing a batch conditional on the batches ahead — for example a `Score` variant taking the path's assumptions, or a pairwise interaction term the generator applies on top of the per-batch price — and that belongs in a revision of [outcome-scorer.md](outcome-scorer.md) when tier 4 is built.
+
+## What no code tier can see
+
+Failures with no cause in the code — flaky tests, infrastructure outages, quota or environment drift — are invisible to any amount of code context. They are what the evidence layer's `pathFailed` factor and the speculator's failure-assumption paths absorb. A code-aware scorer should not be expected to price them, and an evaluation should report them separately so they do not read as model misses.
+
+## Budgets and caching
+
+Every tier shares one per-batch byte budget, spent in tier order so the diff is never displaced by context. The price cache is keyed by queue and batch ID, and a batch's contents are fixed at creation, so context is gathered at most once per batch however many speculation runs score it. Tiers 2–3 read the change's head; tier 4 also depends on which batches are ahead, which changes as the queue moves and therefore shortens how long a cached price holds.
+
+## Placement
+
+Tiers 2 and 3 are further reads against the same local copy, so they belong beside the git differ — either a richer `changediff` git implementation or a separate context extension with the same input contract (`change.Change` in, per-URI content out). Tier 4 composes existing diffs and needs the batches ahead, which the scorer can resolve through `changeset.Resolver` from the path set it is already handed.
+
+## Security
+
+Every tier sends code to the LiteLLM proxy and the provider behind it. Each is opt-in per queue in configuration, defaults off, and should be enabled on a repository only after sign-off for that repository.
+
+## Evaluation
+
+Each tier is justified by the failures it newly catches without new false alarms. The local evaluation uses a serialized ten-change workload on the git provider with the fake build runner, where failures are injected by a branch-name marker that is hidden from the model, and each failure is chosen to need a specific tier:
+
+| Change | Outcome | Needs |
+|---|---|---|
+| Off-by-one loop with a test that catches it | fails | tier 1 |
+| Signature change whose caller is outside the diff | fails | tier 3 |
+| New call to a function a batch ahead renames | fails | tier 4 |
+| Comment-only change that fails anyway | fails | nothing can see it |
+| Large correct addition; new BUILD file; correct rename; deleted test; error wrapping; small helper | pass | decoys for false alarms |
+
+Reported per run: per-change price, separation between failing and passing prices, builds dispatched, builds on false assumptions, and time to land, at build budgets 4 and 2 against a flat heuristic and the model without code. Beyond the synthetic workload, the next level replays real change history with recorded CI outcomes, which measures calibration on failures nobody designed.
+
+## Findings with tier 1
+
+Over six runs per arm of the workload above (budgets 4 and 2):
+
+- **The diff-visible failure is caught** (mean price 0.02). Correct changes that are large, rename symbols with their callers, or wrap errors are priced 0.94–0.97.
+- **The cross-file signature break is missed** (0.85). The diff shows a correct-looking signature change; the broken caller is outside it. This is the case tier 3 exists for.
+- **The interaction failure looks caught (0.07), but only through a shortcut.** A correct change that calls a function defined outside the diff is priced the same (0.08). The model penalizes calls it cannot resolve, whether or not they are broken. Tier 2–3 context that shows the callee exists is the fix for the false alarm; tier 4 is still the only way to catch the real interaction.
+- **A change that deletes a test is a noisy false alarm** (0.20–0.72), with or without the diff.
+- **Infrastructure flakiness is invisible**, as expected (0.97).
+- Without code the model separates nothing: its failing changes are priced no lower than its passing ones, and the deleted-test decoy drags it below chance (mean AUC 0.29 against 0.67 with the diff).
+
+The queue-level effect follows the separation. With blatant failures, tier 1 reached the minimum number of builds at budget 2. With these subtler ones it roughly ties the flat heuristic at budget 2 and spends more builds at budget 4, because its two confident false alarms send speculation down "fails" paths that then break. Tier 1 is necessary but not sufficient; tiers 2–3 are needed before the price is trustworthy on changes that call into the rest of the codebase, which is most of them.
+
+## Findings with tiers 2–3
+
+The same workload over four arms (no model, model without code, tier 1, tiers 1–3), two rounds at each of budgets 4 and 2, prices averaged over the four runs per arm:
+
+| Change | Truth | No code | Tier 1 | Tiers 1–3 |
+|---|---|---|---|---|
+| Off-by-one, visible in the diff | fails | 0.87 | 0.03 | 0.02 |
+| Signature change, caller outside the diff | fails | 0.82 | 0.88 | 0.91 |
+| Calls a function a batch ahead renames | fails | 0.83 | 0.04 | 0.11 |
+| Flaky infrastructure | fails | 0.81 | 0.93 | 0.96 |
+| Calls a function defined outside the diff | passes | 0.86 | 0.06 | 0.85 |
+| Deletes a test | passes | 0.18 | 0.55 | 0.71 |
+| Large addition, BUILD file, rename with callers, error wrapping | pass | 0.71–0.81 | 0.84–0.97 | 0.89–0.98 |
+| Separation (AUC) | | 0.27 | 0.71 | 0.79 |
+
+- **Definitions fix the out-of-diff false alarm**: the correct change calling an existing function moves from 0.06 to 0.85, and passing changes rise across the board.
+- **The caller does not flip the signature break.** The model is handed the one untouched line still calling the old signature and still prices the change at 0.91. Providing the evidence is not sufficient; the fast model does not carry out the cross-reference. Candidate next steps are an explicit structural check that compares call arity against the new signature before asking, or a question that names the comparison (for example a second noul question per caller).
+- **The interaction failure is still priced low with its callee's definition in view**, so whatever the model keys on there, it is not the missing definition; tier 4 remains the only principled way to see it.
+- Queue cost follows: tiers 1–3 spent the fewest builds at budget 4 (22.0 against 22.5 for no model) and was fastest first-accept-to-last-land at both budgets, though at budget 2 tier 1 alone spent half a build fewer.
diff --git a/doc/rfc/submitqueue/replay-benchmark.md b/doc/rfc/submitqueue/replay-benchmark.md
new file mode 100644
index 000000000..b71f763e4
--- /dev/null
+++ b/doc/rfc/submitqueue/replay-benchmark.md
@@ -0,0 +1,42 @@
+# Replay Benchmark
+
+Benchmarking outcome scorers against real merge-queue history from open-source repositories, instead of synthetic workloads whose failures we wrote ourselves.
+
+**Status:** Implemented for repositories that use GitHub merge queue: `tool/replaybench` builds datasets and sandboxes, `submitqueue/extension/buildrunner/replay` answers builds from the record, and the `oracle` scorer can read the same record. Benchmarks only — none of it may configure a production queue.
+
+See [code-aware-scoring.md](code-aware-scoring.md) for the scorers being compared and [outcome-scorer.md](outcome-scorer.md) for where a price is used.
+
+## Why replay
+
+A synthetic workload measures what its author anticipated. Real history supplies failures nobody designed — dependency bumps that break CI, refactors that collide, flaky infrastructure — at their real rate and with their real CI durations, which also makes time to land meaningful where a fake runner that finishes instantly does not.
+
+## Source: GitHub merge queue
+
+A repository using GitHub merge queue runs CI on a `merge_group` event for every queue attempt, on a commit GitHub builds as `gh-readonly-queue//pr--`. That run history is exactly a submit queue's record: which PR, tested on which queue head, with what result, for how long — including attempts that failed and never landed. The merge-group branches are deleted afterwards, but GitHub still serves their commits by SHA, so failed attempts are recoverable.
+
+An attempt passes if every recorded run for its merge-group commit passed and fails if any failed. Groups whose runs were cancelled or unfinished have no outcome and are dropped: GitHub cancels the groups behind a failure, and those say nothing about their own PR.
+
+## Rebuilding changes on one base
+
+Upstream merge-group commits descend from queue heads the sandbox never has, so used as-is the change provider would measure each one against an ancient merge base and report every earlier PR as part of it. Each attempt is therefore rebuilt as one commit on a fixed base: the PR's own effect (merge-group commit minus its parent) three-way merged onto the base with `git merge-tree`. Every change then diverges from the sandbox's main at the base, and the queue lands them for real.
+
+A long window drifts: later PRs touch code earlier ones changed after the base and no longer apply. Datasets are therefore split into short **epochs**, each rebuilt on the commit just before its first attempt and replayed as an independent run. Attempts that still conflict are kept in the dataset, marked skipped, and left out of the run.
+
+## Answering builds
+
+The `replay` build runner returns the recorded outcome after the recorded duration divided by a time scale. A build fails if any change in its base or head is recorded as failing, because a build containing a broken change is broken whatever path it was testing. The outcome and due time are encoded in the build ID, so the runner holds no state. The `oracle` scorer configured with the same dataset prices every recorded change at 0.99 or 0.01, giving the ceiling every other scorer is measured against.
+
+## Known approximations
+
+- **Attribution.** A merge-group commit contains the PRs queued ahead of it, so an upstream failure can belong to a PR ahead. Replay attributes it to the group's own PR.
+- **Flakiness.** A failure whose exact change (same `git patch-id`) later passed for the same PR is marked `flaky_suspect`. These are recorded failures no code-based scorer could predict, and analyses should report them separately.
+- **Selection.** Conflicting attempts are dropped, and they are disproportionately failures — PRs that needed rework against newer code. Epoch size trades kept attempts against how far each epoch is from upstream.
+- **Context.** A rebuilt change's surrounding code is the epoch base, not the upstream queue head it was really tested on.
+
+## Candidate repositories
+
+Go repositories with merge-queue history, sampled when this was written: `jaegertracing/jaeger` (43 MB, about 24% of attempts fail), `open-telemetry/opentelemetry-collector` (73 MB, about 31%), `containerd/containerd` (138 MB, about 23%), `cilium/cilium` (565 MB, about 11%), and `pulumi/pulumi` (384 MB, about 49%, multi-language). Go keeps the reference heuristics in `changediff` applicable.
+
+## Workflow
+
+`replaybench fetch` lists a window's merge-group runs through the GitHub API (`GITHUB_TOKEN`) and writes a dataset; `split` cuts it into epochs; `materialize` clones or reuses an upstream mirror, fetches the commits no branch holds, rebuilds each attempt on the epoch base, marks flaky suspects, and creates a bare sandbox holding the base as `main` and one branch per attempt; `uris` prints the change URIs in queue order. A run points the git compose overlay at the sandbox (`SQ_GIT_SANDBOX_DIR`) and at a provider directory holding the dataset (`SQ_PROVIDER_CONFIG_DIR`), with `buildRunner: {type: replay, dataset: ...}`.
diff --git a/doc/rfc/submitqueue/synthetic-corpus.md b/doc/rfc/submitqueue/synthetic-corpus.md
new file mode 100644
index 000000000..fde0c16c3
--- /dev/null
+++ b/doc/rfc/submitqueue/synthetic-corpus.md
@@ -0,0 +1,57 @@
+# Synthetic Scorer Corpus
+
+Grading outcome scorers on generated Go changes whose outcome is caused by their code and confirmed by building and testing them, as a counterpart to the noisy labels of replayed history.
+
+**Status:** Phase 1 is implemented as `tool/jevcorpus`: ten mutators, size-tiered sweeps with planted needles, build-verified labels, a replay-format dataset and sandbox, and offline grading of `jev` against a size baseline. Interaction pairs (phase 2) and queue runs at scale (phase 3) are proposed.
+
+See [replay-benchmark.md](replay-benchmark.md) for the real-history benchmark this complements and [code-aware-scoring.md](code-aware-scoring.md) for what the `jev` scorer sends.
+
+## Why a synthetic corpus
+
+The demo workload writes text files and fails builds through URI markers, so nothing in its diffs decides an outcome and no content scorer can do better than chance on it. Replay supplies real failures, but its labels carry misattribution, flakiness and context drift, its failure mix is whatever the upstream repository happened to suffer, and its changes are mostly small. Neither can say *why* a scorer missed a failure, or whether it would have seen a bug placed beyond its diff budget.
+
+A synthetic corpus controls exactly those variables. Every change is a known edit to real code, its label is the outcome of actually building and testing it, and the corpus is balanced so that size, position and surface features cannot stand in for reading the code.
+
+## Design
+
+### Seed
+
+Changes are made to a real Go repository at a fixed commit, so that diffs, touched files and repository references look like real work. Any Go module with tests serves; a replay epoch base is a natural choice because it lets the two benchmarks share a codebase. Only packages whose tests pass at the base are mutated, and generated files are left alone, so every failure in the corpus is one the mutation caused.
+
+### Mutators
+
+Each mutator is a deterministic edit spliced into the source at token positions, so its diff is only the lines it means to change. Five are designed to be safe — renaming an unexported function with every use, renaming locals, adding doc comments, deleting an unreferenced function, adding a tested helper in a new file — and five to fail — inverting a comparison, an off-by-one bound, dropping an error check, deleting a function still called, and adding a parameter without updating callers. They span compile failures, test failures, and changes that look alarming but are harmless (a deleted function) or look trivial but are not (a one-character bound).
+
+### Size tiers and needles
+
+Changes are planned evenly across four tiers by measured diff: S (under 50 lines), M (under 500), L (500 or more), and XL, which exceeds the `jev` scorer's diff budget or file cap. Larger changes are sweeps: mechanical, behavior-preserving edits across many files, occasionally with a safe package-level edit as a decoy. An unsafe sweep carries one unsafe edit — the needle — in the first or the last touched file in path order. Because the diff is path-ordered, a late needle in an XL change sits past the point the scorer truncates, and the manifest records whether it does. That measures directly what truncation costs, and so what the proposed tiers of [code-aware-scoring.md](code-aware-scoring.md) would buy.
+
+### Labels
+
+A change's label is the exit status of a verify command run on the base plus the change — by default the whole module builds and the touched packages' tests pass. A change whose observed outcome contradicts its design is regenerated rather than relabeled, so a category such as "dropped error check" contains only drops that a test actually catches, and the regeneration rate per mutator shows which ones are weak.
+
+### Leakage
+
+Nothing a scorer can read names the mutation. Branches are numbered, commit subjects and authors are drawn from fixed pools independently of the label, and mutators write ordinary identifiers and comments. The answer key — intent, mutators, tier, needle placement, failing output — lives in a manifest beside the dataset, which no part of the queue reads.
+
+### Output
+
+The dataset is in the replay format and the sandbox has the same shape `replaybench materialize` produces, so `replaybench uris`, the `replay` build runner, the `oracle` scorer and the git compose overlay all work on a synthetic corpus unchanged. A verified change's build duration becomes its recorded duration.
+
+## Grading
+
+`jevcorpus eval` scores every change, and optionally sampled multi-change batches, through the real `jev` scorer, the git change provider and the git diff provider, with no fallback and no cache so that a failure is visible as one. It reports AUC, Brier score, log loss, expected calibration error and a calibration table for each scorer, overall and per tier, per category, per needle position, and for needles past the diff budget. A size baseline that prices batches by lines changed sits beside it: on a balanced corpus it should rank no better than chance, and a model that does no better than it has learned only size. The oracle, which prices from the record, is the ceiling by construction.
+
+Slices holding one outcome, such as a single unsafe mutator, have no AUC; their mean price per outcome still shows how far a scorer moved.
+
+## Phases
+
+1. **Single changes** (implemented): mutators, tiers, needles, verified labels, offline grading.
+2. **Interaction pairs** (proposed): two changes that each pass alone and fail together — a rename against a new caller of the old name, two changes adding the same symbol — which is what batch scoring exists to catch. The replay runner fails a build only when a member is recorded as failing, so pairs need either a dataset record of which combinations fail, verified at generation time, or a build runner that builds the merged tree.
+3. **Queue runs** (proposed): the corpus through the git compose overlay with the replay runner, for time to land and wasted builds, compared with replay results on the same seed.
+
+## Known limits
+
+- Verification runs the touched packages' tests, not their dependents', so a change that breaks only a downstream package's tests is labeled as landing. A broader verify command trades generation time for that.
+- Mutations are syntactic. They do not produce semantic conflicts across packages, concurrency bugs, or performance regressions, and a model could in principle learn their shapes; holding out a seed or a mutator from any tuning guards against that.
+- Sweeps are mechanical, which real large changes rarely are. They test whether a scorer can find one consequential line in a large diff, not whether it understands a large redesign.
diff --git a/service/submitqueue/demo/provider/git/profiles.yaml b/service/submitqueue/demo/provider/git/profiles.yaml
index 00911d9a2..a21f6d643 100644
--- a/service/submitqueue/demo/provider/git/profiles.yaml
+++ b/service/submitqueue/demo/provider/git/profiles.yaml
@@ -43,6 +43,31 @@ queues:
target: main
repoPath: /var/submitqueue/changerepos/demo
analyzer: {type: pathoverlap, by: directory}
+ # Prices each batch with a System One decision model behind a LiteLLM
+ # proxy, read from LITE_URL and LITE_KEY. Without them every batch takes
+ # the fallback, the same flat price the defaults give, so the queue behaves
+ # exactly as it does without this block. The CoreWeave proxy serves the
+ # TypeSafe pass-through rather than LiteLLM's model-routed /v1/systemone.
+ #
+ # diff sends each change's code with its file list — without it the model
+ # sees only line counts — plus touched files in full and related lines
+ # from elsewhere in the tree. It reads the change provider's own copy (same
+ # repoPath), so the two share one clone and its lock.
+ scorer:
+ base:
+ type: jev
+ jev:
+ model: jev
+ path: /typesafe/v1/systemone
+ timeout: 5s
+ diff:
+ git:
+ remoteUrl: file:///srv/git/sandbox.git
+ target: main
+ repoPath: /var/submitqueue/changerepos/demo
+ files: true
+ references: true
+ fallback: {type: heuristic}
- name: e2e-git-queue
# Its own copy, separate from demo-queue's: the two agree about the
diff --git a/service/submitqueue/docker-compose.git.yml b/service/submitqueue/docker-compose.git.yml
index 1de039f0e..65d10741e 100644
--- a/service/submitqueue/docker-compose.git.yml
+++ b/service/submitqueue/docker-compose.git.yml
@@ -31,6 +31,10 @@ services:
# resolves to. The build runner stays fake here; the change provider and
# the merge are real.
- PROFILES_CONFIG_PATH=/etc/submitqueue/profiles.yaml
+ # Optional LiteLLM proxy for demo-queue's jev scorer. Unset, the scorer
+ # falls back to the heuristic and the stack starts as before.
+ - LITE_URL=${LITE_URL:-}
+ - LITE_KEY=${LITE_KEY:-}
volumes:
- ${SQ_PROVIDER_CONFIG_DIR}:/etc/submitqueue:ro
# The same bare repository Runway merges into, read-only: the orchestrator
diff --git a/service/submitqueue/orchestrator/server/BUILD.bazel b/service/submitqueue/orchestrator/server/BUILD.bazel
index c4b4d887e..5351ca604 100644
--- a/service/submitqueue/orchestrator/server/BUILD.bazel
+++ b/service/submitqueue/orchestrator/server/BUILD.bazel
@@ -43,6 +43,9 @@ go_library(
"//submitqueue/extension/buildrunner/buildkite:go_default_library",
"//submitqueue/extension/buildrunner/fake:go_default_library",
"//submitqueue/extension/buildrunner/githubactions:go_default_library",
+ "//submitqueue/extension/buildrunner/replay:go_default_library",
+ "//submitqueue/extension/changediff:go_default_library",
+ "//submitqueue/extension/changediff/git:go_default_library",
"//submitqueue/extension/changeprovider:go_default_library",
"//submitqueue/extension/changeprovider/fake:go_default_library",
"//submitqueue/extension/changeprovider/git:go_default_library",
@@ -61,6 +64,8 @@ go_library(
"//submitqueue/extension/speculation/scorer/evidence:go_default_library",
"//submitqueue/extension/speculation/scorer/fake:go_default_library",
"//submitqueue/extension/speculation/scorer/heuristic:go_default_library",
+ "//submitqueue/extension/speculation/scorer/jev:go_default_library",
+ "//submitqueue/extension/speculation/scorer/oracle:go_default_library",
"//submitqueue/extension/speculation/speculator:go_default_library",
"//submitqueue/extension/speculation/speculator/standard:go_default_library",
"//submitqueue/extension/validator:go_default_library",
@@ -114,7 +119,9 @@ go_test(
name = "go_default_test",
srcs = [
"config_test.go",
+ "jev_config_test.go",
"main_test.go",
+ "oracle_config_test.go",
"profiles_test.go",
],
# The pinned git, for the tests that resolve a git change provider against a
@@ -130,11 +137,14 @@ go_test(
"//platform/git/exectest:go_default_library",
"//platform/git/repo:go_default_library",
"//platform/githubactions:go_default_library",
+ "//submitqueue/core/changeset/fake:go_default_library",
"//submitqueue/entity:go_default_library",
"//submitqueue/extension/buildrunner:go_default_library",
+ "//submitqueue/extension/buildrunner/replay:go_default_library",
"//submitqueue/extension/changeprovider:go_default_library",
"//submitqueue/extension/conflict:go_default_library",
"//submitqueue/extension/speculation/scorer:go_default_library",
+ "//submitqueue/extension/speculation/scorer/oracle:go_default_library",
"//submitqueue/extension/speculation/speculator:go_default_library",
"//submitqueue/orchestrator/extension/storage:go_default_library",
"@com_github_stretchr_testify//assert:go_default_library",
diff --git a/service/submitqueue/orchestrator/server/config.go b/service/submitqueue/orchestrator/server/config.go
index af891cfe8..7936d0c70 100644
--- a/service/submitqueue/orchestrator/server/config.go
+++ b/service/submitqueue/orchestrator/server/config.go
@@ -19,9 +19,12 @@ import (
"maps"
"math"
"os"
+ "strings"
"time"
yamlv3 "gopkg.in/yaml.v3"
+
+ jevscorer "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer/jev"
)
// Change provider types selectable from configuration.
@@ -38,6 +41,9 @@ const (
buildRunnerTypeFake = "fake"
buildRunnerTypeGitHubActions = "githubactions"
buildRunnerTypeBuildkite = "buildkite"
+ // buildRunnerTypeReplay answers builds from a recorded CI history.
+ // Benchmarks only.
+ buildRunnerTypeReplay = "replay"
)
// Conflict analyzer types selectable from configuration.
@@ -58,6 +64,20 @@ const (
scorerTypeEvidence = "evidence"
scorerTypeHeuristic = "heuristic"
scorerTypeComposite = "composite"
+ scorerTypeJev = "jev"
+ // scorerTypeOracle prices batches from their recorded outcomes. Benchmarks
+ // only: it reads the answer the build will give.
+ scorerTypeOracle = "oracle"
+)
+
+// Defaults for the jev scorer. The environment variable names match the ones
+// the LiteLLM proxy credentials are distributed under.
+const (
+ defaultJevBaseURLEnv = "LITE_URL"
+ defaultJevTokenEnv = "LITE_KEY"
+ defaultJevModel = "jev"
+ defaultJevTimeout = 2 * time.Second
+ defaultJevCacheSize = 4096
)
// maxBucket is the open upper bound of a size bucket, and defaultFlatScore the
@@ -210,6 +230,10 @@ type buildRunnerConfig struct {
// ExtraInputs are passed to every dispatch alongside the reserved sq_*
// inputs (githubactions only).
ExtraInputs map[string]string `yaml:"extraInputs"`
+ // Dataset is the replay dataset path (replay only).
+ Dataset string `yaml:"dataset"`
+ // TimeScale divides recorded build durations (replay only); 0 means 1.
+ TimeScale float64 `yaml:"timeScale"`
}
// analyzerConfig selects how conflicts between concurrent batches are detected.
@@ -229,6 +253,8 @@ type analyzerConfig struct {
// composite belong on base (and on composite components), not at the top level.
type scorerConfig struct {
Type string `yaml:"type"`
+ // Oracle configures where recorded outcomes come from (oracle only).
+ Oracle *oracleScorerConfig `yaml:"oracle"`
// Factors revise the base price, one per piece of evidence (evidence only).
// An omitted key keeps the inherited value, or 1 if neither defaults nor
// the queue named it.
@@ -243,6 +269,58 @@ type scorerConfig struct {
Components map[string]scorerConfig `yaml:"components"`
// Combine names how a composite reduces its components. Defaults to "avg".
Combine string `yaml:"combine"`
+ // Jev configures the model endpoint (jev only).
+ Jev *jevScorerConfig `yaml:"jev"`
+ // Fallback prices a batch the model could not (jev only). Defaults to the
+ // flat heuristic.
+ Fallback *scorerConfig `yaml:"fallback"`
+}
+
+// jevScorerConfig points a jev scorer at a LiteLLM proxy serving /v1/systemone.
+type jevScorerConfig struct {
+ // BaseURL is the proxy root. When empty it is read from BaseURLEnv.
+ BaseURL string `yaml:"baseUrl"`
+ // BaseURLEnv names the environment variable holding the proxy root.
+ BaseURLEnv string `yaml:"baseUrlEnv"`
+ // TokenEnv names the environment variable holding the proxy virtual key.
+ TokenEnv string `yaml:"tokenEnv"`
+ // Path is the System One route on the proxy. Defaults to LiteLLM's
+ // model-routed /v1/systemone; a proxy exposing only the TypeSafe
+ // pass-through takes /typesafe/v1/systemone.
+ Path string `yaml:"path"`
+ // Model is the proxy's model name for the decision model.
+ Model string `yaml:"model"`
+ // Timeout bounds one model call, as a Go duration ("2s").
+ Timeout string `yaml:"timeout"`
+ // CacheSize caps how many batch prices are kept in memory.
+ CacheSize int `yaml:"cacheSize"`
+ // Diff, when set, sends each change's code to the model alongside its file
+ // list. Absent, the model sees file names and line counts only.
+ Diff *jevDiffConfig `yaml:"diff"`
+}
+
+// jevDiffConfig says where a jev scorer reads diffs from. A git block naming
+// the same repoPath as the queue's git change provider shares its copy.
+type jevDiffConfig struct {
+ // Git reads diffs from a local copy of a git remote.
+ Git *gitProviderConfig `yaml:"git"`
+ // Files adds each touched file's full post-change text.
+ Files bool `yaml:"files"`
+ // References adds untouched callers of symbols the change modifies or
+ // removes, and definitions of symbols it calls but does not define.
+ References bool `yaml:"references"`
+ // MaxDiffBytes and MaxContextBytes cap the code sent per batch. Zero keeps
+ // the scorer's defaults; size them to the model's context window.
+ MaxDiffBytes int `yaml:"maxDiffBytes"`
+ MaxContextBytes int `yaml:"maxContextBytes"`
+}
+
+// oracleScorerConfig selects an oracle's source of truth. Empty reads the fake
+// build runner's URI markers.
+type oracleScorerConfig struct {
+ // Dataset is a replay dataset; the oracle then agrees with a replay build
+ // runner reading the same file.
+ Dataset string `yaml:"dataset"`
}
// bucketConfig is one band of a heuristic scorer: batches whose size falls in
@@ -344,25 +422,45 @@ func (c profilesConfig) validateGitRepoPaths() error {
for _, q := range c.Queues {
resolved := c.resolve(q)
- if resolved.ChangeProvider.Type != changeProviderTypeGit || resolved.ChangeProvider.Git == nil {
- continue
- }
- git := *resolved.ChangeProvider.Git
- previous, seen := byPath[git.RepoPath]
- if !seen {
- byPath[git.RepoPath] = owner{queue: q.Name, cfg: git}
- continue
+ var gits []gitProviderConfig
+ if resolved.ChangeProvider.Type == changeProviderTypeGit && resolved.ChangeProvider.Git != nil {
+ gits = append(gits, *resolved.ChangeProvider.Git)
}
- if previous.cfg.RemoteURL != git.RemoteURL || previous.cfg.Target != git.Target {
- return fmt.Errorf(
- "queues %q and %q share repoPath %q but read different repositories (%s@%s vs %s@%s); give each its own path",
- previous.queue, q.Name, git.RepoPath,
- previous.cfg.RemoteURL, previous.cfg.Target, git.RemoteURL, git.Target)
+ gits = append(gits, resolved.Scorer.jevDiffGits()...)
+ for _, git := range gits {
+ previous, seen := byPath[git.RepoPath]
+ if !seen {
+ byPath[git.RepoPath] = owner{queue: q.Name, cfg: git}
+ continue
+ }
+ if previous.cfg.RemoteURL != git.RemoteURL || previous.cfg.Target != git.Target {
+ return fmt.Errorf(
+ "queues %q and %q share repoPath %q but read different repositories (%s@%s vs %s@%s); give each its own path",
+ previous.queue, q.Name, git.RepoPath,
+ previous.cfg.RemoteURL, previous.cfg.Target, git.RemoteURL, git.Target)
+ }
}
}
return nil
}
+// jevDiffGits returns the git diff sources of every jev scorer in the tree.
+func (s scorerConfig) jevDiffGits() []gitProviderConfig {
+ var gits []gitProviderConfig
+ if s.Jev != nil && s.Jev.Diff != nil && s.Jev.Diff.Git != nil {
+ gits = append(gits, *s.Jev.Diff.Git)
+ }
+ for _, child := range []*scorerConfig{s.Base, s.Fallback} {
+ if child != nil {
+ gits = append(gits, child.jevDiffGits()...)
+ }
+ }
+ for _, component := range s.Components {
+ gits = append(gits, component.jevDiffGits()...)
+ }
+ return gits
+}
+
// resolve returns the full profile for a queue: its own overrides where it has
// them, the defaults everywhere else.
func (c profilesConfig) resolve(q namedQueueProfileConfig) queueProfileConfig {
@@ -528,6 +626,13 @@ func (b *buildRunnerConfig) normalizeAndValidate(where string) error {
if b.BaseURL == "" {
b.BaseURL = defaultGitHubBaseURL
}
+ case buildRunnerTypeReplay:
+ if b.Dataset == "" {
+ return fmt.Errorf("%s: replay build runner requires dataset", where)
+ }
+ if b.TimeScale < 0 {
+ return fmt.Errorf("%s: replay timeScale %v is negative", where, b.TimeScale)
+ }
case buildRunnerTypeBuildkite:
// A Buildkite client is bound to one pipeline by its base URL, so there
// is no meaningful default to fall back on.
@@ -583,8 +688,8 @@ func (s *scorerConfig) normalizeRanking(where string, fillBase bool) error {
if s.Type != scorerTypeEvidence {
return fmt.Errorf("%s: scorer type %q belongs under base, not at the ranking layer", where, s.Type)
}
- if len(s.Buckets) > 0 || len(s.Components) > 0 || s.Combine != "" {
- return fmt.Errorf("%s: buckets, components, and combine belong under base", where)
+ if len(s.Buckets) > 0 || len(s.Components) > 0 || s.Combine != "" || s.Jev != nil || s.Fallback != nil {
+ return fmt.Errorf("%s: buckets, components, combine, jev, and fallback belong under base", where)
}
if err := validateFactors(where, s.Factors); err != nil {
return err
@@ -622,7 +727,68 @@ func (s *scorerConfig) normalizeContent(where string) error {
if s.Type == "" {
s.Type = scorerTypeHeuristic
}
+ if s.Type != scorerTypeOracle && s.Oracle != nil {
+ return fmt.Errorf("%s: oracle belongs on an oracle scorer, not %q", where, s.Type)
+ }
+ if s.Type != scorerTypeJev && s.Jev != nil {
+ return fmt.Errorf("%s: jev belongs on a jev scorer, not %q", where, s.Type)
+ }
+ if s.Type != scorerTypeJev && s.Type != scorerTypeOracle && s.Fallback != nil {
+ return fmt.Errorf("%s: fallback belongs on a jev or oracle scorer, not %q", where, s.Type)
+ }
switch s.Type {
+ case scorerTypeJev:
+ if s.Jev == nil {
+ s.Jev = &jevScorerConfig{}
+ }
+ if s.Jev.BaseURLEnv == "" {
+ s.Jev.BaseURLEnv = defaultJevBaseURLEnv
+ }
+ if s.Jev.TokenEnv == "" {
+ s.Jev.TokenEnv = defaultJevTokenEnv
+ }
+ if s.Jev.Model == "" {
+ s.Jev.Model = defaultJevModel
+ }
+ if s.Jev.Path == "" {
+ s.Jev.Path = jevscorer.DefaultPath
+ }
+ if !strings.HasPrefix(s.Jev.Path, "/") {
+ return fmt.Errorf("%s: jev path %q must start with /", where, s.Jev.Path)
+ }
+ if s.Jev.CacheSize < 0 {
+ return fmt.Errorf("%s: jev cacheSize %d is negative", where, s.Jev.CacheSize)
+ }
+ if s.Jev.CacheSize == 0 {
+ s.Jev.CacheSize = defaultJevCacheSize
+ }
+ if s.Jev.Diff != nil {
+ if s.Jev.Diff.Git == nil {
+ return fmt.Errorf("%s: jev diff needs a git block", where)
+ }
+ diffSource := changeProviderConfig{Type: changeProviderTypeGit, Git: s.Jev.Diff.Git}
+ if err := diffSource.ensureGit(where + " jev diff"); err != nil {
+ return err
+ }
+ if s.Jev.Diff.MaxDiffBytes < 0 || s.Jev.Diff.MaxContextBytes < 0 {
+ return fmt.Errorf("%s: jev diff byte caps must not be negative", where)
+ }
+ }
+ if s.Fallback == nil {
+ s.Fallback = &scorerConfig{}
+ }
+ if s.Fallback.Type == scorerTypeJev {
+ return fmt.Errorf("%s: a jev scorer cannot fall back to another jev scorer", where)
+ }
+ return s.Fallback.normalizeContent(where + " fallback")
+ case scorerTypeOracle:
+ if s.Fallback == nil {
+ s.Fallback = &scorerConfig{}
+ }
+ if s.Fallback.Type == scorerTypeOracle {
+ return fmt.Errorf("%s: an oracle scorer cannot fall back to another oracle", where)
+ }
+ return s.Fallback.normalizeContent(where + " fallback")
case scorerTypeHeuristic:
if len(s.Buckets) == 0 {
s.Buckets = []bucketConfig{{Min: 0, Max: maxBucket, Score: defaultFlatScore}}
diff --git a/service/submitqueue/orchestrator/server/jev_config_test.go b/service/submitqueue/orchestrator/server/jev_config_test.go
new file mode 100644
index 000000000..c7314c990
--- /dev/null
+++ b/service/submitqueue/orchestrator/server/jev_config_test.go
@@ -0,0 +1,215 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ nethttp "net/http"
+ "net/http/httptest"
+ "net/url"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+ "github.com/uber-go/tally"
+ "go.uber.org/zap/zaptest"
+
+ "github.com/uber/submitqueue/platform/base/change"
+ gitexectest "github.com/uber/submitqueue/platform/git/exectest"
+ changesetfake "github.com/uber/submitqueue/submitqueue/core/changeset/fake"
+ "github.com/uber/submitqueue/submitqueue/entity"
+ "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer"
+)
+
+func TestLoadProfilesConfig_JevScorerDefaults(t *testing.T) {
+ cfg, err := loadProfilesConfig(writeProfiles(t, "defaults:\n scorer:\n base: {type: jev}\n"))
+ require.NoError(t, err)
+
+ base := cfg.Defaults.Scorer.Base
+ require.NotNil(t, base)
+ require.NotNil(t, base.Jev)
+ assert.Equal(t, defaultJevBaseURLEnv, base.Jev.BaseURLEnv)
+ assert.Equal(t, defaultJevTokenEnv, base.Jev.TokenEnv)
+ assert.Equal(t, defaultJevModel, base.Jev.Model)
+ assert.Equal(t, "/v1/systemone", base.Jev.Path)
+ require.NotNil(t, base.Fallback)
+ assert.Equal(t, scorerTypeHeuristic, base.Fallback.Type)
+}
+
+func TestLoadProfilesConfig_RejectsBadJevScorers(t *testing.T) {
+ tests := []struct {
+ name string
+ contents string
+ }{
+ {name: "jev at the ranking layer", contents: "defaults:\n scorer: {type: jev}\n"},
+ {name: "jev block on evidence", contents: "defaults:\n scorer:\n jev: {model: jev}\n"},
+ {name: "jev block under heuristic", contents: "defaults:\n scorer:\n base:\n type: heuristic\n jev: {model: jev}\n"},
+ {name: "jev falling back to jev", contents: "defaults:\n scorer:\n base:\n type: jev\n fallback: {type: jev}\n"},
+ {name: "negative cache size", contents: "defaults:\n scorer:\n base:\n type: jev\n jev: {cacheSize: -1}\n"},
+ {name: "relative path", contents: "defaults:\n scorer:\n base:\n type: jev\n jev: {path: v1/systemone}\n"},
+ {name: "bad fallback", contents: "defaults:\n scorer:\n base:\n type: jev\n fallback: {type: vibes}\n"},
+ {name: "fallback under heuristic", contents: "defaults:\n scorer:\n base:\n type: heuristic\n fallback: {type: heuristic}\n"},
+ {name: "oracle falling back to oracle", contents: "defaults:\n scorer:\n base:\n type: oracle\n fallback: {type: oracle}\n"},
+ {name: "diff without a source", contents: "defaults:\n scorer:\n base:\n type: jev\n jev: {diff: {}}\n"},
+ {name: "diff git without repoPath", contents: "defaults:\n scorer:\n base:\n type: jev\n jev: {diff: {git: {remoteUrl: /r.git, target: main}}}\n"},
+ {
+ name: "diff sharing a repoPath with a different remote",
+ contents: "queues:\n" +
+ " - name: q\n" +
+ " changeProvider: {type: git, git: {remoteUrl: /a.git, target: main, repoPath: /copy}}\n" +
+ " scorer:\n" +
+ " base:\n" +
+ " type: jev\n" +
+ " jev: {diff: {git: {remoteUrl: /b.git, target: main, repoPath: /copy}}}\n",
+ },
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ _, err := loadProfilesConfig(writeProfiles(t, tt.contents))
+ require.Error(t, err)
+ })
+ }
+}
+
+func TestNewProfiles_JevScorer(t *testing.T) {
+ batch := entity.Batch{ID: "1", Queue: "q", Contains: []string{"1"}}
+ profilesYAML := "" +
+ "defaults:\n" +
+ " scorer:\n" +
+ " base:\n" +
+ " type: jev\n" +
+ " jev: {baseUrlEnv: TEST_JEV_URL, tokenEnv: TEST_JEV_KEY, model: test-jev, path: /typesafe/v1/systemone}\n" +
+ " fallback:\n" +
+ " type: heuristic\n" +
+ " buckets: [{min: 0, max: 2147483647, score: 0.25}]\n"
+
+ scoreThroughProfiles := func(t *testing.T) float64 {
+ t.Helper()
+ cfg, err := loadProfilesConfig(writeProfiles(t, profilesYAML))
+ require.NoError(t, err)
+ profiles, err := newProfiles(context.Background(), zaptest.NewLogger(t), tally.NoopScope, changesetfake.New(), nil, cfg)
+ require.NoError(t, err)
+ s, err := profiles.ScorerFactory().For(scorer.Config{QueueName: "q"})
+ require.NoError(t, err)
+ got, err := s.Score(context.Background(), batch, entity.SpeculationPathSet{})
+ require.NoError(t, err)
+ return got
+ }
+
+ t.Run("asks the proxy with the configured key and model", func(t *testing.T) {
+ srv := httptest.NewServer(nethttp.HandlerFunc(func(w nethttp.ResponseWriter, r *nethttp.Request) {
+ assert.Equal(t, "/typesafe/v1/systemone", r.URL.Path)
+ assert.Equal(t, "Bearer sk-test", r.Header.Get("Authorization"))
+ _, _ = w.Write([]byte(`{"model":"test-jev","answers":{"batch_lands":{"type":"noul","noul":0.8}}}`))
+ }))
+ t.Cleanup(srv.Close)
+ t.Setenv("TEST_JEV_URL", srv.URL)
+ t.Setenv("TEST_JEV_KEY", "sk-test")
+
+ assert.InDelta(t, 0.8, scoreThroughProfiles(t), 1e-9)
+ })
+
+ t.Run("falls back without credentials", func(t *testing.T) {
+ t.Setenv("TEST_JEV_URL", "")
+ t.Setenv("TEST_JEV_KEY", "")
+
+ assert.InDelta(t, 0.25, scoreThroughProfiles(t), 1e-9)
+ })
+}
+
+func TestNewProfiles_JevScorerSendsDiffsFromTheChangeRepository(t *testing.T) {
+ git := gitexectest.Git(t)
+ root := t.TempDir()
+ remote := filepath.Join(root, "remote.git")
+ work := filepath.Join(root, "work")
+ run := func(dir string, args ...string) string {
+ t.Helper()
+ cmd := exec.Command(git, args...)
+ cmd.Dir = dir
+ out, err := cmd.CombinedOutput()
+ require.NoError(t, err, "git %v: %s", args, out)
+ return strings.TrimSpace(string(out))
+ }
+ run("", "init", "--bare", "-b", "main", remote)
+ run("", "clone", remote, work)
+ run(work, "config", "user.name", "Test")
+ run(work, "config", "user.email", "test@example.invalid")
+ require.NoError(t, os.WriteFile(filepath.Join(work, "seed.txt"), []byte("seed\n"), 0o644))
+ run(work, "add", ".")
+ run(work, "commit", "-m", "seed")
+ run(work, "push", "origin", "main")
+ run(work, "checkout", "-B", "feature/x", "main")
+ require.NoError(t, os.WriteFile(filepath.Join(work, "one.go"), []byte("package one\n\nfunc Broken( {\n"), 0o644))
+ require.NoError(t, os.WriteFile(filepath.Join(work, "seed.txt"), []byte("seed, edited\n"), 0o644))
+ run(work, "add", "-A")
+ run(work, "commit", "-m", "change")
+ run(work, "push", "origin", "feature/x")
+ uri := fmt.Sprintf("git://git.example.com/demo/%s/%s", url.PathEscape("refs/heads/feature/x"), run(work, "rev-parse", "HEAD"))
+
+ var sentDiff string
+ var sentFiles int
+ srv := httptest.NewServer(nethttp.HandlerFunc(func(w nethttp.ResponseWriter, r *nethttp.Request) {
+ var body struct {
+ State struct {
+ Changes []struct {
+ Diff string `json:"diff"`
+ FileContents []json.RawMessage `json:"modified_files_after_change"`
+ } `json:"changes"`
+ } `json:"state"`
+ }
+ assert.NoError(t, json.NewDecoder(r.Body).Decode(&body))
+ if assert.Len(t, body.State.Changes, 1) {
+ sentDiff = body.State.Changes[0].Diff
+ sentFiles = len(body.State.Changes[0].FileContents)
+ }
+ _, _ = w.Write([]byte(`{"answers":{"batch_lands":{"type":"noul","noul":0.1}}}`))
+ }))
+ t.Cleanup(srv.Close)
+ t.Setenv("TEST_JEV_URL", srv.URL)
+ t.Setenv("TEST_JEV_KEY", "sk-test")
+
+ gitBlock := fmt.Sprintf("{remoteUrl: %s, target: main, repoPath: %s}", remote, filepath.Join(root, "copy.git"))
+ cfg, err := loadProfilesConfig(writeProfiles(t, fmt.Sprintf(`
+queues:
+ - name: q
+ changeProvider: {type: git, git: %s}
+ scorer:
+ base:
+ type: jev
+ jev: {baseUrlEnv: TEST_JEV_URL, tokenEnv: TEST_JEV_KEY, diff: {git: %s, files: true}}
+`, gitBlock, gitBlock)))
+ require.NoError(t, err)
+
+ batch := entity.Batch{ID: "1", Queue: "q", Contains: []string{"1"}}
+ resolver := changesetfake.New().
+ Set(batch.ID, change.Change{URIs: []string{uri}}).
+ SetDetailed(batch.ID, entity.BatchChanges{BatchID: batch.ID, Queue: batch.Queue, Changes: []entity.ChangeInfo{{URI: uri}}})
+ profiles, err := newProfiles(context.Background(), zaptest.NewLogger(t), tally.NoopScope, resolver, nil, cfg)
+ require.NoError(t, err)
+ s, err := profiles.ScorerFactory().For(scorer.Config{QueueName: "q"})
+ require.NoError(t, err)
+
+ got, err := s.Score(context.Background(), batch, entity.SpeculationPathSet{})
+ require.NoError(t, err)
+ assert.InDelta(t, 0.1, got, 1e-9)
+ assert.Contains(t, sentDiff, "+func Broken( {", "the model must see the change's code")
+ assert.Equal(t, 1, sentFiles, "files: true sends the modified file in full; the added one is already whole in the diff")
+}
diff --git a/service/submitqueue/orchestrator/server/oracle_config_test.go b/service/submitqueue/orchestrator/server/oracle_config_test.go
new file mode 100644
index 000000000..c50ab02ef
--- /dev/null
+++ b/service/submitqueue/orchestrator/server/oracle_config_test.go
@@ -0,0 +1,130 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "os"
+ "path/filepath"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+ "github.com/uber-go/tally"
+ "go.uber.org/zap/zaptest"
+
+ "github.com/uber/submitqueue/platform/base/change"
+ changesetfake "github.com/uber/submitqueue/submitqueue/core/changeset/fake"
+ "github.com/uber/submitqueue/submitqueue/entity"
+ "github.com/uber/submitqueue/submitqueue/extension/buildrunner"
+ "github.com/uber/submitqueue/submitqueue/extension/buildrunner/replay"
+ "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer"
+ "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer/oracle"
+)
+
+func TestNewProfiles_OracleScorer(t *testing.T) {
+ const uri = "github://github.example.com/o/r/pull/1/1111111111111111111111111111111111111111"
+ cfg, err := loadProfilesConfig(writeProfiles(t, "defaults:\n scorer:\n base: {type: oracle}\n"))
+ require.NoError(t, err)
+
+ for name, tt := range map[string]struct {
+ uri string
+ want float64
+ }{
+ "a marked failure is priced as one": {uri: uri + "?sq-fake=build-fail", want: oracle.PriceFails},
+ "an unmarked change is priced as a pass": {uri: uri, want: oracle.PricePasses},
+ } {
+ t.Run(name, func(t *testing.T) {
+ batch := entity.Batch{ID: "1", Queue: "q", Contains: []string{"1"}}
+ resolver := changesetfake.New().Set(batch.ID, change.Change{URIs: []string{tt.uri}})
+ profiles, err := newProfiles(context.Background(), zaptest.NewLogger(t), tally.NoopScope, resolver, nil, cfg)
+ require.NoError(t, err)
+ s, err := profiles.ScorerFactory().For(scorer.Config{QueueName: "q"})
+ require.NoError(t, err)
+ got, err := s.Score(context.Background(), batch, entity.SpeculationPathSet{})
+ require.NoError(t, err)
+ assert.InDelta(t, tt.want, got, 1e-9)
+ })
+ }
+}
+
+func TestNewProfiles_ReplayRunnerAndOracleShareADataset(t *testing.T) {
+ const (
+ passSHA = "1111111111111111111111111111111111111111"
+ failSHA = "2222222222222222222222222222222222222222"
+ )
+ dir := t.TempDir()
+ dsPath := filepath.Join(dir, "ds.json")
+ raw, err := json.Marshal(replay.Dataset{Attempts: []replay.Attempt{
+ {PR: 1, ReplaySHA: passSHA, Outcome: replay.OutcomePass},
+ {PR: 2, ReplaySHA: failSHA, Outcome: replay.OutcomeFail},
+ }})
+ require.NoError(t, err)
+ require.NoError(t, os.WriteFile(dsPath, raw, 0o644))
+
+ cfg, err := loadProfilesConfig(writeProfiles(t, fmt.Sprintf(`
+defaults:
+ buildRunner: {type: replay, dataset: %s}
+ scorer:
+ base: {type: oracle, oracle: {dataset: %s}}
+`, dsPath, dsPath)))
+ require.NoError(t, err)
+
+ for name, tt := range map[string]struct {
+ sha string
+ wantPrice float64
+ wantStatus entity.BuildStatus
+ }{
+ "recorded pass": {sha: passSHA, wantPrice: oracle.PricePasses, wantStatus: entity.BuildStatusSucceeded},
+ "recorded failure": {sha: failSHA, wantPrice: oracle.PriceFails, wantStatus: entity.BuildStatusFailed},
+ } {
+ t.Run(name, func(t *testing.T) {
+ batch := entity.Batch{ID: "1", Queue: "q", Contains: []string{"1"}}
+ uri := "git://git.example.com/sandbox/refs%2Fheads%2Freplay%2Fpr-1/" + tt.sha
+ resolver := changesetfake.New().Set(batch.ID, change.Change{URIs: []string{uri}})
+ profiles, err := newProfiles(context.Background(), zaptest.NewLogger(t), tally.NoopScope, resolver, nil, cfg)
+ require.NoError(t, err)
+
+ s, err := profiles.ScorerFactory().For(scorer.Config{QueueName: "q"})
+ require.NoError(t, err)
+ price, err := s.Score(context.Background(), batch, entity.SpeculationPathSet{})
+ require.NoError(t, err)
+ assert.InDelta(t, tt.wantPrice, price, 1e-9)
+
+ runner, err := profiles.BuildRunnerFactory().For(buildrunner.Config{QueueName: "q"})
+ require.NoError(t, err)
+ id, err := runner.Trigger(context.Background(), nil, batch, nil)
+ require.NoError(t, err)
+ status, _, err := runner.Status(context.Background(), id)
+ require.NoError(t, err)
+ assert.Equal(t, tt.wantStatus, status)
+ })
+ }
+}
+
+func TestLoadProfilesConfig_RejectsBadReplayAndOracle(t *testing.T) {
+ for name, contents := range map[string]string{
+ "replay without a dataset": "defaults:\n buildRunner: {type: replay}\n",
+ "negative time scale": "defaults:\n buildRunner: {type: replay, dataset: /d.json, timeScale: -1}\n",
+ "oracle block on heuristic": "defaults:\n scorer:\n base: {type: heuristic, oracle: {dataset: /d.json}}\n",
+ } {
+ t.Run(name, func(t *testing.T) {
+ _, err := loadProfilesConfig(writeProfiles(t, contents))
+ require.Error(t, err)
+ })
+ }
+}
diff --git a/service/submitqueue/orchestrator/server/profiles.go b/service/submitqueue/orchestrator/server/profiles.go
index 83c1ec3a5..45d28b09a 100644
--- a/service/submitqueue/orchestrator/server/profiles.go
+++ b/service/submitqueue/orchestrator/server/profiles.go
@@ -17,6 +17,7 @@ package main
import (
"fmt"
nethttp "net/http"
+ "sync"
orchstorage "github.com/uber/submitqueue/submitqueue/orchestrator/extension/storage"
@@ -28,6 +29,7 @@ import (
"context"
platformbuildkite "github.com/uber/submitqueue/platform/buildkite"
+ gitrepo "github.com/uber/submitqueue/platform/git/repo"
platformgithubactions "github.com/uber/submitqueue/platform/githubactions"
"github.com/uber/submitqueue/platform/http"
"github.com/uber/submitqueue/submitqueue/core/changeset"
@@ -36,6 +38,9 @@ import (
buildkiterunner "github.com/uber/submitqueue/submitqueue/extension/buildrunner/buildkite"
buildfake "github.com/uber/submitqueue/submitqueue/extension/buildrunner/fake"
githubactionsrunner "github.com/uber/submitqueue/submitqueue/extension/buildrunner/githubactions"
+ "github.com/uber/submitqueue/submitqueue/extension/buildrunner/replay"
+ "github.com/uber/submitqueue/submitqueue/extension/changediff"
+ gitdiffer "github.com/uber/submitqueue/submitqueue/extension/changediff/git"
"github.com/uber/submitqueue/submitqueue/extension/changeprovider"
cpfake "github.com/uber/submitqueue/submitqueue/extension/changeprovider/fake"
gitprovider "github.com/uber/submitqueue/submitqueue/extension/changeprovider/git"
@@ -54,6 +59,8 @@ import (
"github.com/uber/submitqueue/submitqueue/extension/speculation/scorer/evidence"
scorerfake "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer/fake"
"github.com/uber/submitqueue/submitqueue/extension/speculation/scorer/heuristic"
+ jevscorer "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer/jev"
+ "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer/oracle"
"github.com/uber/submitqueue/submitqueue/extension/speculation/speculator"
specstandard "github.com/uber/submitqueue/submitqueue/extension/speculation/speculator/standard"
)
@@ -248,6 +255,35 @@ type profileBuilder struct {
resolver changeset.Resolver
stores orchstorage.Factory
built map[string]any
+
+ // jevMu guards jevShared, which buildScorer reaches from concurrent
+ // speculation runs rather than only at startup.
+ jevMu sync.Mutex
+ jevShared map[string]jevShared
+
+ // replayMu guards replayIndexes: one load per dataset path, shared by the
+ // replay runner and the oracle reading it.
+ replayMu sync.Mutex
+ replayIndexes map[string]replay.Index
+
+ // repoMu guards changeRepos: one copy per repoPath, so a change provider
+ // and a jev diff source naming the same path share the copy and its lock.
+ repoMu sync.Mutex
+ changeRepos map[string]*gitrepo.Repo
+}
+
+// jevShared is what every jev scorer built from one configuration shares: one
+// HTTP client and one price cache. A nil client means the endpoint or key was
+// not set, so those scorers always fall back.
+type jevShared struct {
+ client *nethttp.Client
+ cache *jevscorer.Cache
+ // diffRepo is the copy diffs are read from; nil sends no code.
+ diffRepo *gitrepo.Repo
+ // diffFiles and diffReferences select the context sent with each diff.
+ diffFiles, diffReferences bool
+ // maxDiffBytes and maxContextBytes cap the code sent per batch.
+ maxDiffBytes, maxContextBytes int
}
func (b *profileBuilder) build(cfg queueProfileConfig, where string) (Profile, error) {
@@ -375,11 +411,164 @@ func (b *profileBuilder) buildScorer(c scorer.Config, cfg scorerConfig, where, s
// avg is the only combine the schema admits.
return composite.New(c, components, composite.Avg, b.scope.SubScope(scopeName)), nil
+ case scorerTypeJev:
+ fallback, err := b.buildScorer(c, *cfg.Fallback, where, scopeName+".fallback")
+ if err != nil {
+ return nil, err
+ }
+ shared, err := b.jevSharedFor(*cfg.Jev, where)
+ if err != nil {
+ return nil, err
+ }
+ var differ changediff.Provider
+ if shared.diffRepo != nil {
+ differ = gitdiffer.New(gitdiffer.Params{
+ Config: changediff.Config{QueueName: c.QueueName},
+ Repo: shared.diffRepo,
+ MetricsScope: b.scope.SubScope(scopeName),
+ IncludeFiles: shared.diffFiles,
+ IncludeReferences: shared.diffReferences,
+ })
+ }
+ return jevscorer.New(jevscorer.Params{
+ Config: c,
+ HTTPClient: shared.client,
+ Differ: differ,
+ MaxDiffBytes: shared.maxDiffBytes,
+ MaxContextBytes: shared.maxContextBytes,
+ Path: cfg.Jev.Path,
+ Model: cfg.Jev.Model,
+ Timeout: timeoutOr(cfg.Jev.Timeout, defaultJevTimeout),
+ Resolver: b.resolver,
+ Cache: shared.cache,
+ Fallback: fallback,
+ Logger: b.logger.Sugar(),
+ MetricsScope: b.scope.SubScope(scopeName),
+ }), nil
+
+ case scorerTypeOracle:
+ fallback, err := b.buildScorer(c, *cfg.Fallback, where, scopeName+".fallback")
+ if err != nil {
+ return nil, err
+ }
+ b.logger.Warn("oracle scorer configured: it reads recorded outcomes and is for benchmarks only",
+ zap.String("profile", where), zap.String("queue", c.QueueName))
+ truth := oracle.MarkerTruth()
+ if cfg.Oracle != nil && cfg.Oracle.Dataset != "" {
+ idx, err := b.replayIndexFor(cfg.Oracle.Dataset)
+ if err != nil {
+ return nil, fmt.Errorf("%s: %w", where, err)
+ }
+ truth = replay.Truth(idx)
+ }
+ return oracle.New(oracle.Params{Config: c, Resolver: b.resolver, Truth: truth, Fallback: fallback}), nil
+
default:
return nil, fmt.Errorf("%s: unknown scorer type %q", where, cfg.Type)
}
}
+// jevSharedFor returns the client and cache for a jev configuration, building
+// them on first use. A missing endpoint or key is logged once and leaves the
+// client nil rather than failing startup, so a stack without credentials still
+// runs on the fallback.
+func (b *profileBuilder) jevSharedFor(cfg jevScorerConfig, where string) (jevShared, error) {
+ encoded, err := yamlv3.Marshal(cfg)
+ if err != nil {
+ return jevShared{}, fmt.Errorf("%s: failed to key jev config: %w", where, err)
+ }
+ key := string(encoded)
+
+ b.jevMu.Lock()
+ defer b.jevMu.Unlock()
+ if shared, ok := b.jevShared[key]; ok {
+ return shared, nil
+ }
+
+ shared := jevShared{cache: jevscorer.NewCache(cfg.CacheSize)}
+ if cfg.Diff != nil {
+ repo, err := b.changeRepoFor(*cfg.Diff.Git)
+ if err != nil {
+ return jevShared{}, fmt.Errorf("%s: jev diff: %w", where, err)
+ }
+ shared.diffRepo = repo
+ shared.diffFiles = cfg.Diff.Files
+ shared.diffReferences = cfg.Diff.References
+ shared.maxDiffBytes = cfg.Diff.MaxDiffBytes
+ shared.maxContextBytes = cfg.Diff.MaxContextBytes
+ }
+ baseURL := cfg.BaseURL
+ if baseURL == "" {
+ baseURL, _ = tokenFrom(cfg.BaseURLEnv)
+ }
+ token, hasToken := tokenFrom(cfg.TokenEnv)
+ switch {
+ case baseURL == "":
+ b.logger.Warn("jev scorer has no endpoint; every batch uses the fallback scorer",
+ zap.String("profile", where), zap.String("base_url_env", cfg.BaseURLEnv))
+ case !hasToken:
+ b.logger.Warn("jev scorer has no key; every batch uses the fallback scorer",
+ zap.String("profile", where), zap.String("token_env", cfg.TokenEnv))
+ default:
+ client, err := http.NewClient(baseURL)
+ if err != nil {
+ return jevShared{}, fmt.Errorf("%s: failed to build jev HTTP client: %w", where, err)
+ }
+ client.Timeout = timeoutOr(cfg.Timeout, defaultJevTimeout)
+ client.Transport = &oauth2.Transport{
+ Source: oauth2.StaticTokenSource(&oauth2.Token{AccessToken: token}),
+ Base: client.Transport,
+ }
+ shared.client = client
+ b.logger.Info("jev scorer configured",
+ zap.String("profile", where), zap.String("base_url", baseURL), zap.String("path", cfg.Path), zap.String("model", cfg.Model))
+ }
+
+ if b.jevShared == nil {
+ b.jevShared = make(map[string]jevShared)
+ }
+ b.jevShared[key] = shared
+ return shared, nil
+}
+
+// replayIndexFor loads the replay dataset at path once.
+func (b *profileBuilder) replayIndexFor(path string) (replay.Index, error) {
+ b.replayMu.Lock()
+ defer b.replayMu.Unlock()
+ if idx, ok := b.replayIndexes[path]; ok {
+ return idx, nil
+ }
+ ds, err := replay.Load(path)
+ if err != nil {
+ return nil, err
+ }
+ idx := replay.NewIndex(ds)
+ if b.replayIndexes == nil {
+ b.replayIndexes = make(map[string]replay.Index)
+ }
+ b.replayIndexes[path] = idx
+ return idx, nil
+}
+
+// changeRepoFor returns the provisioned copy at cfg.RepoPath, provisioning it on
+// first use. validateGitRepoPaths guarantees one path never names two remotes.
+func (b *profileBuilder) changeRepoFor(cfg gitProviderConfig) (*gitrepo.Repo, error) {
+ b.repoMu.Lock()
+ defer b.repoMu.Unlock()
+ if repo, ok := b.changeRepos[cfg.RepoPath]; ok {
+ return repo, nil
+ }
+ repo, err := newChangeRepo(b.ctx, cfg)
+ if err != nil {
+ return nil, err
+ }
+ if b.changeRepos == nil {
+ b.changeRepos = make(map[string]*gitrepo.Repo)
+ }
+ b.changeRepos[cfg.RepoPath] = repo
+ return repo, nil
+}
+
// reuse returns the instance already built for an identical configuration, or
// builds and remembers one. Identity is the configuration's own YAML encoding,
// which compares nested optional blocks by value rather than by pointer.
@@ -410,10 +599,9 @@ func (b *profileBuilder) newChangeProviderFactory(cfg changeProviderConfig, wher
}), nil
case changeProviderTypeGit:
- // One copy per distinct configuration, provisioned now. Queues whose
- // blocks are identical share it — and therefore share its lock — because
- // reuse hands them the same factory.
- repo, err := newChangeRepo(b.ctx, *cfg.Git)
+ // One copy per repoPath, provisioned now and shared — with its lock —
+ // by every queue and jev diff source naming that path.
+ repo, err := b.changeRepoFor(*cfg.Git)
if err != nil {
return nil, fmt.Errorf("%s: %w", where, err)
}
@@ -563,6 +751,17 @@ func (b *profileBuilder) newBuildRunnerFactory(cfg buildRunnerConfig, where stri
return buildfake.New(c, b.resolver), nil
}), nil
+ case buildRunnerTypeReplay:
+ idx, err := b.replayIndexFor(cfg.Dataset)
+ if err != nil {
+ return nil, fmt.Errorf("%s: %w", where, err)
+ }
+ b.logger.Warn("replay build runner configured: builds answer from recorded CI history; benchmarks only",
+ zap.String("profile", where), zap.String("dataset", cfg.Dataset), zap.Int("recorded", len(idx)))
+ return buildRunnerFunc(func(c buildrunner.Config) (buildrunner.BuildRunner, error) {
+ return replay.New(replay.Params{Config: c, Resolver: b.resolver, Index: idx, TimeScale: cfg.TimeScale}), nil
+ }), nil
+
case buildRunnerTypeGitHubActions:
token, ok := tokenFrom(cfg.TokenEnv)
if !ok {
diff --git a/submitqueue/entity/BUILD.bazel b/submitqueue/entity/BUILD.bazel
index d3fcf8eb0..b5a9e39a0 100644
--- a/submitqueue/entity/BUILD.bazel
+++ b/submitqueue/entity/BUILD.bazel
@@ -8,6 +8,7 @@ go_library(
"batch_dependent.go",
"build.go",
"cancel.go",
+ "change_diff.go",
"change_provider.go",
"change_record.go",
"conflict.go",
diff --git a/submitqueue/entity/change_diff.go b/submitqueue/entity/change_diff.go
new file mode 100644
index 000000000..258ba1660
--- /dev/null
+++ b/submitqueue/entity/change_diff.go
@@ -0,0 +1,77 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package entity
+
+// ChangeDiff is the code content of one change: the unified diff from where the
+// change diverged from the change before it in its stack (or from the target,
+// for the first) to the change's head, plus optional surrounding context read
+// at the change's head.
+type ChangeDiff struct {
+ // URI is the change URI this diff describes.
+ URI string `json:"uri"`
+ // Patch is the unified diff text. Binary files appear as git's one-line
+ // "Binary files ... differ" marker rather than content.
+ Patch string `json:"patch"`
+ // Files is the post-change content of each text file the change modifies,
+ // in path order. Added files are omitted: they are already whole in Patch.
+ // Empty unless the provider was asked for files.
+ Files []FileContent `json:"files,omitempty"`
+ // References are lines elsewhere in the repository, at the change's head,
+ // related to symbols the change touches. Empty unless the provider was
+ // asked for references.
+ References []CodeReference `json:"references,omitempty"`
+}
+
+// FileContent is the full text of one file at a commit.
+type FileContent struct {
+ // Path is the file path relative to the repository root.
+ Path string `json:"path"`
+ // Content is the file's text.
+ Content string `json:"content"`
+}
+
+// CodeReferenceKind says how a reference relates to the change.
+type CodeReferenceKind string
+
+const (
+ // CodeReferenceKindUnknown is the zero-value sentinel.
+ CodeReferenceKindUnknown CodeReferenceKind = ""
+ // CodeReferenceKindCaller is a use of a symbol whose definition the change
+ // modifies or removes, in a file the change does not touch.
+ CodeReferenceKindCaller CodeReferenceKind = "caller"
+ // CodeReferenceKindDefinition is the definition of a symbol the change calls
+ // but does not itself define.
+ CodeReferenceKindDefinition CodeReferenceKind = "definition"
+)
+
+// CodeReference is one line of code related to a symbol a change touches.
+type CodeReference struct {
+ // Kind says how the line relates to the change.
+ Kind CodeReferenceKind `json:"kind"`
+ // Symbol is the identifier the line was found for.
+ Symbol string `json:"symbol"`
+ // Path is the file path relative to the repository root.
+ Path string `json:"path"`
+ // Line is the 1-based line number within Path.
+ Line int `json:"line"`
+ // Text is the line's content, without its trailing newline.
+ Text string `json:"text"`
+ // DeclarationBefore is, for a caller, the symbol's declaration line as the
+ // change removed it. Empty for a definition.
+ DeclarationBefore string `json:"declaration_before,omitempty"`
+ // DeclarationAfter is, for a caller, the symbol's declaration line as the
+ // change rewrote it, or empty when the change removes the symbol.
+ DeclarationAfter string `json:"declaration_after,omitempty"`
+}
diff --git a/submitqueue/extension/buildrunner/replay/BUILD.bazel b/submitqueue/extension/buildrunner/replay/BUILD.bazel
new file mode 100644
index 000000000..3baeb84f8
--- /dev/null
+++ b/submitqueue/extension/buildrunner/replay/BUILD.bazel
@@ -0,0 +1,33 @@
+load("@rules_go//go:def.bzl", "go_library", "go_test")
+
+go_library(
+ name = "go_default_library",
+ srcs = [
+ "dataset.go",
+ "runner.go",
+ "truth.go",
+ ],
+ importpath = "github.com/uber/submitqueue/submitqueue/extension/buildrunner/replay",
+ visibility = ["//visibility:public"],
+ deps = [
+ "//platform/base/change:go_default_library",
+ "//platform/base/change/git:go_default_library",
+ "//submitqueue/core/changeset:go_default_library",
+ "//submitqueue/entity:go_default_library",
+ "//submitqueue/extension/buildrunner:go_default_library",
+ "//submitqueue/extension/speculation/scorer/oracle:go_default_library",
+ ],
+)
+
+go_test(
+ name = "go_default_test",
+ srcs = ["replay_test.go"],
+ embed = [":go_default_library"],
+ deps = [
+ "//platform/base/change:go_default_library",
+ "//submitqueue/core/changeset/fake:go_default_library",
+ "//submitqueue/entity:go_default_library",
+ "@com_github_stretchr_testify//assert:go_default_library",
+ "@com_github_stretchr_testify//require:go_default_library",
+ ],
+)
diff --git a/submitqueue/extension/buildrunner/replay/README.md b/submitqueue/extension/buildrunner/replay/README.md
new file mode 100644
index 000000000..cc1df064a
--- /dev/null
+++ b/submitqueue/extension/buildrunner/replay/README.md
@@ -0,0 +1,9 @@
+# replay
+
+A `BuildRunner` that answers every build from a recorded CI history, and an `oracle.Truth` reading the same record. Benchmarks only: it never runs anything, and must never configure a production queue. See [doc/rfc/submitqueue/replay-benchmark.md](../../../../doc/rfc/submitqueue/replay-benchmark.md).
+
+A dataset (`Dataset`, written by `tool/replaybench`) lists one repository's merge-queue attempts with their recorded outcome and duration, each rebuilt as a sandbox commit (`ReplaySHA`). The runner looks changes up by the commit SHA in their `git://` URI. A build fails if any change in its base or head is recorded as failing, and finishes after the longest recorded duration among them divided by `TimeScale`. Unrecorded changes pass instantly. The outcome and due time are encoded in the build ID, so `Status` holds no state and any instance answers for any build.
+
+`Truth` reports a batch failing if any of its changes is recorded as failing, and unknown if none is recorded, so an `oracle` configured with the same dataset agrees with the runner on every head.
+
+Configured in the orchestrator's `profiles.yaml` as `buildRunner: {type: replay, dataset: , timeScale: }` and, for the ceiling, `scorer: {base: {type: oracle, oracle: {dataset: }}}`.
diff --git a/submitqueue/extension/buildrunner/replay/dataset.go b/submitqueue/extension/buildrunner/replay/dataset.go
new file mode 100644
index 000000000..b6907a1e7
--- /dev/null
+++ b/submitqueue/extension/buildrunner/replay/dataset.go
@@ -0,0 +1,108 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package replay replays recorded CI outcomes: a buildrunner.BuildRunner that
+// answers each build with the result and duration a real CI system recorded
+// for the same change, and an oracle Truth backed by the same record. It
+// exists for benchmarking scorers against real history and must never run a
+// production queue. See README.md and doc/rfc/submitqueue/replay-benchmark.md.
+package replay
+
+import (
+ "encoding/json"
+ "fmt"
+ "os"
+)
+
+// Outcome is the recorded CI result of one attempt.
+type Outcome string
+
+const (
+ // OutcomeUnknown is the zero-value sentinel.
+ OutcomeUnknown Outcome = ""
+ // OutcomePass means every recorded CI run for the attempt succeeded.
+ OutcomePass Outcome = "pass"
+ // OutcomeFail means at least one recorded CI run for the attempt failed.
+ OutcomeFail Outcome = "fail"
+)
+
+// Dataset is a window of one repository's merge-queue history.
+type Dataset struct {
+ // Repo is the upstream repository as owner/name.
+ Repo string `json:"repo"`
+ // BaseBranch is the upstream branch the queue landed onto.
+ BaseBranch string `json:"base_branch"`
+ // BaseSHA is the upstream commit every replayed change is rebuilt on.
+ BaseSHA string `json:"base_sha"`
+ // Attempts are the queue attempts in the window, oldest first.
+ Attempts []Attempt `json:"attempts"`
+}
+
+// Attempt is one change's trip through the upstream merge queue.
+type Attempt struct {
+ // PR is the upstream pull request number.
+ PR int `json:"pr"`
+ // Title is the upstream merge-group commit subject.
+ Title string `json:"title,omitempty"`
+ // GroupSHA is the upstream merge-group commit CI tested.
+ GroupSHA string `json:"group_sha"`
+ // ParentSHA is GroupSHA's parent: the queue head the PR was tested on.
+ ParentSHA string `json:"parent_sha"`
+ // Outcome is the recorded CI result for GroupSHA.
+ Outcome Outcome `json:"outcome"`
+ // DurationMs is the longest recorded CI run for GroupSHA, in milliseconds.
+ DurationMs int64 `json:"duration_ms"`
+ // QueuedAtMs is when the first CI run for GroupSHA was created, in Unix ms.
+ QueuedAtMs int64 `json:"queued_at_ms"`
+ // ReplaySHA is the sandbox commit carrying only this PR's change on
+ // BaseSHA. Empty until the sandbox is materialized, or if it was skipped.
+ ReplaySHA string `json:"replay_sha,omitempty"`
+ // Branch is the sandbox branch whose head is ReplaySHA.
+ Branch string `json:"branch,omitempty"`
+ // Skipped says why the attempt has no ReplaySHA, when it has none.
+ Skipped string `json:"skipped,omitempty"`
+ // PatchID is git's stable patch ID of the PR's own change.
+ PatchID string `json:"patch_id,omitempty"`
+ // FlakySuspect marks a failure whose change is identical to a later
+ // passing attempt of the same PR: the failure came from something other
+ // than the code, such as infrastructure or another change in its group.
+ FlakySuspect bool `json:"flaky_suspect,omitempty"`
+}
+
+// Load reads a dataset written by tool/replaybench.
+func Load(path string) (Dataset, error) {
+ raw, err := os.ReadFile(path)
+ if err != nil {
+ return Dataset{}, fmt.Errorf("read replay dataset: %w", err)
+ }
+ var ds Dataset
+ if err := json.Unmarshal(raw, &ds); err != nil {
+ return Dataset{}, fmt.Errorf("parse replay dataset %s: %w", path, err)
+ }
+ return ds, nil
+}
+
+// Index looks attempts up by their replayed commit.
+type Index map[string]Attempt
+
+// NewIndex indexes the materialized attempts of ds by ReplaySHA.
+func NewIndex(ds Dataset) Index {
+ idx := make(Index, len(ds.Attempts))
+ for _, a := range ds.Attempts {
+ if a.ReplaySHA != "" {
+ idx[a.ReplaySHA] = a
+ }
+ }
+ return idx
+}
diff --git a/submitqueue/extension/buildrunner/replay/replay_test.go b/submitqueue/extension/buildrunner/replay/replay_test.go
new file mode 100644
index 000000000..e7f5aae80
--- /dev/null
+++ b/submitqueue/extension/buildrunner/replay/replay_test.go
@@ -0,0 +1,140 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package replay
+
+import (
+ "context"
+ "encoding/json"
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+
+ "github.com/uber/submitqueue/platform/base/change"
+ changesetfake "github.com/uber/submitqueue/submitqueue/core/changeset/fake"
+ "github.com/uber/submitqueue/submitqueue/entity"
+)
+
+const (
+ passSHA = "1111111111111111111111111111111111111111"
+ failSHA = "2222222222222222222222222222222222222222"
+ unknownSHA = "3333333333333333333333333333333333333333"
+)
+
+func uriFor(sha string) string {
+ return "git://git.example.com/sandbox/refs%2Fheads%2Freplay%2Fpr-1/" + sha
+}
+
+func testIndex() Index {
+ return NewIndex(Dataset{Attempts: []Attempt{
+ {PR: 1, ReplaySHA: passSHA, Outcome: OutcomePass, DurationMs: 60_000},
+ {PR: 2, ReplaySHA: failSHA, Outcome: OutcomeFail, DurationMs: 120_000},
+ {PR: 3, Outcome: OutcomePass, Skipped: "conflict"},
+ }})
+}
+
+type clock struct{ now time.Time }
+
+func (c *clock) Now() time.Time { return c.now }
+
+func TestRunner(t *testing.T) {
+ head := entity.Batch{ID: "h", Queue: "q"}
+ dep := entity.Batch{ID: "d", Queue: "q"}
+ tests := []struct {
+ name string
+ headSHA string
+ baseSHA string
+ wantDone entity.BuildStatus
+ wantWait time.Duration
+ }{
+ {name: "recorded pass", headSHA: passSHA, wantDone: entity.BuildStatusSucceeded, wantWait: 6 * time.Second},
+ {name: "recorded failure", headSHA: failSHA, wantDone: entity.BuildStatusFailed, wantWait: 12 * time.Second},
+ // A path that assumes a failing dependency succeeds builds that
+ // dependency in, and so fails whatever the head is.
+ {name: "failing change in the base fails the build", headSHA: passSHA, baseSHA: failSHA, wantDone: entity.BuildStatusFailed, wantWait: 12 * time.Second},
+ {name: "unrecorded change passes at once", headSHA: unknownSHA, wantDone: entity.BuildStatusSucceeded},
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ resolver := changesetfake.New().Set(head.ID, change.Change{URIs: []string{uriFor(tt.headSHA)}})
+ var base []entity.Batch
+ if tt.baseSHA != "" {
+ resolver.Set(dep.ID, change.Change{URIs: []string{uriFor(tt.baseSHA)}})
+ base = []entity.Batch{dep}
+ }
+ c := &clock{now: time.UnixMilli(1_000_000)}
+ r := New(Params{Resolver: resolver, Index: testIndex(), TimeScale: 10, Now: c.Now})
+
+ id, err := r.Trigger(context.Background(), base, head, nil)
+ require.NoError(t, err)
+ if tt.wantWait > 0 {
+ c.now = c.now.Add(tt.wantWait - time.Millisecond)
+ status, _, err := r.Status(context.Background(), id)
+ require.NoError(t, err)
+ assert.Equal(t, entity.BuildStatusRunning, status, "the scaled recorded duration has not elapsed")
+ }
+ c.now = c.now.Add(time.Millisecond)
+ status, _, err := r.Status(context.Background(), id)
+ require.NoError(t, err)
+ assert.Equal(t, tt.wantDone, status)
+ })
+ }
+
+ t.Run("a foreign build ID is an error", func(t *testing.T) {
+ _, _, err := New(Params{}).Status(context.Background(), entity.BuildID{ID: "fake-ok-abc"})
+ require.Error(t, err)
+ })
+}
+
+func TestTruth(t *testing.T) {
+ tr := Truth(testIndex())
+ for name, tt := range map[string]struct {
+ shas []string
+ wantFails bool
+ wantKnown bool
+ }{
+ "recorded pass": {shas: []string{passSHA}, wantKnown: true},
+ "recorded failure": {shas: []string{failSHA}, wantFails: true, wantKnown: true},
+ "one failing change fails the batch": {shas: []string{passSHA, failSHA}, wantFails: true, wantKnown: true},
+ "unrecorded": {shas: []string{unknownSHA}},
+ } {
+ t.Run(name, func(t *testing.T) {
+ var changes []change.Change
+ for _, sha := range tt.shas {
+ changes = append(changes, change.Change{URIs: []string{uriFor(sha)}})
+ }
+ fails, known, err := tr.Outcome(context.Background(), changes)
+ require.NoError(t, err)
+ assert.Equal(t, tt.wantFails, fails)
+ assert.Equal(t, tt.wantKnown, known)
+ })
+ }
+}
+
+func TestLoad(t *testing.T) {
+ path := filepath.Join(t.TempDir(), "ds.json")
+ want := Dataset{Repo: "o/r", BaseSHA: passSHA, Attempts: []Attempt{{PR: 1, ReplaySHA: passSHA, Outcome: OutcomePass}}}
+ raw, err := json.Marshal(want)
+ require.NoError(t, err)
+ require.NoError(t, os.WriteFile(path, raw, 0o644))
+
+ got, err := Load(path)
+ require.NoError(t, err)
+ assert.Equal(t, want, got)
+ assert.Contains(t, NewIndex(got), passSHA)
+}
diff --git a/submitqueue/extension/buildrunner/replay/runner.go b/submitqueue/extension/buildrunner/replay/runner.go
new file mode 100644
index 000000000..9234e83ba
--- /dev/null
+++ b/submitqueue/extension/buildrunner/replay/runner.go
@@ -0,0 +1,132 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package replay
+
+import (
+ "context"
+ "crypto/rand"
+ "encoding/hex"
+ "fmt"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/uber/submitqueue/platform/base/change"
+ changegit "github.com/uber/submitqueue/platform/base/change/git"
+ "github.com/uber/submitqueue/submitqueue/core/changeset"
+ "github.com/uber/submitqueue/submitqueue/entity"
+ "github.com/uber/submitqueue/submitqueue/extension/buildrunner"
+)
+
+// Params configures a replay runner.
+type Params struct {
+ // Config is the per-queue identity this runner serves.
+ Config buildrunner.Config
+ // Resolver resolves batches into their changes.
+ Resolver changeset.Resolver
+ // Index holds the recorded attempts.
+ Index Index
+ // TimeScale divides recorded durations, so a replay of hours of CI
+ // finishes in minutes. Values below 1 are treated as 1.
+ TimeScale float64
+ // Now returns the current time. Nil means time.Now.
+ Now func() time.Time
+}
+
+type runner struct {
+ p Params
+}
+
+// New returns a replay build runner. A build fails if any change in its base
+// or head is recorded as failing — a build containing a broken change is
+// broken whatever it was meant to test — and takes the longest recorded
+// duration among them. Changes with no record pass instantly.
+func New(p Params) buildrunner.BuildRunner {
+ if p.TimeScale < 1 {
+ p.TimeScale = 1
+ }
+ if p.Now == nil {
+ p.Now = time.Now
+ }
+ return &runner{p: p}
+}
+
+func (r *runner) Trigger(ctx context.Context, base []entity.Batch, head entity.Batch, _ entity.BuildMetadata) (entity.BuildID, error) {
+ batches := append(append([]entity.Batch{}, base...), head)
+ fails := false
+ var longest int64
+ for _, b := range batches {
+ changes, err := r.p.Resolver.ChangesForBatch(ctx, b)
+ if err != nil {
+ return entity.BuildID{}, fmt.Errorf("replay: resolve batch %s: %w", b.ID, err)
+ }
+ for _, a := range r.recorded(changes) {
+ fails = fails || a.Outcome == OutcomeFail
+ longest = max(longest, a.DurationMs)
+ }
+ }
+ outcome := OutcomePass
+ if fails {
+ outcome = OutcomeFail
+ }
+ due := r.p.Now().UnixMilli() + int64(float64(longest)/r.p.TimeScale)
+
+ var suffix [4]byte
+ if _, err := rand.Read(suffix[:]); err != nil {
+ return entity.BuildID{}, fmt.Errorf("replay: generating build id: %w", err)
+ }
+ // The outcome and due time live in the ID, so Status needs no state and
+ // any runner instance can answer for any build.
+ return entity.BuildID{ID: fmt.Sprintf("replay-%s-%d-%s", outcome, due, hex.EncodeToString(suffix[:]))}, nil
+}
+
+func (r *runner) recorded(changes []change.Change) []Attempt {
+ var found []Attempt
+ for _, c := range changes {
+ for _, uri := range c.URIs {
+ id, err := changegit.ParseChangeID(uri)
+ if err != nil {
+ continue
+ }
+ if a, ok := r.p.Index[id.CommitSHA]; ok {
+ found = append(found, a)
+ }
+ }
+ }
+ return found
+}
+
+func (r *runner) Status(_ context.Context, buildID entity.BuildID) (entity.BuildStatus, entity.BuildMetadata, error) {
+ parts := strings.SplitN(buildID.ID, "-", 4)
+ if len(parts) != 4 || parts[0] != "replay" {
+ return entity.BuildStatusUnknown, nil, fmt.Errorf("replay: build %q was not minted by this runner", buildID.ID)
+ }
+ due, err := strconv.ParseInt(parts[2], 10, 64)
+ if err != nil {
+ return entity.BuildStatusUnknown, nil, fmt.Errorf("replay: build %q has no due time: %w", buildID.ID, err)
+ }
+ if r.p.Now().UnixMilli() < due {
+ return entity.BuildStatusRunning, nil, nil
+ }
+ if Outcome(parts[1]) == OutcomeFail {
+ return entity.BuildStatusFailed, nil, nil
+ }
+ return entity.BuildStatusSucceeded, nil, nil
+}
+
+// Cancel is a no-op: a replayed build has no CI behind it to stop.
+func (r *runner) Cancel(context.Context, entity.BuildID) error {
+ return nil
+}
diff --git a/submitqueue/extension/buildrunner/replay/truth.go b/submitqueue/extension/buildrunner/replay/truth.go
new file mode 100644
index 000000000..bb726be55
--- /dev/null
+++ b/submitqueue/extension/buildrunner/replay/truth.go
@@ -0,0 +1,43 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package replay
+
+import (
+ "context"
+
+ "github.com/uber/submitqueue/platform/base/change"
+ "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer/oracle"
+)
+
+type truth struct {
+ r *runner
+}
+
+// Truth returns an oracle.Truth backed by the recorded outcomes in idx, so the
+// oracle agrees with the replay runner on every head. A batch is known only
+// if at least one of its changes is recorded.
+func Truth(idx Index) oracle.Truth {
+ return truth{r: &runner{p: Params{Index: idx}}}
+}
+
+func (t truth) Outcome(_ context.Context, changes []change.Change) (bool, bool, error) {
+ found := t.r.recorded(changes)
+ for _, a := range found {
+ if a.Outcome == OutcomeFail {
+ return true, true, nil
+ }
+ }
+ return false, len(found) > 0, nil
+}
diff --git a/submitqueue/extension/changediff/BUILD.bazel b/submitqueue/extension/changediff/BUILD.bazel
new file mode 100644
index 000000000..f63404a12
--- /dev/null
+++ b/submitqueue/extension/changediff/BUILD.bazel
@@ -0,0 +1,12 @@
+load("@rules_go//go:def.bzl", "go_library")
+
+go_library(
+ name = "go_default_library",
+ srcs = ["change_diff.go"],
+ importpath = "github.com/uber/submitqueue/submitqueue/extension/changediff",
+ visibility = ["//visibility:public"],
+ deps = [
+ "//platform/base/change:go_default_library",
+ "//submitqueue/entity:go_default_library",
+ ],
+)
diff --git a/submitqueue/extension/changediff/README.md b/submitqueue/extension/changediff/README.md
new file mode 100644
index 000000000..01c600f98
--- /dev/null
+++ b/submitqueue/extension/changediff/README.md
@@ -0,0 +1,23 @@
+# changediff
+
+Reads the code content of a change — its unified diff — as opposed to the metadata a `changeprovider` reports (author, files, line counts).
+
+## Why it is separate from changeprovider
+
+Change metadata is small, persisted in the change store, and read by every stage. A diff can be megabytes and is wanted only by consumers that judge the code itself, such as the `jev` scorer. Keeping diffs out of `entity.ChangeDetails` keeps them out of the change store and out of every consumer that never reads them. Nothing persists a diff; a consumer that wants one reads it on demand and caches whatever it derives.
+
+## Contract
+
+`Provider.Diff` takes a `change.Change` and returns one `entity.ChangeDiff` per URI, in URI order. For a stack, each diff covers only its own change, so the diffs concatenate to the stack's whole effect. Diffs are returned in full; the caller decides how much of them it can afford to use.
+
+## Implementations
+
+- `git` — reads from a local copy of a git remote through `platform/git/repo`. It uses the same baseline as the git change provider: the first change from where it diverged from the target, each later change from where it diverged from the change before it, so a diff and its numstat agree. It refreshes the target before diffing, because a change rebased onto a newer target would otherwise be diffed against a stale merge base and appear to contain the target's new commits. Sharing a repository copy with the git change provider shares its lock.
+
+## Context beyond the diff
+
+The git implementation can also return, per change, the full post-change text of every text file it adds or modifies (`IncludeFiles`), and references found by searching the tree at the change's head (`IncludeReferences`): untouched *callers* of symbols the diff modifies or removes, and *definitions* of functions the diff calls but does not define. Symbols are found by a line-oriented heuristic over the diff — Go declarations first, Python and JavaScript as a best effort — and searches follow Go scoping: a bare call resolves only in the caller's package directory, and a use from another package must be qualified. Searches are capped per symbol and per change. See [doc/rfc/submitqueue/code-aware-scoring.md](../../../doc/rfc/submitqueue/code-aware-scoring.md).
+
+## Wiring
+
+The orchestrator builds a `git` provider for a `jev` scorer whose `jev.diff.git` block names a repository. A block naming the same `repoPath` as the queue's git change provider shares that copy; one path naming two different remotes is rejected at startup.
diff --git a/submitqueue/extension/changediff/change_diff.go b/submitqueue/extension/changediff/change_diff.go
new file mode 100644
index 000000000..bd642f643
--- /dev/null
+++ b/submitqueue/extension/changediff/change_diff.go
@@ -0,0 +1,52 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package changediff defines the contract for reading the code content of a
+// change — its diff — as opposed to the metadata a changeprovider reports.
+//
+// The two are separate because their costs differ by orders of magnitude: change
+// metadata is small, persisted, and read by every stage, while a diff can be
+// megabytes and is wanted only by consumers that judge the code itself (for
+// example a model-backed scorer). Keeping diffs out of entity.ChangeDetails keeps
+// them out of the change store and out of every consumer that never reads them.
+package changediff
+
+//go:generate mockgen -source=change_diff.go -destination=mock/change_diff_mock.go -package=mock
+
+import (
+ "context"
+
+ "github.com/uber/submitqueue/platform/base/change"
+ "github.com/uber/submitqueue/submitqueue/entity"
+)
+
+// Provider reads the diffs of a change.
+type Provider interface {
+ // Diff returns one entity.ChangeDiff per URI of c, in URI order. For a stack
+ // each diff covers only its own change, so the diffs concatenate to the
+ // stack's whole effect. Diffs are returned in full; callers bound their size.
+ Diff(ctx context.Context, c change.Change) ([]entity.ChangeDiff, error)
+}
+
+// Config carries the per-queue identity handed to a Factory.
+type Config struct {
+ // QueueName identifies the queue this Provider serves.
+ QueueName string
+}
+
+// Factory builds the Provider for a queue.
+type Factory interface {
+ // For returns the Provider for the given queue.
+ For(cfg Config) (Provider, error)
+}
diff --git a/submitqueue/extension/changediff/git/BUILD.bazel b/submitqueue/extension/changediff/git/BUILD.bazel
new file mode 100644
index 000000000..623b98e24
--- /dev/null
+++ b/submitqueue/extension/changediff/git/BUILD.bazel
@@ -0,0 +1,42 @@
+load("@rules_go//go:def.bzl", "go_library", "go_test")
+
+go_library(
+ name = "go_default_library",
+ srcs = [
+ "context.go",
+ "differ.go",
+ ],
+ importpath = "github.com/uber/submitqueue/submitqueue/extension/changediff/git",
+ visibility = ["//visibility:public"],
+ deps = [
+ "//platform/base/change:go_default_library",
+ "//platform/base/change/git:go_default_library",
+ "//platform/metrics:go_default_library",
+ "//submitqueue/entity:go_default_library",
+ "//submitqueue/extension/changediff:go_default_library",
+ "@com_github_uber_go_tally//:go_default_library",
+ ],
+)
+
+go_test(
+ name = "go_default_test",
+ srcs = ["differ_test.go"],
+ # The pinned git, so these assertions describe the build the services run
+ # rather than whatever the host happens to have.
+ data = ["@git"],
+ embed = [":go_default_library"],
+ env = {
+ "SUBMITQUEUE_TEST_GIT": "$(location @git//:git)",
+ },
+ deps = [
+ "//platform/base/change:go_default_library",
+ "//platform/git/exec:go_default_library",
+ "//platform/git/exectest:go_default_library",
+ "//platform/git/repo:go_default_library",
+ "//submitqueue/entity:go_default_library",
+ "//submitqueue/extension/changediff:go_default_library",
+ "@com_github_stretchr_testify//assert:go_default_library",
+ "@com_github_stretchr_testify//require:go_default_library",
+ "@com_github_uber_go_tally//:go_default_library",
+ ],
+)
diff --git a/submitqueue/extension/changediff/git/context.go b/submitqueue/extension/changediff/git/context.go
new file mode 100644
index 000000000..4337dca9b
--- /dev/null
+++ b/submitqueue/extension/changediff/git/context.go
@@ -0,0 +1,310 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package git
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "os/exec"
+ "path"
+ "regexp"
+ "sort"
+ "strconv"
+ "strings"
+
+ "github.com/uber/submitqueue/submitqueue/entity"
+)
+
+// Bounds on how much context one change can pull out of the repository. They
+// keep a sweeping change from turning into thousands of git invocations; the
+// consumer applies its own, tighter, byte budget on top.
+const (
+ maxFileBytes = 256 << 10
+ maxSymbolsPerKind = 20
+ maxCallersPerSymbol = 10
+ maxDefinitionsPerName = 3
+)
+
+// Symbol extraction is a line-oriented heuristic over the unified diff, not a
+// parser: it recognizes Go declarations first, with Python and JavaScript
+// forms as a best effort. A missed symbol costs context, never correctness.
+var (
+ definitionPatterns = []*regexp.Regexp{
+ regexp.MustCompile(`^\s*func\s+(?:\([^)]*\)\s*)?([A-Za-z_]\w*)\s*[\[(]`),
+ regexp.MustCompile(`^\s*type\s+([A-Za-z_]\w*)\b`),
+ regexp.MustCompile(`^\s*(?:var|const)\s+([A-Za-z_]\w*)\b`),
+ regexp.MustCompile(`^\s*(?:def|class)\s+([A-Za-z_]\w*)\b`),
+ regexp.MustCompile(`^\s*(?:export\s+)?function\s+([A-Za-z_]\w*)\s*\(`),
+ }
+ // A bare call: an identifier followed by "(", not preceded by "." (a
+ // method or package-qualified call, whose definition cannot be found by
+ // name alone) or by another identifier character.
+ callPattern = regexp.MustCompile(`(?:^|[^.\w])([A-Za-z_]\w*)\s*\(`)
+)
+
+// notSymbols are language keywords and builtins that look like calls.
+var notSymbols = map[string]bool{}
+
+func init() {
+ for _, w := range strings.Fields(`if for switch return func go defer select case range else
+ make new len cap append copy delete panic recover print println close complex real imag
+ string int int8 int16 int32 int64 uint uint8 uint16 uint32 uint64 uintptr float32 float64
+ byte rune bool error any min max clear struct interface map chan
+ def class print isinstance super self function typeof`) {
+ notSymbols[w] = true
+ }
+}
+
+// changedLines returns the removed and added code lines of a unified diff,
+// without their +/- markers and without the file headers.
+func changedLines(patch string) (removed, added []string) {
+ for _, line := range strings.Split(patch, "\n") {
+ switch {
+ case strings.HasPrefix(line, "---"), strings.HasPrefix(line, "+++"):
+ case strings.HasPrefix(line, "-"):
+ removed = append(removed, line[1:])
+ case strings.HasPrefix(line, "+"):
+ added = append(added, line[1:])
+ }
+ }
+ return removed, added
+}
+
+// declaration is a symbol and the line that declares it.
+type declaration struct {
+ name string
+ line string
+}
+
+func declaredSymbols(lines []string) []declaration {
+ seen := map[string]bool{}
+ var decls []declaration
+ for _, line := range lines {
+ for _, re := range definitionPatterns {
+ if m := re.FindStringSubmatch(line); m != nil && !seen[m[1]] {
+ seen[m[1]] = true
+ decls = append(decls, declaration{name: m[1], line: strings.TrimSpace(line)})
+ }
+ }
+ }
+ return decls
+}
+
+func calledSymbols(lines []string, defined map[string]bool) []string {
+ seen := map[string]bool{}
+ var names []string
+ for _, line := range lines {
+ if isDefinition(line) {
+ continue
+ }
+ for _, m := range callPattern.FindAllStringSubmatch(line, -1) {
+ name := m[1]
+ if notSymbols[name] || defined[name] || seen[name] {
+ continue
+ }
+ seen[name] = true
+ names = append(names, name)
+ }
+ }
+ return names
+}
+
+func isDefinition(line string) bool {
+ for _, re := range definitionPatterns {
+ if re.MatchString(line) {
+ return true
+ }
+ }
+ return false
+}
+
+// touchedFile is one entry of `git diff --name-status`.
+type touchedFile struct {
+ path string
+ added bool
+ deleted bool
+}
+
+// touchedFiles parses `git diff --name-status -M -z`: a status field, then one
+// path, or two (old, new) for a rename or copy.
+func touchedFiles(out string) []touchedFile {
+ fields := strings.Split(out, "\x00")
+ var files []touchedFile
+ for i := 0; i < len(fields); i++ {
+ status := fields[i]
+ if status == "" {
+ continue
+ }
+ if (status[0] == 'R' || status[0] == 'C') && i+2 < len(fields) {
+ files = append(files, touchedFile{path: fields[i+2]})
+ i += 2
+ continue
+ }
+ if i+1 >= len(fields) {
+ break
+ }
+ files = append(files, touchedFile{path: fields[i+1], added: status[0] == 'A', deleted: status[0] == 'D'})
+ i++
+ }
+ return files
+}
+
+// changeContext reads the files and references for one change at sha.
+func (d *differ) changeContext(ctx context.Context, base, sha, patch string) ([]entity.FileContent, []entity.CodeReference, error) {
+ if !d.includeFiles && !d.includeReferences {
+ return nil, nil, nil
+ }
+ nameStatus, err := d.repo.RunRaw(ctx, "diff", "--name-status", "-M", "-z", base, sha)
+ if err != nil {
+ return nil, nil, err
+ }
+ touched := touchedFiles(nameStatus)
+ isTouched := make(map[string]bool, len(touched))
+ for _, f := range touched {
+ isTouched[f.path] = true
+ }
+
+ var files []entity.FileContent
+ if d.includeFiles {
+ for _, f := range touched {
+ // An added file is already whole in the diff.
+ if f.added || f.deleted {
+ continue
+ }
+ content, err := d.repo.RunRaw(ctx, "show", sha+":"+f.path)
+ if err != nil {
+ return nil, nil, err
+ }
+ if len(content) > maxFileBytes || strings.ContainsRune(content, 0) {
+ continue
+ }
+ files = append(files, entity.FileContent{Path: f.path, Content: content})
+ }
+ sort.Slice(files, func(i, j int) bool { return files[i].Path < files[j].Path })
+ }
+
+ var refs []entity.CodeReference
+ if d.includeReferences {
+ refs, err = d.references(ctx, sha, patch, isTouched)
+ if err != nil {
+ return nil, nil, err
+ }
+ }
+ return files, refs, nil
+}
+
+// references follows Go's scoping to keep same-named symbols in unrelated
+// packages out: a bare call resolves only within the caller's package
+// directory, and a use from another package is qualified ("pkg.Name"). Other
+// languages get the same rule, which errs toward reporting too little.
+func (d *differ) references(ctx context.Context, sha, patch string, isTouched map[string]bool) ([]entity.CodeReference, error) {
+ removed, added := changedLines(patch)
+ touchedDirs := make(map[string]bool, len(isTouched))
+ for p := range isTouched {
+ touchedDirs[path.Dir(p)] = true
+ }
+
+ // A symbol defined on a removed line was deleted, renamed, or had its
+ // declaration rewritten; any untouched use of it may no longer compile.
+ after := map[string]string{}
+ definedHere := map[string]bool{}
+ for _, decl := range declaredSymbols(added) {
+ after[decl.name] = decl.line
+ definedHere[decl.name] = true
+ }
+
+ var refs []entity.CodeReference
+ for _, decl := range capped(declaredSymbols(removed), maxSymbolsPerKind) {
+ name := decl.name
+ hits, err := d.grep(ctx, sha, "-w", "-e", name)
+ if err != nil {
+ return nil, err
+ }
+ n := 0
+ for _, h := range hits {
+ if isTouched[h.Path] || n == maxCallersPerSymbol {
+ continue
+ }
+ if !touchedDirs[path.Dir(h.Path)] && !strings.Contains(h.Text, "."+name) {
+ continue
+ }
+ h.Kind, h.Symbol = entity.CodeReferenceKindCaller, name
+ h.DeclarationBefore, h.DeclarationAfter = decl.line, after[name]
+ refs = append(refs, h)
+ n++
+ }
+ }
+
+ for _, name := range capped(calledSymbols(added, definedHere), maxSymbolsPerKind) {
+ quoted := regexp.QuoteMeta(name)
+ hits, err := d.grep(ctx, sha, "-E",
+ "-e", `^func (\([^)]*\) )?`+quoted+`[[(]`,
+ "-e", `^(type|var|const) `+quoted+`([^A-Za-z0-9_]|$)`,
+ "-e", `^[[:space:]]*(def|class) `+quoted+`([^A-Za-z0-9_]|$)`,
+ "-e", `^(export )?function `+quoted+`[[:space:]]*\(`,
+ )
+ if err != nil {
+ return nil, err
+ }
+ n := 0
+ for _, h := range hits {
+ if !touchedDirs[path.Dir(h.Path)] || n == maxDefinitionsPerName {
+ continue
+ }
+ h.Kind, h.Symbol = entity.CodeReferenceKindDefinition, name
+ refs = append(refs, h)
+ n++
+ }
+ }
+ return refs, nil
+}
+
+// grep runs `git grep` over the tree at sha. No match is an empty result, not
+// an error: git reports it with exit status 1.
+func (d *differ) grep(ctx context.Context, sha string, args ...string) ([]entity.CodeReference, error) {
+ full := append([]string{"grep", "-n", "-I", "--null"}, args...)
+ full = append(full, sha, "--")
+ out, err := d.repo.RunRaw(ctx, full...)
+ if err != nil {
+ var exitErr *exec.ExitError
+ if errors.As(err, &exitErr) && exitErr.ExitCode() == 1 {
+ return nil, nil
+ }
+ return nil, fmt.Errorf("failed to search %s: %w", sha, err)
+ }
+ var hits []entity.CodeReference
+ for _, line := range strings.Split(out, "\n") {
+ // ":\x00\x00"
+ parts := strings.SplitN(line, "\x00", 3)
+ if len(parts) != 3 {
+ continue
+ }
+ path := strings.TrimPrefix(parts[0], sha+":")
+ n, err := strconv.Atoi(parts[1])
+ if err != nil {
+ continue
+ }
+ hits = append(hits, entity.CodeReference{Path: path, Line: n, Text: parts[2]})
+ }
+ return hits, nil
+}
+
+func capped[T any](xs []T, n int) []T {
+ if len(xs) > n {
+ return xs[:n]
+ }
+ return xs
+}
diff --git a/submitqueue/extension/changediff/git/differ.go b/submitqueue/extension/changediff/git/differ.go
new file mode 100644
index 000000000..bc9480aae
--- /dev/null
+++ b/submitqueue/extension/changediff/git/differ.go
@@ -0,0 +1,132 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package git provides a changediff.Provider that reads diffs out of a local
+// copy of a git remote.
+//
+// Each change is diffed from where it diverged from the change before it in its
+// stack, and the first from the target — the same baseline the git change
+// provider measures line counts against, so a diff and its numstat agree.
+//
+// Optionally it also reads context at the change's head: the full post-change
+// text of touched files, and references found by searching the tree for the
+// symbols the diff touches (see context.go). See
+// doc/rfc/submitqueue/code-aware-scoring.md for the tiers this implements.
+package git
+
+import (
+ "context"
+ "fmt"
+ "sync"
+
+ "github.com/uber-go/tally"
+
+ "github.com/uber/submitqueue/platform/base/change"
+ changegit "github.com/uber/submitqueue/platform/base/change/git"
+ coremetrics "github.com/uber/submitqueue/platform/metrics"
+ "github.com/uber/submitqueue/submitqueue/entity"
+ "github.com/uber/submitqueue/submitqueue/extension/changediff"
+)
+
+const opName = "git_changediff"
+
+// Repository is the git plumbing the differ needs. platform/git/repo.Repo
+// satisfies it; sharing one with the git change provider shares its lock.
+type Repository interface {
+ sync.Locker
+ FetchTarget(ctx context.Context) error
+ EnsureCommit(ctx context.Context, sha, ref string) error
+ MergeBase(ctx context.Context, a, b string) (string, error)
+ RunRaw(ctx context.Context, args ...string) (string, error)
+ Remote() string
+ Target() string
+}
+
+// Params carries what a differ needs.
+type Params struct {
+ Config changediff.Config
+ Repo Repository
+ MetricsScope tally.Scope
+ // IncludeFiles adds each touched file's post-change content.
+ IncludeFiles bool
+ // IncludeReferences adds untouched callers of symbols the change modifies
+ // or removes, and definitions of symbols it calls but does not define.
+ IncludeReferences bool
+}
+
+type differ struct {
+ repo Repository
+ metricsScope tally.Scope
+ includeFiles bool
+ includeReferences bool
+}
+
+// New returns a changediff.Provider reading from repo.
+func New(params Params) changediff.Provider {
+ scope := params.MetricsScope
+ if scope == nil {
+ scope = tally.NoopScope
+ }
+ return &differ{
+ repo: params.Repo,
+ metricsScope: scope.SubScope(opName),
+ includeFiles: params.IncludeFiles,
+ includeReferences: params.IncludeReferences,
+ }
+}
+
+func (d *differ) Diff(ctx context.Context, c change.Change) (_ []entity.ChangeDiff, retErr error) {
+ op := coremetrics.Begin(d.metricsScope, "diff", coremetrics.LongLatencyBuckets)
+ defer func() { op.Complete(retErr) }()
+
+ d.repo.Lock()
+ defer d.repo.Unlock()
+
+ // Without a fresh target, a change rebased onto a newer target would diff
+ // against a stale merge base and include the target's own new commits.
+ if err := d.repo.FetchTarget(ctx); err != nil {
+ return nil, fmt.Errorf("failed to update target branch %s: %w", d.repo.Target(), err)
+ }
+
+ diffs := make([]entity.ChangeDiff, 0, len(c.URIs))
+ previous := ""
+ for _, uri := range c.URIs {
+ id, err := changegit.ParseChangeID(uri)
+ if err != nil {
+ return nil, fmt.Errorf("failed to parse change URI: %w", err)
+ }
+ if err := d.repo.EnsureCommit(ctx, id.CommitSHA, id.Ref); err != nil {
+ return nil, err
+ }
+ against := d.repo.Remote() + "/" + d.repo.Target()
+ if previous != "" {
+ against = previous
+ }
+ base, err := d.repo.MergeBase(ctx, against, id.CommitSHA)
+ if err != nil {
+ return nil, err
+ }
+ patch, err := d.repo.RunRaw(ctx, "diff", "-M", "--no-color", "--no-ext-diff", base, id.CommitSHA)
+ if err != nil {
+ return nil, fmt.Errorf("failed to diff change %s: %w", uri, err)
+ }
+ files, refs, err := d.changeContext(ctx, base, id.CommitSHA, patch)
+ if err != nil {
+ return nil, fmt.Errorf("failed to read context for change %s: %w", uri, err)
+ }
+ diffs = append(diffs, entity.ChangeDiff{URI: uri, Patch: patch, Files: files, References: refs})
+ previous = id.CommitSHA
+ }
+ return diffs, nil
+}
diff --git a/submitqueue/extension/changediff/git/differ_test.go b/submitqueue/extension/changediff/git/differ_test.go
new file mode 100644
index 000000000..a8c16dfd6
--- /dev/null
+++ b/submitqueue/extension/changediff/git/differ_test.go
@@ -0,0 +1,277 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package git
+
+import (
+ "context"
+ "fmt"
+ "net/url"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+ "github.com/uber-go/tally"
+
+ "github.com/uber/submitqueue/platform/base/change"
+ gitexec "github.com/uber/submitqueue/platform/git/exec"
+ gitexectest "github.com/uber/submitqueue/platform/git/exectest"
+ gitrepo "github.com/uber/submitqueue/platform/git/repo"
+ "github.com/uber/submitqueue/submitqueue/entity"
+ "github.com/uber/submitqueue/submitqueue/extension/changediff"
+)
+
+type fixture struct {
+ t *testing.T
+ git string
+ remote string
+ work string
+ differ changediff.Provider
+ repo *gitrepo.Repo
+}
+
+func newFixture(t *testing.T) *fixture {
+ t.Helper()
+ root := t.TempDir()
+ f := &fixture{
+ t: t,
+ git: gitexectest.Git(t),
+ remote: filepath.Join(root, "remote.git"),
+ work: filepath.Join(root, "work"),
+ }
+ f.run("", "init", "--bare", "-b", "main", f.remote)
+ f.run("", "clone", f.remote, f.work)
+ f.run(f.work, "config", "user.name", "Test")
+ f.run(f.work, "config", "user.email", "test@example.invalid")
+ f.write("seed.txt", "seed\n")
+ f.run(f.work, "add", ".")
+ f.run(f.work, "commit", "-m", "seed")
+ f.run(f.work, "push", "origin", "main")
+
+ repo, err := gitrepo.NewRepo(gitrepo.RepoConfig{
+ Git: f.git,
+ Path: filepath.Join(root, "copy.git"),
+ RemoteURL: f.remote,
+ Target: "main",
+ })
+ require.NoError(t, err)
+ require.NoError(t, repo.Provision(context.Background()))
+ f.repo = repo
+ f.differ = New(Params{Config: changediff.Config{QueueName: "q"}, Repo: repo, MetricsScope: tally.NoopScope})
+ return f
+}
+
+func (f *fixture) run(dir string, args ...string) string {
+ f.t.Helper()
+ out, err := gitexec.Command(context.Background(), f.git, dir, args...).CombinedOutput()
+ require.NoError(f.t, err, "git %s: %s", strings.Join(args, " "), out)
+ return strings.TrimSpace(string(out))
+}
+
+func (f *fixture) write(path, contents string) {
+ f.t.Helper()
+ full := filepath.Join(f.work, path)
+ require.NoError(f.t, os.MkdirAll(filepath.Dir(full), 0o755))
+ require.NoError(f.t, os.WriteFile(full, []byte(contents), 0o644))
+}
+
+func (f *fixture) push(branch, base string, files map[string]string) string {
+ f.t.Helper()
+ f.run(f.work, "checkout", "-B", branch, base)
+ for path, contents := range files {
+ f.write(path, contents)
+ }
+ f.run(f.work, "add", "-A")
+ f.run(f.work, "commit", "-m", branch)
+ f.run(f.work, "push", "-f", "origin", branch)
+ return f.run(f.work, "rev-parse", "HEAD")
+}
+
+func (f *fixture) mainSHA() string {
+ return f.run(f.work, "rev-parse", "origin/main")
+}
+
+func (f *fixture) uri(branch, sha string) string {
+ return fmt.Sprintf("git://git.example.com/demo/%s/%s", url.PathEscape("refs/heads/"+branch), sha)
+}
+
+func (f *fixture) diff(uris ...string) ([]entity.ChangeDiff, error) {
+ return f.differ.Diff(context.Background(), change.Change{URIs: uris})
+}
+
+func TestDiff(t *testing.T) {
+ t.Run("single change carries its code", func(t *testing.T) {
+ f := newFixture(t)
+ head := f.push("feature/one", f.mainSHA(), map[string]string{"pkg/a/one.go": "package a\n\nfunc One() int { return 1 }\n"})
+
+ diffs, err := f.diff(f.uri("feature/one", head))
+ require.NoError(t, err)
+ require.Len(t, diffs, 1)
+ assert.Equal(t, f.uri("feature/one", head), diffs[0].URI)
+ assert.Contains(t, diffs[0].Patch, "+++ b/pkg/a/one.go")
+ assert.Contains(t, diffs[0].Patch, "+func One() int { return 1 }")
+ })
+
+ // Diffing each change against the target would repeat the earlier changes'
+ // code in every later one.
+ t.Run("stack is diffed per change", func(t *testing.T) {
+ f := newFixture(t)
+ first := f.push("feature/s1", f.mainSHA(), map[string]string{"a.txt": "first\n"})
+ second := f.push("feature/s2", first, map[string]string{"b.txt": "second\n"})
+
+ diffs, err := f.diff(f.uri("feature/s1", first), f.uri("feature/s2", second))
+ require.NoError(t, err)
+ require.Len(t, diffs, 2)
+ assert.Contains(t, diffs[0].Patch, "+first")
+ assert.Contains(t, diffs[1].Patch, "+second")
+ assert.NotContains(t, diffs[1].Patch, "+first")
+ })
+
+ // A change cut from a target commit the copy has not fetched yet must not
+ // be diffed against the stale target, or it would carry that commit too.
+ t.Run("target advanced since last fetch", func(t *testing.T) {
+ f := newFixture(t)
+ _, err := f.diff(f.uri("feature/warm", f.push("feature/warm", f.mainSHA(), map[string]string{"w.txt": "w\n"})))
+ require.NoError(t, err)
+
+ f.run(f.work, "checkout", "main")
+ f.write("upstream.txt", "upstream\n")
+ f.run(f.work, "add", "-A")
+ f.run(f.work, "commit", "-m", "upstream")
+ f.run(f.work, "push", "origin", "main")
+ head := f.push("feature/rebased", f.mainSHA(), map[string]string{"mine.txt": "mine\n"})
+
+ diffs, err := f.diff(f.uri("feature/rebased", head))
+ require.NoError(t, err)
+ assert.Contains(t, diffs[0].Patch, "+mine")
+ assert.NotContains(t, diffs[0].Patch, "upstream")
+ })
+
+ t.Run("malformed URI is an error", func(t *testing.T) {
+ _, err := newFixture(t).diff("not-a-change-uri")
+ require.Error(t, err)
+ })
+
+ t.Run("unknown commit is an error", func(t *testing.T) {
+ f := newFixture(t)
+ _, err := f.diff(f.uri("feature/nope", "0123456789abcdef0123456789abcdef01234567"))
+ require.Error(t, err)
+ })
+}
+
+func (f *fixture) contextDiff(uri string) entity.ChangeDiff {
+ f.t.Helper()
+ d := New(Params{Repo: f.repo, IncludeFiles: true, IncludeReferences: true})
+ diffs, err := d.Diff(context.Background(), change.Change{URIs: []string{uri}})
+ require.NoError(f.t, err)
+ require.Len(f.t, diffs, 1)
+ return diffs[0]
+}
+
+func refsOf(diff entity.ChangeDiff, kind entity.CodeReferenceKind) []string {
+ var out []string
+ for _, r := range diff.References {
+ if r.Kind == kind {
+ out = append(out, fmt.Sprintf("%s %s:%d", r.Symbol, r.Path, r.Line))
+ }
+ }
+ return out
+}
+
+func TestDiffContext(t *testing.T) {
+ seed := map[string]string{
+ "pkg/scale.go": "package pkg\n\nfunc Scale(v int) int {\n\treturn v * 10\n}\n",
+ "pkg/report.go": "package pkg\n\nfunc Report(v int) int {\n\treturn Scale(v)\n}\n",
+ "pkg/parse.go": "package pkg\n\nfunc ParsePort(s string) (int, error) {\n\treturn 0, nil\n}\n",
+ "pkg/old.go": "package pkg\n",
+ // Same names in an unrelated package: never a reference to pkg's.
+ "other/other.go": "package other\n\nfunc Scale(v int) int {\n\treturn Scale(v)\n}\n\nfunc ParsePort(s string) (int, error) {\n\treturn 0, nil\n}\n",
+ // A qualified use from another package is a real caller.
+ "cmd/main.go": "package main\n\nfunc main() {\n\t_ = pkg.Scale(1)\n}\n",
+ }
+ newSeeded := func(t *testing.T) (*fixture, string) {
+ f := newFixture(t)
+ return f, f.push("main", f.mainSHA(), seed)
+ }
+
+ t.Run("modified files are returned in full, added and deleted ones omitted", func(t *testing.T) {
+ f, base := newSeeded(t)
+ f.run(f.work, "checkout", "-B", "feature/files", base)
+ f.write("pkg/scale.go", "package pkg\n\nfunc Scale(v int) int {\n\treturn v * 100\n}\n")
+ f.run(f.work, "rm", "-q", "pkg/old.go")
+ f.write("pkg/new.go", "package pkg\n\nconst Added = 1\n")
+ f.run(f.work, "add", "pkg/new.go")
+ f.run(f.work, "commit", "-qam", "files")
+ f.run(f.work, "push", "-f", "origin", "feature/files")
+ head := f.run(f.work, "rev-parse", "HEAD")
+
+ diff := f.contextDiff(f.uri("feature/files", head))
+ require.Len(t, diff.Files, 1)
+ assert.Equal(t, "pkg/scale.go", diff.Files[0].Path)
+ assert.Contains(t, diff.Files[0].Content, "return v * 100")
+ })
+
+ // The caller is outside the diff, so only a search of the tree can show
+ // that it still calls the old signature.
+ t.Run("a changed signature reports its untouched callers", func(t *testing.T) {
+ f, base := newSeeded(t)
+ head := f.push("feature/sig", base, map[string]string{
+ "pkg/scale.go": "package pkg\n\nfunc Scale(v, factor int) int {\n\treturn v * factor\n}\n",
+ })
+
+ diff := f.contextDiff(f.uri("feature/sig", head))
+ assert.Equal(t, []string{"Scale cmd/main.go:4", "Scale pkg/report.go:4"}, refsOf(diff, entity.CodeReferenceKindCaller),
+ "the unqualified Scale in package other is a different function")
+ for _, r := range diff.References {
+ assert.Equal(t, "func Scale(v int) int {", r.DeclarationBefore)
+ assert.Equal(t, "func Scale(v, factor int) int {", r.DeclarationAfter)
+ }
+ })
+
+ t.Run("a rename that updates its callers reports none", func(t *testing.T) {
+ f, base := newSeeded(t)
+ head := f.push("feature/rename", base, map[string]string{
+ "pkg/scale.go": "package pkg\n\nfunc Rescale(v int) int {\n\treturn v * 10\n}\n",
+ "pkg/report.go": "package pkg\n\nfunc Report(v int) int {\n\treturn Rescale(v)\n}\n",
+ "cmd/main.go": "package main\n\nfunc main() {\n\t_ = pkg.Rescale(1)\n}\n",
+ })
+
+ diff := f.contextDiff(f.uri("feature/rename", head))
+ assert.Empty(t, refsOf(diff, entity.CodeReferenceKindCaller))
+ })
+
+ t.Run("a call to a function defined elsewhere reports its definition", func(t *testing.T) {
+ f, base := newSeeded(t)
+ head := f.push("feature/call", base, map[string]string{
+ "pkg/addr.go": "package pkg\n\nfunc Addr(p string) error {\n\t_, err := ParsePort(p)\n\treturn err\n}\n",
+ })
+
+ diff := f.contextDiff(f.uri("feature/call", head))
+ assert.Equal(t, []string{"ParsePort pkg/parse.go:3"}, refsOf(diff, entity.CodeReferenceKindDefinition),
+ "a bare call cannot reach package other's ParsePort")
+ })
+
+ t.Run("a call to a function defined nowhere reports nothing", func(t *testing.T) {
+ f, base := newSeeded(t)
+ head := f.push("feature/missing", base, map[string]string{
+ "pkg/addr.go": "package pkg\n\nfunc Addr(p string) error {\n\treturn Missing(p)\n}\n",
+ })
+
+ diff := f.contextDiff(f.uri("feature/missing", head))
+ assert.Empty(t, refsOf(diff, entity.CodeReferenceKindDefinition))
+ })
+}
diff --git a/submitqueue/extension/changediff/mock/BUILD.bazel b/submitqueue/extension/changediff/mock/BUILD.bazel
new file mode 100644
index 000000000..d430c7979
--- /dev/null
+++ b/submitqueue/extension/changediff/mock/BUILD.bazel
@@ -0,0 +1,14 @@
+load("@rules_go//go:def.bzl", "go_library")
+
+go_library(
+ name = "go_default_library",
+ srcs = ["change_diff_mock.go"],
+ importpath = "github.com/uber/submitqueue/submitqueue/extension/changediff/mock",
+ visibility = ["//visibility:public"],
+ deps = [
+ "//platform/base/change:go_default_library",
+ "//submitqueue/entity:go_default_library",
+ "//submitqueue/extension/changediff:go_default_library",
+ "@org_uber_go_mock//gomock:go_default_library",
+ ],
+)
diff --git a/submitqueue/extension/changediff/mock/change_diff_mock.go b/submitqueue/extension/changediff/mock/change_diff_mock.go
new file mode 100644
index 000000000..fbef59334
--- /dev/null
+++ b/submitqueue/extension/changediff/mock/change_diff_mock.go
@@ -0,0 +1,98 @@
+// Code generated by MockGen. DO NOT EDIT.
+// Source: change_diff.go
+//
+// Generated by this command:
+//
+// mockgen -source=change_diff.go -destination=mock/change_diff_mock.go -package=mock
+//
+
+// Package mock is a generated GoMock package.
+package mock
+
+import (
+ context "context"
+ reflect "reflect"
+
+ change "github.com/uber/submitqueue/platform/base/change"
+ entity "github.com/uber/submitqueue/submitqueue/entity"
+ changediff "github.com/uber/submitqueue/submitqueue/extension/changediff"
+ gomock "go.uber.org/mock/gomock"
+)
+
+// MockProvider is a mock of Provider interface.
+type MockProvider struct {
+ ctrl *gomock.Controller
+ recorder *MockProviderMockRecorder
+ isgomock struct{}
+}
+
+// MockProviderMockRecorder is the mock recorder for MockProvider.
+type MockProviderMockRecorder struct {
+ mock *MockProvider
+}
+
+// NewMockProvider creates a new mock instance.
+func NewMockProvider(ctrl *gomock.Controller) *MockProvider {
+ mock := &MockProvider{ctrl: ctrl}
+ mock.recorder = &MockProviderMockRecorder{mock}
+ return mock
+}
+
+// EXPECT returns an object that allows the caller to indicate expected use.
+func (m *MockProvider) EXPECT() *MockProviderMockRecorder {
+ return m.recorder
+}
+
+// Diff mocks base method.
+func (m *MockProvider) Diff(ctx context.Context, c change.Change) ([]entity.ChangeDiff, error) {
+ m.ctrl.T.Helper()
+ ret := m.ctrl.Call(m, "Diff", ctx, c)
+ ret0, _ := ret[0].([]entity.ChangeDiff)
+ ret1, _ := ret[1].(error)
+ return ret0, ret1
+}
+
+// Diff indicates an expected call of Diff.
+func (mr *MockProviderMockRecorder) Diff(ctx, c any) *gomock.Call {
+ mr.mock.ctrl.T.Helper()
+ return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Diff", reflect.TypeOf((*MockProvider)(nil).Diff), ctx, c)
+}
+
+// MockFactory is a mock of Factory interface.
+type MockFactory struct {
+ ctrl *gomock.Controller
+ recorder *MockFactoryMockRecorder
+ isgomock struct{}
+}
+
+// MockFactoryMockRecorder is the mock recorder for MockFactory.
+type MockFactoryMockRecorder struct {
+ mock *MockFactory
+}
+
+// NewMockFactory creates a new mock instance.
+func NewMockFactory(ctrl *gomock.Controller) *MockFactory {
+ mock := &MockFactory{ctrl: ctrl}
+ mock.recorder = &MockFactoryMockRecorder{mock}
+ return mock
+}
+
+// EXPECT returns an object that allows the caller to indicate expected use.
+func (m *MockFactory) EXPECT() *MockFactoryMockRecorder {
+ return m.recorder
+}
+
+// For mocks base method.
+func (m *MockFactory) For(cfg changediff.Config) (changediff.Provider, error) {
+ m.ctrl.T.Helper()
+ ret := m.ctrl.Call(m, "For", cfg)
+ ret0, _ := ret[0].(changediff.Provider)
+ ret1, _ := ret[1].(error)
+ return ret0, ret1
+}
+
+// For indicates an expected call of For.
+func (mr *MockFactoryMockRecorder) For(cfg any) *gomock.Call {
+ mr.mock.ctrl.T.Helper()
+ return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "For", reflect.TypeOf((*MockFactory)(nil).For), cfg)
+}
diff --git a/submitqueue/extension/speculation/scorer/README.md b/submitqueue/extension/speculation/scorer/README.md
index ccdddb654..11e88e191 100644
--- a/submitqueue/extension/speculation/scorer/README.md
+++ b/submitqueue/extension/speculation/scorer/README.md
@@ -16,6 +16,10 @@ See [doc/rfc/submitqueue/outcome-scorer.md](../../../../doc/rfc/submitqueue/outc
**`heuristic`** scores a batch by extracting one number from its changes and matching that against ordered buckets, each mapping a `[Min, Max]` range to a probability. The extraction is a caller-supplied `ValueFunc` over the resolved `entity.BatchChanges`, so the same bucketing works for files touched, lines changed, or any other metric. It ignores `paths`.
+**`oracle`** knows each batch's recorded outcome and prices it 0.99 or 0.01. It is a benchmarking ceiling, never a production scorer. See [oracle/README.md](oracle/README.md).
+
+**`jev`** asks a System One decision model behind a LiteLLM proxy for the probability that the batch builds and lands, caching successful prices by batch and falling back to another content scorer on failure. It ignores `paths` except to forward them to its fallback. See [jev/README.md](jev/README.md).
+
**`composite`** runs several named scorers and reduces their scores to one. The reduce function receives the scores keyed by scorer name, so it can weigh sources differently rather than treating them as interchangeable; `Min`, `Max`, and `Avg` are provided. It ignores `paths` except to forward them to children.
## Adding a backend
diff --git a/submitqueue/extension/speculation/scorer/jev/BUILD.bazel b/submitqueue/extension/speculation/scorer/jev/BUILD.bazel
new file mode 100644
index 000000000..34b3a6073
--- /dev/null
+++ b/submitqueue/extension/speculation/scorer/jev/BUILD.bazel
@@ -0,0 +1,39 @@
+load("@rules_go//go:def.bzl", "go_library", "go_test")
+
+go_library(
+ name = "go_default_library",
+ srcs = [
+ "cache.go",
+ "client.go",
+ "jev.go",
+ ],
+ importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer/jev",
+ visibility = ["//visibility:public"],
+ deps = [
+ "//platform/http:go_default_library",
+ "//platform/metrics:go_default_library",
+ "//submitqueue/core/changeset:go_default_library",
+ "//submitqueue/entity:go_default_library",
+ "//submitqueue/extension/changediff:go_default_library",
+ "//submitqueue/extension/speculation/scorer:go_default_library",
+ "@com_github_uber_go_tally//:go_default_library",
+ "@org_uber_go_zap//:go_default_library",
+ ],
+)
+
+go_test(
+ name = "go_default_test",
+ srcs = ["jev_test.go"],
+ embed = [":go_default_library"],
+ deps = [
+ "//platform/base/change:go_default_library",
+ "//platform/http:go_default_library",
+ "//submitqueue/core/changeset/fake:go_default_library",
+ "//submitqueue/entity:go_default_library",
+ "//submitqueue/extension/changediff/mock:go_default_library",
+ "//submitqueue/extension/speculation/scorer:go_default_library",
+ "@com_github_stretchr_testify//assert:go_default_library",
+ "@com_github_stretchr_testify//require:go_default_library",
+ "@org_uber_go_mock//gomock:go_default_library",
+ ],
+)
diff --git a/submitqueue/extension/speculation/scorer/jev/README.md b/submitqueue/extension/speculation/scorer/jev/README.md
new file mode 100644
index 000000000..05a7e9b27
--- /dev/null
+++ b/submitqueue/extension/speculation/scorer/jev/README.md
@@ -0,0 +1,41 @@
+# jev
+
+`jev` is a content `Scorer` that asks a System One decision model — TypeSafe Jev, reached through a LiteLLM proxy — for the probability that a batch builds and lands. It is meant to sit under `evidence` as the base price, replacing or averaging with `heuristic`.
+
+## Request
+
+Each score is one `POST` of a System One body to the configured path — LiteLLM's model-routed `/v1/systemone` by default, or the TypeSafe pass-through `/typesafe/v1/systemone` on proxies that expose only that (see the [LiteLLM decisions docs](https://docs.litellm.ai/docs/decisions)) — with a single noul question, `batch_lands`. The state is a JSON description of the batch built from `changeset.Resolver.DetailedForBatch`: the queue, change count, total files and lines changed, and per change its URI, author and changed files with line counts. Changes are sorted by URI so the same batch always produces the same request, and at most 200 files are described per batch; beyond that the state carries totals only and sets `files_truncated`. With a `changediff.Provider` configured, each change also carries its unified diff as `diff`, up to 64KB per batch; a change past the cap is cut at a line boundary and marked `diff_truncated`. Without one the model sees only file names and line counts, which carries little more signal than the `heuristic` scorer. A diff that cannot be read is a failure, so the batch falls back rather than being scored without its code.
+
+With `jev.diff.files` and `jev.diff.references`, each change also carries `file_contents` (touched files in full) and `references` (`caller` and `definition` lines from elsewhere in the tree), from a separate 64KB budget spent only after every diff.
+
+Sending diffs sends the code to the proxy and its model provider. It is opt-in for that reason.
+
+The answer's `noul` is the price. A refusal, a missing answer, a value outside `[0, 1]`, a non-2xx status or a transport error is a failure.
+
+## Caching and failure
+
+A batch's requests are fixed at creation, so a price never goes stale. Successful prices are cached by queue and batch ID in a `Cache` that the wiring shares across every scorer built from one configuration; the speculator builds a fresh scorer per run, so a per-scorer cache would never be hit. Failures are not cached, so the next speculation run asks again.
+
+On failure the scorer returns the configured fallback scorer's price and logs a warning. With no fallback it returns the error, and the `bestfirst` generator substitutes its own default. A missing endpoint or key is a failure on every call rather than a startup error, so a stack without credentials runs on the fallback.
+
+Each call is bounded by the configured timeout (2s by default). The speculator scores every unresolved dependency on every run, so a slow model slows speculation; the cache makes that a one-time cost per batch.
+
+## Configuration
+
+In the orchestrator's `profiles.yaml`, under a scorer `base` or a composite component:
+
+| Field | Default | Meaning |
+|---|---|---|
+| `jev.baseUrl` | — | Proxy root. When empty, read from `baseUrlEnv`. |
+| `jev.baseUrlEnv` | `LITE_URL` | Environment variable holding the proxy root. |
+| `jev.tokenEnv` | `LITE_KEY` | Environment variable holding the LiteLLM virtual key, sent as a bearer token. |
+| `jev.path` | `/v1/systemone` | System One route on the proxy. Use `/typesafe/v1/systemone` for the TypeSafe pass-through. |
+| `jev.model` | `jev` | The proxy's model name for the decision model. The pass-through may ignore it. |
+| `jev.timeout` | `2s` | Bound on one call. |
+| `jev.cacheSize` | `4096` | Batch prices kept in memory. |
+| `jev.diff.git` | — | Repository to read diffs from, in the same shape as the git change provider's `git` block. Absent, no code is sent. Name the change provider's `repoPath` to share its copy. |
+| `jev.diff.files` | `false` | Also send each touched file's full post-change text. |
+| `jev.diff.references` | `false` | Also send untouched callers of changed symbols and definitions of called ones. |
+| `fallback` | flat `heuristic` | Content scorer used when the model fails. Cannot itself be `jev`. |
+
+The git demo enables it on `demo-queue`; see `service/submitqueue/demo/provider/git/profiles.yaml`.
diff --git a/submitqueue/extension/speculation/scorer/jev/cache.go b/submitqueue/extension/speculation/scorer/jev/cache.go
new file mode 100644
index 000000000..49ef47e4c
--- /dev/null
+++ b/submitqueue/extension/speculation/scorer/jev/cache.go
@@ -0,0 +1,69 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package jev
+
+import "sync"
+
+// Cache holds model prices by queue and batch ID. It is safe for concurrent
+// use and is meant to be shared by every scorer built from one configuration:
+// the wiring builds a scorer per speculation run, so a per-scorer cache would
+// never be hit.
+//
+// When full, the oldest entry is evicted. A batch's requests are fixed at
+// creation, so a price never goes stale; eviction exists only to bound memory.
+type Cache struct {
+ mu sync.Mutex
+ max int
+ prices map[string]float64
+ order []string
+ nextOut int
+}
+
+// NewCache returns a Cache holding at most maxEntries prices. A non-positive
+// maxEntries is treated as 1.
+func NewCache(maxEntries int) *Cache {
+ if maxEntries < 1 {
+ maxEntries = 1
+ }
+ return &Cache{
+ max: maxEntries,
+ prices: make(map[string]float64, maxEntries),
+ order: make([]string, 0, maxEntries),
+ }
+}
+
+func (c *Cache) get(key string) (float64, bool) {
+ c.mu.Lock()
+ defer c.mu.Unlock()
+ p, ok := c.prices[key]
+ return p, ok
+}
+
+func (c *Cache) put(key string, price float64) {
+ c.mu.Lock()
+ defer c.mu.Unlock()
+ if _, ok := c.prices[key]; ok {
+ c.prices[key] = price
+ return
+ }
+ if len(c.order) < c.max {
+ c.order = append(c.order, key)
+ } else {
+ delete(c.prices, c.order[c.nextOut])
+ c.order[c.nextOut] = key
+ c.nextOut = (c.nextOut + 1) % c.max
+ }
+ c.prices[key] = price
+}
diff --git a/submitqueue/extension/speculation/scorer/jev/client.go b/submitqueue/extension/speculation/scorer/jev/client.go
new file mode 100644
index 000000000..ac4b06b19
--- /dev/null
+++ b/submitqueue/extension/speculation/scorer/jev/client.go
@@ -0,0 +1,113 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package jev
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ nethttp "net/http"
+
+ phttp "github.com/uber/submitqueue/platform/http"
+)
+
+// DefaultPath is LiteLLM's model-routed System One route. Proxies without it
+// may expose the TypeSafe pass-through, "/typesafe/v1/systemone", instead;
+// both take the same body and return the same answers.
+const DefaultPath = "/v1/systemone"
+
+// Question types and answer types of the System One format.
+const (
+ questionTypeNoul = "noul"
+ answerTypeNoul = "noul"
+)
+
+// systemOneRequest is a POST /v1/systemone body.
+type systemOneRequest struct {
+ // Model is the proxy's model name, not the provider's.
+ Model string `json:"model"`
+ // State is the thing being judged. Any JSON value is accepted.
+ State any `json:"state"`
+ // Questions are keyed by name; each key comes back as a key in answers.
+ Questions map[string]systemOneQuestion `json:"questions"`
+}
+
+// systemOneQuestion is one named question in a System One request.
+type systemOneQuestion struct {
+ // Type is the question type, e.g. "noul".
+ Type string `json:"type"`
+ // Instructions is the question put to the model.
+ Instructions string `json:"instructions,omitempty"`
+ // Criteria describes the answers. For a noul it maps "true" and "false" to
+ // what each means.
+ Criteria map[string]string `json:"criteria,omitempty"`
+}
+
+// systemOneResponse is the body of a successful POST /v1/systemone.
+type systemOneResponse struct {
+ // Model is the concrete model version that answered.
+ Model string `json:"model"`
+ // Answers holds one answer per question name.
+ Answers map[string]systemOneAnswer `json:"answers"`
+}
+
+// systemOneAnswer is one answer in a System One response. A question the model
+// declines comes back with Type "refusal" and no value.
+type systemOneAnswer struct {
+ // Type is the answer type, e.g. "noul" or "refusal".
+ Type string `json:"type"`
+ // Noul is the probability that a noul question's answer is true.
+ Noul *float64 `json:"noul,omitempty"`
+}
+
+// askNoul sends one noul question about state and returns the probability the
+// answer is true. client must be rooted at the proxy base URL and authenticated.
+func askNoul(ctx context.Context, client *nethttp.Client, path, model, name string, question systemOneQuestion, state any) (float64, string, error) {
+ body, err := json.Marshal(systemOneRequest{
+ Model: model,
+ State: state,
+ Questions: map[string]systemOneQuestion{name: question},
+ })
+ if err != nil {
+ return 0, "", fmt.Errorf("marshal system one request: %w", err)
+ }
+
+ status, respBody, err := phttp.SendRequest(ctx, client, nethttp.MethodPost, path, body, func(r *nethttp.Request) {
+ r.Header.Set("Content-Type", "application/json")
+ })
+ if err != nil {
+ return 0, "", fmt.Errorf("post %s: %w", path, err)
+ }
+ if status < 200 || status >= 300 {
+ return 0, "", fmt.Errorf("post %s: %w", path, phttp.NewStatusError(status, respBody))
+ }
+
+ var resp systemOneResponse
+ if err := json.Unmarshal(respBody, &resp); err != nil {
+ return 0, "", fmt.Errorf("decode system one response: %w", err)
+ }
+ answer, ok := resp.Answers[name]
+ if !ok {
+ return 0, resp.Model, fmt.Errorf("system one response has no answer for %q", name)
+ }
+ if answer.Type != answerTypeNoul || answer.Noul == nil {
+ return 0, resp.Model, fmt.Errorf("system one answer for %q is %q, not a noul", name, answer.Type)
+ }
+ p := *answer.Noul
+ if !(p >= 0 && p <= 1) {
+ return 0, resp.Model, fmt.Errorf("system one noul for %q is %v, not a probability", name, p)
+ }
+ return p, resp.Model, nil
+}
diff --git a/submitqueue/extension/speculation/scorer/jev/jev.go b/submitqueue/extension/speculation/scorer/jev/jev.go
new file mode 100644
index 000000000..6822eecd3
--- /dev/null
+++ b/submitqueue/extension/speculation/scorer/jev/jev.go
@@ -0,0 +1,370 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package jev prices a batch by asking a System One decision model (TypeSafe
+// Jev, reached through a LiteLLM proxy) for the probability that the batch
+// builds and lands. See README.md.
+package jev
+
+import (
+ "context"
+ "fmt"
+ nethttp "net/http"
+ "sort"
+ "strings"
+ "time"
+
+ "github.com/uber-go/tally"
+ "go.uber.org/zap"
+
+ "github.com/uber/submitqueue/platform/metrics"
+ "github.com/uber/submitqueue/submitqueue/core/changeset"
+ "github.com/uber/submitqueue/submitqueue/entity"
+ "github.com/uber/submitqueue/submitqueue/extension/changediff"
+ "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer"
+)
+
+// questionName keys the single noul question in every request.
+const questionName = "batch_lands"
+
+// maxFilesInState caps the files described to the model per batch. Beyond it
+// the state carries only totals, so one sweeping refactor cannot blow the
+// request size or the per-token cost.
+const maxFilesInState = 200
+
+// DefaultMaxDiffBytes and DefaultMaxContextBytes cap the diff text, and the
+// file contents and references, sent per batch. Context is spent only after
+// every diff, so it never displaces a change's own code. A diff past its cap is
+// cut at a line boundary and marked truncated. Size them to the model's context
+// window: the request also carries JSON escaping, file stats and the question.
+const (
+ DefaultMaxDiffBytes = 64 << 10
+ DefaultMaxContextBytes = 64 << 10
+)
+
+var landsQuestion = systemOneQuestion{
+ Type: questionTypeNoul,
+ Instructions: "Will this batch of code changes pass its CI build and merge cleanly into the target branch? Judge the code in each change's diff. A change may also include modified_files_after_change, the full text of files it modifies, and repository_context, unmodified code elsewhere in the repository that relates to the change.",
+ Criteria: map[string]string{
+ "true": "The batch builds, its tests pass, and it merges without conflict.",
+ "false": "The batch fails to build, breaks tests, or cannot be merged.",
+ },
+}
+
+// Params configures a jev scorer.
+type Params struct {
+ // Config is the per-queue identity this scorer serves.
+ Config scorer.Config
+ // HTTPClient must be rooted at the LiteLLM base URL and authenticate every
+ // request. Nil means the model is unreachable and every score falls back.
+ HTTPClient *nethttp.Client
+ // Path is the System One route on the proxy. Empty means DefaultPath.
+ Path string
+ // Model is the LiteLLM model name to ask.
+ Model string
+ // Timeout bounds one model call. Zero means only ctx bounds it.
+ Timeout time.Duration
+ // Resolver resolves a batch into its changes. Nil makes every score fall
+ // back.
+ Resolver changeset.Resolver
+ // Differ supplies each change's code. Nil sends file names and line counts
+ // only; a Differ error makes the score fall back.
+ Differ changediff.Provider
+ // MaxDiffBytes and MaxContextBytes cap the code sent per batch. Zero means
+ // DefaultMaxDiffBytes and DefaultMaxContextBytes.
+ MaxDiffBytes, MaxContextBytes int
+ // Cache is shared across scorers built from one configuration. Nil disables
+ // caching.
+ Cache *Cache
+ // Fallback prices a batch the model could not. Nil returns the error
+ // instead, which the generator replaces with its own default.
+ Fallback scorer.Scorer
+ // Logger receives one warning per fallback.
+ Logger *zap.SugaredLogger
+ // MetricsScope receives score latency and cache/fallback counters.
+ MetricsScope tally.Scope
+}
+
+type jevScorer struct {
+ p Params
+}
+
+// New returns a scorer.Scorer backed by a System One model. Model prices are
+// cached by queue and batch ID; failures are not cached, so the next run asks
+// again.
+func New(p Params) scorer.Scorer {
+ if p.Logger == nil {
+ p.Logger = zap.NewNop().Sugar()
+ }
+ if p.MetricsScope == nil {
+ p.MetricsScope = tally.NoopScope
+ }
+ if p.Path == "" {
+ p.Path = DefaultPath
+ }
+ if p.MaxDiffBytes <= 0 {
+ p.MaxDiffBytes = DefaultMaxDiffBytes
+ }
+ if p.MaxContextBytes <= 0 {
+ p.MaxContextBytes = DefaultMaxContextBytes
+ }
+ return &jevScorer{p: p}
+}
+
+func (s *jevScorer) Score(ctx context.Context, batch entity.Batch, paths entity.SpeculationPathSet) (ret float64, retErr error) {
+ op := metrics.Begin(s.p.MetricsScope, "score", metrics.FastLatencyBuckets)
+ defer func() { op.Complete(retErr) }()
+
+ key := batch.Queue + "/" + batch.ID
+ if s.p.Cache != nil {
+ if price, ok := s.p.Cache.get(key); ok {
+ metrics.NamedCounter(s.p.MetricsScope, "score", "cache_hit", 1)
+ return price, nil
+ }
+ }
+
+ price, err := s.askModel(ctx, batch)
+ if err == nil {
+ if s.p.Cache != nil {
+ s.p.Cache.put(key, price)
+ }
+ return price, nil
+ }
+ if s.p.Fallback == nil {
+ return 0, err
+ }
+ metrics.NamedCounter(s.p.MetricsScope, "score", "fallback", 1)
+ s.p.Logger.Warnw("jev scorer falling back", "queue", batch.Queue, "batch", batch.ID, "error", err)
+ return s.p.Fallback.Score(ctx, batch, paths)
+}
+
+func (s *jevScorer) askModel(ctx context.Context, batch entity.Batch) (float64, error) {
+ if s.p.HTTPClient == nil {
+ return 0, fmt.Errorf("jev scorer has no endpoint configured")
+ }
+ if s.p.Resolver == nil {
+ return 0, fmt.Errorf("jev scorer has no change resolver")
+ }
+ changes, err := s.p.Resolver.DetailedForBatch(ctx, batch)
+ if err != nil {
+ return 0, fmt.Errorf("resolve changes for batch %s: %w", batch.ID, err)
+ }
+ content, err := s.contentForBatch(ctx, batch)
+ if err != nil {
+ return 0, err
+ }
+ if s.p.Timeout > 0 {
+ var cancel context.CancelFunc
+ ctx, cancel = context.WithTimeout(ctx, s.p.Timeout)
+ defer cancel()
+ }
+ price, model, err := askNoul(ctx, s.p.HTTPClient, s.p.Path, s.p.Model, questionName, landsQuestion, batchState(changes, content, s.p.MaxDiffBytes, s.p.MaxContextBytes))
+ if err != nil {
+ return 0, err
+ }
+ s.p.Logger.Infow("jev scored batch", "queue", batch.Queue, "batch", batch.ID, "model", model, "probability", price)
+ return price, nil
+}
+
+// contentForBatch returns each change's code keyed by URI, or nil without a
+// Differ.
+func (s *jevScorer) contentForBatch(ctx context.Context, batch entity.Batch) (map[string]entity.ChangeDiff, error) {
+ if s.p.Differ == nil {
+ return nil, nil
+ }
+ raw, err := s.p.Resolver.ChangesForBatch(ctx, batch)
+ if err != nil {
+ return nil, fmt.Errorf("resolve change URIs for batch %s: %w", batch.ID, err)
+ }
+ content := make(map[string]entity.ChangeDiff)
+ for _, c := range raw {
+ diffs, err := s.p.Differ.Diff(ctx, c)
+ if err != nil {
+ return nil, fmt.Errorf("diff changes for batch %s: %w", batch.ID, err)
+ }
+ for _, d := range diffs {
+ content[d.URI] = d
+ }
+ }
+ return content, nil
+}
+
+// The types below are the JSON the model reads. Field names are chosen to be
+// self-describing to it, and the change is kept apart from the repository
+// context around it so neither can be mistaken for the other.
+type changeState struct {
+ URI string `json:"uri"`
+ Author string `json:"author,omitempty"`
+ FileStats []fileState `json:"file_stats,omitempty"`
+ Diff string `json:"diff,omitempty"`
+ DiffTruncated bool `json:"diff_truncated,omitempty"`
+ // ModifiedFiles and RepositoryContext are absent without a Differ asked
+ // for them.
+ ModifiedFiles []fileContentState `json:"modified_files_after_change,omitempty"`
+ RepositoryContext *repositoryContextState `json:"repository_context,omitempty"`
+ ContextTruncated bool `json:"context_truncated,omitempty"`
+}
+
+type fileState struct {
+ Path string `json:"path"`
+ Added int `json:"lines_added"`
+ Deleted int `json:"lines_deleted"`
+ Modified int `json:"lines_modified,omitempty"`
+}
+
+type fileContentState struct {
+ Path string `json:"path"`
+ Content string `json:"content"`
+ Truncated bool `json:"truncated,omitempty"`
+}
+
+// repositoryContextDescription labels the block for the model, which otherwise
+// sees it nested under a change and could read it as part of that change.
+const repositoryContextDescription = "Unmodified code elsewhere in the repository, read at this change's head. This change does not modify any of it."
+
+type repositoryContextState struct {
+ Description string `json:"description"`
+ Callers []callerState `json:"callers_of_changed_declarations,omitempty"`
+ Definitions []definitionState `json:"definitions_of_called_functions,omitempty"`
+}
+
+type callerState struct {
+ Symbol string `json:"symbol"`
+ DeclarationBefore string `json:"declaration_before_change"`
+ DeclarationAfter string `json:"declaration_after_change"`
+ CallSite codeLineState `json:"call_site"`
+}
+
+type definitionState struct {
+ Symbol string `json:"symbol"`
+ Definition codeLineState `json:"definition"`
+}
+
+type codeLineState struct {
+ Path string `json:"path"`
+ Line int `json:"line"`
+ Text string `json:"text"`
+}
+
+// removedDeclaration stands in for an empty after-declaration, which would
+// otherwise read as missing data rather than as a deletion.
+const removedDeclaration = "(removed by this change)"
+
+type batchStateDoc struct {
+ Queue string `json:"queue"`
+ ChangeCount int `json:"change_count"`
+ TotalFiles int `json:"total_files_changed"`
+ TotalLinesChanged int `json:"total_lines_changed"`
+ FilesTruncated bool `json:"files_truncated,omitempty"`
+ Changes []changeState `json:"changes"`
+}
+
+// batchState describes a batch to the model. Changes are sorted by URI so the
+// same batch always produces the same request.
+func batchState(changes entity.BatchChanges, content map[string]entity.ChangeDiff, maxDiffBytes, maxContextBytes int) batchStateDoc {
+ infos := append([]entity.ChangeInfo(nil), changes.Changes...)
+ sort.Slice(infos, func(i, j int) bool { return infos[i].URI < infos[j].URI })
+
+ doc := batchStateDoc{
+ Queue: changes.Queue,
+ ChangeCount: len(infos),
+ TotalFiles: changes.TotalFiles(),
+ TotalLinesChanged: changes.TotalLinesChanged(),
+ Changes: make([]changeState, 0, len(infos)),
+ }
+ budget := maxFilesInState
+ diffBudget := maxDiffBytes
+ for _, info := range infos {
+ cs := changeState{URI: info.URI, Author: info.Details.Author.Email}
+ if cs.Author == "" {
+ cs.Author = info.Details.Author.Name
+ }
+ for _, f := range info.Details.ChangedFiles {
+ if budget == 0 {
+ doc.FilesTruncated = true
+ break
+ }
+ cs.FileStats = append(cs.FileStats, fileState{Path: f.Path, Added: f.LinesAdded, Deleted: f.LinesDeleted, Modified: f.LinesModified})
+ budget--
+ }
+ cs.Diff, cs.DiffTruncated = truncatePatch(content[info.URI].Patch, diffBudget)
+ diffBudget -= len(cs.Diff)
+ doc.Changes = append(doc.Changes, cs)
+ }
+
+ contextBudget := maxContextBytes
+ for i := range doc.Changes {
+ contextBudget = addContext(&doc.Changes[i], content[doc.Changes[i].URI], contextBudget)
+ }
+ return doc
+}
+
+// addContext fills cs's files and repository context from code within budget
+// bytes and returns what is left. The fixed per-reference cost approximates
+// its JSON keys.
+func addContext(cs *changeState, code entity.ChangeDiff, budget int) int {
+ for _, f := range code.Files {
+ text, truncated := truncatePatch(f.Content, budget-len(f.Path))
+ if text == "" {
+ cs.ContextTruncated = true
+ break
+ }
+ cs.ModifiedFiles = append(cs.ModifiedFiles, fileContentState{Path: f.Path, Content: text, Truncated: truncated})
+ budget -= len(f.Path) + len(text)
+ }
+ if len(code.References) == 0 {
+ return budget
+ }
+ repoCtx := &repositoryContextState{Description: repositoryContextDescription}
+ for _, r := range code.References {
+ cost := len(r.Symbol) + len(r.Path) + len(r.Text) + len(r.DeclarationBefore) + len(r.DeclarationAfter) + 96
+ if cost > budget {
+ cs.ContextTruncated = true
+ break
+ }
+ budget -= cost
+ site := codeLineState{Path: r.Path, Line: r.Line, Text: r.Text}
+ switch r.Kind {
+ case entity.CodeReferenceKindCaller:
+ after := r.DeclarationAfter
+ if after == "" {
+ after = removedDeclaration
+ }
+ repoCtx.Callers = append(repoCtx.Callers, callerState{Symbol: r.Symbol, DeclarationBefore: r.DeclarationBefore, DeclarationAfter: after, CallSite: site})
+ case entity.CodeReferenceKindDefinition:
+ repoCtx.Definitions = append(repoCtx.Definitions, definitionState{Symbol: r.Symbol, Definition: site})
+ }
+ }
+ if len(repoCtx.Callers)+len(repoCtx.Definitions) > 0 {
+ cs.RepositoryContext = repoCtx
+ }
+ return budget
+}
+
+// truncatePatch cuts patch to at most limit bytes at a line boundary.
+func truncatePatch(patch string, limit int) (string, bool) {
+ if len(patch) <= limit {
+ return patch, false
+ }
+ if limit <= 0 {
+ return "", true
+ }
+ cut := patch[:limit]
+ if i := strings.LastIndexByte(cut, '\n'); i >= 0 {
+ cut = cut[:i+1]
+ } else {
+ cut = ""
+ }
+ return cut, true
+}
diff --git a/submitqueue/extension/speculation/scorer/jev/jev_test.go b/submitqueue/extension/speculation/scorer/jev/jev_test.go
new file mode 100644
index 000000000..143ab3afb
--- /dev/null
+++ b/submitqueue/extension/speculation/scorer/jev/jev_test.go
@@ -0,0 +1,404 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package jev
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ nethttp "net/http"
+ "net/http/httptest"
+ "strings"
+ "sync/atomic"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+ "go.uber.org/mock/gomock"
+
+ "github.com/uber/submitqueue/platform/base/change"
+ phttp "github.com/uber/submitqueue/platform/http"
+ changesetfake "github.com/uber/submitqueue/submitqueue/core/changeset/fake"
+ "github.com/uber/submitqueue/submitqueue/entity"
+ changediffmock "github.com/uber/submitqueue/submitqueue/extension/changediff/mock"
+ "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer"
+)
+
+var testBatch = entity.Batch{ID: "7", Queue: "test-queue", Contains: []string{"7"}}
+
+type staticScorer float64
+
+func (s staticScorer) Score(context.Context, entity.Batch, entity.SpeculationPathSet) (float64, error) {
+ return float64(s), nil
+}
+
+// fakeProxy serves /v1/systemone with the given handler and counts calls.
+func fakeProxy(t *testing.T, handle func(w nethttp.ResponseWriter, req systemOneRequest)) (*nethttp.Client, *atomic.Int32) {
+ t.Helper()
+ var calls atomic.Int32
+ srv := httptest.NewServer(nethttp.HandlerFunc(func(w nethttp.ResponseWriter, r *nethttp.Request) {
+ calls.Add(1)
+ assert.Equal(t, DefaultPath, r.URL.Path)
+ assert.Equal(t, nethttp.MethodPost, r.Method)
+ var req systemOneRequest
+ require.NoError(t, json.NewDecoder(r.Body).Decode(&req))
+ handle(w, req)
+ }))
+ t.Cleanup(srv.Close)
+ client, err := phttp.NewClient(srv.URL)
+ require.NoError(t, err)
+ return client, &calls
+}
+
+func noulResponse(p float64) func(nethttp.ResponseWriter, systemOneRequest) {
+ return func(w nethttp.ResponseWriter, _ systemOneRequest) {
+ _, _ = fmt.Fprintf(w, `{"model":"jev-1.13.0","answers":{%q:{"type":"noul","noul":%v}}}`, questionName, p)
+ }
+}
+
+func TestScore(t *testing.T) {
+ tests := []struct {
+ name string
+ handle func(nethttp.ResponseWriter, systemOneRequest)
+ fallback scorer.Scorer
+ want float64
+ wantErr bool
+ }{
+ {
+ name: "returns the model's noul",
+ handle: noulResponse(0.82),
+ want: 0.82,
+ },
+ {
+ name: "non-2xx falls back",
+ handle: func(w nethttp.ResponseWriter, _ systemOneRequest) {
+ w.WriteHeader(nethttp.StatusUnauthorized)
+ },
+ fallback: staticScorer(0.5),
+ want: 0.5,
+ },
+ {
+ name: "refusal falls back",
+ handle: func(w nethttp.ResponseWriter, _ systemOneRequest) {
+ _, _ = fmt.Fprintf(w, `{"answers":{%q:{"type":"refusal"}}}`, questionName)
+ },
+ fallback: staticScorer(0.4),
+ want: 0.4,
+ },
+ {
+ name: "out-of-range noul without fallback is an error",
+ handle: noulResponse(1.5),
+ wantErr: true,
+ },
+ {
+ name: "missing answer without fallback is an error",
+ handle: func(w nethttp.ResponseWriter, _ systemOneRequest) {
+ _, _ = w.Write([]byte(`{"answers":{}}`))
+ },
+ wantErr: true,
+ },
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ client, _ := fakeProxy(t, tt.handle)
+ s := New(Params{
+ Config: scorer.Config{QueueName: "test-queue"},
+ HTTPClient: client,
+ Model: "jev",
+ Resolver: changesetfake.New(),
+ Fallback: tt.fallback,
+ })
+ got, err := s.Score(context.Background(), testBatch, entity.SpeculationPathSet{})
+ if tt.wantErr {
+ require.Error(t, err)
+ return
+ }
+ require.NoError(t, err)
+ assert.InDelta(t, tt.want, got, 1e-9)
+ })
+ }
+}
+
+func TestScoreSendsBatchStateAndModel(t *testing.T) {
+ resolver := changesetfake.New().SetDetailed(testBatch.ID, entity.BatchChanges{
+ BatchID: testBatch.ID,
+ Queue: testBatch.Queue,
+ Changes: []entity.ChangeInfo{{
+ URI: "git://example.com/repo/refs%2Fheads%2Fa/1111111111111111111111111111111111111111",
+ Details: entity.ChangeDetails{
+ Author: entity.Author{Email: "dev@example.com"},
+ ChangedFiles: []entity.ChangedFile{{Path: "demo/a.go", LinesAdded: 10, LinesDeleted: 2}},
+ },
+ }},
+ })
+ var got systemOneRequest
+ client, _ := fakeProxy(t, func(w nethttp.ResponseWriter, req systemOneRequest) {
+ got = req
+ noulResponse(0.9)(w, req)
+ })
+ s := New(Params{HTTPClient: client, Model: "jev", Resolver: resolver})
+
+ _, err := s.Score(context.Background(), testBatch, entity.SpeculationPathSet{})
+ require.NoError(t, err)
+
+ assert.Equal(t, "jev", got.Model)
+ require.Contains(t, got.Questions, questionName)
+ assert.Equal(t, questionTypeNoul, got.Questions[questionName].Type)
+ state, err := json.Marshal(got.State)
+ require.NoError(t, err)
+ assert.JSONEq(t, `{
+ "queue": "test-queue",
+ "change_count": 1,
+ "total_files_changed": 1,
+ "total_lines_changed": 12,
+ "changes": [{
+ "uri": "git://example.com/repo/refs%2Fheads%2Fa/1111111111111111111111111111111111111111",
+ "author": "dev@example.com",
+ "file_stats": [{"path": "demo/a.go", "lines_added": 10, "lines_deleted": 2}]
+ }]
+ }`, string(state))
+}
+
+func TestScoreSendsDiffs(t *testing.T) {
+ const (
+ uriA = "git://example.com/repo/refs%2Fheads%2Fa/1111111111111111111111111111111111111111"
+ uriB = "git://example.com/repo/refs%2Fheads%2Fb/2222222222222222222222222222222222222222"
+ )
+ newResolver := func() *changesetfake.Resolver {
+ return changesetfake.New().
+ Set(testBatch.ID, change.Change{URIs: []string{uriA}}, change.Change{URIs: []string{uriB}}).
+ SetDetailed(testBatch.ID, entity.BatchChanges{
+ BatchID: testBatch.ID,
+ Queue: testBatch.Queue,
+ Changes: []entity.ChangeInfo{{URI: uriA}, {URI: uriB}},
+ })
+ }
+
+ t.Run("each change carries its own diff", func(t *testing.T) {
+ differ := changediffmock.NewMockProvider(gomock.NewController(t))
+ differ.EXPECT().Diff(gomock.Any(), change.Change{URIs: []string{uriA}}).
+ Return([]entity.ChangeDiff{{URI: uriA, Patch: "+func A() {}\n"}}, nil)
+ differ.EXPECT().Diff(gomock.Any(), change.Change{URIs: []string{uriB}}).
+ Return([]entity.ChangeDiff{{URI: uriB, Patch: "+func B() {}\n"}}, nil)
+
+ var got systemOneRequest
+ client, _ := fakeProxy(t, func(w nethttp.ResponseWriter, req systemOneRequest) {
+ got = req
+ noulResponse(0.9)(w, req)
+ })
+ s := New(Params{HTTPClient: client, Resolver: newResolver(), Differ: differ})
+ _, err := s.Score(context.Background(), testBatch, entity.SpeculationPathSet{})
+ require.NoError(t, err)
+
+ var state batchStateDoc
+ raw, err := json.Marshal(got.State)
+ require.NoError(t, err)
+ require.NoError(t, json.Unmarshal(raw, &state))
+ require.Len(t, state.Changes, 2)
+ assert.Equal(t, "+func A() {}\n", state.Changes[0].Diff)
+ assert.Equal(t, "+func B() {}\n", state.Changes[1].Diff)
+ })
+
+ // Scoring without the code would silently be the heuristic the model is
+ // meant to beat, so a diff failure falls back visibly instead.
+ t.Run("diff error falls back without asking the model", func(t *testing.T) {
+ differ := changediffmock.NewMockProvider(gomock.NewController(t))
+ differ.EXPECT().Diff(gomock.Any(), gomock.Any()).Return(nil, errors.New("fetch failed"))
+
+ client, calls := fakeProxy(t, noulResponse(0.9))
+ s := New(Params{HTTPClient: client, Resolver: newResolver(), Differ: differ, Fallback: staticScorer(0.4)})
+ got, err := s.Score(context.Background(), testBatch, entity.SpeculationPathSet{})
+ require.NoError(t, err)
+ assert.InDelta(t, 0.4, got, 1e-9)
+ assert.Zero(t, calls.Load())
+ })
+}
+
+func TestBatchStateContext(t *testing.T) {
+ const uri = "git://example.com/repo/refs%2Fheads%2Fa/1111111111111111111111111111111111111111"
+ changes := entity.BatchChanges{Queue: "q", Changes: []entity.ChangeInfo{{URI: uri}}}
+ caller := entity.CodeReference{
+ Kind: entity.CodeReferenceKindCaller, Symbol: "Scale", Path: "report.go", Line: 4, Text: "return Scale(v)",
+ DeclarationBefore: "func Scale(v int) int {", DeclarationAfter: "func Scale(v, f int) int {",
+ }
+ definition := entity.CodeReference{Kind: entity.CodeReferenceKindDefinition, Symbol: "ParsePort", Path: "parse.go", Line: 9, Text: "func ParsePort(s string) (int, error) {"}
+
+ t.Run("the change and its repository context are kept apart", func(t *testing.T) {
+ doc := batchStateWithDefaults(changes, map[string]entity.ChangeDiff{uri: {
+ URI: uri,
+ Patch: "+func Scale(v, f int) int {\n",
+ Files: []entity.FileContent{{Path: "scale.go", Content: "package p\n"}},
+ References: []entity.CodeReference{caller, definition},
+ }})
+ require.Len(t, doc.Changes, 1)
+ cs := doc.Changes[0]
+ assert.Equal(t, "+func Scale(v, f int) int {\n", cs.Diff)
+ assert.Equal(t, []fileContentState{{Path: "scale.go", Content: "package p\n"}}, cs.ModifiedFiles)
+ require.NotNil(t, cs.RepositoryContext)
+ assert.Equal(t, repositoryContextDescription, cs.RepositoryContext.Description)
+ assert.Equal(t, []callerState{{
+ Symbol: "Scale", DeclarationBefore: "func Scale(v int) int {", DeclarationAfter: "func Scale(v, f int) int {",
+ CallSite: codeLineState{Path: "report.go", Line: 4, Text: "return Scale(v)"},
+ }}, cs.RepositoryContext.Callers)
+ assert.Equal(t, []definitionState{{
+ Symbol: "ParsePort", Definition: codeLineState{Path: "parse.go", Line: 9, Text: "func ParsePort(s string) (int, error) {"},
+ }}, cs.RepositoryContext.Definitions)
+ assert.False(t, cs.ContextTruncated)
+ })
+
+ t.Run("a removed declaration says so", func(t *testing.T) {
+ removed := caller
+ removed.DeclarationAfter = ""
+ doc := batchStateWithDefaults(changes, map[string]entity.ChangeDiff{uri: {URI: uri, References: []entity.CodeReference{removed}}})
+ assert.Equal(t, removedDeclaration, doc.Changes[0].RepositoryContext.Callers[0].DeclarationAfter)
+ })
+
+ t.Run("no references means no repository context", func(t *testing.T) {
+ doc := batchStateWithDefaults(changes, map[string]entity.ChangeDiff{uri: {URI: uri, Patch: "+x\n"}})
+ assert.Nil(t, doc.Changes[0].RepositoryContext)
+ })
+
+ t.Run("configured caps bound the diff", func(t *testing.T) {
+ doc := batchState(changes, map[string]entity.ChangeDiff{uri: {URI: uri, Patch: "aaaa\nbbbb\ncccc\n"}}, 10, 0)
+ assert.Equal(t, "aaaa\nbbbb\n", doc.Changes[0].Diff)
+ assert.True(t, doc.Changes[0].DiffTruncated)
+ })
+
+ t.Run("context past the budget is cut and flagged", func(t *testing.T) {
+ big := strings.Repeat("x\n", DefaultMaxContextBytes)
+ doc := batchStateWithDefaults(changes, map[string]entity.ChangeDiff{uri: {
+ URI: uri,
+ Files: []entity.FileContent{{Path: "big.go", Content: big}},
+ References: []entity.CodeReference{caller},
+ }})
+ cs := doc.Changes[0]
+ require.Len(t, cs.ModifiedFiles, 1)
+ assert.True(t, cs.ModifiedFiles[0].Truncated)
+ assert.LessOrEqual(t, len(cs.ModifiedFiles[0].Content), DefaultMaxContextBytes)
+ assert.Nil(t, cs.RepositoryContext)
+ assert.True(t, cs.ContextTruncated)
+ })
+}
+
+func TestTruncatePatch(t *testing.T) {
+ tests := []struct {
+ name string
+ patch string
+ limit int
+ want string
+ wantTruncated bool
+ }{
+ {name: "fits", patch: "a\nb\n", limit: 10, want: "a\nb\n"},
+ {name: "cut at a line boundary", patch: "aaa\nbbb\nccc\n", limit: 9, want: "aaa\nbbb\n", wantTruncated: true},
+ {name: "no whole line fits", patch: "aaaaaa\n", limit: 3, want: "", wantTruncated: true},
+ {name: "budget spent", patch: "a\n", limit: 0, want: "", wantTruncated: true},
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ got, truncated := truncatePatch(tt.patch, tt.limit)
+ assert.Equal(t, tt.want, got)
+ assert.Equal(t, tt.wantTruncated, truncated)
+ })
+ }
+}
+
+func TestScoreUsesConfiguredPath(t *testing.T) {
+ srv := httptest.NewServer(nethttp.HandlerFunc(func(w nethttp.ResponseWriter, r *nethttp.Request) {
+ if r.URL.Path != "/typesafe/v1/systemone" {
+ w.WriteHeader(nethttp.StatusNotFound)
+ return
+ }
+ noulResponse(0.61)(w, systemOneRequest{})
+ }))
+ t.Cleanup(srv.Close)
+ client, err := phttp.NewClient(srv.URL)
+ require.NoError(t, err)
+
+ s := New(Params{HTTPClient: client, Path: "/typesafe/v1/systemone", Resolver: changesetfake.New()})
+ got, err := s.Score(context.Background(), testBatch, entity.SpeculationPathSet{})
+ require.NoError(t, err)
+ assert.InDelta(t, 0.61, got, 1e-9)
+}
+
+func TestScoreCachesSuccessesOnly(t *testing.T) {
+ var fail atomic.Bool
+ fail.Store(true)
+ client, calls := fakeProxy(t, func(w nethttp.ResponseWriter, req systemOneRequest) {
+ if fail.Load() {
+ w.WriteHeader(nethttp.StatusBadGateway)
+ return
+ }
+ noulResponse(0.7)(w, req)
+ })
+ cache := NewCache(10)
+ newScorer := func() scorer.Scorer {
+ return New(Params{HTTPClient: client, Model: "jev", Resolver: changesetfake.New(), Cache: cache, Fallback: staticScorer(0.5)})
+ }
+
+ got, err := newScorer().Score(context.Background(), testBatch, entity.SpeculationPathSet{})
+ require.NoError(t, err)
+ assert.InDelta(t, 0.5, got, 1e-9)
+
+ fail.Store(false)
+ for range 3 {
+ got, err = newScorer().Score(context.Background(), testBatch, entity.SpeculationPathSet{})
+ require.NoError(t, err)
+ assert.InDelta(t, 0.7, got, 1e-9)
+ }
+ assert.Equal(t, int32(2), calls.Load())
+}
+
+func TestScoreWithoutEndpoint(t *testing.T) {
+ t.Run("falls back", func(t *testing.T) {
+ s := New(Params{Resolver: changesetfake.New(), Fallback: staticScorer(0.6)})
+ got, err := s.Score(context.Background(), testBatch, entity.SpeculationPathSet{})
+ require.NoError(t, err)
+ assert.InDelta(t, 0.6, got, 1e-9)
+ })
+ t.Run("errors without fallback", func(t *testing.T) {
+ s := New(Params{Resolver: changesetfake.New()})
+ _, err := s.Score(context.Background(), testBatch, entity.SpeculationPathSet{})
+ require.Error(t, err)
+ })
+}
+
+func TestScoreResolverError(t *testing.T) {
+ client, calls := fakeProxy(t, noulResponse(0.9))
+ s := New(Params{HTTPClient: client, Resolver: changesetfake.New().FailWith(errors.New("boom")), Fallback: staticScorer(0.3)})
+ got, err := s.Score(context.Background(), testBatch, entity.SpeculationPathSet{})
+ require.NoError(t, err)
+ assert.InDelta(t, 0.3, got, 1e-9)
+ assert.Zero(t, calls.Load())
+}
+
+func TestCacheEvictsOldest(t *testing.T) {
+ c := NewCache(2)
+ c.put("a", 0.1)
+ c.put("b", 0.2)
+ c.put("c", 0.3)
+
+ _, ok := c.get("a")
+ assert.False(t, ok)
+ for key, want := range map[string]float64{"b": 0.2, "c": 0.3} {
+ got, ok := c.get(key)
+ require.True(t, ok, key)
+ assert.InDelta(t, want, got, 1e-9)
+ }
+}
+
+func batchStateWithDefaults(changes entity.BatchChanges, content map[string]entity.ChangeDiff) batchStateDoc {
+ return batchState(changes, content, DefaultMaxDiffBytes, DefaultMaxContextBytes)
+}
diff --git a/submitqueue/extension/speculation/scorer/oracle/BUILD.bazel b/submitqueue/extension/speculation/scorer/oracle/BUILD.bazel
new file mode 100644
index 000000000..d1b874c9d
--- /dev/null
+++ b/submitqueue/extension/speculation/scorer/oracle/BUILD.bazel
@@ -0,0 +1,28 @@
+load("@rules_go//go:def.bzl", "go_library", "go_test")
+
+go_library(
+ name = "go_default_library",
+ srcs = ["oracle.go"],
+ importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer/oracle",
+ visibility = ["//visibility:public"],
+ deps = [
+ "//platform/base/change:go_default_library",
+ "//platform/fakemarker:go_default_library",
+ "//submitqueue/core/changeset:go_default_library",
+ "//submitqueue/entity:go_default_library",
+ "//submitqueue/extension/speculation/scorer:go_default_library",
+ ],
+)
+
+go_test(
+ name = "go_default_test",
+ srcs = ["oracle_test.go"],
+ embed = [":go_default_library"],
+ deps = [
+ "//platform/base/change:go_default_library",
+ "//submitqueue/core/changeset/fake:go_default_library",
+ "//submitqueue/entity:go_default_library",
+ "@com_github_stretchr_testify//assert:go_default_library",
+ "@com_github_stretchr_testify//require:go_default_library",
+ ],
+)
diff --git a/submitqueue/extension/speculation/scorer/oracle/README.md b/submitqueue/extension/speculation/scorer/oracle/README.md
new file mode 100644
index 000000000..0724c2970
--- /dev/null
+++ b/submitqueue/extension/speculation/scorer/oracle/README.md
@@ -0,0 +1,9 @@
+# oracle
+
+`oracle` is a content `Scorer` that already knows each batch's outcome: it prices a batch that will fail at 0.01 and one that will pass at 0.99. It is the ceiling for benchmarking scorers — the fewest builds and shortest time to land speculation can reach on a workload whose outcomes are recorded — and must never be configured on a production queue.
+
+Outcomes come from a `Truth`. `MarkerTruth` reads the fake build runner's `sq-fake=build-fail` URI marker, so on a fake-build workload the oracle agrees with the runner by construction. A replayed workload supplies a `Truth` backed by its recorded CI results. A batch fails if any of its changes fails. A batch whose outcome is not recorded takes the fallback scorer's price, or an error without one.
+
+The prices stop short of 0 and 1 because the `bestfirst` generator ranks paths by summed log-probability, which a certain price would make infinite. Sitting under `evidence`, the oracle's price is still revised by build results; that only pushes it further toward the outcome it already predicted.
+
+In `profiles.yaml`, under a scorer `base`: `{type: oracle}`, optionally with `fallback`.
diff --git a/submitqueue/extension/speculation/scorer/oracle/oracle.go b/submitqueue/extension/speculation/scorer/oracle/oracle.go
new file mode 100644
index 000000000..33f965549
--- /dev/null
+++ b/submitqueue/extension/speculation/scorer/oracle/oracle.go
@@ -0,0 +1,108 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package oracle provides a scorer.Scorer that already knows each batch's
+// outcome. It exists to measure the ceiling of speculation — the fewest builds
+// and shortest time to land any scorer could reach on a workload whose
+// outcomes are recorded — and must never be configured on a production queue.
+// See README.md.
+package oracle
+
+import (
+ "context"
+ "fmt"
+
+ "github.com/uber/submitqueue/platform/base/change"
+ "github.com/uber/submitqueue/platform/fakemarker"
+ "github.com/uber/submitqueue/submitqueue/core/changeset"
+ "github.com/uber/submitqueue/submitqueue/entity"
+ "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer"
+)
+
+// Prices for a known outcome. They stop short of 0 and 1 because the
+// generator ranks by log-probability, which a certain price would make
+// infinite.
+const (
+ PricePasses = 0.99
+ PriceFails = 0.01
+)
+
+// Truth reports the recorded outcome of a batch's changes.
+type Truth interface {
+ // Outcome reports whether a batch made of changes fails, and whether its
+ // outcome is recorded at all. An unknown outcome is not an error.
+ Outcome(ctx context.Context, changes []change.Change) (fails, known bool, err error)
+}
+
+// Params configures an oracle scorer.
+type Params struct {
+ // Config is the per-queue identity this scorer serves.
+ Config scorer.Config
+ // Resolver resolves a batch into its changes.
+ Resolver changeset.Resolver
+ // Truth supplies the recorded outcomes.
+ Truth Truth
+ // Fallback prices a batch whose outcome is not recorded. Nil returns an
+ // error instead, which the generator replaces with its own default.
+ Fallback scorer.Scorer
+}
+
+type oracleScorer struct {
+ p Params
+}
+
+// New returns an oracle scorer.
+func New(p Params) scorer.Scorer {
+ return &oracleScorer{p: p}
+}
+
+func (s *oracleScorer) Score(ctx context.Context, batch entity.Batch, paths entity.SpeculationPathSet) (float64, error) {
+ changes, err := s.p.Resolver.ChangesForBatch(ctx, batch)
+ if err != nil {
+ return 0, fmt.Errorf("resolve changes for batch %s: %w", batch.ID, err)
+ }
+ fails, known, err := s.p.Truth.Outcome(ctx, changes)
+ if err != nil {
+ return 0, fmt.Errorf("look up outcome of batch %s: %w", batch.ID, err)
+ }
+ switch {
+ case known && fails:
+ return PriceFails, nil
+ case known:
+ return PricePasses, nil
+ case s.p.Fallback != nil:
+ return s.p.Fallback.Score(ctx, batch, paths)
+ default:
+ return 0, fmt.Errorf("outcome of batch %s is not recorded", batch.ID)
+ }
+}
+
+// markerTruth reads outcomes from the fake build runner's URI markers, so the
+// oracle agrees with that runner by construction.
+type markerTruth struct{}
+
+// MarkerTruth returns a Truth that matches the fake build runner: a batch
+// fails if any change URI carries "sq-fake=build-fail". Every outcome is known.
+func MarkerTruth() Truth {
+ return markerTruth{}
+}
+
+func (markerTruth) Outcome(_ context.Context, changes []change.Change) (bool, bool, error) {
+ for _, c := range changes {
+ if fakemarker.Token(c.URIs) == "build-fail" {
+ return true, true, nil
+ }
+ }
+ return false, true, nil
+}
diff --git a/submitqueue/extension/speculation/scorer/oracle/oracle_test.go b/submitqueue/extension/speculation/scorer/oracle/oracle_test.go
new file mode 100644
index 000000000..8ec36620d
--- /dev/null
+++ b/submitqueue/extension/speculation/scorer/oracle/oracle_test.go
@@ -0,0 +1,83 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package oracle
+
+import (
+ "context"
+ "errors"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+
+ "github.com/uber/submitqueue/platform/base/change"
+ changesetfake "github.com/uber/submitqueue/submitqueue/core/changeset/fake"
+ "github.com/uber/submitqueue/submitqueue/entity"
+)
+
+type staticScorer float64
+
+func (s staticScorer) Score(context.Context, entity.Batch, entity.SpeculationPathSet) (float64, error) {
+ return float64(s), nil
+}
+
+type unknownTruth struct{}
+
+func (unknownTruth) Outcome(context.Context, []change.Change) (bool, bool, error) {
+ return false, false, nil
+}
+
+var batch = entity.Batch{ID: "1", Queue: "q", Contains: []string{"1"}}
+
+func TestScore(t *testing.T) {
+ const passing = "github://github.example.com/o/r/pull/1/1111111111111111111111111111111111111111"
+ failing := passing + "?sq-fake=build-fail"
+
+ tests := []struct {
+ name string
+ uris []string
+ truth Truth
+ fallback float64
+ want float64
+ wantErr bool
+ }{
+ {name: "known pass", uris: []string{passing}, truth: MarkerTruth(), want: PricePasses},
+ {name: "known fail", uris: []string{failing}, truth: MarkerTruth(), want: PriceFails},
+ {name: "one failing change fails the batch", uris: []string{passing, failing}, truth: MarkerTruth(), want: PriceFails},
+ {name: "unknown falls back", uris: []string{passing}, truth: unknownTruth{}, fallback: 0.6, want: 0.6},
+ {name: "unknown without fallback is an error", uris: []string{passing}, truth: unknownTruth{}, wantErr: true},
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ p := Params{Resolver: changesetfake.New().Set(batch.ID, change.Change{URIs: tt.uris}), Truth: tt.truth}
+ if tt.fallback > 0 {
+ p.Fallback = staticScorer(tt.fallback)
+ }
+ got, err := New(p).Score(context.Background(), batch, entity.SpeculationPathSet{})
+ if tt.wantErr {
+ require.Error(t, err)
+ return
+ }
+ require.NoError(t, err)
+ assert.InDelta(t, tt.want, got, 1e-9)
+ })
+ }
+
+ t.Run("resolver error", func(t *testing.T) {
+ _, err := New(Params{Resolver: changesetfake.New().FailWith(errors.New("boom")), Truth: MarkerTruth()}).
+ Score(context.Background(), batch, entity.SpeculationPathSet{})
+ require.Error(t, err)
+ })
+}
diff --git a/submitqueue/orchestrator/controller/speculate/doc.go b/submitqueue/orchestrator/controller/speculate/doc.go
index 79ec04ee7..eee44dabe 100644
--- a/submitqueue/orchestrator/controller/speculate/doc.go
+++ b/submitqueue/orchestrator/controller/speculate/doc.go
@@ -83,7 +83,10 @@
//
// Cancelling is intent, not fact: the build keeps its slot until CI actually
// stops it, and only an observation of that stop (or proof nothing was ever
-// dispatched) moves the path to cancelled. The intent needs no dispatch of its
+// dispatched) moves the path to cancelled. A pending path the run's own read
+// found no build for skips cancelling and is cancelled at once: no build will
+// ever report on it, so the next run is the only thing that could free its
+// slot, and a quiet queue may never send one. The intent needs no dispatch of its
// own — the poll loop reads it off the set and asks the runner to stop the
// build. A terminal path can be resurrected by a new build proposal — status
// returns to pending and Attempt increments, the one backwards step in the
diff --git a/submitqueue/orchestrator/controller/speculate/finalize.go b/submitqueue/orchestrator/controller/speculate/finalize.go
index 420acd7f3..9386741ee 100644
--- a/submitqueue/orchestrator/controller/speculate/finalize.go
+++ b/submitqueue/orchestrator/controller/speculate/finalize.go
@@ -104,7 +104,7 @@ func (c *Controller) finalize(ctx context.Context, snap *snapshot) error {
if !ok {
winner, bypassed = bypassablePath(batch, set, *snap)
}
- if supersede(&set, winner.ID, nowMs) {
+ if supersede(&set, winner.ID, nowMs, snap.undispatched) {
snap.pathSets[batch.ID] = set
snap.markDirty(batch.ID)
}
@@ -210,7 +210,7 @@ func (c *Controller) finalizeCancellations(ctx context.Context, snap *snapshot,
metrics.NamedCounter(c.metricsScope, opName, "cancel_batch", 1)
set, exists := snap.pathSets[batch.ID]
- if exists && cancelAllPaths(&set, nowMs) {
+ if exists && cancelAllPaths(&set, nowMs, snap.undispatched) {
snap.pathSets[batch.ID] = set
snap.markDirty(batch.ID)
}
@@ -428,10 +428,14 @@ func (c *Controller) recoverable(ctx context.Context, store orchstorage.Storage,
// is one neither terminal nor already cancelling.
//
// Cancelling is intent, not fact: the build may still be occupying CI, and
-// only an observation of it actually stopping (or of nothing ever having been
-// dispatched) moves the path on to cancelled. cancelBrokenPathsInSet,
-// supersede, and cancelAllPaths are the three reasons to ask.
-func markCancelling(set *entity.SpeculationPathSet, nowMs int64, keep func(entity.SpeculationPathEntry) bool) bool {
+// only an observation of it actually stopping moves the path on to cancelled.
+// A path in undispatched has no build to observe, so it is cancelled outright;
+// leaving it cancelling would hold its budget slot until some later run, and
+// nothing guarantees one comes — a quiet queue would wedge behind it. A build
+// dispatched after the read is stopped by the poll loop, which treats
+// cancelled like cancelling. cancelBrokenPathsInSet, supersede, and
+// cancelAllPaths are the three reasons to ask.
+func markCancelling(set *entity.SpeculationPathSet, nowMs int64, undispatched map[string]bool, keep func(entity.SpeculationPathEntry) bool) bool {
changed := false
for i := range set.Paths {
entry := &set.Paths[i]
@@ -442,6 +446,9 @@ func markCancelling(set *entity.SpeculationPathSet, nowMs int64, keep func(entit
continue
}
entry.Status = entity.SpeculationPathStatusCancelling
+ if undispatched[entry.ID] {
+ entry.Status = entity.SpeculationPathStatusCancelled
+ }
entry.UpdatedAtMs = nowMs
changed = true
}
@@ -452,7 +459,7 @@ func markCancelling(set *entity.SpeculationPathSet, nowMs int64, keep func(entit
// Those guesses can no longer come true, so their builds are only spending
// budget.
func cancelBrokenPathsInSet(set *entity.SpeculationPathSet, snap snapshot, nowMs int64) bool {
- return markCancelling(set, nowMs, func(entry entity.SpeculationPathEntry) bool {
+ return markCancelling(set, nowMs, snap.undispatched, func(entry entity.SpeculationPathEntry) bool {
return !assumptionBroken(entry.Path, snap)
})
}
@@ -460,16 +467,16 @@ func cancelBrokenPathsInSet(set *entity.SpeculationPathSet, snap snapshot, nowMs
// supersede stops every path other than the winner once the head can land.
// Its live siblings cannot help any more but still hold CI slots the rest of
// the queue could use.
-func supersede(set *entity.SpeculationPathSet, winnerID string, nowMs int64) bool {
- return markCancelling(set, nowMs, func(entry entity.SpeculationPathEntry) bool {
+func supersede(set *entity.SpeculationPathSet, winnerID string, nowMs int64, undispatched map[string]bool) bool {
+ return markCancelling(set, nowMs, undispatched, func(entry entity.SpeculationPathEntry) bool {
return entry.ID == winnerID
})
}
// cancelAllPaths stops every path that is still running. Used when the head
// itself was cancelled by the user, so no path can help it any more.
-func cancelAllPaths(set *entity.SpeculationPathSet, nowMs int64) bool {
- return markCancelling(set, nowMs, func(entity.SpeculationPathEntry) bool {
+func cancelAllPaths(set *entity.SpeculationPathSet, nowMs int64, undispatched map[string]bool) bool {
+ return markCancelling(set, nowMs, undispatched, func(entity.SpeculationPathEntry) bool {
return false
})
}
diff --git a/submitqueue/orchestrator/controller/speculate/outcome_test.go b/submitqueue/orchestrator/controller/speculate/outcome_test.go
index e14dced6e..663852140 100644
--- a/submitqueue/orchestrator/controller/speculate/outcome_test.go
+++ b/submitqueue/orchestrator/controller/speculate/outcome_test.go
@@ -433,14 +433,14 @@ func TestSupersede(t *testing.T) {
entity.SpeculationPathStatusFailed)
set := setOf(winner, sibling, finished)
- assert.True(t, supersede(&set, winner.ID, 99))
+ assert.True(t, supersede(&set, winner.ID, 99, nil))
assert.Equal(t, entity.SpeculationPathStatusPassed, set.Paths[0].Status, "the winner is untouched")
assert.Equal(t, entity.SpeculationPathStatusCancelling, set.Paths[1].Status)
assert.Equal(t, int64(99), set.Paths[1].UpdatedAtMs)
assert.Equal(t, entity.SpeculationPathStatusFailed, set.Paths[2].Status, "a finished path is left alone")
- assert.False(t, supersede(&set, winner.ID, 100), "a second pass changes nothing")
+ assert.False(t, supersede(&set, winner.ID, 100, nil), "a second pass changes nothing")
}
func TestAllPathsStopped(t *testing.T) {
@@ -465,10 +465,10 @@ func TestCancelAllPaths(t *testing.T) {
entity.SpeculationPathStatusPassed)
set := setOf(running, done)
- assert.True(t, cancelAllPaths(&set, 7))
+ assert.True(t, cancelAllPaths(&set, 7, nil))
assert.Equal(t, entity.SpeculationPathStatusCancelling, set.Paths[0].Status)
assert.Equal(t, entity.SpeculationPathStatusPassed, set.Paths[1].Status,
"a finished build has no slot to release")
- assert.False(t, cancelAllPaths(&set, 8), "a second pass changes nothing")
+ assert.False(t, cancelAllPaths(&set, 8, nil), "a second pass changes nothing")
}
diff --git a/submitqueue/orchestrator/controller/speculate/run.go b/submitqueue/orchestrator/controller/speculate/run.go
index c1e4ee99e..b69966b14 100644
--- a/submitqueue/orchestrator/controller/speculate/run.go
+++ b/submitqueue/orchestrator/controller/speculate/run.go
@@ -124,11 +124,12 @@ func (c *Controller) read(ctx context.Context, store orchstorage.Storage, queue
}
snap := snapshot{
- store: store,
- batches: make(map[string]entity.Batch, len(inFlight)),
- inFlight: inFlight,
- pathSets: make(map[string]entity.SpeculationPathSet, len(inFlight)),
- dirty: make(map[string]bool, len(inFlight)),
+ store: store,
+ batches: make(map[string]entity.Batch, len(inFlight)),
+ inFlight: inFlight,
+ pathSets: make(map[string]entity.SpeculationPathSet, len(inFlight)),
+ undispatched: make(map[string]bool),
+ dirty: make(map[string]bool, len(inFlight)),
}
for _, batch := range inFlight {
@@ -167,7 +168,7 @@ func (c *Controller) read(ctx context.Context, store orchstorage.Storage, queue
return snapshot{}, c.attributed(fmt.Errorf("failed to get path set for batch %s: %w", batch.ID, err),
entity.BatchSubject(batch.ID))
}
- changed, err := c.updatePathsFromBuilds(ctx, store, &set)
+ changed, err := c.updatePathsFromBuilds(ctx, store, &set, snap.undispatched)
if err != nil {
return snapshot{}, err
}
@@ -194,7 +195,7 @@ func (c *Controller) read(ctx context.Context, store orchstorage.Storage, queue
// the run makes later, which keeps this controller the path set's single
// writer: the build stages record what CI did on per-build records, and the
// run alone decides when that becomes the path's status.
-func (c *Controller) updatePathsFromBuilds(ctx context.Context, store orchstorage.Storage, set *entity.SpeculationPathSet) (bool, error) {
+func (c *Controller) updatePathsFromBuilds(ctx context.Context, store orchstorage.Storage, set *entity.SpeculationPathSet, undispatched map[string]bool) (bool, error) {
changed := false
for i := range set.Paths {
entry := &set.Paths[i]
@@ -220,6 +221,10 @@ func (c *Controller) updatePathsFromBuilds(ctx context.Context, store orchstorag
entry.Status = entity.SpeculationPathStatusCancelled
changed = true
metrics.NamedCounter(c.metricsScope, opName, "path_cancelled_undispatched", 1)
+ continue
+ }
+ if entry.Status == entity.SpeculationPathStatusPending {
+ undispatched[entry.ID] = true
}
continue
}
diff --git a/submitqueue/orchestrator/controller/speculate/run_test.go b/submitqueue/orchestrator/controller/speculate/run_test.go
index c40b172ce..12064d845 100644
--- a/submitqueue/orchestrator/controller/speculate/run_test.go
+++ b/submitqueue/orchestrator/controller/speculate/run_test.go
@@ -573,6 +573,37 @@ func TestRun_BrokenPathsAreVisibleToTheSpeculator(t *testing.T) {
"the Speculator must see the broken path as already cancelling")
}
+// A funded path whose build was never dispatched has nothing to wait for once
+// it breaks. Left cancelling it would hold its budget slot until some later
+// run, and a quiet queue sends none: the stalled batch is the only thing that
+// could produce the next signal, and it is waiting on the slot.
+func TestRun_BrokenUndispatchedPathIsCancelledInTheSameRun(t *testing.T) {
+ ctrl := gomock.NewController(t)
+ broken := pathOver(entity.DependencyAssumptionSucceeds, entity.DependencyAssumptionFails)
+ spec := &scriptedSpeculator{}
+
+ h := newRunHarness(t, ctrl, spec, []entity.Batch{speculatingHead()})
+ h.noBuildsDispatched()
+ h.batches.EXPECT().Get(gomock.Any(), dep1).Return(entity.Batch{ID: dep1, State: entity.BatchStateFailed}, nil)
+ h.batches.EXPECT().Get(gomock.Any(), dep2).Return(entity.Batch{ID: dep2, State: entity.BatchStateSpeculating}, nil)
+ h.pathSets.EXPECT().Get(gomock.Any(), head).Return(entity.SpeculationPathSet{
+ Head: head,
+ Paths: []entity.SpeculationPathEntry{entryFor(broken, entity.SpeculationPathStatusPending)},
+ Version: 2,
+ }, nil)
+ h.pathSets.EXPECT().Update(gomock.Any(), gomock.Any(), int32(2), int32(3)).
+ DoAndReturn(func(_ context.Context, set entity.SpeculationPathSet, _, _ int32) error {
+ assert.Equal(t, entity.SpeculationPathStatusCancelled, set.Paths[0].Status)
+ return nil
+ })
+
+ require.NoError(t, h.run(head))
+
+ require.Len(t, spec.gotSets, 1)
+ assert.Equal(t, entity.SpeculationPathStatusCancelled, spec.gotSets[0].Paths[0].Status,
+ "the Speculator must see the slot as free in the run that broke the path")
+}
+
// cancelBrokenPathsInSet marks broken paths cancelling, not cancelled: their builds
// may still be occupying CI, and only the signal that sees them stop can call
// it done.
diff --git a/submitqueue/orchestrator/controller/speculate/snapshot.go b/submitqueue/orchestrator/controller/speculate/snapshot.go
index 31781b981..8ba7acf3a 100644
--- a/submitqueue/orchestrator/controller/speculate/snapshot.go
+++ b/submitqueue/orchestrator/controller/speculate/snapshot.go
@@ -49,6 +49,10 @@ type snapshot struct {
// already reflect what each path's build actually did — see
// (*Controller).updatePathsFromBuilds.
pathSets map[string]entity.SpeculationPathSet
+ // undispatched holds the IDs of live paths whose current attempt had no
+ // build at read time. Cancelling one of these has nothing to wait for, so
+ // it goes straight to cancelled — see markCancelling.
+ undispatched map[string]bool
// dirty marks heads whose in-memory set differs from what is stored.
// Always touch it through markDirty / markClean / isDirty so every step
// of the handshake is greppable; see those methods for the contract.
diff --git a/tool/jevcorpus/BUILD.bazel b/tool/jevcorpus/BUILD.bazel
new file mode 100644
index 000000000..0e8e43e0e
--- /dev/null
+++ b/tool/jevcorpus/BUILD.bazel
@@ -0,0 +1,52 @@
+load("@rules_go//go:def.bzl", "go_binary", "go_library", "go_test")
+
+go_library(
+ name = "go_default_library",
+ srcs = [
+ "eval.go",
+ "generate.go",
+ "main.go",
+ "manifest.go",
+ "plan.go",
+ "resolver.go",
+ ],
+ importpath = "github.com/uber/submitqueue/tool/jevcorpus",
+ visibility = ["//visibility:private"],
+ deps = [
+ "//platform/base/change:go_default_library",
+ "//platform/base/change/git:go_default_library",
+ "//platform/git/repo:go_default_library",
+ "//platform/http:go_default_library",
+ "//submitqueue/core/changeset:go_default_library",
+ "//submitqueue/entity:go_default_library",
+ "//submitqueue/extension/buildrunner/replay:go_default_library",
+ "//submitqueue/extension/changediff:go_default_library",
+ "//submitqueue/extension/changediff/git:go_default_library",
+ "//submitqueue/extension/changeprovider:go_default_library",
+ "//submitqueue/extension/changeprovider/git:go_default_library",
+ "//submitqueue/extension/speculation/scorer:go_default_library",
+ "//submitqueue/extension/speculation/scorer/jev:go_default_library",
+ "//tool/jevcorpus/grade:go_default_library",
+ "//tool/jevcorpus/mutate:go_default_library",
+ "@com_github_uber_go_tally//:go_default_library",
+ "@org_golang_x_oauth2//:go_default_library",
+ "@org_uber_go_zap//:go_default_library",
+ ],
+)
+
+go_binary(
+ name = "jevcorpus",
+ embed = [":go_default_library"],
+ visibility = ["//visibility:public"],
+)
+
+go_test(
+ name = "go_default_test",
+ srcs = ["plan_test.go"],
+ embed = [":go_default_library"],
+ deps = [
+ "//tool/jevcorpus/mutate:go_default_library",
+ "@com_github_stretchr_testify//assert:go_default_library",
+ "@com_github_stretchr_testify//require:go_default_library",
+ ],
+)
diff --git a/tool/jevcorpus/README.md b/tool/jevcorpus/README.md
new file mode 100644
index 000000000..7db96608c
--- /dev/null
+++ b/tool/jevcorpus/README.md
@@ -0,0 +1,27 @@
+# jevcorpus
+
+Builds a synthetic corpus of safe and unsafe Go changes with build-verified labels, and grades outcome scorers against it. Benchmarks only. See [doc/rfc/submitqueue/synthetic-corpus.md](../../doc/rfc/submitqueue/synthetic-corpus.md) for the design.
+
+## generate
+
+`jevcorpus generate` clones a seed repository, runs the tests under `-pkgs` once at `-base`, and keeps the packages that pass as mutation targets. It then plans `-count` changes split by `-unsafe-ratio` and spread evenly over the S, M, L and XL tiers. Each change is built from the `mutate` package's mutators on a clean base, verified with `-verify` (run at the module root with the touched packages in `$PKGS`), and committed as `corpus/c-NNNN`. A change whose outcome contradicts its intent is rebuilt, up to a few tries, unless `-keep-mismatch` is set. A slot that cannot be filled is skipped and logged; L and XL slots need a seed with hundreds of candidate files.
+
+It writes the dataset at `-out`, the manifest beside it as `.manifest.json`, and a bare sandbox at `-sandbox` holding the base as `main` and one branch per kept change. One seed, base and flag set always plan the same corpus, though verification time and flaky tests can change which attempts are kept.
+
+Example, with a replay candidate as the seed:
+
+```bash
+git clone --quiet https://github.com/jaegertracing/jaeger /tmp/seed/jaeger
+bazel run //tool/jevcorpus -- generate -repo /tmp/seed/jaeger -pkgs "./internal/... ./cmd/..." \
+ -count 80 -out /tmp/corpus/jaeger.json -sandbox /tmp/corpus/sandbox.git
+```
+
+## eval
+
+`jevcorpus eval` scores one batch per change, plus `-batches` sampled batches of `-batch-size` changes, with each scorer in `-scorers`. `size` prices by lines changed; `jev` is the real scorer over the git change and diff providers, reading the proxy root and key from `LITE_URL` and `LITE_KEY` by default, with no fallback or cache. `-files` and `-references` match the scorer's `jev.diff` options, and `-path /typesafe/v1/systemone` selects the TypeSafe pass-through. It prints a report per scorer and slice, and with `-out` writes every row and report as JSON.
+
+Sending diffs sends the seed's code to the proxy and its model provider.
+
+## In a queue
+
+The dataset is in the replay format, so a corpus runs through the git compose overlay exactly as a replay epoch does: `replaybench uris -dataset ` lists the change URIs, `buildRunner: {type: replay, dataset: }` answers builds from the labels, and `oracle` reads them as the ceiling. See [doc/rfc/submitqueue/replay-benchmark.md](../../doc/rfc/submitqueue/replay-benchmark.md).
diff --git a/tool/jevcorpus/eval.go b/tool/jevcorpus/eval.go
new file mode 100644
index 000000000..094214e04
--- /dev/null
+++ b/tool/jevcorpus/eval.go
@@ -0,0 +1,419 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "context"
+ "errors"
+ "flag"
+ "fmt"
+ "math/rand"
+ nethttp "net/http"
+ "os"
+ "path/filepath"
+ "sort"
+ "strconv"
+ "strings"
+ "sync"
+ "time"
+
+ "github.com/uber-go/tally"
+ "go.uber.org/zap"
+ "golang.org/x/oauth2"
+
+ changegit "github.com/uber/submitqueue/platform/base/change/git"
+ gitrepo "github.com/uber/submitqueue/platform/git/repo"
+ phttp "github.com/uber/submitqueue/platform/http"
+ "github.com/uber/submitqueue/submitqueue/core/changeset"
+ "github.com/uber/submitqueue/submitqueue/entity"
+ "github.com/uber/submitqueue/submitqueue/extension/buildrunner/replay"
+ "github.com/uber/submitqueue/submitqueue/extension/changediff"
+ gitdiffer "github.com/uber/submitqueue/submitqueue/extension/changediff/git"
+ "github.com/uber/submitqueue/submitqueue/extension/changeprovider"
+ gitprovider "github.com/uber/submitqueue/submitqueue/extension/changeprovider/git"
+ "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer"
+ "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer/jev"
+ "github.com/uber/submitqueue/tool/jevcorpus/grade"
+)
+
+const (
+ evalQueue = "corpus-queue"
+ evalRemote = "git.example.com"
+ evalRepo = "sandbox"
+)
+
+// sizeScale is the line count at which the size baseline prices a batch at
+// 0.5. Only its ranking matters to AUC; it stands for a scorer that has
+// learned nothing but "bigger is riskier".
+const sizeScale = 250
+
+type evalFlags struct {
+ dataset, manifest, sandbox, scorers, out string
+ baseURL, baseURLEnv, tokenEnv, path, model string
+ timeout time.Duration
+ diff, files, references bool
+ maxDiffBytes, maxContextBytes int
+ batchSize, batches, concurrency, limit, bins int
+ seed int64
+}
+
+// evalRow is one scored batch.
+type evalRow struct {
+ Batch string `json:"batch"`
+ Branches []string `json:"branches"`
+ Lands bool `json:"lands"`
+ // Slices are the result groups the row is graded in.
+ Slices []string `json:"slices"`
+ DiffBytes int `json:"diff_bytes"`
+ Scores map[string]float64 `json:"scores"`
+ Errors map[string]string `json:"errors,omitempty"`
+ LatencyMs map[string]int64 `json:"latency_ms"`
+}
+
+type evalResult struct {
+ Dataset string `json:"dataset"`
+ Scorers []string `json:"scorers"`
+ Rows []evalRow `json:"rows"`
+ Reports map[string]map[string]grade.Report `json:"reports"`
+}
+
+func runEval(args []string) error {
+ var f evalFlags
+ fs := flag.NewFlagSet("eval", flag.ExitOnError)
+ fs.StringVar(&f.dataset, "dataset", "", "dataset written by generate (required)")
+ fs.StringVar(&f.manifest, "manifest", "", "manifest written by generate; defaults to the one beside -dataset")
+ fs.StringVar(&f.sandbox, "sandbox", "", "sandbox repository written by generate (required)")
+ fs.StringVar(&f.scorers, "scorers", "size,jev", "comma-separated scorers to grade: size, jev")
+ fs.StringVar(&f.out, "out", "", "write every row and report as JSON here")
+ fs.StringVar(&f.baseURL, "base-url", "", "jev: proxy root; defaults to $LITE_URL")
+ fs.StringVar(&f.baseURLEnv, "base-url-env", "LITE_URL", "jev: environment variable holding the proxy root")
+ fs.StringVar(&f.tokenEnv, "token-env", "LITE_KEY", "jev: environment variable holding the virtual key")
+ fs.StringVar(&f.path, "path", jev.DefaultPath, "jev: System One route")
+ fs.StringVar(&f.model, "model", "jev", "jev: proxy model name")
+ fs.DurationVar(&f.timeout, "timeout", 60*time.Second, "jev: bound on one model call")
+ fs.BoolVar(&f.diff, "diff", true, "jev: send each change's diff")
+ fs.BoolVar(&f.files, "files", false, "jev: also send touched files in full")
+ fs.BoolVar(&f.references, "references", false, "jev: also send callers and definitions")
+ fs.IntVar(&f.maxDiffBytes, "max-diff-bytes", 0, "jev: diff budget per batch; zero is the scorer's default")
+ fs.IntVar(&f.maxContextBytes, "max-context-bytes", 0, "jev: context budget per batch; zero is the scorer's default")
+ fs.IntVar(&f.batchSize, "batch-size", 3, "changes per multi-change batch")
+ fs.IntVar(&f.batches, "batches", 0, "multi-change batches to sample in addition to one batch per change")
+ fs.IntVar(&f.concurrency, "concurrency", 4, "batches scored at once")
+ fs.IntVar(&f.limit, "limit", 0, "score only the first n changes; zero scores all")
+ fs.IntVar(&f.bins, "bins", 10, "calibration bins")
+ fs.Int64Var(&f.seed, "seed", 1, "random seed for sampling multi-change batches")
+ if err := fs.Parse(args); err != nil {
+ return err
+ }
+ if f.dataset == "" || f.sandbox == "" {
+ return errors.New("-dataset and -sandbox are required")
+ }
+ if f.manifest == "" {
+ f.manifest = manifestPathFor(f.dataset)
+ }
+ return evaluate(context.Background(), f)
+}
+
+func evaluate(ctx context.Context, f evalFlags) error {
+ ds, err := replay.Load(f.dataset)
+ if err != nil {
+ return err
+ }
+ m, err := loadManifest(f.manifest)
+ if err != nil {
+ return err
+ }
+ bySHA := make(map[string]ManifestChange, len(m.Changes))
+ for _, c := range m.Changes {
+ bySHA[c.SHA] = c
+ }
+
+ sandbox, err := filepath.Abs(f.sandbox)
+ if err != nil {
+ return err
+ }
+ tmp, err := os.MkdirTemp("", "jevcorpus-eval-")
+ if err != nil {
+ return err
+ }
+ defer os.RemoveAll(tmp)
+ repo, err := gitrepo.NewRepo(gitrepo.RepoConfig{Path: filepath.Join(tmp, "copy"), RemoteURL: sandbox, Target: ds.BaseBranch})
+ if err != nil {
+ return err
+ }
+ if err := repo.Provision(ctx); err != nil {
+ return err
+ }
+ provider := gitprovider.New(gitprovider.Params{
+ Config: changeprovider.Config{QueueName: evalQueue}, Repo: repo,
+ Logger: zap.NewNop().Sugar(), MetricsScope: tally.NoopScope,
+ })
+
+ var changes []ManifestChange
+ resolver := corpusResolver{queue: evalQueue, provider: provider, uriByRequest: map[string]string{}}
+ for _, a := range ds.Attempts {
+ if a.ReplaySHA == "" {
+ continue
+ }
+ mc, ok := bySHA[a.ReplaySHA]
+ if !ok {
+ return fmt.Errorf("dataset change %s is missing from the manifest", a.ReplaySHA)
+ }
+ if f.limit > 0 && len(changes) == f.limit {
+ break
+ }
+ changes = append(changes, mc)
+ resolver.uriByRequest[strconv.Itoa(len(changes))] = changegit.ChangeID{
+ Scheme: "git", Remote: evalRemote, Repo: evalRepo, Ref: "refs/heads/" + mc.Branch, CommitSHA: mc.SHA,
+ }.String()
+ }
+ if len(changes) == 0 {
+ return errors.New("dataset has no materialized changes")
+ }
+
+ names, scorers, err := buildScorers(f, resolver, repo)
+ if err != nil {
+ return err
+ }
+ batches, rows := planEvalBatches(changes, f, rand.New(rand.NewSource(f.seed)))
+ logf("scoring %d batches with %s", len(batches), strings.Join(names, ", "))
+ scoreRows(ctx, batches, rows, names, scorers, f.concurrency)
+
+ result := evalResult{Dataset: f.dataset, Scorers: names, Rows: rows, Reports: gradeRows(rows, names, f.bins)}
+ printReports(result)
+ if f.out != "" {
+ return writeJSON(f.out, result)
+ }
+ return nil
+}
+
+func buildScorers(f evalFlags, resolver changeset.Resolver, repo *gitrepo.Repo) ([]string, map[string]scorer.Scorer, error) {
+ cfg := scorer.Config{QueueName: evalQueue}
+ scorers := map[string]scorer.Scorer{}
+ var names []string
+ for _, name := range strings.Split(f.scorers, ",") {
+ name = strings.TrimSpace(name)
+ switch name {
+ case "":
+ continue
+ case "size":
+ scorers[name] = sizeScorer{resolver: resolver}
+ case "jev":
+ client, err := jevClient(f)
+ if err != nil {
+ return nil, nil, err
+ }
+ var differ changediff.Provider
+ if f.diff {
+ differ = gitdiffer.New(gitdiffer.Params{
+ Config: changediff.Config{QueueName: evalQueue}, Repo: repo, MetricsScope: tally.NoopScope,
+ IncludeFiles: f.files, IncludeReferences: f.references,
+ })
+ }
+ // No fallback and no cache: a failure must show as a failure, and
+ // every batch is asked once.
+ scorers[name] = jev.New(jev.Params{
+ Config: cfg, HTTPClient: client, Path: f.path, Model: f.model, Timeout: f.timeout,
+ Resolver: resolver, Differ: differ, MaxDiffBytes: f.maxDiffBytes, MaxContextBytes: f.maxContextBytes,
+ })
+ default:
+ return nil, nil, fmt.Errorf("unknown scorer %q; want size or jev", name)
+ }
+ names = append(names, name)
+ }
+ if len(names) == 0 {
+ return nil, nil, errors.New("-scorers names no scorer")
+ }
+ return names, scorers, nil
+}
+
+func jevClient(f evalFlags) (*nethttp.Client, error) {
+ baseURL := f.baseURL
+ if baseURL == "" {
+ baseURL = os.Getenv(f.baseURLEnv)
+ }
+ token := os.Getenv(f.tokenEnv)
+ if baseURL == "" || token == "" {
+ return nil, fmt.Errorf("jev needs a proxy root (-base-url or $%s) and a key ($%s); drop it from -scorers to grade baselines only", f.baseURLEnv, f.tokenEnv)
+ }
+ client, err := phttp.NewClient(baseURL)
+ if err != nil {
+ return nil, err
+ }
+ client.Timeout = f.timeout
+ client.Transport = &oauth2.Transport{Source: oauth2.StaticTokenSource(&oauth2.Token{AccessToken: token}), Base: client.Transport}
+ return client, nil
+}
+
+// sizeScorer prices a batch by its size alone, falling as lines grow.
+type sizeScorer struct {
+ resolver changeset.Resolver
+}
+
+func (s sizeScorer) Score(ctx context.Context, batch entity.Batch, _ entity.SpeculationPathSet) (float64, error) {
+ changes, err := s.resolver.DetailedForBatch(ctx, batch)
+ if err != nil {
+ return 0, err
+ }
+ return 1 / (1 + float64(changes.TotalLinesChanged())/sizeScale), nil
+}
+
+// planEvalBatches returns one batch per change, then f.batches sampled
+// batches of f.batchSize distinct changes. A batch lands only if every change
+// in it does, matching the replay runner.
+func planEvalBatches(changes []ManifestChange, f evalFlags, r *rand.Rand) ([]entity.Batch, []evalRow) {
+ var batches []entity.Batch
+ var rows []evalRow
+ add := func(members []int, slices []string) {
+ b := entity.Batch{ID: strconv.Itoa(len(batches) + 1), Queue: evalQueue}
+ row := evalRow{Batch: b.ID, Lands: true, Slices: slices}
+ for _, i := range members {
+ b.Contains = append(b.Contains, strconv.Itoa(i+1))
+ row.Branches = append(row.Branches, changes[i].Branch)
+ row.Lands = row.Lands && changes[i].Lands
+ row.DiffBytes += changes[i].DiffBytes
+ }
+ batches = append(batches, b)
+ rows = append(rows, row)
+ }
+ for i, c := range changes {
+ add([]int{i}, singleSlices(c))
+ }
+ if f.batchSize > 1 && f.batchSize <= len(changes) {
+ for n := 0; n < f.batches; n++ {
+ add(r.Perm(len(changes))[:f.batchSize], []string{fmt.Sprintf("batch of %d", f.batchSize)})
+ }
+ }
+ return batches, rows
+}
+
+func singleSlices(c ManifestChange) []string {
+ slices := []string{"all", "tier " + string(c.Tier), "category " + c.Category()}
+ if c.Needle.Position != NeedleNone {
+ slices = append(slices, "needle "+string(c.Needle.Position))
+ if c.Needle.BeyondDiffBudget {
+ slices = append(slices, "needle beyond diff budget")
+ }
+ }
+ return slices
+}
+
+func scoreRows(ctx context.Context, batches []entity.Batch, rows []evalRow, names []string, scorers map[string]scorer.Scorer, concurrency int) {
+ work := make(chan int)
+ var wg sync.WaitGroup
+ var progressMu sync.Mutex
+ finished := 0
+ for w := 0; w < max(concurrency, 1); w++ {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ for i := range work {
+ row := &rows[i]
+ row.Scores, row.Errors, row.LatencyMs = map[string]float64{}, map[string]string{}, map[string]int64{}
+ for _, name := range names {
+ began := time.Now()
+ p, err := scorers[name].Score(ctx, batches[i], entity.SpeculationPathSet{})
+ row.LatencyMs[name] = time.Since(began).Milliseconds()
+ if err != nil {
+ row.Errors[name] = err.Error()
+ continue
+ }
+ row.Scores[name] = p
+ }
+ progressMu.Lock()
+ finished++
+ if finished%10 == 0 || finished == len(rows) {
+ logf("scored %d/%d", finished, len(rows))
+ }
+ progressMu.Unlock()
+ }
+ }()
+ }
+ for i := range rows {
+ work <- i
+ }
+ close(work)
+ wg.Wait()
+}
+
+func gradeRows(rows []evalRow, names []string, bins int) map[string]map[string]grade.Report {
+ out := map[string]map[string]grade.Report{}
+ for _, name := range names {
+ bySlice := map[string][]grade.Prediction{}
+ for _, row := range rows {
+ p, ok := row.Scores[name]
+ if !ok {
+ continue
+ }
+ for _, s := range row.Slices {
+ bySlice[s] = append(bySlice[s], grade.Prediction{P: p, Lands: row.Lands})
+ }
+ }
+ out[name] = map[string]grade.Report{}
+ for s, preds := range bySlice {
+ out[name][s] = grade.Grade(preds, bins)
+ }
+ }
+ return out
+}
+
+func printReports(res evalResult) {
+ for _, name := range res.Scorers {
+ failed := 0
+ for _, row := range res.Rows {
+ if _, ok := row.Errors[name]; ok {
+ failed++
+ }
+ }
+ fmt.Printf("\n%s: %d batches, %d failed to score\n", name, len(res.Rows), failed)
+ fmt.Printf("%-34s %4s %5s %6s %6s %7s %6s %7s %7s\n", "slice", "n", "lands", "auc", "brier", "logloss", "ece", "p|lands", "p|fails")
+ reports := res.Reports[name]
+ slices := make([]string, 0, len(reports))
+ for s := range reports {
+ slices = append(slices, s)
+ }
+ sort.Slice(slices, func(i, j int) bool { return sliceOrder(slices[i]) < sliceOrder(slices[j]) })
+ for _, s := range slices {
+ r := reports[s]
+ auc := " -"
+ if r.AUCDefined {
+ auc = fmt.Sprintf("%6.3f", r.AUC)
+ }
+ fmt.Printf("%-34s %4d %5d %6s %6.3f %7.3f %6.3f %7s %7s\n", s, r.N, r.Lands, auc, r.Brier, r.LogLoss, r.ECE,
+ meanOrDash(r.MeanLands, r.Lands), meanOrDash(r.MeanFails, r.N-r.Lands))
+ }
+ }
+}
+
+// sliceOrder sorts "all" first, then tiers in size order, then everything
+// else alphabetically.
+func sliceOrder(s string) string {
+ if s == "all" {
+ return "0"
+ }
+ for i, t := range tiers {
+ if s == "tier "+string(t) {
+ return "1" + strconv.Itoa(i)
+ }
+ }
+ return "2" + s
+}
+
+func meanOrDash(mean float64, n int) string {
+ if n == 0 {
+ return "-"
+ }
+ return fmt.Sprintf("%.3f", mean)
+}
diff --git a/tool/jevcorpus/generate.go b/tool/jevcorpus/generate.go
new file mode 100644
index 000000000..dd4cd5cb1
--- /dev/null
+++ b/tool/jevcorpus/generate.go
@@ -0,0 +1,505 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "bufio"
+ "bytes"
+ "errors"
+ "flag"
+ "fmt"
+ "io/fs"
+ "math/rand"
+ "os"
+ "os/exec"
+ "path"
+ "path/filepath"
+ "sort"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/uber/submitqueue/submitqueue/extension/buildrunner/replay"
+ "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer/jev"
+ "github.com/uber/submitqueue/tool/jevcorpus/mutate"
+)
+
+// defaultVerify is the corpus's CI: the whole module builds and the touched
+// packages' tests pass. $PKGS holds the touched packages relative to the
+// module root. go vet is left out: the baseline does not check it, so a
+// package with existing vet findings would fail every safe change.
+const defaultVerify = "go build ./... && go test -count=1 $PKGS"
+
+// triesPerSpec bounds how many changes are built and verified for one planned
+// slot before the slot is given up; most retries are intent mismatches.
+const triesPerSpec = 6
+
+// verifyTailLines is how much failing output a manifest keeps for debugging.
+const verifyTailLines = 15
+
+// queuedAtStepMs spaces synthetic attempts a minute apart in the dataset.
+const queuedAtStepMs = 60_000
+
+type generateFlags struct {
+ repo, base, moduleDir, pkgs, out, sandbox, work, verify string
+ count int
+ unsafeRatio float64
+ seed int64
+ keepMismatch bool
+}
+
+func runGenerate(args []string) error {
+ var f generateFlags
+ flags := flag.NewFlagSet("generate", flag.ExitOnError)
+ flags.StringVar(&f.repo, "repo", "", "seed git repository: a path or clone URL (required)")
+ flags.StringVar(&f.base, "base", "HEAD", "seed commit every change is built on")
+ flags.StringVar(&f.moduleDir, "module-dir", ".", "Go module root inside the repository")
+ flags.StringVar(&f.pkgs, "pkgs", "./...", "space-separated package patterns, relative to the module root, to mutate")
+ flags.StringVar(&f.out, "out", "", "dataset to write; the manifest is written beside it (required)")
+ flags.StringVar(&f.sandbox, "sandbox", "", "bare sandbox repository to create; must not exist (required)")
+ flags.StringVar(&f.work, "work", "", "working clone to build in; defaults to a temporary directory")
+ flags.StringVar(&f.verify, "verify", defaultVerify, "shell command run at the module root; exit 0 means the change lands")
+ flags.IntVar(&f.count, "count", 40, "changes to generate")
+ flags.Float64Var(&f.unsafeRatio, "unsafe-ratio", 0.5, "share of changes designed to fail")
+ flags.Int64Var(&f.seed, "seed", 1, "random seed; one seed and base always plan the same corpus")
+ flags.BoolVar(&f.keepMismatch, "keep-mismatch", false, "keep changes whose observed outcome contradicts their intent")
+ if err := flags.Parse(args); err != nil {
+ return err
+ }
+ if f.repo == "" || f.out == "" || f.sandbox == "" {
+ return errors.New("-repo, -out, and -sandbox are required")
+ }
+ if f.count < 1 || f.unsafeRatio < 0 || f.unsafeRatio > 1 {
+ return errors.New("-count must be positive and -unsafe-ratio within [0, 1]")
+ }
+ sandbox, err := filepath.Abs(f.sandbox)
+ if err != nil {
+ return err
+ }
+ if _, err := os.Stat(sandbox); err == nil {
+ return fmt.Errorf("sandbox %s already exists; remove it first", sandbox)
+ }
+ return generate(f, sandbox)
+}
+
+func generate(f generateFlags, sandbox string) error {
+ work := f.work
+ if work == "" {
+ tmp, err := os.MkdirTemp("", "jevcorpus-work-")
+ if err != nil {
+ return err
+ }
+ defer os.RemoveAll(tmp)
+ work = filepath.Join(tmp, "repo")
+ }
+ if _, err := os.Stat(work); err != nil {
+ logf("cloning %s into %s", f.repo, work)
+ if _, err := git("", "clone", "--quiet", f.repo, work); err != nil {
+ return err
+ }
+ }
+ base, err := git(work, "rev-parse", f.base+"^{commit}")
+ if err != nil {
+ return err
+ }
+ if err := resetWorkTree(work, base); err != nil {
+ return err
+ }
+ moduleRoot := filepath.Join(work, f.moduleDir)
+
+ tree, err := loadTree(work, moduleRoot)
+ if err != nil {
+ return err
+ }
+ logf("base %s: %d Go files; running baseline tests over %s", base[:12], tree.Len(), f.pkgs)
+ cands, err := healthyCandidates(work, moduleRoot, f.pkgs, tree)
+ if err != nil {
+ return err
+ }
+ logf("%d candidate files in %d packages whose tests pass at the base", len(cands.files), len(cands.dirs))
+ if len(cands.files) == 0 {
+ return errors.New("no package under -pkgs has passing tests at the base")
+ }
+
+ r := rand.New(rand.NewSource(f.seed))
+ specs := planSpecs(f.count, f.unsafeRatio, r)
+ manifest := Manifest{Repo: f.repo, BaseSHA: base, Seed: f.seed, Verify: f.verify}
+ ds := replay.Dataset{Repo: f.repo, BaseBranch: "main", BaseSHA: base}
+ start := time.Now().UnixMilli()
+ mismatches := 0
+
+ for i, s := range specs {
+ branch := fmt.Sprintf("corpus/c-%04d", i+1)
+ var kept bool
+ unbuildable := 0
+ for try := 0; try < triesPerSpec && !kept; try++ {
+ b, ok := buildChange(tree, cands, s, r)
+ if !ok {
+ unbuildable++
+ continue
+ }
+ mc, title, err := commitChange(work, moduleRoot, base, branch, f.verify, tree, b, r)
+ if err != nil {
+ return fmt.Errorf("%s: %w", branch, err)
+ }
+ mc.Intent, mc.TargetTier = s.intent, s.tier
+ if mismatch := (s.intent == mutate.IntentSafe) != mc.Lands; mismatch {
+ mismatches++
+ if !f.keepMismatch {
+ logf("%s: %s %s change (%s) observed lands=%v; retrying", branch, s.intent, s.tier, strings.Join(b.mutators, ","), mc.Lands)
+ continue
+ }
+ }
+ kept = true
+ manifest.Changes = append(manifest.Changes, mc)
+ ds.Attempts = append(ds.Attempts, replay.Attempt{
+ PR: i + 1,
+ Title: title,
+ GroupSHA: mc.SHA,
+ ParentSHA: base,
+ Outcome: outcomeOf(mc.Lands),
+ DurationMs: mc.VerifyMs,
+ QueuedAtMs: start + int64(i)*queuedAtStepMs,
+ ReplaySHA: mc.SHA,
+ Branch: branch,
+ })
+ logf("%s: %s %s lands=%v (%d files, %d lines, %s)", branch, s.intent, mc.Tier, mc.Lands, mc.Files, mc.LinesChanged, mc.Category())
+ }
+ if !kept {
+ logf("%s: gave up on a %s %s change after %d tries; %d found too few mutation sites in the candidate files",
+ branch, s.intent, s.tier, triesPerSpec, unbuildable)
+ }
+ }
+ if len(ds.Attempts) == 0 {
+ return errors.New("no change was kept")
+ }
+
+ if err := createSandbox(work, sandbox, base, ds.Attempts); err != nil {
+ return err
+ }
+ if err := writeDataset(f.out, ds); err != nil {
+ return err
+ }
+ if err := writeJSON(manifestPathFor(f.out), manifest); err != nil {
+ return err
+ }
+ logf("kept %d of %d planned changes; %d built changes contradicted their intent", len(ds.Attempts), len(specs), mismatches)
+ printGrid(manifest)
+ return nil
+}
+
+// commitChange writes b onto a clean base, verifies it, and commits it to
+// branch whatever the outcome; the caller decides whether to keep it.
+func commitChange(work, moduleRoot, base, branch, verify string, tree mutate.Tree, b built, r *rand.Rand) (ManifestChange, string, error) {
+ if err := resetWorkTree(work, base); err != nil {
+ return ManifestChange{}, "", err
+ }
+ for p, src := range b.patch {
+ full := filepath.Join(work, filepath.FromSlash(p))
+ if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
+ return ManifestChange{}, "", err
+ }
+ if err := os.WriteFile(full, []byte(src), 0o644); err != nil {
+ return ManifestChange{}, "", err
+ }
+ }
+ if _, err := git(work, "add", "-A"); err != nil {
+ return ManifestChange{}, "", err
+ }
+ mc := ManifestChange{Branch: branch, Mutators: b.mutators}
+ if err := measureStaged(work, &mc, b.needle); err != nil {
+ return ManifestChange{}, "", err
+ }
+
+ pkgs := touchedPackages(work, moduleRoot, b.patch)
+ began := time.Now()
+ cmd := exec.Command("sh", "-c", verify)
+ cmd.Dir = moduleRoot
+ cmd.Env = append(os.Environ(), "PKGS="+strings.Join(pkgs, " "))
+ out, verr := cmd.CombinedOutput()
+ mc.VerifyMs = time.Since(began).Milliseconds()
+ mc.Lands = verr == nil
+ if !mc.Lands {
+ var exitErr *exec.ExitError
+ if !errors.As(verr, &exitErr) {
+ return ManifestChange{}, "", fmt.Errorf("run verify: %w", verr)
+ }
+ mc.VerifyTail = tailLines(string(out), verifyTailLines)
+ }
+
+ title := commitSubject(b.patch, r)
+ who := authors[r.Intn(len(authors))]
+ commit := exec.Command("git", "-c", "commit.gpgsign=false", "commit", "--quiet", "--no-verify", "-m", title)
+ commit.Dir = work
+ commit.Env = append(os.Environ(),
+ "GIT_AUTHOR_NAME="+who.name, "GIT_AUTHOR_EMAIL="+who.email,
+ "GIT_COMMITTER_NAME=jevcorpus", "GIT_COMMITTER_EMAIL=jevcorpus@submitqueue.invalid")
+ if _, err := run(commit); err != nil {
+ return ManifestChange{}, "", err
+ }
+ sha, err := git(work, "rev-parse", "HEAD")
+ if err != nil {
+ return ManifestChange{}, "", err
+ }
+ mc.SHA = sha
+ if _, err := git(work, "branch", "--force", branch, sha); err != nil {
+ return ManifestChange{}, "", err
+ }
+ return mc, title, nil
+}
+
+// measureStaged records the staged change's size and where its needle sits
+// in the diff.
+func measureStaged(work string, mc *ManifestChange, n needle) error {
+ numstat, err := git(work, "diff", "--cached", "--numstat")
+ if err != nil {
+ return err
+ }
+ for _, line := range strings.Split(numstat, "\n") {
+ fields := strings.Fields(line)
+ if len(fields) < 3 {
+ continue
+ }
+ mc.Files++
+ added, _ := strconv.Atoi(fields[0])
+ deleted, _ := strconv.Atoi(fields[1])
+ mc.LinesChanged += added + deleted
+ }
+ patch, err := gitRaw(work, "diff", "--cached")
+ if err != nil {
+ return err
+ }
+ mc.DiffBytes = len(patch)
+ mc.Tier = tierFor(mc.LinesChanged, mc.DiffBytes, mc.Files)
+ if n.mutator != "" {
+ mc.Needle = ManifestNeedle{Mutator: n.mutator, File: n.file, Position: n.position}
+ if at := strings.Index(patch, "diff --git a/"+n.file+" "); at >= 0 {
+ mc.Needle.OffsetBytes = at
+ mc.Needle.BeyondDiffBudget = at >= jev.DefaultMaxDiffBytes
+ }
+ }
+ return nil
+}
+
+// touchedPackages returns the package directories p edits, relative to the
+// module root in the "./dir" form go commands take.
+func touchedPackages(work, moduleRoot string, p mutate.Patch) []string {
+ seen := map[string]bool{}
+ var out []string
+ for _, file := range p.Paths() {
+ rel, err := filepath.Rel(moduleRoot, filepath.Join(work, filepath.FromSlash(path.Dir(file))))
+ if err != nil || strings.HasPrefix(rel, "..") {
+ continue
+ }
+ pkg := "./" + filepath.ToSlash(rel)
+ if !seen[pkg] {
+ seen[pkg] = true
+ out = append(out, pkg)
+ }
+ }
+ return out
+}
+
+func resetWorkTree(work, base string) error {
+ if _, err := git(work, "checkout", "--quiet", "--force", "--detach", base); err != nil {
+ return err
+ }
+ _, err := git(work, "clean", "-fdq")
+ return err
+}
+
+// loadTree reads every Go file under moduleRoot, keyed by its path relative
+// to the repository root. Vendored code, testdata, and hidden or underscore
+// directories are skipped, as the go command skips them.
+func loadTree(work, moduleRoot string) (mutate.Tree, error) {
+ files := map[string]string{}
+ err := filepath.WalkDir(moduleRoot, func(p string, d fs.DirEntry, err error) error {
+ if err != nil {
+ return err
+ }
+ name := d.Name()
+ if d.IsDir() {
+ if p != moduleRoot && (name == "vendor" || name == "testdata" || strings.HasPrefix(name, ".") || strings.HasPrefix(name, "_")) {
+ return filepath.SkipDir
+ }
+ return nil
+ }
+ if !strings.HasSuffix(name, ".go") {
+ return nil
+ }
+ src, err := os.ReadFile(p)
+ if err != nil {
+ return err
+ }
+ rel, err := filepath.Rel(work, p)
+ if err != nil {
+ return err
+ }
+ files[filepath.ToSlash(rel)] = string(src)
+ return nil
+ })
+ return mutate.NewTree(files), err
+}
+
+// healthyCandidates runs the base's tests once and keeps the packages that
+// pass, so every failure in the corpus is one the mutation caused.
+func healthyCandidates(work, moduleRoot, pattern string, tree mutate.Tree) (candidates, error) {
+ if _, err := goCmd(moduleRoot, "build", "./..."); err != nil {
+ return candidates{}, fmt.Errorf("the base does not build: %w", err)
+ }
+ patterns := strings.Fields(pattern)
+ listing, err := goCmd(moduleRoot, append([]string{"list", "-f", "{{.ImportPath}}\t{{.Dir}}"}, patterns...)...)
+ if err != nil {
+ return candidates{}, err
+ }
+ dirOf := map[string]string{}
+ for _, line := range strings.Split(listing, "\n") {
+ if imp, dir, ok := strings.Cut(line, "\t"); ok {
+ dirOf[imp] = dir
+ }
+ }
+ // A failing package exits non-zero but its "ok" siblings are still
+ // reported, which is all this reads.
+ out, _ := goCmd(moduleRoot, append([]string{"test", "-count=1"}, patterns...)...)
+ var c candidates
+ sc := bufio.NewScanner(strings.NewReader(out))
+ for sc.Scan() {
+ fields := strings.Fields(sc.Text())
+ if len(fields) < 2 || fields[0] != "ok" {
+ continue
+ }
+ dir, ok := dirOf[fields[1]]
+ if !ok {
+ continue
+ }
+ rel, err := filepath.Rel(work, dir)
+ if err != nil {
+ continue
+ }
+ rel = filepath.ToSlash(rel)
+ var files []string
+ for _, p := range tree.GoFiles(rel) {
+ if src, _ := tree.File(p); !mutate.IsTestFile(p) && !isGenerated(src) {
+ files = append(files, p)
+ }
+ }
+ if len(files) > 0 {
+ c.dirs = append(c.dirs, rel)
+ c.files = append(c.files, files...)
+ }
+ }
+ sort.Strings(c.dirs)
+ sort.Strings(c.files)
+ return c, nil
+}
+
+// isGenerated follows the Go convention for generated files: a "Code
+// generated ... DO NOT EDIT." line before the package clause.
+func isGenerated(src string) bool {
+ head, _, _ := strings.Cut(src, "\npackage ")
+ return strings.Contains(head, "Code generated") && strings.Contains(head, "DO NOT EDIT")
+}
+
+func createSandbox(work, sandbox, base string, attempts []replay.Attempt) error {
+ if err := os.MkdirAll(filepath.Dir(sandbox), 0o755); err != nil {
+ return err
+ }
+ if _, err := git("", "init", "--bare", "--quiet", "-b", "main", sandbox); err != nil {
+ return err
+ }
+ refspecs := []string{base + ":refs/heads/main"}
+ for _, a := range attempts {
+ refspecs = append(refspecs, a.ReplaySHA+":refs/heads/"+a.Branch)
+ }
+ _, err := git(work, append([]string{"push", "--quiet", sandbox}, refspecs...)...)
+ return err
+}
+
+func printGrid(m Manifest) {
+ type cell struct{ lands, fails int }
+ grid := map[Tier]*cell{}
+ for _, t := range tiers {
+ grid[t] = &cell{}
+ }
+ for _, c := range m.Changes {
+ if c.Lands {
+ grid[c.Tier].lands++
+ } else {
+ grid[c.Tier].fails++
+ }
+ }
+ fmt.Println("tier lands fails")
+ for _, t := range tiers {
+ fmt.Printf("%-4s %5d %5d\n", t, grid[t].lands, grid[t].fails)
+ }
+}
+
+func outcomeOf(lands bool) replay.Outcome {
+ if lands {
+ return replay.OutcomePass
+ }
+ return replay.OutcomeFail
+}
+
+func tailLines(s string, n int) string {
+ lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
+ if len(lines) > n {
+ lines = lines[len(lines)-n:]
+ }
+ return strings.Join(lines, "\n")
+}
+
+func logf(format string, args ...any) {
+ fmt.Fprintf(os.Stderr, format+"\n", args...)
+}
+
+func git(dir string, args ...string) (string, error) {
+ out, err := gitRaw(dir, args...)
+ return strings.TrimSpace(out), err
+}
+
+func gitRaw(dir string, args ...string) (string, error) {
+ cmd := exec.Command("git", args...)
+ cmd.Dir = dir
+ return runRaw(cmd)
+}
+
+func goCmd(dir string, args ...string) (string, error) {
+ cmd := exec.Command("go", args...)
+ cmd.Dir = dir
+ var out bytes.Buffer
+ cmd.Stdout = &out
+ cmd.Stderr = &out
+ err := cmd.Run()
+ if err != nil {
+ return out.String(), fmt.Errorf("go %s: %w: %s", strings.Join(args, " "), err, tailLines(out.String(), verifyTailLines))
+ }
+ return out.String(), nil
+}
+
+func run(cmd *exec.Cmd) (string, error) {
+ out, err := runRaw(cmd)
+ return strings.TrimSpace(out), err
+}
+
+func runRaw(cmd *exec.Cmd) (string, error) {
+ var stderr bytes.Buffer
+ cmd.Stderr = &stderr
+ out, err := cmd.Output()
+ if err != nil {
+ return "", fmt.Errorf("%s: %w: %s", strings.Join(cmd.Args, " "), err, strings.TrimSpace(stderr.String()))
+ }
+ return string(out), nil
+}
diff --git a/tool/jevcorpus/grade/BUILD.bazel b/tool/jevcorpus/grade/BUILD.bazel
new file mode 100644
index 000000000..d66d9d764
--- /dev/null
+++ b/tool/jevcorpus/grade/BUILD.bazel
@@ -0,0 +1,15 @@
+load("@rules_go//go:def.bzl", "go_library", "go_test")
+
+go_library(
+ name = "go_default_library",
+ srcs = ["grade.go"],
+ importpath = "github.com/uber/submitqueue/tool/jevcorpus/grade",
+ visibility = ["//visibility:public"],
+)
+
+go_test(
+ name = "go_default_test",
+ srcs = ["grade_test.go"],
+ embed = [":go_default_library"],
+ deps = ["@com_github_stretchr_testify//assert:go_default_library"],
+)
diff --git a/tool/jevcorpus/grade/grade.go b/tool/jevcorpus/grade/grade.go
new file mode 100644
index 000000000..212aa0898
--- /dev/null
+++ b/tool/jevcorpus/grade/grade.go
@@ -0,0 +1,226 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package grade scores probability predictions against binary outcomes.
+//
+// AUC measures ranking alone: the chance a random landing batch is priced above
+// a random failing one. Brier score and log loss measure the probabilities
+// themselves, log loss punishing confident mistakes far harder. Calibration
+// bins compare mean prediction with the observed landing rate, and ECE
+// summarises their gaps. A scorer can rank perfectly and be badly calibrated,
+// or be perfectly calibrated and rank no better than chance, so a report
+// carries all of them.
+package grade
+
+import (
+ "math"
+ "sort"
+)
+
+// logLossEpsilon clips predictions away from 0 and 1 so one certain mistake
+// is a large finite penalty rather than infinity.
+const logLossEpsilon = 1e-6
+
+// Prediction is one scored outcome.
+type Prediction struct {
+ // P is the predicted probability that the batch lands, in [0, 1].
+ P float64 `json:"p"`
+ // Lands is the observed outcome.
+ Lands bool `json:"lands"`
+}
+
+// Bin is one calibration bucket over [Lo, Hi).
+type Bin struct {
+ // Lo is the inclusive lower bound of predictions in the bin.
+ Lo float64 `json:"lo"`
+ // Hi is the upper bound: exclusive, except 1 in the last bin.
+ Hi float64 `json:"hi"`
+ // Count is the number of predictions in the bin.
+ Count int `json:"count"`
+ // MeanP is the mean prediction in the bin; zero when Count is zero.
+ MeanP float64 `json:"mean_p"`
+ // LandRate is the observed fraction that landed; zero when Count is zero.
+ LandRate float64 `json:"land_rate"`
+}
+
+// Report grades one set of predictions.
+type Report struct {
+ // N is the number of predictions.
+ N int `json:"n"`
+ // Lands is how many of them landed.
+ Lands int `json:"lands"`
+ // AUC is the ROC area; meaningful only when AUCDefined.
+ AUC float64 `json:"auc"`
+ // AUCDefined is false when every outcome is the same, which leaves
+ // ranking unmeasurable.
+ AUCDefined bool `json:"auc_defined"`
+ // Brier is the mean squared error of the predictions; 0 is perfect and
+ // 0.25 is a constant 0.5.
+ Brier float64 `json:"brier"`
+ // LogLoss is the mean negative log-likelihood of the outcomes.
+ LogLoss float64 `json:"log_loss"`
+ // ECE is the count-weighted mean gap between MeanP and LandRate across
+ // Bins.
+ ECE float64 `json:"ece"`
+ // MeanLands and MeanFails are the mean prediction for each outcome; zero
+ // when it has none. Their gap is the separation a one-outcome slice, where
+ // AUC is undefined, can still show.
+ MeanLands float64 `json:"mean_lands"`
+ MeanFails float64 `json:"mean_fails"`
+ // Bins is the calibration table.
+ Bins []Bin `json:"bins"`
+}
+
+// Grade returns the report for preds with the given number of equal-width
+// calibration bins. Fewer than one bin means one.
+func Grade(preds []Prediction, bins int) Report {
+ r := Report{N: len(preds)}
+ if len(preds) == 0 {
+ return r
+ }
+ var sumLands, sumFails float64
+ for _, p := range preds {
+ if p.Lands {
+ r.Lands++
+ sumLands += p.P
+ } else {
+ sumFails += p.P
+ }
+ }
+ if r.Lands > 0 {
+ r.MeanLands = sumLands / float64(r.Lands)
+ }
+ if fails := r.N - r.Lands; fails > 0 {
+ r.MeanFails = sumFails / float64(fails)
+ }
+ r.AUC, r.AUCDefined = AUC(preds)
+ r.Brier = Brier(preds)
+ r.LogLoss = LogLoss(preds)
+ r.Bins = Calibration(preds, bins)
+ r.ECE = ECE(r.Bins)
+ return r
+}
+
+// AUC returns the probability that a random landing prediction is priced above
+// a random failing one, counting ties as half. It is false when preds lack
+// either outcome.
+func AUC(preds []Prediction) (float64, bool) {
+ sorted := append([]Prediction(nil), preds...)
+ sort.Slice(sorted, func(i, j int) bool { return sorted[i].P < sorted[j].P })
+ var pos, neg int
+ var rankSum float64
+ for i := 0; i < len(sorted); {
+ j := i
+ for j < len(sorted) && sorted[j].P == sorted[i].P {
+ j++
+ }
+ // Tied predictions share the mean of the ranks they span (1-based).
+ meanRank := float64(i+1+j) / 2
+ for k := i; k < j; k++ {
+ if sorted[k].Lands {
+ pos++
+ rankSum += meanRank
+ } else {
+ neg++
+ }
+ }
+ i = j
+ }
+ if pos == 0 || neg == 0 {
+ return 0, false
+ }
+ return (rankSum - float64(pos)*float64(pos+1)/2) / (float64(pos) * float64(neg)), true
+}
+
+// Brier returns the mean of (p - outcome)².
+func Brier(preds []Prediction) float64 {
+ if len(preds) == 0 {
+ return 0
+ }
+ var sum float64
+ for _, p := range preds {
+ d := p.P - outcome(p)
+ sum += d * d
+ }
+ return sum / float64(len(preds))
+}
+
+// LogLoss returns the mean of -[y·ln(p) + (1-y)·ln(1-p)], with p clipped to
+// [ε, 1-ε].
+func LogLoss(preds []Prediction) float64 {
+ if len(preds) == 0 {
+ return 0
+ }
+ var sum float64
+ for _, p := range preds {
+ q := math.Min(math.Max(p.P, logLossEpsilon), 1-logLossEpsilon)
+ if p.Lands {
+ sum -= math.Log(q)
+ } else {
+ sum -= math.Log(1 - q)
+ }
+ }
+ return sum / float64(len(preds))
+}
+
+// Calibration buckets preds into n equal-width bins over [0, 1].
+func Calibration(preds []Prediction, n int) []Bin {
+ if n < 1 {
+ n = 1
+ }
+ bins := make([]Bin, n)
+ sumP := make([]float64, n)
+ landed := make([]int, n)
+ for i := range bins {
+ bins[i].Lo = float64(i) / float64(n)
+ bins[i].Hi = float64(i+1) / float64(n)
+ }
+ for _, p := range preds {
+ i := int(p.P * float64(n))
+ i = min(max(i, 0), n-1)
+ bins[i].Count++
+ sumP[i] += p.P
+ if p.Lands {
+ landed[i]++
+ }
+ }
+ for i := range bins {
+ if bins[i].Count > 0 {
+ bins[i].MeanP = sumP[i] / float64(bins[i].Count)
+ bins[i].LandRate = float64(landed[i]) / float64(bins[i].Count)
+ }
+ }
+ return bins
+}
+
+// ECE returns the count-weighted mean |MeanP - LandRate| across bins.
+func ECE(bins []Bin) float64 {
+ total := 0
+ var sum float64
+ for _, b := range bins {
+ total += b.Count
+ sum += float64(b.Count) * math.Abs(b.MeanP-b.LandRate)
+ }
+ if total == 0 {
+ return 0
+ }
+ return sum / float64(total)
+}
+
+func outcome(p Prediction) float64 {
+ if p.Lands {
+ return 1
+ }
+ return 0
+}
diff --git a/tool/jevcorpus/grade/grade_test.go b/tool/jevcorpus/grade/grade_test.go
new file mode 100644
index 000000000..c15ddc31a
--- /dev/null
+++ b/tool/jevcorpus/grade/grade_test.go
@@ -0,0 +1,92 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package grade
+
+import (
+ "math"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+)
+
+func preds(pairs ...any) []Prediction {
+ var out []Prediction
+ for i := 0; i < len(pairs); i += 2 {
+ out = append(out, Prediction{P: pairs[i].(float64), Lands: pairs[i+1].(bool)})
+ }
+ return out
+}
+
+func TestAUC(t *testing.T) {
+ tests := []struct {
+ name string
+ preds []Prediction
+ want float64
+ wantDefined bool
+ }{
+ {name: "perfect ranking", preds: preds(0.9, true, 0.8, true, 0.2, false, 0.1, false), want: 1, wantDefined: true},
+ {name: "inverted ranking", preds: preds(0.1, true, 0.9, false), want: 0, wantDefined: true},
+ {name: "constant prediction is chance", preds: preds(0.5, true, 0.5, false, 0.5, true, 0.5, false), want: 0.5, wantDefined: true},
+ {name: "one swapped pair of four", preds: preds(0.9, true, 0.4, true, 0.6, false, 0.1, false), want: 0.75, wantDefined: true},
+ {name: "ranking ignores scale", preds: preds(0.52, true, 0.51, false), want: 1, wantDefined: true},
+ {name: "single outcome is undefined", preds: preds(0.9, true, 0.1, true)},
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ got, defined := AUC(tt.preds)
+ assert.Equal(t, tt.wantDefined, defined)
+ assert.InDelta(t, tt.want, got, 1e-9)
+ })
+ }
+}
+
+func TestBrier(t *testing.T) {
+ assert.InDelta(t, 0, Brier(preds(1.0, true, 0.0, false)), 1e-9)
+ assert.InDelta(t, 0.25, Brier(preds(0.5, true, 0.5, false)), 1e-9)
+ assert.InDelta(t, 0.81, Brier(preds(0.9, false)), 1e-9)
+}
+
+func TestLogLoss(t *testing.T) {
+ assert.InDelta(t, -math.Log(0.6), LogLoss(preds(0.4, false)), 1e-9)
+ assert.Greater(t, LogLoss(preds(0.99, false)), LogLoss(preds(0.6, false)), "confident mistakes cost more")
+ assert.False(t, math.IsInf(LogLoss(preds(1.0, false)), 0), "a certain mistake is clipped, not infinite")
+}
+
+func TestCalibration(t *testing.T) {
+ bins := Calibration(preds(0.05, false, 0.75, true, 0.75, false, 1.0, true), 4)
+ assert.Len(t, bins, 4)
+ assert.Equal(t, 1, bins[0].Count)
+ assert.Equal(t, 0, bins[1].Count)
+ assert.Equal(t, 3, bins[3].Count, "1.0 joins the last bin rather than overflowing it")
+ assert.InDelta(t, 2.0/3, bins[3].LandRate, 1e-9)
+}
+
+func TestECE(t *testing.T) {
+ calibrated := Calibration(preds(0.75, true, 0.75, true, 0.75, true, 0.75, false), 10)
+ assert.InDelta(t, 0, ECE(calibrated), 1e-9)
+ overconfident := Calibration(preds(0.95, true, 0.95, false), 10)
+ assert.InDelta(t, 0.45, ECE(overconfident), 1e-9)
+}
+
+func TestGrade(t *testing.T) {
+ r := Grade(preds(0.9, true, 0.2, false), 10)
+ assert.Equal(t, 2, r.N)
+ assert.Equal(t, 1, r.Lands)
+ assert.True(t, r.AUCDefined)
+ assert.Len(t, r.Bins, 10)
+ assert.InDelta(t, 0.9, r.MeanLands, 1e-9)
+ assert.InDelta(t, 0.2, r.MeanFails, 1e-9)
+ assert.Equal(t, Report{}, Grade(nil, 10))
+}
diff --git a/tool/jevcorpus/main.go b/tool/jevcorpus/main.go
new file mode 100644
index 000000000..1207634b6
--- /dev/null
+++ b/tool/jevcorpus/main.go
@@ -0,0 +1,52 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Command jevcorpus builds a synthetic, build-verified corpus of safe and unsafe
+// Go changes and grades outcome scorers against it. See
+// doc/rfc/submitqueue/synthetic-corpus.md.
+//
+// jevcorpus generate -repo /path/to/seed -count 80 -out corpus.json -sandbox /tmp/c/sandbox.git
+// jevcorpus eval -dataset corpus.json -sandbox /tmp/c/sandbox.git -scorers size,jev
+//
+// The dataset is in the replay format, so `replaybench uris` lists its change
+// URIs and the replay build runner and oracle scorer read it unchanged.
+package main
+
+import (
+ "fmt"
+ "os"
+)
+
+const usage = `usage: jevcorpus [flags]`
+
+func main() {
+ if len(os.Args) < 2 {
+ fmt.Fprintln(os.Stderr, usage)
+ os.Exit(2)
+ }
+ var err error
+ switch os.Args[1] {
+ case "generate":
+ err = runGenerate(os.Args[2:])
+ case "eval":
+ err = runEval(os.Args[2:])
+ default:
+ fmt.Fprintln(os.Stderr, usage)
+ os.Exit(2)
+ }
+ if err != nil {
+ fmt.Fprintln(os.Stderr, "Error:", err)
+ os.Exit(1)
+ }
+}
diff --git a/tool/jevcorpus/manifest.go b/tool/jevcorpus/manifest.go
new file mode 100644
index 000000000..e2f906ce2
--- /dev/null
+++ b/tool/jevcorpus/manifest.go
@@ -0,0 +1,130 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "encoding/json"
+ "fmt"
+ "os"
+ "path/filepath"
+ "strings"
+
+ "github.com/uber/submitqueue/submitqueue/extension/buildrunner/replay"
+ "github.com/uber/submitqueue/tool/jevcorpus/mutate"
+)
+
+// Manifest is the corpus's answer key: what each change was built to do and
+// what it did. It sits beside the replay dataset rather than in it, so nothing
+// a scorer reads through the queue — branch, commit, dataset title — carries
+// the label or the mutation that produced it.
+type Manifest struct {
+ // Repo is the seed repository the corpus was generated from.
+ Repo string `json:"repo"`
+ // BaseSHA is the seed commit every change is built on.
+ BaseSHA string `json:"base_sha"`
+ // Seed is the generator's random seed.
+ Seed int64 `json:"seed"`
+ // Verify is the shell command whose exit status is the label.
+ Verify string `json:"verify"`
+ // Changes are in dataset order.
+ Changes []ManifestChange `json:"changes"`
+}
+
+// ManifestChange is one generated change.
+type ManifestChange struct {
+ // Branch is the sandbox branch holding the change.
+ Branch string `json:"branch"`
+ // SHA is the change's commit on the base.
+ SHA string `json:"sha"`
+ // Intent is what the change was designed to do.
+ Intent mutate.Intent `json:"intent"`
+ // Lands is the observed label: Verify passed.
+ Lands bool `json:"lands"`
+ // TargetTier is the size the planner aimed for.
+ TargetTier Tier `json:"target_tier"`
+ // Tier is the size measured from git's diff.
+ Tier Tier `json:"tier"`
+ // Mutators are the mutators applied, in order.
+ Mutators []string `json:"mutators"`
+ // Files is the number of files changed.
+ Files int `json:"files"`
+ // LinesChanged is lines added plus lines deleted.
+ LinesChanged int `json:"lines_changed"`
+ // DiffBytes is the size of the change's unified diff.
+ DiffBytes int `json:"diff_bytes"`
+ // Needle is the unsafe edit in a large change; empty Mutator when none.
+ Needle ManifestNeedle `json:"needle,omitzero"`
+ // VerifyMs is how long Verify took, in milliseconds.
+ VerifyMs int64 `json:"verify_ms"`
+ // VerifyTail is the end of Verify's output when it failed.
+ VerifyTail string `json:"verify_tail,omitempty"`
+}
+
+// ManifestNeedle locates the unsafe edit in a large change.
+type ManifestNeedle struct {
+ // Mutator is the unsafe mutator applied.
+ Mutator string `json:"mutator,omitempty"`
+ // File is the file it edited.
+ File string `json:"file,omitempty"`
+ // Position is where File sits among the change's files.
+ Position NeedlePosition `json:"position,omitempty"`
+ // OffsetBytes is where File's section starts in the change's diff.
+ OffsetBytes int `json:"offset_bytes,omitempty"`
+ // BeyondDiffBudget is true when OffsetBytes is past the jev scorer's
+ // default diff budget, so a single-change batch never shows the model the
+ // edit.
+ BeyondDiffBudget bool `json:"beyond_diff_budget,omitempty"`
+}
+
+// Category names the kind of change for slicing results: the mutator of a
+// single-edit change, or "sweep" with its needle for a large one.
+func (c ManifestChange) Category() string {
+ if c.Needle.Mutator != "" {
+ return "sweep+" + c.Needle.Mutator
+ }
+ if c.TargetTier != TierS {
+ return "sweep"
+ }
+ if len(c.Mutators) == 0 {
+ return "unknown"
+ }
+ return c.Mutators[0]
+}
+
+func manifestPathFor(datasetPath string) string {
+ return strings.TrimSuffix(datasetPath, filepath.Ext(datasetPath)) + ".manifest.json"
+}
+
+func loadManifest(path string) (Manifest, error) {
+ raw, err := os.ReadFile(path)
+ if err != nil {
+ return Manifest{}, fmt.Errorf("read manifest: %w", err)
+ }
+ var m Manifest
+ if err := json.Unmarshal(raw, &m); err != nil {
+ return Manifest{}, fmt.Errorf("parse manifest %s: %w", path, err)
+ }
+ return m, nil
+}
+
+func writeJSON(path string, v any) error {
+ out, err := json.MarshalIndent(v, "", " ")
+ if err != nil {
+ return err
+ }
+ return os.WriteFile(path, append(out, '\n'), 0o644)
+}
+
+func writeDataset(path string, ds replay.Dataset) error { return writeJSON(path, ds) }
diff --git a/tool/jevcorpus/mutate/BUILD.bazel b/tool/jevcorpus/mutate/BUILD.bazel
new file mode 100644
index 000000000..7b717bf2c
--- /dev/null
+++ b/tool/jevcorpus/mutate/BUILD.bazel
@@ -0,0 +1,24 @@
+load("@rules_go//go:def.bzl", "go_library", "go_test")
+
+go_library(
+ name = "go_default_library",
+ srcs = [
+ "mutator.go",
+ "safe.go",
+ "source.go",
+ "tree.go",
+ "unsafe.go",
+ ],
+ importpath = "github.com/uber/submitqueue/tool/jevcorpus/mutate",
+ visibility = ["//visibility:public"],
+)
+
+go_test(
+ name = "go_default_test",
+ srcs = ["mutate_test.go"],
+ embed = [":go_default_library"],
+ deps = [
+ "@com_github_stretchr_testify//assert:go_default_library",
+ "@com_github_stretchr_testify//require:go_default_library",
+ ],
+)
diff --git a/tool/jevcorpus/mutate/mutate_test.go b/tool/jevcorpus/mutate/mutate_test.go
new file mode 100644
index 000000000..0d86d023c
--- /dev/null
+++ b/tool/jevcorpus/mutate/mutate_test.go
@@ -0,0 +1,239 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package mutate
+
+import (
+ "go/ast"
+ "go/parser"
+ "go/token"
+ "go/types"
+ "math/rand"
+ "sort"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+const fixtureA = `package sample
+
+// Total sums the values no greater than limit.
+func Total(values []int, limit int) int {
+ sum := 0
+ for i := 0; i < len(values); i++ {
+ if values[i] > limit {
+ continue
+ }
+ sum += values[i]
+ }
+ return sum
+}
+
+func helper(n int) int {
+ if n == 3 {
+ return 0
+ }
+ return n * 2
+}
+
+func unused(x int) int { return x + 1 }
+
+func Double(n int) int { return helper(n) }
+
+func check(err error) error {
+ if err != nil {
+ return err
+ }
+ return nil
+}
+
+type counter struct{ hits int }
+
+func (c *counter) bump(by int) {
+ total := c.hits + by
+ c.hits = total
+}
+`
+
+const fixtureB = `package sample
+
+func Use() int {
+ res := helper(4) + Total([]int{1, 2}, 5)
+ for _, v := range []int{1, 2} {
+ res += v
+ }
+ return res
+}
+`
+
+func fixtureTree() Tree {
+ return NewTree(map[string]string{"pkg/a.go": fixtureA, "pkg/b.go": fixtureB})
+}
+
+// typeCheck type-checks the non-test files in dir. The fixture imports
+// nothing, so no importer is needed.
+func typeCheck(t Tree, dir string) error {
+ fset := token.NewFileSet()
+ var files []*ast.File
+ for _, p := range t.GoFiles(dir) {
+ if IsTestFile(p) {
+ continue
+ }
+ src, _ := t.File(p)
+ f, err := parser.ParseFile(fset, p, src, 0)
+ if err != nil {
+ return err
+ }
+ files = append(files, f)
+ }
+ _, err := (&types.Config{}).Check(dir, fset, files, nil)
+ return err
+}
+
+func TestFixtureTypeChecks(t *testing.T) {
+ require.NoError(t, typeCheck(fixtureTree(), "pkg"))
+}
+
+func TestMutators(t *testing.T) {
+ tests := []struct {
+ mutator Mutator
+ target string
+ wantTypeChecks bool
+ }{
+ {mutator: RenameFunc, target: "pkg", wantTypeChecks: true},
+ {mutator: RenameLocals, target: "pkg/a.go", wantTypeChecks: true},
+ {mutator: RenameLocals, target: "pkg/b.go", wantTypeChecks: true},
+ {mutator: DocumentFuncs, target: "pkg/a.go", wantTypeChecks: true},
+ {mutator: DeleteDeadFunc, target: "pkg", wantTypeChecks: true},
+ {mutator: AddHelper, target: "pkg", wantTypeChecks: true},
+ {mutator: FlipComparison, target: "pkg/a.go", wantTypeChecks: true},
+ {mutator: OffByOne, target: "pkg/a.go", wantTypeChecks: true},
+ {mutator: DropErrorCheck, target: "pkg/a.go", wantTypeChecks: true},
+ {mutator: DeleteCalledFunc, target: "pkg", wantTypeChecks: false},
+ {mutator: AddParam, target: "pkg", wantTypeChecks: false},
+ }
+ for _, tt := range tests {
+ t.Run(tt.mutator.Name+" "+tt.target, func(t *testing.T) {
+ for seed := int64(0); seed < 20; seed++ {
+ tree := fixtureTree()
+ patch, ok := tt.mutator.Apply(tree, tt.target, rand.New(rand.NewSource(seed)))
+ require.True(t, ok, "seed %d found no site", seed)
+ for p, src := range patch {
+ old, _ := tree.File(p)
+ assert.NotEqual(t, old, src, "seed %d left %s unchanged", seed, p)
+ }
+ err := typeCheck(tree.With(patch), "pkg")
+ if tt.wantTypeChecks {
+ assert.NoError(t, err, "seed %d", seed)
+ } else {
+ assert.Error(t, err, "seed %d", seed)
+ }
+ }
+ })
+ }
+}
+
+func TestMutatorsAreDeterministic(t *testing.T) {
+ for _, m := range All {
+ t.Run(m.Name, func(t *testing.T) {
+ target := "pkg/a.go"
+ if m.Scope == ScopePackage {
+ target = "pkg"
+ }
+ first, ok1 := m.Apply(fixtureTree(), target, rand.New(rand.NewSource(7)))
+ second, ok2 := m.Apply(fixtureTree(), target, rand.New(rand.NewSource(7)))
+ assert.Equal(t, ok1, ok2)
+ assert.Equal(t, first, second)
+ })
+ }
+}
+
+func TestMutatorsWithoutSites(t *testing.T) {
+ bare := NewTree(map[string]string{"pkg/a.go": "package sample\n\n// V is a value.\nvar V = 1\n"})
+ for _, m := range All {
+ if m.Name == AddHelper.Name {
+ continue
+ }
+ t.Run(m.Name, func(t *testing.T) {
+ target := "pkg/a.go"
+ if m.Scope == ScopePackage {
+ target = "pkg"
+ }
+ _, ok := m.Apply(bare, target, rand.New(rand.NewSource(1)))
+ assert.False(t, ok)
+ })
+ }
+}
+
+func TestFileMutatorsSkipTestFiles(t *testing.T) {
+ tree := NewTree(map[string]string{"pkg/a_test.go": fixtureA})
+ for _, m := range ByIntent(IntentUnsafe, ScopeFile) {
+ t.Run(m.Name, func(t *testing.T) {
+ _, ok := m.Apply(tree, "pkg/a_test.go", rand.New(rand.NewSource(1)))
+ assert.False(t, ok)
+ })
+ }
+}
+
+func TestRenameFuncRenamesEveryUse(t *testing.T) {
+ tree := NewTree(map[string]string{
+ "pkg/a.go": "package sample\n\n// helper doubles n.\nfunc helper(n int) int { return n * 2 }\n",
+ "pkg/b.go": "package sample\n\nfunc Use() int { return helper(4) }\n",
+ })
+ patch, ok := RenameFunc.Apply(tree, "pkg", rand.New(rand.NewSource(3)))
+ require.True(t, ok)
+ assert.Equal(t, []string{"pkg/a.go", "pkg/b.go"}, patch.Paths())
+ assert.NotContains(t, patch["pkg/a.go"], "// helper doubles", "the doc comment follows the rename")
+ assert.NoError(t, typeCheck(tree.With(patch), "pkg"))
+}
+
+func TestAddParamHandlesTrailingComma(t *testing.T) {
+ src := "package sample\n\nfunc f(\n\ta int,\n) int {\n\treturn a\n}\n\nfunc g() int { return f(1) }\n"
+ tree := NewTree(map[string]string{"pkg/a.go": src})
+ patch, ok := AddParam.Apply(tree, "pkg", rand.New(rand.NewSource(1)))
+ require.True(t, ok)
+ assert.Error(t, typeCheck(tree.With(patch), "pkg"))
+ _, parses := gofmt(patch["pkg/a.go"])
+ assert.True(t, parses)
+}
+
+func TestByIntent(t *testing.T) {
+ var names []string
+ for _, m := range ByIntent(IntentUnsafe, ScopeFile) {
+ names = append(names, m.Name)
+ }
+ sort.Strings(names)
+ assert.Equal(t, []string{"drop-error-check", "flip-comparison", "off-by-one"}, names)
+}
+
+func TestSplitWords(t *testing.T) {
+ tests := map[string][]string{
+ "parseHTTPHeader": {"parse", "http", "header"},
+ "newClient": {"new", "client"},
+ "ID": {"id"},
+ "snake_case": {"snake", "case"},
+ }
+ for in, want := range tests {
+ t.Run(in, func(t *testing.T) {
+ assert.Equal(t, want, splitWords(in))
+ })
+ }
+}
+
+func TestDocSentence(t *testing.T) {
+ assert.Equal(t, "// parseHeader parses the header.", docSentence("parseHeader"))
+ assert.Equal(t, "// isValid reports whether valid.", docSentence("isValid"))
+ assert.Equal(t, "// Total implements the total step.", docSentence("Total"))
+}
diff --git a/tool/jevcorpus/mutate/mutator.go b/tool/jevcorpus/mutate/mutator.go
new file mode 100644
index 000000000..3a9d7f8df
--- /dev/null
+++ b/tool/jevcorpus/mutate/mutator.go
@@ -0,0 +1,97 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package mutate
+
+import (
+ "math/rand"
+)
+
+// Intent is what a mutation is designed to do to the build.
+type Intent string
+
+const (
+ // IntentUnknown is the zero-value sentinel.
+ IntentUnknown Intent = ""
+ // IntentSafe mutations are meant to keep the module building and its
+ // tests passing.
+ IntentSafe Intent = "safe"
+ // IntentUnsafe mutations are meant to break the build or a test.
+ IntentUnsafe Intent = "unsafe"
+)
+
+// Scope is what a mutator's target names.
+type Scope string
+
+const (
+ // ScopeUnknown is the zero-value sentinel.
+ ScopeUnknown Scope = ""
+ // ScopeFile mutators take a Go file path.
+ ScopeFile Scope = "file"
+ // ScopePackage mutators take a package directory.
+ ScopePackage Scope = "package"
+)
+
+// Mutator is one kind of edit.
+type Mutator struct {
+ // Name identifies the mutator in a corpus manifest; stable across releases.
+ Name string
+ // Intent is what the edit is designed to do to the build.
+ Intent Intent
+ // Scope is what Apply's target names.
+ Scope Scope
+ apply func(t Tree, target string, r *rand.Rand) (Patch, bool)
+}
+
+// Apply edits target in t. It returns false when target has no site for this
+// mutator. Every returned file is gofmt-clean and differs from t's.
+func (m Mutator) Apply(t Tree, target string, r *rand.Rand) (Patch, bool) {
+ p, ok := m.apply(t, target, r)
+ if !ok || len(p) == 0 {
+ return nil, false
+ }
+ out := make(Patch, len(p))
+ for path, src := range p {
+ formatted, ok := gofmt(src)
+ if !ok {
+ return nil, false
+ }
+ if old, exists := t.File(path); exists && old == formatted {
+ continue
+ }
+ out[path] = formatted
+ }
+ if len(out) == 0 {
+ return nil, false
+ }
+ return out, true
+}
+
+// All is every mutator, safe first.
+var All = []Mutator{
+ RenameFunc, RenameLocals, DocumentFuncs, DeleteDeadFunc, AddHelper,
+ FlipComparison, OffByOne, DropErrorCheck, DeleteCalledFunc, AddParam,
+}
+
+// ByIntent returns the mutators with intent i and scope s, in All order. An
+// empty s matches every scope.
+func ByIntent(i Intent, s Scope) []Mutator {
+ var out []Mutator
+ for _, m := range All {
+ if m.Intent == i && (s == ScopeUnknown || m.Scope == s) {
+ out = append(out, m)
+ }
+ }
+ return out
+}
diff --git a/tool/jevcorpus/mutate/safe.go b/tool/jevcorpus/mutate/safe.go
new file mode 100644
index 000000000..ce982e183
--- /dev/null
+++ b/tool/jevcorpus/mutate/safe.go
@@ -0,0 +1,475 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package mutate
+
+import (
+ "go/ast"
+ "go/token"
+ "math/rand"
+ "path"
+ "regexp"
+ "strings"
+)
+
+// RenameFunc renames an unexported function and every use of it.
+var RenameFunc = Mutator{Name: "rename-func", Intent: IntentSafe, Scope: ScopePackage, apply: renameFunc}
+
+// RenameLocals renames local variables in some of a file's functions.
+var RenameLocals = Mutator{Name: "rename-locals", Intent: IntentSafe, Scope: ScopeFile, apply: renameLocals}
+
+// DocumentFuncs adds doc comments to undocumented functions.
+var DocumentFuncs = Mutator{Name: "document-funcs", Intent: IntentSafe, Scope: ScopeFile, apply: documentFuncs}
+
+// DeleteDeadFunc deletes an unexported function nothing references.
+var DeleteDeadFunc = Mutator{Name: "delete-dead-func", Intent: IntentSafe, Scope: ScopePackage, apply: deleteDeadFunc}
+
+// AddHelper adds a small, tested, unused helper in a new file.
+var AddHelper = Mutator{Name: "add-helper", Intent: IntentSafe, Scope: ScopePackage, apply: addHelper}
+
+// hasDirective reports whether a doc comment carries a compiler or cgo
+// directive, which can reference a function by name from outside Go.
+func hasDirective(doc *ast.CommentGroup) bool {
+ if doc == nil {
+ return false
+ }
+ for _, c := range doc.List {
+ if strings.HasPrefix(c.Text, "//go:") || strings.HasPrefix(c.Text, "//export") {
+ return true
+ }
+ }
+ return false
+}
+
+var renamedFuncSuffixes = []string{"Impl", "Internal", "Helper", "Once"}
+
+func renameFunc(t Tree, dir string, r *rand.Rand) (Patch, bool) {
+ files := packageFiles(t, dir)
+ counts := map[string]int{}
+ for _, pf := range files {
+ declarationCounts(pf.file, counts)
+ }
+ names := allIdentNames(files)
+ var sites []funcSite
+ for _, s := range unexportedFuncs(files) {
+ if counts[s.decl.Name.Name] == 1 && !hasDirective(s.decl.Doc) {
+ sites = append(sites, s)
+ }
+ }
+ if len(sites) == 0 {
+ return nil, false
+ }
+ s := sites[r.Intn(len(sites))]
+ old := s.decl.Name.Name
+ renamed := ""
+ for _, i := range r.Perm(len(renamedFuncSuffixes)) {
+ if cand := old + renamedFuncSuffixes[i]; !names[cand] {
+ renamed = cand
+ break
+ }
+ }
+ if renamed == "" {
+ return nil, false
+ }
+
+ out := Patch{}
+ word := regexp.MustCompile(`\b` + regexp.QuoteMeta(old) + `\b`)
+ for _, pf := range files {
+ sel := selectorNames(pf.file)
+ var edits []textEdit
+ ast.Inspect(pf.file, func(n ast.Node) bool {
+ if id, ok := n.(*ast.Ident); ok && id.Name == old && !sel[id] {
+ at := pf.offset(id.Pos())
+ edits = append(edits, textEdit{start: at, end: at + len(old), text: renamed})
+ }
+ return true
+ })
+ if pf.path == s.pf.path && s.decl.Doc != nil {
+ start, end := pf.offset(s.decl.Doc.Pos()), pf.offset(s.decl.Doc.End())
+ edits = append(edits, textEdit{start: start, end: end, text: word.ReplaceAllString(pf.src[start:end], renamed)})
+ }
+ if len(edits) > 0 {
+ out[pf.path] = splice(pf.src, edits)
+ }
+ }
+ return out, true
+}
+
+// shortLocalNames maps terse local names to the longer spellings a rename
+// sweep would choose.
+var shortLocalNames = map[string]string{
+ "i": "idx", "j": "jdx", "n": "count", "s": "str", "b": "buf", "k": "key",
+ "v": "val", "c": "cur", "m": "mapping", "r": "reader", "w": "writer",
+ "p": "ptr", "t": "tmp", "e": "elem", "f": "fn", "x": "item", "res": "result",
+ "req": "request", "resp": "response", "cfg": "config", "msg": "message",
+ "buf": "buffer", "idx": "index", "tmp": "scratch", "val": "value",
+}
+
+func renamedLocal(name string) string {
+ if long, ok := shortLocalNames[name]; ok {
+ return long
+ }
+ return name + "Value"
+}
+
+func renameLocals(t Tree, target string, r *rand.Rand) (Patch, bool) {
+ pf, ok := parseTarget(t, target)
+ if !ok {
+ return nil, false
+ }
+ topLevel := map[string]int{}
+ for _, f := range packageFiles(t, path.Dir(target)) {
+ for _, d := range f.file.Decls {
+ switch d := d.(type) {
+ case *ast.FuncDecl:
+ if d.Recv == nil {
+ topLevel[d.Name.Name]++
+ }
+ case *ast.GenDecl:
+ for _, spec := range d.Specs {
+ switch spec := spec.(type) {
+ case *ast.TypeSpec:
+ topLevel[spec.Name.Name]++
+ case *ast.ValueSpec:
+ for _, id := range spec.Names {
+ topLevel[id.Name]++
+ }
+ }
+ }
+ }
+ }
+ }
+ for _, imp := range pf.file.Imports {
+ topLevel[importName(imp)]++
+ }
+ fileNames := allIdentNames([]parsedFile{pf})
+ sel := selectorNames(pf.file)
+
+ var edits []textEdit
+ var eligible [][]textEdit
+ for _, d := range pf.file.Decls {
+ fd, ok := d.(*ast.FuncDecl)
+ if !ok || fd.Body == nil {
+ continue
+ }
+ if fe := localRenames(pf, fd, topLevel, fileNames, sel); len(fe) > 0 {
+ eligible = append(eligible, fe)
+ }
+ }
+ if len(eligible) == 0 {
+ return nil, false
+ }
+ for i, fe := range eligible {
+ if i == 0 || r.Intn(2) == 0 {
+ edits = append(edits, fe...)
+ }
+ }
+ return Patch{pf.path: splice(pf.src, edits)}, true
+}
+
+// localRenames returns the edits renaming up to three of fd's locals, skipping
+// any whose new or old spelling could collide with something else in scope.
+func localRenames(pf parsedFile, fd *ast.FuncDecl, topLevel map[string]int, fileNames map[string]bool, sel map[*ast.Ident]bool) []textEdit {
+ unsafeNames := map[string]bool{"_": true, "err": true, "ok": true}
+ for _, fl := range []*ast.FieldList{fd.Recv, fd.Type.Params, fd.Type.Results, fd.Type.TypeParams} {
+ if fl == nil {
+ continue
+ }
+ for _, field := range fl.List {
+ for _, n := range field.Names {
+ unsafeNames[n.Name] = true
+ }
+ }
+ }
+ var defined []string
+ seen := map[string]bool{}
+ define := func(e ast.Expr) {
+ if id, ok := e.(*ast.Ident); ok && !seen[id.Name] {
+ seen[id.Name] = true
+ defined = append(defined, id.Name)
+ }
+ }
+ ast.Inspect(fd.Body, func(n ast.Node) bool {
+ switch n := n.(type) {
+ case *ast.AssignStmt:
+ if n.Tok == token.DEFINE {
+ for _, l := range n.Lhs {
+ define(l)
+ }
+ }
+ case *ast.RangeStmt:
+ if n.Tok == token.DEFINE {
+ define(n.Key)
+ define(n.Value)
+ }
+ case *ast.Field:
+ for _, id := range n.Names {
+ unsafeNames[id.Name] = true
+ }
+ case *ast.KeyValueExpr:
+ if id, ok := n.Key.(*ast.Ident); ok {
+ unsafeNames[id.Name] = true
+ }
+ case *ast.LabeledStmt:
+ unsafeNames[n.Label.Name] = true
+ }
+ return true
+ })
+
+ var edits []textEdit
+ renamed := 0
+ for _, name := range defined {
+ if renamed == 3 {
+ break
+ }
+ to := renamedLocal(name)
+ if unsafeNames[name] || topLevel[name] > 0 || fileNames[to] || topLevel[to] > 0 {
+ continue
+ }
+ fileNames[to] = true
+ renamed++
+ ast.Inspect(fd.Body, func(n ast.Node) bool {
+ if id, ok := n.(*ast.Ident); ok && id.Name == name && !sel[id] {
+ at := pf.offset(id.Pos())
+ edits = append(edits, textEdit{start: at, end: at + len(name), text: to})
+ }
+ return true
+ })
+ }
+ return edits
+}
+
+var docVerbs = map[string]string{
+ "get": "returns the", "new": "returns a new", "make": "makes the", "build": "builds the",
+ "parse": "parses the", "load": "loads the", "read": "reads the", "write": "writes the",
+ "handle": "handles the", "create": "creates the", "update": "updates the",
+ "delete": "deletes the", "add": "adds the", "remove": "removes the", "find": "finds the",
+ "list": "lists the", "run": "runs the", "start": "starts the", "stop": "stops the",
+ "close": "closes the", "open": "opens the", "validate": "validates the",
+ "check": "checks the", "apply": "applies the", "format": "formats the",
+ "convert": "converts the", "set": "sets the", "to": "converts to", "with": "returns a copy with the",
+ "is": "reports whether", "has": "reports whether it has", "can": "reports whether it can",
+}
+
+func docSentence(name string) string {
+ words := splitWords(name)
+ if len(words) == 0 {
+ return "// " + name + " is part of this package's implementation."
+ }
+ if verb, ok := docVerbs[words[0]]; ok && len(words) > 1 {
+ return "// " + name + " " + verb + " " + strings.Join(words[1:], " ") + "."
+ }
+ return "// " + name + " implements the " + strings.Join(words, " ") + " step."
+}
+
+func documentFuncs(t Tree, target string, r *rand.Rand) (Patch, bool) {
+ pf, ok := parseTarget(t, target)
+ if !ok {
+ return nil, false
+ }
+ var undocumented []*ast.FuncDecl
+ for _, d := range pf.file.Decls {
+ if fd, ok := d.(*ast.FuncDecl); ok && fd.Doc == nil {
+ undocumented = append(undocumented, fd)
+ }
+ }
+ if len(undocumented) == 0 {
+ return nil, false
+ }
+ var edits []textEdit
+ for i, fd := range undocumented {
+ if i > 0 && r.Intn(10) >= 7 {
+ continue
+ }
+ at, _ := wholeLines(pf.src, pf.offset(fd.Pos()), pf.offset(fd.Pos()))
+ edits = append(edits, textEdit{start: at, end: at, text: docSentence(fd.Name.Name) + "\n"})
+ }
+ return Patch{pf.path: splice(pf.src, edits)}, true
+}
+
+func deleteDeadFunc(t Tree, dir string, r *rand.Rand) (Patch, bool) {
+ files := packageFiles(t, dir)
+ var sites []funcSite
+ for _, s := range unexportedFuncs(files) {
+ if !hasDirective(s.decl.Doc) && funcReferences(files, s) == 0 {
+ sites = append(sites, s)
+ }
+ }
+ if len(sites) == 0 {
+ return nil, false
+ }
+ return deleteFuncDecl(sites[r.Intn(len(sites))]), true
+}
+
+type helperTemplate struct {
+ name, stem, src, test string
+}
+
+var helperTemplates = []helperTemplate{
+ {
+ name: "clampInt", stem: "clamp",
+ src: `package {{pkg}}
+
+// {{name}} limits v to the closed range [lo, hi].
+func {{name}}(v, lo, hi int) int {
+ if v < lo {
+ return lo
+ }
+ if v > hi {
+ return hi
+ }
+ return v
+}
+`,
+ test: `package {{pkg}}
+
+import "testing"
+
+func {{test}}(t *testing.T) {
+ cases := []struct{ v, lo, hi, want int }{
+ {v: -1, lo: 0, hi: 10, want: 0},
+ {v: 5, lo: 0, hi: 10, want: 5},
+ {v: 11, lo: 0, hi: 10, want: 10},
+ }
+ for _, c := range cases {
+ if got := {{name}}(c.v, c.lo, c.hi); got != c.want {
+ t.Errorf("got %d, want %d", got, c.want)
+ }
+ }
+}
+`,
+ },
+ {
+ name: "uniqueStrings", stem: "unique",
+ src: `package {{pkg}}
+
+// {{name}} returns values without duplicates, keeping first occurrences in
+// order.
+func {{name}}(values []string) []string {
+ seen := make(map[string]bool, len(values))
+ out := make([]string, 0, len(values))
+ for _, v := range values {
+ if seen[v] {
+ continue
+ }
+ seen[v] = true
+ out = append(out, v)
+ }
+ return out
+}
+`,
+ test: `package {{pkg}}
+
+import "testing"
+
+func {{test}}(t *testing.T) {
+ got := {{name}}([]string{"b", "a", "b", "c", "a"})
+ want := []string{"b", "a", "c"}
+ if len(got) != len(want) {
+ t.Fatalf("got %v, want %v", got, want)
+ }
+ for i := range want {
+ if got[i] != want[i] {
+ t.Fatalf("got %v, want %v", got, want)
+ }
+ }
+}
+`,
+ },
+ {
+ name: "chunkStrings", stem: "chunk",
+ src: `package {{pkg}}
+
+// {{name}} splits values into consecutive slices of at most size elements.
+// A non-positive size returns nil.
+func {{name}}(values []string, size int) [][]string {
+ if size <= 0 {
+ return nil
+ }
+ var out [][]string
+ for start := 0; start < len(values); start += size {
+ end := start + size
+ if end > len(values) {
+ end = len(values)
+ }
+ out = append(out, values[start:end])
+ }
+ return out
+}
+`,
+ test: `package {{pkg}}
+
+import "testing"
+
+func {{test}}(t *testing.T) {
+ got := {{name}}([]string{"a", "b", "c", "d", "e"}, 2)
+ if len(got) != 3 || len(got[0]) != 2 || len(got[2]) != 1 || got[2][0] != "e" {
+ t.Fatalf("unexpected chunks %v", got)
+ }
+ if {{name}}([]string{"a"}, 0) != nil {
+ t.Fatal("non-positive size must return nil")
+ }
+}
+`,
+ },
+ {
+ name: "reversedStrings", stem: "reverse",
+ src: `package {{pkg}}
+
+// {{name}} returns a reversed copy of values, leaving values unchanged.
+func {{name}}(values []string) []string {
+ out := make([]string, len(values))
+ for i, v := range values {
+ out[len(values)-1-i] = v
+ }
+ return out
+}
+`,
+ test: `package {{pkg}}
+
+import "testing"
+
+func {{test}}(t *testing.T) {
+ in := []string{"a", "b", "c"}
+ got := {{name}}(in)
+ if got[0] != "c" || got[2] != "a" || in[0] != "a" {
+ t.Fatalf("got %v from %v", got, in)
+ }
+}
+`,
+ },
+}
+
+func addHelper(t Tree, dir string, r *rand.Rand) (Patch, bool) {
+ files := packageFiles(t, dir)
+ if len(files) == 0 {
+ return nil, false
+ }
+ pkg := files[0].file.Name.Name
+ names := allIdentNames(files)
+ for _, i := range r.Perm(len(helperTemplates)) {
+ h := helperTemplates[i]
+ test := "Test" + upperFirst(h.name)
+ srcPath := path.Join(dir, h.stem+"_helper.go")
+ testPath := path.Join(dir, h.stem+"_helper_test.go")
+ _, srcExists := t.File(srcPath)
+ _, testExists := t.File(testPath)
+ if names[h.name] || names[test] || srcExists || testExists {
+ continue
+ }
+ fill := strings.NewReplacer("{{pkg}}", pkg, "{{name}}", h.name, "{{test}}", test)
+ return Patch{srcPath: fill.Replace(h.src), testPath: fill.Replace(h.test)}, true
+ }
+ return nil, false
+}
diff --git a/tool/jevcorpus/mutate/source.go b/tool/jevcorpus/mutate/source.go
new file mode 100644
index 000000000..95d2691f7
--- /dev/null
+++ b/tool/jevcorpus/mutate/source.go
@@ -0,0 +1,260 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package mutate
+
+import (
+ "go/ast"
+ "go/format"
+ "go/parser"
+ "go/token"
+ "sort"
+ "strings"
+ "unicode"
+)
+
+type parsedFile struct {
+ path string
+ src string
+ fset *token.FileSet
+ file *ast.File
+}
+
+func parseFile(p, src string) (parsedFile, bool) {
+ fset := token.NewFileSet()
+ f, err := parser.ParseFile(fset, p, src, parser.ParseComments|parser.SkipObjectResolution)
+ if err != nil {
+ return parsedFile{}, false
+ }
+ return parsedFile{path: p, src: src, fset: fset, file: f}, true
+}
+
+func (pf parsedFile) offset(pos token.Pos) int { return pf.fset.File(pos).Offset(pos) }
+
+// packageFiles parses the files in dir that belong to its primary package —
+// the one its first non-test file declares — so an external _test package,
+// which cannot see unexported names, is left out.
+func packageFiles(t Tree, dir string) []parsedFile {
+ var parsed []parsedFile
+ for _, p := range t.GoFiles(dir) {
+ src, _ := t.File(p)
+ if pf, ok := parseFile(p, src); ok {
+ parsed = append(parsed, pf)
+ }
+ }
+ name := ""
+ for _, pf := range parsed {
+ if !IsTestFile(pf.path) {
+ name = pf.file.Name.Name
+ break
+ }
+ }
+ if name == "" {
+ return nil
+ }
+ out := parsed[:0]
+ for _, pf := range parsed {
+ if pf.file.Name.Name == name {
+ out = append(out, pf)
+ }
+ }
+ return out
+}
+
+type textEdit struct {
+ start, end int
+ text string
+}
+
+// splice applies non-overlapping edits to src.
+func splice(src string, edits []textEdit) string {
+ sorted := append([]textEdit(nil), edits...)
+ sort.Slice(sorted, func(i, j int) bool { return sorted[i].start > sorted[j].start })
+ for _, e := range sorted {
+ src = src[:e.start] + e.text + src[e.end:]
+ }
+ return src
+}
+
+// wholeLines widens [start, end) to the full lines it touches, with their
+// trailing newline, so removing it leaves no stray indentation.
+func wholeLines(src string, start, end int) (int, int) {
+ for start > 0 && src[start-1] != '\n' {
+ start--
+ }
+ if i := strings.IndexByte(src[end:], '\n'); i >= 0 {
+ end += i + 1
+ } else {
+ end = len(src)
+ }
+ return start, end
+}
+
+// gofmt returns src formatted, or false when it no longer parses.
+func gofmt(src string) (string, bool) {
+ out, err := format.Source([]byte(src))
+ if err != nil {
+ return "", false
+ }
+ return string(out), true
+}
+
+func isExported(name string) bool {
+ for _, r := range name {
+ return unicode.IsUpper(r)
+ }
+ return false
+}
+
+func isFuncCandidate(fd *ast.FuncDecl) bool {
+ return fd.Recv == nil && fd.Body != nil && fd.Name.Name != "init" && fd.Name.Name != "main" && fd.Name.Name != "_"
+}
+
+// selectorNames returns the identifiers that appear as the selected name of a
+// selector expression (x.Name) or as a composite-literal key, where an
+// identifier names a field or method rather than the declaration in scope.
+func selectorNames(f *ast.File) map[*ast.Ident]bool {
+ out := map[*ast.Ident]bool{}
+ ast.Inspect(f, func(n ast.Node) bool {
+ switch n := n.(type) {
+ case *ast.SelectorExpr:
+ out[n.Sel] = true
+ case *ast.KeyValueExpr:
+ if id, ok := n.Key.(*ast.Ident); ok {
+ out[id] = true
+ }
+ }
+ return true
+ })
+ return out
+}
+
+// declarationCounts counts the identifiers declared under each name in f:
+// functions, methods, types, values, parameters, results, fields, and local
+// definitions.
+func declarationCounts(f *ast.File, counts map[string]int) {
+ addFields := func(fl *ast.FieldList) {
+ if fl == nil {
+ return
+ }
+ for _, field := range fl.List {
+ for _, n := range field.Names {
+ counts[n.Name]++
+ }
+ }
+ }
+ ast.Inspect(f, func(n ast.Node) bool {
+ switch n := n.(type) {
+ case *ast.FuncDecl:
+ counts[n.Name.Name]++
+ addFields(n.Recv)
+ case *ast.FuncType:
+ addFields(n.TypeParams)
+ addFields(n.Params)
+ addFields(n.Results)
+ case *ast.StructType:
+ addFields(n.Fields)
+ case *ast.InterfaceType:
+ addFields(n.Methods)
+ case *ast.TypeSpec:
+ counts[n.Name.Name]++
+ case *ast.ValueSpec:
+ for _, id := range n.Names {
+ counts[id.Name]++
+ }
+ case *ast.AssignStmt:
+ if n.Tok == token.DEFINE {
+ for _, l := range n.Lhs {
+ if id, ok := l.(*ast.Ident); ok {
+ counts[id.Name]++
+ }
+ }
+ }
+ case *ast.RangeStmt:
+ if n.Tok == token.DEFINE {
+ for _, e := range []ast.Expr{n.Key, n.Value} {
+ if id, ok := e.(*ast.Ident); ok {
+ counts[id.Name]++
+ }
+ }
+ }
+ case *ast.ImportSpec:
+ counts[importName(n)]++
+ case *ast.LabeledStmt:
+ counts[n.Label.Name]++
+ }
+ return true
+ })
+}
+
+// importName is the name an import binds in its file. For an unnamed import it
+// is the path's last element, which matches the package name by convention.
+func importName(imp *ast.ImportSpec) string {
+ if imp.Name != nil {
+ return imp.Name.Name
+ }
+ p := strings.Trim(imp.Path.Value, `"`)
+ return p[strings.LastIndexByte(p, '/')+1:]
+}
+
+// allIdentNames is every identifier spelling in files, selectors included.
+func allIdentNames(files []parsedFile) map[string]bool {
+ out := map[string]bool{}
+ for _, pf := range files {
+ ast.Inspect(pf.file, func(n ast.Node) bool {
+ if id, ok := n.(*ast.Ident); ok {
+ out[id.Name] = true
+ }
+ return true
+ })
+ }
+ return out
+}
+
+// splitWords splits a Go identifier into lower-case words: "parseHTTPHeader"
+// becomes "parse http header".
+func splitWords(name string) []string {
+ var words []string
+ var cur []rune
+ runes := []rune(name)
+ for i, r := range runes {
+ if r == '_' {
+ if len(cur) > 0 {
+ words = append(words, strings.ToLower(string(cur)))
+ cur = nil
+ }
+ continue
+ }
+ boundary := i > 0 && unicode.IsUpper(r) &&
+ (unicode.IsLower(runes[i-1]) || (i+1 < len(runes) && unicode.IsLower(runes[i+1]) && unicode.IsUpper(runes[i-1])))
+ if boundary && len(cur) > 0 {
+ words = append(words, strings.ToLower(string(cur)))
+ cur = nil
+ }
+ cur = append(cur, r)
+ }
+ if len(cur) > 0 {
+ words = append(words, strings.ToLower(string(cur)))
+ }
+ return words
+}
+
+func upperFirst(s string) string {
+ if s == "" {
+ return s
+ }
+ r := []rune(s)
+ r[0] = unicode.ToUpper(r[0])
+ return string(r)
+}
diff --git a/tool/jevcorpus/mutate/tree.go b/tool/jevcorpus/mutate/tree.go
new file mode 100644
index 000000000..cea7c02f6
--- /dev/null
+++ b/tool/jevcorpus/mutate/tree.go
@@ -0,0 +1,120 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package mutate makes deterministic, labeled edits to Go source: changes meant
+// to keep a module building and its tests passing (safe) and changes meant to
+// break one or the other (unsafe). Edits are spliced into the original text at
+// token positions rather than reprinted from the syntax tree, so a mutation's
+// diff is only the lines it means to change.
+//
+// The intent is a hypothesis, not a label. A caller turns it into a label by
+// building and testing the result; see tool/jevcorpus.
+package mutate
+
+import (
+ "path"
+ "sort"
+ "strings"
+)
+
+// Tree is an immutable snapshot of Go source files keyed by slash-separated
+// path relative to the repository root.
+type Tree struct {
+ files map[string]string
+}
+
+// NewTree returns a Tree holding a copy of files.
+func NewTree(files map[string]string) Tree {
+ cp := make(map[string]string, len(files))
+ for p, src := range files {
+ cp[p] = src
+ }
+ return Tree{files: cp}
+}
+
+// File returns the content at p.
+func (t Tree) File(p string) (string, bool) {
+ src, ok := t.files[p]
+ return src, ok
+}
+
+// Len is the number of files in the tree.
+func (t Tree) Len() int { return len(t.files) }
+
+// Dirs returns every directory holding a Go file, sorted.
+func (t Tree) Dirs() []string {
+ seen := map[string]bool{}
+ for p := range t.files {
+ seen[path.Dir(p)] = true
+ }
+ dirs := make([]string, 0, len(seen))
+ for d := range seen {
+ dirs = append(dirs, d)
+ }
+ sort.Strings(dirs)
+ return dirs
+}
+
+// GoFiles returns the Go files directly in dir, sorted.
+func (t Tree) GoFiles(dir string) []string {
+ var out []string
+ for p := range t.files {
+ if path.Dir(p) == dir && strings.HasSuffix(p, ".go") {
+ out = append(out, p)
+ }
+ }
+ sort.Strings(out)
+ return out
+}
+
+// With returns a new Tree with p's files replacing or adding to t's.
+func (t Tree) With(p Patch) Tree {
+ cp := make(map[string]string, len(t.files)+len(p))
+ for k, v := range t.files {
+ cp[k] = v
+ }
+ for k, v := range p {
+ cp[k] = v
+ }
+ return Tree{files: cp}
+}
+
+// Patch is a set of whole-file contents keyed by path. A path absent from the
+// tree it applies to is a new file.
+type Patch map[string]string
+
+// Merge returns a new Patch holding p's files overlaid by q's.
+func (p Patch) Merge(q Patch) Patch {
+ out := make(Patch, len(p)+len(q))
+ for k, v := range p {
+ out[k] = v
+ }
+ for k, v := range q {
+ out[k] = v
+ }
+ return out
+}
+
+// Paths returns the patched paths, sorted.
+func (p Patch) Paths() []string {
+ out := make([]string, 0, len(p))
+ for k := range p {
+ out = append(out, k)
+ }
+ sort.Strings(out)
+ return out
+}
+
+// IsTestFile reports whether p is a Go test file.
+func IsTestFile(p string) bool { return strings.HasSuffix(p, "_test.go") }
diff --git a/tool/jevcorpus/mutate/unsafe.go b/tool/jevcorpus/mutate/unsafe.go
new file mode 100644
index 000000000..8610da82a
--- /dev/null
+++ b/tool/jevcorpus/mutate/unsafe.go
@@ -0,0 +1,372 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package mutate
+
+import (
+ "go/ast"
+ "go/token"
+ "math/rand"
+ "strconv"
+)
+
+// FlipComparison inverts one comparison in an if or for condition. Nil checks
+// are left to DropErrorCheck so this stays a logic bug rather than a crash.
+var FlipComparison = Mutator{Name: "flip-comparison", Intent: IntentUnsafe, Scope: ScopeFile, apply: flipComparison}
+
+// OffByOne relaxes or tightens a bound against len(), or moves an integer
+// literal in a comparison by one.
+var OffByOne = Mutator{Name: "off-by-one", Intent: IntentUnsafe, Scope: ScopeFile, apply: offByOne}
+
+// DropErrorCheck removes an `if err != nil { ... return ... }` block.
+var DropErrorCheck = Mutator{Name: "drop-error-check", Intent: IntentUnsafe, Scope: ScopeFile, apply: dropErrorCheck}
+
+// DeleteCalledFunc deletes an unexported function that the package still
+// references.
+var DeleteCalledFunc = Mutator{Name: "delete-called-func", Intent: IntentUnsafe, Scope: ScopePackage, apply: deleteCalledFunc}
+
+// AddParam adds a parameter to a function without updating its callers.
+var AddParam = Mutator{Name: "add-param", Intent: IntentUnsafe, Scope: ScopePackage, apply: addParam}
+
+var flippedComparison = map[token.Token]token.Token{
+ token.EQL: token.NEQ, token.NEQ: token.EQL,
+ token.LSS: token.GEQ, token.GEQ: token.LSS,
+ token.GTR: token.LEQ, token.LEQ: token.GTR,
+}
+
+var relaxedBound = map[token.Token]token.Token{
+ token.LSS: token.LEQ, token.LEQ: token.LSS,
+ token.GTR: token.GEQ, token.GEQ: token.GTR,
+}
+
+func isNil(e ast.Expr) bool {
+ id, ok := e.(*ast.Ident)
+ return ok && id.Name == "nil"
+}
+
+func isLenCall(e ast.Expr) bool {
+ call, ok := e.(*ast.CallExpr)
+ if !ok {
+ return false
+ }
+ id, ok := call.Fun.(*ast.Ident)
+ return ok && id.Name == "len"
+}
+
+// conditionComparisons returns the comparisons inside if and for conditions.
+func conditionComparisons(f *ast.File) []*ast.BinaryExpr {
+ var out []*ast.BinaryExpr
+ visit := func(cond ast.Expr) {
+ if cond == nil {
+ return
+ }
+ ast.Inspect(cond, func(n ast.Node) bool {
+ if _, ok := n.(*ast.FuncLit); ok {
+ return false
+ }
+ if be, ok := n.(*ast.BinaryExpr); ok {
+ if _, cmp := flippedComparison[be.Op]; cmp {
+ out = append(out, be)
+ }
+ }
+ return true
+ })
+ }
+ ast.Inspect(f, func(n ast.Node) bool {
+ switch n := n.(type) {
+ case *ast.IfStmt:
+ visit(n.Cond)
+ case *ast.ForStmt:
+ visit(n.Cond)
+ }
+ return true
+ })
+ return out
+}
+
+func replaceOperator(pf parsedFile, be *ast.BinaryExpr, op token.Token) Patch {
+ start := pf.offset(be.OpPos)
+ return Patch{pf.path: splice(pf.src, []textEdit{{start: start, end: start + len(be.Op.String()), text: op.String()}})}
+}
+
+func flipComparison(t Tree, target string, r *rand.Rand) (Patch, bool) {
+ pf, ok := parseTarget(t, target)
+ if !ok {
+ return nil, false
+ }
+ var sites []*ast.BinaryExpr
+ for _, be := range conditionComparisons(pf.file) {
+ if !isNil(be.X) && !isNil(be.Y) {
+ sites = append(sites, be)
+ }
+ }
+ if len(sites) == 0 {
+ return nil, false
+ }
+ be := sites[r.Intn(len(sites))]
+ return replaceOperator(pf, be, flippedComparison[be.Op]), true
+}
+
+func offByOne(t Tree, target string, r *rand.Rand) (Patch, bool) {
+ pf, ok := parseTarget(t, target)
+ if !ok {
+ return nil, false
+ }
+ type site struct {
+ be *ast.BinaryExpr
+ lit *ast.BasicLit
+ }
+ var sites []site
+ for _, be := range conditionComparisons(pf.file) {
+ if _, bound := relaxedBound[be.Op]; !bound {
+ continue
+ }
+ if isLenCall(be.X) || isLenCall(be.Y) {
+ sites = append(sites, site{be: be})
+ continue
+ }
+ for _, side := range []ast.Expr{be.X, be.Y} {
+ if lit, ok := side.(*ast.BasicLit); ok && lit.Kind == token.INT {
+ if _, err := strconv.Atoi(lit.Value); err == nil {
+ sites = append(sites, site{be: be, lit: lit})
+ }
+ }
+ }
+ }
+ if len(sites) == 0 {
+ return nil, false
+ }
+ s := sites[r.Intn(len(sites))]
+ if s.lit == nil {
+ return replaceOperator(pf, s.be, relaxedBound[s.be.Op]), true
+ }
+ v, _ := strconv.Atoi(s.lit.Value)
+ start := pf.offset(s.lit.Pos())
+ return Patch{pf.path: splice(pf.src, []textEdit{{start: start, end: start + len(s.lit.Value), text: strconv.Itoa(v + 1)}})}, true
+}
+
+func dropErrorCheck(t Tree, target string, r *rand.Rand) (Patch, bool) {
+ pf, ok := parseTarget(t, target)
+ if !ok {
+ return nil, false
+ }
+ var sites []*ast.IfStmt
+ ast.Inspect(pf.file, func(n ast.Node) bool {
+ is, ok := n.(*ast.IfStmt)
+ if !ok || is.Init != nil || is.Else != nil {
+ return true
+ }
+ be, ok := is.Cond.(*ast.BinaryExpr)
+ if !ok || be.Op != token.NEQ || !isNil(be.Y) {
+ return true
+ }
+ if id, ok := be.X.(*ast.Ident); !ok || id.Name != "err" {
+ return true
+ }
+ for _, stmt := range is.Body.List {
+ if _, ok := stmt.(*ast.ReturnStmt); ok {
+ sites = append(sites, is)
+ break
+ }
+ }
+ return true
+ })
+ if len(sites) == 0 {
+ return nil, false
+ }
+ is := sites[r.Intn(len(sites))]
+ start, end := wholeLines(pf.src, pf.offset(is.Pos()), pf.offset(is.End()))
+ return Patch{pf.path: splice(pf.src, []textEdit{{start: start, end: end}})}, true
+}
+
+type funcSite struct {
+ pf parsedFile
+ decl *ast.FuncDecl
+}
+
+// funcReferences counts identifiers spelled name across files, outside the
+// declaration itself and outside selectors and literal keys.
+func funcReferences(files []parsedFile, site funcSite) int {
+ name := site.decl.Name.Name
+ refs := 0
+ for _, pf := range files {
+ sel := selectorNames(pf.file)
+ ast.Inspect(pf.file, func(n ast.Node) bool {
+ id, ok := n.(*ast.Ident)
+ if !ok || id.Name != name || sel[id] {
+ return true
+ }
+ if pf.path == site.pf.path && id.Pos() >= site.decl.Pos() && id.End() <= site.decl.End() {
+ return true
+ }
+ refs++
+ return true
+ })
+ }
+ return refs
+}
+
+// unexportedFuncs returns the package's unexported top-level functions in
+// non-test files.
+func unexportedFuncs(files []parsedFile) []funcSite {
+ var out []funcSite
+ for _, pf := range files {
+ if IsTestFile(pf.path) {
+ continue
+ }
+ for _, d := range pf.file.Decls {
+ fd, ok := d.(*ast.FuncDecl)
+ if ok && isFuncCandidate(fd) && !isExported(fd.Name.Name) {
+ out = append(out, funcSite{pf: pf, decl: fd})
+ }
+ }
+ }
+ return out
+}
+
+func deleteFuncDecl(site funcSite) Patch {
+ startPos := site.decl.Pos()
+ if site.decl.Doc != nil {
+ startPos = site.decl.Doc.Pos()
+ }
+ start, end := wholeLines(site.pf.src, site.pf.offset(startPos), site.pf.offset(site.decl.End()))
+ return Patch{site.pf.path: splice(site.pf.src, []textEdit{{start: start, end: end}})}
+}
+
+func deleteCalledFunc(t Tree, dir string, r *rand.Rand) (Patch, bool) {
+ files := packageFiles(t, dir)
+ var sites []funcSite
+ for _, s := range unexportedFuncs(files) {
+ if funcReferences(files, s) > 0 {
+ sites = append(sites, s)
+ }
+ }
+ if len(sites) == 0 {
+ return nil, false
+ }
+ return deleteFuncDecl(sites[r.Intn(len(sites))]), true
+}
+
+var extraParams = []struct{ name, typ string }{
+ {"limit", "int"}, {"verbose", "bool"}, {"retries", "int"}, {"flags", "uint32"},
+}
+
+func addParam(t Tree, dir string, r *rand.Rand) (Patch, bool) {
+ files := packageFiles(t, dir)
+ var sites []funcSite
+ for _, pf := range files {
+ if IsTestFile(pf.path) {
+ continue
+ }
+ for _, d := range pf.file.Decls {
+ fd, ok := d.(*ast.FuncDecl)
+ if !ok || !isFuncCandidate(fd) || isVariadic(fd) || fd.Type.TypeParams != nil {
+ continue
+ }
+ if s := (funcSite{pf: pf, decl: fd}); calledIn(files, s) {
+ sites = append(sites, s)
+ }
+ }
+ }
+ if len(sites) == 0 {
+ return nil, false
+ }
+ s := sites[r.Intn(len(sites))]
+ used := map[string]bool{}
+ ast.Inspect(s.decl, func(n ast.Node) bool {
+ if id, ok := n.(*ast.Ident); ok {
+ used[id.Name] = true
+ }
+ return true
+ })
+ for _, off := range r.Perm(len(extraParams)) {
+ p := extraParams[off]
+ if used[p.name] {
+ continue
+ }
+ params := s.decl.Type.Params
+ text := p.name + " " + p.typ
+ if params.NumFields() > 0 {
+ text = ", " + text
+ if trailingComma(s.pf.src, s.pf.offset(params.Closing)) {
+ text = p.name + " " + p.typ + ","
+ }
+ }
+ at := s.pf.offset(params.Closing)
+ return Patch{s.pf.path: splice(s.pf.src, []textEdit{{start: at, end: at, text: text}})}, true
+ }
+ return nil, false
+}
+
+func isVariadic(fd *ast.FuncDecl) bool {
+ list := fd.Type.Params.List
+ if len(list) == 0 {
+ return false
+ }
+ _, ok := list[len(list)-1].Type.(*ast.Ellipsis)
+ return ok
+}
+
+// calledIn reports whether site's function is called by name somewhere
+// outside its own body.
+func calledIn(files []parsedFile, site funcSite) bool {
+ fd := site.decl
+ for _, pf := range files {
+ sameFile := pf.path == site.pf.path
+ found := false
+ ast.Inspect(pf.file, func(n ast.Node) bool {
+ if found {
+ return false
+ }
+ call, ok := n.(*ast.CallExpr)
+ if !ok {
+ return true
+ }
+ if id, ok := call.Fun.(*ast.Ident); ok && id.Name == fd.Name.Name && !(sameFile && call.Pos() >= fd.Pos() && call.End() <= fd.End()) {
+ found = true
+ }
+ return true
+ })
+ if found {
+ return true
+ }
+ }
+ return false
+}
+
+func trailingComma(src string, closing int) bool {
+ for i := closing - 1; i >= 0; i-- {
+ switch src[i] {
+ case ' ', '\t', '\n', '\r':
+ continue
+ case ',':
+ return true
+ default:
+ return false
+ }
+ }
+ return false
+}
+
+// parseTarget parses a non-test file target.
+func parseTarget(t Tree, target string) (parsedFile, bool) {
+ if IsTestFile(target) {
+ return parsedFile{}, false
+ }
+ src, ok := t.File(target)
+ if !ok {
+ return parsedFile{}, false
+ }
+ return parseFile(target, src)
+}
diff --git a/tool/jevcorpus/plan.go b/tool/jevcorpus/plan.go
new file mode 100644
index 000000000..5c8dd7e49
--- /dev/null
+++ b/tool/jevcorpus/plan.go
@@ -0,0 +1,326 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "fmt"
+ "math/rand"
+ "path"
+ "sort"
+ "strings"
+
+ "github.com/uber/submitqueue/submitqueue/extension/speculation/scorer/jev"
+ "github.com/uber/submitqueue/tool/jevcorpus/mutate"
+)
+
+// Tier is a change's size class. XL is defined by the jev scorer's diff budget
+// and file cap, so an XL change is one the model cannot see in full.
+type Tier string
+
+const (
+ // TierUnknown is the zero-value sentinel.
+ TierUnknown Tier = ""
+ // TierS is under 50 changed lines.
+ TierS Tier = "S"
+ // TierM is 50 to 499 changed lines.
+ TierM Tier = "M"
+ // TierL is 500 or more changed lines within the diff budget.
+ TierL Tier = "L"
+ // TierXL exceeds the diff budget or the file cap.
+ TierXL Tier = "XL"
+)
+
+var tiers = []Tier{TierS, TierM, TierL, TierXL}
+
+// maxFilesInState mirrors the jev scorer's per-batch file cap.
+const maxFilesInState = 200
+
+func tierFor(lines, diffBytes, files int) Tier {
+ switch {
+ case diffBytes > jev.DefaultMaxDiffBytes || files > maxFilesInState:
+ return TierXL
+ case lines >= 500:
+ return TierL
+ case lines >= 50:
+ return TierM
+ default:
+ return TierS
+ }
+}
+
+// NeedlePosition is where an unsafe edit sits among a large change's files.
+type NeedlePosition string
+
+const (
+ // NeedleNone is the zero-value sentinel: no needle.
+ NeedleNone NeedlePosition = ""
+ // NeedleEarly puts the needle in the first touched file in path order,
+ // which is where the diff starts.
+ NeedleEarly NeedlePosition = "early"
+ // NeedleLate puts it in the last, where a truncated diff is cut.
+ NeedleLate NeedlePosition = "late"
+)
+
+// spec is one change the corpus should contain.
+type spec struct {
+ intent mutate.Intent
+ tier Tier
+}
+
+// planSpecs balances count changes across intents and tiers: the unsafe share
+// is unsafeRatio, and each intent is spread evenly over the tiers so size never
+// predicts the label.
+func planSpecs(count int, unsafeRatio float64, r *rand.Rand) []spec {
+ unsafe := int(float64(count)*unsafeRatio + 0.5)
+ out := make([]spec, 0, count)
+ for i := 0; i < count; i++ {
+ intent, k := mutate.IntentSafe, i-unsafe
+ if i < unsafe {
+ intent, k = mutate.IntentUnsafe, i
+ }
+ out = append(out, spec{intent: intent, tier: tiers[k%len(tiers)]})
+ }
+ r.Shuffle(len(out), func(i, j int) { out[i], out[j] = out[j], out[i] })
+ return out
+}
+
+// candidates are the mutation targets: non-generated, non-test files and their
+// directories, in packages whose tests pass at the base.
+type candidates struct {
+ files []string
+ dirs []string
+}
+
+// built is a change ready to write to the work tree.
+type built struct {
+ patch mutate.Patch
+ mutators []string
+ needle needle
+}
+
+// needle locates the one unsafe edit in an otherwise safe sweep.
+type needle struct {
+ mutator string
+ file string
+ position NeedlePosition
+}
+
+// sweepTarget is the approximate size a sweep grows to before it stops: lines
+// for M and L, estimated diff bytes for XL.
+type sweepTarget struct {
+ lines, bytes int
+}
+
+var sweepTargets = map[Tier]sweepTarget{
+ TierM: {lines: 120},
+ TierL: {lines: 900},
+ TierXL: {bytes: jev.DefaultMaxDiffBytes * 3 / 2},
+}
+
+// maxSweepFiles bounds how far a sweep searches for files it can grow by.
+const maxSweepFiles = 600
+
+func buildChange(t mutate.Tree, c candidates, s spec, r *rand.Rand) (built, bool) {
+ if s.tier == TierS {
+ return buildSingle(t, c, s.intent, r)
+ }
+ return buildSweep(t, c, s, r)
+}
+
+func buildSingle(t mutate.Tree, c candidates, intent mutate.Intent, r *rand.Rand) (built, bool) {
+ ms := mutate.ByIntent(intent, mutate.ScopeUnknown)
+ for try := 0; try < 40; try++ {
+ m := ms[r.Intn(len(ms))]
+ target, ok := pickTarget(c, m.Scope, r)
+ if !ok {
+ continue
+ }
+ if p, ok := m.Apply(t, target, r); ok {
+ return built{patch: p, mutators: []string{m.Name}}, true
+ }
+ }
+ return built{}, false
+}
+
+func pickTarget(c candidates, s mutate.Scope, r *rand.Rand) (string, bool) {
+ pool := c.files
+ if s == mutate.ScopePackage {
+ pool = c.dirs
+ }
+ if len(pool) == 0 {
+ return "", false
+ }
+ return pool[r.Intn(len(pool))], true
+}
+
+// buildSweep grows a mechanical, behavior-preserving change file by file until
+// it reaches the tier's size, sometimes adding a safe package-level edit as a
+// decoy, then for an unsafe spec plants one unsafe edit at the first or last
+// touched file.
+func buildSweep(t mutate.Tree, c candidates, s spec, r *rand.Rand) (built, bool) {
+ target := sweepTargets[s.tier]
+ patch := mutate.Patch{}
+ est := newDiffEstimate(t)
+ swept := 0
+ var applied []string
+ if r.Intn(10) < 3 {
+ decoys := []mutate.Mutator{mutate.DeleteDeadFunc, mutate.RenameFunc, mutate.AddHelper}
+ m := decoys[r.Intn(len(decoys))]
+ if dir, ok := pickTarget(c, mutate.ScopePackage, r); ok {
+ if p, ok := m.Apply(t, dir, r); ok {
+ patch = patch.Merge(p)
+ est.update(patch, p)
+ applied = append(applied, m.Name)
+ }
+ }
+ }
+ for _, i := range r.Perm(len(c.files)) {
+ if est.reached(target) || swept >= maxSweepFiles {
+ break
+ }
+ file := c.files[i]
+ swept++
+ for _, m := range []mutate.Mutator{mutate.DocumentFuncs, mutate.RenameLocals} {
+ if p, ok := m.Apply(t.With(patch), file, r); ok {
+ patch = patch.Merge(p)
+ est.update(patch, p)
+ applied = appendOnce(applied, m.Name)
+ }
+ }
+ }
+ if len(patch) == 0 || !est.reached(target) {
+ return built{}, false
+ }
+ b := built{patch: patch, mutators: applied}
+ if s.intent == mutate.IntentSafe {
+ return b, true
+ }
+ return plantNeedle(t, b, r)
+}
+
+func plantNeedle(t mutate.Tree, b built, r *rand.Rand) (built, bool) {
+ files := b.patch.Paths()
+ position := NeedleEarly
+ if r.Intn(2) == 0 {
+ position = NeedleLate
+ sort.Sort(sort.Reverse(sort.StringSlice(files)))
+ }
+ needles := mutate.ByIntent(mutate.IntentUnsafe, mutate.ScopeFile)
+ for _, file := range files {
+ for _, i := range r.Perm(len(needles)) {
+ m := needles[i]
+ if p, ok := m.Apply(t.With(b.patch), file, r); ok {
+ return built{
+ patch: b.patch.Merge(p),
+ mutators: append(append([]string(nil), b.mutators...), m.Name),
+ needle: needle{mutator: m.Name, file: file, position: position},
+ }, true
+ }
+ }
+ }
+ return built{}, false
+}
+
+func appendOnce(list []string, s string) []string {
+ for _, x := range list {
+ if x == s {
+ return list
+ }
+ }
+ return append(list, s)
+}
+
+// diffEstimate tracks a growing patch's estimated size file by file, so a sweep
+// re-estimates only what each step touched.
+type diffEstimate struct {
+ base mutate.Tree
+ perFile map[string][2]int
+ lines, bytes int
+}
+
+func newDiffEstimate(base mutate.Tree) *diffEstimate {
+ return &diffEstimate{base: base, perFile: map[string][2]int{}}
+}
+
+func (d *diffEstimate) update(full, step mutate.Patch) {
+ for path := range step {
+ prev := d.perFile[path]
+ lines, bytes := estimateDiff(d.base, mutate.Patch{path: full[path]})
+ d.perFile[path] = [2]int{lines, bytes}
+ d.lines += lines - prev[0]
+ d.bytes += bytes - prev[1]
+ }
+}
+
+func (d *diffEstimate) reached(target sweepTarget) bool {
+ if target.bytes > 0 {
+ return d.bytes >= target.bytes
+ }
+ return d.lines >= target.lines
+}
+
+// estimateDiff approximates a patch's changed lines and unified-diff size
+// without running git: changed lines are the per-file multiset difference of
+// old and new lines, and each carries roughly its own length again in context.
+// It only steers sweep growth; the corpus records git's own numbers.
+func estimateDiff(t mutate.Tree, p mutate.Patch) (lines, bytes int) {
+ for path, src := range p {
+ old, _ := t.File(path)
+ counts := map[string]int{}
+ for _, l := range strings.Split(old, "\n") {
+ counts[l]++
+ }
+ for _, l := range strings.Split(src, "\n") {
+ counts[l]--
+ }
+ for l, n := range counts {
+ if n < 0 {
+ n = -n
+ }
+ lines += n
+ bytes += n * 2 * (len(l) + 2)
+ }
+ bytes += 2*len(path) + 64
+ }
+ return lines, bytes
+}
+
+var commitSubjects = []string{
+ "refactor(%s): tidy internals",
+ "%s: clean up helpers",
+ "chore(%s): small cleanups",
+ "%s: simplify control flow",
+ "fix(%s): adjust edge-case handling",
+ "%s: update docs and naming",
+ "perf(%s): avoid redundant work",
+ "%s: follow-up to review comments",
+}
+
+var authors = []struct{ name, email string }{
+ {"Avery Lin", "avery.lin@corpus.invalid"},
+ {"Jordan Patel", "jordan.patel@corpus.invalid"},
+ {"Sam Okafor", "sam.okafor@corpus.invalid"},
+ {"Riley Novak", "riley.novak@corpus.invalid"},
+ {"Casey Moreau", "casey.moreau@corpus.invalid"},
+ {"Morgan Ito", "morgan.ito@corpus.invalid"},
+}
+
+// commitSubject names a change from where it is, never from what it does: the
+// subject is drawn independently of the mutation so it cannot leak the label.
+func commitSubject(p mutate.Patch, r *rand.Rand) string {
+ paths := p.Paths()
+ scope := path.Base(path.Dir(paths[r.Intn(len(paths))]))
+ return fmt.Sprintf(commitSubjects[r.Intn(len(commitSubjects))], scope)
+}
diff --git a/tool/jevcorpus/plan_test.go b/tool/jevcorpus/plan_test.go
new file mode 100644
index 000000000..a5a111e16
--- /dev/null
+++ b/tool/jevcorpus/plan_test.go
@@ -0,0 +1,225 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "fmt"
+ "math/rand"
+ "strings"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+
+ "github.com/uber/submitqueue/tool/jevcorpus/mutate"
+)
+
+// syntheticModule is a tree of undocumented functions with locals, comparisons
+// and error checks — enough sites for every mutator and for sweeps of any
+// tier.
+func syntheticModule(packages, filesPerPackage int) (mutate.Tree, candidates) {
+ files := map[string]string{}
+ var c candidates
+ for p := 0; p < packages; p++ {
+ dir := fmt.Sprintf("pkg%02d", p)
+ c.dirs = append(c.dirs, dir)
+ for f := 0; f < filesPerPackage; f++ {
+ var b strings.Builder
+ fmt.Fprintf(&b, "package pkg%02d\n\n", p)
+ for fn := 0; fn < 6; fn++ {
+ fmt.Fprintf(&b, `func computeTotal%[1]d%[2]d(values []int, limit int) (int, error) {
+ res := 0
+ for i := 0; i < len(values); i++ {
+ v := values[i]
+ if v > limit {
+ continue
+ }
+ res += v
+ }
+ err := validateTotal%[1]d%[2]d(res)
+ if err != nil {
+ return 0, err
+ }
+ return res, nil
+}
+
+func validateTotal%[1]d%[2]d(n int) error {
+ if n == 42 {
+ return nil
+ }
+ return nil
+}
+
+func unusedHelper%[1]d%[2]d(x int) int { return x + 1 }
+
+`, f, fn)
+ }
+ path := fmt.Sprintf("%s/file%02d.go", dir, f)
+ files[path] = b.String()
+ c.files = append(c.files, path)
+ }
+ }
+ return mutate.NewTree(files), c
+}
+
+func TestPlanSpecsBalancesIntentAndTier(t *testing.T) {
+ specs := planSpecs(80, 0.5, rand.New(rand.NewSource(1)))
+ require.Len(t, specs, 80)
+ cells := map[spec]int{}
+ for _, s := range specs {
+ cells[s]++
+ }
+ for _, intent := range []mutate.Intent{mutate.IntentSafe, mutate.IntentUnsafe} {
+ for _, tier := range tiers {
+ assert.Equal(t, 10, cells[spec{intent: intent, tier: tier}], "%s %s", intent, tier)
+ }
+ }
+}
+
+func TestPlanSpecsHonorsUnsafeRatio(t *testing.T) {
+ unsafe := 0
+ for _, s := range planSpecs(40, 0.25, rand.New(rand.NewSource(1))) {
+ if s.intent == mutate.IntentUnsafe {
+ unsafe++
+ }
+ }
+ assert.Equal(t, 10, unsafe)
+}
+
+func TestTierFor(t *testing.T) {
+ tests := []struct {
+ name string
+ lines, diffBytes, files int
+ want Tier
+ }{
+ {name: "small", lines: 10, diffBytes: 800, files: 1, want: TierS},
+ {name: "medium", lines: 50, diffBytes: 4000, files: 3, want: TierM},
+ {name: "large", lines: 500, diffBytes: 40_000, files: 20, want: TierL},
+ {name: "past the diff budget", lines: 900, diffBytes: 70_000, files: 40, want: TierXL},
+ {name: "past the file cap", lines: 300, diffBytes: 20_000, files: 201, want: TierXL},
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ assert.Equal(t, tt.want, tierFor(tt.lines, tt.diffBytes, tt.files))
+ })
+ }
+}
+
+func TestBuildChange(t *testing.T) {
+ tree, c := syntheticModule(30, 12)
+ for _, intent := range []mutate.Intent{mutate.IntentSafe, mutate.IntentUnsafe} {
+ for _, tier := range tiers {
+ t.Run(string(intent)+" "+string(tier), func(t *testing.T) {
+ r := rand.New(rand.NewSource(5))
+ b, ok := buildChange(tree, c, spec{intent: intent, tier: tier}, r)
+ require.True(t, ok)
+ require.NotEmpty(t, b.patch)
+ for _, name := range b.mutators {
+ m := mutatorNamed(t, name)
+ if m.Intent == mutate.IntentUnsafe {
+ assert.Equal(t, mutate.IntentUnsafe, intent, "a safe change applied %s", name)
+ }
+ }
+ if tier != TierS {
+ lines, bytes := estimateDiff(tree, b.patch)
+ if target := sweepTargets[tier]; target.bytes > 0 {
+ assert.GreaterOrEqual(t, bytes, target.bytes)
+ } else {
+ assert.GreaterOrEqual(t, lines, target.lines)
+ }
+ }
+ if intent == mutate.IntentUnsafe && tier != TierS {
+ assert.NotEmpty(t, b.needle.mutator)
+ assert.Contains(t, b.patch, b.needle.file)
+ }
+ })
+ }
+ }
+}
+
+func TestPlantNeedlePosition(t *testing.T) {
+ tree, c := syntheticModule(4, 4)
+ sweep, ok := buildSweep(tree, c, spec{intent: mutate.IntentSafe, tier: TierM}, rand.New(rand.NewSource(2)))
+ require.True(t, ok)
+ paths := sweep.patch.Paths()
+ seen := map[NeedlePosition]bool{}
+ for seed := int64(0); seed < 20; seed++ {
+ b, ok := plantNeedle(tree, sweep, rand.New(rand.NewSource(seed)))
+ require.True(t, ok)
+ seen[b.needle.position] = true
+ switch b.needle.position {
+ case NeedleEarly:
+ assert.Equal(t, paths[0], b.needle.file)
+ case NeedleLate:
+ assert.Equal(t, paths[len(paths)-1], b.needle.file)
+ }
+ }
+ assert.True(t, seen[NeedleEarly] && seen[NeedleLate], "both positions are planted")
+}
+
+func TestCommitSubjectDoesNotNameTheMutation(t *testing.T) {
+ p := mutate.Patch{"pkg/store/a.go": ""}
+ for seed := int64(0); seed < 50; seed++ {
+ subject := commitSubject(p, rand.New(rand.NewSource(seed)))
+ assert.Contains(t, subject, "store")
+ for _, m := range mutate.All {
+ assert.NotContains(t, subject, m.Name)
+ }
+ }
+}
+
+func TestManifestChangeCategory(t *testing.T) {
+ tests := []struct {
+ name string
+ change ManifestChange
+ want string
+ }{
+ {name: "single edit", change: ManifestChange{TargetTier: TierS, Mutators: []string{"flip-comparison"}}, want: "flip-comparison"},
+ {name: "safe sweep", change: ManifestChange{TargetTier: TierL, Mutators: []string{"document-funcs"}}, want: "sweep"},
+ {name: "sweep with a needle", change: ManifestChange{TargetTier: TierXL, Needle: ManifestNeedle{Mutator: "off-by-one"}}, want: "sweep+off-by-one"},
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ assert.Equal(t, tt.want, tt.change.Category())
+ })
+ }
+}
+
+func TestPlanEvalBatches(t *testing.T) {
+ changes := []ManifestChange{
+ {Branch: "a", Lands: true, Tier: TierS, TargetTier: TierS, Mutators: []string{"rename-func"}},
+ {Branch: "b", Lands: false, Tier: TierXL, TargetTier: TierXL, Needle: ManifestNeedle{Mutator: "off-by-one", Position: NeedleLate, BeyondDiffBudget: true}},
+ {Branch: "c", Lands: true, Tier: TierM, TargetTier: TierM},
+ }
+ batches, rows := planEvalBatches(changes, evalFlags{batchSize: 3, batches: 2}, rand.New(rand.NewSource(1)))
+ require.Len(t, batches, 5)
+ require.Len(t, rows, 5)
+ assert.True(t, rows[0].Lands)
+ assert.False(t, rows[1].Lands)
+ assert.Contains(t, rows[1].Slices, "needle beyond diff budget")
+ assert.Equal(t, []string{"1"}, batches[0].Contains)
+ assert.False(t, rows[3].Lands, "a batch holding a failing change fails")
+ assert.Len(t, batches[3].Contains, 3)
+}
+
+func mutatorNamed(t *testing.T, name string) mutate.Mutator {
+ for _, m := range mutate.All {
+ if m.Name == name {
+ return m
+ }
+ }
+ require.Failf(t, "unknown mutator", "%q", name)
+ return mutate.Mutator{}
+}
diff --git a/tool/jevcorpus/resolver.go b/tool/jevcorpus/resolver.go
new file mode 100644
index 000000000..e4499017e
--- /dev/null
+++ b/tool/jevcorpus/resolver.go
@@ -0,0 +1,63 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "context"
+ "fmt"
+
+ "github.com/uber/submitqueue/platform/base/change"
+ "github.com/uber/submitqueue/submitqueue/entity"
+ "github.com/uber/submitqueue/submitqueue/extension/changeprovider"
+)
+
+// corpusResolver is a changeset.Resolver over a fixed set of single-change
+// requests, standing in for the orchestrator's request and change stores. It
+// asks the change provider on every call, as the orchestrator's stores are
+// filled from the provider at admission.
+type corpusResolver struct {
+ queue string
+ provider changeprovider.ChangeProvider
+ // uriByRequest maps a request ID to its one change URI.
+ uriByRequest map[string]string
+}
+
+func (r corpusResolver) ChangesForBatch(_ context.Context, batch entity.Batch) ([]change.Change, error) {
+ out := make([]change.Change, 0, len(batch.Contains))
+ for _, id := range batch.Contains {
+ uri, ok := r.uriByRequest[id]
+ if !ok {
+ return nil, fmt.Errorf("batch %s names unknown request %s", batch.ID, id)
+ }
+ out = append(out, change.Change{URIs: []string{uri}})
+ }
+ return out, nil
+}
+
+func (r corpusResolver) DetailedForBatch(ctx context.Context, batch entity.Batch) (entity.BatchChanges, error) {
+ changes, err := r.ChangesForBatch(ctx, batch)
+ if err != nil {
+ return entity.BatchChanges{}, err
+ }
+ out := entity.BatchChanges{BatchID: batch.ID, Queue: batch.Queue}
+ for i, c := range changes {
+ infos, err := r.provider.Get(ctx, entity.Request{ID: batch.Contains[i], Queue: r.queue, Change: c})
+ if err != nil {
+ return entity.BatchChanges{}, err
+ }
+ out.Changes = append(out.Changes, infos...)
+ }
+ return out, nil
+}
diff --git a/tool/replaybench/BUILD.bazel b/tool/replaybench/BUILD.bazel
new file mode 100644
index 000000000..a53f415be
--- /dev/null
+++ b/tool/replaybench/BUILD.bazel
@@ -0,0 +1,33 @@
+load("@rules_go//go:def.bzl", "go_binary", "go_library", "go_test")
+
+go_library(
+ name = "go_default_library",
+ srcs = [
+ "fetch.go",
+ "main.go",
+ "materialize.go",
+ ],
+ importpath = "github.com/uber/submitqueue/tool/replaybench",
+ visibility = ["//visibility:private"],
+ deps = [
+ "//platform/base/change/git:go_default_library",
+ "//submitqueue/extension/buildrunner/replay:go_default_library",
+ ],
+)
+
+go_binary(
+ name = "replaybench",
+ embed = [":go_default_library"],
+ visibility = ["//visibility:public"],
+)
+
+go_test(
+ name = "go_default_test",
+ srcs = ["fetch_test.go"],
+ embed = [":go_default_library"],
+ deps = [
+ "//submitqueue/extension/buildrunner/replay:go_default_library",
+ "@com_github_stretchr_testify//assert:go_default_library",
+ "@com_github_stretchr_testify//require:go_default_library",
+ ],
+)
diff --git a/tool/replaybench/fetch.go b/tool/replaybench/fetch.go
new file mode 100644
index 000000000..78c931afe
--- /dev/null
+++ b/tool/replaybench/fetch.go
@@ -0,0 +1,184 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "encoding/json"
+ "errors"
+ "flag"
+ "fmt"
+ "io"
+ "net/http"
+ "net/url"
+ "os"
+ "regexp"
+ "sort"
+ "strconv"
+ "time"
+
+ "github.com/uber/submitqueue/submitqueue/extension/buildrunner/replay"
+)
+
+// workflowRun is the subset of a GitHub Actions run the dataset needs.
+type workflowRun struct {
+ HeadSHA string `json:"head_sha"`
+ HeadBranch string `json:"head_branch"`
+ Status string `json:"status"`
+ Conclusion string `json:"conclusion"`
+ CreatedAt time.Time `json:"created_at"`
+ RunStartedAt time.Time `json:"run_started_at"`
+ UpdatedAt time.Time `json:"updated_at"`
+}
+
+// groupBranch is GitHub's merge-group branch: gh-readonly-queue//pr--.
+var groupBranch = regexp.MustCompile(`^gh-readonly-queue/(.+)/pr-(\d+)-([0-9a-f]{40})$`)
+
+func runFetch(args []string) error {
+ fs := flag.NewFlagSet("fetch", flag.ExitOnError)
+ repo := fs.String("repo", "", "upstream repository, owner/name (required)")
+ baseBranch := fs.String("base-branch", "main", "branch the merge queue lands onto")
+ since := fs.String("since", "", "first day of the window, YYYY-MM-DD (required)")
+ until := fs.String("until", "", "last day of the window, YYYY-MM-DD (required)")
+ maxAttempts := fs.Int("max", 40, "keep at most this many attempts, oldest first")
+ out := fs.String("out", "", "dataset path to write (required)")
+ if err := fs.Parse(args); err != nil {
+ return err
+ }
+ if *repo == "" || *since == "" || *until == "" || *out == "" {
+ return errors.New("-repo, -since, -until, and -out are required")
+ }
+ token := os.Getenv("GITHUB_TOKEN")
+ if token == "" {
+ return errors.New("GITHUB_TOKEN must be set (for example: export GITHUB_TOKEN=$(gh auth token))")
+ }
+
+ runs, err := listMergeGroupRuns(token, *repo, *since+".."+*until)
+ if err != nil {
+ return err
+ }
+ ds := replay.Dataset{Repo: *repo, BaseBranch: *baseBranch, Attempts: attemptsFromRuns(runs, *baseBranch)}
+ if len(ds.Attempts) > *maxAttempts {
+ ds.Attempts = ds.Attempts[:*maxAttempts]
+ }
+ fails := 0
+ for _, a := range ds.Attempts {
+ if a.Outcome == replay.OutcomeFail {
+ fails++
+ }
+ }
+ fmt.Printf("%s: %d attempts (%d failed) from %d runs\n", *repo, len(ds.Attempts), fails, len(runs))
+ return writeDataset(*out, ds)
+}
+
+// attemptsFromRuns groups runs by the merge-group commit they tested. A group
+// fails if any run failed and passes if every run passed; a group whose runs
+// were cancelled or are unfinished has no outcome and is dropped — GitHub
+// cancels the groups behind a failure, and those say nothing about their PR.
+func attemptsFromRuns(runs []workflowRun, baseBranch string) []replay.Attempt {
+ type group struct {
+ attempt replay.Attempt
+ verdict string
+ }
+ groups := map[string]*group{}
+ for _, r := range runs {
+ m := groupBranch.FindStringSubmatch(r.HeadBranch)
+ if m == nil || m[1] != baseBranch {
+ continue
+ }
+ g, ok := groups[r.HeadSHA]
+ if !ok {
+ pr, _ := strconv.Atoi(m[2])
+ g = &group{attempt: replay.Attempt{PR: pr, GroupSHA: r.HeadSHA, ParentSHA: m[3], QueuedAtMs: r.CreatedAt.UnixMilli()}, verdict: "pass"}
+ groups[r.HeadSHA] = g
+ }
+ g.attempt.QueuedAtMs = min(g.attempt.QueuedAtMs, r.CreatedAt.UnixMilli())
+ if !r.RunStartedAt.IsZero() {
+ g.attempt.DurationMs = max(g.attempt.DurationMs, r.UpdatedAt.Sub(r.RunStartedAt).Milliseconds())
+ }
+ switch {
+ case r.Status != "completed":
+ g.verdict = "unknown"
+ case r.Conclusion == "failure" || r.Conclusion == "timed_out" || r.Conclusion == "startup_failure":
+ if g.verdict != "unknown" {
+ g.verdict = "fail"
+ }
+ case r.Conclusion == "success" || r.Conclusion == "skipped" || r.Conclusion == "neutral":
+ default:
+ if g.verdict == "pass" {
+ g.verdict = "unknown"
+ }
+ }
+ }
+ var attempts []replay.Attempt
+ for _, g := range groups {
+ switch g.verdict {
+ case "pass":
+ g.attempt.Outcome = replay.OutcomePass
+ case "fail":
+ g.attempt.Outcome = replay.OutcomeFail
+ default:
+ continue
+ }
+ attempts = append(attempts, g.attempt)
+ }
+ sort.Slice(attempts, func(i, j int) bool { return attempts[i].QueuedAtMs < attempts[j].QueuedAtMs })
+ return attempts
+}
+
+// listMergeGroupRuns pages through the window's merge_group runs. GitHub
+// returns at most 1000 runs per query, so keep windows small.
+func listMergeGroupRuns(token, repo, created string) ([]workflowRun, error) {
+ var all []workflowRun
+ for page := 1; page <= 10; page++ {
+ q := url.Values{"event": {"merge_group"}, "created": {created}, "per_page": {"100"}, "page": {strconv.Itoa(page)}}
+ req, err := http.NewRequest(http.MethodGet, "https://api.github.com/repos/"+repo+"/actions/runs?"+q.Encode(), nil)
+ if err != nil {
+ return nil, err
+ }
+ req.Header.Set("Authorization", "Bearer "+token)
+ req.Header.Set("Accept", "application/vnd.github+json")
+ resp, err := http.DefaultClient.Do(req)
+ if err != nil {
+ return nil, fmt.Errorf("list runs: %w", err)
+ }
+ body, err := io.ReadAll(resp.Body)
+ resp.Body.Close()
+ if err != nil {
+ return nil, err
+ }
+ if resp.StatusCode != http.StatusOK {
+ return nil, fmt.Errorf("list runs: %s: %s", resp.Status, body)
+ }
+ var payload struct {
+ WorkflowRuns []workflowRun `json:"workflow_runs"`
+ }
+ if err := json.Unmarshal(body, &payload); err != nil {
+ return nil, fmt.Errorf("decode runs: %w", err)
+ }
+ all = append(all, payload.WorkflowRuns...)
+ if len(payload.WorkflowRuns) < 100 {
+ break
+ }
+ }
+ return all, nil
+}
+
+func writeDataset(path string, ds replay.Dataset) error {
+ raw, err := json.MarshalIndent(ds, "", " ")
+ if err != nil {
+ return err
+ }
+ return os.WriteFile(path, append(raw, '\n'), 0o644)
+}
diff --git a/tool/replaybench/fetch_test.go b/tool/replaybench/fetch_test.go
new file mode 100644
index 000000000..fbbbdad42
--- /dev/null
+++ b/tool/replaybench/fetch_test.go
@@ -0,0 +1,53 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+
+ "github.com/uber/submitqueue/submitqueue/extension/buildrunner/replay"
+)
+
+func TestAttemptsFromRuns(t *testing.T) {
+ const parent = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
+ t0 := time.Unix(1_700_000_000, 0)
+ run := func(sha string, pr string, conclusion string, at time.Time, minutes int) workflowRun {
+ return workflowRun{
+ HeadSHA: sha, HeadBranch: "gh-readonly-queue/main/pr-" + pr + "-" + parent,
+ Status: "completed", Conclusion: conclusion,
+ CreatedAt: at, RunStartedAt: at, UpdatedAt: at.Add(time.Duration(minutes) * time.Minute),
+ }
+ }
+ runs := []workflowRun{
+ run("pass1", "1", "success", t0, 10),
+ run("pass1", "1", "skipped", t0, 1),
+ run("fail2", "2", "success", t0.Add(time.Minute), 5),
+ run("fail2", "2", "failure", t0.Add(time.Minute), 20),
+ // GitHub cancels the groups behind a failure; they say nothing about
+ // their own PR.
+ run("cancel3", "3", "cancelled", t0.Add(2*time.Minute), 1),
+ {HeadSHA: "other", HeadBranch: "gh-readonly-queue/release/pr-4-" + parent, Status: "completed", Conclusion: "success", CreatedAt: t0},
+ {HeadSHA: "pending5", HeadBranch: "gh-readonly-queue/main/pr-5-" + parent, Status: "in_progress", CreatedAt: t0},
+ }
+
+ got := attemptsFromRuns(runs, "main")
+ require.Len(t, got, 2)
+ assert.Equal(t, replay.Attempt{PR: 1, GroupSHA: "pass1", ParentSHA: parent, Outcome: replay.OutcomePass, DurationMs: 10 * 60_000, QueuedAtMs: t0.UnixMilli()}, got[0])
+ assert.Equal(t, replay.Attempt{PR: 2, GroupSHA: "fail2", ParentSHA: parent, Outcome: replay.OutcomeFail, DurationMs: 20 * 60_000, QueuedAtMs: t0.Add(time.Minute).UnixMilli()}, got[1])
+}
diff --git a/tool/replaybench/main.go b/tool/replaybench/main.go
new file mode 100644
index 000000000..91a7288f3
--- /dev/null
+++ b/tool/replaybench/main.go
@@ -0,0 +1,55 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Command replaybench builds replay benchmarks from an open-source
+// repository's GitHub merge-queue history. See
+// doc/rfc/submitqueue/replay-benchmark.md.
+//
+// replaybench fetch -repo owner/name -since 2026-01-01 -until 2026-02-01 -out ds.json
+// replaybench materialize -dataset ds.json -mirror /tmp/m.git -sandbox /tmp/s/sandbox.git
+// replaybench split -dataset ds.json -size 12
+// replaybench uris -dataset ds.json
+package main
+
+import (
+ "fmt"
+ "os"
+)
+
+const usage = `usage: replaybench [flags]`
+
+func main() {
+ if len(os.Args) < 2 {
+ fmt.Fprintln(os.Stderr, usage)
+ os.Exit(2)
+ }
+ var err error
+ switch os.Args[1] {
+ case "fetch":
+ err = runFetch(os.Args[2:])
+ case "split":
+ err = runSplit(os.Args[2:])
+ case "materialize":
+ err = runMaterialize(os.Args[2:])
+ case "uris":
+ err = runURIs(os.Args[2:])
+ default:
+ fmt.Fprintln(os.Stderr, usage)
+ os.Exit(2)
+ }
+ if err != nil {
+ fmt.Fprintln(os.Stderr, "Error:", err)
+ os.Exit(1)
+ }
+}
diff --git a/tool/replaybench/materialize.go b/tool/replaybench/materialize.go
new file mode 100644
index 000000000..947e1d54b
--- /dev/null
+++ b/tool/replaybench/materialize.go
@@ -0,0 +1,274 @@
+// Copyright (c) 2026 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "bytes"
+ "errors"
+ "flag"
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+
+ changegit "github.com/uber/submitqueue/platform/base/change/git"
+ "github.com/uber/submitqueue/submitqueue/extension/buildrunner/replay"
+)
+
+// Every replayed change is rebuilt as one commit on the window's base: the
+// upstream PR's own effect (its merge-group commit minus that commit's parent),
+// three-way merged onto the base. Rebuilding on one base, rather than keeping
+// upstream's queue topology, is what lets the change provider measure each
+// change against the sandbox's own main: upstream group commits descend from
+// queue heads the sandbox never has.
+func runMaterialize(args []string) error {
+ fs := flag.NewFlagSet("materialize", flag.ExitOnError)
+ dsPath := fs.String("dataset", "", "dataset written by fetch; rewritten in place (required)")
+ mirror := fs.String("mirror", "", "bare clone of the upstream repository, created if missing (required)")
+ sandbox := fs.String("sandbox", "", "bare sandbox repository to create (required); must not exist")
+ if err := fs.Parse(args); err != nil {
+ return err
+ }
+ if *dsPath == "" || *mirror == "" || *sandbox == "" {
+ return errors.New("-dataset, -mirror, and -sandbox are required")
+ }
+ // The push runs inside the mirror, so a relative sandbox path would
+ // resolve against the wrong directory.
+ sandboxPath, err := filepath.Abs(*sandbox)
+ if err != nil {
+ return err
+ }
+ *sandbox = sandboxPath
+ ds, err := replay.Load(*dsPath)
+ if err != nil {
+ return err
+ }
+ if len(ds.Attempts) == 0 {
+ return errors.New("dataset has no attempts")
+ }
+ if _, err := os.Stat(*sandbox); err == nil {
+ return fmt.Errorf("sandbox %s already exists; remove it first", *sandbox)
+ }
+
+ if _, err := os.Stat(*mirror); err != nil {
+ fmt.Printf("cloning %s into %s\n", ds.Repo, *mirror)
+ if _, err := git("", "clone", "--bare", "--quiet", "https://github.com/"+ds.Repo+".git", *mirror); err != nil {
+ return err
+ }
+ }
+ if err := fetchMissing(*mirror, ds); err != nil {
+ return err
+ }
+
+ ds.BaseSHA = ds.Attempts[0].ParentSHA
+ baseTree, err := git(*mirror, "rev-parse", ds.BaseSHA+"^{tree}")
+ if err != nil {
+ return err
+ }
+ if err := markFlakySuspects(*mirror, ds.Attempts); err != nil {
+ return err
+ }
+ kept := 0
+ for i := range ds.Attempts {
+ a := &ds.Attempts[i]
+ a.ReplaySHA, a.Branch, a.Skipped = "", "", ""
+ if err := rebuildOnBase(*mirror, ds.BaseSHA, baseTree, a); err != nil {
+ return fmt.Errorf("PR %d: %w", a.PR, err)
+ }
+ if a.Skipped == "" {
+ kept++
+ }
+ }
+
+ refspecs := []string{ds.BaseSHA + ":refs/heads/" + ds.BaseBranch}
+ for _, a := range ds.Attempts {
+ if a.ReplaySHA != "" {
+ refspecs = append(refspecs, a.ReplaySHA+":refs/heads/"+a.Branch)
+ }
+ }
+ if err := os.MkdirAll(filepath.Dir(*sandbox), 0o755); err != nil {
+ return err
+ }
+ if _, err := git("", "init", "--bare", "--quiet", "-b", ds.BaseBranch, *sandbox); err != nil {
+ return err
+ }
+ if _, err := git(*mirror, append([]string{"push", "--quiet", *sandbox}, refspecs...)...); err != nil {
+ return err
+ }
+ fmt.Printf("sandbox %s: base %s, %d of %d attempts materialized\n", *sandbox, ds.BaseSHA[:12], kept, len(ds.Attempts))
+ return writeDataset(*dsPath, ds)
+}
+
+// fetchMissing fetches group and parent commits the mirror lacks. GitHub
+// serves commits by SHA even after the merge-group branch is deleted, which
+// is what makes failed attempts — never on any branch — recoverable.
+func fetchMissing(mirror string, ds replay.Dataset) error {
+ var missing []string
+ seen := map[string]bool{}
+ for _, a := range ds.Attempts {
+ for _, sha := range []string{a.GroupSHA, a.ParentSHA} {
+ if seen[sha] {
+ continue
+ }
+ seen[sha] = true
+ if _, err := git(mirror, "cat-file", "-e", sha+"^{commit}"); err != nil {
+ missing = append(missing, sha)
+ }
+ }
+ }
+ if len(missing) == 0 {
+ return nil
+ }
+ fmt.Printf("fetching %d commits not on any branch\n", len(missing))
+ _, err := git(mirror, append([]string{"fetch", "--quiet", "origin"}, missing...)...)
+ return err
+}
+
+func rebuildOnBase(mirror, base, baseTree string, a *replay.Attempt) error {
+ tree, err := git(mirror, "merge-tree", "--write-tree", "--merge-base="+a.ParentSHA, base, a.GroupSHA)
+ if err != nil {
+ var exitErr *exec.ExitError
+ if errors.As(err, &exitErr) && exitErr.ExitCode() == 1 {
+ a.Skipped = "conflicts with the window base"
+ return nil
+ }
+ return err
+ }
+ tree = strings.SplitN(tree, "\n", 2)[0]
+ if tree == baseTree {
+ a.Skipped = "no change relative to the window base"
+ return nil
+ }
+ meta, err := git(mirror, "log", "-1", "--format=%an%x00%ae%x00%aI%x00%s", a.GroupSHA)
+ if err != nil {
+ return err
+ }
+ f := strings.SplitN(meta, "\x00", 4)
+ if len(f) != 4 {
+ return fmt.Errorf("unreadable metadata for %s", a.GroupSHA)
+ }
+ a.Title = f[3]
+ cmd := exec.Command("git", "commit-tree", tree, "-p", base, "-m", a.Title)
+ cmd.Dir = mirror
+ cmd.Env = append(os.Environ(),
+ "GIT_AUTHOR_NAME="+f[0], "GIT_AUTHOR_EMAIL="+f[1], "GIT_AUTHOR_DATE="+f[2],
+ "GIT_COMMITTER_NAME=replaybench", "GIT_COMMITTER_EMAIL=replaybench@submitqueue.invalid", "GIT_COMMITTER_DATE="+f[2])
+ out, err := run(cmd)
+ if err != nil {
+ return err
+ }
+ a.ReplaySHA = out
+ a.Branch = fmt.Sprintf("replay/pr-%d/%s", a.PR, a.GroupSHA[:8])
+ return nil
+}
+
+// markFlakySuspects records each attempt's patch ID and flags failures whose
+// exact change later passed for the same PR.
+func markFlakySuspects(mirror string, attempts []replay.Attempt) error {
+ for i := range attempts {
+ a := &attempts[i]
+ cmd := exec.Command("sh", "-c", `git diff "$1" "$2" | git patch-id --stable`, "sh", a.ParentSHA, a.GroupSHA)
+ cmd.Dir = mirror
+ out, err := run(cmd)
+ if err != nil {
+ return fmt.Errorf("PR %d: patch id: %w", a.PR, err)
+ }
+ a.PatchID, _, _ = strings.Cut(out, " ")
+ }
+ for i := range attempts {
+ a := &attempts[i]
+ a.FlakySuspect = false
+ if a.Outcome != replay.OutcomeFail || a.PatchID == "" {
+ continue
+ }
+ for _, later := range attempts[i+1:] {
+ if later.PR == a.PR && later.PatchID == a.PatchID && later.Outcome == replay.OutcomePass {
+ a.FlakySuspect = true
+ break
+ }
+ }
+ }
+ return nil
+}
+
+// runSplit cuts a fetched dataset into consecutive epochs of -size attempts,
+// written next to it as .e01.json, .e02.json, and so on. Each
+// epoch is materialized on its own base, so a short epoch drifts less from
+// upstream and loses fewer attempts to conflicts.
+func runSplit(args []string) error {
+ fs := flag.NewFlagSet("split", flag.ExitOnError)
+ dsPath := fs.String("dataset", "", "dataset written by fetch (required)")
+ size := fs.Int("size", 12, "attempts per epoch")
+ if err := fs.Parse(args); err != nil {
+ return err
+ }
+ if *dsPath == "" || *size < 1 {
+ return errors.New("-dataset and a positive -size are required")
+ }
+ ds, err := replay.Load(*dsPath)
+ if err != nil {
+ return err
+ }
+ stem := strings.TrimSuffix(*dsPath, filepath.Ext(*dsPath))
+ for n, start := 1, 0; start < len(ds.Attempts); n, start = n+1, start+*size {
+ epoch := ds
+ epoch.BaseSHA = ""
+ epoch.Attempts = append([]replay.Attempt(nil), ds.Attempts[start:min(start+*size, len(ds.Attempts))]...)
+ path := fmt.Sprintf("%s.e%02d.json", stem, n)
+ if err := writeDataset(path, epoch); err != nil {
+ return err
+ }
+ fmt.Println(path)
+ }
+ return nil
+}
+
+func runURIs(args []string) error {
+ fs := flag.NewFlagSet("uris", flag.ExitOnError)
+ dsPath := fs.String("dataset", "", "materialized dataset (required)")
+ remote := fs.String("remote", "git.example.com", "authority in the printed change URIs")
+ repoName := fs.String("repo-name", "sandbox", "repository segment in the printed change URIs")
+ if err := fs.Parse(args); err != nil {
+ return err
+ }
+ ds, err := replay.Load(*dsPath)
+ if err != nil {
+ return err
+ }
+ for _, a := range ds.Attempts {
+ if a.ReplaySHA == "" {
+ continue
+ }
+ fmt.Println(changegit.ChangeID{Scheme: "git", Remote: *remote, Repo: *repoName, Ref: "refs/heads/" + a.Branch, CommitSHA: a.ReplaySHA}.String())
+ }
+ return nil
+}
+
+func git(dir string, args ...string) (string, error) {
+ cmd := exec.Command("git", args...)
+ cmd.Dir = dir
+ return run(cmd)
+}
+
+func run(cmd *exec.Cmd) (string, error) {
+ var stderr bytes.Buffer
+ cmd.Stderr = &stderr
+ out, err := cmd.Output()
+ if err != nil {
+ return "", fmt.Errorf("git %s: %w: %s", strings.Join(cmd.Args[1:], " "), err, strings.TrimSpace(stderr.String()))
+ }
+ return strings.TrimSpace(string(out)), nil
+}