Skip to content

Commit 319ecb0

Browse files
committed
feat(stovepipe): define repair coordination contracts and RFC
1 parent 2d379da commit 319ecb0

24 files changed

Lines changed: 1306 additions & 2 deletions

‎Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -629,6 +629,7 @@ GO_GENERATE_PACKAGES := \
629629
./stovepipe/controller/... \
630630
./stovepipe/core/requestlog/... \
631631
./stovepipe/extension/buildrunner/... \
632+
./stovepipe/extension/fixer/... \
632633
./stovepipe/extension/projectstatus/... \
633634
./stovepipe/extension/queueconfig/... \
634635
./stovepipe/extension/sourcecontrol/... \

‎doc/rfc/stovepipe/fixer.md‎

Lines changed: 119 additions & 0 deletions
Large diffs are not rendered by default.

‎doc/rfc/stovepipe/workflow.md‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ The pipeline is a queue-driven chain of small, single-purpose controllers, in th
88
99
The planned `buildanalysis` step runs alongside that chain: it extracts details periodically while a build runs and performs a final pass after completion. The [workflow diagram](#workflow) shows the full intended flow through breakage tracking, repair decisions, execution, and recovery; [Build analysis](#build-analysis) describes the extraction step.
1010

11+
The [repair workflow](#repair-workflow) branches from build observations: planned breakage tracking requests repairs for broken checks and supplies later recovery evidence. Queue messages and Go extension contracts are defined; the stage implementations, persistence, and wiring follow the repair RFC.
12+
1113
There is no `analyze` controller and no `analyze` topic. A later project-analysis stage that would map a target graph onto another `build` pass is design only, described under [Designed, not built](#designed-not-built-project-analysis). `record` resolves project results inline on the same delivery that writes the repository fact.
1214

1315
## What Stovepipe is agnostic about
@@ -53,13 +55,18 @@ A Queue may have a **promotion ref**: a stable branch name (say `verified-main`
5355

5456
The ref is a *cache* of the last-green URI, not a second record of greenness. It only ever moves where the bookmark already points, so it inherits the same forward-only rule, and a commit that a history rewrite has dropped from the branch is skipped rather than retried — the next green commit corrects the ref. Which ref a Queue promotes to, and whether it has one at all, is `SourceControl` configuration resolved from the Queue name alongside the repo and credentials; the pipeline names only the commit, never a branch.
5557

58+
### Breakage and fix — failure episodes and shared repairs
59+
60+
A [`Breakage`](../../../stovepipe/entity/fix.go) identifies one episode of a failing check by `(Queue, Key, Generation)` and references its build and failure observation. Planned breakage tracking owns health and generations. The repair design adds an evaluator that decides whether to start a fix, attach the episode to an existing fix, or wait for more evidence. A `Fix` can cover several episodes, with recovery recorded independently for each.
61+
5662
## Extensions
5763

5864
| Extension | Responsibility |
5965
|---|---|
6066
| **SourceControl** | Resolve a Queue name to its current head URI; answer ancestry/comparison questions between two URIs (is the new head a fast-forward descendant of the last green, or was history rewritten?); enumerate commits in a range; advance the Queue's **promotion ref** to a commit. The sole owner of URI semantics, including which refs a Queue name resolves to. |
6167
| **build-runner** | Build a scope at a URI, optionally relative to a baseline URI. `Trigger` returns a build id; `Status` returns pass/fail and the caller-supplied metadata it echoed. It does not return a target graph. See [build-runner.md](../submitqueue/build-runner.md). |
6268
| **buildanalysis.Analyzer** (planned) | Extract available check results and evidence from a referenced Build. Resolve provider-specific artifacts through injected dependencies and report completeness and coverage. |
69+
| **fixer** (design) | `RepairEvaluator` recommends Start, Attach, or Defer before dispatch. [`RepairExecutor`](../../../stovepipe/extension/fixer/fixer.go) submits and observes the selected repair. Host wiring supplies both implementations. Evaluation and runtime coordination remain to be implemented; see [fixer.md](fixer.md#implementation). |
6370
| **Hooks** | Deliver Stovepipe's validation events to downstream systems. What is published today is repository-scoped: validation of this URI has begun, this URI is green or not green, and a cancelled validation ended without a fact. Fire-and-forget notification, decoupled so Stovepipe does not know or care who consumes the event. The shared cross-domain hook seam rather than a Stovepipe-specific extension. See [hook-framework.md](../hook-framework.md). |
6471
| **Storage** | Persist Queues (incl. last-green URI), Requests, build records, and per-URI / per-project greenness. Key/value-shaped per the extension-design rules in [AGENTS.md](../../../AGENTS.md). |
6572

@@ -155,6 +162,18 @@ Add `stovepipe/controller/buildanalysis.Controller` implementing [`consumer.Cont
155162

156163
The initial and completion handoffs follow the existing persist-then-publish retry pattern. Analysis holds only its own per-build partition. Versioned writes make repeated polls and completion signals converge while preserving final results against stale updates. Downstream breakage tracking determines failure episodes and recovery from this evidence.
157164

165+
### Repair workflow
166+
167+
The repair branch defines asynchronous work alongside validation. The queue contracts are present; the `fix` and `fixsignal` implementations and registration follow the [repair RFC](fixer.md#implementation). Both stages use [`consumer.Controller.Process`](../../../platform/consumer/controller.go). The planned [buildanalysis step](#build-analysis) persists observations during and after execution. Breakage tracking uses that evidence to maintain failure episodes and recovery; it is a planned responsibility, with its runtime integration defined separately. Validation capacity, result recording, hooks, and promotion remain independent of repair progress.
168+
169+
1. **Persist and publish the episode.** Breakage tracking stores a pending `FixBinding` with the check key, generation, and failure evidence, then publishes [`FixRequest`](../../../stovepipe/core/messagequeue/proto/fix.proto) to `TopicKeyFix`. The message carries only the queue and episode identity. A failed shard can initiate this step while the build is running.
170+
2. **Evaluate existing work.** `fix.Controller.Process` reloads the episode and invokes [`fixer.RepairEvaluator`](../../../stovepipe/extension/fixer/fixer.go) for an unassigned failure. The evaluator compares the failure with compatible active repairs and dispatch reservations. It recommends Start, Attach with a fix ID, or Defer. Resolve detailed evidence through host-supplied dependencies.
171+
3. **Commit the decision.** Recheck the episode and selected repair, persist ownership and membership, then publish [`FixSignal`](../../../stovepipe/core/messagequeue/proto/fixsignal.proto) to `TopicKeyFixSignal`. A versioned active-work record guards new reservations; conflicting admissions reevaluate with the accepted reservation visible. Deferred requests use `delivery.Hold(...)` under a bounded waiting policy.
172+
4. **Execute and coordinate.** `fixsignal.Controller.Process` reloads the repair and episode bindings, submits accepted starts through `RepairExecutor`, and holds the delivery between observations. An attachment follows the existing repair. A worker that discovers overlap later records a reassociation intent and signals the affected episodes; the host protocol coordinates publication ownership.
173+
5. **Apply recovery.** Breakage tracking persists recovery evidence and publishes a new `FixRequest` intent. `fix` resolves an unassigned episode or signals its assigned repair; `fixsignal` reloads the bindings and updates resolution. With failure-only reports, absence establishes recovery after complete, comparable validation. Continue observing execution until it is settled or stopped.
174+
175+
The [repair RFC](fixer.md) defines the decisions, concurrency rules, persistence, and integration requirements.
176+
158177
### Designed, not built: project analysis
159178

160179
An `analyze` stage is not part of the pipeline. `stovepipe/core/messagequeue` has no analyze topic, and `stovepipe/controller` has no analyze package. The design, still only a design, is a stage that would take a build's target graph, map broken or at-risk targets to projects, and publish project-scoped builds back through `build` → `buildsignal` → `record`. That second pass, a per-project hook event, and intermediate greenness degrees are not what the controllers do. Open questions for that design stay at the bottom of this doc.
@@ -170,15 +189,18 @@ An `analyze` stage is not part of the pipeline. `stovepipe/core/messagequeue` ha
170189
| **buildanalysis** (planned) | BuildID after creation or completion; periodic redelivery | Persisted build observations | Extract check results and evidence while active and at completion; publish references for downstream analysis and breakage tracking |
171190
| **record** | RequestID | hook topic | Write whole-repo greenness; resolve project results inline; on green advance last-green URI and the promotion ref; publish the recorded or cancelled repository hook event |
172191
| **hook** | HookEvent | — | Invoke the hooks the resolver returns. The example server resolves every event to noop |
192+
| **fix** (planned) | `FixRequest`: queue, check key, generation | `FixSignal` or held delivery | Reload episode; apply Start/Attach/Defer or hand off recovery → [fixer.md](fixer.md) |
193+
| **fixsignal** (planned) | `FixSignal`: queue, fix ID | Persisted execution/resolution or held delivery | Submit accepted repairs, observe execution, and apply recorded episode recovery → [fixer.md](fixer.md) |
173194

174195
## Step RFCs
175196

176-
Per-stage design detail lives under `steps/` so this doc stays a pipeline overview:
197+
Detailed workflow contracts and behavior:
177198

178199
- [process.md](steps/process.md) — build-strategy decision, concurrency gate, backlog coalescing, [concurrency lifecycle](steps/process.md#concurrency-lifecycle), entity changes, [waiting for a slot](steps/process.md#waiting-for-a-slot)
179200
- [build.md](steps/build.md) — trigger-only stage: reads the decided scope off the Request, triggers the build-runner, hands off to buildsignal; the stovepipe `BuildRunner` contract and why it differs from SubmitQueue's
180201
- [buildsignal.md](steps/buildsignal.md) — the poll loop: hold-based re-poll cadence, per-build partitioning, and the fail-closed handoff to record
181202
- [record.md](steps/record.md) — turning a terminal build outcome into an immutable validation fact, monotonic last-green advancement and ref promotion, and the hook event announcing the outcome. Its Phase 2 / analyze handoff is the unbuilt design, not a topic `record` publishes
203+
- [fixer.md](fixer.md) — `fix` and `fixsignal` queue stages, Start/Attach/Defer evaluation, durable ownership, execution, and per-episode recovery
182204

183205
## Dedup, idempotency, and history rewrites
184206

‎stovepipe/core/messagequeue/README.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,12 @@ Payloads are defined in proto3 (`proto/`, generated into `protopb/`) and seriali
1010
- **build** (`TopicKeyBuild`, `BuildRequest`) — process publishes the request id here once its build scope (`BuildStrategy`/`BaseURI`) is decided; the build controller reloads the `Request`, triggers the build, and persists the resulting `Build`. Partitioned by request id.
1111
- **buildsignal** (`TopicKeyBuildSignal`, `BuildSignal`) — build publishes the build id here after triggering; buildsignal holds the delivery between polls until the build reaches a terminal status. Partitioned by build id, so each build's poll loop is an independent partition.
1212
- **record** (`TopicKeyRecord`, `Record`) — buildsignal publishes the request id after persisting the terminal outcome; record reloads the request and records its validation fact. Partitioned by request id.
13+
14+
## Repair stage contracts
15+
16+
The repair messages and topic keys are defined for the [repair design](../../../doc/rfc/stovepipe/fixer.md). Controller implementations and runtime registration follow in the implementation layers. Both stages use the existing `consumer.Controller` interface.
17+
18+
- **fix** (`TopicKeyFix`, [`FixRequest`](proto/fix.proto)) — breakage tracking persists an episode's repair request or recovery evidence, then publishes its queue, check key, and generation. Processing reloads the episode, evaluates Start/Attach/Defer for an unassigned failure, or signals the existing repair. Partitioned by episode identity.
19+
- **fixsignal** (`TopicKeyFixSignal`, [`FixSignal`](proto/fixsignal.proto)) — fix processing publishes the repair ID after persisting a decision or observing an assigned episode's recovery. Processing reloads the repair and its episode bindings to submit, observe execution, and apply recovery. Partitioned by repair ID, with held deliveries between observations.
20+
21+
Each durable episode update has a distinct publication intent; retries reuse that intent. A recovery update re-enters through `fix`, so processing can resolve either a pending request or an already assigned repair using the current stored ownership.

‎stovepipe/core/messagequeue/messagequeue.go‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,12 @@ type (
5353
// Record is the payload buildsignal publishes to the record stage once a
5454
// build reaches a terminal status: the build id to record.
5555
Record = protopb.Record
56+
57+
// FixRequest identifies a persisted failure episode whose assignment needs processing.
58+
FixRequest = protopb.FixRequest
59+
60+
// FixSignal identifies a persisted repair whose execution and recovery need processing.
61+
FixSignal = protopb.FixSignal
5662
)
5763

5864
// marshalOpts keeps the JSON field names identical to the proto field names

‎stovepipe/core/messagequeue/messagequeue_test.go‎

Lines changed: 42 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,45 @@ func TestRecordRoundTrip(t *testing.T) {
6666
assert.True(t, proto.Equal(rec, got), "round-tripped Record should equal the original")
6767
}
6868

69+
func TestFixMessagesRoundTrip(t *testing.T) {
70+
for _, tt := range []struct {
71+
name string
72+
message proto.Message
73+
}{
74+
{
75+
name: "episode request",
76+
message: &FixRequest{
77+
QueueName: "repo/main",
78+
Key: "checks/package:compile",
79+
Generation: 9007199254740993,
80+
},
81+
},
82+
{
83+
name: "repair signal",
84+
message: &FixSignal{QueueName: "repo/main", Id: "repair-42"},
85+
},
86+
} {
87+
t.Run(tt.name, func(t *testing.T) {
88+
data, err := Marshal(tt.message)
89+
require.NoError(t, err)
90+
got := tt.message.ProtoReflect().New().Interface()
91+
require.NoError(t, Unmarshal(data, got))
92+
assert.True(t, proto.Equal(tt.message, got))
93+
})
94+
}
95+
}
96+
97+
func TestFixRequestWireFormat(t *testing.T) {
98+
message := &FixRequest{QueueName: "repo/main", Key: "check-a", Generation: 9007199254740993}
99+
data, err := Marshal(message)
100+
require.NoError(t, err)
101+
assert.JSONEq(t, `{"queue_name":"repo/main","key":"check-a","generation":"9007199254740993"}`, string(data))
102+
103+
var got FixRequest
104+
require.NoError(t, Unmarshal([]byte(`{"queue_name":"repo/main","key":"check-a","generation":"9007199254740993","future_field":true}`), &got))
105+
assert.True(t, proto.Equal(message, &got))
106+
}
107+
69108
// TestWireFormat locks the protojson encoding decision the contract relies on:
70109
// snake_case field names (UseProtoNames).
71110
func TestWireFormat(t *testing.T) {
@@ -80,7 +119,7 @@ func TestWireFormat(t *testing.T) {
80119
// no topic_keys option names an unknown key.
81120
func TestTopicKeysBindEveryTopicKey(t *testing.T) {
82121
bound := map[string]int{}
83-
for _, m := range []proto.Message{&ProcessRequest{}, &BuildRequest{}, &BuildSignal{}, &Record{}} {
122+
for _, m := range []proto.Message{&ProcessRequest{}, &BuildRequest{}, &BuildSignal{}, &Record{}, &FixRequest{}, &FixSignal{}} {
84123
keys := TopicKeys(m)
85124
require.NotEmpty(t, keys, "message must declare a non-empty topic_keys option")
86125
for _, key := range keys {
@@ -93,6 +132,8 @@ func TestTopicKeysBindEveryTopicKey(t *testing.T) {
93132
TopicKeyBuild,
94133
TopicKeyBuildSignal,
95134
TopicKeyRecord,
135+
TopicKeyFix,
136+
TopicKeyFixSignal,
96137
}
97138

98139
valid := map[string]bool{}

‎stovepipe/core/messagequeue/proto/BUILD.bazel‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@ exports_files(
22
[
33
"build.proto",
44
"buildsignal.proto",
5+
"fix.proto",
6+
"fixsignal.proto",
57
"process.proto",
68
"record.proto",
79
],
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
// Copyright (c) 2026 Uber Technologies, Inc.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
syntax = "proto3";
16+
17+
package uber.stovepipe.messagequeue;
18+
19+
import "api/base/messagequeue/proto/messagequeue.proto";
20+
21+
option go_package = "github.com/uber/submitqueue/stovepipe/core/messagequeue/protopb";
22+
option java_multiple_files = true;
23+
option java_outer_classname = "FixRequestProto";
24+
option java_package = "com.uber.submitqueue.stovepipe.messagequeue";
25+
26+
// FixRequest references a durable episode request; evidence and assignment are loaded from storage.
27+
// Publishing requires the episode state to have been persisted first.
28+
message FixRequest {
29+
option (uber.base.messagequeue.topic_keys) = "fix";
30+
31+
// queue_name identifies the validation queue containing the episode.
32+
string queue_name = 1;
33+
// key is the opaque check identity within the queue.
34+
string key = 2;
35+
// generation distinguishes a recurrence after recovery from the previous failure episode.
36+
int64 generation = 3;
37+
}
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
// Copyright (c) 2026 Uber Technologies, Inc.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
syntax = "proto3";
16+
17+
package uber.stovepipe.messagequeue;
18+
19+
import "api/base/messagequeue/proto/messagequeue.proto";
20+
21+
option go_package = "github.com/uber/submitqueue/stovepipe/core/messagequeue/protopb";
22+
option java_multiple_files = true;
23+
option java_outer_classname = "FixSignalProto";
24+
option java_package = "com.uber.submitqueue.stovepipe.messagequeue";
25+
26+
// FixSignal references a durable repair; execution and episode recovery are loaded from storage.
27+
// Publishing requires the repair and its episode associations to have been persisted first.
28+
message FixSignal {
29+
option (uber.base.messagequeue.topic_keys) = "fixsignal";
30+
31+
// id is the queue-scoped repair identifier.
32+
string id = 1;
33+
// queue_name identifies the validation queue containing the repair.
34+
string queue_name = 2;
35+
}

‎stovepipe/core/messagequeue/protopb/BUILD.bazel‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ go_library(
55
srcs = [
66
"build.pb.go",
77
"buildsignal.pb.go",
8+
"fix.pb.go",
9+
"fixsignal.pb.go",
810
"process.pb.go",
911
"record.pb.go",
1012
],

0 commit comments

Comments
 (0)