From b16c34a329554930cdcd7218c828057b61a49735 Mon Sep 17 00:00:00 2001 From: dmmdea Date: Thu, 12 Nov 2026 09:54:58 -0500 Subject: [PATCH 1/5] Stamp the build on ledger and corpus rows; split failures by build Ledger.Record, the one path every ledger row takes to disk, now stamps the build of the writing process from the same self-hash the delegation log uses, under the same field names: harness_build_sha256 on a row for work the process ran, delegator_build_sha256 on a delegator's row. A delegation row also takes the executing node's harness_build_sha256 and seat_config_sha256 from the node's result, and the node's own agent row carries seat_config_sha256, only when the seat served. All fields are omitted when empty, so older rows read as before. trace-corpus harvest copies the three fields from the delegation-log row into the corpus row. None of the corpus rows kept the delegator build before. ledger --failures --by-build adds a builds key and a stderr table: rows are bucketed by the executing build, else delegator:, else unstamped, and the buckets add up to the unsplit totals. Output without the flag is unchanged. The failure-register, mcp-server and self-improvement pages now state exactly which rows carry which stamp. Co-Authored-By: Claude Fable 5.2 --- .../unreleased/feat-rsi-register-followups.md | 7 +++ docs/systems/failure-register.md | 20 +++++- docs/systems/mcp-server.md | 13 +++- docs/systems/self-improvement.md | 5 ++ internal/core/types.go | 3 + internal/delegate/run.go | 30 +++++---- internal/ledger/build_stamp.go | 32 ++++++++++ internal/ledger/failures.go | 63 ++++++++++++++++++- internal/ledger/ledger.go | 17 ++++- internal/pipeline/agenttask.go | 1 + internal/pipeline/pipeline.go | 2 + ledger_failures.go | 23 ++++++- main.go | 2 +- trace_corpus.go | 23 ++++++- 14 files changed, 213 insertions(+), 28 deletions(-) create mode 100644 changes/unreleased/feat-rsi-register-followups.md create mode 100644 internal/ledger/build_stamp.go diff --git a/changes/unreleased/feat-rsi-register-followups.md b/changes/unreleased/feat-rsi-register-followups.md new file mode 100644 index 000000000..2df611b27 --- /dev/null +++ b/changes/unreleased/feat-rsi-register-followups.md @@ -0,0 +1,7 @@ +### Added - ledger rows and trace-corpus rows record which build produced them, and `ledger --failures --by-build` splits failures by build + +- **Ledger stamps.** `Record`, the one path every ledger row takes to disk, stamps the build of the writing process from the same self-hash and under the same names as the delegation log: `harness_build_sha256` on a row for work the process ran itself, `delegator_build_sha256` on a delegator's row (a delegation row and its dispatch marker, and any row naming a remote node). A delegation row also carries the executing node's `harness_build_sha256` and `seat_config_sha256` from the node's result, and the node's own agent row carries `seat_config_sha256`, both only when the seat served. All three are omitted when empty; older rows still read and simply have none. No caller of `Record` changed except the delegate runner and the agent task, which hand over the two values only they hold. +- **Trace corpus.** `trace-corpus harvest` copies `delegator_build_sha256`, `harness_build_sha256` and `seat_config_sha256` from the delegation-log row into the corpus row, omitted when the log row has none. Before this the delegator build was on 17,054 of 17,095 log rows and on none of the 12,322 corpus rows. +- **Reader.** `local-offload ledger --failures --by-build` adds a `builds` key (build, job rows, failure rows, per-class counts) and a short table on stderr. A row is bucketed by the build that ran the work, else `delegator:`, else `unstamped`; the buckets add up to the report's job and failure totals. Without the flag the output is unchanged. +- **Docs.** The failure-register, mcp-server and self-improvement pages now say exactly which rows carry which stamp. +- **Check.** Built binary on the real ledger, read-only: `--by-build` puts all 10,942 failures (of 27,908 job rows) under `unstamped`, totals equal to the unsplit report, every other key identical. A classify call and two local delegations through the same binary into a scratch ledger wrote rows with the stamps, and the split showed them under that hash. `trace-corpus harvest` of the real delegation log into a scratch file: 12,291 of 12,329 corpus rows carry the delegator build, 11,652 the executing build, 10,645 the seat config. diff --git a/docs/systems/failure-register.md b/docs/systems/failure-register.md index 3f7ffd928..4ef82a9a7 100644 --- a/docs/systems/failure-register.md +++ b/docs/systems/failure-register.md @@ -109,6 +109,19 @@ a share of `failure_rows` and counts only the classes filed as stack faults: an | `classes[].cards_tokens`, `rows_with_tokens` | the card work the class spent, and how many rows measured it | | `classes[].documented`, `investigated`, `meaning`, `root_cause`, `structural_fix` | the joined table row, and whether a person wrote a root cause or a fix in it | +**Failures by build (`--by-build`).** The ledger stamps the build that wrote each row, in `Record`, from the same +self-hash the delegation log uses and under its names: `harness_build_sha256` on a row for work the writing process ran +itself, `delegator_build_sha256` on a delegator's row (a delegation row and its dispatch marker, and any row naming a +remote node), and on a delegation row also the executing node's `harness_build_sha256` and, when its seat served, +`seat_config_sha256`, as the node's result reported them. The `builds` key holds one entry per bucket, +`{build, job_rows, failure_rows, classes}`, failure rows descending: its `job_rows` add up to the report's `job_rows` +and its `failure_rows` to `failure_rows`. A row is bucketed by `harness_build_sha256` when it has one (the build that +ran the work); else by `delegator_build_sha256`, named `delegator:` (the job never reached a node that reported a +build, or the node predates the stamp, so only the delegator's build is known); else `unstamped`. Every row written +before the stamp, and any row of a process whose self-hash failed, is `unstamped`: it is unknown, never a guess, and +that bucket shrinks only as new rows come in. The split is by build only; `seat_config_sha256` is on the rows but not a +bucket. + **Door, seat and model.** The ledger has no seat or model column. `model_tier` is its one identity column: on a delegation row (task `agent_delegate`) it is `:`, on every other row the model alias. The report reads the **door** as recorded (`(unknown)` when a row has none) and the **seat** as `model_tier` as recorded. The @@ -232,9 +245,10 @@ code; this table is checked against both. ## Interfaces and entry points -- `local-offload ledger --failures [--since DAYS] [--include-test-traffic] [--table FILE] [--config FILE]`: the - report, JSON on stdout and nothing else. A note about the left-out traffic, a missing ledger or lines it could not - read goes to stderr. `--failures=false` is the savings summary, as for any boolean flag. +- `local-offload ledger --failures [--by-build] [--since DAYS] [--include-test-traffic] [--table FILE] [--config FILE]`: + the report, JSON on stdout and nothing else. A note about the left-out traffic, a missing ledger or lines it could not + read goes to stderr. `--failures=false` is the savings summary, as for any boolean flag. `--by-build` adds the `builds` + key (below) and prints the same split as a short table on stderr; without it the output is what it was. ## Invariants and assumptions diff --git a/docs/systems/mcp-server.md b/docs/systems/mcp-server.md index 42c39ea20..88e21a6dc 100644 --- a/docs/systems/mcp-server.md +++ b/docs/systems/mcp-server.md @@ -1412,8 +1412,14 @@ read-only unless deliberately widened. See second swap landing in that window fail on a sharing violation, with node-swap's holder diagnosis unable to see the stale server (its image path was renamed away). **Counting a rollout (runbook).** A rollout's effect (a gate that reads ledger rows after a deploy) counts only the rows whose - `harness_build_sha256` equals the deployed build's hash: a session on an old image keeps writing the old hash until it is - reconnected, so the rows themselves say which sessions are not on the build. The operator's deploy script reports the MCP servers + build stamp equals the deployed build's hash: a session on an old image keeps writing the old hash until it is + reconnected, so the rows themselves say which sessions are not on the build. The ledger stamps the build on every row it + writes, in `Record`, from the same hash the delegation log uses: a row for work this process ran carries + `harness_build_sha256`; a delegation row (task `agent_delegate`, its dispatch marker included) and any row naming a remote + node, written by the delegator, carry `delegator_build_sha256`, and a delegation row also carries the executing node's + `harness_build_sha256` and `seat_config_sha256` as the node's result reported them (a run whose seat served). A row written + before the stamp, or by a process whose self-hash failed, carries none. `local-offload ledger --failures --by-build` splits + the failures by these stamps ([failure-register.md](failure-register.md)). The operator's deploy script reports the MCP servers it finds still on an older image at the end of each Windows leg and in its deploy record, under "Sessions still on an old image" (process id and start time, and the session when the owners registry names it). It is information only: it never changes a node's verdict and never stops anyone's server. @@ -1474,7 +1480,8 @@ read-only unless deliberately widened. See deploy replaces the installed binary while every MCP server already running keeps the image it started from, so a session could write rows with old behaviour for days and nothing said so (the register counted 13 of 18 MCP processes older than the deployed build). The block is `{version, build_sha256, installed_version?, stale_image?, note?}`: - the version and the build hash this process stamps on its rows, and the comparison with the file now at the path it + the version and the build hash this process stamps on the rows it writes (`harness_build_sha256`, or + `delegator_build_sha256` on a delegator's row), and the comparison with the file now at the path it started from. `stale_image` is `true` when the installed bytes differ, `false` when they are equal, and **absent with a `note` when the installed file cannot be read** (never "fresh" on a failure); `installed_version` is the build marker the installed file carries. `mcpserver.Run` takes the identity from `buildinfo.CaptureStartup()` and holds it on diff --git a/docs/systems/self-improvement.md b/docs/systems/self-improvement.md index 2659e114d..c1ba8cde9 100644 --- a/docs/systems/self-improvement.md +++ b/docs/systems/self-improvement.md @@ -122,6 +122,11 @@ refuses one), so a baseline is never replaced and a new one goes to a new `--out for byte and refuses a baseline taken from a different corpus. `readout`, `freeze` and `check` are deterministic and model-free, and the report carries no timestamp. +A harvested delegation row carries three stamps over from its log row, under the log's names and omitted when the log row +has none: `delegator_build_sha256` (the delegator's build), `harness_build_sha256` (the executing node's build) and +`seat_config_sha256` (the seat's config pin). A row from before the stamps, or whose seat never served, carries none: +unknown, never a value. The readout does not use them yet; they let a reader split a corpus by build. + `harvest --since` and `--until` bound the delegation shards by their day, inclusive. Each must be a real `YYYY-MM-DD` date and `--since` must not be after `--until`; anything else is refused with exit 2 before a shard is read, so a typo cannot silently widen or narrow the window. `trace-corpus help`, `--help` and `-h` print the usage with the flags of every diff --git a/internal/core/types.go b/internal/core/types.go index 7c1e81e98..db76b4a61 100644 --- a/internal/core/types.go +++ b/internal/core/types.go @@ -363,6 +363,9 @@ type Meta struct { Steps int `json:"steps,omitempty"` StopReason string `json:"stop_reason,omitempty"` RepackMs int64 `json:"repack_ms,omitempty"` + // SeatConfigSHA256 is the seat-config pin of an agent run whose seat served (the wire result's field of the same + // name), carried to the ledger row. In-process only (`json:"-"`): the wire result publishes it already. + SeatConfigSHA256 string `json:"-"` // RepackAttempts: the re-pack's completion count beside its wall, so the // ledger row can separate one slow attempt from a three-attempt loop — // the figure the D-129 engine route is measured on. diff --git a/internal/delegate/run.go b/internal/delegate/run.go index 512d3574b..bd93e713e 100644 --- a/internal/delegate/run.go +++ b/internal/delegate/run.go @@ -1406,19 +1406,23 @@ func (r *runner) record(contract core.AgentContract, pr PlacedResult) { // on this box has none) and the closed-set reason code — so a reader // never joins on latency or greps prose. The reason above is stored // whole: the ledger no longer cuts it. - Door: doorOf(contract), - FleetJobID: fleetJobIDOf(pr), - ReasonCode: reasonCodeFor(pr), - Route: r.route, - RouteAsked: routeAsked, - PinReason: pinReason, - Placement: placement, - Steps: pr.Result.Steps, - StopReason: pr.Result.StopReason, - RepackMs: pr.Result.RepackMs, - RepackAttempts: pr.Result.RepackAttempts, - QueuedMs: pr.Result.QueuedMs, - AcceptanceResult: acceptanceResult(pr), + Door: doorOf(contract), + FleetJobID: fleetJobIDOf(pr), + ReasonCode: reasonCodeFor(pr), + Route: r.route, + RouteAsked: routeAsked, + PinReason: pinReason, + Placement: placement, + // The executing node's build and seat config, as its result reported them (the delegation log's + // result block carries the same two); Record stamps this process's own build as the delegator's. + HarnessBuildSHA256: pr.Result.HarnessBuildSHA256, + SeatConfigSHA256: pr.Result.SeatConfigSHA256, + Steps: pr.Result.Steps, + StopReason: pr.Result.StopReason, + RepackMs: pr.Result.RepackMs, + RepackAttempts: pr.Result.RepackAttempts, + QueuedMs: pr.Result.QueuedMs, + AcceptanceResult: acceptanceResult(pr), // The same mark the corpus line carries (delegationLogLine.Refusal): a place in line, not the subtask's outcome. Refusal: pr.refusalRow, }); err != nil { diff --git a/internal/ledger/build_stamp.go b/internal/ledger/build_stamp.go new file mode 100644 index 000000000..a5a8d09c9 --- /dev/null +++ b/internal/ledger/build_stamp.go @@ -0,0 +1,32 @@ +package ledger + +import "github.com/dmmdea/offload-harness/internal/buildinfo" + +// stampBuild puts this process's build on a row, on the one path every writer takes (Record), so no record site +// can forget it. It reads the same source the delegation log does (buildinfo.BuildSHA256, computed once per +// process) and uses the delegation log's field names. +// +// Which field depends on whose work the row records: +// +// - a delegation row (task agent_delegate, its dispatch marker included) and any row naming a remote node are +// written by the DELEGATOR, so this process's build is delegator_build_sha256, as in the delegation log; +// - every other row records work this process ran, so it is harness_build_sha256. +// +// A caller that holds the executing node's own build (the node's result carries it) sets harness_build_sha256 +// itself; a value already on the row is never replaced. A process whose self-hash failed stamps nothing (the +// failure was logged once where it happened): the row is then unstamped, never stamped with a guess. +func stampBuild(e *Entry) { + build := buildinfo.BuildSHA256() + if build == "" { + return + } + if e.Task == delegationTask || e.Node != "" || e.NodeID != "" { + if e.DelegatorBuildSHA256 == "" { + e.DelegatorBuildSHA256 = build + } + return + } + if e.HarnessBuildSHA256 == "" { + e.HarnessBuildSHA256 = build + } +} diff --git a/internal/ledger/failures.go b/internal/ledger/failures.go index 240087e49..65b00bb2c 100644 --- a/internal/ledger/failures.go +++ b/internal/ledger/failures.go @@ -123,6 +123,44 @@ type FailureRegister struct { ExcludedNonProduction map[string]int `json:"excluded_non_production,omitempty"` // Classes are ordered by Count descending, then Class: the same ledger gives the same bytes. Classes []FailureClass `json:"classes"` + // Builds is the same job rows split by the build that produced them (see FailureBuild), failure rows descending, + // then job rows, then name. Its failure rows add up to FailureRows and its job rows to JobRows. + Builds []FailureBuild `json:"builds"` +} + +// UnstampedBuild is the bucket of the rows that carry no build stamp: every row written before the ledger stamped +// them, and a row written by a process whose self-hash failed. +const UnstampedBuild = "unstamped" + +// delegatorBuildPrefix marks a bucket named by the DELEGATOR's build: a row that carries no executing build (the +// job never reached a node that reported one, or the node predates the stamp) and so can only say which build +// of the delegator wrote it. +const delegatorBuildPrefix = "delegator:" + +// FailureBuild is one build's part of the register: the job rows it produced, the failed ones, and their classes. +// Build is the executing build (harness_build_sha256) when the row has one, else delegator_build_sha256 with a +// "delegator:" prefix, else "unstamped". A name is never a guess: a row is in "unstamped" because it holds neither. +type FailureBuild struct { + Build string `json:"build"` + JobRows int `json:"job_rows"` + FailureRows int `json:"failure_rows"` + Classes []FailureCount `json:"classes"` +} + +// buildBucket is the register's build bucket for a row (see FailureBuild). +func buildBucket(e Entry) string { + switch { + case e.HarnessBuildSHA256 != "": + return e.HarnessBuildSHA256 + case e.DelegatorBuildSHA256 != "": + return delegatorBuildPrefix + e.DelegatorBuildSHA256 + } + return UnstampedBuild +} + +type buildAcc struct { + jobs, failures int + classes map[string]int } // failureRow is a ledger row and the one fact Entry cannot carry: whether the line had the cards_tokens key. @@ -301,7 +339,7 @@ type classAcc struct { // file readers: a last line still being written is not a row, and is counted in UnparsedLines with every other line // that does not decode. func FailureRegisterFile(path string, since int64, includeTraffic bool) (FailureRegister, error) { - reg := FailureRegister{Classes: []FailureClass{}, Kinds: []FailureKindCount{}} + reg := FailureRegister{Classes: []FailureClass{}, Kinds: []FailureKindCount{}, Builds: []FailureBuild{}} f, err := os.Open(path) if err != nil { if os.IsNotExist(err) { @@ -334,6 +372,7 @@ func FailureRegisterFile(path string, since int64, includeTraffic bool) (Failure jobs := jobClasses(all) accs := map[string]*classAcc{} + builds := map[string]*buildAcc{} var total tokenSum for _, r := range rows { e := r.Entry @@ -359,10 +398,18 @@ func FailureRegisterFile(path string, since int64, includeTraffic bool) (Failure continue } reg.JobRows++ + ba := builds[buildBucket(e)] + if ba == nil { + ba = &buildAcc{classes: map[string]int{}} + builds[buildBucket(e)] = ba + } + ba.jobs++ class, derived, failed := failureClassOf(e) if !failed { continue } + ba.failures++ + ba.classes[class]++ reg.FailureRows++ if derived { reg.ClassDerivedRows++ @@ -400,6 +447,20 @@ func FailureRegisterFile(path string, since int64, includeTraffic bool) (Failure } reg.CardsTokens, reg.RowsWithTokens = total.ptr(), total.measured + reg.Builds = make([]FailureBuild, 0, len(builds)) + for name, b := range builds { + reg.Builds = append(reg.Builds, FailureBuild{Build: name, JobRows: b.jobs, FailureRows: b.failures, Classes: sortedCounts(b.classes)}) + } + sort.Slice(reg.Builds, func(i, j int) bool { + x, y := reg.Builds[i], reg.Builds[j] + switch { + case x.FailureRows != y.FailureRows: + return x.FailureRows > y.FailureRows + case x.JobRows != y.JobRows: + return x.JobRows > y.JobRows + } + return x.Build < y.Build + }) kinds := map[string]int{} for class, a := range accs { c := FailureClass{ diff --git a/internal/ledger/ledger.go b/internal/ledger/ledger.go index 9eac398a9..dfe56507c 100644 --- a/internal/ledger/ledger.go +++ b/internal/ledger/ledger.go @@ -239,9 +239,19 @@ type Entry struct { RouteAsked string `json:"route_asked,omitempty"` PinReason string `json:"pin_reason,omitempty"` Placement string `json:"placement,omitempty"` - Steps int `json:"steps,omitempty"` - StopReason string `json:"stop_reason,omitempty"` - RepackMs int64 `json:"repack_ms,omitempty"` + // HarnessBuildSHA256 is the build (the SHA-256 of the executable) that RAN the work the row records, the field + // the delegation log's node result calls by the same name. Record stamps this process's own on a row it wrote + // for work it ran itself; a delegator's row (delegation, or any row naming a remote node) is stamped + // DelegatorBuildSHA256 instead and takes this one from the node's result when the caller has it. + // SeatConfigSHA256 pins the seat's server config (the node's seat-config probe) and is set only by a writer that + // holds it: a row for a run whose seat served. All three are omitted when empty; a row from an older build has + // none, which reads as unknown, never as a value (build_stamp.go). + HarnessBuildSHA256 string `json:"harness_build_sha256,omitempty"` + DelegatorBuildSHA256 string `json:"delegator_build_sha256,omitempty"` + SeatConfigSHA256 string `json:"seat_config_sha256,omitempty"` + Steps int `json:"steps,omitempty"` + StopReason string `json:"stop_reason,omitempty"` + RepackMs int64 `json:"repack_ms,omitempty"` // RepackAttempts is how many seat completions the structured re-pack // spent. It rides beside RepackMs because the wall alone cannot tell a // slow single attempt from a loop that re-generated three times, and the @@ -412,6 +422,7 @@ func (l *Ledger) Record(e Entry) error { // Provenance and the one token figure are stamped HERE, on the one path // every writer takes, so no record site can forget them (D-101). stampOrigin(&e) + stampBuild(&e) settleRunClass(&e, l.class) switch { case e.ParentJobID != "": diff --git a/internal/pipeline/agenttask.go b/internal/pipeline/agenttask.go index 04da03f30..54c0e2a34 100644 --- a/internal/pipeline/agenttask.go +++ b/internal/pipeline/agenttask.go @@ -204,6 +204,7 @@ func (p *Pipeline) runAgentTask(ctx context.Context, req core.Request, meta core meta.StopReason = w.StopReason meta.RepackMs = w.RepackMs meta.RepackAttempts = w.RepackAttempts + meta.SeatConfigSHA256 = w.SeatConfigSHA256 if jid, _ := req.Params["job_id"].(string); jid != "" { meta.JobID = jid } diff --git a/internal/pipeline/pipeline.go b/internal/pipeline/pipeline.go index 24589d7cf..1555fa6ab 100644 --- a/internal/pipeline/pipeline.go +++ b/internal/pipeline/pipeline.go @@ -1014,6 +1014,8 @@ func entryFrom(task core.TaskType, meta core.Meta, deferred bool, inputChars int StopReason: meta.StopReason, RepackMs: meta.RepackMs, RepackAttempts: meta.RepackAttempts, + // The seat-config pin of an agent run whose seat served (empty — omitted — on every other row). + SeatConfigSHA256: meta.SeatConfigSHA256, // The surface that admitted the call (A-102): "offload_summarize", // "cli:summarize", "fleet". Empty when no door stamped the request. Door: meta.Door, diff --git a/ledger_failures.go b/ledger_failures.go index 7ce63a9c8..2ab8073f2 100644 --- a/ledger_failures.go +++ b/ledger_failures.go @@ -81,6 +81,13 @@ type failureReport struct { Classes []failureClassOut `json:"classes"` } +// failureReportByBuild is the report with --by-build: the same object, and one more key at the end. Without the flag +// the report is failureReport and carries no "builds" key, so its bytes are what they were. +type failureReportByBuild struct { + failureReport + Builds []ledger.FailureBuild `json:"builds"` +} + // runLedgerMode is the ledger verb's entry (the one line main.go changed): the failure register when --failures is // asked for, the savings summary otherwise. The summary's own flag set does not define --failures, so a // --failures=false that reaches it would exit 2 with a usage dump; the flag is a mode switch decided here, and @@ -148,6 +155,7 @@ func runLedgerFailures(args []string) error { fs.Bool("failures", true, "print the failure-class register instead of the savings summary") since := fs.Int("since", 0, "only count entries from the last N days") includeTraffic := includeTrafficFlag(fs) + byBuild := fs.Bool("by-build", false, "also split the failures by the build that produced the row (JSON key \"builds\", and a short table on stderr); rows with no stamp are one bucket named \"unstamped\"") table := fs.String("table", "", "markdown file holding the failure-class table (default: docs/systems/failure-register.md as embedded in this build)") _ = fs.Parse(args) cfg := loadCfg(fs) @@ -178,7 +186,12 @@ func runLedgerFailures(args []string) error { if *since > 0 { report.SinceDays = *since } - b, err := json.MarshalIndent(report, "", " ") + var out any = report + if *byBuild { + out = failureReportByBuild{failureReport: report, Builds: reg.Builds} + noteBuilds(reg) + } + b, err := json.MarshalIndent(out, "", " ") if err != nil { return err } @@ -186,6 +199,14 @@ func runLedgerFailures(args []string) error { return nil } +// noteBuilds prints the build split as a table on stderr, for a person at the terminal; stdout stays the one JSON value. +func noteBuilds(reg ledger.FailureRegister) { + fmt.Fprintf(os.Stderr, "failures by build (%d failed of %d job rows):\n", reg.FailureRows, reg.JobRows) + for _, b := range reg.Builds { + fmt.Fprintf(os.Stderr, " %-72s %6d failed of %6d job rows\n", b.Build, b.FailureRows, b.JobRows) + } +} + // noteLedgerReading says on stderr what the JSON also says, so a person looking at the terminal sees it: the ledger // was not there (every count is then zero because nothing was read), or some of its lines could not be read. func noteLedgerReading(path string, reg ledger.FailureRegister) { diff --git a/main.go b/main.go index 2ff4980d9..020925c5f 100644 --- a/main.go +++ b/main.go @@ -341,7 +341,7 @@ Usage: local-offload fleet-smoke [--remote URL]... [--timeout 120] [--json] send one grounded one-step contract to EVERY node and print where each landed (node, seat, placement, wall, verdict); non-zero unless all PASS local-offload top [--ui http://127.0.0.1:18813] [--interval 5s] terminal view of the fleet overview (for headless boxes; reads a running fleet-ui) local-offload rig --seat [--since 7d] [--node ID] [--out report.json] [--json] the seat rigger (ADR 0036 P3a): classify this box's delegation-log failures for a seat onto ONE axis each (published precedence), weights over eligible rows, evidence job ids, the pre-authored remedy per axis — proposes and applies nothing - local-offload ledger [--since DAYS] [--failures] [--table FILE] [--include-test-traffic] token-savings report; --failures prints the failure-class register instead (docs/systems/failure-register.md); --table (with --failures) names the register's class table, --include-test-traffic counts bench and test rows too + local-offload ledger [--since DAYS] [--failures [--by-build]] [--table FILE] [--include-test-traffic] token-savings report; --failures prints the failure-class register instead (docs/systems/failure-register.md), --by-build adds the split by the build that produced each row; --table (with --failures) names the register's class table, --include-test-traffic counts bench and test rows too local-offload doctor [--config FILE] [--probe-completion] check endpoint health + config, then the engine answering each lane and seat (--probe-completion sends a 1-token completion and can load one idle seat on an idle box) local-offload audit-yaml [--against-render] [--other-box] FILE... check live llama-swap config(s) against the operator rules (INV-1/INV-2: no -ngl 0, no empty CUDA_VISIBLE_DEVICES, ttl 300 everywhere, no persistent group, no preload). diff --git a/trace_corpus.go b/trace_corpus.go index e2584a051..da5d7107d 100644 --- a/trace_corpus.go +++ b/trace_corpus.go @@ -123,6 +123,14 @@ type tcRun struct { // compaction ladder (elision markers, skeletons). It exists only for agent traces, which keep // a transcript; nil on a delegation row. It is a LOWER BOUND on whether the ladder fired. LadderArtifacts *int `json:"ladder_artifacts"` + // DelegatorBuildSHA256 is the build of the delegator that wrote the delegation-log row, HarnessBuildSHA256 the build + // of the node that ran the work and SeatConfigSHA256 the pin of the seat's server config, copied from the row under + // the names the row uses. They let a reader split the corpus by build. Delegation rows only, omitted when the row + // carries none (a row from before the stamps, an agent trace, or a run whose seat never served), which a reader + // must treat as unknown, never as a value. + DelegatorBuildSHA256 string `json:"delegator_build_sha256,omitempty"` + HarnessBuildSHA256 string `json:"harness_build_sha256,omitempty"` + SeatConfigSHA256 string `json:"seat_config_sha256,omitempty"` } // ---- reading the records ---- @@ -181,9 +189,11 @@ type tcTraceIn struct { // tcDelegationIn is the part of a delegation-log row the harvest reads. type tcDelegationIn struct { - Seat string `json:"seat"` - Node string `json:"node"` - Refusal *struct { + Seat string `json:"seat"` + Node string `json:"node"` + // The delegator's own build, stamped on the row itself; the executing node's build and seat pin are in Result. + DelegatorBuildSHA256 string `json:"delegator_build_sha256"` + Refusal *struct { Status int `json:"status"` } `json:"refusal"` Result *tcDelegationResult `json:"result"` @@ -199,6 +209,9 @@ type tcDelegationResult struct { CacheTokens int64 `json:"cache_tokens"` SeatTokensIn int64 `json:"seat_tokens_in"` Pager *tcPagerIn `json:"pager"` + // The node's build and seat-config pin (omitted by a node that predates them). + HarnessBuildSHA256 string `json:"harness_build_sha256"` + SeatConfigSHA256 string `json:"seat_config_sha256"` } // neverRan reports a logged subtask whose agent never executed: a deferral (capacity, @@ -490,6 +503,10 @@ func tcHarvestDelegation(o tcHarvestOpts, stats *tcHarvestStats) ([]tcRun, error TraceBearing: len(res.Trace) > 0, Steps: res.Steps, StopReason: res.StopReason, + + DelegatorBuildSHA256: in.DelegatorBuildSHA256, + HarnessBuildSHA256: res.HarnessBuildSHA256, + SeatConfigSHA256: res.SeatConfigSHA256, } pg, ok := res.Pager.run() if !ok { From 515135c8374d6164674a734a711467a1646c2364 Mon Sep 17 00:00:00 2001 From: dmmdea Date: Thu, 12 Nov 2026 10:10:07 -0500 Subject: [PATCH 2/5] State the proposer trigger as a printed count of measured runs, and record a root cause per failure class ADR 0036's trigger for building the recipe proposer read "trace-bearing runs in the hundreds" and read as met on 10,766 runs, while 31 carry a measured pager report, the measurement the proposer's design needs. A dated amendment restates it on that count (needed 100), and `trace-corpus trigger` (and a stderr line of `trace-corpus readout`) prints it as needed / have / met. The readout report is unchanged, so frozen baselines still reproduce. ADR 0091 is Accepted with its release and a status note on what its data now shows. `internal/ledger/rootcause.go` records, for each of the 31 failure classes, a cause kind, a root cause, the evidence and an action, from the emitting code and the real 30-day rows. `ledger --failures` prints them with each class (JSON `cause`, text on stderr) and lists the classes still not investigated (15 of 31; 4 of the 19 in the window). Existing fields are unchanged; the fixture golden gains the new keys. The table is on the failure-register page. Co-Authored-By: Claude Fable 5.2 --- .../unreleased/feat-rsi-trigger-rootcause.md | 6 + ...e-agent-lane-is-a-harnessed-environment.md | 20 ++ ...ger-refetch-compaction-fire-and-prefill.md | 15 +- docs/architecture/decisions/README.md | 2 +- docs/systems/coding-agent.md | 2 +- docs/systems/failure-register.md | 77 +++++++- docs/systems/self-improvement.md | 20 +- internal/ledger/rootcause.go | 175 ++++++++++++++++++ ledger_failures.go | 26 ++- testdata/failure-register/expected.json | 96 +++++++++- trace_corpus.go | 73 +++++++- 11 files changed, 480 insertions(+), 32 deletions(-) create mode 100644 changes/unreleased/feat-rsi-trigger-rootcause.md create mode 100644 internal/ledger/rootcause.go diff --git a/changes/unreleased/feat-rsi-trigger-rootcause.md b/changes/unreleased/feat-rsi-trigger-rootcause.md new file mode 100644 index 000000000..7e88f98a6 --- /dev/null +++ b/changes/unreleased/feat-rsi-trigger-rootcause.md @@ -0,0 +1,6 @@ +### Added - the recipe proposer's trigger is a printed count of measured runs, and every failure class carries a root cause and an action + +- **Proposer trigger (ADR 0036, amendment 2026-11-12).** The trigger for building the recipe proposer read "trace-bearing runs in the hundreds", and the corpus has 10,766 of those. The runs a proposer can learn from carry a measured pager report (`pager.measured_runs`: the loop evicted something, so the re-fetch question has an answer for that run), and the corpus has 31 of 12,332. The trigger is restated on that count, as a dated amendment, and the verb prints it: `trace-corpus trigger --corpus FILE` prints JSON (`needed`, `have`, `met`, the trace-bearing and total counts, the largest single seat's count) and `trace-corpus readout` prints the same as one stderr line. On the real corpus it reads `needed 100 / have 31 / NOT MET`; the largest single named seat has 10. The readout report itself is unchanged, so a frozen baseline still reproduces. The seat's frozen pre-registration, the other half of the trigger, is not in the corpus and is not judged. No proposer is built. +- **ADR 0091** is Accepted: the verb, the harvest, the readout and the pager and pass-count recording shipped in 0.206.0. Its status note records that the pager gate now reads `BELOW_GATE` on 31 runs (8 of 86 evicted payloads re-fetched) where its first readout read `NO_DATA`, and that 31 runs do not settle the `edit_context` question. +- **Root causes.** `internal/ledger/rootcause.go` holds, for each of the 31 failure classes, a cause kind (capacity signal, node configuration, harness defect, caller contract, upstream service, unknown), a root cause, the evidence (the emitting function and the 30-day count) and an action. `ledger --failures` prints them with each class as `classes[].cause` in the JSON and as text on stderr, with `causes_not_investigated` listing the open classes present in the window; the existing fields are unchanged. Where the code and the rows do not settle a cause the entry says `not investigated` and what would settle it: 15 of the 31 classes (4 of the 19 in the last 30 days), never a guessed cause. The table is on the failure-register page. +- **Check.** Built binary on the real corpus and ledger, read-only: `trace-corpus trigger` prints `needed 100 / have 31 / NOT MET` of 10,766 trace-bearing and 12,332 runs, matching the installed 0.232.0 build's counts; `ledger --failures --since 30` prints all 19 classes in the window with cause and action, 4 not investigated (`failed_verification`, `other`, `poll_deadline`, `stall_cold_load`), and every pre-existing field matches the build before the change apart from live rows written in between. diff --git a/docs/architecture/decisions/0036-the-agent-lane-is-a-harnessed-environment.md b/docs/architecture/decisions/0036-the-agent-lane-is-a-harnessed-environment.md index 4b5160329..1a7e467c0 100644 --- a/docs/architecture/decisions/0036-the-agent-lane-is-a-harnessed-environment.md +++ b/docs/architecture/decisions/0036-the-agent-lane-is-a-harnessed-environment.md @@ -173,6 +173,26 @@ one (`local-offload recipe-audit`). Decided, with the reasons on record: readout` reports trace-bearing runs in the hundreds, and the seat the proposer is for has a frozen pre-registration to be audited against. +### Amendment 2026-11-12 — the proposer trigger counts measured runs, and a verb prints it + +The 2026-10-15 trigger is written on "trace-bearing runs ... in the hundreds". That count answers whether a run kept a +step trace, not whether the run can teach a proposer anything: the proposer's candidates (`deny_tools`, `allow_tools`, +`max_calls_per_tool`, and a live A/B for the rest) are judged on what the pager and the trace say about the +run, and a run whose pager recorded no eviction says nothing about whether paged-out context was needed again. On the +operator's machine on 2026-11-12 the corpus held 12,332 runs, 10,766 of them trace-bearing (the raw count that read as +"in the hundreds") and 31 with a measured pager report (`pager.measured_runs`: the loop evicted something, so the +re-fetch question has an answer for that run); 5,227 recorded a pager report with nothing evicted and 7,074 recorded none. +The largest single named seat had 10. Both figures were read with the installed 0.232.0 build. + +Restated, as a number a verb prints. The reopen trigger is now: **`trace-corpus trigger` (and the stderr line of +`trace-corpus readout`) reports `have` at or above `needed`**, where `needed` is 100, the lowest reading of "in the +hundreds", and `have` is the corpus's `pager.measured_runs`; and the seat the proposer is for has a frozen +pre-registration (not in the corpus, so not judged by the verb). The verb prints `needed N / have M / met | NOT MET` +and, beside it, the trace-bearing and total run counts and the largest single named seat's count, so a reader sees +the gap and that a total across seats can overstate what one seat has. On the 2026-11-12 corpus it prints +`needed 100 / have 31 / NOT MET`. The rest of the 2026-10-15 amendment, including "the proposer proposes and never +applies", stands; no proposer is built by this amendment. + ## Consequences - A nil/zero table is byte-identical to the pre-key loop; every field is additive and `omitempty`, and a diff --git a/docs/architecture/decisions/0091-agent-trace-corpus-readout-pager-refetch-compaction-fire-and-prefill.md b/docs/architecture/decisions/0091-agent-trace-corpus-readout-pager-refetch-compaction-fire-and-prefill.md index cc0faf787..f9cda17f7 100644 --- a/docs/architecture/decisions/0091-agent-trace-corpus-readout-pager-refetch-compaction-fire-and-prefill.md +++ b/docs/architecture/decisions/0091-agent-trace-corpus-readout-pager-refetch-compaction-fire-and-prefill.md @@ -1,11 +1,11 @@ --- -status: Proposed +status: Accepted date: "2026-10-14" --- # 0091 — Agent-trace corpus readout: pager re-fetch, compaction-fire and prefill -Release: unreleased +Release: 0.206.0 (the verb, the harvest, the readout and the recording of the pager report and compaction pass count; `trace-corpus trigger` and the stderr trigger line are in the next release, see ADR 0036's 2026-11-12 amendment) ## Context @@ -137,6 +137,17 @@ A recommendation for the human decision on the planned context-editing measureme stating the counts beside any verdict. This ADR sets no minimum sample size; whoever reads the gate states how many runs it rests on. +## Status note 2026-11-12 + +Accepted, because what it decides is built and in use: the verb, the harvest, the null-not-zero rule, the run +definition, the pool-alias bucket and the redaction shipped in 0.206.0 (PR 588), and the recording of the pager report +and compaction pass count ships in the agent wire result, so recorded runs now exist. The "What the numbers decide" list +stays a set of recommendations for a human decision, not decisions of this ADR, and the first-readout table is a +dated snapshot. What it has not produced is the evidence it was built to collect: a harvest on 2026-11-12 of 12,332 runs +(installed 0.232.0) holds 31 runs with a measured pager report, of 5,258 that recorded one, so the pager gate reads +`BELOW_GATE` on 31 runs (8 of 86 evicted payloads re-fetched, 9.3 % against the 10 % gate) rather than `NO_DATA`, and no seat has more than 10 of them. Thirty-one runs do not settle the +`edit_context` question, which stays open. The proposer trigger in ADR 0036 is stated on this count. + ## Consequences - The three questions now have a reproducible answer path: a hash-stamped corpus, a frozen readout and a byte-exact check. diff --git a/docs/architecture/decisions/README.md b/docs/architecture/decisions/README.md index 3465397f8..5d12f6207 100644 --- a/docs/architecture/decisions/README.md +++ b/docs/architecture/decisions/README.md @@ -100,7 +100,7 @@ code, treat everything else as background. | [0088](0088-one-measurement-gate.md) | Proposed | One measurement gate for every adoption: `eval.Gate` over a frozen, hashed baseline (mode, primary metric and margin, judge, efficiency metrics, `min_pairs`, bootstrap size and seed, held-out ids, declared tasks), with the primary read as the mean paired task difference on Tango's score interval for pass/fail pairs and on the score form of the Student-t interval over a sample with two pseudo-pairs at each end of the range for every other score (no cell over 4% false ADOPT in 166 null cells, the strong-incumbent cells at 61 to 1000 tasks included, where the version 2 score form read up to 17.7%; at most 3.1% for pass/fail pairs over every size from 16 to 500; the price is power on scores that are not pass/fail; the paired bootstrap, `BootstrapDeltaMean`, stays unchanged for its legacy callers and reads the efficiency metrics): superiority adopts iff `lo > 0`, non-inferiority blocks iff `hi < -m` and adopts iff `lo > -m` with a declared efficiency metric improving and none worse or unmeasured (tokens, money or memory in GiB, never a wall-time term, INV-5); a candidate's unfinished attempts enter at the worst score and their recorded cost is spent, the incumbent's are skipped; fewer than `min_pairs` usable pairs or an excluded share above 0.20 is INCONCLUSIVE and the incumbent keeps every non-ADOPT (the remedy is more tasks, which the power note counts); a held-out slice can veto an ADOPT; a model judge needs a passed receipt; an era marker pins the runs; a baseline of another gate version (this is version 3) is refused first, for its version; `local-offload gate ` exits 0 for a verdict (read it from stdout), 1 for a non-ADOPT under `--require-adopt`, 2 for a refusal; the legacy callers (`agent-trajectory-gate`, `compeval.RunAB`) are not migrated; [0080](0080-a-ram-spill-moe-is-the-agent-seat-when-it-earns-it.md)'s three gates stay as the resource gates | | [0089](0089-strata-is-an-opt-in-agent-seat-backend-that-earns-its-place-through-the-gate.md) | Proposed | Strata is an opt-in agent-seat backend that earns its place through the gate: the agent class only, never a cascade rung, and it displaces no declared `vllm_seat`; a dormant slot in every tier (`strata_seat` or `strata_debt`), rendered and seeded by one predicate (`tierseed.StrataEligible`) whose floors were re-read on 2026-10-14 and are hard-coded only by the predicate change, each with its `source` and `verified_at`; the default checkpoint (Flash-Next IQ3_S, experts spilled to host RAM) reverses the shape 0.115.4 removed, under the operator's 2026-10-07 and 2026-10-10 rules, and only if the spike measures Strata's split to match the clause (attention and dense layers on the cards, only expert matmuls on the host, per-process CPU share within 10 percentage points of the `--n-cpu-moe` arm), with `--kv-streaming off` pinned on the clause arm and the KV cache's placement recorded on every arm; a llama-swap-launched Strata seat is a harness seat, the spike is hand-launched, lease-gated, outside llama-swap with `idle_unload_s=300`; production is `internal/strataseat` with `ttl: 300` and a generalized seat-guard roster; the pair is resolved by role and pinned by GPU UUID prefix at run time; the Qwen Community License 1.0 and Swift Open License 1.0 rows read `commercial_use: unverified`; promotion is a ladder through the one measurement gate (ADR 0088) and ADR 0080's resource gates, with quality the only gate | | [0090](0090-capability-review-2026-10-dispositions-and-reopen-triggers.md) | Proposed (the pinning mechanism is superseded by ADR 0099) | Capability review 2026-10: dispositions and reopen triggers: the operator's eight mandatory links each end in one terminal state (INSTALLED, PORTED, REPLICATED, DECLINED or PARKED, or a conditional pair; the Headroom rungs, the CLM paper and GitSwarm read pending, naming what decides them); a Parked table gives fifteen further items a reason and a reopen trigger; and DeepGEMM is parked until a Blackwell sm_120 tier adopts an FP8-block or FP8xFP4/NVFP4 vLLM checkpoint, with the checklist question that watches for it (ADR 0089's NVFP4 arm carries it; the cascade bake checklist will carry the same question once that checklist is committed, so that cross-link is pending) and the procedure if it fires (CUDA 13.2 toolchain, wheel built in a scratch copy of the seat environment, one-card A/B with output equality first and throughput as a side column, explicit approval) | -| [0091](0091-agent-trace-corpus-readout-pager-refetch-compaction-fire-and-prefill.md) | Proposed | The agent-trace corpus readout: `local-offload trace-corpus` reads recorded agent runs (standalone agent traces and delegation-log rows, never the ledger) for the pager re-fetch rate against the 10 % gate of R2-13, the compaction fire rate and prefill, and never reads an unrecorded value as 0 (every rate is null with the unrecorded count beside it). A delegation row counts as a run only if an agent executed, runs on a pool alias are their own bucket keyed by node (a per-seat reading cannot reach them), and a path-shaped seat, node id or stop reason is replaced at harvest. The first readout (7,057 runs, one machine) reads `NO_DATA` on the gate and a null fire rate, so it neither builds nor shelves the proposed `edit_context` tool (one that would let the agent seat edit its own transcript), and it leaves 29.3 % of runs unattributable to a seat; the ADR carries aggregates only and recommends the measurement wait for instrumented runs or use a selected set of long runs. | +| [0091](0091-agent-trace-corpus-readout-pager-refetch-compaction-fire-and-prefill.md) | Accepted | The agent-trace corpus readout: `local-offload trace-corpus` reads recorded agent runs (standalone agent traces and delegation-log rows, never the ledger) for the pager re-fetch rate against the 10 % gate of R2-13, the compaction fire rate and prefill, and never reads an unrecorded value as 0 (every rate is null with the unrecorded count beside it). A delegation row counts as a run only if an agent executed, runs on a pool alias are their own bucket keyed by node (a per-seat reading cannot reach them), and a path-shaped seat, node id or stop reason is replaced at harvest. The first readout (7,057 runs, one machine) reads `NO_DATA` on the gate and a null fire rate, so it neither builds nor shelves the proposed `edit_context` tool (one that would let the agent seat edit its own transcript), and it leaves 29.3 % of runs unattributable to a seat; the ADR carries aggregates only and recommends the measurement wait for instrumented runs or use a selected set of long runs. | | [0094](0094-a-lease-names-its-cards-a-card-has-one-holder.md) | Accepted | A lease names its cards; a card has one holder: a lease carries a list of card ids (empty is the whole node), two leases conflict when either is whole-node or their lists share a card, the grant creates each card's claim exclusively under the epoch lock and a whole-node grant verifies the other side, the fence is per epoch, nothing frees a card from a holder that may still be using it except an explicit release by epoch that checks no liveness, the writer is a per-host switch behind a green reader audit, every consumer asks about its own cards and a doubt reads as every card, and a busy card is a place in line; the takeover, the release that waits for a dead tree, an overdue policy beyond the request, a media plan that always names its cards, a local worker pool across cards and a scripted acceptance are listed as not built and not decided | | [0095](0095-humming-serves-the-weight-formats-marlin-cannot.md) | Accepted | Humming serves the weight formats Marlin cannot; Marlin keeps the 4-bit seat: the seat launchers keep Humming's JIT cache and generated modules under the seat's work directory and cap its parallel builds at eight; `--linear-backend humming` is a documented opt-in that moves no seat; a higher-precision seat is a saved 5 or 6-bit compressed-tensors checkpoint (vLLM's WNA16 scheme loads it onto Humming with no flag), never online quantization at every cold start and never `--quantization humming` on a compressed-tensors checkpoint; no Humming on a vLLM older than 0.29; Humming's version is the one vLLM pins | | [0096](0096-a-live-driver-fault-repeat-demotes-a-cards-placement.md) | Accepted | A live driver-fault repeat demotes a card's placement: a repeat (NVSentinel's rule, one fault class in 5 or more bursts on one card within 24 h, restart and sleep bursts left out) is live for 24 h after its last burst; a remote whose every seat for the contract sits on such a card ranks right after saturation in `betterRemote` and `fitPickWith`, demoted and never excluded; the local seat reads busy for the deal when its cards carry one and a remote could take the work, never for route=local; cause-agnostic; driver events still move no verdict, gate, exclusion, drain or release (amends register C-83-B for placement only) | diff --git a/docs/systems/coding-agent.md b/docs/systems/coding-agent.md index 1076b24db..50bd43fad 100644 --- a/docs/systems/coding-agent.md +++ b/docs/systems/coding-agent.md @@ -1484,7 +1484,7 @@ replaces the default. `cmd/local-agent/serve_test.go` covers the loopback guard. ## Source map - [`internal/agent/loop.go`](../../internal/agent/loop.go) — loop, budget, `dispatchOrThrottle` -- [`trace_corpus.go`](../../trace_corpus.go) — the `trace-corpus` verb (harvest, readout, freeze, check) +- [`trace_corpus.go`](../../trace_corpus.go) — the `trace-corpus` verb (harvest, readout, trigger, freeze, check) over recorded agent runs; [`internal/agent/contextpager.go`](../../internal/agent/contextpager.go) — the pager instrument and the compaction pass count it records - [`internal/agent/policy.go`](../../internal/agent/policy.go) — broker, `.git` denial, audit append diff --git a/docs/systems/failure-register.md b/docs/systems/failure-register.md index 4ef82a9a7..54fb97368 100644 --- a/docs/systems/failure-register.md +++ b/docs/systems/failure-register.md @@ -13,7 +13,8 @@ The split is the point. The counts come from the ledger. The **root cause** and a person's: they are written in the table below, and no code derives, guesses or fills them in. A class with no row here reads `undocumented`, and a class whose row holds no root cause and no fix a person wrote reads as not investigated (`uninvestigated_classes`), so a failure the harness keeps hitting without anyone having looked into it -is visible as a gap, not hidden. The table ships with those cells empty: nothing on this page has been investigated yet. +is visible as a gap, not hidden. The table ships with those cells empty: nothing in it has been written by a person yet. The root causes found by reading +the code and the real rows are a separate catalogue, in code, printed with each class (see Root causes). It is read-only. It never writes the ledger, starts nothing and calls no model. @@ -87,6 +88,69 @@ policy, or a write set past a door cap; neither a broken stack nor an unplaceabl a share of `failure_rows` and counts only the classes filed as stack faults: an `unclassified` failure is in `failure_rows` but never in `stack_fault_rows`, so while unclassified rows exist the share is a lower bound. +### Root causes + +Every class of the closed set has an entry in `internal/ledger/rootcause.go`, beside the class definitions: a **cause kind**, +a **root cause** (one or two sentences), the **evidence** (the function that emits the class and the count read) and an +**action**. `ledger --failures` prints it with each class: in the JSON as `classes[].cause` (`kind`, `root_cause`, +`evidence`, `action`, `investigated`) and as text on stderr, one block per class, with `causes_not_investigated` naming +the classes in the window that are still open. The person-written table below it is a different thing and is unchanged: +its `root_cause` and `structural_fix` fields still come only from the page, and `cause.root_cause` comes only from +the catalogue. + +The cause kinds are `capacity signal` (the stack was healthy and full), `node configuration`, `harness defect`, +`caller contract` (the request, its limits or its deadline), `upstream service` (an engine, a node host or a network +path failed) and `unknown`. They say whose fix the cause is, which `kind` (above) does not. + +How the table was made (2026-11-12): the code that emits each class was read, then the rows of the 30 days to +2026-11-12 in the real ledger, and the whole ledger for a class with no row in the window. An entry is `investigated` only +when the emitter and those rows together settle the cause it states. Where they do not, the cause reads +`not investigated:` and says what would settle it. The counts are that snapshot; the live count of a class is its `count`. +A class with no entry reads as not investigated. **15 of the 31 classes are not investigated**: 8 have rows that do not +settle the cause (`failed_verification`, `poll_deadline`, `stall_admission`, `stall_cold_load`, `stall_prefill`, +`stall_decode`, `stall_repack`, `other`) and 7 have no row in the whole ledger (`job_lost`, `remote_error`, `shed`, +`stall_tool`, `stall_engine`, `stall_other`, `write`). In the 30 days to 2026-11-12, 4 of the 19 classes present were +not investigated. + +| failure class | cause kind | root cause | evidence | action | +|---|---|---|---|---| +| `failed_verification` | unknown | not investigated: the rows say which check failed (`min_items` on plans, quotes or numbers; a regex) but not whether the seat's answer was wrong or the check too strict; the acceptance list is shape-only (delegate/lint.go warns of it) | delegate/run.go (failed verification); 357 rows in 30 days, 345 of them from research subtasks | not investigated: read the saved output of a sample of failed research subtasks against their acceptance lists | +| `queue_full` | capacity signal | Every node asked refused the dispatch for room: its queue was at fleet_max_queue_depth (the sampled bodies read `queue full (N jobs accepted+running, limit N)`) or its card was leased. The delegator files every 503 or 429 so. | delegate/run_capacity.go capacityRefusal; 49 rows in 30 days (of the three commonest reason groups, 22 rows read queue full and 1 node leased) | none: working as designed. A node with spare capacity can raise fleet_max_queue_depth. | +| `queue_deadline` | capacity signal | A node accepted the job and its backlog never reached it inside the queue patience, and the node did not confirm taking it back. Jobs queue behind other work on a full node. | delegate/run_remote.go (queue deadline after); 0 rows in 30 days, 318 in the whole ledger, all written before the withdrawal stamp existed | none: working as designed (the delegator withdraws and re-places since ADR 0064; queue_withdrawn is the same event confirmed). | +| `queue_withdrawn` | capacity signal | The same event as queue_deadline, and the node confirmed taking the job back: it sat in the node's backlog (`accepted`, never `running`) past the queue patience and was withdrawn. | delegate/run_remote.go; 186 rows in 30 days, 182 from research subtasks, 111 on one node | none: working as designed. If one node keeps most of them (111 of 186), check its concurrency against the headroom it advertises. | +| `poll_deadline` | unknown | not investigated: the class does not separate a job longer than the poll budget from a node that stopped making progress (failurekinds.go says so), and the 3 rows differ (a poll error, a stale last-progress, the node's own wall) | delegate/run_remote.go (poll deadline after); 3 rows in 30 days | not investigated: for each row, compare the node's own record of the job with the delegator's poll budget | +| `canceled` | caller contract | The caller's context ended (the caller went away, or the call's own deadline passed) before or while a subtask ran. | delegate/calldeadline.go; 29 rows in 30 days: `canceled: context canceled` and `the call's context ended before this subtask started` | none: working as designed. | +| `node_unreachable` | upstream service | A node host did not answer on the network: a dial that timed out (`the connected party did not properly respond`) or a poll that never got an answer. Why the host did not answer (asleep, off, network) is not recorded on the row. | delegate/run_remote.go; 3 rows in 30 days, 2 on one node | none from the harness: the node host or path is the fault; `local-offload doctor` shows which nodes answer now. | +| `job_lost` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | delegate/run_remote.go (a poll 404 after an acknowledged dispatch); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | +| `dispatch_refused` | node configuration | A node refused the dispatch itself, and the sampled rows give two node-side causes: the node rejected the delegator's bearer token (status 401 `unauthorized`, 112 rows, one node: a fleet_auth_token mismatch) and a node whose configured data folder could not be created (`creating pipeline-jobs dir`, 33 rows, one node: the path is not on that machine now). | delegate/calldeadline.go (401 unauthorized) and fleetnode/tasks.go:1031 (creating pipeline-jobs dir); 145 rows in 30 days, 145 on one node's seat | re-pair that node's token (`local-offload doctor` checks fleet tokens) and fix or remove the node's missing data folder; both are edits on the node. | +| `remote_error` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | delegate/run_remote.go (a node's job reported errored, or its answer was unusable); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | +| `capacity_wait` | capacity signal | No node had room for the subtask before the call's deadline: 248 of 267 waited until the call's own deadline (not agent_placement_wait_sec), 259 for one node's agent pool alias. | delegate/run_wait.go:436 and :442; 267 rows in 30 days, 259 on one pool alias | none: working as designed. Reducing it needs more worker slots on the pool's node, or fewer research subtasks per call. | +| `node_busy` | capacity signal | The seat or card the call needed was held by other work when it arrived: the seat was at its local run cap (`seat busy:`), a GPU lease of another class held the card (`gpu busy: held by ...`), or a media job waited for host RAM. The harness defers rather than overload the card. | pipeline/agenttask.go (seat busy, defer class capacity) and pipeline/pipeline_gpu.go (gpu busy); 628 rows in 30 days: video 247, research 205, audio 109 (the agent pool at its cap, a media lease on the cards, host RAM) | none: working as designed. Fewer deferrals need more free card time (keep a card free for media while text leases run) or a higher run cap on a box with room. | +| `shed` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | delegate (sheddable work, priority -1, found no idle node); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | +| `no_eligible_node` | upstream service | A lane found no fleet node able to take the call. In the sampled rows each node probed failed its health check with a timeout (`context deadline exceeded`), so the nodes were not answering, not refusing for room. Why they did not answer is not recorded. | the remote lanes' placement sentences (forecastremote and the others) and delegate route=remote; 76 rows in 30 days, 48 of them the vision lane | none from the harness: bring the nodes back (`local-offload doctor`); not investigated: why the probes timed out at the failure times. | +| `seat_down` | upstream service | The inference engine behind the seat was gone from llama-swap's /running, or refused connections, when the run needed it, and did not return inside the node's recovery wait. What stopped the engine is not recorded. | agent/seatdown.go:185; 89 rows in 30 days, 87 on one node's pool alias | not investigated: why that node's engine goes away; read its llama-swap log at the failure times. The harness already waits and re-issues (ADR 0066). | +| `stall_admission` | unknown | not investigated: the monitor reports a run silent in the admission phase; why is not on the row | agent/liveness.go; 0 rows in 30 days, 1 in the whole ledger | not investigated: read the node's engine log at the time of the row | +| `stall_cold_load` | unknown | not investigated: the rows say no byte came for 120 s after the seat read ready, on its first completion since the load; whether the bound is too tight for a cold seat or the engine was wedged is not settled | agent/liveness.go:268; 4 rows in 30 days, all on one node's pool alias | not investigated: time that seat's first completion after a load against the bound | +| `stall_prefill` | unknown | not investigated: the rows say no progress in prefill past an allowance computed from the prompt size and the seat's rate; a prompt larger than the allowance or a wedged engine both read so | agent/liveness.go; 0 rows in 30 days, 246 in the whole ledger (written before the stamp) | not investigated: compare the prompt size of those rows with the seat's measured prefill rate | +| `stall_decode` | unknown | not investigated: no progress while tokens were generated; the rows do not say whether the engine wedged or the allowance was too tight | agent/liveness.go; 0 rows in 30 days, 51 in the whole ledger (written before the stamp) | not investigated: read the engine log at the time of the rows | +| `stall_tool` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | agent/liveness.go (silence inside a tool call); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | +| `stall_repack` | unknown | not investigated: no progress in the structured re-pack of an answer; why is not on the row | agent/liveness.go (`structured re-pack unreachable: stalled`); 0 rows in 30 days, 99 in the whole ledger (written before the stamp) | not investigated: read the re-pack requests of those rows against the seat's log | +| `stall_engine` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | agent/liveness.go (the engine did no work, or kept stepping without a token); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | +| `stall_other` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | delegate/reasoncode.go stallReasonCode (a stall whose phase the reason does not name); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | +| `budget` | caller contract | A limit the call carried ended the run before the agent finished: the call's own deadline (`call deadline reached`, `wall timeout after ... the caller's deadline`), a cancellation filed under this class (`agent loop: canceled`), or the step budget (`step budget exhausted`). Most are the caller's deadline, not a node ceiling. | core/calldeadline.go, pipeline/liveness.go:411, pipeline/agenttask.go:1295; 250 rows in 30 days, 241 from research subtasks | give long research subtasks a longer call deadline or fewer subtasks per call; a run that ends on the step budget needs a larger max_steps. | +| `abstention` | node configuration | The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call. | pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days, 107 on one vision seat's questions | raise the vision lane's output cap for that seat, or ask for shorter answers; not investigated: why those answers run past the cap. | +| `contract` | caller contract | The caller named something the node does not serve: an image family the node does not carry. | config/families.go:730; 4 rows in 30 days, all one family name on generate_image | the caller names a served family, or the node adds that family. | +| `config` | node configuration | A seat was set up so that it cannot serve the call: its tool-call parser does not match its chat template, so a tool call came back as text. | pipeline/agenttask.go:1303; 2 rows in 30 days, both on one seat | set that seat's --tool-call-parser to the one its chat template needs. | +| `write` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | pipeline/agenttask.go (the write door refused: agent_allow_write false, or a write set past a door cap); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | +| `infrastructure` | upstream service | Mostly a seat whose engine was gone from llama-swap's /running (the node's defer class infrastructure carrying the same sentence as seat_down), then a vision seat that was cold and timed out. Some rows are a capacity hold filed here (see the followups). | agent/seatdown.go:185, pipeline_vision.go; 195 rows in 30 days: research 90, fleet 82; about 63 delegate rows read `local seat reserved (gpu lease ...)` | see seat_down; not investigated: why the engine goes away. The lease-hold rows are a classification defect, not an infrastructure fault. | +| `bad_input` | caller contract | The input cannot be used as given: an image path that does not exist, a template variable the template does not declare. | pipeline (image load) and compose_video BAD_INPUT; 2 rows in 30 days | none: working as designed. The caller fixes the input. | +| `other` | unknown | not investigated: the class is the residue no rule names, a mix of causes; the largest sampled sentence is a node's `a text lease holds its card; it may take the call later` (a capacity signal no rule names), and 15 rows are the browse door's own errors | reasoncode.go DeferCode (the residue); 55 rows in 30 days: audio 18, browse 15, video 7, compose 6 | not investigated: list the distinct reasons of the 55 rows and add a rule per sentence that ties to one class | + +Reading the rows also found two misfilings, not changed by the catalogue: a delegate row for a local seat held by a text +lease (`local seat reserved (gpu lease ...)`) is filed `infrastructure` though it is a capacity hold, and the agent loop's +`agent loop: canceled` (the caller gave up, or the box is draining) is filed `budget` though `canceled` exists. A +transport 4xx from a node's own folder failure is filed `dispatch_refused`, a class defined as a fact about the request. + ### The fields | field | meaning | @@ -108,6 +172,8 @@ a share of `failure_rows` and counts only the classes filed as stack faults: an | `classes[].groups` | the rows counted per (door, seat, model) together | | `classes[].cards_tokens`, `rows_with_tokens` | the card work the class spent, and how many rows measured it | | `classes[].documented`, `investigated`, `meaning`, `root_cause`, `structural_fix` | the joined table row, and whether a person wrote a root cause or a fix in it | +| `classes[].cause` | the catalogue entry (Root causes): `kind`, `root_cause`, `evidence`, `action`, `investigated` | +| `causes_not_investigated` | the classes in the window whose catalogue cause is `not investigated` | **Failures by build (`--by-build`).** The ledger stamps the build that wrote each row, in `Record`, from the same self-hash the delegation log uses and under its names: `harness_build_sha256` on a row for work the writing process ran @@ -156,7 +222,7 @@ The table is the human-maintained half of the register. `uninvestigated_classes`. - A new reason code needs a new row. `TestEmbeddedFailureTableCoversEveryReasonCode` fails until there is one, and fails again for a row that names a code the ledger does not define. -- **No row below has been investigated.** `meaning` states what the class is, from the constants' own definitions. The +- **No row of this table has been written by a person.** `meaning` states what the class is, from the constants' own definitions. The `root cause` and `structural fix` cells ship empty on purpose: filling them with a restatement of those definitions made every class look covered and defeated the gap list. A cell gets text only when a person has looked at an observed failure and wrote down what they found; `TestEmbeddedFailureTableShipsOnlyInvestigatedCells` fails for a non-empty @@ -248,7 +314,8 @@ code; this table is checked against both. - `local-offload ledger --failures [--by-build] [--since DAYS] [--include-test-traffic] [--table FILE] [--config FILE]`: the report, JSON on stdout and nothing else. A note about the left-out traffic, a missing ledger or lines it could not read goes to stderr. `--failures=false` is the savings summary, as for any boolean flag. `--by-build` adds the `builds` - key (below) and prints the same split as a short table on stderr; without it the output is what it was. + key (below) and prints the same split as a short table on stderr; without it the output is what it was. Every run also + prints each class with its cause and action as text on stderr (the JSON carries them as `classes[].cause`). ## Invariants and assumptions @@ -260,7 +327,8 @@ code; this table is checked against both. - A line the register cannot read is counted in `unparsed_lines`, never dropped silently; a missing ledger is `ledger_found: false`, never a quiet zero. - An unmeasured token total is `null`, never `0`. -- `root_cause` and `structural_fix` come only from the table. A class with no row says `undocumented`. +- `root_cause` and `structural_fix` come only from the table. A class with no row says `undocumented`. `cause` comes only + from the catalogue in code, and a class with no entry there reads `not investigated`. ## Testing notes @@ -274,6 +342,7 @@ the unreadable lines. - [ledger_failures.go](../../ledger_failures.go): the `--failures` mode, the table parser and the join. - [internal/ledger/failures.go](../../internal/ledger/failures.go): the reader that groups the rows. - [internal/ledger/failurekinds.go](../../internal/ledger/failurekinds.go): the kind of every failure class. +- [internal/ledger/rootcause.go](../../internal/ledger/rootcause.go): the root-cause catalogue. - [internal/ledger/reasoncode.go](../../internal/ledger/reasoncode.go): the closed reason-code set and the classifier. - [internal/core/agentwire.go](../../internal/core/agentwire.go): the wire defer classes. - [ledger_failures_test.go](../../ledger_failures_test.go): the acceptance tests. diff --git a/docs/systems/self-improvement.md b/docs/systems/self-improvement.md index c1ba8cde9..62ee897ed 100644 --- a/docs/systems/self-improvement.md +++ b/docs/systems/self-improvement.md @@ -76,7 +76,7 @@ an unfinished attempt, not a duration the gate reads. | part | runs as | answers | owner page | |---|---|---|---| | failure register | `local-offload ledger --failures` | which failures the stack has, how many, whose they are, what is known about why | [failure-register.md](failure-register.md) | -| trace-corpus readout | `local-offload trace-corpus harvest\|readout\|freeze\|check` | how the agent loop behaves in recorded runs: pager re-fetch, compaction passes, prefill | [coding-agent.md](coding-agent.md), [ADR 0091](../architecture/decisions/0091-agent-trace-corpus-readout-pager-refetch-compaction-fire-and-prefill.md) | +| trace-corpus readout | `local-offload trace-corpus harvest\|readout\|trigger\|freeze\|check` | how the agent loop behaves in recorded runs: pager re-fetch, compaction passes, prefill | [coding-agent.md](coding-agent.md), [ADR 0091](../architecture/decisions/0091-agent-trace-corpus-readout-pager-refetch-compaction-fire-and-prefill.md) | | measurement gate | `local-offload gate freeze\|check\|verdict\|schema\|selftest` | whether a candidate is better, or not worse and cheaper | [measurement-gate.md](measurement-gate.md), [ADR 0088](../architecture/decisions/0088-one-measurement-gate.md) | | SHA-bound review evidence | `offload_review_diff` (MCP tool) | which bytes a review judged, and whether a clean seat judged them | [mcp-server.md](mcp-server.md) | @@ -112,11 +112,19 @@ class with no row reads `undocumented`, and a class whose row holds neither a ro in `uninvestigated_classes`. At this head no row of the table has been investigated, so the register's first product is its gap list. A test fails for a non-empty cell until the class is named in it with what the finding rests on. +Every class also carries a **cause** found by reading its emitting code and the real rows, from a catalogue in code +(`internal/ledger/rootcause.go`): a cause kind, a root cause, the evidence and an action, printed with the class +(`classes[].cause`, and as text on stderr). It says `not investigated` and what would settle it where the evidence does not +settle the cause: 15 of the 31 classes on 2026-11-12, and `causes_not_investigated` lists those in the window. + ### The trace-corpus readout -`local-offload trace-corpus` has four modes. `harvest` reads recorded agent runs, from the standalone agent traces and +`local-offload trace-corpus` has five modes. `harvest` reads recorded agent runs, from the standalone agent traces and the delegation log and never the ledger, reduces each to numbers and enums, redacts the few strings left and writes the -corpus to `--out`, printing its sha256, which every report then carries as `corpus_hash`. `readout` prints the report. +corpus to `--out`, printing its sha256, which every report then carries as `corpus_hash`. `readout` prints the report and, on stderr, the recipe proposer's trigger (ADR 0036) as +`needed N / have M / met | NOT MET`. `trigger` prints that trigger as JSON: `needed` (100), `have` (runs with a measured +pager report, `pager.measured_runs`, not trace-bearing runs), `met`, the trace-bearing and total run counts beside it and the +largest single named seat's count. It does not judge the other half of the trigger, a frozen pre-registration for the seat. `freeze` writes the report as a baseline, created exclusively: a path that exists is refused (exit 2, as `gate freeze` refuses one), so a baseline is never replaced and a new one goes to a new `--out`. `check` demands the report back byte for byte and refuses a baseline taken from a different corpus. `readout`, `freeze` and `check` are deterministic and @@ -150,7 +158,7 @@ node's real seat, so those runs are reported apart (`pool_alias`) and `attributi per-seat reading cannot reach. The first readout, in [ADR 0091](../architecture/decisions/0091-agent-trace-corpus-readout-pager-refetch-compaction-fire-and-prefill.md) -(status Proposed at this head, a snapshot of one machine's log on 2026-10-14; later logs carry recorded runs, so re-run the +(status Accepted since 2026-11-12, a snapshot of one machine's log on 2026-10-14; later logs carry recorded runs, so re-run the readout before quoting it), found no run that had recorded a pager report or a pass count. The pager verdict read `NO_DATA` and the fire rate `null`, so the questions were recorded as unanswered rather than answered "rarely". @@ -256,7 +264,7 @@ flags is kept raw without being shown to the seat. The key stays off until a sea - `local-offload ledger --failures [--since DAYS] [--include-test-traffic] [--table FILE]` - `local-offload trace-corpus harvest [--traces DIR] [--delegation-log DIR] --out FILE [--seat LIST] [--exclude-seat LIST] [--since YYYY-MM-DD] [--until YYYY-MM-DD] [--config PATH]`, then - `readout|freeze|check --corpus FILE` (`check` also takes `--baseline FILE`; `readout` and `freeze` take `--out FILE`) + `readout|trigger|freeze|check --corpus FILE` (`check` also takes `--baseline FILE`; `readout` and `freeze` take `--out FILE`) - `local-offload gate freeze|check|verdict|schema|selftest` - `local-offload recipe-audit --recipe ID --seat ALIAS --seat-tag TAG --contracts TASKS.json --baseline BASELINE.json --engine-build STR [--store DIR] [--attempts N] [--input-out FILE] [--config PATH]` @@ -328,7 +336,7 @@ flags is kept raw without being shown to the seat. The key stays off until a sea named head. It does not block or allow a merge, and the lane cannot show the head belongs to the diff. - **Bind a verdict to the hash `freeze` printed**, not to a later `gate check` of the file: an edit that leaves a valid canonical document is another pre-registration, and only the hash tells it from the frozen one. -- **ADR 0088, ADR 0090 and ADR 0091 are Proposed at this head.** Only `Accepted` ADRs are current guidance, so the code +- **ADR 0088 and ADR 0090 are Proposed at this head** (ADR 0091 is Accepted). Only `Accepted` ADRs are current guidance, so the code and the system pages above are what this page rests on. ## Source map diff --git a/internal/ledger/rootcause.go b/internal/ledger/rootcause.go new file mode 100644 index 000000000..afba36a23 --- /dev/null +++ b/internal/ledger/rootcause.go @@ -0,0 +1,175 @@ +package ledger + +// The root-cause catalogue: for every failure class of the closed reason-code set (reasoncode.go), why it happens, +// what that rests on and what would reduce it. It sits beside the class definitions because a class and its cause +// are read together, and the failure register (failures.go, `ledger --failures`) prints it with each class. +// +// How an entry was written (2026-11-12): the code that emits the class was read, then the rows of the 30 days to +// 2026-11-12 in the real ledger (their reasons, doors and seats), then the all-time counts for a class with no row +// in the window. An entry is Investigated only when the emitter and the rows together settle the cause it states. +// Where they do not, RootCause starts with "not investigated" and says what would settle it; no cause is guessed. +// The counts in Evidence are that snapshot, not live figures: the live count is the class's own `count`. +// +// These are findings about the harness at that date, kept in code so a class cannot be listed without one; they are +// not read from a row, and a row's text never decides them. When a cause changes, change the entry. + +// The cause kinds. A kind says whose fix the cause is, which is not what failurekinds.go's Kind says (that one +// says who owns the failure for the stack-fault share). +const ( + // CauseCapacity: the stack was healthy and full; the signal is the design working. + CauseCapacity = "capacity signal" + // CauseNodeConfig: a node (or this box) is set up in a way that cannot serve the call. + CauseNodeConfig = "node configuration" + // CauseHarnessDefect: the harness itself does the wrong thing. + CauseHarnessDefect = "harness defect" + // CauseCallerContract: the request, its limits or its deadline, is what ended the call. + CauseCallerContract = "caller contract" + // CauseUpstream: a service the harness depends on (an engine, a node host, a network path) failed. + CauseUpstream = "upstream service" + // CauseUnknown: nothing read so far settles it. + CauseUnknown = "unknown" +) + +// notInvestigated opens the RootCause and Action of an entry nobody has settled. +const notInvestigated = "not investigated" + +// FailureCause is the established cause of one failure class. +type FailureCause struct { + // Kind is one of the Cause* constants. + Kind string `json:"kind"` + // RootCause is why the class happens, in one or two sentences, or "not investigated: ..." naming what would + // settle it. + RootCause string `json:"root_cause"` + // Evidence is the function that emits the class and the count read in the 30 days to 2026-11-12. + Evidence string `json:"evidence"` + // Action is what would reduce the class, or "none: working as designed", or "not investigated". + Action string `json:"action"` + // Investigated is false when RootCause starts with "not investigated". + Investigated bool `json:"investigated"` +} + +// FailureCauseOf is the catalogue entry of a class. A class with no entry reads as not investigated, so a new +// reason code is listed as a gap and never as a blank. +func FailureCauseOf(class string) FailureCause { + if c, ok := failureCauses[class]; ok { + return c + } + return FailureCause{Kind: CauseUnknown, Investigated: false, + RootCause: notInvestigated + ": the class has no catalogue entry (internal/ledger/rootcause.go); read its emitter and its rows to settle it", + Evidence: "none", Action: notInvestigated} +} + +// noRow is the entry of a class the ledger has never held a row for: its emitter can be read, but no failure has +// been seen, so no cause is stated. +func noRow(emitter string) FailureCause { + return FailureCause{Kind: CauseUnknown, + RootCause: notInvestigated + ": the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read", + Evidence: emitter + "; 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger", + Action: notInvestigated} +} + +var failureCauses = map[string]FailureCause{ + ReasonNodeBusy: {Kind: CauseCapacity, Investigated: true, + RootCause: "The seat or card the call needed was held by other work when it arrived: the seat was at its local run cap (`seat busy:`), a GPU lease of another class held the card (`gpu busy: held by ...`), or a media job waited for host RAM. The harness defers rather than overload the card.", + Evidence: "pipeline/agenttask.go (seat busy, defer class capacity) and pipeline/pipeline_gpu.go (gpu busy); 628 rows in 30 days: video 247, research 205, audio 109 (the agent pool at its cap, a media lease on the cards, host RAM)", + Action: "none: working as designed. Fewer deferrals need more free card time (keep a card free for media while text leases run) or a higher run cap on a box with room."}, + ReasonFailedVerification: {Kind: CauseUnknown, + RootCause: notInvestigated + ": the rows say which check failed (`min_items` on plans, quotes or numbers; a regex) but not whether the seat's answer was wrong or the check too strict; the acceptance list is shape-only (delegate/lint.go warns of it)", + Evidence: "delegate/run.go (failed verification); 357 rows in 30 days, 345 of them from research subtasks", + Action: notInvestigated + ": read the saved output of a sample of failed research subtasks against their acceptance lists"}, + ReasonQueueFull: {Kind: CauseCapacity, Investigated: true, + RootCause: "Every node asked refused the dispatch for room: its queue was at fleet_max_queue_depth (the sampled bodies read `queue full (N jobs accepted+running, limit N)`) or its card was leased. The delegator files every 503 or 429 so.", + Evidence: "delegate/run_capacity.go capacityRefusal; 49 rows in 30 days (of the three commonest reason groups, 22 rows read queue full and 1 node leased)", + Action: "none: working as designed. A node with spare capacity can raise fleet_max_queue_depth."}, + ReasonQueueDeadline: {Kind: CauseCapacity, Investigated: true, + RootCause: "A node accepted the job and its backlog never reached it inside the queue patience, and the node did not confirm taking it back. Jobs queue behind other work on a full node.", + Evidence: "delegate/run_remote.go (queue deadline after); 0 rows in 30 days, 318 in the whole ledger, all written before the withdrawal stamp existed", + Action: "none: working as designed (the delegator withdraws and re-places since ADR 0064; queue_withdrawn is the same event confirmed)."}, + ReasonQueueWithdrawn: {Kind: CauseCapacity, Investigated: true, + RootCause: "The same event as queue_deadline, and the node confirmed taking the job back: it sat in the node's backlog (`accepted`, never `running`) past the queue patience and was withdrawn.", + Evidence: "delegate/run_remote.go; 186 rows in 30 days, 182 from research subtasks, 111 on one node", + Action: "none: working as designed. If one node keeps most of them (111 of 186), check its concurrency against the headroom it advertises."}, + ReasonPollDeadline: {Kind: CauseUnknown, + RootCause: notInvestigated + ": the class does not separate a job longer than the poll budget from a node that stopped making progress (failurekinds.go says so), and the 3 rows differ (a poll error, a stale last-progress, the node's own wall)", + Evidence: "delegate/run_remote.go (poll deadline after); 3 rows in 30 days", + Action: notInvestigated + ": for each row, compare the node's own record of the job with the delegator's poll budget"}, + ReasonCanceled: {Kind: CauseCallerContract, Investigated: true, + RootCause: "The caller's context ended (the caller went away, or the call's own deadline passed) before or while a subtask ran.", + Evidence: "delegate/calldeadline.go; 29 rows in 30 days: `canceled: context canceled` and `the call's context ended before this subtask started`", + Action: "none: working as designed."}, + ReasonNodeUnreachable: {Kind: CauseUpstream, Investigated: true, + RootCause: "A node host did not answer on the network: a dial that timed out (`the connected party did not properly respond`) or a poll that never got an answer. Why the host did not answer (asleep, off, network) is not recorded on the row.", + Evidence: "delegate/run_remote.go; 3 rows in 30 days, 2 on one node", + Action: "none from the harness: the node host or path is the fault; `local-offload doctor` shows which nodes answer now."}, + ReasonJobLost: noRow("delegate/run_remote.go (a poll 404 after an acknowledged dispatch)"), + ReasonDispatchRefused: {Kind: CauseNodeConfig, Investigated: true, + RootCause: "A node refused the dispatch itself, and the sampled rows give two node-side causes: the node rejected the delegator's bearer token (status 401 `unauthorized`, 112 rows, one node: a fleet_auth_token mismatch) and a node whose configured data folder could not be created (`creating pipeline-jobs dir`, 33 rows, one node: the path is not on that machine now).", + Evidence: "delegate/calldeadline.go (401 unauthorized) and fleetnode/tasks.go:1031 (creating pipeline-jobs dir); 145 rows in 30 days, 145 on one node's seat", + Action: "re-pair that node's token (`local-offload doctor` checks fleet tokens) and fix or remove the node's missing data folder; both are edits on the node."}, + ReasonRemoteError: noRow("delegate/run_remote.go (a node's job reported errored, or its answer was unusable)"), + ReasonCapacityWait: {Kind: CauseCapacity, Investigated: true, + RootCause: "No node had room for the subtask before the call's deadline: 248 of 267 waited until the call's own deadline (not agent_placement_wait_sec), 259 for one node's agent pool alias.", + Evidence: "delegate/run_wait.go:436 and :442; 267 rows in 30 days, 259 on one pool alias", + Action: "none: working as designed. Reducing it needs more worker slots on the pool's node, or fewer research subtasks per call."}, + ReasonShed: noRow("delegate (sheddable work, priority -1, found no idle node)"), + ReasonNoEligibleNode: {Kind: CauseUpstream, Investigated: true, + RootCause: "A lane found no fleet node able to take the call. In the sampled rows each node probed failed its health check with a timeout (`context deadline exceeded`), so the nodes were not answering, not refusing for room. Why they did not answer is not recorded.", + Evidence: "the remote lanes' placement sentences (forecastremote and the others) and delegate route=remote; 76 rows in 30 days, 48 of them the vision lane", + Action: "none from the harness: bring the nodes back (`local-offload doctor`); not investigated: why the probes timed out at the failure times."}, + ReasonSeatDown: {Kind: CauseUpstream, Investigated: true, + RootCause: "The inference engine behind the seat was gone from llama-swap's /running, or refused connections, when the run needed it, and did not return inside the node's recovery wait. What stopped the engine is not recorded.", + Evidence: "agent/seatdown.go:185; 89 rows in 30 days, 87 on one node's pool alias", + Action: "not investigated: why that node's engine goes away; read its llama-swap log at the failure times. The harness already waits and re-issues (ADR 0066)."}, + ReasonStallAdmission: {Kind: CauseUnknown, + RootCause: notInvestigated + ": the monitor reports a run silent in the admission phase; why is not on the row", + Evidence: "agent/liveness.go; 0 rows in 30 days, 1 in the whole ledger", + Action: notInvestigated + ": read the node's engine log at the time of the row"}, + ReasonStallColdLoad: {Kind: CauseUnknown, + RootCause: notInvestigated + ": the rows say no byte came for 120 s after the seat read ready, on its first completion since the load; whether the bound is too tight for a cold seat or the engine was wedged is not settled", + Evidence: "agent/liveness.go:268; 4 rows in 30 days, all on one node's pool alias", + Action: notInvestigated + ": time that seat's first completion after a load against the bound"}, + ReasonStallPrefill: {Kind: CauseUnknown, + RootCause: notInvestigated + ": the rows say no progress in prefill past an allowance computed from the prompt size and the seat's rate; a prompt larger than the allowance or a wedged engine both read so", + Evidence: "agent/liveness.go; 0 rows in 30 days, 246 in the whole ledger (written before the stamp)", + Action: notInvestigated + ": compare the prompt size of those rows with the seat's measured prefill rate"}, + ReasonStallDecode: {Kind: CauseUnknown, + RootCause: notInvestigated + ": no progress while tokens were generated; the rows do not say whether the engine wedged or the allowance was too tight", + Evidence: "agent/liveness.go; 0 rows in 30 days, 51 in the whole ledger (written before the stamp)", + Action: notInvestigated + ": read the engine log at the time of the rows"}, + ReasonStallTool: noRow("agent/liveness.go (silence inside a tool call)"), + ReasonStallRepack: {Kind: CauseUnknown, + RootCause: notInvestigated + ": no progress in the structured re-pack of an answer; why is not on the row", + Evidence: "agent/liveness.go (`structured re-pack unreachable: stalled`); 0 rows in 30 days, 99 in the whole ledger (written before the stamp)", + Action: notInvestigated + ": read the re-pack requests of those rows against the seat's log"}, + ReasonStallEngine: noRow("agent/liveness.go (the engine did no work, or kept stepping without a token)"), + ReasonStallOther: noRow("delegate/reasoncode.go stallReasonCode (a stall whose phase the reason does not name)"), + ReasonBudget: {Kind: CauseCallerContract, Investigated: true, + RootCause: "A limit the call carried ended the run before the agent finished: the call's own deadline (`call deadline reached`, `wall timeout after ... the caller's deadline`), a cancellation filed under this class (`agent loop: canceled`), or the step budget (`step budget exhausted`). Most are the caller's deadline, not a node ceiling.", + Evidence: "core/calldeadline.go, pipeline/liveness.go:411, pipeline/agenttask.go:1295; 250 rows in 30 days, 241 from research subtasks", + Action: "give long research subtasks a longer call deadline or fewer subtasks per call; a run that ends on the step budget needs a larger max_steps."}, + ReasonAbstention: {Kind: CauseNodeConfig, Investigated: true, + RootCause: "The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call.", + Evidence: "pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days, 107 on one vision seat's questions", + Action: "raise the vision lane's output cap for that seat, or ask for shorter answers; not investigated: why those answers run past the cap."}, + ReasonContract: {Kind: CauseCallerContract, Investigated: true, + RootCause: "The caller named something the node does not serve: an image family the node does not carry.", + Evidence: "config/families.go:730; 4 rows in 30 days, all one family name on generate_image", + Action: "the caller names a served family, or the node adds that family."}, + ReasonConfig: {Kind: CauseNodeConfig, Investigated: true, + RootCause: "A seat was set up so that it cannot serve the call: its tool-call parser does not match its chat template, so a tool call came back as text.", + Evidence: "pipeline/agenttask.go:1303; 2 rows in 30 days, both on one seat", + Action: "set that seat's --tool-call-parser to the one its chat template needs."}, + ReasonWrite: noRow("pipeline/agenttask.go (the write door refused: agent_allow_write false, or a write set past a door cap)"), + ReasonInfrastructure: {Kind: CauseUpstream, Investigated: true, + RootCause: "Mostly a seat whose engine was gone from llama-swap's /running (the node's defer class infrastructure carrying the same sentence as seat_down), then a vision seat that was cold and timed out. Some rows are a capacity hold filed here (see the followups).", + Evidence: "agent/seatdown.go:185, pipeline_vision.go; 195 rows in 30 days: research 90, fleet 82; about 63 delegate rows read `local seat reserved (gpu lease ...)`", + Action: "see seat_down; not investigated: why the engine goes away. The lease-hold rows are a classification defect, not an infrastructure fault."}, + ReasonBadInput: {Kind: CauseCallerContract, Investigated: true, + RootCause: "The input cannot be used as given: an image path that does not exist, a template variable the template does not declare.", + Evidence: "pipeline (image load) and compose_video BAD_INPUT; 2 rows in 30 days", + Action: "none: working as designed. The caller fixes the input."}, + ReasonOther: {Kind: CauseUnknown, + RootCause: notInvestigated + ": the class is the residue no rule names, a mix of causes; the largest sampled sentence is a node's `a text lease holds its card; it may take the call later` (a capacity signal no rule names), and 15 rows are the browse door's own errors", + Evidence: "reasoncode.go DeferCode (the residue); 55 rows in 30 days: audio 18, browse 15, video 7, compose 6", + Action: notInvestigated + ": list the distinct reasons of the 55 rows and add a rule per sentence that ties to one class"}, +} diff --git a/ledger_failures.go b/ledger_failures.go index 2ab8073f2..bd596a647 100644 --- a/ledger_failures.go +++ b/ledger_failures.go @@ -56,6 +56,9 @@ type failureClassOut struct { Meaning string `json:"meaning"` RootCause string `json:"root_cause"` StructuralFix string `json:"structural_fix"` + // Cause is the root-cause catalogue's entry for the class (internal/ledger/rootcause.go): what was found by reading + // the emitter and the rows, as opposed to the three table columns above, which a person writes in the page. + Cause ledger.FailureCause `json:"cause"` } // failureReport is what `ledger --failures` prints: one JSON object, no trailing text. @@ -77,7 +80,10 @@ type failureReport struct { UndocumentedClasses []string `json:"undocumented_classes"` // UninvestigatedClasses are the classes in the ledger with no root cause and no structural fix written by a // person: a class with no row (undocumented) is among them. - UninvestigatedClasses []string `json:"uninvestigated_classes"` + UninvestigatedClasses []string `json:"uninvestigated_classes"` + // CausesNotInvestigated are the classes in the ledger whose catalogue entry says "not investigated" (the + // catalogue's cause, apart from the table's): what is still unknown, by name. + CausesNotInvestigated []string `json:"causes_not_investigated"` Classes []failureClassOut `json:"classes"` } @@ -183,6 +189,7 @@ func runLedgerFailures(args []string) error { noteExcluded(reg.ExcludedNonProduction) noteLedgerReading(cfg.LedgerPath, reg) report := joinFailureDocs(reg, docs) + noteCauses(report) if *since > 0 { report.SinceDays = *since } @@ -207,6 +214,15 @@ func noteBuilds(reg ledger.FailureRegister) { } } +// noteCauses prints each class with its cause and action as text on stderr, for a person at the terminal; the same +// entries are in the JSON as `cause`. stdout stays the one JSON value. +func noteCauses(r failureReport) { + fmt.Fprintf(os.Stderr, "failure causes (%d classes in the window, %d not investigated):\n", len(r.Classes), len(r.CausesNotInvestigated)) + for _, c := range r.Classes { + fmt.Fprintf(os.Stderr, " %-22s %6d rows [%s]\n root cause: %s\n action: %s\n", c.Class, c.Count, c.Cause.Kind, c.Cause.RootCause, c.Cause.Action) + } +} + // noteLedgerReading says on stderr what the JSON also says, so a person looking at the terminal sees it: the ledger // was not there (every count is then zero because nothing was read), or some of its lines could not be read. func noteLedgerReading(path string, reg ledger.FailureRegister) { @@ -225,10 +241,13 @@ func joinFailureDocs(reg ledger.FailureRegister, docs map[string]failureDoc) fai LedgerFound: reg.LedgerFound, UnparsedLines: reg.UnparsedLines, JobRows: reg.JobRows, FailureRows: reg.FailureRows, RefusedAttempts: reg.RefusedAttempts, StackFaultRows: reg.StackFaultRows, StackFaultShare: reg.StackFaultShare, Kinds: reg.Kinds, ClassDerivedRows: reg.ClassDerivedRows, CardsTokens: reg.CardsTokens, RowsWithTokens: reg.RowsWithTokens, ExcludedNonProduction: reg.ExcludedNonProduction, - UndocumentedClasses: []string{}, UninvestigatedClasses: []string{}, Classes: make([]failureClassOut, 0, len(reg.Classes)), + UndocumentedClasses: []string{}, UninvestigatedClasses: []string{}, CausesNotInvestigated: []string{}, Classes: make([]failureClassOut, 0, len(reg.Classes)), } for _, c := range reg.Classes { - row := failureClassOut{FailureClass: c} + row := failureClassOut{FailureClass: c, Cause: ledger.FailureCauseOf(c.Class)} + if !row.Cause.Investigated { + out.CausesNotInvestigated = append(out.CausesNotInvestigated, c.Class) + } if d, ok := docs[c.Class]; ok { row.Documented, row.Investigated, row.Meaning, row.RootCause, row.StructuralFix = true, d.Investigated, d.Meaning, d.RootCause, d.StructuralFix } else { @@ -242,6 +261,7 @@ func joinFailureDocs(reg ledger.FailureRegister, docs map[string]failureDoc) fai } sort.Strings(out.UndocumentedClasses) sort.Strings(out.UninvestigatedClasses) + sort.Strings(out.CausesNotInvestigated) return out } diff --git a/testdata/failure-register/expected.json b/testdata/failure-register/expected.json index 660466b36..412b52e20 100644 --- a/testdata/failure-register/expected.json +++ b/testdata/failure-register/expected.json @@ -52,6 +52,12 @@ "other", "stall_prefill" ], + "causes_not_investigated": [ + "failed_verification", + "other", + "poll_deadline", + "stall_prefill" + ], "classes": [ { "class": "node_busy", @@ -111,7 +117,14 @@ "investigated": true, "meaning": "A seat answered a capacity defer after admission.", "root_cause": "Another job held what the call needed.", - "structural_fix": "open" + "structural_fix": "open", + "cause": { + "kind": "capacity signal", + "root_cause": "The seat or card the call needed was held by other work when it arrived: the seat was at its local run cap (`seat busy:`), a GPU lease of another class held the card (`gpu busy: held by ...`), or a media job waited for host RAM. The harness defers rather than overload the card.", + "evidence": "pipeline/agenttask.go (seat busy, defer class capacity) and pipeline/pipeline_gpu.go (gpu busy); 628 rows in 30 days: video 247, research 205, audio 109 (the agent pool at its cap, a media lease on the cards, host RAM)", + "action": "none: working as designed. Fewer deferrals need more free card time (keep a card free for media while text leases run) or a higher run cap on a box with room.", + "investigated": true + } }, { "class": "abstention", @@ -178,7 +191,14 @@ "investigated": true, "meaning": "The model ran and gave no usable answer.", "root_cause": "The work, not the stack.", - "structural_fix": "open" + "structural_fix": "open", + "cause": { + "kind": "node configuration", + "root_cause": "The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call.", + "evidence": "pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days, 107 on one vision seat's questions", + "action": "raise the vision lane's output cap for that seat, or ask for shorter answers; not investigated: why those answers run past the cap.", + "investigated": true + } }, { "class": "failed_verification", @@ -230,7 +250,14 @@ "investigated": true, "meaning": "A result failed the contract's own checks.", "root_cause": "not recorded", - "structural_fix": "Tighten the contract (synthetic row)." + "structural_fix": "Tighten the contract (synthetic row).", + "cause": { + "kind": "unknown", + "root_cause": "not investigated: the rows say which check failed (`min_items` on plans, quotes or numbers; a regex) but not whether the seat's answer was wrong or the check too strict; the acceptance list is shape-only (delegate/lint.go warns of it)", + "evidence": "delegate/run.go (failed verification); 357 rows in 30 days, 345 of them from research subtasks", + "action": "not investigated: read the saved output of a sample of failed research subtasks against their acceptance lists", + "investigated": false + } }, { "class": "capacity_wait", @@ -271,7 +298,14 @@ "investigated": false, "meaning": "undocumented", "root_cause": "undocumented", - "structural_fix": "undocumented" + "structural_fix": "undocumented", + "cause": { + "kind": "capacity signal", + "root_cause": "No node had room for the subtask before the call's deadline: 248 of 267 waited until the call's own deadline (not agent_placement_wait_sec), 259 for one node's agent pool alias.", + "evidence": "delegate/run_wait.go:436 and :442; 267 rows in 30 days, 259 on one pool alias", + "action": "none: working as designed. Reducing it needs more worker slots on the pool's node, or fewer research subtasks per call.", + "investigated": true + } }, { "class": "budget", @@ -312,7 +346,14 @@ "investigated": true, "meaning": "A ceiling stopped the run.", "root_cause": "A step budget.", - "structural_fix": "open" + "structural_fix": "open", + "cause": { + "kind": "caller contract", + "root_cause": "A limit the call carried ended the run before the agent finished: the call's own deadline (`call deadline reached`, `wall timeout after ... the caller's deadline`), a cancellation filed under this class (`agent loop: canceled`), or the step budget (`step budget exhausted`). Most are the caller's deadline, not a node ceiling.", + "evidence": "core/calldeadline.go, pipeline/liveness.go:411, pipeline/agenttask.go:1295; 250 rows in 30 days, 241 from research subtasks", + "action": "give long research subtasks a longer call deadline or fewer subtasks per call; a run that ends on the step budget needs a larger max_steps.", + "investigated": true + } }, { "class": "config", @@ -353,7 +394,14 @@ "investigated": false, "meaning": "undocumented", "root_cause": "undocumented", - "structural_fix": "undocumented" + "structural_fix": "undocumented", + "cause": { + "kind": "node configuration", + "root_cause": "A seat was set up so that it cannot serve the call: its tool-call parser does not match its chat template, so a tool call came back as text.", + "evidence": "pipeline/agenttask.go:1303; 2 rows in 30 days, both on one seat", + "action": "set that seat's --tool-call-parser to the one its chat template needs.", + "investigated": true + } }, { "class": "node_unreachable", @@ -394,7 +442,14 @@ "investigated": true, "meaning": "No node answered.", "root_cause": "The network path.", - "structural_fix": "open" + "structural_fix": "open", + "cause": { + "kind": "upstream service", + "root_cause": "A node host did not answer on the network: a dial that timed out (`the connected party did not properly respond`) or a poll that never got an answer. Why the host did not answer (asleep, off, network) is not recorded on the row.", + "evidence": "delegate/run_remote.go; 3 rows in 30 days, 2 on one node", + "action": "none from the harness: the node host or path is the fault; `local-offload doctor` shows which nodes answer now.", + "investigated": true + } }, { "class": "other", @@ -435,7 +490,14 @@ "investigated": false, "meaning": "undocumented", "root_cause": "undocumented", - "structural_fix": "undocumented" + "structural_fix": "undocumented", + "cause": { + "kind": "unknown", + "root_cause": "not investigated: the class is the residue no rule names, a mix of causes; the largest sampled sentence is a node's `a text lease holds its card; it may take the call later` (a capacity signal no rule names), and 15 rows are the browse door's own errors", + "evidence": "reasoncode.go DeferCode (the residue); 55 rows in 30 days: audio 18, browse 15, video 7, compose 6", + "action": "not investigated: list the distinct reasons of the 55 rows and add a rule per sentence that ties to one class", + "investigated": false + } }, { "class": "poll_deadline", @@ -476,7 +538,14 @@ "investigated": true, "meaning": "A node owned the job and it did not finish inside the poll budget.", "root_cause": "A job larger than the budget.", - "structural_fix": "open" + "structural_fix": "open", + "cause": { + "kind": "unknown", + "root_cause": "not investigated: the class does not separate a job longer than the poll budget from a node that stopped making progress (failurekinds.go says so), and the 3 rows differ (a poll error, a stale last-progress, the node's own wall)", + "evidence": "delegate/run_remote.go (poll deadline after); 3 rows in 30 days", + "action": "not investigated: for each row, compare the node's own record of the job with the delegator's poll budget", + "investigated": false + } }, { "class": "stall_prefill", @@ -517,7 +586,14 @@ "investigated": false, "meaning": "The run went silent while the prompt was being processed.", "root_cause": "Not recorded.", - "structural_fix": "open" + "structural_fix": "open", + "cause": { + "kind": "unknown", + "root_cause": "not investigated: the rows say no progress in prefill past an allowance computed from the prompt size and the seat's rate; a prompt larger than the allowance or a wedged engine both read so", + "evidence": "agent/liveness.go; 0 rows in 30 days, 246 in the whole ledger (written before the stamp)", + "action": "not investigated: compare the prompt size of those rows with the seat's measured prefill rate", + "investigated": false + } } ] } diff --git a/trace_corpus.go b/trace_corpus.go index da5d7107d..863ecf361 100644 --- a/trace_corpus.go +++ b/trace_corpus.go @@ -955,6 +955,55 @@ func tcMarshalReport(rep tcReport) ([]byte, error) { return append(b, '\n'), nil } +// tcProposerNeeded is the count the recipe proposer's trigger needs (ADR 0036, amendment 2026-11-12): +// runs that carry a measured pager report. ADR 0036 asks for a corpus "in the hundreds", and 100 is +// the lowest reading of that. +const tcProposerNeeded = 100 + +// tcTrigger is ADR 0036's proposer trigger as numbers. Have counts the runs whose pager report is +// measured (the loop evicted something, so the re-fetch question has an answer for that run): the +// runs a proposer could learn from. TraceBearing is the raw count the trigger used to read, kept +// beside it so the gap is visible. The second half of the trigger, a frozen pre-registration for +// the seat the proposer is for, is not in the corpus and is not judged here. +type tcTrigger struct { + Needed int `json:"needed"` + Have int `json:"have"` + Met bool `json:"met"` + Counts string `json:"counts"` + TraceBearing int `json:"trace_bearing_runs"` + Runs int `json:"runs"` + LargestSeat string `json:"largest_seat,omitempty"` + LargestHave int `json:"largest_seat_have"` + Note string `json:"note"` +} + +func tcProposerTrigger(rep tcReport) tcTrigger { + t := tcTrigger{Needed: tcProposerNeeded, Have: rep.All.Pager.MeasuredRuns, + Counts: "runs whose pager report is measured (pager.measured_runs)", + TraceBearing: rep.All.TraceBearingRuns, Runs: rep.All.Runs, + Note: "also needs a frozen pre-registration for the seat the proposer is for; that is not in the corpus and is not judged here"} + t.Met = t.Have >= t.Needed + for seat, st := range rep.BySeat { + if m := st.Pager.MeasuredRuns; m > t.LargestHave || (m == t.LargestHave && m > 0 && seat < t.LargestSeat) { + t.LargestSeat, t.LargestHave = seat, m + } + } + return t +} + +func (t tcTrigger) line() string { + met := "NOT MET" + if t.Met { + met = "met" + } + s := fmt.Sprintf("[trace-corpus] proposer trigger (ADR 0036): needed %d / have %d / %s; have counts runs with a measured pager report, of %d trace-bearing runs and %d runs", + t.Needed, t.Have, met, t.TraceBearing, t.Runs) + if t.LargestSeat != "" { + s += fmt.Sprintf("; the largest single named seat has %d (%s)", t.LargestHave, t.LargestSeat) + } + return s +} + // tcCheck compares the report for the corpus against a frozen baseline, byte for byte. A // baseline from a different corpus is refused rather than diffed. func tcCheck(baseline []byte, rep tcReport) error { @@ -1012,7 +1061,7 @@ func tcSet(csv string) map[string]bool { } func runTraceCorpusTo(args []string, stdout, stderr io.Writer) error { - const usage = "usage: local-offload trace-corpus [flags] (harvest takes --traces/--delegation-log/--out; the others take --corpus)" + const usage = "usage: local-offload trace-corpus [flags] (harvest takes --traces/--delegation-log/--out; the others take --corpus)" if len(args) < 1 { return fmt.Errorf("%s", usage) } @@ -1056,7 +1105,7 @@ func runTraceCorpusTo(args []string, stdout, stderr io.Writer) error { TracesDir: *tracesDir, DelegationDir: *delegationDir, Seats: tcSet(*seats), ExcludeSeats: tcSet(*excludeSeats), Since: *since, Until: *until, }, *outPath) - case "readout", "freeze", "check": + case "readout", "trigger", "freeze", "check": if *corpusPath == "" { return fmt.Errorf("trace-corpus %s: --corpus is required", mode) } @@ -1066,7 +1115,17 @@ func runTraceCorpusTo(args []string, stdout, stderr io.Writer) error { } rep := tcBuildReport(runs, hash) switch mode { + case "trigger": + t := tcProposerTrigger(rep) + fmt.Fprintln(stderr, t.line()) + b, err := json.MarshalIndent(t, "", " ") + if err != nil { + return err + } + _, err = stdout.Write(append(b, '\n')) + return err case "readout": + fmt.Fprintln(stderr, tcProposerTrigger(rep).line()) b, err := tcMarshalReport(rep) if err != nil { return err @@ -1104,19 +1163,23 @@ func runTraceCorpusTo(args []string, stdout, stderr io.Writer) error { return nil } default: - return fmt.Errorf("trace-corpus: unknown mode %q (harvest|readout|freeze|check)", mode) + return fmt.Errorf("trace-corpus: unknown mode %q (harvest|readout|trigger|freeze|check)", mode) } } // tcHelp is what `trace-corpus help`, `--help` and `-h` print before the flag list. -const tcHelp = `usage: local-offload trace-corpus [flags] +const tcHelp = `usage: local-offload trace-corpus [flags] trace-corpus harvest [--traces DIR] [--delegation-log DIR] --out CORPUS [--seat LIST] [--exclude-seat LIST] [--since YYYY-MM-DD] [--until YYYY-MM-DD] [--config PATH] read the recorded runs into one corpus file. --since and --until bound the delegation shards by their day, inclusive; each must be a real YYYY-MM-DD date and --since must not be after --until. trace-corpus readout --corpus CORPUS [--out FILE] - print the readout (or write it to FILE, replacing one that is there). + print the readout (or write it to FILE, replacing one that is there). The report is unchanged; + a line on stderr states the recipe proposer's trigger as needed / have / met. + trace-corpus trigger --corpus CORPUS + print ADR 0036's proposer trigger as JSON (needed, have, met) and the same line on stderr. The count + is runs with a measured pager report, not trace-bearing runs. trace-corpus freeze --corpus CORPUS --out BASELINE write the readout as the baseline. The baseline is created exclusively: an existing path is refused, so a new baseline goes to a new --out. From c0b2a78b742c16f8cdada83edd3645583db9d73e Mon Sep 17 00:00:00 2001 From: dmmdea Date: Thu, 12 Nov 2026 10:30:03 -0500 Subject: [PATCH 3/5] File a lease hold, a loop cancellation and a node's folder fault under the class that names them A delegation to a seat held by a text lease was filed infrastructure (now node_busy), the agent loop's cancellation was filed budget (now canceled) and a node's own pipeline-jobs folder failure was filed dispatch_refused (now config). RefileStamped holds the three, applied by Record to new rows and by the failure register to rows already written. A node's whole "a text lease holds its card; it may take the call later" sentence and four of the browse door's typed errors get read-time rules in DeferCode. Root-cause entries, the register page and its fixture golden are updated; no class is renamed and the JSON shape is unchanged. Co-Authored-By: Claude Fable 5.2 --- .../feat-rsi-register-followups-2.md | 7 +++ docs/systems/failure-register.md | 33 ++++++----- internal/ledger/failures.go | 11 +++- internal/ledger/reasoncode.go | 55 ++++++++++++++++++- internal/ledger/rootcause.go | 36 ++++++------ testdata/failure-register/expected.json | 20 +++---- 6 files changed, 116 insertions(+), 46 deletions(-) create mode 100644 changes/unreleased/feat-rsi-register-followups-2.md diff --git a/changes/unreleased/feat-rsi-register-followups-2.md b/changes/unreleased/feat-rsi-register-followups-2.md new file mode 100644 index 000000000..4c461e716 --- /dev/null +++ b/changes/unreleased/feat-rsi-register-followups-2.md @@ -0,0 +1,7 @@ +### Fixed - the failure register files a lease hold, a loop cancellation and a node's folder fault under the class that names them + +- **What was wrong.** Three writers filed real failure rows under a class that does not describe them, and two sentences sat in `other`. A delegation to a local seat held by a text lease (`local seat reserved (gpu lease ...)`) was `infrastructure`, though it is a capacity hold and not a fault. The agent loop's `agent loop: canceled (the parent context ended ...)` was `budget`, though the closed set has `canceled`. A node's own folder failure (`agent contract: creating pipeline-jobs dir: ...`, which reaches the delegator as an ack-time 400) was `dispatch_refused`, a class defined as a fact about the request or the credentials, though the fault is the node's configuration. In `other`, a node's `a text lease holds its card; it may take the call later` is a capacity signal, and the browse door's own typed errors name classes. +- **Where each now goes.** The lease hold is `node_busy` (the class defined as the seat or card being held by other work; `capacity_wait` is the delegator finding no node with room, a different cause with a different fix). The loop cancellation is `canceled`. The folder fault is `config` (the node's configured home does not exist on that machine; `infrastructure` is for a part of the stack that failed while configured right). The node's lease sentence, matched whole, is `node_busy`; the browse door's `BROWSER_UNAVAILABLE` and `DECISION_UNAVAILABLE` are `infrastructure`, `HOST_NOT_ALLOWED` is `contract` and `budget` is `budget`. The browse door's `blocked` ties to no one class and stays in `other`. +- **How.** One function, `RefileStamped` in `internal/ledger/reasoncode.go`, holds the three misfilings, each needing both the old stamp and the writer's own sentence. `Record` applies it when it settles a row's code, so new rows are stamped right; the register applies it when it reads, so rows already in the ledger are counted right (they count in `class_derived_rows`; the ledger file is not rewritten). The five other sentences are rules in `DeferCode`. The wire defer class of the lease-hold defer is unchanged, so the exit code and the run summary still count it as infrastructure. No class is renamed and the JSON shape of `ledger --failures` is unchanged. +- **Docs.** The root-cause entries in `internal/ledger/rootcause.go` and the table on the failure-register page say what was fixed and that older rows are corrected at read time. +- **Check.** The built binary and the installed one, both read-only on the real ledger, `ledger --failures --since 30`: 2,566 failure rows of 8,784 job rows in both, no row added or lost. `infrastructure` 195 to 123, `budget` 250 to 236, `dispatch_refused` 145 to 115, `other` 55 to 30, and `node_busy` 628 to 721, `canceled` 29 to 46, `config` 2 to 32, `contract` 4 to 5. Every other class is unchanged, and the stack-fault share falls from 0.1142 to 0.0978 because the lease holds were counted as stack faults before. diff --git a/docs/systems/failure-register.md b/docs/systems/failure-register.md index 54fb97368..ba3f84549 100644 --- a/docs/systems/failure-register.md +++ b/docs/systems/failure-register.md @@ -119,13 +119,13 @@ not investigated. | `queue_deadline` | capacity signal | A node accepted the job and its backlog never reached it inside the queue patience, and the node did not confirm taking it back. Jobs queue behind other work on a full node. | delegate/run_remote.go (queue deadline after); 0 rows in 30 days, 318 in the whole ledger, all written before the withdrawal stamp existed | none: working as designed (the delegator withdraws and re-places since ADR 0064; queue_withdrawn is the same event confirmed). | | `queue_withdrawn` | capacity signal | The same event as queue_deadline, and the node confirmed taking the job back: it sat in the node's backlog (`accepted`, never `running`) past the queue patience and was withdrawn. | delegate/run_remote.go; 186 rows in 30 days, 182 from research subtasks, 111 on one node | none: working as designed. If one node keeps most of them (111 of 186), check its concurrency against the headroom it advertises. | | `poll_deadline` | unknown | not investigated: the class does not separate a job longer than the poll budget from a node that stopped making progress (failurekinds.go says so), and the 3 rows differ (a poll error, a stale last-progress, the node's own wall) | delegate/run_remote.go (poll deadline after); 3 rows in 30 days | not investigated: for each row, compare the node's own record of the job with the delegator's poll budget | -| `canceled` | caller contract | The caller's context ended (the caller went away, or the call's own deadline passed) before or while a subtask ran. | delegate/calldeadline.go; 29 rows in 30 days: `canceled: context canceled` and `the call's context ended before this subtask started` | none: working as designed. | +| `canceled` | caller contract | The caller's context ended (the caller went away, or the call's own deadline passed) before or while a subtask ran. It also holds the node's `agent loop: canceled (the parent context ended ...)`, which the wire files as a budget defer because the wire has no canceled class; the ledger had filed it `budget` until 2026-11-12. | delegate/calldeadline.go and ledger RefileStamped (pipeline/agenttask.go:1295); 29 rows in 30 days as stamped: `canceled: context canceled` and `the call's context ended before this subtask started`, and 17 more once the loop cancellations are read here | none: working as designed. | | `node_unreachable` | upstream service | A node host did not answer on the network: a dial that timed out (`the connected party did not properly respond`) or a poll that never got an answer. Why the host did not answer (asleep, off, network) is not recorded on the row. | delegate/run_remote.go; 3 rows in 30 days, 2 on one node | none from the harness: the node host or path is the fault; `local-offload doctor` shows which nodes answer now. | | `job_lost` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | delegate/run_remote.go (a poll 404 after an acknowledged dispatch); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | -| `dispatch_refused` | node configuration | A node refused the dispatch itself, and the sampled rows give two node-side causes: the node rejected the delegator's bearer token (status 401 `unauthorized`, 112 rows, one node: a fleet_auth_token mismatch) and a node whose configured data folder could not be created (`creating pipeline-jobs dir`, 33 rows, one node: the path is not on that machine now). | delegate/calldeadline.go (401 unauthorized) and fleetnode/tasks.go:1031 (creating pipeline-jobs dir); 145 rows in 30 days, 145 on one node's seat | re-pair that node's token (`local-offload doctor` checks fleet tokens) and fix or remove the node's missing data folder; both are edits on the node. | +| `dispatch_refused` | node configuration | A node refused the dispatch itself: it rejected the delegator's bearer token (status 401 `unauthorized`, one node: a fleet_auth_token mismatch). Until 2026-11-12 the class also held a node whose own data folder could not be created (`creating pipeline-jobs dir`, an ack-time 400); that is a fault of the node's configuration and not of the request, and it is now filed `config`. | delegate/calldeadline.go (401 unauthorized); 145 rows in 30 days as stamped, 145 on one node's seat: 115 are the 401 and 30 the folder fault, which ledger RefileStamped now reads as config | re-pair that node's token (`local-offload doctor` checks fleet tokens); an edit on the node. | | `remote_error` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | delegate/run_remote.go (a node's job reported errored, or its answer was unusable); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | | `capacity_wait` | capacity signal | No node had room for the subtask before the call's deadline: 248 of 267 waited until the call's own deadline (not agent_placement_wait_sec), 259 for one node's agent pool alias. | delegate/run_wait.go:436 and :442; 267 rows in 30 days, 259 on one pool alias | none: working as designed. Reducing it needs more worker slots on the pool's node, or fewer research subtasks per call. | -| `node_busy` | capacity signal | The seat or card the call needed was held by other work when it arrived: the seat was at its local run cap (`seat busy:`), a GPU lease of another class held the card (`gpu busy: held by ...`), or a media job waited for host RAM. The harness defers rather than overload the card. | pipeline/agenttask.go (seat busy, defer class capacity) and pipeline/pipeline_gpu.go (gpu busy); 628 rows in 30 days: video 247, research 205, audio 109 (the agent pool at its cap, a media lease on the cards, host RAM) | none: working as designed. Fewer deferrals need more free card time (keep a card free for media while text leases run) or a higher run cap on a box with room. | +| `node_busy` | capacity signal | The seat or card the call needed was held by other work when it arrived: the seat was at its local run cap (`seat busy:`), a GPU lease of another class held the card (`gpu busy: held by ...`), or a media job waited for host RAM. It also holds the two text-lease holds filed elsewhere until 2026-11-12: the delegator's `local seat reserved (gpu lease ...)` (a text lease holds the only seat the call may use and no remote had room; filed `infrastructure`) and a node's `a text lease holds its card; it may take the call later` (filed `other`). The harness defers rather than overload the card. | pipeline/agenttask.go (seat busy, defer class capacity), pipeline/pipeline_gpu.go (gpu busy), ledger RefileStamped and DeferCode (the lease holds); 628 rows in 30 days as stamped: video 247, research 205, audio 109 (the agent pool at its cap, a media lease on the cards, host RAM), and 93 more once the lease holds are read here (79 from infrastructure, 14 from other) | none: working as designed. Fewer deferrals need more free card time (keep a card free for media while text leases run) or a higher run cap on a box with room. | | `shed` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | delegate (sheddable work, priority -1, found no idle node); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | | `no_eligible_node` | upstream service | A lane found no fleet node able to take the call. In the sampled rows each node probed failed its health check with a timeout (`context deadline exceeded`), so the nodes were not answering, not refusing for room. Why they did not answer is not recorded. | the remote lanes' placement sentences (forecastremote and the others) and delegate route=remote; 76 rows in 30 days, 48 of them the vision lane | none from the harness: bring the nodes back (`local-offload doctor`); not investigated: why the probes timed out at the failure times. | | `seat_down` | upstream service | The inference engine behind the seat was gone from llama-swap's /running, or refused connections, when the run needed it, and did not return inside the node's recovery wait. What stopped the engine is not recorded. | agent/seatdown.go:185; 89 rows in 30 days, 87 on one node's pool alias | not investigated: why that node's engine goes away; read its llama-swap log at the failure times. The harness already waits and re-issues (ADR 0066). | @@ -137,19 +137,16 @@ not investigated. | `stall_repack` | unknown | not investigated: no progress in the structured re-pack of an answer; why is not on the row | agent/liveness.go (`structured re-pack unreachable: stalled`); 0 rows in 30 days, 99 in the whole ledger (written before the stamp) | not investigated: read the re-pack requests of those rows against the seat's log | | `stall_engine` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | agent/liveness.go (the engine did no work, or kept stepping without a token); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | | `stall_other` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | delegate/reasoncode.go stallReasonCode (a stall whose phase the reason does not name); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | -| `budget` | caller contract | A limit the call carried ended the run before the agent finished: the call's own deadline (`call deadline reached`, `wall timeout after ... the caller's deadline`), a cancellation filed under this class (`agent loop: canceled`), or the step budget (`step budget exhausted`). Most are the caller's deadline, not a node ceiling. | core/calldeadline.go, pipeline/liveness.go:411, pipeline/agenttask.go:1295; 250 rows in 30 days, 241 from research subtasks | give long research subtasks a longer call deadline or fewer subtasks per call; a run that ends on the step budget needs a larger max_steps. | +| `budget` | caller contract | A limit the call carried ended the run before the agent finished: the call's own deadline (`call deadline reached`, `wall timeout after ... the caller's deadline`), or the step budget (`step budget exhausted`). Most are the caller's deadline, not a node ceiling. The agent loop's `agent loop: canceled` was filed here until 2026-11-12 and is now `canceled`; the browse door's `browse: budget:` (max_actions spent) is read here. | core/calldeadline.go, pipeline/liveness.go:411, pipeline/agenttask.go:1295; 250 rows in 30 days as stamped, 241 from research subtasks; 17 of them are loop cancellations that ledger RefileStamped now reads as canceled | give long research subtasks a longer call deadline or fewer subtasks per call; a run that ends on the step budget needs a larger max_steps. | | `abstention` | node configuration | The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call. | pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days, 107 on one vision seat's questions | raise the vision lane's output cap for that seat, or ask for shorter answers; not investigated: why those answers run past the cap. | | `contract` | caller contract | The caller named something the node does not serve: an image family the node does not carry. | config/families.go:730; 4 rows in 30 days, all one family name on generate_image | the caller names a served family, or the node adds that family. | -| `config` | node configuration | A seat was set up so that it cannot serve the call: its tool-call parser does not match its chat template, so a tool call came back as text. | pipeline/agenttask.go:1303; 2 rows in 30 days, both on one seat | set that seat's --tool-call-parser to the one its chat template needs. | +| `config` | node configuration | A seat or node was set up so that it cannot serve the call: a seat's tool-call parser does not match its chat template, so a tool call came back as text (2 rows), or a node's configured data folder does not exist on that machine, so it could not create its pipeline-jobs folder for a dispatch (30 rows, one node; filed `dispatch_refused` until 2026-11-12). | pipeline/agenttask.go:1303 and fleetnode/tasks.go:1031 (read by ledger RefileStamped); 2 rows in 30 days as stamped, 32 once the folder fault is read here | set that seat's --tool-call-parser to the one its chat template needs; fix or remove the node's missing data folder (its `home`). | | `write` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | pipeline/agenttask.go (the write door refused: agent_allow_write false, or a write set past a door cap); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | -| `infrastructure` | upstream service | Mostly a seat whose engine was gone from llama-swap's /running (the node's defer class infrastructure carrying the same sentence as seat_down), then a vision seat that was cold and timed out. Some rows are a capacity hold filed here (see the followups). | agent/seatdown.go:185, pipeline_vision.go; 195 rows in 30 days: research 90, fleet 82; about 63 delegate rows read `local seat reserved (gpu lease ...)` | see seat_down; not investigated: why the engine goes away. The lease-hold rows are a classification defect, not an infrastructure fault. | +| `infrastructure` | upstream service | Mostly a seat whose engine was gone from llama-swap's /running (the node's defer class infrastructure carrying the same sentence as seat_down), then a vision seat that was cold and timed out, and the browse door's `browse: BROWSER_UNAVAILABLE:` and `browse: DECISION_UNAVAILABLE:`. The delegator's `local seat reserved (gpu lease ...)` was filed here until 2026-11-12; it is a capacity hold and is now `node_busy` (the wire class of that defer stays infrastructure, so the exit code and summary still count it). | agent/seatdown.go:185, pipeline_vision.go; 195 rows in 30 days as stamped: research 90, fleet 82; 79 of them were `local seat reserved (gpu lease ...)`, which ledger RefileStamped now reads as node_busy | see seat_down; not investigated: why the engine goes away. | | `bad_input` | caller contract | The input cannot be used as given: an image path that does not exist, a template variable the template does not declare. | pipeline (image load) and compose_video BAD_INPUT; 2 rows in 30 days | none: working as designed. The caller fixes the input. | -| `other` | unknown | not investigated: the class is the residue no rule names, a mix of causes; the largest sampled sentence is a node's `a text lease holds its card; it may take the call later` (a capacity signal no rule names), and 15 rows are the browse door's own errors | reasoncode.go DeferCode (the residue); 55 rows in 30 days: audio 18, browse 15, video 7, compose 6 | not investigated: list the distinct reasons of the 55 rows and add a rule per sentence that ties to one class | +| `other` | unknown | not investigated: the class is the residue no rule names, a mix of causes. Since 2026-11-12 a node's `a text lease holds its card; it may take the call later` is read as node_busy, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors by their own classes; what is left is mostly the browse door's `blocked` (the model reported it cannot proceed, 4 rows) and media and compose sentences no rule names | reasoncode.go DeferCode (the residue); 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the rules above are read | not investigated: list the distinct reasons of the rows left and add a rule per sentence that ties to one class | -Reading the rows also found two misfilings, not changed by the catalogue: a delegate row for a local seat held by a text -lease (`local seat reserved (gpu lease ...)`) is filed `infrastructure` though it is a capacity hold, and the agent loop's -`agent loop: canceled` (the caller gave up, or the box is draining) is filed `budget` though `canceled` exists. A -transport 4xx from a node's own folder failure is filed `dispatch_refused`, a class defined as a fact about the request. +Reading the rows found three misfilings and two sentences left in `other`, all fixed on 2026-11-12 (`RefileStamped` and `DeferCode` in `internal/ledger/reasoncode.go`): a delegate row for a local seat held by a text lease (`local seat reserved (gpu lease ...)`) was filed `infrastructure` and is now `node_busy`, because the class is defined as the seat or card being held by other work (`capacity_wait` is the delegator finding no node with room, a different cause with a different fix); the agent loop's `agent loop: canceled` was filed `budget` and is now `canceled`; and a node's own folder failure (`agent contract: creating pipeline-jobs dir`, an ack-time 400) was filed `dispatch_refused`, a fact about the request or the credentials, and is now `config` (the node's configured home does not exist on that machine; `infrastructure` is for a stack part that failed while configured right). Rows written before the fix are corrected when the register is read: a row stamped with the old class and carrying the writer's own sentence is counted under the new one and in `class_derived_rows`; the ledger file is not rewritten. Rows written from now are stamped right by `Record`. The wire class of the lease-hold defer stays `infrastructure`, so the exit code and the summary count it as before. In `other`, a node's `a text lease holds its card; it may take the call later` is now `node_busy`, and the browse door's `BROWSER_UNAVAILABLE` and `DECISION_UNAVAILABLE` are `infrastructure`, `HOST_NOT_ALLOWED` is `contract` and `budget` is `budget`; its `blocked` (the model reported it cannot proceed) ties to no one class and stays in `other`. ### The fields @@ -267,9 +264,13 @@ The table is the human-maintained half of the register. ### Reading a row written before the code was stamped -A row with a stamped reason code is read as stamped, with one exception: `other` is what the ledger files when no member +A row with a stamped reason code is read as stamped, with two exceptions. `other` is what the ledger files when no member named the row at write time, so a deferred row stamped `other` is classed again and keeps `other` unless a rule below now -names a member. A deferred row without a code is classed the same way. Both are counted in `class_derived_rows` and classed +names a member. And three stamps were wrong until 2026-11-12 (`RefileStamped`, applied by `Record` to new rows and by the +register to old ones, and counted in `class_derived_rows` too): `infrastructure` on a reason holding `local seat reserved (` +is `node_busy`; `budget` on a reason starting `agent loop: canceled (the parent context ended` is `canceled`; and +`dispatch_refused` on a reason holding `agent contract: creating pipeline-jobs dir` is `config`. Each needs the wrong stamp +and the writer's own sentence, so no other row of those classes moves. A deferred row without a code is classed the same way. Both are counted in `class_derived_rows` and classed by `DeferCode` in [`internal/ledger/reasoncode.go`](../../internal/ledger/reasoncode.go): its error class first, then the prefix of its reason, and `other` when neither names it. The prefix rules cover only sentences the harness itself writes, and each maps to the class its own writer files the same defer under. A rule is added only when a @@ -285,6 +286,10 @@ real sentence in the source ties to an existing class without doubt; the sentenc advertises ...` are `no_eligible_node`. - `dispatch : status 503|429: ...` is `queue_full`: the delegator files every capacity refusal of a dispatch so, whatever the node's body says (queue full, node leased, too many requests). +- `a text lease holds its card; it may take the call later`, the whole sentence of a node's capacity defer on a media lane, is + `node_busy`; the shorter `a text lease holds its card` also ends a probe list that found no node, and is not matched. +- The browse door's `browse: BROWSER_UNAVAILABLE:` and `browse: DECISION_UNAVAILABLE:` are `infrastructure`, + `browse: HOST_NOT_ALLOWED:` is `contract` and `browse: budget:` is `budget`. A door that holds a result's wire defer class (the remote lanes and the accelerator forward) stamps the code when it writes the row: the error class and the sentence first, then the defer class (`capacity` is `node_busy`), so such a defer @@ -292,7 +297,7 @@ is no longer left to read time. A deferred row whose writer gave no reason is fi `deferred: no reason recorded`, which the register reads as `other`. Not yet ruled, because the sentence does not tie to one class without doubt: a bare `node leased`, `no node that answered -is called ...`, `no fleet node can take this edit now` and the browse door's own error classes. +is called ...`, `no fleet node can take this edit now` and the browse door's `blocked` (the model reported it cannot proceed) and its other error classes. ### Wire defer classes diff --git a/internal/ledger/failures.go b/internal/ledger/failures.go index 65b00bb2c..9909bca59 100644 --- a/internal/ledger/failures.go +++ b/internal/ledger/failures.go @@ -112,8 +112,8 @@ type FailureRegister struct { StackFaultRows int `json:"stack_fault_rows"` StackFaultShare *float64 `json:"stack_fault_share"` // ClassDerivedRows is how many failed rows were classed here by DeferCode, the classifier Record itself uses: - // rows with no usable reason code (from before it was stamped, or a code outside the closed set) and deferred - // rows stamped "other" that a rule now names. + // rows with no usable reason code (from before it was stamped, or a code outside the closed set), deferred + // rows stamped "other" that a rule now names, and rows whose stamp is a known misfiling (RefileStamped). ClassDerivedRows int `json:"class_derived_rows"` // CardsTokens and RowsWithTokens total the classes' (same rule, same nil). CardsTokens *int64 `json:"cards_tokens"` @@ -249,12 +249,17 @@ func failureClassOf(e Entry) (class string, derived, failed bool) { if IsReasonCode(e.ReasonCode) && e.ReasonCode != ReasonOK && e.ReasonCode != ReasonStarted { // "other" is the residue Record files when no member named the row at write time, so it is the one stamp a // later build may read better: a deferred row stamped "other" is classed again, and keeps "other" unless a - // sentence the harness wrote now names a member. Every other stamp is read as stamped. + // sentence the harness wrote now names a member. A stamp that was a known misfiling (RefileStamped: a lease + // hold filed infrastructure, a cancellation filed budget, a node's folder fault filed dispatch_refused, all + // until 2026-11-12) reads as the class it should have carried. Every other stamp is read as stamped. if e.ReasonCode == ReasonOther && e.Deferred { if c := DeferCode(e.ErrClass, e.Reason); c != ReasonOther { return c, true, true } } + if c := RefileStamped(e.ReasonCode, e.Reason); c != e.ReasonCode { + return c, true, true + } return e.ReasonCode, false, true } switch { diff --git a/internal/ledger/reasoncode.go b/internal/ledger/reasoncode.go index 2ab11ef1f..829d5ab44 100644 --- a/internal/ledger/reasoncode.go +++ b/internal/ledger/reasoncode.go @@ -135,7 +135,7 @@ var reasonCodes = []string{ func settleReasonCode(e Entry) string { switch { case IsReasonCode(e.ReasonCode): - return e.ReasonCode + return RefileStamped(e.ReasonCode, e.Reason) case e.Phase == PhaseStarted: return ReasonStarted case e.Deferred: @@ -212,6 +212,12 @@ var reasonPrefixCodes = []struct{ prefix, code string }{ // A named family the node does not serve is the request's; a family whose overlay is incomplete is the config's. {"unknown image family", ReasonContract}, {"unknown edit family", ReasonContract}, {"imagegen_families[", ReasonConfig}, {"probe:", ReasonInfrastructure}, {"generate_svg: write:", ReasonInfrastructure}, {"generate_svg:", ReasonInfrastructure}, + // The browse door's own typed errors (pipeline/browse.go: "browse: : "; docs/systems/browse-lane.md), + // the four that name one member: no browser or no decision endpoint to drive it is the stack (as lane_unavailable), + // a page outside allow_hosts is the request's, and max_actions spent is a limit the call carried. The rest of the + // door's classes ("blocked", where the model reported it cannot proceed, and the others) are not named here. + {"browse: BROWSER_UNAVAILABLE:", ReasonInfrastructure}, {"browse: DECISION_UNAVAILABLE:", ReasonInfrastructure}, + {"browse: HOST_NOT_ALLOWED:", ReasonContract}, {"browse: budget:", ReasonBudget}, } // legacyReasonPrefixCodes classify a deferred row by the sentence the harness ITSELF wrote, for the rows @@ -264,10 +270,57 @@ func legacyReasonCode(reason string) string { return ReasonNoEligibleNode case dispatchCapacityRe.MatchString(reason): return ReasonQueueFull + case strings.Contains(reason, textLeaseHoldsCardSentence): + return ReasonNodeBusy } return "" } +// textLeaseHoldsCardSentence is the whole sentence a media lane writes when a node's card is under a text lease and +// the node itself would take the call once the lease ends (internal/mediaremote/named.go, defer class capacity): +// the node's own capacity defer, node_busy. It is matched whole on purpose: the shorter "a text lease holds its +// card" also ends a different sentence (a probe list in which no node served the call: no_eligible_node). +const textLeaseHoldsCardSentence = "a text lease holds its card; it may take the call later" + +// Reasons that were stamped under the wrong class before 2026-11-12, each the sentence one harness writer owns and +// the class its stamp should have been. RefileStamped is the one place that knows them. +const ( + // localSeatReservedSentence opens (or follows "route=spread: " in) the defer the delegator files when the only + // seat it may use is held by a text lease and no remote had room (delegate/run_wait.go reservedDefer). The wire + // class of that defer stays infrastructure (the exit code and the summary count it so); the ledger class is the + // capacity hold it is: the card is held by other work, node_busy. + localSeatReservedSentence = "local seat reserved (" + // agentLoopCanceledSentence is the node's defer when the agent loop's parent context ended (pipeline/agenttask.go). + // The wire class is budget because the wire has no canceled; the ledger has. + agentLoopCanceledSentence = "agent loop: canceled (the parent context ended" + // nodeJobsDirSentence is a node that could not create its own pipeline-jobs folder (fleetnode/tasks.go), which + // the delegator reads as an ack-time 400 and files as dispatch_refused, a fact about the request or the + // credentials. The folder is the node's configured home, so the fault is the node's configuration. + nodeJobsDirSentence = "agent contract: creating pipeline-jobs dir" +) + +// RefileStamped is the class a row's stamped code should read as, given its reason: the stamp itself when the pair +// is not one of the known misfilings. It is applied where a stamp is settled (Record, so new rows are written +// right) and where one is read (failureClassOf, so rows written before 2026-11-12 count right), and nowhere else. +// Each rule needs BOTH the wrong stamp and the writer's own sentence, so no other row of those classes can move. +func RefileStamped(code, reason string) string { + switch code { + case ReasonInfrastructure: + if strings.Contains(reason, localSeatReservedSentence) { + return ReasonNodeBusy + } + case ReasonBudget: + if strings.HasPrefix(reason, agentLoopCanceledSentence) { + return ReasonCanceled + } + case ReasonDispatchRefused: + if strings.Contains(reason, nodeJobsDirSentence) { + return ReasonConfig + } + } + return code +} + // stallCodeOf names the phase a node's liveness verdict ("stalled: no progress for 300s in prefill (...)") went // silent in, by the phase words of the run registry, as the delegator reads them (delegate.stallReasonCode). func stallCodeOf(reason string) string { diff --git a/internal/ledger/rootcause.go b/internal/ledger/rootcause.go index afba36a23..fa2fa2114 100644 --- a/internal/ledger/rootcause.go +++ b/internal/ledger/rootcause.go @@ -70,8 +70,8 @@ func noRow(emitter string) FailureCause { var failureCauses = map[string]FailureCause{ ReasonNodeBusy: {Kind: CauseCapacity, Investigated: true, - RootCause: "The seat or card the call needed was held by other work when it arrived: the seat was at its local run cap (`seat busy:`), a GPU lease of another class held the card (`gpu busy: held by ...`), or a media job waited for host RAM. The harness defers rather than overload the card.", - Evidence: "pipeline/agenttask.go (seat busy, defer class capacity) and pipeline/pipeline_gpu.go (gpu busy); 628 rows in 30 days: video 247, research 205, audio 109 (the agent pool at its cap, a media lease on the cards, host RAM)", + RootCause: "The seat or card the call needed was held by other work when it arrived: the seat was at its local run cap (`seat busy:`), a GPU lease of another class held the card (`gpu busy: held by ...`), or a media job waited for host RAM. It also holds the two text-lease holds filed elsewhere until 2026-11-12: the delegator's `local seat reserved (gpu lease ...)` (a text lease holds the only seat the call may use and no remote had room; filed `infrastructure`) and a node's `a text lease holds its card; it may take the call later` (filed `other`). The harness defers rather than overload the card.", + Evidence: "pipeline/agenttask.go (seat busy, defer class capacity), pipeline/pipeline_gpu.go (gpu busy), ledger RefileStamped and DeferCode (the lease holds); 628 rows in 30 days as stamped: video 247, research 205, audio 109 (the agent pool at its cap, a media lease on the cards, host RAM), and 93 more once the lease holds are read here (79 from infrastructure, 14 from other)", Action: "none: working as designed. Fewer deferrals need more free card time (keep a card free for media while text leases run) or a higher run cap on a box with room."}, ReasonFailedVerification: {Kind: CauseUnknown, RootCause: notInvestigated + ": the rows say which check failed (`min_items` on plans, quotes or numbers; a regex) but not whether the seat's answer was wrong or the check too strict; the acceptance list is shape-only (delegate/lint.go warns of it)", @@ -94,8 +94,8 @@ var failureCauses = map[string]FailureCause{ Evidence: "delegate/run_remote.go (poll deadline after); 3 rows in 30 days", Action: notInvestigated + ": for each row, compare the node's own record of the job with the delegator's poll budget"}, ReasonCanceled: {Kind: CauseCallerContract, Investigated: true, - RootCause: "The caller's context ended (the caller went away, or the call's own deadline passed) before or while a subtask ran.", - Evidence: "delegate/calldeadline.go; 29 rows in 30 days: `canceled: context canceled` and `the call's context ended before this subtask started`", + RootCause: "The caller's context ended (the caller went away, or the call's own deadline passed) before or while a subtask ran. It also holds the node's `agent loop: canceled (the parent context ended ...)`, which the wire files as a budget defer because the wire has no canceled class; the ledger had filed it `budget` until 2026-11-12.", + Evidence: "delegate/calldeadline.go and ledger RefileStamped (pipeline/agenttask.go:1295); 29 rows in 30 days as stamped: `canceled: context canceled` and `the call's context ended before this subtask started`, and 17 more once the loop cancellations are read here", Action: "none: working as designed."}, ReasonNodeUnreachable: {Kind: CauseUpstream, Investigated: true, RootCause: "A node host did not answer on the network: a dial that timed out (`the connected party did not properly respond`) or a poll that never got an answer. Why the host did not answer (asleep, off, network) is not recorded on the row.", @@ -103,9 +103,9 @@ var failureCauses = map[string]FailureCause{ Action: "none from the harness: the node host or path is the fault; `local-offload doctor` shows which nodes answer now."}, ReasonJobLost: noRow("delegate/run_remote.go (a poll 404 after an acknowledged dispatch)"), ReasonDispatchRefused: {Kind: CauseNodeConfig, Investigated: true, - RootCause: "A node refused the dispatch itself, and the sampled rows give two node-side causes: the node rejected the delegator's bearer token (status 401 `unauthorized`, 112 rows, one node: a fleet_auth_token mismatch) and a node whose configured data folder could not be created (`creating pipeline-jobs dir`, 33 rows, one node: the path is not on that machine now).", - Evidence: "delegate/calldeadline.go (401 unauthorized) and fleetnode/tasks.go:1031 (creating pipeline-jobs dir); 145 rows in 30 days, 145 on one node's seat", - Action: "re-pair that node's token (`local-offload doctor` checks fleet tokens) and fix or remove the node's missing data folder; both are edits on the node."}, + RootCause: "A node refused the dispatch itself: it rejected the delegator's bearer token (status 401 `unauthorized`, one node: a fleet_auth_token mismatch). Until 2026-11-12 the class also held a node whose own data folder could not be created (`creating pipeline-jobs dir`, an ack-time 400); that is a fault of the node's configuration and not of the request, and it is now filed `config`.", + Evidence: "delegate/calldeadline.go (401 unauthorized); 145 rows in 30 days as stamped, 145 on one node's seat: 115 are the 401 and 30 the folder fault, which ledger RefileStamped now reads as config", + Action: "re-pair that node's token (`local-offload doctor` checks fleet tokens); an edit on the node."}, ReasonRemoteError: noRow("delegate/run_remote.go (a node's job reported errored, or its answer was unusable)"), ReasonCapacityWait: {Kind: CauseCapacity, Investigated: true, RootCause: "No node had room for the subtask before the call's deadline: 248 of 267 waited until the call's own deadline (not agent_placement_wait_sec), 259 for one node's agent pool alias.", @@ -144,8 +144,8 @@ var failureCauses = map[string]FailureCause{ ReasonStallEngine: noRow("agent/liveness.go (the engine did no work, or kept stepping without a token)"), ReasonStallOther: noRow("delegate/reasoncode.go stallReasonCode (a stall whose phase the reason does not name)"), ReasonBudget: {Kind: CauseCallerContract, Investigated: true, - RootCause: "A limit the call carried ended the run before the agent finished: the call's own deadline (`call deadline reached`, `wall timeout after ... the caller's deadline`), a cancellation filed under this class (`agent loop: canceled`), or the step budget (`step budget exhausted`). Most are the caller's deadline, not a node ceiling.", - Evidence: "core/calldeadline.go, pipeline/liveness.go:411, pipeline/agenttask.go:1295; 250 rows in 30 days, 241 from research subtasks", + RootCause: "A limit the call carried ended the run before the agent finished: the call's own deadline (`call deadline reached`, `wall timeout after ... the caller's deadline`), or the step budget (`step budget exhausted`). Most are the caller's deadline, not a node ceiling. The agent loop's `agent loop: canceled` was filed here until 2026-11-12 and is now `canceled`; the browse door's `browse: budget:` (max_actions spent) is read here.", + Evidence: "core/calldeadline.go, pipeline/liveness.go:411, pipeline/agenttask.go:1295; 250 rows in 30 days as stamped, 241 from research subtasks; 17 of them are loop cancellations that ledger RefileStamped now reads as canceled", Action: "give long research subtasks a longer call deadline or fewer subtasks per call; a run that ends on the step budget needs a larger max_steps."}, ReasonAbstention: {Kind: CauseNodeConfig, Investigated: true, RootCause: "The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call.", @@ -156,20 +156,20 @@ var failureCauses = map[string]FailureCause{ Evidence: "config/families.go:730; 4 rows in 30 days, all one family name on generate_image", Action: "the caller names a served family, or the node adds that family."}, ReasonConfig: {Kind: CauseNodeConfig, Investigated: true, - RootCause: "A seat was set up so that it cannot serve the call: its tool-call parser does not match its chat template, so a tool call came back as text.", - Evidence: "pipeline/agenttask.go:1303; 2 rows in 30 days, both on one seat", - Action: "set that seat's --tool-call-parser to the one its chat template needs."}, + RootCause: "A seat or node was set up so that it cannot serve the call: a seat's tool-call parser does not match its chat template, so a tool call came back as text (2 rows), or a node's configured data folder does not exist on that machine, so it could not create its pipeline-jobs folder for a dispatch (30 rows, one node; filed `dispatch_refused` until 2026-11-12).", + Evidence: "pipeline/agenttask.go:1303 and fleetnode/tasks.go:1031 (read by ledger RefileStamped); 2 rows in 30 days as stamped, 32 once the folder fault is read here", + Action: "set that seat's --tool-call-parser to the one its chat template needs; fix or remove the node's missing data folder (its `home`)."}, ReasonWrite: noRow("pipeline/agenttask.go (the write door refused: agent_allow_write false, or a write set past a door cap)"), ReasonInfrastructure: {Kind: CauseUpstream, Investigated: true, - RootCause: "Mostly a seat whose engine was gone from llama-swap's /running (the node's defer class infrastructure carrying the same sentence as seat_down), then a vision seat that was cold and timed out. Some rows are a capacity hold filed here (see the followups).", - Evidence: "agent/seatdown.go:185, pipeline_vision.go; 195 rows in 30 days: research 90, fleet 82; about 63 delegate rows read `local seat reserved (gpu lease ...)`", - Action: "see seat_down; not investigated: why the engine goes away. The lease-hold rows are a classification defect, not an infrastructure fault."}, + RootCause: "Mostly a seat whose engine was gone from llama-swap's /running (the node's defer class infrastructure carrying the same sentence as seat_down), then a vision seat that was cold and timed out, and the browse door's `browse: BROWSER_UNAVAILABLE:` and `browse: DECISION_UNAVAILABLE:`. The delegator's `local seat reserved (gpu lease ...)` was filed here until 2026-11-12; it is a capacity hold and is now `node_busy` (the wire class of that defer stays infrastructure, so the exit code and summary still count it).", + Evidence: "agent/seatdown.go:185, pipeline_vision.go; 195 rows in 30 days as stamped: research 90, fleet 82; 79 of them were `local seat reserved (gpu lease ...)`, which ledger RefileStamped now reads as node_busy", + Action: "see seat_down; not investigated: why the engine goes away."}, ReasonBadInput: {Kind: CauseCallerContract, Investigated: true, RootCause: "The input cannot be used as given: an image path that does not exist, a template variable the template does not declare.", Evidence: "pipeline (image load) and compose_video BAD_INPUT; 2 rows in 30 days", Action: "none: working as designed. The caller fixes the input."}, ReasonOther: {Kind: CauseUnknown, - RootCause: notInvestigated + ": the class is the residue no rule names, a mix of causes; the largest sampled sentence is a node's `a text lease holds its card; it may take the call later` (a capacity signal no rule names), and 15 rows are the browse door's own errors", - Evidence: "reasoncode.go DeferCode (the residue); 55 rows in 30 days: audio 18, browse 15, video 7, compose 6", - Action: notInvestigated + ": list the distinct reasons of the 55 rows and add a rule per sentence that ties to one class"}, + RootCause: notInvestigated + ": the class is the residue no rule names, a mix of causes. Since 2026-11-12 a node's `a text lease holds its card; it may take the call later` is read as node_busy, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors by their own classes; what is left is mostly the browse door's `blocked` (the model reported it cannot proceed, 4 rows) and media and compose sentences no rule names", + Evidence: "reasoncode.go DeferCode (the residue); 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the rules above are read", + Action: notInvestigated + ": list the distinct reasons of the rows left and add a rule per sentence that ties to one class"}, } diff --git a/testdata/failure-register/expected.json b/testdata/failure-register/expected.json index 412b52e20..2f0164b0c 100644 --- a/testdata/failure-register/expected.json +++ b/testdata/failure-register/expected.json @@ -120,8 +120,8 @@ "structural_fix": "open", "cause": { "kind": "capacity signal", - "root_cause": "The seat or card the call needed was held by other work when it arrived: the seat was at its local run cap (`seat busy:`), a GPU lease of another class held the card (`gpu busy: held by ...`), or a media job waited for host RAM. The harness defers rather than overload the card.", - "evidence": "pipeline/agenttask.go (seat busy, defer class capacity) and pipeline/pipeline_gpu.go (gpu busy); 628 rows in 30 days: video 247, research 205, audio 109 (the agent pool at its cap, a media lease on the cards, host RAM)", + "root_cause": "The seat or card the call needed was held by other work when it arrived: the seat was at its local run cap (`seat busy:`), a GPU lease of another class held the card (`gpu busy: held by ...`), or a media job waited for host RAM. It also holds the two text-lease holds filed elsewhere until 2026-11-12: the delegator's `local seat reserved (gpu lease ...)` (a text lease holds the only seat the call may use and no remote had room; filed `infrastructure`) and a node's `a text lease holds its card; it may take the call later` (filed `other`). The harness defers rather than overload the card.", + "evidence": "pipeline/agenttask.go (seat busy, defer class capacity), pipeline/pipeline_gpu.go (gpu busy), ledger RefileStamped and DeferCode (the lease holds); 628 rows in 30 days as stamped: video 247, research 205, audio 109 (the agent pool at its cap, a media lease on the cards, host RAM), and 93 more once the lease holds are read here (79 from infrastructure, 14 from other)", "action": "none: working as designed. Fewer deferrals need more free card time (keep a card free for media while text leases run) or a higher run cap on a box with room.", "investigated": true } @@ -349,8 +349,8 @@ "structural_fix": "open", "cause": { "kind": "caller contract", - "root_cause": "A limit the call carried ended the run before the agent finished: the call's own deadline (`call deadline reached`, `wall timeout after ... the caller's deadline`), a cancellation filed under this class (`agent loop: canceled`), or the step budget (`step budget exhausted`). Most are the caller's deadline, not a node ceiling.", - "evidence": "core/calldeadline.go, pipeline/liveness.go:411, pipeline/agenttask.go:1295; 250 rows in 30 days, 241 from research subtasks", + "root_cause": "A limit the call carried ended the run before the agent finished: the call's own deadline (`call deadline reached`, `wall timeout after ... the caller's deadline`), or the step budget (`step budget exhausted`). Most are the caller's deadline, not a node ceiling. The agent loop's `agent loop: canceled` was filed here until 2026-11-12 and is now `canceled`; the browse door's `browse: budget:` (max_actions spent) is read here.", + "evidence": "core/calldeadline.go, pipeline/liveness.go:411, pipeline/agenttask.go:1295; 250 rows in 30 days as stamped, 241 from research subtasks; 17 of them are loop cancellations that ledger RefileStamped now reads as canceled", "action": "give long research subtasks a longer call deadline or fewer subtasks per call; a run that ends on the step budget needs a larger max_steps.", "investigated": true } @@ -397,9 +397,9 @@ "structural_fix": "undocumented", "cause": { "kind": "node configuration", - "root_cause": "A seat was set up so that it cannot serve the call: its tool-call parser does not match its chat template, so a tool call came back as text.", - "evidence": "pipeline/agenttask.go:1303; 2 rows in 30 days, both on one seat", - "action": "set that seat's --tool-call-parser to the one its chat template needs.", + "root_cause": "A seat or node was set up so that it cannot serve the call: a seat's tool-call parser does not match its chat template, so a tool call came back as text (2 rows), or a node's configured data folder does not exist on that machine, so it could not create its pipeline-jobs folder for a dispatch (30 rows, one node; filed `dispatch_refused` until 2026-11-12).", + "evidence": "pipeline/agenttask.go:1303 and fleetnode/tasks.go:1031 (read by ledger RefileStamped); 2 rows in 30 days as stamped, 32 once the folder fault is read here", + "action": "set that seat's --tool-call-parser to the one its chat template needs; fix or remove the node's missing data folder (its `home`).", "investigated": true } }, @@ -493,9 +493,9 @@ "structural_fix": "undocumented", "cause": { "kind": "unknown", - "root_cause": "not investigated: the class is the residue no rule names, a mix of causes; the largest sampled sentence is a node's `a text lease holds its card; it may take the call later` (a capacity signal no rule names), and 15 rows are the browse door's own errors", - "evidence": "reasoncode.go DeferCode (the residue); 55 rows in 30 days: audio 18, browse 15, video 7, compose 6", - "action": "not investigated: list the distinct reasons of the 55 rows and add a rule per sentence that ties to one class", + "root_cause": "not investigated: the class is the residue no rule names, a mix of causes. Since 2026-11-12 a node's `a text lease holds its card; it may take the call later` is read as node_busy, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors by their own classes; what is left is mostly the browse door's `blocked` (the model reported it cannot proceed, 4 rows) and media and compose sentences no rule names", + "evidence": "reasoncode.go DeferCode (the residue); 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the rules above are read", + "action": "not investigated: list the distinct reasons of the rows left and add a rule per sentence that ties to one class", "investigated": false } }, From f239c4bd631cd334facf7ddf303b033889355182 Mon Sep 17 00:00:00 2001 From: dmmdea Date: Thu, 12 Nov 2026 10:40:20 -0500 Subject: [PATCH 4/5] docs: say where the ledger class differs from the wire class for a lease hold and a loop cancellation Co-Authored-By: Claude Fable 5.2 --- docs/systems/fleet-node.md | 2 +- internal/delegate/run_wait.go | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/systems/fleet-node.md b/docs/systems/fleet-node.md index da5905f49..07cbbbefb 100644 --- a/docs/systems/fleet-node.md +++ b/docs/systems/fleet-node.md @@ -1837,7 +1837,7 @@ it; the class is what code branches on, because a reason string is prose and an | `structured re-pack attempt /3 (…): re-pack repeated the list item "" times in a row in "" and the answer holds "" time(s): a loop, not content` | `abstention` | Register C14 (ADR 0055 item 10). A COMPLETE object that validated but repeated one list item (or a block of two or three) at least 6 times in a row, where the answer it was written from holds the item fewer times: a loop that closed itself before the cap. It is not published. The next attempt carries the no-repeat instruction and `frequency_penalty` 0.5 (lane `…+no-repeat`); when every attempt does it the call defers here with the finished answer in `output`. The same applies to a cut at the cap that is a loop or that ran on past a budget the answer fits in: the attempt after it is conditioned the same way instead of being the same request again. | | `this node does not open the write door: agent_allow_write is false …` | `write` | 0.122.0, register D-06. The contract asked for `write_root` and this node has not opted in. Its own class because it is neither a broken stack (nothing is wrong with the box) nor an unplaceable contract (another node may have opted in) — the delegator's right reflex is to re-place. On the FLEET path this never appears: `buildAgentRun` refuses at ack with a 400 so the re-placement happens there; the defer is the in-process local path, which has no ack hop. | | `write door: … over the N-file cap` / `… over the N-byte cap` | `write` | The run finished and its write set is past a door cap. **No diff is published** — a truncated or partial patch applies as silent damage. The fix is a smaller leg, not an operator. | -| `agent loop: canceled (the parent context ended — the caller gave up, or this box is draining)` | `budget` | 2026-09-17 (register S-22/W-16). The PARENT context was canceled mid-LOOP — the mirror of the re-pack's own arm below, added to the loop branch which fell through to the generic `agent loop: …` infrastructure shape instead: the failed request looks exactly like a dial refusal (a `*url.Error`), so an operator used to be told to fix a box that never misbehaved (12 `agent loop: context canceled` rows filed as broken hardware). Nothing on the box failed. **The reason names no single cause, on purpose (PR #366 review):** on the FLEET NODE path the only thing that ever cancels this context is `fleetnode.Jobs.DrainAndStop` on a node drain — never a caller abandoning a poll — and that path's own mark (`state=error, err="interrupted"`) is written BEFORE the cancel that releases the run; `finish()` is write-once against an already-terminal job, so **this defer's words never reach a delegator polling a drained node at all** — the delegator reads `interrupted`, not `budget` (proven in `internal/fleetnode/jobs_test.go`, `TestJobsDrainDiscardsALateAgentBudgetDefer`). Only the LOCAL in-process path (`RunAgentContract`, no `Jobs` store in front of it) can ever surface this string to a human, and there a cancel genuinely is the caller's own context ending — hence the wording names both possibilities rather than asserting the one that is false on the fleet path. | +| `agent loop: canceled (the parent context ended — the caller gave up, or this box is draining)` | `budget` | **This column is the wire defer class. Since 2026-11-12 the ledger row and the failure register file this sentence under `canceled`** (`ledger.RefileStamped`; rows written earlier are refiled when read). 2026-09-17 (register S-22/W-16). The PARENT context was canceled mid-LOOP — the mirror of the re-pack's own arm below, added to the loop branch which fell through to the generic `agent loop: …` infrastructure shape instead: the failed request looks exactly like a dial refusal (a `*url.Error`), so an operator used to be told to fix a box that never misbehaved (12 `agent loop: context canceled` rows filed as broken hardware). Nothing on the box failed. **The reason names no single cause, on purpose (PR #366 review):** on the FLEET NODE path the only thing that ever cancels this context is `fleetnode.Jobs.DrainAndStop` on a node drain — never a caller abandoning a poll — and that path's own mark (`state=error, err="interrupted"`) is written BEFORE the cancel that releases the run; `finish()` is write-once against an already-terminal job, so **this defer's words never reach a delegator polling a drained node at all** — the delegator reads `interrupted`, not `budget` (proven in `internal/fleetnode/jobs_test.go`, `TestJobsDrainDiscardsALateAgentBudgetDefer`). Only the LOCAL in-process path (`RunAgentContract`, no `Jobs` store in front of it) can ever surface this string to a human, and there a cancel genuinely is the caller's own context ending — hence the wording names both possibilities rather than asserting the one that is false on the fleet path. | | `canceled during the structured re-pack (the caller's context ended)` | `budget` | The PARENT context was canceled mid-re-pack (the delegator abandoned the poll, the node is shutting down). It arrives as a `*url.Error` exactly like a dial refusal, so it used to read as broken infrastructure — but nothing on the box failed. | More shapes originate on the **delegator**, not the node: diff --git a/internal/delegate/run_wait.go b/internal/delegate/run_wait.go index e6a36e6a2..a4b3f0b7a 100644 --- a/internal/delegate/run_wait.go +++ b/internal/delegate/run_wait.go @@ -546,6 +546,9 @@ func budgetSpent(remaining, budget, placements int) string { // human's timing decision, not a rewritten contract), the holder named so the // caller can wait, route elsewhere, or ask. It never runs the contract. // +// The wire class stays infrastructure (exit code and summary counts depend on it). The ledger row is a +// capacity hold and is filed under node_busy (ledger.RefileStamped), which is what the failure register counts. +// // call is non-nil when the call's deadline bounded the wait (ADR 0073): the advice to set // agent_lease_wait_sec longer would then be wrong, because the wait already ran as long as the // call could spare. From 0daf6e7060d4678ac5fa51e363c54115dc7deedc Mon Sep 17 00:00:00 2001 From: dmmdea Date: Thu, 12 Nov 2026 10:52:59 -0500 Subject: [PATCH 5/5] Keep a lease hold in the fault class when its remote fallback names a fault The one review of this branch found that the lease-hold refile read only the opening sentence, so a held local seat could file a dead or misconfigured fleet under a capacity class. A row whose reason also says a remote failed its health probe, no remote is configured, none passed the capability gate, or one advertises no context ceiling now stays infrastructure. Real ledger, 30 days, read only: 2,566 failure rows before and after; infrastructure 195 to 140 and node_busy 628 to 704 (55 lease holds move, 17 that also name a failed health probe stay). Co-Authored-By: Claude Fable 5.2 --- .../feat-rsi-register-followups-2.md | 3 ++- docs/systems/failure-register.md | 4 ++-- internal/ledger/reasoncode.go | 24 ++++++++++++++++++- internal/ledger/rootcause.go | 2 +- 4 files changed, 28 insertions(+), 5 deletions(-) diff --git a/changes/unreleased/feat-rsi-register-followups-2.md b/changes/unreleased/feat-rsi-register-followups-2.md index 4c461e716..e63557d6f 100644 --- a/changes/unreleased/feat-rsi-register-followups-2.md +++ b/changes/unreleased/feat-rsi-register-followups-2.md @@ -3,5 +3,6 @@ - **What was wrong.** Three writers filed real failure rows under a class that does not describe them, and two sentences sat in `other`. A delegation to a local seat held by a text lease (`local seat reserved (gpu lease ...)`) was `infrastructure`, though it is a capacity hold and not a fault. The agent loop's `agent loop: canceled (the parent context ended ...)` was `budget`, though the closed set has `canceled`. A node's own folder failure (`agent contract: creating pipeline-jobs dir: ...`, which reaches the delegator as an ack-time 400) was `dispatch_refused`, a class defined as a fact about the request or the credentials, though the fault is the node's configuration. In `other`, a node's `a text lease holds its card; it may take the call later` is a capacity signal, and the browse door's own typed errors name classes. - **Where each now goes.** The lease hold is `node_busy` (the class defined as the seat or card being held by other work; `capacity_wait` is the delegator finding no node with room, a different cause with a different fix). The loop cancellation is `canceled`. The folder fault is `config` (the node's configured home does not exist on that machine; `infrastructure` is for a part of the stack that failed while configured right). The node's lease sentence, matched whole, is `node_busy`; the browse door's `BROWSER_UNAVAILABLE` and `DECISION_UNAVAILABLE` are `infrastructure`, `HOST_NOT_ALLOWED` is `contract` and `budget` is `budget`. The browse door's `blocked` ties to no one class and stays in `other`. - **How.** One function, `RefileStamped` in `internal/ledger/reasoncode.go`, holds the three misfilings, each needing both the old stamp and the writer's own sentence. `Record` applies it when it settles a row's code, so new rows are stamped right; the register applies it when it reads, so rows already in the ledger are counted right (they count in `class_derived_rows`; the ledger file is not rewritten). The five other sentences are rules in `DeferCode`. The wire defer class of the lease-hold defer is unchanged, so the exit code and the run summary still count it as infrastructure. No class is renamed and the JSON shape of `ledger --failures` is unchanged. +- **One exception, on purpose.** A lease hold whose remote fallback failed on a fault (every remote failed its health probe, no remote is configured, none passed the capability gate, or one advertises no context ceiling) stays `infrastructure`: a held local seat must not hide a dead or misconfigured fleet. A row that names a fault keeps the fault class even when other remotes were only full. On the real ledger, 30 days: of the 72 lease-hold rows the report counts, 55 name only remotes without room and move to `node_busy`; 17 also say another remote failed its health probe and stay `infrastructure`. - **Docs.** The root-cause entries in `internal/ledger/rootcause.go` and the table on the failure-register page say what was fixed and that older rows are corrected at read time. -- **Check.** The built binary and the installed one, both read-only on the real ledger, `ledger --failures --since 30`: 2,566 failure rows of 8,784 job rows in both, no row added or lost. `infrastructure` 195 to 123, `budget` 250 to 236, `dispatch_refused` 145 to 115, `other` 55 to 30, and `node_busy` 628 to 721, `canceled` 29 to 46, `config` 2 to 32, `contract` 4 to 5. Every other class is unchanged, and the stack-fault share falls from 0.1142 to 0.0978 because the lease holds were counted as stack faults before. +- **Check.** The built binary and the installed one, both read-only on the real ledger, `ledger --failures --since 30`: 2,566 failure rows of 8,784 job rows in both, no row added or lost. `infrastructure` 195 to 140, `budget` 250 to 236, `dispatch_refused` 145 to 115, `other` 55 to 30, and `node_busy` 628 to 704, `canceled` 29 to 46, `config` 2 to 32, `contract` 4 to 5. Every other class is unchanged, and the stack-fault share falls from 0.1142 to 0.1044 because the lease holds were counted as stack faults before. diff --git a/docs/systems/failure-register.md b/docs/systems/failure-register.md index ba3f84549..bda05fb48 100644 --- a/docs/systems/failure-register.md +++ b/docs/systems/failure-register.md @@ -142,7 +142,7 @@ not investigated. | `contract` | caller contract | The caller named something the node does not serve: an image family the node does not carry. | config/families.go:730; 4 rows in 30 days, all one family name on generate_image | the caller names a served family, or the node adds that family. | | `config` | node configuration | A seat or node was set up so that it cannot serve the call: a seat's tool-call parser does not match its chat template, so a tool call came back as text (2 rows), or a node's configured data folder does not exist on that machine, so it could not create its pipeline-jobs folder for a dispatch (30 rows, one node; filed `dispatch_refused` until 2026-11-12). | pipeline/agenttask.go:1303 and fleetnode/tasks.go:1031 (read by ledger RefileStamped); 2 rows in 30 days as stamped, 32 once the folder fault is read here | set that seat's --tool-call-parser to the one its chat template needs; fix or remove the node's missing data folder (its `home`). | | `write` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | pipeline/agenttask.go (the write door refused: agent_allow_write false, or a write set past a door cap); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | -| `infrastructure` | upstream service | Mostly a seat whose engine was gone from llama-swap's /running (the node's defer class infrastructure carrying the same sentence as seat_down), then a vision seat that was cold and timed out, and the browse door's `browse: BROWSER_UNAVAILABLE:` and `browse: DECISION_UNAVAILABLE:`. The delegator's `local seat reserved (gpu lease ...)` was filed here until 2026-11-12; it is a capacity hold and is now `node_busy` (the wire class of that defer stays infrastructure, so the exit code and summary still count it). | agent/seatdown.go:185, pipeline_vision.go; 195 rows in 30 days as stamped: research 90, fleet 82; 79 of them were `local seat reserved (gpu lease ...)`, which ledger RefileStamped now reads as node_busy | see seat_down; not investigated: why the engine goes away. | +| `infrastructure` | upstream service | Mostly a seat whose engine was gone from llama-swap's /running (the node's defer class infrastructure carrying the same sentence as seat_down), then a vision seat that was cold and timed out, and the browse door's `browse: BROWSER_UNAVAILABLE:` and `browse: DECISION_UNAVAILABLE:`. The delegator's `local seat reserved (gpu lease ...)` was filed here until 2026-11-12; it is a capacity hold and is now `node_busy` (the wire class of that defer stays infrastructure, so the exit code and summary still count it). | agent/seatdown.go:185, pipeline_vision.go; 195 rows in 30 days as stamped: research 90, fleet 82; 72 of them were `local seat reserved (gpu lease ...)`: ledger RefileStamped now reads the 55 that name only remotes without room as node_busy, and the 17 that also name a failed health probe stay here | see seat_down; not investigated: why the engine goes away. | | `bad_input` | caller contract | The input cannot be used as given: an image path that does not exist, a template variable the template does not declare. | pipeline (image load) and compose_video BAD_INPUT; 2 rows in 30 days | none: working as designed. The caller fixes the input. | | `other` | unknown | not investigated: the class is the residue no rule names, a mix of causes. Since 2026-11-12 a node's `a text lease holds its card; it may take the call later` is read as node_busy, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors by their own classes; what is left is mostly the browse door's `blocked` (the model reported it cannot proceed, 4 rows) and media and compose sentences no rule names | reasoncode.go DeferCode (the residue); 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the rules above are read | not investigated: list the distinct reasons of the rows left and add a rule per sentence that ties to one class | @@ -267,7 +267,7 @@ The table is the human-maintained half of the register. A row with a stamped reason code is read as stamped, with two exceptions. `other` is what the ledger files when no member named the row at write time, so a deferred row stamped `other` is classed again and keeps `other` unless a rule below now names a member. And three stamps were wrong until 2026-11-12 (`RefileStamped`, applied by `Record` to new rows and by the -register to old ones, and counted in `class_derived_rows` too): `infrastructure` on a reason holding `local seat reserved (` +register to old ones, and counted in `class_derived_rows` too): `infrastructure` on a reason holding `local seat reserved (` (unless the same reason says the remote fallback failed on a fault: a failed health probe, no remote configured, none past the capability gate, or no advertised context ceiling; that row stays `infrastructure`) is `node_busy`; `budget` on a reason starting `agent loop: canceled (the parent context ended` is `canceled`; and `dispatch_refused` on a reason holding `agent contract: creating pipeline-jobs dir` is `config`. Each needs the wrong stamp and the writer's own sentence, so no other row of those classes moves. A deferred row without a code is classed the same way. Both are counted in `class_derived_rows` and classed diff --git a/internal/ledger/reasoncode.go b/internal/ledger/reasoncode.go index 829d5ab44..b558b9f19 100644 --- a/internal/ledger/reasoncode.go +++ b/internal/ledger/reasoncode.go @@ -306,7 +306,7 @@ const ( func RefileStamped(code, reason string) string { switch code { case ReasonInfrastructure: - if strings.Contains(reason, localSeatReservedSentence) { + if strings.Contains(reason, localSeatReservedSentence) && !remoteFallbackFaulted(reason) { return ReasonNodeBusy } case ReasonBudget: @@ -321,6 +321,28 @@ func RefileStamped(code, reason string) string { return code } +// remoteFaultSentences are the delegator's own words for a remote fallback that failed because the fleet is broken +// or misconfigured (delegate/run_probe.go nodeSideVerdict and noEligibleRemote), as opposed to full or leased. +var remoteFaultSentences = []string{ + "failed the health probe", + "no remote fleet nodes are configured", + "no remote passed the capability gate", + "advertise no context ceiling", +} + +// remoteFallbackFaulted reports a lease hold whose remote fallback failed on a fault, not on room. Such a row stays +// infrastructure: the local seat being held must never hide a dead or misconfigured fleet behind a capacity class +// (2026-11-12 review). A row that names a fault keeps the fault class even when other remotes were only full: 17 of +// the 72 lease-hold rows the register then counted also named a failed health probe, and stay infrastructure. +func remoteFallbackFaulted(reason string) bool { + for _, s := range remoteFaultSentences { + if strings.Contains(reason, s) { + return true + } + } + return false +} + // stallCodeOf names the phase a node's liveness verdict ("stalled: no progress for 300s in prefill (...)") went // silent in, by the phase words of the run registry, as the delegator reads them (delegate.stallReasonCode). func stallCodeOf(reason string) string { diff --git a/internal/ledger/rootcause.go b/internal/ledger/rootcause.go index fa2fa2114..4dc5e56e3 100644 --- a/internal/ledger/rootcause.go +++ b/internal/ledger/rootcause.go @@ -162,7 +162,7 @@ var failureCauses = map[string]FailureCause{ ReasonWrite: noRow("pipeline/agenttask.go (the write door refused: agent_allow_write false, or a write set past a door cap)"), ReasonInfrastructure: {Kind: CauseUpstream, Investigated: true, RootCause: "Mostly a seat whose engine was gone from llama-swap's /running (the node's defer class infrastructure carrying the same sentence as seat_down), then a vision seat that was cold and timed out, and the browse door's `browse: BROWSER_UNAVAILABLE:` and `browse: DECISION_UNAVAILABLE:`. The delegator's `local seat reserved (gpu lease ...)` was filed here until 2026-11-12; it is a capacity hold and is now `node_busy` (the wire class of that defer stays infrastructure, so the exit code and summary still count it).", - Evidence: "agent/seatdown.go:185, pipeline_vision.go; 195 rows in 30 days as stamped: research 90, fleet 82; 79 of them were `local seat reserved (gpu lease ...)`, which ledger RefileStamped now reads as node_busy", + Evidence: "agent/seatdown.go:185, pipeline_vision.go; 195 rows in 30 days as stamped: research 90, fleet 82; 72 of them were `local seat reserved (gpu lease ...)`: ledger RefileStamped now reads the 55 that name only remotes without room as node_busy, and the 17 that also name a failed health probe stay here", Action: "see seat_down; not investigated: why the engine goes away."}, ReasonBadInput: {Kind: CauseCallerContract, Investigated: true, RootCause: "The input cannot be used as given: an image path that does not exist, a template variable the template does not declare.",