From 07f1629f903fc1aa76fa98524aa5a8a0a98afb19 Mon Sep 17 00:00:00 2001 From: dmmdea Date: Thu, 12 Nov 2026 12:29:25 -0500 Subject: [PATCH 1/2] Read the misfiled sentences and error classes into their classes, and record the causes of the four uninvestigated ones The failure register still held 30 rows in other and 3 in poll_deadline after the last release. Each is a sentence or error class one harness writer owns, and each now goes to the class that names it: node_busy (a media call naming a node that did not answer), node_unreachable (a dispatch that could not connect, repeated poll failures, a fetch that dropped, a poll deadline that ends on a last poll error), config (no_node_configured, audio QA without ffmpeg), bad_input (media convert without in or out, lint_errors), abstention (low classify confidence) and failed_verification (compose check_failed). Each rule needs the writer's own words, and the two that wrap other text (media convert, reasoning tier) are matched as whole sentences. The browse "blocked" rows stay in other: their cause is not settled. rootcause.go now holds the findings for failed_verification, poll_deadline, stall_cold_load and other, with their counts and what each rests on; the part not settled is stated in the text. The pin test over core.ErrClass* omitted no_node_configured, so it could not have caught the missing table entry; it lists it now. The page and the fixture golden follow the catalogue. Co-Authored-By: Claude Sonnet 5.5 --- .../fix-register-rules-and-causes.md | 10 +++ docs/systems/failure-register.md | 53 ++++++++++------ internal/ledger/reasoncode.go | 63 ++++++++++++++++++- internal/ledger/rootcause.go | 52 +++++++-------- internal/pipeline/transcribe_noaudio_test.go | 2 +- testdata/failure-register/expected.json | 47 +++++++------- 6 files changed, 154 insertions(+), 73 deletions(-) create mode 100644 changes/unreleased/fix-register-rules-and-causes.md diff --git a/changes/unreleased/fix-register-rules-and-causes.md b/changes/unreleased/fix-register-rules-and-causes.md new file mode 100644 index 000000000..29ee6329f --- /dev/null +++ b/changes/unreleased/fix-register-rules-and-causes.md @@ -0,0 +1,10 @@ +### Fixed - the failure register files 27 more rows under the class that names them, and the four classes it had not investigated now carry their causes + +- **What was wrong.** After the last release 30 failure rows still sat in `other` and 3 in `poll_deadline`, and four classes read "not investigated" (`failed_verification` 357 rows, `other`, `poll_deadline` 3, `stall_cold_load` 4). The rows in `other` were lane refusals and transport faults whose sentences or error classes no rule named. The 3 `poll_deadline` rows were not that class: each ends on a last poll error, so the node had stopped answering, not kept working past the allowance. One audio row was filed `failed_verification` though the audio was never judged (ffmpeg or ffprobe was missing). +- **Where each row now goes.** `no node that answered is called ...` (8 rows) is `node_busy`, the class the lane's own defer already files for new rows through its wire class. A dispatch whose connection could not be made (1), `N consecutive poll failures` (1), `rendered the job but fetching it failed` (1) and the 3 poll-deadline rows with a last poll error are `node_unreachable`. Error class `no_node_configured` (2) and the audio QA's missing-ffmpeg dead air (1) are `config`. `media convert: convert: in and out are required` (1) and error class `lint_errors` (1) are `bad_input`. `reasoning tier: low classify confidence ...` (3) is `abstention`. Compose's `check_failed` (5) is `failed_verification`. +- **Two readings that are a choice.** `rendered the job but fetching it failed` is `node_unreachable` (the fetch dropped), with `infrastructure` the other reading because the render itself succeeded. Compose `check_failed` is `failed_verification` (the lane's own checks of the composition failed), with `bad_input` the other reading because the caller wrote the composition. Both are one-line rule changes. +- **Left alone on purpose.** The browse door's `blocked` (4 rows) is proposed as `abstention` but its cause is not settled, so it has no rule; the 3 `generate_svg` rows with no reason, no reason code and no door cannot be traced. `other` is 7 rows. +- **How.** Each rule needs the writer's own fixed words. `media convert: convert: in and out are required` and `reasoning tier: low classify confidence ` are matched as whole sentences, not as the bare prefixes `media convert:` and `reasoning tier: `, because `media : ` also wraps what ffmpeg reported and `reasoning tier: ` wraps other verifier wordings. Three new error classes are in `errClassCodes`; the dispatch, poll-failure and fetch sentences are in `legacyReasonCode`; the poll-deadline and audio rules are in `RefileStamped` and each needs both the old stamp and the writer's sentence, so no other row of those classes moves. The pin test `TestEveryCoreErrClassHasALedgerReasonCode` listed every error class but `no_node_configured`; it does now. +- **Causes.** `internal/ledger/rootcause.go` now holds the findings, with their counts and the day measured, and says where the evidence stops. `failed_verification` is a harness defect for the research lane's own checks: of 357 rows, 307 are those checks rejecting answers that were right or honest, of which 202 are established by comparing the digest to the page and 105 rest on the seat's own statement; 10 are seat misses, 4 a caller regex over-escaped, 1 a misfile, 35 undetermined. `poll_deadline` is an upstream node that stopped answering (why is not settled). `stall_cold_load` is one placement and filing defect on 2026-11-04: the placer put a long-context subtask on a seat that shares the pair's cards with the pool seat, and the monitor filed the wait behind the swap as a cold-load stall. `other` is a bundle of misfiled sentences, not one cause. The follow-ups it names (stop deriving the `min_items` check, count the call's own siblings in `pairLong`, record the answer in the ledger, stop stamping a poll deadline with a last poll error as `poll_deadline` at the writer) are not made here. +- **Docs.** The catalogue copy and the rule lists on `docs/systems/failure-register.md` match the code; `causes_not_investigated` is empty in the window. +- **Check.** The built binary and the installed one, both read-only on the real ledger, `ledger --failures --since 30`: 2,566 failure rows of 8,785 job rows in both, the sum over classes unchanged. `other` 30 to 7, `poll_deadline` 3 to 0, `node_busy` 704 to 712, `node_unreachable` 3 to 9, `failed_verification` 357 to 361, `abstention` 221 to 224, `config` 32 to 35, `bad_input` 2 to 4; every other class is unchanged. `causes_not_investigated` went from `failed_verification`, `other`, `poll_deadline`, `stall_cold_load` to none. Two further rows of the `no_node_configured` class are measurement runs the register leaves out. diff --git a/docs/systems/failure-register.md b/docs/systems/failure-register.md index bda05fb48..3405e954e 100644 --- a/docs/systems/failure-register.md +++ b/docs/systems/failure-register.md @@ -106,31 +106,31 @@ How the table was made (2026-11-12): the code that emits each class was read, th 2026-11-12 in the real ledger, and the whole ledger for a class with no row in the window. An entry is `investigated` only when the emitter and those rows together settle the cause it states. Where they do not, the cause reads `not investigated:` and says what would settle it. The counts are that snapshot; the live count of a class is its `count`. -A class with no entry reads as not investigated. **15 of the 31 classes are not investigated**: 8 have rows that do not -settle the cause (`failed_verification`, `poll_deadline`, `stall_admission`, `stall_cold_load`, `stall_prefill`, -`stall_decode`, `stall_repack`, `other`) and 7 have no row in the whole ledger (`job_lost`, `remote_error`, `shed`, -`stall_tool`, `stall_engine`, `stall_other`, `write`). In the 30 days to 2026-11-12, 4 of the 19 classes present were -not investigated. +A class with no entry reads as not investigated. **11 of the 31 classes are not investigated**: 4 have rows that do not +settle the cause (`stall_admission`, `stall_prefill`, `stall_decode`, `stall_repack`) and 7 have no row in the whole ledger (`job_lost`, `remote_error`, `shed`, +`stall_tool`, `stall_engine`, `stall_other`, `write`). In the 30 days to 2026-11-12, none of the 18 classes present was +not investigated. `failed_verification` and `other` are marked investigated because their dominant cause is settled; the text of each says exactly which +rows it rests on and which it does not (202 of 357 `failed_verification` rows compared with the page, 105 resting on the seat's own statement; 7 of the 30 `other` rows left without a rule). | failure class | cause kind | root cause | evidence | action | |---|---|---|---|---| -| `failed_verification` | unknown | not investigated: the rows say which check failed (`min_items` on plans, quotes or numbers; a regex) but not whether the seat's answer was wrong or the check too strict; the acceptance list is shape-only (delegate/lint.go warns of it) | delegate/run.go (failed verification); 357 rows in 30 days, 345 of them from research subtasks | not investigated: read the saved output of a sample of failed research subtasks against their acceptance lists | +| `failed_verification` | harness defect | Mostly the research lane's own acceptance checks rejecting answers that were right or honest, not seats failing. Of the 357 rows (346 from research subtasks, whose checks the lane writes itself), 307 are those checks rejecting an answer that was right or honest: 256 fail a page-word tripwire that is graded against the loop's final prose and not against the structured digest, and 51 fail a derived `min_items` check on the first required array. Not every one of the 307 is shown to be a false rejection. 202 rows are established by comparing the digest to the page: 188 tripwire rows whose quotes and figures are found verbatim in the page, and 14 `min_items` rows on pages that were empty shells, login walls or placeholders. 105 rest on the seat's own statement that the page lacks the content (68 tripwire, 37 `min_items`) and were not checked against the page. The other 50 are 10 seat misses (it could not read or complete the page, found 1 of 3 items on a long report, returned an empty digest, did not echo an id, or left a status empty), 4 caller regexes that were over-escaped, 1 misfile (an audio row, see config) and 35 undetermined, because the ledger does not keep the answer or the kind of check. The class also reads 5 compose rows (error class `check_failed`: a composition's own runtime or layout checks failed, from one caller iterating on one composition; filed `other` before), with `bad_input` as the other reading since the composition is what the caller wrote. | research/anchors.go AnchorCheck, graded by core.AcceptanceCheck.Eval and evalText; delegate/run.go (failed verification); 357 rows in 30 days as stamped, 361 once the compose `check_failed` rows are read here (5 added, the audio row left). By door: research 345 and the research CLI 1, agent_delegate 8, delegate CLI 2, audio CLI 1; by seat: a 9B-class seat on three nodes 211, a 35B seat 82, the agent-pool seat 58, a 27B seat 5; 264 fall on five days (2026-10-19, -20, -23, -24, -25). Check shapes, counted apart from the causes above (the 282 tripwire rows include 26 that are seat misses or undetermined, and the tripwire and `min_items` shapes share 6 rows): docanchor regex 282, derived `min_items::1` with 0 items 70, `min_items` with 1 item against a wanted 3 on agent_delegate 2, a caller regex naming a drive path 4, `contains:` 2, a caller regex of three alternatives 1, `nonempty:status` 1, audio dead_air 1. Why the tripwire is the suspect: none of the 282 anchor sets appears in the graded prose and 5 of 282 appear in the structured digest; the 13 digests with the lowest page overlap were read and none was about another document; widening the honest-absence words (delegate/absence.go HonestAbsence) rescues 10 of 282; the failure rate among research pages that reached a verdict is 7.0% (345 against 4,604 passed) and does not step down across delegator builds (3 to 14%). Of the tripwire first attempts retried on another node, 17 ended ok, 11 failed again and 2 ended in budget. Measured 2026-11-12 by rebuilding the register's class rule outside the binary and joining the rows to the delegation log | stop deriving a pass/fail `min_items::1` for research subtasks (excuse it when the digest says the page has none, as the existing empty-digest excuse does); do not dispatch near-empty pages (a minimum-prose floor in research.Build); grade the tripwire against the digest too, passing when its quotes, figures or titles are found verbatim in the page (research/quotes.go VerifyQuotes exists), which would have passed at least 188 of the 282; and record the answer and the acceptance kind in the ledger so the 35 undetermined rows can be settled | | `queue_full` | capacity signal | Every node asked refused the dispatch for room: its queue was at fleet_max_queue_depth (the sampled bodies read `queue full (N jobs accepted+running, limit N)`) or its card was leased. The delegator files every 503 or 429 so. | delegate/run_capacity.go capacityRefusal; 49 rows in 30 days (of the three commonest reason groups, 22 rows read queue full and 1 node leased) | none: working as designed. A node with spare capacity can raise fleet_max_queue_depth. | | `queue_deadline` | capacity signal | A node accepted the job and its backlog never reached it inside the queue patience, and the node did not confirm taking it back. Jobs queue behind other work on a full node. | delegate/run_remote.go (queue deadline after); 0 rows in 30 days, 318 in the whole ledger, all written before the withdrawal stamp existed | none: working as designed (the delegator withdraws and re-places since ADR 0064; queue_withdrawn is the same event confirmed). | | `queue_withdrawn` | capacity signal | The same event as queue_deadline, and the node confirmed taking the job back: it sat in the node's backlog (`accepted`, never `running`) past the queue patience and was withdrawn. | delegate/run_remote.go; 186 rows in 30 days, 182 from research subtasks, 111 on one node | none: working as designed. If one node keeps most of them (111 of 186), check its concurrency against the headroom it advertises. | -| `poll_deadline` | unknown | not investigated: the class does not separate a job longer than the poll budget from a node that stopped making progress (failurekinds.go says so), and the 3 rows differ (a poll error, a stale last-progress, the node's own wall) | delegate/run_remote.go (poll deadline after); 3 rows in 30 days | not investigated: for each row, compare the node's own record of the job with the delegator's poll budget | +| `poll_deadline` | upstream service | The class is for a node that answered and kept working past the allowance, and the 3 rows of the 30 days as stamped were not that: each is a research job that ran 681 to 978 s and ends `last poll error ... context deadline exceeded`, so the node stopped answering the delegator's 15 s poll requests while it owned the job. Two also show stale progress (128 s against a 60 s allowance, 922 s against a 79 s allowance) while other nodes completed normally in the same delegator session. They are filed `node_unreachable` here (ledger RefileStamped reads a poll deadline that carries a last poll error so), which leaves 0 in the window. Why the node went silent (asleep, off, network, hung) is not settled: no node-side log was available. | delegate/run_remote.go runRemote (the poll loop and its 15 s poll request timeout) and delegate/reasoncode.go deferReasonCode (`poll deadline` is filed poll_deadline whatever follows it); 3 rows in 30 days as stamped, 0 once the last poll error is read as node_unreachable. One is the same incident as a node_unreachable row and carries `withdraw not confirmed: no answer within 5s`; one ran 877 s on a node whose normal p99 is 281 s and maximum 740 s; the third node's own health probes also timed out three times that day. The 20 older rows (2026-09-16 to -29) are pure budget shapes with no last poll error, and none has occurred since; they stay poll_deadline | none for the filing (RefileStamped). For the node: `local-offload doctor` shows which nodes answer now; not investigated: why that node stopped answering. Not changed here: delegate/reasoncode.go deferReasonCode still stamps a poll deadline with a last poll error as poll_deadline when it writes the row (Record refiles it on the way to disk) | | `canceled` | caller contract | The caller's context ended (the caller went away, or the call's own deadline passed) before or while a subtask ran. It also holds the node's `agent loop: canceled (the parent context ended ...)`, which the wire files as a budget defer because the wire has no canceled class; the ledger had filed it `budget` until 2026-11-12. | delegate/calldeadline.go and ledger RefileStamped (pipeline/agenttask.go:1295); 29 rows in 30 days as stamped: `canceled: context canceled` and `the call's context ended before this subtask started`, and 17 more once the loop cancellations are read here | none: working as designed. | -| `node_unreachable` | upstream service | A node host did not answer on the network: a dial that timed out (`the connected party did not properly respond`) or a poll that never got an answer. Why the host did not answer (asleep, off, network) is not recorded on the row. | delegate/run_remote.go; 3 rows in 30 days, 2 on one node | none from the harness: the node host or path is the fault; `local-offload doctor` shows which nodes answer now. | +| `node_unreachable` | upstream service | A node host did not answer on the network: a dial that timed out (`the connected party did not properly respond`) or a poll that never got an answer. It also holds a dispatch whose connection could not be made (`dial tcp`), a job whose polls were not answered (`N consecutive poll failures`), a node that rendered a job and whose output then could not be fetched (`rendered the job but fetching it failed`, a dropped connection; the render itself succeeded, so infrastructure is the other reading), and the 3 poll-deadline rows that ended on a last poll error (poll_deadline until 2026-11-12). Why the host did not answer (asleep, off, network) is not recorded on the row. | delegate/run_remote.go; 3 rows in 30 days as stamped, 2 on one node, and 6 more once the rules above are read here (3 from other, 3 from poll_deadline) | none from the harness: the node host or path is the fault; `local-offload doctor` shows which nodes answer now. | | `job_lost` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | delegate/run_remote.go (a poll 404 after an acknowledged dispatch); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | | `dispatch_refused` | node configuration | A node refused the dispatch itself: it rejected the delegator's bearer token (status 401 `unauthorized`, one node: a fleet_auth_token mismatch). Until 2026-11-12 the class also held a node whose own data folder could not be created (`creating pipeline-jobs dir`, an ack-time 400); that is a fault of the node's configuration and not of the request, and it is now filed `config`. | delegate/calldeadline.go (401 unauthorized); 145 rows in 30 days as stamped, 145 on one node's seat: 115 are the 401 and 30 the folder fault, which ledger RefileStamped now reads as config | re-pair that node's token (`local-offload doctor` checks fleet tokens); an edit on the node. | | `remote_error` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | delegate/run_remote.go (a node's job reported errored, or its answer was unusable); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | | `capacity_wait` | capacity signal | No node had room for the subtask before the call's deadline: 248 of 267 waited until the call's own deadline (not agent_placement_wait_sec), 259 for one node's agent pool alias. | delegate/run_wait.go:436 and :442; 267 rows in 30 days, 259 on one pool alias | none: working as designed. Reducing it needs more worker slots on the pool's node, or fewer research subtasks per call. | -| `node_busy` | capacity signal | The seat or card the call needed was held by other work when it arrived: the seat was at its local run cap (`seat busy:`), a GPU lease of another class held the card (`gpu busy: held by ...`), or a media job waited for host RAM. It also holds the two text-lease holds filed elsewhere until 2026-11-12: the delegator's `local seat reserved (gpu lease ...)` (a text lease holds the only seat the call may use and no remote had room; filed `infrastructure`) and a node's `a text lease holds its card; it may take the call later` (filed `other`). The harness defers rather than overload the card. | pipeline/agenttask.go (seat busy, defer class capacity), pipeline/pipeline_gpu.go (gpu busy), ledger RefileStamped and DeferCode (the lease holds); 628 rows in 30 days as stamped: video 247, research 205, audio 109 (the agent pool at its cap, a media lease on the cards, host RAM), and 93 more once the lease holds are read here (79 from infrastructure, 14 from other) | none: working as designed. Fewer deferrals need more free card time (keep a card free for media while text leases run) or a higher run cap on a box with room. | +| `node_busy` | capacity signal | The seat or card the call needed was held by other work when it arrived: the seat was at its local run cap (`seat busy:`), a GPU lease of another class held the card (`gpu busy: held by ...`), or a media job waited for host RAM. It also holds the two text-lease holds filed elsewhere until 2026-11-12: the delegator's `local seat reserved (gpu lease ...)` (a text lease holds the only seat the call may use and no remote had room; filed `infrastructure`) and a node's `a text lease holds its card; it may take the call later` (filed `other`). A media call that named a node that did not answer the health probe (`no node that answered is called ...`, the node's own capacity defer; 8 rows, filed `other` before) is read here too. The harness defers rather than overload the card. | pipeline/agenttask.go (seat busy, defer class capacity), pipeline/pipeline_gpu.go (gpu busy), ledger RefileStamped and DeferCode (the lease holds); 628 rows in 30 days as stamped: video 247, research 205, audio 109 (the agent pool at its cap, a media lease on the cards, host RAM), 93 more once the lease holds are read here (79 from infrastructure, 14 from other), and 8 more once `no node that answered is called ...` is read (video 5, audio 3) | none: working as designed. Fewer deferrals need more free card time (keep a card free for media while text leases run) or a higher run cap on a box with room. | | `shed` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | delegate (sheddable work, priority -1, found no idle node); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | | `no_eligible_node` | upstream service | A lane found no fleet node able to take the call. In the sampled rows each node probed failed its health check with a timeout (`context deadline exceeded`), so the nodes were not answering, not refusing for room. Why they did not answer is not recorded. | the remote lanes' placement sentences (forecastremote and the others) and delegate route=remote; 76 rows in 30 days, 48 of them the vision lane | none from the harness: bring the nodes back (`local-offload doctor`); not investigated: why the probes timed out at the failure times. | | `seat_down` | upstream service | The inference engine behind the seat was gone from llama-swap's /running, or refused connections, when the run needed it, and did not return inside the node's recovery wait. What stopped the engine is not recorded. | agent/seatdown.go:185; 89 rows in 30 days, 87 on one node's pool alias | not investigated: why that node's engine goes away; read its llama-swap log at the failure times. The harness already waits and re-issues (ADR 0066). | | `stall_admission` | unknown | not investigated: the monitor reports a run silent in the admission phase; why is not on the row | agent/liveness.go; 0 rows in 30 days, 1 in the whole ledger | not investigated: read the node's engine log at the time of the row | -| `stall_cold_load` | unknown | not investigated: the rows say no byte came for 120 s after the seat read ready, on its first completion since the load; whether the bound is too tight for a cold seat or the engine was wedged is not settled | agent/liveness.go:268; 4 rows in 30 days, all on one node's pool alias | not investigated: time that seat's first completion after a load against the bound | +| `stall_cold_load` | harness defect | A placement and filing defect, from one event on 2026-11-04: the 4 rows are two delegations 7 minutes apart on the delegator's own pair of cards. In the same call and the same second the placer put a fourth subtask on the pair's long-context seat, which shares the two cards with the pool seat, so loading one evicts the other. It read the pool seat as not loaded (cold, nothing to evict) and ignored the call's own pending siblings. The displaced subtasks then waited behind the swap (a warm request answered HTTP 500 after 117 s, `mid-swap`; about 145 s of seat-down wait and about 226 s of admission wait; 0 steps) and the liveness monitor filed that as a cold-load stall and not as a capacity hold. It was not a wedge and the 120 s bound was not too tight. Established from the ledger rows and the node's notes; not from an engine log, which was not available. A wedge is excluded only because the long-context seat completed in the same window. | placement/decide.go pairLong (reads occupancy at decision time); agent/liveness.go onStall, fileStallLocked and StallError.Error; seatload/activity.go Summary (`seat not loaded`); 4 rows in 30 days, all on one node's pool alias. The long-context job of that call ran ok (943 s, 7,660 tokens) and that seat appears exactly once in the whole ledger, in that call; one older cold-load stall exists (2026-09-28). Of 87 seat_down rows on the same seat, 85 overlap no other seat's run, so they are a different cause and are not covered here | placement/decide.go pairLong should count the same call's pending siblings on the pair and queue the long subtask behind them, or refuse the co-schedule; the liveness monitor should file a wait behind a known displacement as a capacity hold (node_busy). No sentence rule can move these rows: the sentence is true of what the monitor saw. | | `stall_prefill` | unknown | not investigated: the rows say no progress in prefill past an allowance computed from the prompt size and the seat's rate; a prompt larger than the allowance or a wedged engine both read so | agent/liveness.go; 0 rows in 30 days, 246 in the whole ledger (written before the stamp) | not investigated: compare the prompt size of those rows with the seat's measured prefill rate | | `stall_decode` | unknown | not investigated: no progress while tokens were generated; the rows do not say whether the engine wedged or the allowance was too tight | agent/liveness.go; 0 rows in 30 days, 51 in the whole ledger (written before the stamp) | not investigated: read the engine log at the time of the rows | | `stall_tool` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | agent/liveness.go (silence inside a tool call); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | @@ -138,16 +138,18 @@ not investigated. | `stall_engine` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | agent/liveness.go (the engine did no work, or kept stepping without a token); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | | `stall_other` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | delegate/reasoncode.go stallReasonCode (a stall whose phase the reason does not name); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | | `budget` | caller contract | A limit the call carried ended the run before the agent finished: the call's own deadline (`call deadline reached`, `wall timeout after ... the caller's deadline`), or the step budget (`step budget exhausted`). Most are the caller's deadline, not a node ceiling. The agent loop's `agent loop: canceled` was filed here until 2026-11-12 and is now `canceled`; the browse door's `browse: budget:` (max_actions spent) is read here. | core/calldeadline.go, pipeline/liveness.go:411, pipeline/agenttask.go:1295; 250 rows in 30 days as stamped, 241 from research subtasks; 17 of them are loop cancellations that ledger RefileStamped now reads as canceled | give long research subtasks a longer call deadline or fewer subtasks per call; a run that ends on the step budget needs a larger max_steps. | -| `abstention` | node configuration | The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call. | pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days, 107 on one vision seat's questions | raise the vision lane's output cap for that seat, or ask for shorter answers; not investigated: why those answers run past the cap. | +| `abstention` | node configuration | The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call. The reasoning tier's `reasoning tier: low classify confidence ...` (the model's own confidence in a classify answer under the minimum, 3 rows, filed `other` before) is read here too. | pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days as stamped, 107 on one vision seat's questions, 224 once the classify confidence defers are read (pipeline/pipeline_attempt.go) | raise the vision lane's output cap for that seat, or ask for shorter answers; not investigated: why those answers run past the cap. | | `contract` | caller contract | The caller named something the node does not serve: an image family the node does not carry. | config/families.go:730; 4 rows in 30 days, all one family name on generate_image | the caller names a served family, or the node adds that family. | -| `config` | node configuration | A seat or node was set up so that it cannot serve the call: a seat's tool-call parser does not match its chat template, so a tool call came back as text (2 rows), or a node's configured data folder does not exist on that machine, so it could not create its pipeline-jobs folder for a dispatch (30 rows, one node; filed `dispatch_refused` until 2026-11-12). | pipeline/agenttask.go:1303 and fleetnode/tasks.go:1031 (read by ledger RefileStamped); 2 rows in 30 days as stamped, 32 once the folder fault is read here | set that seat's --tool-call-parser to the one its chat template needs; fix or remove the node's missing data folder (its `home`). | +| `config` | node configuration | A seat or node was set up so that it cannot serve the call: a seat's tool-call parser does not match its chat template, so a tool call came back as text (2 rows), or a node's configured data folder does not exist on that machine, so it could not create its pipeline-jobs folder for a dispatch (30 rows, one node; filed `dispatch_refused` until 2026-11-12), a node that has not opened the media-job door a call with input files needs (error class `no_node_configured`, 2 rows, filed `other`), or the audio QA could not measure the audio because ffmpeg or ffprobe is not installed (1 row, filed `failed_verification`). | pipeline/agenttask.go:1303 and fleetnode/tasks.go:1031 (read by ledger RefileStamped); 2 rows in 30 days as stamped, 32 once the folder fault is read here, 35 once the media-job and audio QA rows are read (render/audio-qa.mjs, ledger RefileStamped and errClassCodes) | set that seat's --tool-call-parser to the one its chat template needs; fix or remove the node's missing data folder (its `home`). | | `write` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | pipeline/agenttask.go (the write door refused: agent_allow_write false, or a write set past a door cap); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | | `infrastructure` | upstream service | Mostly a seat whose engine was gone from llama-swap's /running (the node's defer class infrastructure carrying the same sentence as seat_down), then a vision seat that was cold and timed out, and the browse door's `browse: BROWSER_UNAVAILABLE:` and `browse: DECISION_UNAVAILABLE:`. The delegator's `local seat reserved (gpu lease ...)` was filed here until 2026-11-12; it is a capacity hold and is now `node_busy` (the wire class of that defer stays infrastructure, so the exit code and summary still count it). | agent/seatdown.go:185, pipeline_vision.go; 195 rows in 30 days as stamped: research 90, fleet 82; 72 of them were `local seat reserved (gpu lease ...)`: ledger RefileStamped now reads the 55 that name only remotes without room as node_busy, and the 17 that also name a failed health probe stay here | see seat_down; not investigated: why the engine goes away. | -| `bad_input` | caller contract | The input cannot be used as given: an image path that does not exist, a template variable the template does not declare. | pipeline (image load) and compose_video BAD_INPUT; 2 rows in 30 days | none: working as designed. The caller fixes the input. | -| `other` | unknown | not investigated: the class is the residue no rule names, a mix of causes. Since 2026-11-12 a node's `a text lease holds its card; it may take the call later` is read as node_busy, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors by their own classes; what is left is mostly the browse door's `blocked` (the model reported it cannot proceed, 4 rows) and media and compose sentences no rule names | reasoncode.go DeferCode (the residue); 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the rules above are read | not investigated: list the distinct reasons of the rows left and add a rule per sentence that ties to one class | +| `bad_input` | caller contract | The input cannot be used as given: an image path that does not exist, a template variable the template does not declare, a media call with no in or out (`media convert: convert: in and out are required`), a composition the compose lint refuses (error class `lint_errors`). | pipeline (image load) and compose_video BAD_INPUT and LINT_ERRORS; 2 rows in 30 days as stamped, 4 once the two added rules are read (1 media convert, 1 lint) | none: working as designed. The caller fixes the input. | +| `other` | unknown | The class is the residue no rule names, a bundle of misfiled sentences and not one cause; it stays kind unknown on purpose. Of the 55 rows as stamped, 25 are read by the rules of 2026-11-12 that came first (a node's `a text lease holds its card; it may take the call later`, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors), which leaves 30. 23 of those 30 are lane refusals and transport faults whose sentences or error classes nobody had mapped, and each is now settled by a sentence or error-class rule: `no node that answered is called ...` (8, node_busy), a dispatch whose connection could not be made (1), `N consecutive poll failures` (1) and `rendered the job but fetching it failed` (1), all node_unreachable; error class `no_node_configured` (2, config); `media convert: convert: in and out are required` (1) and error class `lint_errors` (1), bad_input; `reasoning tier: low classify confidence ...` (3, abstention); and the compose error class `check_failed` (5, failed_verification, with bad_input as the other reading, since the composition is what the caller wrote). 7 rows are left: 4 of the browse door's `blocked` (the model or the loop reported no progress, 9 to 18 s), which is proposed as abstention but whose cause is not settled and so has no rule, and 3 generate_svg rows with no reason, no reason code and no door, which cannot be traced. | reasoncode.go DeferCode (the residue) and errClassCodes; 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the first rules are read, 7 once the rules above are read. Two further rows of those error classes are measurement runs, which the register leaves out | read the 4 browse `blocked` rows against the pages they drove before ruling on them; the 3 rows with no reason need the writer to record one. Not changed here: the other verifier wordings behind `reasoning tier: ...` stay in other until a row shows them. | Reading the rows found three misfilings and two sentences left in `other`, all fixed on 2026-11-12 (`RefileStamped` and `DeferCode` in `internal/ledger/reasoncode.go`): a delegate row for a local seat held by a text lease (`local seat reserved (gpu lease ...)`) was filed `infrastructure` and is now `node_busy`, because the class is defined as the seat or card being held by other work (`capacity_wait` is the delegator finding no node with room, a different cause with a different fix); the agent loop's `agent loop: canceled` was filed `budget` and is now `canceled`; and a node's own folder failure (`agent contract: creating pipeline-jobs dir`, an ack-time 400) was filed `dispatch_refused`, a fact about the request or the credentials, and is now `config` (the node's configured home does not exist on that machine; `infrastructure` is for a stack part that failed while configured right). Rows written before the fix are corrected when the register is read: a row stamped with the old class and carrying the writer's own sentence is counted under the new one and in `class_derived_rows`; the ledger file is not rewritten. Rows written from now are stamped right by `Record`. The wire class of the lease-hold defer stays `infrastructure`, so the exit code and the summary count it as before. In `other`, a node's `a text lease holds its card; it may take the call later` is now `node_busy`, and the browse door's `BROWSER_UNAVAILABLE` and `DECISION_UNAVAILABLE` are `infrastructure`, `HOST_NOT_ALLOWED` is `contract` and `budget` is `budget`; its `blocked` (the model reported it cannot proceed) ties to no one class and stays in `other`. +A second pass over the rows left in `other`, `failed_verification` and `poll_deadline` (2026-11-12) moved 27 rows in the 30-day register without changing the totals: `other` 30 to 7 (23 rows), `poll_deadline` 3 to 0, and `node_busy` +8, `node_unreachable` +6 (3 of them from `poll_deadline`), `failed_verification` +4 (5 compose rows in, 1 audio row out), `abstention` +3, `config` +3, `bad_input` +2. The rules are the ones listed below and the two `RefileStamped` rules named in the section below; each needs the writer's own words, so no other row moves. Rows written before the rules are corrected when the register is read, as before. + ### The fields | field | meaning | @@ -266,10 +268,13 @@ The table is the human-maintained half of the register. A row with a stamped reason code is read as stamped, with two exceptions. `other` is what the ledger files when no member named the row at write time, so a deferred row stamped `other` is classed again and keeps `other` unless a rule below now -names a member. And three stamps were wrong until 2026-11-12 (`RefileStamped`, applied by `Record` to new rows and by the +names a member. And five stamps were wrong until 2026-11-12 (`RefileStamped`, applied by `Record` to new rows and by the register to old ones, and counted in `class_derived_rows` too): `infrastructure` on a reason holding `local seat reserved (` (unless the same reason says the remote fallback failed on a fault: a failed health probe, no remote configured, none past the capability gate, or no advertised context ceiling; that row stays `infrastructure`) -is `node_busy`; `budget` on a reason starting `agent loop: canceled (the parent context ended` is `canceled`; and -`dispatch_refused` on a reason holding `agent contract: creating pipeline-jobs dir` is `config`. Each needs the wrong stamp +is `node_busy`; `budget` on a reason starting `agent loop: canceled (the parent context ended` is `canceled`; +`dispatch_refused` on a reason holding `agent contract: creating pipeline-jobs dir` is `config`; `poll_deadline` on a reason holding +` (last poll error: ` is `node_unreachable` (the node stopped answering its polls while it owned the job; the class is for a node that answered and kept working past +the allowance); and `failed_verification` on a reason holding both `could not be measured` and `no measurement (ffmpeg/ffprobe unavailable)` is `config` (the audio QA never +judged the audio: the box lacks a tool). Each needs the wrong stamp and the writer's own sentence, so no other row of those classes moves. A deferred row without a code is classed the same way. Both are counted in `class_derived_rows` and classed by `DeferCode` in [`internal/ledger/reasoncode.go`](../../internal/ledger/reasoncode.go): its error class first, then the prefix of its reason, and `other` when neither names it. The prefix rules cover only sentences the @@ -290,14 +295,24 @@ real sentence in the source ties to an existing class without doubt; the sentenc `node_busy`; the shorter `a text lease holds its card` also ends a probe list that found no node, and is not matched. - The browse door's `browse: BROWSER_UNAVAILABLE:` and `browse: DECISION_UNAVAILABLE:` are `infrastructure`, `browse: HOST_NOT_ALLOWED:` is `contract` and `browse: budget:` is `budget`. +- `no node that answered is called ...` (a media call that named a node which did not answer the health probe) is `node_busy`, as the lane's own capacity defer. +- `dispatch : ... dial tcp ...` (a dispatch whose connection could not be made), ` consecutive poll failures, last: ...` behind `: job : ` and + `node rendered the job but fetching it failed: ...` are `node_unreachable`. The last one is the doubtful rule: the render succeeded and the fetch dropped, + so `infrastructure` is the other reading. +- `media convert: convert: in and out are required` is `bad_input`, and `reasoning tier: low classify confidence ...` (the model's confidence in a classify answer + under the minimum) is `abstention`. Only these whole sentences are named: `media : ` also wraps what ffmpeg reported, and `reasoning tier: ` wraps other verifier + wordings. +- Error classes (these come before the prefix): `no_node_configured` is `config`, `lint_errors` is `bad_input` and compose's `check_failed` is + `failed_verification` (the lane's own checks of the composition failed; `bad_input` is the other reading, since the caller wrote the composition). A door that holds a result's wire defer class (the remote lanes and the accelerator forward) stamps the code when it writes the row: the error class and the sentence first, then the defer class (`capacity` is `node_busy`), so such a defer is no longer left to read time. A deferred row whose writer gave no reason is filed with the reason `deferred: no reason recorded`, which the register reads as `other`. -Not yet ruled, because the sentence does not tie to one class without doubt: a bare `node leased`, `no node that answered -is called ...`, `no fleet node can take this edit now` and the browse door's `blocked` (the model reported it cannot proceed) and its other error classes. +Not yet ruled, because the sentence does not tie to one class without doubt: a bare `node leased`, `no fleet node can take this edit now`, the browse door's +`blocked` (the model or the loop reported no progress; 4 rows, proposed as `abstention` but the cause is not settled) and its other error classes, and the other +wordings behind `reasoning tier: `. Three `generate_svg` rows carry no reason, no reason code and no door, and cannot be ruled on. ### Wire defer classes diff --git a/internal/ledger/reasoncode.go b/internal/ledger/reasoncode.go index b558b9f19..676dbc00b 100644 --- a/internal/ledger/reasoncode.go +++ b/internal/ledger/reasoncode.go @@ -163,6 +163,9 @@ var errClassCodes = map[string]string{ // The box or the binding cannot do what was asked as configured. "not_configured": ReasonConfig, "binary_not_absolute": ReasonConfig, "model_incompatible": ReasonConfig, "device_invalid": ReasonConfig, "ffmpeg_unavailable": ReasonConfig, + // A call that names a node (or a lane's node roster) this box is not set up to dial or that has not opened the + // door the call needs (a node without media-job, a roster left out for configuration): the setup, not the load. + "no_node_configured": ReasonConfig, // A route the config binds whose file is gone (CT-54): another node may serve it, this one is misconfigured now. "route_not_ready": ReasonConfig, // The request itself is what no node can run. @@ -170,8 +173,12 @@ var errClassCodes = map[string]string{ // The engine ran and its output failed the quality gate. "black_clip": ReasonFailedVerification, "frozen_clip": ReasonFailedVerification, "depth_frames_invalid": ReasonFailedVerification, "dead_air": ReasonFailedVerification, "unmeasurable": ReasonFailedVerification, - // The input file. - "bad_input": ReasonBadInput, + // compose_video's own checks of a composition it rendered (a page error, a layout check) failed. The composition + // is what the caller wrote, so bad_input is the alternative reading; the checks are the lane's acceptance of the + // result, which is what failed_verification names. + "check_failed": ReasonFailedVerification, + // The input file, or a composition the caller wrote that the lint refuses. + "bad_input": ReasonBadInput, "lint_errors": ReasonBadInput, } // reasonPrefixCodes classify a deferred row whose err_class names nothing, by the stable prefix of the @@ -185,6 +192,9 @@ var reasonPrefixCodes = []struct{ prefix, code string }{ {"audio convert: audioio: tempdir:", ReasonInfrastructure}, {"the input has no audio stream", ReasonBadInput}, {"audio convert:", ReasonBadInput}, {"image load:", ReasonBadInput}, {"frame sampling:", ReasonBadInput}, {"empty image prompt", ReasonBadInput}, + // The media verb's own refusal of a call with no in or out. Only this whole sentence is named: "media : " also + // wraps what ffmpeg itself reported, which is not the caller's input. + {"media convert: convert: in and out are required", ReasonBadInput}, {"generate_svg: missing kind", ReasonBadInput}, {"generate_svg: bad spec", ReasonBadInput}, // The media doors' own refusals of an input that cannot be run as given (the sentence each door writes). {"empty audio prompt", ReasonBadInput}, {"empty video prompt", ReasonBadInput}, {"empty character/background prompt", ReasonBadInput}, @@ -194,6 +204,10 @@ var reasonPrefixCodes = []struct{ prefix, code string }{ // transcribe --words: the upstream answered but gave no per-word timings, so the words door has nothing to return. {"the whisper upstream returned no word timings", ReasonAbstention}, {"non-JSON output", ReasonAbstention}, + // The reasoning tier's own gate: the classify answer arrived and the model's own confidence in it was below the + // configured minimum, so it declined to commit (pipeline_attempt.go). The rest of "reasoning tier: " is the + // verifier's wording for other faults and is not named. + {"reasoning tier: low classify confidence ", ReasonAbstention}, {"transcribe call failed:", ReasonInfrastructure}, {"forecast call failed:", ReasonInfrastructure}, {"image generation failed:", ReasonInfrastructure}, {"audio generation failed:", ReasonInfrastructure}, {"video generation failed:", ReasonInfrastructure}, {"character animation failed:", ReasonInfrastructure}, {"run-graph failed:", ReasonInfrastructure}, @@ -234,12 +248,16 @@ var reasonPrefixCodes = []struct{ prefix, code string }{ // - "seat down: ": core.SeatDownReason, the node's typed seat-down outcome (ADR 0066), the code seat_down. // - the lane placement defers (a probe of every fleet node found none able to take the call), the code // no_eligible_node; "no fleet node advertises" is the accelerator forward's own. +// - "no node that answered is called ": a media call that named a node (mediaremote/named.go) which was not among +// the entries that answered the health probe; the sentence lists the entries that did not answer. It is the same +// defer CodeForLaneDefer files through the wire class capacity, so the code is node_busy. var legacyReasonPrefixCodes = []struct{ prefix, code string }{ {"wall timeout after", ReasonBudget}, {"step budget exhausted", ReasonBudget}, {"output failed schema", ReasonAbstention}, {"failed verification:", ReasonFailedVerification}, {"seat down: ", ReasonSeatDown}, {"no fleet node is eligible for the ", ReasonNoEligibleNode}, {"no fleet node serves ", ReasonNoEligibleNode}, + {"no node that answered is called ", ReasonNodeBusy}, } // queueWithdrawnClause ends a "queue deadline after" reason the node confirmed it took back @@ -251,6 +269,20 @@ const queueWithdrawnClause = "the job was withdrawn from the node, which will ne // as queue_full (capacityRefusal), whatever the body says (queue full, node leased, too many requests). var dispatchCapacityRe = regexp.MustCompile(`dispatch \S+: status (?:503|429): `) +// dialFailureWords is the Go net package's own wording of a connection that could not be made, inside the delegator's +// "dispatch : Post ...: dial tcp : " wrapper. +const dialFailureWords = " dial tcp " + +// pollFailuresRe matches the media and the other remote lanes' own give-up on a node whose job polls stopped being +// answered: "node : job : consecutive poll failures, last: " (mediaremote, textremote, +// visionremote and the others). The node stopped answering, which is the definition of node_unreachable. +var pollFailuresRe = regexp.MustCompile(`: job \S+: \d+ consecutive poll failures, last: `) + +// renderedFetchFailedSentence is the media lane's defer when a node finished the render and the fetch of its output +// then failed (mediaremote.go: "node rendered the job but fetching it failed: "). The reply the node gave +// was cut or never came back, so it reads as the node not answering; the render itself succeeded. +const renderedFetchFailedSentence = " rendered the job but fetching it failed: " + // legacyReasonCode is the code of a reason one of the sentences above names, or "". func legacyReasonCode(reason string) string { for _, p := range legacyReasonPrefixCodes { @@ -272,6 +304,15 @@ func legacyReasonCode(reason string) string { return ReasonQueueFull case strings.Contains(reason, textLeaseHoldsCardSentence): return ReasonNodeBusy + case strings.HasPrefix(reason, "dispatch ") && strings.Contains(reason, dialFailureWords): + // The delegator's own wrapper of a dispatch POST that could not connect ("dispatch : Post ...: dial tcp ..."). + // The prefix is what keeps an agent loop's own "agent loop: Post ...: dial tcp ..." (a seat's engine, seat_down or + // infrastructure) out of it: that is the box's own seat, not a node host. + return ReasonNodeUnreachable + case pollFailuresRe.MatchString(reason): + return ReasonNodeUnreachable + case strings.HasPrefix(reason, "node ") && strings.Contains(reason, renderedFetchFailedSentence): + return ReasonNodeUnreachable } return "" } @@ -297,6 +338,16 @@ const ( // the delegator reads as an ack-time 400 and files as dispatch_refused, a fact about the request or the // credentials. The folder is the node's configured home, so the fault is the node's configuration. nodeJobsDirSentence = "agent contract: creating pipeline-jobs dir" + // lastPollErrorClause is appended by the delegator to a "poll deadline after" reason when the last poll of the job + // got no answer (delegate/run_remote.go): the node stopped answering while it owned the job. The class + // poll_deadline is for a node that answered and kept working past the allowance, so this one is node_unreachable. + lastPollErrorClause = " (last poll error: " + // audioQAUnmeasuredSentence and audioQANoToolSentence are the audio QA's own words when it could not measure the + // delivered audio because ffmpeg or ffprobe is not there (render/audio-qa.mjs: "the delivered audio could not be + // measured (no measurement (ffmpeg/ffprobe unavailable))"). The audio was never judged, so nothing failed its check: + // the box is missing a tool, which is configuration. + audioQAUnmeasuredSentence = "could not be measured" + audioQANoToolSentence = "no measurement (ffmpeg/ffprobe unavailable)" ) // RefileStamped is the class a row's stamped code should read as, given its reason: the stamp itself when the pair @@ -317,6 +368,14 @@ func RefileStamped(code, reason string) string { if strings.Contains(reason, nodeJobsDirSentence) { return ReasonConfig } + case ReasonPollDeadline: + if strings.Contains(reason, lastPollErrorClause) { + return ReasonNodeUnreachable + } + case ReasonFailedVerification: + if strings.Contains(reason, audioQAUnmeasuredSentence) && strings.Contains(reason, audioQANoToolSentence) { + return ReasonConfig + } } return code } diff --git a/internal/ledger/rootcause.go b/internal/ledger/rootcause.go index 4dc5e56e3..c9543e123 100644 --- a/internal/ledger/rootcause.go +++ b/internal/ledger/rootcause.go @@ -70,13 +70,13 @@ func noRow(emitter string) FailureCause { var failureCauses = map[string]FailureCause{ ReasonNodeBusy: {Kind: CauseCapacity, Investigated: true, - RootCause: "The seat or card the call needed was held by other work when it arrived: the seat was at its local run cap (`seat busy:`), a GPU lease of another class held the card (`gpu busy: held by ...`), or a media job waited for host RAM. It also holds the two text-lease holds filed elsewhere until 2026-11-12: the delegator's `local seat reserved (gpu lease ...)` (a text lease holds the only seat the call may use and no remote had room; filed `infrastructure`) and a node's `a text lease holds its card; it may take the call later` (filed `other`). The harness defers rather than overload the card.", - Evidence: "pipeline/agenttask.go (seat busy, defer class capacity), pipeline/pipeline_gpu.go (gpu busy), ledger RefileStamped and DeferCode (the lease holds); 628 rows in 30 days as stamped: video 247, research 205, audio 109 (the agent pool at its cap, a media lease on the cards, host RAM), and 93 more once the lease holds are read here (79 from infrastructure, 14 from other)", + RootCause: "The seat or card the call needed was held by other work when it arrived: the seat was at its local run cap (`seat busy:`), a GPU lease of another class held the card (`gpu busy: held by ...`), or a media job waited for host RAM. It also holds the two text-lease holds filed elsewhere until 2026-11-12: the delegator's `local seat reserved (gpu lease ...)` (a text lease holds the only seat the call may use and no remote had room; filed `infrastructure`) and a node's `a text lease holds its card; it may take the call later` (filed `other`). A media call that named a node that did not answer the health probe (`no node that answered is called ...`, the node's own capacity defer; 8 rows, filed `other` before) is read here too. The harness defers rather than overload the card.", + Evidence: "pipeline/agenttask.go (seat busy, defer class capacity), pipeline/pipeline_gpu.go (gpu busy), ledger RefileStamped and DeferCode (the lease holds); 628 rows in 30 days as stamped: video 247, research 205, audio 109 (the agent pool at its cap, a media lease on the cards, host RAM), 93 more once the lease holds are read here (79 from infrastructure, 14 from other), and 8 more once `no node that answered is called ...` is read (video 5, audio 3)", Action: "none: working as designed. Fewer deferrals need more free card time (keep a card free for media while text leases run) or a higher run cap on a box with room."}, - ReasonFailedVerification: {Kind: CauseUnknown, - RootCause: notInvestigated + ": the rows say which check failed (`min_items` on plans, quotes or numbers; a regex) but not whether the seat's answer was wrong or the check too strict; the acceptance list is shape-only (delegate/lint.go warns of it)", - Evidence: "delegate/run.go (failed verification); 357 rows in 30 days, 345 of them from research subtasks", - Action: notInvestigated + ": read the saved output of a sample of failed research subtasks against their acceptance lists"}, + ReasonFailedVerification: {Kind: CauseHarnessDefect, Investigated: true, + RootCause: "Mostly the research lane's own acceptance checks rejecting answers that were right or honest, not seats failing. Of the 357 rows (346 from research subtasks, whose checks the lane writes itself), 307 are those checks rejecting an answer that was right or honest: 256 fail a page-word tripwire that is graded against the loop's final prose and not against the structured digest, and 51 fail a derived `min_items` check on the first required array. Not every one of the 307 is shown to be a false rejection. 202 rows are established by comparing the digest to the page: 188 tripwire rows whose quotes and figures are found verbatim in the page, and 14 `min_items` rows on pages that were empty shells, login walls or placeholders. 105 rest on the seat's own statement that the page lacks the content (68 tripwire, 37 `min_items`) and were not checked against the page. The other 50 are 10 seat misses (it could not read or complete the page, found 1 of 3 items on a long report, returned an empty digest, did not echo an id, or left a status empty), 4 caller regexes that were over-escaped, 1 misfile (an audio row, see config) and 35 undetermined, because the ledger does not keep the answer or the kind of check. The class also reads 5 compose rows (error class `check_failed`: a composition's own runtime or layout checks failed, from one caller iterating on one composition; filed `other` before), with `bad_input` as the other reading since the composition is what the caller wrote.", + Evidence: "research/anchors.go AnchorCheck, graded by core.AcceptanceCheck.Eval and evalText; delegate/run.go (failed verification); 357 rows in 30 days as stamped, 361 once the compose `check_failed` rows are read here (5 added, the audio row left). By door: research 345 and the research CLI 1, agent_delegate 8, delegate CLI 2, audio CLI 1; by seat: a 9B-class seat on three nodes 211, a 35B seat 82, the agent-pool seat 58, a 27B seat 5; 264 fall on five days (2026-10-19, -20, -23, -24, -25). Check shapes, counted apart from the causes above (the 282 tripwire rows include 26 that are seat misses or undetermined, and the tripwire and `min_items` shapes share 6 rows): docanchor regex 282, derived `min_items::1` with 0 items 70, `min_items` with 1 item against a wanted 3 on agent_delegate 2, a caller regex naming a drive path 4, `contains:` 2, a caller regex of three alternatives 1, `nonempty:status` 1, audio dead_air 1. Why the tripwire is the suspect: none of the 282 anchor sets appears in the graded prose and 5 of 282 appear in the structured digest; the 13 digests with the lowest page overlap were read and none was about another document; widening the honest-absence words (delegate/absence.go HonestAbsence) rescues 10 of 282; the failure rate among research pages that reached a verdict is 7.0% (345 against 4,604 passed) and does not step down across delegator builds (3 to 14%). Of the tripwire first attempts retried on another node, 17 ended ok, 11 failed again and 2 ended in budget. Measured 2026-11-12 by rebuilding the register's class rule outside the binary and joining the rows to the delegation log", + Action: "stop deriving a pass/fail `min_items::1` for research subtasks (excuse it when the digest says the page has none, as the existing empty-digest excuse does); do not dispatch near-empty pages (a minimum-prose floor in research.Build); grade the tripwire against the digest too, passing when its quotes, figures or titles are found verbatim in the page (research/quotes.go VerifyQuotes exists), which would have passed at least 188 of the 282; and record the answer and the acceptance kind in the ledger so the 35 undetermined rows can be settled"}, ReasonQueueFull: {Kind: CauseCapacity, Investigated: true, RootCause: "Every node asked refused the dispatch for room: its queue was at fleet_max_queue_depth (the sampled bodies read `queue full (N jobs accepted+running, limit N)`) or its card was leased. The delegator files every 503 or 429 so.", Evidence: "delegate/run_capacity.go capacityRefusal; 49 rows in 30 days (of the three commonest reason groups, 22 rows read queue full and 1 node leased)", @@ -89,17 +89,17 @@ var failureCauses = map[string]FailureCause{ RootCause: "The same event as queue_deadline, and the node confirmed taking the job back: it sat in the node's backlog (`accepted`, never `running`) past the queue patience and was withdrawn.", Evidence: "delegate/run_remote.go; 186 rows in 30 days, 182 from research subtasks, 111 on one node", Action: "none: working as designed. If one node keeps most of them (111 of 186), check its concurrency against the headroom it advertises."}, - ReasonPollDeadline: {Kind: CauseUnknown, - RootCause: notInvestigated + ": the class does not separate a job longer than the poll budget from a node that stopped making progress (failurekinds.go says so), and the 3 rows differ (a poll error, a stale last-progress, the node's own wall)", - Evidence: "delegate/run_remote.go (poll deadline after); 3 rows in 30 days", - Action: notInvestigated + ": for each row, compare the node's own record of the job with the delegator's poll budget"}, + ReasonPollDeadline: {Kind: CauseUpstream, Investigated: true, + RootCause: "The class is for a node that answered and kept working past the allowance, and the 3 rows of the 30 days as stamped were not that: each is a research job that ran 681 to 978 s and ends `last poll error ... context deadline exceeded`, so the node stopped answering the delegator's 15 s poll requests while it owned the job. Two also show stale progress (128 s against a 60 s allowance, 922 s against a 79 s allowance) while other nodes completed normally in the same delegator session. They are filed `node_unreachable` here (ledger RefileStamped reads a poll deadline that carries a last poll error so), which leaves 0 in the window. Why the node went silent (asleep, off, network, hung) is not settled: no node-side log was available.", + Evidence: "delegate/run_remote.go runRemote (the poll loop and its 15 s poll request timeout) and delegate/reasoncode.go deferReasonCode (`poll deadline` is filed poll_deadline whatever follows it); 3 rows in 30 days as stamped, 0 once the last poll error is read as node_unreachable. One is the same incident as a node_unreachable row and carries `withdraw not confirmed: no answer within 5s`; one ran 877 s on a node whose normal p99 is 281 s and maximum 740 s; the third node's own health probes also timed out three times that day. The 20 older rows (2026-09-16 to -29) are pure budget shapes with no last poll error, and none has occurred since; they stay poll_deadline", + Action: "none for the filing (RefileStamped). For the node: `local-offload doctor` shows which nodes answer now; not investigated: why that node stopped answering. Not changed here: delegate/reasoncode.go deferReasonCode still stamps a poll deadline with a last poll error as poll_deadline when it writes the row (Record refiles it on the way to disk)"}, ReasonCanceled: {Kind: CauseCallerContract, Investigated: true, RootCause: "The caller's context ended (the caller went away, or the call's own deadline passed) before or while a subtask ran. It also holds the node's `agent loop: canceled (the parent context ended ...)`, which the wire files as a budget defer because the wire has no canceled class; the ledger had filed it `budget` until 2026-11-12.", Evidence: "delegate/calldeadline.go and ledger RefileStamped (pipeline/agenttask.go:1295); 29 rows in 30 days as stamped: `canceled: context canceled` and `the call's context ended before this subtask started`, and 17 more once the loop cancellations are read here", Action: "none: working as designed."}, ReasonNodeUnreachable: {Kind: CauseUpstream, Investigated: true, - RootCause: "A node host did not answer on the network: a dial that timed out (`the connected party did not properly respond`) or a poll that never got an answer. Why the host did not answer (asleep, off, network) is not recorded on the row.", - Evidence: "delegate/run_remote.go; 3 rows in 30 days, 2 on one node", + RootCause: "A node host did not answer on the network: a dial that timed out (`the connected party did not properly respond`) or a poll that never got an answer. It also holds a dispatch whose connection could not be made (`dial tcp`), a job whose polls were not answered (`N consecutive poll failures`), a node that rendered a job and whose output then could not be fetched (`rendered the job but fetching it failed`, a dropped connection; the render itself succeeded, so infrastructure is the other reading), and the 3 poll-deadline rows that ended on a last poll error (poll_deadline until 2026-11-12). Why the host did not answer (asleep, off, network) is not recorded on the row.", + Evidence: "delegate/run_remote.go; 3 rows in 30 days as stamped, 2 on one node, and 6 more once the rules above are read here (3 from other, 3 from poll_deadline)", Action: "none from the harness: the node host or path is the fault; `local-offload doctor` shows which nodes answer now."}, ReasonJobLost: noRow("delegate/run_remote.go (a poll 404 after an acknowledged dispatch)"), ReasonDispatchRefused: {Kind: CauseNodeConfig, Investigated: true, @@ -124,10 +124,10 @@ var failureCauses = map[string]FailureCause{ RootCause: notInvestigated + ": the monitor reports a run silent in the admission phase; why is not on the row", Evidence: "agent/liveness.go; 0 rows in 30 days, 1 in the whole ledger", Action: notInvestigated + ": read the node's engine log at the time of the row"}, - ReasonStallColdLoad: {Kind: CauseUnknown, - RootCause: notInvestigated + ": the rows say no byte came for 120 s after the seat read ready, on its first completion since the load; whether the bound is too tight for a cold seat or the engine was wedged is not settled", - Evidence: "agent/liveness.go:268; 4 rows in 30 days, all on one node's pool alias", - Action: notInvestigated + ": time that seat's first completion after a load against the bound"}, + ReasonStallColdLoad: {Kind: CauseHarnessDefect, Investigated: true, + RootCause: "A placement and filing defect, from one event on 2026-11-04: the 4 rows are two delegations 7 minutes apart on the delegator's own pair of cards. In the same call and the same second the placer put a fourth subtask on the pair's long-context seat, which shares the two cards with the pool seat, so loading one evicts the other. It read the pool seat as not loaded (cold, nothing to evict) and ignored the call's own pending siblings. The displaced subtasks then waited behind the swap (a warm request answered HTTP 500 after 117 s, `mid-swap`; about 145 s of seat-down wait and about 226 s of admission wait; 0 steps) and the liveness monitor filed that as a cold-load stall and not as a capacity hold. It was not a wedge and the 120 s bound was not too tight. Established from the ledger rows and the node's notes; not from an engine log, which was not available. A wedge is excluded only because the long-context seat completed in the same window.", + Evidence: "placement/decide.go pairLong (reads occupancy at decision time); agent/liveness.go onStall, fileStallLocked and StallError.Error; seatload/activity.go Summary (`seat not loaded`); 4 rows in 30 days, all on one node's pool alias. The long-context job of that call ran ok (943 s, 7,660 tokens) and that seat appears exactly once in the whole ledger, in that call; one older cold-load stall exists (2026-09-28). Of 87 seat_down rows on the same seat, 85 overlap no other seat's run, so they are a different cause and are not covered here", + Action: "placement/decide.go pairLong should count the same call's pending siblings on the pair and queue the long subtask behind them, or refuse the co-schedule; the liveness monitor should file a wait behind a known displacement as a capacity hold (node_busy). No sentence rule can move these rows: the sentence is true of what the monitor saw."}, ReasonStallPrefill: {Kind: CauseUnknown, RootCause: notInvestigated + ": the rows say no progress in prefill past an allowance computed from the prompt size and the seat's rate; a prompt larger than the allowance or a wedged engine both read so", Evidence: "agent/liveness.go; 0 rows in 30 days, 246 in the whole ledger (written before the stamp)", @@ -148,16 +148,16 @@ var failureCauses = map[string]FailureCause{ Evidence: "core/calldeadline.go, pipeline/liveness.go:411, pipeline/agenttask.go:1295; 250 rows in 30 days as stamped, 241 from research subtasks; 17 of them are loop cancellations that ledger RefileStamped now reads as canceled", Action: "give long research subtasks a longer call deadline or fewer subtasks per call; a run that ends on the step budget needs a larger max_steps."}, ReasonAbstention: {Kind: CauseNodeConfig, Investigated: true, - RootCause: "The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call.", - Evidence: "pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days, 107 on one vision seat's questions", + RootCause: "The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call. The reasoning tier's `reasoning tier: low classify confidence ...` (the model's own confidence in a classify answer under the minimum, 3 rows, filed `other` before) is read here too.", + Evidence: "pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days as stamped, 107 on one vision seat's questions, 224 once the classify confidence defers are read (pipeline/pipeline_attempt.go)", Action: "raise the vision lane's output cap for that seat, or ask for shorter answers; not investigated: why those answers run past the cap."}, ReasonContract: {Kind: CauseCallerContract, Investigated: true, RootCause: "The caller named something the node does not serve: an image family the node does not carry.", Evidence: "config/families.go:730; 4 rows in 30 days, all one family name on generate_image", Action: "the caller names a served family, or the node adds that family."}, ReasonConfig: {Kind: CauseNodeConfig, Investigated: true, - RootCause: "A seat or node was set up so that it cannot serve the call: a seat's tool-call parser does not match its chat template, so a tool call came back as text (2 rows), or a node's configured data folder does not exist on that machine, so it could not create its pipeline-jobs folder for a dispatch (30 rows, one node; filed `dispatch_refused` until 2026-11-12).", - Evidence: "pipeline/agenttask.go:1303 and fleetnode/tasks.go:1031 (read by ledger RefileStamped); 2 rows in 30 days as stamped, 32 once the folder fault is read here", + RootCause: "A seat or node was set up so that it cannot serve the call: a seat's tool-call parser does not match its chat template, so a tool call came back as text (2 rows), or a node's configured data folder does not exist on that machine, so it could not create its pipeline-jobs folder for a dispatch (30 rows, one node; filed `dispatch_refused` until 2026-11-12), a node that has not opened the media-job door a call with input files needs (error class `no_node_configured`, 2 rows, filed `other`), or the audio QA could not measure the audio because ffmpeg or ffprobe is not installed (1 row, filed `failed_verification`).", + Evidence: "pipeline/agenttask.go:1303 and fleetnode/tasks.go:1031 (read by ledger RefileStamped); 2 rows in 30 days as stamped, 32 once the folder fault is read here, 35 once the media-job and audio QA rows are read (render/audio-qa.mjs, ledger RefileStamped and errClassCodes)", Action: "set that seat's --tool-call-parser to the one its chat template needs; fix or remove the node's missing data folder (its `home`)."}, ReasonWrite: noRow("pipeline/agenttask.go (the write door refused: agent_allow_write false, or a write set past a door cap)"), ReasonInfrastructure: {Kind: CauseUpstream, Investigated: true, @@ -165,11 +165,11 @@ var failureCauses = map[string]FailureCause{ Evidence: "agent/seatdown.go:185, pipeline_vision.go; 195 rows in 30 days as stamped: research 90, fleet 82; 72 of them were `local seat reserved (gpu lease ...)`: ledger RefileStamped now reads the 55 that name only remotes without room as node_busy, and the 17 that also name a failed health probe stay here", Action: "see seat_down; not investigated: why the engine goes away."}, ReasonBadInput: {Kind: CauseCallerContract, Investigated: true, - RootCause: "The input cannot be used as given: an image path that does not exist, a template variable the template does not declare.", - Evidence: "pipeline (image load) and compose_video BAD_INPUT; 2 rows in 30 days", + RootCause: "The input cannot be used as given: an image path that does not exist, a template variable the template does not declare, a media call with no in or out (`media convert: convert: in and out are required`), a composition the compose lint refuses (error class `lint_errors`).", + Evidence: "pipeline (image load) and compose_video BAD_INPUT and LINT_ERRORS; 2 rows in 30 days as stamped, 4 once the two added rules are read (1 media convert, 1 lint)", Action: "none: working as designed. The caller fixes the input."}, - ReasonOther: {Kind: CauseUnknown, - RootCause: notInvestigated + ": the class is the residue no rule names, a mix of causes. Since 2026-11-12 a node's `a text lease holds its card; it may take the call later` is read as node_busy, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors by their own classes; what is left is mostly the browse door's `blocked` (the model reported it cannot proceed, 4 rows) and media and compose sentences no rule names", - Evidence: "reasoncode.go DeferCode (the residue); 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the rules above are read", - Action: notInvestigated + ": list the distinct reasons of the rows left and add a rule per sentence that ties to one class"}, + ReasonOther: {Kind: CauseUnknown, Investigated: true, + RootCause: "The class is the residue no rule names, a bundle of misfiled sentences and not one cause; it stays kind unknown on purpose. Of the 55 rows as stamped, 25 are read by the rules of 2026-11-12 that came first (a node's `a text lease holds its card; it may take the call later`, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors), which leaves 30. 23 of those 30 are lane refusals and transport faults whose sentences or error classes nobody had mapped, and each is now settled by a sentence or error-class rule: `no node that answered is called ...` (8, node_busy), a dispatch whose connection could not be made (1), `N consecutive poll failures` (1) and `rendered the job but fetching it failed` (1), all node_unreachable; error class `no_node_configured` (2, config); `media convert: convert: in and out are required` (1) and error class `lint_errors` (1), bad_input; `reasoning tier: low classify confidence ...` (3, abstention); and the compose error class `check_failed` (5, failed_verification, with bad_input as the other reading, since the composition is what the caller wrote). 7 rows are left: 4 of the browse door's `blocked` (the model or the loop reported no progress, 9 to 18 s), which is proposed as abstention but whose cause is not settled and so has no rule, and 3 generate_svg rows with no reason, no reason code and no door, which cannot be traced.", + Evidence: "reasoncode.go DeferCode (the residue) and errClassCodes; 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the first rules are read, 7 once the rules above are read. Two further rows of those error classes are measurement runs, which the register leaves out", + Action: "read the 4 browse `blocked` rows against the pages they drove before ruling on them; the 3 rows with no reason need the writer to record one. Not changed here: the other verifier wordings behind `reasoning tier: ...` stay in other until a row shows them."}, } diff --git a/internal/pipeline/transcribe_noaudio_test.go b/internal/pipeline/transcribe_noaudio_test.go index a7e0e8bac..772255744 100644 --- a/internal/pipeline/transcribe_noaudio_test.go +++ b/internal/pipeline/transcribe_noaudio_test.go @@ -63,7 +63,7 @@ func TestTranscribeOfAFileWithNoAudioStreamSaysSoAndIsFiledAsABadInput(t *testin func TestEveryCoreErrClassHasALedgerReasonCode(t *testing.T) { for _, c := range []string{ core.ErrClassGPUBusy, core.ErrClassGPUQueued, core.ErrClassComposeBusy, core.ErrClassGPULeaseUnavailable, core.ErrClassBadInput, core.ErrClassFFmpegUnavailable, - errClassGPUBusy, errClassLaneGone, + core.ErrClassNoNodeConfigured, errClassGPUBusy, errClassLaneGone, } { if got := ledger.DeferCode(c, ""); got == ledger.ReasonOther { t.Errorf("err_class %q files as %q: add it to the ledger's table", c, got) diff --git a/testdata/failure-register/expected.json b/testdata/failure-register/expected.json index 2f0164b0c..bf29b08aa 100644 --- a/testdata/failure-register/expected.json +++ b/testdata/failure-register/expected.json @@ -53,9 +53,6 @@ "stall_prefill" ], "causes_not_investigated": [ - "failed_verification", - "other", - "poll_deadline", "stall_prefill" ], "classes": [ @@ -120,8 +117,8 @@ "structural_fix": "open", "cause": { "kind": "capacity signal", - "root_cause": "The seat or card the call needed was held by other work when it arrived: the seat was at its local run cap (`seat busy:`), a GPU lease of another class held the card (`gpu busy: held by ...`), or a media job waited for host RAM. It also holds the two text-lease holds filed elsewhere until 2026-11-12: the delegator's `local seat reserved (gpu lease ...)` (a text lease holds the only seat the call may use and no remote had room; filed `infrastructure`) and a node's `a text lease holds its card; it may take the call later` (filed `other`). The harness defers rather than overload the card.", - "evidence": "pipeline/agenttask.go (seat busy, defer class capacity), pipeline/pipeline_gpu.go (gpu busy), ledger RefileStamped and DeferCode (the lease holds); 628 rows in 30 days as stamped: video 247, research 205, audio 109 (the agent pool at its cap, a media lease on the cards, host RAM), and 93 more once the lease holds are read here (79 from infrastructure, 14 from other)", + "root_cause": "The seat or card the call needed was held by other work when it arrived: the seat was at its local run cap (`seat busy:`), a GPU lease of another class held the card (`gpu busy: held by ...`), or a media job waited for host RAM. It also holds the two text-lease holds filed elsewhere until 2026-11-12: the delegator's `local seat reserved (gpu lease ...)` (a text lease holds the only seat the call may use and no remote had room; filed `infrastructure`) and a node's `a text lease holds its card; it may take the call later` (filed `other`). A media call that named a node that did not answer the health probe (`no node that answered is called ...`, the node's own capacity defer; 8 rows, filed `other` before) is read here too. The harness defers rather than overload the card.", + "evidence": "pipeline/agenttask.go (seat busy, defer class capacity), pipeline/pipeline_gpu.go (gpu busy), ledger RefileStamped and DeferCode (the lease holds); 628 rows in 30 days as stamped: video 247, research 205, audio 109 (the agent pool at its cap, a media lease on the cards, host RAM), 93 more once the lease holds are read here (79 from infrastructure, 14 from other), and 8 more once `no node that answered is called ...` is read (video 5, audio 3)", "action": "none: working as designed. Fewer deferrals need more free card time (keep a card free for media while text leases run) or a higher run cap on a box with room.", "investigated": true } @@ -194,8 +191,8 @@ "structural_fix": "open", "cause": { "kind": "node configuration", - "root_cause": "The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call.", - "evidence": "pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days, 107 on one vision seat's questions", + "root_cause": "The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call. The reasoning tier's `reasoning tier: low classify confidence ...` (the model's own confidence in a classify answer under the minimum, 3 rows, filed `other` before) is read here too.", + "evidence": "pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days as stamped, 107 on one vision seat's questions, 224 once the classify confidence defers are read (pipeline/pipeline_attempt.go)", "action": "raise the vision lane's output cap for that seat, or ask for shorter answers; not investigated: why those answers run past the cap.", "investigated": true } @@ -252,11 +249,11 @@ "root_cause": "not recorded", "structural_fix": "Tighten the contract (synthetic row).", "cause": { - "kind": "unknown", - "root_cause": "not investigated: the rows say which check failed (`min_items` on plans, quotes or numbers; a regex) but not whether the seat's answer was wrong or the check too strict; the acceptance list is shape-only (delegate/lint.go warns of it)", - "evidence": "delegate/run.go (failed verification); 357 rows in 30 days, 345 of them from research subtasks", - "action": "not investigated: read the saved output of a sample of failed research subtasks against their acceptance lists", - "investigated": false + "kind": "harness defect", + "root_cause": "Mostly the research lane's own acceptance checks rejecting answers that were right or honest, not seats failing. Of the 357 rows (346 from research subtasks, whose checks the lane writes itself), 307 are those checks rejecting an answer that was right or honest: 256 fail a page-word tripwire that is graded against the loop's final prose and not against the structured digest, and 51 fail a derived `min_items` check on the first required array. Not every one of the 307 is shown to be a false rejection. 202 rows are established by comparing the digest to the page: 188 tripwire rows whose quotes and figures are found verbatim in the page, and 14 `min_items` rows on pages that were empty shells, login walls or placeholders. 105 rest on the seat's own statement that the page lacks the content (68 tripwire, 37 `min_items`) and were not checked against the page. The other 50 are 10 seat misses (it could not read or complete the page, found 1 of 3 items on a long report, returned an empty digest, did not echo an id, or left a status empty), 4 caller regexes that were over-escaped, 1 misfile (an audio row, see config) and 35 undetermined, because the ledger does not keep the answer or the kind of check. The class also reads 5 compose rows (error class `check_failed`: a composition's own runtime or layout checks failed, from one caller iterating on one composition; filed `other` before), with `bad_input` as the other reading since the composition is what the caller wrote.", + "evidence": "research/anchors.go AnchorCheck, graded by core.AcceptanceCheck.Eval and evalText; delegate/run.go (failed verification); 357 rows in 30 days as stamped, 361 once the compose `check_failed` rows are read here (5 added, the audio row left). By door: research 345 and the research CLI 1, agent_delegate 8, delegate CLI 2, audio CLI 1; by seat: a 9B-class seat on three nodes 211, a 35B seat 82, the agent-pool seat 58, a 27B seat 5; 264 fall on five days (2026-10-19, -20, -23, -24, -25). Check shapes, counted apart from the causes above (the 282 tripwire rows include 26 that are seat misses or undetermined, and the tripwire and `min_items` shapes share 6 rows): docanchor regex 282, derived `min_items:\u003cfield\u003e:1` with 0 items 70, `min_items` with 1 item against a wanted 3 on agent_delegate 2, a caller regex naming a drive path 4, `contains:\u003chex\u003e` 2, a caller regex of three alternatives 1, `nonempty:status` 1, audio dead_air 1. Why the tripwire is the suspect: none of the 282 anchor sets appears in the graded prose and 5 of 282 appear in the structured digest; the 13 digests with the lowest page overlap were read and none was about another document; widening the honest-absence words (delegate/absence.go HonestAbsence) rescues 10 of 282; the failure rate among research pages that reached a verdict is 7.0% (345 against 4,604 passed) and does not step down across delegator builds (3 to 14%). Of the tripwire first attempts retried on another node, 17 ended ok, 11 failed again and 2 ended in budget. Measured 2026-11-12 by rebuilding the register's class rule outside the binary and joining the rows to the delegation log", + "action": "stop deriving a pass/fail `min_items:\u003cfirst required array\u003e:1` for research subtasks (excuse it when the digest says the page has none, as the existing empty-digest excuse does); do not dispatch near-empty pages (a minimum-prose floor in research.Build); grade the tripwire against the digest too, passing when its quotes, figures or titles are found verbatim in the page (research/quotes.go VerifyQuotes exists), which would have passed at least 188 of the 282; and record the answer and the acceptance kind in the ledger so the 35 undetermined rows can be settled", + "investigated": true } }, { @@ -397,8 +394,8 @@ "structural_fix": "undocumented", "cause": { "kind": "node configuration", - "root_cause": "A seat or node was set up so that it cannot serve the call: a seat's tool-call parser does not match its chat template, so a tool call came back as text (2 rows), or a node's configured data folder does not exist on that machine, so it could not create its pipeline-jobs folder for a dispatch (30 rows, one node; filed `dispatch_refused` until 2026-11-12).", - "evidence": "pipeline/agenttask.go:1303 and fleetnode/tasks.go:1031 (read by ledger RefileStamped); 2 rows in 30 days as stamped, 32 once the folder fault is read here", + "root_cause": "A seat or node was set up so that it cannot serve the call: a seat's tool-call parser does not match its chat template, so a tool call came back as text (2 rows), or a node's configured data folder does not exist on that machine, so it could not create its pipeline-jobs folder for a dispatch (30 rows, one node; filed `dispatch_refused` until 2026-11-12), a node that has not opened the media-job door a call with input files needs (error class `no_node_configured`, 2 rows, filed `other`), or the audio QA could not measure the audio because ffmpeg or ffprobe is not installed (1 row, filed `failed_verification`).", + "evidence": "pipeline/agenttask.go:1303 and fleetnode/tasks.go:1031 (read by ledger RefileStamped); 2 rows in 30 days as stamped, 32 once the folder fault is read here, 35 once the media-job and audio QA rows are read (render/audio-qa.mjs, ledger RefileStamped and errClassCodes)", "action": "set that seat's --tool-call-parser to the one its chat template needs; fix or remove the node's missing data folder (its `home`).", "investigated": true } @@ -445,8 +442,8 @@ "structural_fix": "open", "cause": { "kind": "upstream service", - "root_cause": "A node host did not answer on the network: a dial that timed out (`the connected party did not properly respond`) or a poll that never got an answer. Why the host did not answer (asleep, off, network) is not recorded on the row.", - "evidence": "delegate/run_remote.go; 3 rows in 30 days, 2 on one node", + "root_cause": "A node host did not answer on the network: a dial that timed out (`the connected party did not properly respond`) or a poll that never got an answer. It also holds a dispatch whose connection could not be made (`dial tcp`), a job whose polls were not answered (`N consecutive poll failures`), a node that rendered a job and whose output then could not be fetched (`rendered the job but fetching it failed`, a dropped connection; the render itself succeeded, so infrastructure is the other reading), and the 3 poll-deadline rows that ended on a last poll error (poll_deadline until 2026-11-12). Why the host did not answer (asleep, off, network) is not recorded on the row.", + "evidence": "delegate/run_remote.go; 3 rows in 30 days as stamped, 2 on one node, and 6 more once the rules above are read here (3 from other, 3 from poll_deadline)", "action": "none from the harness: the node host or path is the fault; `local-offload doctor` shows which nodes answer now.", "investigated": true } @@ -493,10 +490,10 @@ "structural_fix": "undocumented", "cause": { "kind": "unknown", - "root_cause": "not investigated: the class is the residue no rule names, a mix of causes. Since 2026-11-12 a node's `a text lease holds its card; it may take the call later` is read as node_busy, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors by their own classes; what is left is mostly the browse door's `blocked` (the model reported it cannot proceed, 4 rows) and media and compose sentences no rule names", - "evidence": "reasoncode.go DeferCode (the residue); 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the rules above are read", - "action": "not investigated: list the distinct reasons of the rows left and add a rule per sentence that ties to one class", - "investigated": false + "root_cause": "The class is the residue no rule names, a bundle of misfiled sentences and not one cause; it stays kind unknown on purpose. Of the 55 rows as stamped, 25 are read by the rules of 2026-11-12 that came first (a node's `a text lease holds its card; it may take the call later`, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors), which leaves 30. 23 of those 30 are lane refusals and transport faults whose sentences or error classes nobody had mapped, and each is now settled by a sentence or error-class rule: `no node that answered is called ...` (8, node_busy), a dispatch whose connection could not be made (1), `N consecutive poll failures` (1) and `rendered the job but fetching it failed` (1), all node_unreachable; error class `no_node_configured` (2, config); `media convert: convert: in and out are required` (1) and error class `lint_errors` (1), bad_input; `reasoning tier: low classify confidence ...` (3, abstention); and the compose error class `check_failed` (5, failed_verification, with bad_input as the other reading, since the composition is what the caller wrote). 7 rows are left: 4 of the browse door's `blocked` (the model or the loop reported no progress, 9 to 18 s), which is proposed as abstention but whose cause is not settled and so has no rule, and 3 generate_svg rows with no reason, no reason code and no door, which cannot be traced.", + "evidence": "reasoncode.go DeferCode (the residue) and errClassCodes; 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the first rules are read, 7 once the rules above are read. Two further rows of those error classes are measurement runs, which the register leaves out", + "action": "read the 4 browse `blocked` rows against the pages they drove before ruling on them; the 3 rows with no reason need the writer to record one. Not changed here: the other verifier wordings behind `reasoning tier: ...` stay in other until a row shows them.", + "investigated": true } }, { @@ -540,11 +537,11 @@ "root_cause": "A job larger than the budget.", "structural_fix": "open", "cause": { - "kind": "unknown", - "root_cause": "not investigated: the class does not separate a job longer than the poll budget from a node that stopped making progress (failurekinds.go says so), and the 3 rows differ (a poll error, a stale last-progress, the node's own wall)", - "evidence": "delegate/run_remote.go (poll deadline after); 3 rows in 30 days", - "action": "not investigated: for each row, compare the node's own record of the job with the delegator's poll budget", - "investigated": false + "kind": "upstream service", + "root_cause": "The class is for a node that answered and kept working past the allowance, and the 3 rows of the 30 days as stamped were not that: each is a research job that ran 681 to 978 s and ends `last poll error ... context deadline exceeded`, so the node stopped answering the delegator's 15 s poll requests while it owned the job. Two also show stale progress (128 s against a 60 s allowance, 922 s against a 79 s allowance) while other nodes completed normally in the same delegator session. They are filed `node_unreachable` here (ledger RefileStamped reads a poll deadline that carries a last poll error so), which leaves 0 in the window. Why the node went silent (asleep, off, network, hung) is not settled: no node-side log was available.", + "evidence": "delegate/run_remote.go runRemote (the poll loop and its 15 s poll request timeout) and delegate/reasoncode.go deferReasonCode (`poll deadline` is filed poll_deadline whatever follows it); 3 rows in 30 days as stamped, 0 once the last poll error is read as node_unreachable. One is the same incident as a node_unreachable row and carries `withdraw not confirmed: no answer within 5s`; one ran 877 s on a node whose normal p99 is 281 s and maximum 740 s; the third node's own health probes also timed out three times that day. The 20 older rows (2026-09-16 to -29) are pure budget shapes with no last poll error, and none has occurred since; they stay poll_deadline", + "action": "none for the filing (RefileStamped). For the node: `local-offload doctor` shows which nodes answer now; not investigated: why that node stopped answering. Not changed here: delegate/reasoncode.go deferReasonCode still stamps a poll deadline with a last poll error as poll_deadline when it writes the row (Record refiles it on the way to disk)", + "investigated": true } }, { From 28812dbee65665dce3e0ff04302860b55d8f5146 Mon Sep 17 00:00:00 2001 From: dmmdea Date: Thu, 12 Nov 2026 12:47:41 -0500 Subject: [PATCH 2/2] Rule the browse "blocked" rows, state the research acceptance fix in the failed_verification entry, and keep the misfiling comment from going stale The browse door's "browse: blocked:" (the model or the loop reported no progress) is abstention: the loop ended with no usable answer. It is a prefix rule on the door's own sentence and not an errClassCodes entry, because the word means other things elsewhere; the prefix is reached since the error class is not in the table. other falls to the 3 rows with no reason, and abstention takes the 4 browse rows. The failed_verification entry now says the research acceptance fix ships in the same release, with the replay numbers, what still fails and that rows already written keep their class. The comment in failureClassOf points at RefileStamped's own list instead of naming the pairs, so it cannot go stale. The page, the fragment and the fixture golden follow the catalogue. Co-Authored-By: Claude Sonnet 5.5 --- changes/unreleased/fix-register-rules-and-causes.md | 11 ++++++----- docs/systems/failure-register.md | 12 ++++++------ internal/ledger/failures.go | 5 ++--- internal/ledger/reasoncode.go | 9 ++++++--- internal/ledger/rootcause.go | 12 ++++++------ testdata/failure-register/expected.json | 12 ++++++------ 6 files changed, 32 insertions(+), 29 deletions(-) diff --git a/changes/unreleased/fix-register-rules-and-causes.md b/changes/unreleased/fix-register-rules-and-causes.md index 29ee6329f..6b0676381 100644 --- a/changes/unreleased/fix-register-rules-and-causes.md +++ b/changes/unreleased/fix-register-rules-and-causes.md @@ -1,10 +1,11 @@ -### Fixed - the failure register files 27 more rows under the class that names them, and the four classes it had not investigated now carry their causes +### Fixed - the failure register files 31 more rows under the class that names them, and the four classes it had not investigated now carry their causes - **What was wrong.** After the last release 30 failure rows still sat in `other` and 3 in `poll_deadline`, and four classes read "not investigated" (`failed_verification` 357 rows, `other`, `poll_deadline` 3, `stall_cold_load` 4). The rows in `other` were lane refusals and transport faults whose sentences or error classes no rule named. The 3 `poll_deadline` rows were not that class: each ends on a last poll error, so the node had stopped answering, not kept working past the allowance. One audio row was filed `failed_verification` though the audio was never judged (ffmpeg or ffprobe was missing). -- **Where each row now goes.** `no node that answered is called ...` (8 rows) is `node_busy`, the class the lane's own defer already files for new rows through its wire class. A dispatch whose connection could not be made (1), `N consecutive poll failures` (1), `rendered the job but fetching it failed` (1) and the 3 poll-deadline rows with a last poll error are `node_unreachable`. Error class `no_node_configured` (2) and the audio QA's missing-ffmpeg dead air (1) are `config`. `media convert: convert: in and out are required` (1) and error class `lint_errors` (1) are `bad_input`. `reasoning tier: low classify confidence ...` (3) is `abstention`. Compose's `check_failed` (5) is `failed_verification`. +- **Where each row now goes.** `no node that answered is called ...` (8 rows) is `node_busy`, the class the lane's own defer already files for new rows through its wire class. A dispatch whose connection could not be made (1), `N consecutive poll failures` (1), `rendered the job but fetching it failed` (1) and the 3 poll-deadline rows with a last poll error are `node_unreachable`. Error class `no_node_configured` (2) and the audio QA's missing-ffmpeg dead air (1) are `config`. `media convert: convert: in and out are required` (1) and error class `lint_errors` (1) are `bad_input`. `reasoning tier: low classify confidence ...` (3) is `abstention`. Compose's `check_failed` (5) is `failed_verification`, and the browse door's `blocked` (4) is `abstention`. - **Two readings that are a choice.** `rendered the job but fetching it failed` is `node_unreachable` (the fetch dropped), with `infrastructure` the other reading because the render itself succeeded. Compose `check_failed` is `failed_verification` (the lane's own checks of the composition failed), with `bad_input` the other reading because the caller wrote the composition. Both are one-line rule changes. -- **Left alone on purpose.** The browse door's `blocked` (4 rows) is proposed as `abstention` but its cause is not settled, so it has no rule; the 3 `generate_svg` rows with no reason, no reason code and no door cannot be traced. `other` is 7 rows. +- **The browse `blocked` rows.** `browse: blocked: the model or the loop reported no progress` (4 rows, one caller retrying within 4 minutes) is `abstention`: the loop ended with no usable answer. Why the page could not be driven is a separate question the row cannot answer. It is a prefix rule on the door's own sentence and not an error-class entry, because the word `blocked` means other things elsewhere; the prefix is reached because the error class is not in the table. +- **Left alone on purpose.** The 3 `generate_svg` rows with no reason, no reason code and no door cannot be traced. `other` is 3 rows. - **How.** Each rule needs the writer's own fixed words. `media convert: convert: in and out are required` and `reasoning tier: low classify confidence ` are matched as whole sentences, not as the bare prefixes `media convert:` and `reasoning tier: `, because `media : ` also wraps what ffmpeg reported and `reasoning tier: ` wraps other verifier wordings. Three new error classes are in `errClassCodes`; the dispatch, poll-failure and fetch sentences are in `legacyReasonCode`; the poll-deadline and audio rules are in `RefileStamped` and each needs both the old stamp and the writer's sentence, so no other row of those classes moves. The pin test `TestEveryCoreErrClassHasALedgerReasonCode` listed every error class but `no_node_configured`; it does now. -- **Causes.** `internal/ledger/rootcause.go` now holds the findings, with their counts and the day measured, and says where the evidence stops. `failed_verification` is a harness defect for the research lane's own checks: of 357 rows, 307 are those checks rejecting answers that were right or honest, of which 202 are established by comparing the digest to the page and 105 rest on the seat's own statement; 10 are seat misses, 4 a caller regex over-escaped, 1 a misfile, 35 undetermined. `poll_deadline` is an upstream node that stopped answering (why is not settled). `stall_cold_load` is one placement and filing defect on 2026-11-04: the placer put a long-context subtask on a seat that shares the pair's cards with the pool seat, and the monitor filed the wait behind the swap as a cold-load stall. `other` is a bundle of misfiled sentences, not one cause. The follow-ups it names (stop deriving the `min_items` check, count the call's own siblings in `pairLong`, record the answer in the ledger, stop stamping a poll deadline with a last poll error as `poll_deadline` at the writer) are not made here. +- **Causes.** `internal/ledger/rootcause.go` now holds the findings, with their counts and the day measured, and says where the evidence stops. `failed_verification` is a harness defect for the research lane's own checks: of 357 rows, 307 are those checks rejecting answers that were right or honest, of which 202 are established by comparing the digest to the page and 105 rest on the seat's own statement; 10 are seat misses, 4 a caller regex over-escaped, 1 a misfile, 35 undetermined. `poll_deadline` is an upstream node that stopped answering (why is not settled). `stall_cold_load` is one placement and filing defect on 2026-11-04: the placer put a long-context subtask on a seat that shares the pair's cards with the pool seat, and the monitor filed the wait behind the swap as a cold-load stall. `other` is a bundle of misfiled sentences, not one cause. The research acceptance fix ships in the same release: a digest that quotes its page is excused from the page-word tripwire, `research.Build` derives a `nonempty:` check over the caller's required fields in place of `min_items::1`, and an HTML page under 150 characters of text is refused before dispatch as a skipped source. An offline replay of the saved pages and answers (which reproduced every saved verdict, 356 fail and 4,622 pass) gives, of the 356 failed rows with a job id, 167 now pass, 16 are refused as pages with no content and 173 still fail (10 not research); of the 4,622 passes, 4,513 still pass, 109 are refused as pages with no content and none fails. Rows already written keep their class, so none of the register's 30-day counts changes from it. The follow-ups the entry still names (count the call's own siblings in `pairLong`, record the answer and the acceptance kind on the ledger row, stop stamping a poll deadline with a last poll error as `poll_deadline` at the writer) are not made here. - **Docs.** The catalogue copy and the rule lists on `docs/systems/failure-register.md` match the code; `causes_not_investigated` is empty in the window. -- **Check.** The built binary and the installed one, both read-only on the real ledger, `ledger --failures --since 30`: 2,566 failure rows of 8,785 job rows in both, the sum over classes unchanged. `other` 30 to 7, `poll_deadline` 3 to 0, `node_busy` 704 to 712, `node_unreachable` 3 to 9, `failed_verification` 357 to 361, `abstention` 221 to 224, `config` 32 to 35, `bad_input` 2 to 4; every other class is unchanged. `causes_not_investigated` went from `failed_verification`, `other`, `poll_deadline`, `stall_cold_load` to none. Two further rows of the `no_node_configured` class are measurement runs the register leaves out. +- **Check.** The built binary and the installed one, both read-only on the real ledger, `ledger --failures --since 30`: 2,566 failure rows of 8,785 job rows in both, the sum over classes unchanged. `other` 30 to 3, `poll_deadline` 3 to 0, `node_busy` 704 to 712, `node_unreachable` 3 to 9, `failed_verification` 357 to 361, `abstention` 221 to 228, `config` 32 to 35, `bad_input` 2 to 4; every other class is unchanged. `causes_not_investigated` went from `failed_verification`, `other`, `poll_deadline`, `stall_cold_load` to none. Two further rows of the `no_node_configured` class are measurement runs the register leaves out. diff --git a/docs/systems/failure-register.md b/docs/systems/failure-register.md index 3405e954e..0e7953f90 100644 --- a/docs/systems/failure-register.md +++ b/docs/systems/failure-register.md @@ -114,7 +114,7 @@ rows it rests on and which it does not (202 of 357 `failed_verification` rows co | failure class | cause kind | root cause | evidence | action | |---|---|---|---|---| -| `failed_verification` | harness defect | Mostly the research lane's own acceptance checks rejecting answers that were right or honest, not seats failing. Of the 357 rows (346 from research subtasks, whose checks the lane writes itself), 307 are those checks rejecting an answer that was right or honest: 256 fail a page-word tripwire that is graded against the loop's final prose and not against the structured digest, and 51 fail a derived `min_items` check on the first required array. Not every one of the 307 is shown to be a false rejection. 202 rows are established by comparing the digest to the page: 188 tripwire rows whose quotes and figures are found verbatim in the page, and 14 `min_items` rows on pages that were empty shells, login walls or placeholders. 105 rest on the seat's own statement that the page lacks the content (68 tripwire, 37 `min_items`) and were not checked against the page. The other 50 are 10 seat misses (it could not read or complete the page, found 1 of 3 items on a long report, returned an empty digest, did not echo an id, or left a status empty), 4 caller regexes that were over-escaped, 1 misfile (an audio row, see config) and 35 undetermined, because the ledger does not keep the answer or the kind of check. The class also reads 5 compose rows (error class `check_failed`: a composition's own runtime or layout checks failed, from one caller iterating on one composition; filed `other` before), with `bad_input` as the other reading since the composition is what the caller wrote. | research/anchors.go AnchorCheck, graded by core.AcceptanceCheck.Eval and evalText; delegate/run.go (failed verification); 357 rows in 30 days as stamped, 361 once the compose `check_failed` rows are read here (5 added, the audio row left). By door: research 345 and the research CLI 1, agent_delegate 8, delegate CLI 2, audio CLI 1; by seat: a 9B-class seat on three nodes 211, a 35B seat 82, the agent-pool seat 58, a 27B seat 5; 264 fall on five days (2026-10-19, -20, -23, -24, -25). Check shapes, counted apart from the causes above (the 282 tripwire rows include 26 that are seat misses or undetermined, and the tripwire and `min_items` shapes share 6 rows): docanchor regex 282, derived `min_items::1` with 0 items 70, `min_items` with 1 item against a wanted 3 on agent_delegate 2, a caller regex naming a drive path 4, `contains:` 2, a caller regex of three alternatives 1, `nonempty:status` 1, audio dead_air 1. Why the tripwire is the suspect: none of the 282 anchor sets appears in the graded prose and 5 of 282 appear in the structured digest; the 13 digests with the lowest page overlap were read and none was about another document; widening the honest-absence words (delegate/absence.go HonestAbsence) rescues 10 of 282; the failure rate among research pages that reached a verdict is 7.0% (345 against 4,604 passed) and does not step down across delegator builds (3 to 14%). Of the tripwire first attempts retried on another node, 17 ended ok, 11 failed again and 2 ended in budget. Measured 2026-11-12 by rebuilding the register's class rule outside the binary and joining the rows to the delegation log | stop deriving a pass/fail `min_items::1` for research subtasks (excuse it when the digest says the page has none, as the existing empty-digest excuse does); do not dispatch near-empty pages (a minimum-prose floor in research.Build); grade the tripwire against the digest too, passing when its quotes, figures or titles are found verbatim in the page (research/quotes.go VerifyQuotes exists), which would have passed at least 188 of the 282; and record the answer and the acceptance kind in the ledger so the 35 undetermined rows can be settled | +| `failed_verification` | harness defect | Mostly the research lane's own acceptance checks rejecting answers that were right or honest, not seats failing. Of the 357 rows (346 from research subtasks, whose checks the lane writes itself), 307 are those checks rejecting an answer that was right or honest: 256 fail a page-word tripwire that is graded against the loop's final prose and not against the structured digest, and 51 fail a derived `min_items` check on the first required array. Not every one of the 307 is shown to be a false rejection. 202 rows are established by comparing the digest to the page: 188 tripwire rows whose quotes and figures are found verbatim in the page, and 14 `min_items` rows on pages that were empty shells, login walls or placeholders. 105 rest on the seat's own statement that the page lacks the content (68 tripwire, 37 `min_items`) and were not checked against the page. The other 50 are 10 seat misses (it could not read or complete the page, found 1 of 3 items on a long report, returned an empty digest, did not echo an id, or left a status empty), 4 caller regexes that were over-escaped, 1 misfile (an audio row, see config) and 35 undetermined, because the ledger does not keep the answer or the kind of check. The class also reads 5 compose rows (error class `check_failed`: a composition's own runtime or layout checks failed, from one caller iterating on one composition; filed `other` before), with `bad_input` as the other reading since the composition is what the caller wrote. | research/anchors.go AnchorCheck, graded by core.AcceptanceCheck.Eval and evalText; delegate/run.go (failed verification); 357 rows in 30 days as stamped, 361 once the compose `check_failed` rows are read here (5 added, the audio row left). By door: research 345 and the research CLI 1, agent_delegate 8, delegate CLI 2, audio CLI 1; by seat: a 9B-class seat on three nodes 211, a 35B seat 82, the agent-pool seat 58, a 27B seat 5; 264 fall on five days (2026-10-19, -20, -23, -24, -25). Check shapes, counted apart from the causes above (the 282 tripwire rows include 26 that are seat misses or undetermined, and the tripwire and `min_items` shapes share 6 rows): docanchor regex 282, derived `min_items::1` with 0 items 70, `min_items` with 1 item against a wanted 3 on agent_delegate 2, a caller regex naming a drive path 4, `contains:` 2, a caller regex of three alternatives 1, `nonempty:status` 1, audio dead_air 1. Why the tripwire is the suspect: none of the 282 anchor sets appears in the graded prose and 5 of 282 appear in the structured digest; the 13 digests with the lowest page overlap were read and none was about another document; widening the honest-absence words (delegate/absence.go HonestAbsence) rescues 10 of 282; the failure rate among research pages that reached a verdict is 7.0% (345 against 4,604 passed) and does not step down across delegator builds (3 to 14%). Of the tripwire first attempts retried on another node, 17 ended ok, 11 failed again and 2 ended in budget. Measured 2026-11-12 by rebuilding the register's class rule outside the binary and joining the rows to the delegation log | the research acceptance is fixed in the same release (measured 2026-11-12 by an offline replay of the saved pages and answers through research.Build and delegate.EvalAcceptanceGraded, read-only; the original code reproduced every saved verdict, 356 fail and 4,622 pass). Three changes: a digest that quotes its page is excused from the page-word tripwire (delegate.DigestGrounded: one string of 60 or more characters, or two different ones of 15 or more, found in the contract's documents and not in the goal); research.Build derives a `nonempty:` check over the caller's required fields, any one of them filled, in place of `min_items::1`; and an HTML page under 150 characters of text is refused before dispatch as a skipped source (`error_kind` `empty_page`, no ledger row). Of the 356 failed rows with a job id, 167 now pass (125 tripwire failures by quoting the page, 42 item-count failures), 16 are refused as pages with no content and 173 still fail, 10 of them not research. Of the 4,622 passes, 4,513 still pass, 109 are refused as pages with no content and none fails. Rows already written keep their class, so the counts above and the register's 30-day count do not change; only rows written after the release are affected. Still open: the 173 (163 research rows: 67 digests that say the page lacks what was asked but also make a claim, 62 that hold strings of the page but under the grounding thresholds, 23 residual, 6 item-count rows silent in every required field, 3 where the seat could not read, 2 all-empty digests; plus the 10 that are not research), and recording the answer and the acceptance kind on the ledger row so the 35 undetermined rows can be settled. Known limit: a seat that says none for a page that holds the items is believed, now under a caller's own schema too; a caller who needs items writes `min_items:` in their own acceptance | | `queue_full` | capacity signal | Every node asked refused the dispatch for room: its queue was at fleet_max_queue_depth (the sampled bodies read `queue full (N jobs accepted+running, limit N)`) or its card was leased. The delegator files every 503 or 429 so. | delegate/run_capacity.go capacityRefusal; 49 rows in 30 days (of the three commonest reason groups, 22 rows read queue full and 1 node leased) | none: working as designed. A node with spare capacity can raise fleet_max_queue_depth. | | `queue_deadline` | capacity signal | A node accepted the job and its backlog never reached it inside the queue patience, and the node did not confirm taking it back. Jobs queue behind other work on a full node. | delegate/run_remote.go (queue deadline after); 0 rows in 30 days, 318 in the whole ledger, all written before the withdrawal stamp existed | none: working as designed (the delegator withdraws and re-places since ADR 0064; queue_withdrawn is the same event confirmed). | | `queue_withdrawn` | capacity signal | The same event as queue_deadline, and the node confirmed taking the job back: it sat in the node's backlog (`accepted`, never `running`) past the queue patience and was withdrawn. | delegate/run_remote.go; 186 rows in 30 days, 182 from research subtasks, 111 on one node | none: working as designed. If one node keeps most of them (111 of 186), check its concurrency against the headroom it advertises. | @@ -138,17 +138,17 @@ rows it rests on and which it does not (202 of 357 `failed_verification` rows co | `stall_engine` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | agent/liveness.go (the engine did no work, or kept stepping without a token); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | | `stall_other` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | delegate/reasoncode.go stallReasonCode (a stall whose phase the reason does not name); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | | `budget` | caller contract | A limit the call carried ended the run before the agent finished: the call's own deadline (`call deadline reached`, `wall timeout after ... the caller's deadline`), or the step budget (`step budget exhausted`). Most are the caller's deadline, not a node ceiling. The agent loop's `agent loop: canceled` was filed here until 2026-11-12 and is now `canceled`; the browse door's `browse: budget:` (max_actions spent) is read here. | core/calldeadline.go, pipeline/liveness.go:411, pipeline/agenttask.go:1295; 250 rows in 30 days as stamped, 241 from research subtasks; 17 of them are loop cancellations that ledger RefileStamped now reads as canceled | give long research subtasks a longer call deadline or fewer subtasks per call; a run that ends on the step budget needs a larger max_steps. | -| `abstention` | node configuration | The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call. The reasoning tier's `reasoning tier: low classify confidence ...` (the model's own confidence in a classify answer under the minimum, 3 rows, filed `other` before) is read here too. | pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days as stamped, 107 on one vision seat's questions, 224 once the classify confidence defers are read (pipeline/pipeline_attempt.go) | raise the vision lane's output cap for that seat, or ask for shorter answers; not investigated: why those answers run past the cap. | +| `abstention` | node configuration | The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call. The reasoning tier's `reasoning tier: low classify confidence ...` (the model's own confidence in a classify answer under the minimum, 3 rows, filed `other` before) is read here too, as is the browse door's `browse: blocked:` (the model or the loop reported no progress, 4 rows, filed `other` before; why the page could not be driven is not on the row). | pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days as stamped, 107 on one vision seat's questions, 228 once the classify confidence defers and the browse `blocked` rows are read (pipeline/pipeline_attempt.go, pipeline/browse.go) | raise the vision lane's output cap for that seat, or ask for shorter answers; not investigated: why those answers run past the cap. | | `contract` | caller contract | The caller named something the node does not serve: an image family the node does not carry. | config/families.go:730; 4 rows in 30 days, all one family name on generate_image | the caller names a served family, or the node adds that family. | | `config` | node configuration | A seat or node was set up so that it cannot serve the call: a seat's tool-call parser does not match its chat template, so a tool call came back as text (2 rows), or a node's configured data folder does not exist on that machine, so it could not create its pipeline-jobs folder for a dispatch (30 rows, one node; filed `dispatch_refused` until 2026-11-12), a node that has not opened the media-job door a call with input files needs (error class `no_node_configured`, 2 rows, filed `other`), or the audio QA could not measure the audio because ffmpeg or ffprobe is not installed (1 row, filed `failed_verification`). | pipeline/agenttask.go:1303 and fleetnode/tasks.go:1031 (read by ledger RefileStamped); 2 rows in 30 days as stamped, 32 once the folder fault is read here, 35 once the media-job and audio QA rows are read (render/audio-qa.mjs, ledger RefileStamped and errClassCodes) | set that seat's --tool-call-parser to the one its chat template needs; fix or remove the node's missing data folder (its `home`). | | `write` | unknown | not investigated: the ledger holds no row of this class, so no failure of it has been observed; it settles when a row appears and its reason and node are read | pipeline/agenttask.go (the write door refused: agent_allow_write false, or a write set past a door cap); 0 rows in the 30 days to 2026-11-12 and 0 in the whole ledger | not investigated | | `infrastructure` | upstream service | Mostly a seat whose engine was gone from llama-swap's /running (the node's defer class infrastructure carrying the same sentence as seat_down), then a vision seat that was cold and timed out, and the browse door's `browse: BROWSER_UNAVAILABLE:` and `browse: DECISION_UNAVAILABLE:`. The delegator's `local seat reserved (gpu lease ...)` was filed here until 2026-11-12; it is a capacity hold and is now `node_busy` (the wire class of that defer stays infrastructure, so the exit code and summary still count it). | agent/seatdown.go:185, pipeline_vision.go; 195 rows in 30 days as stamped: research 90, fleet 82; 72 of them were `local seat reserved (gpu lease ...)`: ledger RefileStamped now reads the 55 that name only remotes without room as node_busy, and the 17 that also name a failed health probe stay here | see seat_down; not investigated: why the engine goes away. | | `bad_input` | caller contract | The input cannot be used as given: an image path that does not exist, a template variable the template does not declare, a media call with no in or out (`media convert: convert: in and out are required`), a composition the compose lint refuses (error class `lint_errors`). | pipeline (image load) and compose_video BAD_INPUT and LINT_ERRORS; 2 rows in 30 days as stamped, 4 once the two added rules are read (1 media convert, 1 lint) | none: working as designed. The caller fixes the input. | -| `other` | unknown | The class is the residue no rule names, a bundle of misfiled sentences and not one cause; it stays kind unknown on purpose. Of the 55 rows as stamped, 25 are read by the rules of 2026-11-12 that came first (a node's `a text lease holds its card; it may take the call later`, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors), which leaves 30. 23 of those 30 are lane refusals and transport faults whose sentences or error classes nobody had mapped, and each is now settled by a sentence or error-class rule: `no node that answered is called ...` (8, node_busy), a dispatch whose connection could not be made (1), `N consecutive poll failures` (1) and `rendered the job but fetching it failed` (1), all node_unreachable; error class `no_node_configured` (2, config); `media convert: convert: in and out are required` (1) and error class `lint_errors` (1), bad_input; `reasoning tier: low classify confidence ...` (3, abstention); and the compose error class `check_failed` (5, failed_verification, with bad_input as the other reading, since the composition is what the caller wrote). 7 rows are left: 4 of the browse door's `blocked` (the model or the loop reported no progress, 9 to 18 s), which is proposed as abstention but whose cause is not settled and so has no rule, and 3 generate_svg rows with no reason, no reason code and no door, which cannot be traced. | reasoncode.go DeferCode (the residue) and errClassCodes; 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the first rules are read, 7 once the rules above are read. Two further rows of those error classes are measurement runs, which the register leaves out | read the 4 browse `blocked` rows against the pages they drove before ruling on them; the 3 rows with no reason need the writer to record one. Not changed here: the other verifier wordings behind `reasoning tier: ...` stay in other until a row shows them. | +| `other` | unknown | The class is the residue no rule names, a bundle of misfiled sentences and not one cause; it stays kind unknown on purpose. Of the 55 rows as stamped, 25 are read by the rules of 2026-11-12 that came first (a node's `a text lease holds its card; it may take the call later`, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors), which leaves 30. 27 of those 30 are lane refusals and transport faults whose sentences or error classes nobody had mapped, and each is now settled by a sentence or error-class rule: `no node that answered is called ...` (8, node_busy), a dispatch whose connection could not be made (1), `N consecutive poll failures` (1) and `rendered the job but fetching it failed` (1), all node_unreachable; error class `no_node_configured` (2, config); `media convert: convert: in and out are required` (1) and error class `lint_errors` (1), bad_input; `reasoning tier: low classify confidence ...` (3, abstention); the compose error class `check_failed` (5, failed_verification, with bad_input as the other reading, since the composition is what the caller wrote); and the browse door's `browse: blocked: the model or the loop reported no progress` (4, abstention: the loop ended with no usable answer; why the page could not be driven is not on the row). 3 rows are left: generate_svg rows with no reason, no reason code and no door, which cannot be traced. | reasoncode.go DeferCode (the residue) and errClassCodes; 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the first rules are read, 3 once the rules above are read. Two further rows of those error classes are measurement runs, which the register leaves out | the 3 rows with no reason need the writer to record one; why the browse door could not drive its page is a separate question the 4 `blocked` rows cannot answer. Not changed here: the other verifier wordings behind `reasoning tier: ...` stay in other until a row shows them. | Reading the rows found three misfilings and two sentences left in `other`, all fixed on 2026-11-12 (`RefileStamped` and `DeferCode` in `internal/ledger/reasoncode.go`): a delegate row for a local seat held by a text lease (`local seat reserved (gpu lease ...)`) was filed `infrastructure` and is now `node_busy`, because the class is defined as the seat or card being held by other work (`capacity_wait` is the delegator finding no node with room, a different cause with a different fix); the agent loop's `agent loop: canceled` was filed `budget` and is now `canceled`; and a node's own folder failure (`agent contract: creating pipeline-jobs dir`, an ack-time 400) was filed `dispatch_refused`, a fact about the request or the credentials, and is now `config` (the node's configured home does not exist on that machine; `infrastructure` is for a stack part that failed while configured right). Rows written before the fix are corrected when the register is read: a row stamped with the old class and carrying the writer's own sentence is counted under the new one and in `class_derived_rows`; the ledger file is not rewritten. Rows written from now are stamped right by `Record`. The wire class of the lease-hold defer stays `infrastructure`, so the exit code and the summary count it as before. In `other`, a node's `a text lease holds its card; it may take the call later` is now `node_busy`, and the browse door's `BROWSER_UNAVAILABLE` and `DECISION_UNAVAILABLE` are `infrastructure`, `HOST_NOT_ALLOWED` is `contract` and `budget` is `budget`; its `blocked` (the model reported it cannot proceed) ties to no one class and stays in `other`. -A second pass over the rows left in `other`, `failed_verification` and `poll_deadline` (2026-11-12) moved 27 rows in the 30-day register without changing the totals: `other` 30 to 7 (23 rows), `poll_deadline` 3 to 0, and `node_busy` +8, `node_unreachable` +6 (3 of them from `poll_deadline`), `failed_verification` +4 (5 compose rows in, 1 audio row out), `abstention` +3, `config` +3, `bad_input` +2. The rules are the ones listed below and the two `RefileStamped` rules named in the section below; each needs the writer's own words, so no other row moves. Rows written before the rules are corrected when the register is read, as before. +A second pass over the rows left in `other`, `failed_verification` and `poll_deadline` (2026-11-12) moved 31 rows in the 30-day register without changing the totals: `other` 30 to 3 (27 rows), `poll_deadline` 3 to 0, and `node_busy` +8, `node_unreachable` +6 (3 of them from `poll_deadline`), `failed_verification` +4 (5 compose rows in, 1 audio row out), `abstention` +7 (3 low classify confidence, 4 browse `blocked`), `config` +3, `bad_input` +2. The rules are the ones listed below and the two `RefileStamped` rules named in the section below; each needs the writer's own words, so no other row moves. Rows written before the rules are corrected when the register is read, as before. ### The fields @@ -299,6 +299,7 @@ real sentence in the source ties to an existing class without doubt; the sentenc - `dispatch : ... dial tcp ...` (a dispatch whose connection could not be made), ` consecutive poll failures, last: ...` behind `: job : ` and `node rendered the job but fetching it failed: ...` are `node_unreachable`. The last one is the doubtful rule: the render succeeded and the fetch dropped, so `infrastructure` is the other reading. +- `browse: blocked:` (the model or the loop reported no progress) is `abstention`: the loop ended with no usable answer, and why the page could not be driven is not on the row. It is matched on the door's own sentence, not as an error class. - `media convert: convert: in and out are required` is `bad_input`, and `reasoning tier: low classify confidence ...` (the model's confidence in a classify answer under the minimum) is `abstention`. Only these whole sentences are named: `media : ` also wraps what ffmpeg reported, and `reasoning tier: ` wraps other verifier wordings. @@ -311,8 +312,7 @@ is no longer left to read time. A deferred row whose writer gave no reason is fi `deferred: no reason recorded`, which the register reads as `other`. Not yet ruled, because the sentence does not tie to one class without doubt: a bare `node leased`, `no fleet node can take this edit now`, the browse door's -`blocked` (the model or the loop reported no progress; 4 rows, proposed as `abstention` but the cause is not settled) and its other error classes, and the other -wordings behind `reasoning tier: `. Three `generate_svg` rows carry no reason, no reason code and no door, and cannot be ruled on. +other error classes, and the other wordings behind `reasoning tier: `. Three `generate_svg` rows carry no reason, no reason code and no door, and cannot be ruled on. ### Wire defer classes diff --git a/internal/ledger/failures.go b/internal/ledger/failures.go index 9909bca59..dd7d0d324 100644 --- a/internal/ledger/failures.go +++ b/internal/ledger/failures.go @@ -249,9 +249,8 @@ func failureClassOf(e Entry) (class string, derived, failed bool) { if IsReasonCode(e.ReasonCode) && e.ReasonCode != ReasonOK && e.ReasonCode != ReasonStarted { // "other" is the residue Record files when no member named the row at write time, so it is the one stamp a // later build may read better: a deferred row stamped "other" is classed again, and keeps "other" unless a - // sentence the harness wrote now names a member. A stamp that was a known misfiling (RefileStamped: a lease - // hold filed infrastructure, a cancellation filed budget, a node's folder fault filed dispatch_refused, all - // until 2026-11-12) reads as the class it should have carried. Every other stamp is read as stamped. + // sentence the harness wrote now names a member. A stamp that was a known misfiling (the stamp and sentence + // pairs RefileStamped lists) reads as the class it should have carried. Every other stamp is read as stamped. if e.ReasonCode == ReasonOther && e.Deferred { if c := DeferCode(e.ErrClass, e.Reason); c != ReasonOther { return c, true, true diff --git a/internal/ledger/reasoncode.go b/internal/ledger/reasoncode.go index 676dbc00b..fd476cd8d 100644 --- a/internal/ledger/reasoncode.go +++ b/internal/ledger/reasoncode.go @@ -227,11 +227,14 @@ var reasonPrefixCodes = []struct{ prefix, code string }{ {"unknown image family", ReasonContract}, {"unknown edit family", ReasonContract}, {"imagegen_families[", ReasonConfig}, {"probe:", ReasonInfrastructure}, {"generate_svg: write:", ReasonInfrastructure}, {"generate_svg:", ReasonInfrastructure}, // The browse door's own typed errors (pipeline/browse.go: "browse: : "; docs/systems/browse-lane.md), - // the four that name one member: no browser or no decision endpoint to drive it is the stack (as lane_unavailable), - // a page outside allow_hosts is the request's, and max_actions spent is a limit the call carried. The rest of the - // door's classes ("blocked", where the model reported it cannot proceed, and the others) are not named here. + // the five that name one member: no browser or no decision endpoint to drive it is the stack (as lane_unavailable), + // a page outside allow_hosts is the request's, max_actions spent is a limit the call carried, and "blocked" (the model + // or the loop reported no progress) is a loop that ended with no usable answer, abstention; why the page could not be + // driven is not on the row. "blocked" is matched on the door's own sentence and is not an errClassCodes entry: the + // word means other things elsewhere. The door's other classes are not named here. {"browse: BROWSER_UNAVAILABLE:", ReasonInfrastructure}, {"browse: DECISION_UNAVAILABLE:", ReasonInfrastructure}, {"browse: HOST_NOT_ALLOWED:", ReasonContract}, {"browse: budget:", ReasonBudget}, + {"browse: blocked:", ReasonAbstention}, } // legacyReasonPrefixCodes classify a deferred row by the sentence the harness ITSELF wrote, for the rows diff --git a/internal/ledger/rootcause.go b/internal/ledger/rootcause.go index c9543e123..353779fae 100644 --- a/internal/ledger/rootcause.go +++ b/internal/ledger/rootcause.go @@ -76,7 +76,7 @@ var failureCauses = map[string]FailureCause{ ReasonFailedVerification: {Kind: CauseHarnessDefect, Investigated: true, RootCause: "Mostly the research lane's own acceptance checks rejecting answers that were right or honest, not seats failing. Of the 357 rows (346 from research subtasks, whose checks the lane writes itself), 307 are those checks rejecting an answer that was right or honest: 256 fail a page-word tripwire that is graded against the loop's final prose and not against the structured digest, and 51 fail a derived `min_items` check on the first required array. Not every one of the 307 is shown to be a false rejection. 202 rows are established by comparing the digest to the page: 188 tripwire rows whose quotes and figures are found verbatim in the page, and 14 `min_items` rows on pages that were empty shells, login walls or placeholders. 105 rest on the seat's own statement that the page lacks the content (68 tripwire, 37 `min_items`) and were not checked against the page. The other 50 are 10 seat misses (it could not read or complete the page, found 1 of 3 items on a long report, returned an empty digest, did not echo an id, or left a status empty), 4 caller regexes that were over-escaped, 1 misfile (an audio row, see config) and 35 undetermined, because the ledger does not keep the answer or the kind of check. The class also reads 5 compose rows (error class `check_failed`: a composition's own runtime or layout checks failed, from one caller iterating on one composition; filed `other` before), with `bad_input` as the other reading since the composition is what the caller wrote.", Evidence: "research/anchors.go AnchorCheck, graded by core.AcceptanceCheck.Eval and evalText; delegate/run.go (failed verification); 357 rows in 30 days as stamped, 361 once the compose `check_failed` rows are read here (5 added, the audio row left). By door: research 345 and the research CLI 1, agent_delegate 8, delegate CLI 2, audio CLI 1; by seat: a 9B-class seat on three nodes 211, a 35B seat 82, the agent-pool seat 58, a 27B seat 5; 264 fall on five days (2026-10-19, -20, -23, -24, -25). Check shapes, counted apart from the causes above (the 282 tripwire rows include 26 that are seat misses or undetermined, and the tripwire and `min_items` shapes share 6 rows): docanchor regex 282, derived `min_items::1` with 0 items 70, `min_items` with 1 item against a wanted 3 on agent_delegate 2, a caller regex naming a drive path 4, `contains:` 2, a caller regex of three alternatives 1, `nonempty:status` 1, audio dead_air 1. Why the tripwire is the suspect: none of the 282 anchor sets appears in the graded prose and 5 of 282 appear in the structured digest; the 13 digests with the lowest page overlap were read and none was about another document; widening the honest-absence words (delegate/absence.go HonestAbsence) rescues 10 of 282; the failure rate among research pages that reached a verdict is 7.0% (345 against 4,604 passed) and does not step down across delegator builds (3 to 14%). Of the tripwire first attempts retried on another node, 17 ended ok, 11 failed again and 2 ended in budget. Measured 2026-11-12 by rebuilding the register's class rule outside the binary and joining the rows to the delegation log", - Action: "stop deriving a pass/fail `min_items::1` for research subtasks (excuse it when the digest says the page has none, as the existing empty-digest excuse does); do not dispatch near-empty pages (a minimum-prose floor in research.Build); grade the tripwire against the digest too, passing when its quotes, figures or titles are found verbatim in the page (research/quotes.go VerifyQuotes exists), which would have passed at least 188 of the 282; and record the answer and the acceptance kind in the ledger so the 35 undetermined rows can be settled"}, + Action: "the research acceptance is fixed in the same release (measured 2026-11-12 by an offline replay of the saved pages and answers through research.Build and delegate.EvalAcceptanceGraded, read-only; the original code reproduced every saved verdict, 356 fail and 4,622 pass). Three changes: a digest that quotes its page is excused from the page-word tripwire (delegate.DigestGrounded: one string of 60 or more characters, or two different ones of 15 or more, found in the contract's documents and not in the goal); research.Build derives a `nonempty:` check over the caller's required fields, any one of them filled, in place of `min_items::1`; and an HTML page under 150 characters of text is refused before dispatch as a skipped source (`error_kind` `empty_page`, no ledger row). Of the 356 failed rows with a job id, 167 now pass (125 tripwire failures by quoting the page, 42 item-count failures), 16 are refused as pages with no content and 173 still fail, 10 of them not research. Of the 4,622 passes, 4,513 still pass, 109 are refused as pages with no content and none fails. Rows already written keep their class, so the counts above and the register's 30-day count do not change; only rows written after the release are affected. Still open: the 173 (163 research rows: 67 digests that say the page lacks what was asked but also make a claim, 62 that hold strings of the page but under the grounding thresholds, 23 residual, 6 item-count rows silent in every required field, 3 where the seat could not read, 2 all-empty digests; plus the 10 that are not research), and recording the answer and the acceptance kind on the ledger row so the 35 undetermined rows can be settled. Known limit: a seat that says none for a page that holds the items is believed, now under a caller's own schema too; a caller who needs items writes `min_items:` in their own acceptance"}, ReasonQueueFull: {Kind: CauseCapacity, Investigated: true, RootCause: "Every node asked refused the dispatch for room: its queue was at fleet_max_queue_depth (the sampled bodies read `queue full (N jobs accepted+running, limit N)`) or its card was leased. The delegator files every 503 or 429 so.", Evidence: "delegate/run_capacity.go capacityRefusal; 49 rows in 30 days (of the three commonest reason groups, 22 rows read queue full and 1 node leased)", @@ -148,8 +148,8 @@ var failureCauses = map[string]FailureCause{ Evidence: "core/calldeadline.go, pipeline/liveness.go:411, pipeline/agenttask.go:1295; 250 rows in 30 days as stamped, 241 from research subtasks; 17 of them are loop cancellations that ledger RefileStamped now reads as canceled", Action: "give long research subtasks a longer call deadline or fewer subtasks per call; a run that ends on the step budget needs a larger max_steps."}, ReasonAbstention: {Kind: CauseNodeConfig, Investigated: true, - RootCause: "The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call. The reasoning tier's `reasoning tier: low classify confidence ...` (the model's own confidence in a classify answer under the minimum, 3 rows, filed `other` before) is read here too.", - Evidence: "pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days as stamped, 107 on one vision seat's questions, 224 once the classify confidence defers are read (pipeline/pipeline_attempt.go)", + RootCause: "The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call. The reasoning tier's `reasoning tier: low classify confidence ...` (the model's own confidence in a classify answer under the minimum, 3 rows, filed `other` before) is read here too, as is the browse door's `browse: blocked:` (the model or the loop reported no progress, 4 rows, filed `other` before; why the page could not be driven is not on the row).", + Evidence: "pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days as stamped, 107 on one vision seat's questions, 228 once the classify confidence defers and the browse `blocked` rows are read (pipeline/pipeline_attempt.go, pipeline/browse.go)", Action: "raise the vision lane's output cap for that seat, or ask for shorter answers; not investigated: why those answers run past the cap."}, ReasonContract: {Kind: CauseCallerContract, Investigated: true, RootCause: "The caller named something the node does not serve: an image family the node does not carry.", @@ -169,7 +169,7 @@ var failureCauses = map[string]FailureCause{ Evidence: "pipeline (image load) and compose_video BAD_INPUT and LINT_ERRORS; 2 rows in 30 days as stamped, 4 once the two added rules are read (1 media convert, 1 lint)", Action: "none: working as designed. The caller fixes the input."}, ReasonOther: {Kind: CauseUnknown, Investigated: true, - RootCause: "The class is the residue no rule names, a bundle of misfiled sentences and not one cause; it stays kind unknown on purpose. Of the 55 rows as stamped, 25 are read by the rules of 2026-11-12 that came first (a node's `a text lease holds its card; it may take the call later`, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors), which leaves 30. 23 of those 30 are lane refusals and transport faults whose sentences or error classes nobody had mapped, and each is now settled by a sentence or error-class rule: `no node that answered is called ...` (8, node_busy), a dispatch whose connection could not be made (1), `N consecutive poll failures` (1) and `rendered the job but fetching it failed` (1), all node_unreachable; error class `no_node_configured` (2, config); `media convert: convert: in and out are required` (1) and error class `lint_errors` (1), bad_input; `reasoning tier: low classify confidence ...` (3, abstention); and the compose error class `check_failed` (5, failed_verification, with bad_input as the other reading, since the composition is what the caller wrote). 7 rows are left: 4 of the browse door's `blocked` (the model or the loop reported no progress, 9 to 18 s), which is proposed as abstention but whose cause is not settled and so has no rule, and 3 generate_svg rows with no reason, no reason code and no door, which cannot be traced.", - Evidence: "reasoncode.go DeferCode (the residue) and errClassCodes; 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the first rules are read, 7 once the rules above are read. Two further rows of those error classes are measurement runs, which the register leaves out", - Action: "read the 4 browse `blocked` rows against the pages they drove before ruling on them; the 3 rows with no reason need the writer to record one. Not changed here: the other verifier wordings behind `reasoning tier: ...` stay in other until a row shows them."}, + RootCause: "The class is the residue no rule names, a bundle of misfiled sentences and not one cause; it stays kind unknown on purpose. Of the 55 rows as stamped, 25 are read by the rules of 2026-11-12 that came first (a node's `a text lease holds its card; it may take the call later`, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors), which leaves 30. 27 of those 30 are lane refusals and transport faults whose sentences or error classes nobody had mapped, and each is now settled by a sentence or error-class rule: `no node that answered is called ...` (8, node_busy), a dispatch whose connection could not be made (1), `N consecutive poll failures` (1) and `rendered the job but fetching it failed` (1), all node_unreachable; error class `no_node_configured` (2, config); `media convert: convert: in and out are required` (1) and error class `lint_errors` (1), bad_input; `reasoning tier: low classify confidence ...` (3, abstention); the compose error class `check_failed` (5, failed_verification, with bad_input as the other reading, since the composition is what the caller wrote); and the browse door's `browse: blocked: the model or the loop reported no progress` (4, abstention: the loop ended with no usable answer; why the page could not be driven is not on the row). 3 rows are left: generate_svg rows with no reason, no reason code and no door, which cannot be traced.", + Evidence: "reasoncode.go DeferCode (the residue) and errClassCodes; 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the first rules are read, 3 once the rules above are read. Two further rows of those error classes are measurement runs, which the register leaves out", + Action: "the 3 rows with no reason need the writer to record one; why the browse door could not drive its page is a separate question the 4 `blocked` rows cannot answer. Not changed here: the other verifier wordings behind `reasoning tier: ...` stay in other until a row shows them."}, } diff --git a/testdata/failure-register/expected.json b/testdata/failure-register/expected.json index bf29b08aa..083a609b6 100644 --- a/testdata/failure-register/expected.json +++ b/testdata/failure-register/expected.json @@ -191,8 +191,8 @@ "structural_fix": "open", "cause": { "kind": "node configuration", - "root_cause": "The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call. The reasoning tier's `reasoning tier: low classify confidence ...` (the model's own confidence in a classify answer under the minimum, 3 rows, filed `other` before) is read here too.", - "evidence": "pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days as stamped, 107 on one vision seat's questions, 224 once the classify confidence defers are read (pipeline/pipeline_attempt.go)", + "root_cause": "The answer was cut at its completion token cap, so there was nothing usable to return: the vision lane's `vision output truncated` (117 rows) and an agent answer cut at the completion budget so the re-pack was skipped (about 94 rows, `output_truncated`). The cap is a setting of the seat or the call. The reasoning tier's `reasoning tier: low classify confidence ...` (the model's own confidence in a classify answer under the minimum, 3 rows, filed `other` before) is read here too, as is the browse door's `browse: blocked:` (the model or the loop reported no progress, 4 rows, filed `other` before; why the page could not be driven is not on the row).", + "evidence": "pipeline/pipeline_vision.go:315 and pipeline/agentlistcap.go:75; 221 rows in 30 days as stamped, 107 on one vision seat's questions, 228 once the classify confidence defers and the browse `blocked` rows are read (pipeline/pipeline_attempt.go, pipeline/browse.go)", "action": "raise the vision lane's output cap for that seat, or ask for shorter answers; not investigated: why those answers run past the cap.", "investigated": true } @@ -252,7 +252,7 @@ "kind": "harness defect", "root_cause": "Mostly the research lane's own acceptance checks rejecting answers that were right or honest, not seats failing. Of the 357 rows (346 from research subtasks, whose checks the lane writes itself), 307 are those checks rejecting an answer that was right or honest: 256 fail a page-word tripwire that is graded against the loop's final prose and not against the structured digest, and 51 fail a derived `min_items` check on the first required array. Not every one of the 307 is shown to be a false rejection. 202 rows are established by comparing the digest to the page: 188 tripwire rows whose quotes and figures are found verbatim in the page, and 14 `min_items` rows on pages that were empty shells, login walls or placeholders. 105 rest on the seat's own statement that the page lacks the content (68 tripwire, 37 `min_items`) and were not checked against the page. The other 50 are 10 seat misses (it could not read or complete the page, found 1 of 3 items on a long report, returned an empty digest, did not echo an id, or left a status empty), 4 caller regexes that were over-escaped, 1 misfile (an audio row, see config) and 35 undetermined, because the ledger does not keep the answer or the kind of check. The class also reads 5 compose rows (error class `check_failed`: a composition's own runtime or layout checks failed, from one caller iterating on one composition; filed `other` before), with `bad_input` as the other reading since the composition is what the caller wrote.", "evidence": "research/anchors.go AnchorCheck, graded by core.AcceptanceCheck.Eval and evalText; delegate/run.go (failed verification); 357 rows in 30 days as stamped, 361 once the compose `check_failed` rows are read here (5 added, the audio row left). By door: research 345 and the research CLI 1, agent_delegate 8, delegate CLI 2, audio CLI 1; by seat: a 9B-class seat on three nodes 211, a 35B seat 82, the agent-pool seat 58, a 27B seat 5; 264 fall on five days (2026-10-19, -20, -23, -24, -25). Check shapes, counted apart from the causes above (the 282 tripwire rows include 26 that are seat misses or undetermined, and the tripwire and `min_items` shapes share 6 rows): docanchor regex 282, derived `min_items:\u003cfield\u003e:1` with 0 items 70, `min_items` with 1 item against a wanted 3 on agent_delegate 2, a caller regex naming a drive path 4, `contains:\u003chex\u003e` 2, a caller regex of three alternatives 1, `nonempty:status` 1, audio dead_air 1. Why the tripwire is the suspect: none of the 282 anchor sets appears in the graded prose and 5 of 282 appear in the structured digest; the 13 digests with the lowest page overlap were read and none was about another document; widening the honest-absence words (delegate/absence.go HonestAbsence) rescues 10 of 282; the failure rate among research pages that reached a verdict is 7.0% (345 against 4,604 passed) and does not step down across delegator builds (3 to 14%). Of the tripwire first attempts retried on another node, 17 ended ok, 11 failed again and 2 ended in budget. Measured 2026-11-12 by rebuilding the register's class rule outside the binary and joining the rows to the delegation log", - "action": "stop deriving a pass/fail `min_items:\u003cfirst required array\u003e:1` for research subtasks (excuse it when the digest says the page has none, as the existing empty-digest excuse does); do not dispatch near-empty pages (a minimum-prose floor in research.Build); grade the tripwire against the digest too, passing when its quotes, figures or titles are found verbatim in the page (research/quotes.go VerifyQuotes exists), which would have passed at least 188 of the 282; and record the answer and the acceptance kind in the ledger so the 35 undetermined rows can be settled", + "action": "the research acceptance is fixed in the same release (measured 2026-11-12 by an offline replay of the saved pages and answers through research.Build and delegate.EvalAcceptanceGraded, read-only; the original code reproduced every saved verdict, 356 fail and 4,622 pass). Three changes: a digest that quotes its page is excused from the page-word tripwire (delegate.DigestGrounded: one string of 60 or more characters, or two different ones of 15 or more, found in the contract's documents and not in the goal); research.Build derives a `nonempty:` check over the caller's required fields, any one of them filled, in place of `min_items:\u003cfirst required array\u003e:1`; and an HTML page under 150 characters of text is refused before dispatch as a skipped source (`error_kind` `empty_page`, no ledger row). Of the 356 failed rows with a job id, 167 now pass (125 tripwire failures by quoting the page, 42 item-count failures), 16 are refused as pages with no content and 173 still fail, 10 of them not research. Of the 4,622 passes, 4,513 still pass, 109 are refused as pages with no content and none fails. Rows already written keep their class, so the counts above and the register's 30-day count do not change; only rows written after the release are affected. Still open: the 173 (163 research rows: 67 digests that say the page lacks what was asked but also make a claim, 62 that hold strings of the page but under the grounding thresholds, 23 residual, 6 item-count rows silent in every required field, 3 where the seat could not read, 2 all-empty digests; plus the 10 that are not research), and recording the answer and the acceptance kind on the ledger row so the 35 undetermined rows can be settled. Known limit: a seat that says none for a page that holds the items is believed, now under a caller's own schema too; a caller who needs items writes `min_items:` in their own acceptance", "investigated": true } }, @@ -490,9 +490,9 @@ "structural_fix": "undocumented", "cause": { "kind": "unknown", - "root_cause": "The class is the residue no rule names, a bundle of misfiled sentences and not one cause; it stays kind unknown on purpose. Of the 55 rows as stamped, 25 are read by the rules of 2026-11-12 that came first (a node's `a text lease holds its card; it may take the call later`, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors), which leaves 30. 23 of those 30 are lane refusals and transport faults whose sentences or error classes nobody had mapped, and each is now settled by a sentence or error-class rule: `no node that answered is called ...` (8, node_busy), a dispatch whose connection could not be made (1), `N consecutive poll failures` (1) and `rendered the job but fetching it failed` (1), all node_unreachable; error class `no_node_configured` (2, config); `media convert: convert: in and out are required` (1) and error class `lint_errors` (1), bad_input; `reasoning tier: low classify confidence ...` (3, abstention); and the compose error class `check_failed` (5, failed_verification, with bad_input as the other reading, since the composition is what the caller wrote). 7 rows are left: 4 of the browse door's `blocked` (the model or the loop reported no progress, 9 to 18 s), which is proposed as abstention but whose cause is not settled and so has no rule, and 3 generate_svg rows with no reason, no reason code and no door, which cannot be traced.", - "evidence": "reasoncode.go DeferCode (the residue) and errClassCodes; 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the first rules are read, 7 once the rules above are read. Two further rows of those error classes are measurement runs, which the register leaves out", - "action": "read the 4 browse `blocked` rows against the pages they drove before ruling on them; the 3 rows with no reason need the writer to record one. Not changed here: the other verifier wordings behind `reasoning tier: ...` stay in other until a row shows them.", + "root_cause": "The class is the residue no rule names, a bundle of misfiled sentences and not one cause; it stays kind unknown on purpose. Of the 55 rows as stamped, 25 are read by the rules of 2026-11-12 that came first (a node's `a text lease holds its card; it may take the call later`, and the browse door's `BROWSER_UNAVAILABLE`, `DECISION_UNAVAILABLE`, `HOST_NOT_ALLOWED` and `budget` errors), which leaves 30. 27 of those 30 are lane refusals and transport faults whose sentences or error classes nobody had mapped, and each is now settled by a sentence or error-class rule: `no node that answered is called ...` (8, node_busy), a dispatch whose connection could not be made (1), `N consecutive poll failures` (1) and `rendered the job but fetching it failed` (1), all node_unreachable; error class `no_node_configured` (2, config); `media convert: convert: in and out are required` (1) and error class `lint_errors` (1), bad_input; `reasoning tier: low classify confidence ...` (3, abstention); the compose error class `check_failed` (5, failed_verification, with bad_input as the other reading, since the composition is what the caller wrote); and the browse door's `browse: blocked: the model or the loop reported no progress` (4, abstention: the loop ended with no usable answer; why the page could not be driven is not on the row). 3 rows are left: generate_svg rows with no reason, no reason code and no door, which cannot be traced.", + "evidence": "reasoncode.go DeferCode (the residue) and errClassCodes; 55 rows in 30 days as stamped (audio 18, browse 15, video 7, compose 6), 30 once the first rules are read, 3 once the rules above are read. Two further rows of those error classes are measurement runs, which the register leaves out", + "action": "the 3 rows with no reason need the writer to record one; why the browse door could not drive its page is a separate question the 4 `blocked` rows cannot answer. Not changed here: the other verifier wordings behind `reasoning tier: ...` stay in other until a row shows them.", "investigated": true } },