Repository navigation
feat(incidents): relay upstream release webhooks to the repair worker - #63
Merged
Merged
Conversation
…es and stuck holds A failed route pin renamed ENABLED to PAUSED-route-unavailable and nothing renamed it back. Any run without a verified result became an uncertain hold with no time limit; in serial mode the hold also renamed ENABLED, which disabled the only bookkeeping check that could end it. Remediation then stayed dead until someone noticed. ENABLED now belongs to the operator alone and holds live in SQLite: - Failures before the model is invoked (readback, evidence files, pin, unexpected errors, restart) return the batch to the queue. A new events.invoked column is the only thing that turns an interruption into a hold. - A failed pin, or an invocation that ends within three minutes without a result, opens a route breaker that backs off 1/2/5/10/30 minutes and retries the same route. No fallback model or account. - A hold ends by receipt as before, or once its lease has expired (15 minutes after an immediate failure, two hours otherwise) and the Gateway twice shows the session inactive. Its events are abandoned, never replayed. - The bookkeeping check gets up to three attempts and is not consumed by an unavailable route. - New events wait five minutes before admission so the producer can withdraw faults caused by an upgrade in progress; withdrawn batches consume no dispatch budget. - The ACK carries an attention list (open breaker, long hold, stalled queue, errors) for out-of-band alerting. Startup restores ENABLED from a single automatic PAUSED-* marker left by an earlier release. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ation stalls Only operations started from the dashboard, or holding the dashnet journal, kept expected downtime out of the repair queue, and faults were queued the moment the operation ended. A hand-run Testnet or quorum server upgrade went straight to the repair agent. server/change-activity.js recognises planned change from the fresh probe: a recreated container (recent start, RestartCount 0) or a reboot holds that host, the network's endpoint checks and its EC2 status check; two or more hosts changing at once hold the network. A succeeded dashboard operation and a released journal claim now settle for 20 minutes; failed operations get no settle window. A crash loop is not a planned change, Mainnet is never held, and every hold is capped. Held issues stay open and are emitted with fresh evidence if the fault outlives the hold. A stalled worker cannot report its stall through the queue it is not draining, so the producer raises it out of band: unreachable receiver, stale worker heartbeat, paused dispatch with a backlog, or the receiver's attention list. After 15 minutes it posts to an optional Slack-compatible webhook (INCIDENT_ALERT_WEBHOOK_FILE), repeats every six hours and reports recovery. The remediation board shows the reasons. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…worker The remediation board was read-only, so pausing dispatch or dropping a backlog needed shell access to the receiver host. Administrators now have two controls on /remediation: - Switch off / on. Off by default: a new or upgraded receiver dispatches nothing until an administrator switches it on. Switching off starts nothing new; work in progress continues and holds are still reconciled. The host's ENABLED file stays the master switch, so the dashboard can never enable dispatch the host has disabled. - Clear queue. Marks queued events cleared without repairing them. Running and held runs are untouched; an issue that is still open returns on the producer's next revision. The web process records the request (who, when) in data/remediation-control.json behind the admin session and CSRF token. The incident sidecar forwards three fields (seq, dispatch, clearSeq) on its signed heartbeat, only to a receiver that has announced support, so an older receiver never rejects an envelope and no event is quarantined. The receiver validates that closed vocabulary, applies it inside the ingest transaction and reports what it applied; monotonic counters make redelivery and replay harmless. Switching off from the dashboard is not reported as a stall. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ll notices - An operator's own PAUSED-* file now keeps dispatch off when a legacy PAUSED-<run> marker is restored at the end of its run. - A successful bookkeeping pin closes a breaker that a failed pin opened, so a route-unavailable notice does not linger with an empty queue. - A journal claim holds its network only through the bounded activity result: a runner that crashed without releasing the claim no longer silences the network indefinitely. - A journal that records an error ends rolling inference for that operation, so the hosts it recreated do not keep the whole network held. - Held instance IDs are collected before the network freshness exits, so a collector gap does not release a held host's EC2 status check. - Endpoint issues held because one host is changing say so instead of naming a network operation. - A failed recovery notice is retried until delivered (for at most the repeat interval); a stall that begins meanwhile starts afresh. - The board keeps showing why remediation cannot progress when the worker's own status is stale, which is when it matters most. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Core 24 changed wallet behaviour the deployment tooling relied on, and nothing told the worker a new version existed until a deployment failed. The only release webhook (on dashpay/platform) drives the CI release builders; Core and Tenderdash had none, and the console's release hint ignores new majors by design. The worker host has no public ingress, so the status server relays: - POST /api/releases/github-webhook verifies X-Hub-Signature-256 with RELEASE_WEBHOOK_SECRET and accepts published, released and prereleased events for dashpay/dash, dashpay/platform and dashpay/tenderdash. It records repository, tag, pre-release flag and a release URL it builds itself. Titles and release notes are never stored or forwarded. - The incident sidecar turns each record into a release_published event (domain release) in the signed, durable outbox, sent only to a receiver that announces the domain. An attempt the worker abandoned is re-sent up to three times, an hour apart; a cleared notice is not. - The receiver accepts the release domain and dispatches these notices without the fault readback. The policy makes them a read-only compatibility assessment of the three components as a set, and authorizes no upgrade or deployment. - /remediation lists recent releases with the outcome. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 52 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Review found three defects. Lock release. A hold past its lease was released when the parent session looked idle. An idle parent proves nothing about children or remote commands underneath it: a receipt with pendingChildren 1 was abandoned and its locks freed, so another repair could overlap that work. Idleness no longer releases anything. The worker snapshots the session (lastRunId, endedAt, lastActivityAt) immediately before invoking the agent. Past the lease, a hold returns to the queue only when the Gateway twice shows the session inactive and unchanged since that snapshot, with no receipt file and no runtime reply in the CLI output: positive evidence that no model turn ran, so nothing was started. That still recovers from an unavailable account. Once a turn has run, only a terminal receipt declaring no pending work releases the locks; a pending-children receipt, a changed session, a missing snapshot, an active session and an unreadable Gateway all keep the hold, which is reported after 30 minutes. The abandoned state is gone: an attempt that never ran is queued again, since nothing was played. Crash loops. A clean container start opened a maintenance hold that survived for the rest of the settle window even after the container began crash-looping. A restart loop seen in a fresh probe now ends that host's hold at once. Stall notices. The notice repeated every six hours and a brief recovery reset its history. It is now sent at most twice per stall: once after 15 minutes and once when the stall is over, which requires it to stay clear for 15 minutes. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Review found that trimming the release history erased deduplication state: with 301 recent releases an old event ID was emitted again with different contents, which the receiver rejects as a conflicting ID. Two changes remove the failure rather than raise the limit: - The event is built only from the release (repository, tag, pre-release flag and GitHub's publication time), never from the clock or local state. The same ID always carries the same bytes, so a redelivered webhook, a restart or lost sidecar state can only repeat the identical event, which the receiver accepts as a duplicate. A webhook without a usable publication time is refused. - Delivery identity (releaseAnnounced) is kept apart from the display list (releases). It covers every release whose record still exists and is never trimmed for the board. The sidecar's re-send of abandoned attempts is removed with the abandoned state: the receiver now queues an attempt that never ran again by itself, and a turn that ran ends with a receipt. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…eports The previous commit released a hold only when the session was unchanged since a snapshot taken before the attempt. Reading the installed Gateway (OpenClaw 2026.11.3) shows that is never true for the case it was meant for: when the account is unavailable the invocation still registers a run. The CLI prints ok false with a cli_error, and the session gets a new lastRunId, status failed and a runtime of about 50 ms. Only lastActivityAt stays where it was. Two such attempts hold slots on the production broker today. The Gateway also reports directly what the review asked about: whether a run is executing and whether child sessions exist. A hold now returns to the queue only when, past its lease and on two observations, the Gateway states hasActiveRun false, activeRunIds empty and childOwnerSessionKeys empty, lastActivityAt shows no activity since the attempt was claimed (and equals the pre-invocation snapshot where one exists), there is no receipt file, and the CLI output reports no turn. A Gateway that does not report those fields releases nothing. Activity after the claim, a child session, an executing run, a turn reported by the CLI and any receipt file all keep the hold. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…the repository Production has run receiver and status-server code since October that was never pushed here. Deploying this branch as it stood would have removed it. This merges that code (branch feat/mainnet-services-20261030 for the receiver, deploy/latest-20261103T2240 for the status server) with the changes on this branch, so nothing the running system does is lost. From production, unchanged in behaviour: - maintenance domain with the platform_release_compatibility task: signed GitHub release webhook for dashpay/platform, durable release inbox, code-and-tests-only instructions, version-bound compatibility proof - known CI blocker correlation before admission (status deduplicated) - separate dispatch budgets for ordinary and compatibility work, with an audited local reset - up to three lanes, killed/error sessions eligible for bookkeeping, release only on status done, stalled-progress health - board: compatibility cases, stalled execution, blocked stage Reconciled with this branch: - completion checks use production's completion_checks table and schedule (three attempts: at once, after 15 minutes, after an hour). This branch's attempt counter is gone. The route is still pinned before an attempt is recorded, and ownership is checked before the pin. - claim() takes both the administrator switch and the CI correlation - the release-compatibility instructions move into prepare() - holds, route failures and legacy PAUSED markers keep this branch's behaviour: ENABLED is never renamed by the receiver Left out on purpose: Mainnet service and Evo observation, devnet key access, AWS cost privacy and other production overlays unrelated to remediation. They remain unpublished. Co-authored-by: infraclaw-dash <283232465+infraclaw-dash@users.noreply.github.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Production already has a release-compatibility task for dashpay/platform, now in the base branch. This branch had added a second mechanism beside it (a `release` domain, `release_published`, /api/releases/github-webhook and a releases panel). That is withdrawn here: the tree is the base branch's, and the next commit extends the existing task to Dash Core and Tenderdash instead. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A cleared fault returns on the next monitoring update if it is still open. A release compatibility task is delivered once and never raised again, so clearing it lost the work for good. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… Tenderdash A network is built from Core, Platform and Tenderdash, but only Platform releases produced a compatibility task. Core 24 changed wallet behaviour the deployment tooling relied on and nothing checked it until a deployment failed. The existing mechanism now covers all three repositories, each bound to its fixed GitHub ID and its own issue code. Platform keeps its code, file names and issue identity, so earlier tasks and receipts stay valid. - one signed route, /api/webhooks/github/release; the original /api/webhooks/github/platform-release is the same handler - Core and Tenderdash tasks are sent only to a receiver that announces releaseRepositories, so an older receiver never rejects a batch - worker instructions are built locally from the component table, tell the worker to test the three components as a set and to keep the release lines still deployed working - Core and Tenderdash proof names its repository (releaseTag, releaseCommit); Platform proof is unchanged and may use either form Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… loop is never held by a rolling change Two findings from review, both reproduced by the new tests against the previous code. The dispatch switch. The receiver keeps the highest sequence it applied and ignores anything not above it. When remediation-control.json was lost or damaged its numbering restarted at zero, so the OFF it then meant, and every OFF pressed afterwards, was discarded as stale and the receiver stayed on. The file is now read against the control block of the receiver's last acknowledgement. When the receiver is ahead, the request is OFF one above the receiver's sequence, and the next switch or clear is numbered above it too. No clear is inferred: the clear counter sent is the one the receiver already applied. The board says the saved setting was lost. The sidecar still never writes the file, which it has read-only; the receiver is unchanged. Recognised change. A host in a crash loop already lost its own hold, but two other hosts restarting cleanly held the whole network by rolling inference, and that hold covered the crashing host's fault and its EC2 status check. Activity recognition now names crashing hosts, and a network hold that is only inferred from other hosts no longer covers them. A dashboard operation or a journal claim still holds every host. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
ktechmidas
changed the base branch from
fix/remediation-self-recovery
to
master
November 15, 2026 20:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #62 (base branch
fix/remediation-self-recovery), which it depends on. The change here is the last commit; the two merges before it bring in the base and withdraw this PR's first approach.What changed since the first version
The first version added a second release mechanism (a
releasedomain,release_published,/api/releases/github-webhook, a releases panel) without knowing production already had one fordashpay/platform. That is withdrawn. Now that the production task is in the base branch (#62), this PR extends it to the two repositories it did not cover.Problem
A network is built from Dash Core, Platform and Tenderdash, but only Platform releases produced a compatibility task. Core 24 changed wallet behaviour the deployment tooling relied on, and nothing checked it until a deployment failed.
Change
dashpay/dashanddashpay/tenderdashjoindashpay/platform, each bound to its fixed GitHub ID and its own issue code (core_release_compatibility,tenderdash_release_compatibility). Platform keeps its code, file names and issue identity, so tasks and receipts recorded before this stay valid.POST /api/webhooks/github/releaseaccepts all three. The original/api/webhooks/github/platform-releaseis the same handler and keeps working, so the existing Platform webhook needs no change. A payload whose repository name and ID do not both match the table is refused.releaseRepositoriesin its acknowledgement; the sidecar creates Core and Tenderdash tasks only then. Until then their records wait in the inbox. Platform never waits.repository,releaseTagandreleaseCommit; Platform receipts keepplatformTagandplatformCommitand may use either form. A Platform-form receipt never proves a Core or Tenderdash release.publishedis acted on, a redelivered or later-promoted release creates nothing new, and all three share themaintenance:dash-network-goscope (one task per attempt, one admission an hour, four a day).deploy/INCIDENTS.mddocuments the mechanism, which was undocumented.Deploying
dashpay/dashanddashpay/tenderdashexist and point at/api/webhooks/github/release. Until this is deployed their deliveries get 404. They were created with a new secret, not the one production holds inGITHUB_RELEASE_WEBHOOK_SECRET_FILE, so after deployment deliveries are refused with 401 until one of these is done:Not covered
dash-network-gostill pins one Core version in CI.Testing
Run locally on Windows (Python 3.12, Node 22):
deploy/incident-broker: 65 passed (3 added: the three sources and what is refused, the instructions per component, the proof forms).node --test: 211 tests, 195 passed, 11 skipped, 5 failed, with one call shimmed: the release inbox fsyncs a directory, which Windows does not permit. The 5 fail on unmodifiedmasteron that machine too. Without the shim the release-inbox tests fail there as well, so CI on Linux is the reference. 3 tests added.eslint .clean,vite buildsucceeds.🤖 Generated with Claude Code