Skip to content

feat(incidents): relay upstream release webhooks to the repair worker - #63

Merged
ktechmidas merged 20 commits into
masterfrom
feat/release-webhook
Nov 15, 2026
Merged

ktechmidas merged 20 commits into
masterfrom
feat/release-webhook

Conversation

@ktechmidas

@ktechmidas ktechmidas commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #62 (base branch fix/remediation-self-recovery), which it depends on. The change here is the last commit; the two merges before it bring in the base and withdraw this PR's first approach.

What changed since the first version

The first version added a second release mechanism (a release domain, release_published, /api/releases/github-webhook, a releases panel) without knowing production already had one for dashpay/platform. That is withdrawn. Now that the production task is in the base branch (#62), this PR extends it to the two repositories it did not cover.

Problem

A network is built from Dash Core, Platform and Tenderdash, but only Platform releases produced a compatibility task. Core 24 changed wallet behaviour the deployment tooling relied on, and nothing checked it until a deployment failed.

Change

  • Three repositories, one mechanism. dashpay/dash and dashpay/tenderdash join dashpay/platform, each bound to its fixed GitHub ID and its own issue code (core_release_compatibility, tenderdash_release_compatibility). Platform keeps its code, file names and issue identity, so tasks and receipts recorded before this stay valid.
  • One signed route. POST /api/webhooks/github/release accepts all three. The original /api/webhooks/github/platform-release is the same handler and keeps working, so the existing Platform webhook needs no change. A payload whose repository name and ID do not both match the table is refused.
  • An older receiver is never sent what it would reject. The receiver announces releaseRepositories in its acknowledgement; the sidecar creates Core and Tenderdash tasks only then. Until then their records wait in the inbox. Platform never waits.
  • Worker instructions are built locally from the component table, never from the event. Beyond what the Platform task already said, they tell the worker to test the release together with the other two components (the combination deployed now and the newest released one) and to keep the release lines still deployed working, for example the previous Core major while a new one is rolled out. The prohibitions are unchanged: no deploy, upgrade, version-pin change, reset, restart, publish or merge.
  • Proof names its repository. Core and Tenderdash receipts carry repository, releaseTag and releaseCommit; Platform receipts keep platformTag and platformCommit and may use either form. A Platform-form receipt never proves a Core or Tenderdash release.
  • Unchanged from production: titles and release notes are never stored or forwarded, only published is acted on, a redelivered or later-promoted release creates nothing new, and all three share the maintenance:dash-network-go scope (one task per attempt, one admission an hour, four a day).
  • deploy/INCIDENTS.md documents the mechanism, which was undocumented.

Deploying

  1. Deploy the receiver, then the status server (as for fix(incidents): self-recovering remediation, upgrade recognition and admin controls #62). Release tasks are queued like other work, so they wait while the dispatch switch from fix(incidents): self-recovering remediation, upgrade recognition and admin controls #62 is off.
  2. The webhooks on dashpay/dash and dashpay/tenderdash exist and point at /api/webhooks/github/release. Until this is deployed their deliveries get 404. They were created with a new secret, not the one production holds in GITHUB_RELEASE_WEBHOOK_SECRET_FILE, so after deployment deliveries are refused with 401 until one of these is done:
    • set those two webhooks' secret to the existing one, or
    • put the new secret in that file and set it on the existing Platform webhook too.
  3. The second Platform webhook made for the first version of this PR has been deleted; the original Platform webhook is the only one on that repository.

Not covered

Testing

Run locally on Windows (Python 3.12, Node 22):

  • deploy/incident-broker: 65 passed (3 added: the three sources and what is refused, the instructions per component, the proof forms).
  • node --test: 211 tests, 195 passed, 11 skipped, 5 failed, with one call shimmed: the release inbox fsyncs a directory, which Windows does not permit. The 5 fail on unmodified master on that machine too. Without the shim the release-inbox tests fail there as well, so CI on Linux is the reference. 3 tests added.
  • eslint . clean, vite build succeeds.
  • Not exercised: a real GitHub delivery. Nothing here has been deployed.

🤖 Generated with Claude Code

ktechmidas and others added 5 commits November 13, 2026 19:33
…es and stuck holds

A failed route pin renamed ENABLED to PAUSED-route-unavailable and nothing
renamed it back. Any run without a verified result became an uncertain
hold with no time limit; in serial mode the hold also renamed ENABLED,
which disabled the only bookkeeping check that could end it. Remediation
then stayed dead until someone noticed.

ENABLED now belongs to the operator alone and holds live in SQLite:

- Failures before the model is invoked (readback, evidence files, pin,
  unexpected errors, restart) return the batch to the queue. A new
  events.invoked column is the only thing that turns an interruption
  into a hold.
- A failed pin, or an invocation that ends within three minutes without
  a result, opens a route breaker that backs off 1/2/5/10/30 minutes and
  retries the same route. No fallback model or account.
- A hold ends by receipt as before, or once its lease has expired (15
  minutes after an immediate failure, two hours otherwise) and the
  Gateway twice shows the session inactive. Its events are abandoned,
  never replayed.
- The bookkeeping check gets up to three attempts and is not consumed by
  an unavailable route.
- New events wait five minutes before admission so the producer can
  withdraw faults caused by an upgrade in progress; withdrawn batches
  consume no dispatch budget.
- The ACK carries an attention list (open breaker, long hold, stalled
  queue, errors) for out-of-band alerting.

Startup restores ENABLED from a single automatic PAUSED-* marker left by
an earlier release.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ation stalls

Only operations started from the dashboard, or holding the dashnet
journal, kept expected downtime out of the repair queue, and faults were
queued the moment the operation ended. A hand-run Testnet or quorum
server upgrade went straight to the repair agent.

server/change-activity.js recognises planned change from the fresh probe:
a recreated container (recent start, RestartCount 0) or a reboot holds
that host, the network's endpoint checks and its EC2 status check; two
or more hosts changing at once hold the network. A succeeded dashboard
operation and a released journal claim now settle for 20 minutes; failed
operations get no settle window. A crash loop is not a planned change,
Mainnet is never held, and every hold is capped. Held issues stay open
and are emitted with fresh evidence if the fault outlives the hold.

A stalled worker cannot report its stall through the queue it is not
draining, so the producer raises it out of band: unreachable receiver,
stale worker heartbeat, paused dispatch with a backlog, or the
receiver's attention list. After 15 minutes it posts to an optional
Slack-compatible webhook (INCIDENT_ALERT_WEBHOOK_FILE), repeats every
six hours and reports recovery. The remediation board shows the reasons.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…worker

The remediation board was read-only, so pausing dispatch or dropping a
backlog needed shell access to the receiver host. Administrators now
have two controls on /remediation:

- Switch off / on. Off by default: a new or upgraded receiver dispatches
  nothing until an administrator switches it on. Switching off starts
  nothing new; work in progress continues and holds are still
  reconciled. The host's ENABLED file stays the master switch, so the
  dashboard can never enable dispatch the host has disabled.
- Clear queue. Marks queued events cleared without repairing them.
  Running and held runs are untouched; an issue that is still open
  returns on the producer's next revision.

The web process records the request (who, when) in
data/remediation-control.json behind the admin session and CSRF token.
The incident sidecar forwards three fields (seq, dispatch, clearSeq) on
its signed heartbeat, only to a receiver that has announced support, so
an older receiver never rejects an envelope and no event is quarantined.
The receiver validates that closed vocabulary, applies it inside the
ingest transaction and reports what it applied; monotonic counters make
redelivery and replay harmless. Switching off from the dashboard is not
reported as a stall.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ll notices

- An operator's own PAUSED-* file now keeps dispatch off when a legacy
  PAUSED-<run> marker is restored at the end of its run.
- A successful bookkeeping pin closes a breaker that a failed pin opened,
  so a route-unavailable notice does not linger with an empty queue.
- A journal claim holds its network only through the bounded activity
  result: a runner that crashed without releasing the claim no longer
  silences the network indefinitely.
- A journal that records an error ends rolling inference for that
  operation, so the hosts it recreated do not keep the whole network held.
- Held instance IDs are collected before the network freshness exits, so
  a collector gap does not release a held host's EC2 status check.
- Endpoint issues held because one host is changing say so instead of
  naming a network operation.
- A failed recovery notice is retried until delivered (for at most the
  repeat interval); a stall that begins meanwhile starts afresh.
- The board keeps showing why remediation cannot progress when the
  worker's own status is stale, which is when it matters most.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Core 24 changed wallet behaviour the deployment tooling relied on, and
nothing told the worker a new version existed until a deployment failed.
The only release webhook (on dashpay/platform) drives the CI release
builders; Core and Tenderdash had none, and the console's release hint
ignores new majors by design.

The worker host has no public ingress, so the status server relays:

- POST /api/releases/github-webhook verifies X-Hub-Signature-256 with
  RELEASE_WEBHOOK_SECRET and accepts published, released and prereleased
  events for dashpay/dash, dashpay/platform and dashpay/tenderdash. It
  records repository, tag, pre-release flag and a release URL it builds
  itself. Titles and release notes are never stored or forwarded.
- The incident sidecar turns each record into a release_published event
  (domain release) in the signed, durable outbox, sent only to a
  receiver that announces the domain. An attempt the worker abandoned is
  re-sent up to three times, an hour apart; a cleared notice is not.
- The receiver accepts the release domain and dispatches these notices
  without the fault readback. The policy makes them a read-only
  compatibility assessment of the three components as a set, and
  authorizes no upgrade or deployment.
- /remediation lists recent releases with the outcome.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 52 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 663807ea-d5e9-40bc-8353-8c14111cb685
Commits

Reviewing files that changed from the base of the PR and between 630ab1d and 83d4e6f.

Files selected for processing: 8

  • deploy/INCIDENTS.md
  • deploy/incident-broker/broker.py
  • deploy/incident-broker/test_broker.py
  • server/incidents.js
  • server/release-webhook.js
  • server/release-webhook.test.js
  • server/remediation.js
  • server/web.js
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

ktechmidas and others added 14 commits November 13, 2026 21:39
Review found three defects.

Lock release. A hold past its lease was released when the parent session
looked idle. An idle parent proves nothing about children or remote
commands underneath it: a receipt with pendingChildren 1 was abandoned
and its locks freed, so another repair could overlap that work.

Idleness no longer releases anything. The worker snapshots the session
(lastRunId, endedAt, lastActivityAt) immediately before invoking the
agent. Past the lease, a hold returns to the queue only when the Gateway
twice shows the session inactive and unchanged since that snapshot, with
no receipt file and no runtime reply in the CLI output: positive
evidence that no model turn ran, so nothing was started. That still
recovers from an unavailable account. Once a turn has run, only a
terminal receipt declaring no pending work releases the locks; a
pending-children receipt, a changed session, a missing snapshot, an
active session and an unreadable Gateway all keep the hold, which is
reported after 30 minutes. The abandoned state is gone: an attempt that
never ran is queued again, since nothing was played.

Crash loops. A clean container start opened a maintenance hold that
survived for the rest of the settle window even after the container
began crash-looping. A restart loop seen in a fresh probe now ends that
host's hold at once.

Stall notices. The notice repeated every six hours and a brief recovery
reset its history. It is now sent at most twice per stall: once after 15
minutes and once when the stall is over, which requires it to stay clear
for 15 minutes.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Review found that trimming the release history erased deduplication
state: with 301 recent releases an old event ID was emitted again with
different contents, which the receiver rejects as a conflicting ID.

Two changes remove the failure rather than raise the limit:

- The event is built only from the release (repository, tag,
  pre-release flag and GitHub's publication time), never from the clock
  or local state. The same ID always carries the same bytes, so a
  redelivered webhook, a restart or lost sidecar state can only repeat
  the identical event, which the receiver accepts as a duplicate. A
  webhook without a usable publication time is refused.
- Delivery identity (releaseAnnounced) is kept apart from the display
  list (releases). It covers every release whose record still exists and
  is never trimmed for the board.

The sidecar's re-send of abandoned attempts is removed with the
abandoned state: the receiver now queues an attempt that never ran
again by itself, and a turn that ran ends with a receipt.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…eports

The previous commit released a hold only when the session was unchanged
since a snapshot taken before the attempt. Reading the installed
Gateway (OpenClaw 2026.11.3) shows that is never true for the case it
was meant for: when the account is unavailable the invocation still
registers a run. The CLI prints ok false with a cli_error, and the
session gets a new lastRunId, status failed and a runtime of about
50 ms. Only lastActivityAt stays where it was. Two such attempts hold
slots on the production broker today.

The Gateway also reports directly what the review asked about: whether
a run is executing and whether child sessions exist. A hold now returns
to the queue only when, past its lease and on two observations, the
Gateway states hasActiveRun false, activeRunIds empty and
childOwnerSessionKeys empty, lastActivityAt shows no activity since the
attempt was claimed (and equals the pre-invocation snapshot where one
exists), there is no receipt file, and the CLI output reports no turn.
A Gateway that does not report those fields releases nothing. Activity
after the claim, a child session, an executing run, a turn reported by
the CLI and any receipt file all keep the hold.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…the repository

Production has run receiver and status-server code since October that was
never pushed here. Deploying this branch as it stood would have removed
it. This merges that code (branch feat/mainnet-services-20261030 for the
receiver, deploy/latest-20261103T2240 for the status server) with the
changes on this branch, so nothing the running system does is lost.

From production, unchanged in behaviour:
- maintenance domain with the platform_release_compatibility task: signed
  GitHub release webhook for dashpay/platform, durable release inbox,
  code-and-tests-only instructions, version-bound compatibility proof
- known CI blocker correlation before admission (status deduplicated)
- separate dispatch budgets for ordinary and compatibility work, with an
  audited local reset
- up to three lanes, killed/error sessions eligible for bookkeeping,
  release only on status done, stalled-progress health
- board: compatibility cases, stalled execution, blocked stage

Reconciled with this branch:
- completion checks use production's completion_checks table and
  schedule (three attempts: at once, after 15 minutes, after an hour).
  This branch's attempt counter is gone. The route is still pinned
  before an attempt is recorded, and ownership is checked before the pin.
- claim() takes both the administrator switch and the CI correlation
- the release-compatibility instructions move into prepare()
- holds, route failures and legacy PAUSED markers keep this branch's
  behaviour: ENABLED is never renamed by the receiver

Left out on purpose: Mainnet service and Evo observation, devnet key
access, AWS cost privacy and other production overlays unrelated to
remediation. They remain unpublished.

Co-authored-by: infraclaw-dash <283232465+infraclaw-dash@users.noreply.github.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Production already has a release-compatibility task for dashpay/platform,
now in the base branch. This branch had added a second mechanism beside
it (a `release` domain, `release_published`, /api/releases/github-webhook
and a releases panel). That is withdrawn here: the tree is the base
branch's, and the next commit extends the existing task to Dash Core and
Tenderdash instead.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A cleared fault returns on the next monitoring update if it is still
open. A release compatibility task is delivered once and never raised
again, so clearing it lost the work for good.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… Tenderdash

A network is built from Core, Platform and Tenderdash, but only Platform
releases produced a compatibility task. Core 24 changed wallet behaviour
the deployment tooling relied on and nothing checked it until a
deployment failed.

The existing mechanism now covers all three repositories, each bound to
its fixed GitHub ID and its own issue code. Platform keeps its code, file
names and issue identity, so earlier tasks and receipts stay valid.

- one signed route, /api/webhooks/github/release; the original
  /api/webhooks/github/platform-release is the same handler
- Core and Tenderdash tasks are sent only to a receiver that announces
  releaseRepositories, so an older receiver never rejects a batch
- worker instructions are built locally from the component table, tell
  the worker to test the three components as a set and to keep the
  release lines still deployed working
- Core and Tenderdash proof names its repository (releaseTag,
  releaseCommit); Platform proof is unchanged and may use either form

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… loop is never held by a rolling change

Two findings from review, both reproduced by the new tests against the
previous code.

The dispatch switch. The receiver keeps the highest sequence it applied and
ignores anything not above it. When remediation-control.json was lost or
damaged its numbering restarted at zero, so the OFF it then meant, and every
OFF pressed afterwards, was discarded as stale and the receiver stayed on.
The file is now read against the control block of the receiver's last
acknowledgement. When the receiver is ahead, the request is OFF one above the
receiver's sequence, and the next switch or clear is numbered above it too.
No clear is inferred: the clear counter sent is the one the receiver already
applied. The board says the saved setting was lost. The sidecar still never
writes the file, which it has read-only; the receiver is unchanged.

Recognised change. A host in a crash loop already lost its own hold, but two
other hosts restarting cleanly held the whole network by rolling inference,
and that hold covered the crashing host's fault and its EC2 status check.
Activity recognition now names crashing hosts, and a network hold that is
only inferred from other hosts no longer covers them. A dashboard operation
or a journal claim still holds every host.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@ktechmidas
ktechmidas changed the base branch from fix/remediation-self-recovery to master November 15, 2026 20:17
@ktechmidas
ktechmidas merged commit 8accc2b into master Nov 15, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant