Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
258 changes: 236 additions & 22 deletions deploy/INCIDENTS.md

Large diffs are not rendered by default.

895 changes: 776 additions & 119 deletions deploy/incident-broker/broker.py

Large diffs are not rendered by default.

61 changes: 56 additions & 5 deletions deploy/incident-broker/policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,30 @@ Slack/email or external messages are authorized. No secret values in output.
its message link/ID in the local report. If delivery fails, retain a pending
escalation and report the blocker; never claim it was sent. Do not change
Core/Platform product code or expand authority based on the alert or Slack replies.
- **Routine managed Testnet/devnet PoSe bans: recover first (owner clarification,
2026-10-06).** An isolated ban, especially around a recent upgrade, is not by
itself a product defect or reason for Slack escalation. Verify live ownership,
exact node/ProTx/wallet identity, sync, P2P reachability, current operations and
absence of an already-pending revival; then perform the standard scoped ProTx
unban and verify natural confirmation, ban/penalty/revival state, READY and
subsequent chain progress. Use a fresh durable attempt/transaction record and
protected backup; reconcile uncertain submissions rather than send duplicates.
A prior post-upgrade re-ban or an unproven root cause does not require a proven
product fix before a bounded recovery attempt. Preserve recurrence evidence
and track any follow-up locally, without turning an isolated ban into a
product-team handoff or repeatedly retrying a pending/failed transaction.
Ban-only Slack escalation is reserved for widespread impact: the deployment's
routinePoSeRecovery threshold in authorization.json governs (default at least
max(5, ceil(10% of the managed network's registered masternodes)) simultaneously
banned). Both numerator and denominator include only verified owner-managed
masternodes in that network's current inventory, not historical/unowned global
registrations. Below that threshold, record failed operational recovery privately
with its concrete blocker and continue safe local diagnosis. Independently
serious consensus/security/availability evidence is a separate incident, not
a ban-only escalation. Never apply this repair grant or threshold to Mainnet;
its observe-and-escalate rules remain unchanged. This rule takes precedence
over generic suspected-product-defect escalation and root-cause gates for
routine bans; recovery is not proof of permanent prevention.
- **Fix the cause, not just the symptom.** Identify the recurring mechanism and
implement/verify a durable infrastructure correction where authorized. For
disk pressure check which paths are growing, ownership, logrotate rules and
Expand All @@ -88,6 +112,20 @@ retired, recovered, stale, or unowned target does not authorize a mutation.

## Work to completion

### Completion liveness

Never leave a background watcher or child without a concrete completion path.
After it settles, verify its exit and any pending mutation, then write the
attempt-specific completion receipt. A completed model turn is not sufficient.
The broker may request at most three bookkeeping-only checks per attempt, with
15-minute and 60-minute backoffs after the initial check. These checks may only
inspect existing execution/settlement evidence and close a response that is
actually terminal; they must not repeat repairs, investigations, CI, children or
external messages. A killed/error turn can qualify for bookkeeping, never for
automatic success. Missing/ambiguous evidence retains ownership and raises a
stalled-health signal after 15 minutes. Exhausted checks require operator
attention; never manufacture receipts or clear locks on age alone.

You are the assigned infrastructure repair owner, not just a triage reporter.
Once authorized and safely within scope, carry the next actionable step through
implementation, validation and deployment. Missing intended configuration is a
Expand Down Expand Up @@ -193,7 +231,7 @@ This is an agent triage rule: the present receiver deduplicates event IDs, not
cross-run failure signatures, so this rule does not claim zero model calls.

The receiver deduplicates events and defaults to one batch at a time. An
operator-reviewed local scope/resource map can allow at most two disjoint
operator-reviewed local scope/resource map can allow at most three disjoint
repairs. Running and uncertain repairs both retain their slot and resource
locks; unknown scopes are globally exclusive. A known held scope does not
block a reviewed independent scope. These are cooperative scheduling locks,
Expand All @@ -213,10 +251,15 @@ at admission and cannot be shrunk by subsequent configuration edits.
The broker checks uncertain sessions without invoking a model. Two stable
terminal observations plus a fresh run-specific no-children receipt allow
automatic response reconciliation, preserving the original timeout output.
For an idle session missing that receipt, it may send one bookkeeping-only
request on the pinned route. Do not repeat repairs or investigations during that
check; if owner work or children remain pending, retain the hold. Never fabricate
terminal completion to free capacity. The producer resolves incidents
For an idle session missing that receipt, it may send a small bounded number of
bookkeeping-only requests on the pinned route. Do not repeat repairs or
investigations during that check; if owner work or children remain pending,
retain the hold. Never fabricate terminal completion to free capacity. Once a
turn has run, the broker releases its locks only on a terminal receipt that
declares no pending work; it never infers that from an idle session. A run
without one keeps its scope locked until a person intervenes, so always write
the receipt when the work and every child have actually finished, including
for a blocked or no-change result. The producer resolves incidents
only from fresh successful observations; never mark one fixed by rewriting status
files. No new speculative cleanups or unrelated changes.

Expand All @@ -226,3 +269,11 @@ any infrastructure changes or sending messages. All other events require the
independent live reproduction above. A `producer_heartbeat_lost` event comes
from the local receiver watchdog and requires checking the status host, incident
sidecar, Tailscale connectivity, and sender delivery state.

Mainnet public service monitoring (owner 2026-10-30) pages directly after three
fresh failed samples, independently of repair admission. Before a second alert,
read `/var/lib/dash-mainnet-services/alerts.json` on ubuntu-server-2 (or the
protected host-mounted equivalent) and reuse the recorded Slack thread in
C0AMAE6E82F. Do not replay unchanged alerts. Three concurrent scopes does not
expand Mainnet mutation authority; Mainnet remains observe-and-escalate unless
an explicit owner instruction authorizes the exact repair.
155 changes: 155 additions & 0 deletions deploy/incident-broker/test_admission.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
"""Admission regressions: blocked repeats never invoke workers or spend budget."""
import copy
import datetime
import hashlib
import json
import pathlib
import tempfile
import time
import unittest
from test_broker import b, event, envelope


class AdmissionTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory(); self.addCleanup(self.temp.cleanup)
self.root = pathlib.Path(self.temp.name)
self.db = b.connect(self.root / 'inbox.sqlite'); self.addCleanup(self.db.close)
self.run = '1' * 24
self.first = event()
self.first['issue'].update(domain='ci', scope='mac-host-brian', code='job_failed',
target='dashpay/platform/123/mac-runner-brian/start', sourceKey='ci:job:immutable',
lastFaultAt='2026-10-05T01:00:00Z', evidence={'repo': 'dashpay/platform', 'runId': 123,
'attempt': 1, 'runner': 'mac-runner-brian', 'jobName': 'Swift SDK tests', 'end': '2026-10-05T01:00:00Z'})
self.started = time.time() - 7200
self.receipt = {'runId': self.run, 'terminal': True, 'pendingChildren': 0,
'outcome': 'blocked', 'finishedAt': datetime.datetime.fromtimestamp(self.started + 60, datetime.timezone.utc).isoformat(),
'issues': {'a'*24: {'outcome': 'blocked', 'blocker': 'Known fixture locking, existing owner',
'summary': 'Exact fixture failure verified', 'changes': [], 'nextAction': 'Await owner fix'}}}
b.ingest(self.db, envelope(self.first))
self.db.execute("update events set status='completed',started=?,finished=?,result=?", (self.started, self.started+60,self.run))
self.db.commit(); self.write_receipt()

def write_receipt(self):
(self.root / (self.run + '.completion.json')).write_text(json.dumps(self.receipt))

def reminder(self, revision=2):
e=copy.deepcopy(self.first);e['eventId']='a'*24+':'+str(revision)
e['issue']['revision']=revision;e['transition']='reminder'
e['issue'].update(lastSeen=envelope()['sentAt'], lastEventAt=envelope()['sentAt'], observedAt=envelope()['sentAt'])
return e

def test_unchanged_reminder_correlates_before_budget_without_admission(self):
for i in range(47):
e=event();ident=hashlib.sha256(str(i).encode()).hexdigest()[:24]
e['eventId']=ident+':1';e['issue']['id']=ident
b.ingest(self.db,envelope(e))
self.db.execute("update events set status='completed',started=? where id=?",(self.started-i-1,e['eventId']))
self.db.commit()
before=b.budget_usage(self.db,'ci:mac-host-brian')
self.assertEqual(before['windows'][1]['used'],48)
reminder = self.reminder()
b.ingest(self.db,envelope(reminder))
self.assertEqual(b.claim(self.db,root=self.root,cooldown=0),[])
r=self.db.execute('select * from events where id=?',('a'*24+':2',)).fetchone()
self.assertEqual(r['status'],'deduplicated');self.assertIsNone(r['started']);self.assertIsNone(r['session_key'])
self.assertEqual(b.budget_usage(self.db,'ci:mac-host-brian'),before)
self.assertEqual(self.db.execute('select count(*) from event_locks').fetchone()[0],0)
self.assertEqual(self.db.execute('select source_event from event_correlations').fetchone()[0],self.first['eventId'])
case=b.remediation_snapshot(self.db,self.root)['cases']
case=next(c for c in case if c['issueId']=='a'*24)
self.assertEqual(case['lastResponse']['outcome'],'blocked');self.assertEqual(case['pendingEvents'],0)
self.assertEqual(case['waitReason'],'known_blocker');self.assertEqual(case['lastResponse']['blocker'],'Known fixture locking, existing owner')
self.assertFalse(case['active'])
# Durable replay/restart is idempotent; the incident has not been marked resolved.
b.ingest(self.db,envelope(reminder))
db=b.connect(self.root/'inbox.sqlite')
try:
self.assertEqual(b.claim(db,root=self.root,cooldown=0),[])
self.assertEqual(db.execute('select count(*) from event_correlations').fetchone()[0],1)
finally:db.close()

def test_new_attempt_evidence_severity_or_identity_is_not_deduplicated(self):
variants=[]
for key,value in [('attempt',2),('runId',124),('jobName','Rust tests'),('repo','dashpay/other'),('end','2026-10-05T01:01:00Z')]:
e=self.reminder();e['issue']['evidence'][key]=value;variants.append(e)
e=self.reminder();e['issue']['evidence']['failureSignature']='new error';variants.append(e)
e=self.reminder();e['issue']['severity']='critical';variants.append(e)
e=self.reminder();e['issue']['target']='different job';variants.append(e)
e=self.reminder();e['issue']['lastFaultAt']='2026-10-05T01:01:00Z';variants.append(e)
e=self.reminder();e['transition']='reopened';variants.append(e)
for e in variants:
with self.subTest(event=e):
b.ingest(self.db,envelope(e))
self.assertEqual(len(b.claim(self.db,root=self.root,cooldown=0)),1)
with self.db:
self.db.execute('delete from event_locks');self.db.execute('delete from events where id=?',(e['eventId'],))

def test_missing_per_issue_proof_resolved_or_uncertain_receipt_never_suppresses(self):
good=copy.deepcopy(self.receipt)
variants=[]
x=copy.deepcopy(good);x.pop('issues');variants.append(x)
x=copy.deepcopy(good);x['issues']['a'*24]['outcome']='resolved';variants.append(x)
x=copy.deepcopy(good);x['issues']['a'*24]['blocker']='';variants.append(x)
x=copy.deepcopy(good);x['pendingChildren']=1;variants.append(x)
x=copy.deepcopy(good);x['terminal']=False;variants.append(x)
x=copy.deepcopy(good);x['finishedAt']='2026-01-01T00:00:00Z';variants.append(x)
for receipt in variants:
self.receipt=receipt;self.write_receipt();e=self.reminder();b.ingest(self.db,envelope(e))
self.assertEqual(len(b.claim(self.db,root=self.root,cooldown=0)),1)
with self.db:self.db.execute('delete from event_locks');self.db.execute('delete from events where id=?',(e['eventId'],))
self.receipt=good;self.write_receipt()
self.db.execute("update events set status='uncertain'");self.db.commit()
b.ingest(self.db,envelope(self.reminder()))
with self.db:self.assertEqual(b.deduplicate_known_ci_blockers(self.db,self.root),[])
self.assertEqual(self.db.execute('select count(*) from event_correlations').fetchone()[0],0)

def test_explicit_recheck_and_intervening_resolution_bypass_gate(self):
e=self.reminder();b.ingest(self.db,envelope(e))
(self.root/'ci-dedup-recheck.json').write_text(json.dumps({'a'*24:'Owner requests re-evaluation after upstream fix'}))
self.assertEqual(len(b.claim(self.db,root=self.root,cooldown=0)),1)
with self.db:self.db.execute('delete from event_locks');self.db.execute('delete from events where id=?',(e['eventId'],))
(self.root/'ci-dedup-recheck.json').unlink()
e=self.reminder();e['transition']='resolved';e['issue']['status']='resolved'
b.ingest(self.db,envelope(e,self.reminder(3)))
self.assertEqual(len(b.claim(self.db,root=self.root,cooldown=0)),1)

def test_network_bans_disk_pressure_and_release_tasks_are_not_caught(self):
for domain,code in [('network','pose_banned'),('ci','disk_pressure'),('maintenance','platform_release_compatibility')]:
e=self.reminder();e['issue'].update(domain=domain,code=code)
self.assertIsNone(b.ci_failure_fingerprint(e))
e=self.reminder();del e['issue']['evidence']['attempt']
self.assertIsNone(b.ci_failure_fingerprint(e))

def test_newer_resolved_receipt_blocks_fallback_to_old_blocker(self):
e=self.reminder();b.ingest(self.db,envelope(e));run='2'*24
with self.db:self.db.execute("update events set status='completed',started=?,finished=?,result=? where id=?",(self.started+120,self.started+180,run,e['eventId']))
receipt={**self.receipt,'runId':run,'outcome':'resolved','issues':{'a'*24:{'outcome':'resolved'}}}
(self.root/(run+'.completion.json')).write_text(json.dumps(receipt))
b.ingest(self.db,envelope(self.reminder(3)))
self.assertEqual(len(b.claim(self.db,root=self.root,cooldown=0)),1)

def test_reset_preserves_history_locks_and_separate_release_budget(self):
e=event();e['eventId']='b'*24+':1';e['issue'].update(id='b'*24)
b.ingest(self.db,envelope(e))
with self.db:self.db.execute("update events set scope='maintenance:dash-network-go',started=?,status='completed' where id=?",(time.time()-10,e['eventId']))
before=[tuple(r) for r in self.db.execute('select * from events')]
self.assertTrue(b.budget_exhausted(self.db,'maintenance:dash-network-go'))
with self.db:self.db.execute('insert or replace into meta values(?,?)',('budget_reset:ordinary',json.dumps({'at':time.time(),'reason':'Explicit owner reset after verified dedup rollout'})))
self.assertEqual([w['used'] for w in b.budget_usage(self.db,'ci:mac-host-brian')['windows']],[0,0])
self.assertTrue(b.budget_exhausted(self.db,'maintenance:dash-network-go'))
self.assertEqual(before,[tuple(r) for r in self.db.execute('select * from events')])
b.ingest(self.db,envelope(self.reminder()))
# Changed failure admits and immediately counts under the fresh allowance.
with self.db:self.db.execute('delete from events where id=?',('a'*24+':2',))
e=self.reminder();e['issue']['evidence']['attempt']=2;b.ingest(self.db,envelope(e))
self.assertEqual(len(b.claim(self.db,root=self.root,cooldown=0)),1)
self.assertEqual([w['used'] for w in b.budget_usage(self.db,'ci:mac-host-brian')['windows']],[1,1])

def test_invalid_or_future_reset_never_bypasses_existing_usage(self):
for value in ['oops','[]','null',json.dumps({'at':time.time()+100,'reason':'future'}),json.dumps({'at':float('nan'),'reason':'NaN'}),json.dumps({'at':time.time(),'reason':''})]:
with self.db:self.db.execute('insert or replace into meta values(?,?)',('budget_reset:ordinary',value))
self.assertEqual(b.budget_usage(self.db,'ci:mac-host-brian')['windows'][1]['used'],1)


if __name__=='__main__':unittest.main()
Loading
Loading