bdevierno1 · bdevierno1 · Jun 19, 2026
diff --git a/docs/postmortem-template.md b/docs/postmortem-template.md
@@ -0,0 +1,133 @@
+# Blameless Postmortem Template
+
+> **How to use this template:** Copy this file to `docs/postmortems/YYYY-MM-DD-<slug>.md`, fill every section, and open it as a PR within 3 business days of a SEV1 or SEV2 incident. Systems fail; people do their best with the information available. This document focuses on what happened and how to prevent recurrence — not on who made a mistake.
+
+---
+
+## Incident summary
+
+| Field | Value |
+|-------|-------|
+| **Title** | `<short descriptive title, e.g. "API server down — migration removed a required column">` |
+| **Date / time (UTC)** | `YYYY-MM-DD HH:MM – HH:MM UTC` |
+| **Duration** | `<X h Y min>` |
+| **Severity** | `SEV1` / `SEV2` |
+| **Incident commander** | `<name>` |
+| **Scribe** | `<name>` |
+| **Participants** | `<names>` |
+| **Affected service(s)** | `<e.g. API Server, Agent Runner>` |
+| **Ticket / incident channel** | `#inc-YYYYMMDD-<slug>` |
+
+---
+
+## Impact
+
+_Quantify the impact. Be precise._
+
+- **Operations affected:** `<N>` operations failed to start / complete during the window.
+- **Users affected:** `<N>` users experienced errors or degraded experience.
+- **Data lost:** `<none / describe if any>`
+- **External visibility:** `<yes/no — was this visible to users? How did they notice?>`
+
+---
+
+## Timeline
+
+_List events in UTC. Include detection, escalation, key actions, and resolution. One line per event._
+
+| Time (UTC) | Event |
+|------------|-------|
+| HH:MM | Alert fired / user reported issue |
+| HH:MM | On-call acknowledged |
+| HH:MM | SEV declared |
+| HH:MM | Diagnosis: root cause identified |
+| HH:MM | Mitigation applied (rollback / config change / restart) |
+| HH:MM | Service restored; verification checks passed |
+| HH:MM | Incident closed |
+
+---
+
+## Root cause
+
+_One to three sentences. What was the direct technical cause? Avoid blame language._
+
+Example: "A schema migration in `schema_migrate.py` dropped the `findings_json` column instead of adding it, causing all reads of `Operation.findings_json` to raise `OperationalError`."
+
+```
+<root cause here>
+```
+
+---
+
+## Contributing factors
+
+_What conditions allowed the root cause to have this impact? Use "5 Whys" or similar._
+
+1. **Why did this happen?** `<e.g. The migration was not tested against an existing non-empty database.>`
+2. **Why was it not caught before deploy?** `<e.g. CI only tests against a fresh schema; no migration regression test existed.>`
+3. **Why did detection take N minutes?** `<e.g. The liveness check probes /health, which does not exercise the DB schema.>`
+4. (Add more as needed)
+
+---
+
+## What went well
+
+_Genuinely good practices that limited impact or aided recovery._
+
+- `<e.g. The deep health endpoint surfaced the database failure within 2 minutes.>`
+- `<e.g. The on-call engineer had the rollback runbook open and executed it in under 10 minutes.>`
+
+---
+
+## What went wrong
+
+_Things that made this harder than it needed to be._
+
+- `<e.g. No database backup existed; rollback required a schema-only repair.>`
+- `<e.g. The SEV1 runbook lacked a step for cancelling stuck in-flight operations.>`
+
+---
+
+## Where we got lucky
+
+_Things that could have made this worse but didn't._
+
+- `<e.g. The incident occurred during low-traffic hours; only 3 users were affected.>`
+- `<e.g. The SQLite file was not corrupted — only the schema was wrong.>`
+
+---
+
+## Action items
+
+_Each item has an owner and a due date. Track these as issues in the repo._
+
+| # | Action | Owner | Due date | Issue |
+|---|--------|-------|----------|-------|
+| 1 | `<e.g. Add migration regression test that runs against an existing DB fixture>` | `<name>` | `YYYY-MM-DD` | `#<n>` |
+| 2 | `<e.g. Add PRAGMA integrity_check to the /api/health/deep endpoint>` | `<name>` | `YYYY-MM-DD` | `#<n>` |
+| 3 | `<e.g. Set up automated 15-minute SQLite backups per database-rollback.md §6>` | `<name>` | `YYYY-MM-DD` | `#<n>` |
+
+---
+
+## Detection gap
+
+_How long did it take to detect? What would have caught this faster?_
+
+- **Time to detect:** `<N min>`
+- **Alert that fired (or didn't):** `<e.g. A02 deep health — but only after 2-min threshold>`
+- **Missing alert:** `<e.g. No alert for schema column presence; adding A05 integrity check would have fired sooner>`
+
+---
+
+## Appendix: supporting data
+
+_Paste relevant log snippets, queries, and metrics here._
+
+```
+<paste log lines, stack traces, or SQL output>
+```
+
+---
+
+_Filed by:_ `<name>` on `YYYY-MM-DD`  
+_Reviewed by:_ `<names>` on `YYYY-MM-DD`
diff --git a/docs/postmortems/.gitkeep b/docs/postmortems/.gitkeep
diff --git a/docs/runbooks/alert-definitions.md b/docs/runbooks/alert-definitions.md
@@ -0,0 +1,120 @@
+# Alert Definitions — Vecna Operations Console
+
+This file is the single source of truth for alert conditions and their linked runbooks.  
+Every alert entry maps a **signal** to a **severity**, a **runbook link**, and a **threshold**.
+
+> **Rule:** No alert may be wired to monitoring without an entry here. Each entry links to a runbook with restart/backfill commands. Do not page on minor wobbles.
+
+---
+
+## A01 — API Server Liveness
+
+| Field | Value |
+|-------|-------|
+| **Signal** | `GET /health` returns non-200 or connection refused |
+| **Probe interval** | 30 s |
+| **Threshold** | 3 consecutive failures (90 s) |
+| **Severity** | SEV1 |
+| **Runbook** | [sev1-runbook.md](sev1-runbook.md) → [tier0/api-server-rollback.md](tier0/api-server-rollback.md) |
+| **Sample check** | `curl -sf http://127.0.0.1:8000/health` |
+
+**Do not page on:** a single transient 502 from a load balancer restart.
+
+---
+
+## A02 — Deep Health Degradation
+
+| Field | Value |
+|-------|-------|
+| **Signal** | `GET /api/health/deep` returns `database: false` or `model: ""` |
+| **Probe interval** | 60 s |
+| **Threshold** | 2 consecutive failures (2 min) |
+| **Severity** | SEV1 (database false) / SEV2 (model empty) |
+| **Runbook — database** | [sev1-runbook.md](sev1-runbook.md) → [tier0/database-rollback.md](tier0/database-rollback.md) |
+| **Runbook — model** | [sev2-runbook.md](sev2-runbook.md) §3a |
+| **Sample check** | `curl -s http://127.0.0.1:8000/api/health/deep \| jq '{db:.database,model:.model}'` |
+
+---
+
+## A03 — Operation Error Rate
+
+| Field | Value |
+|-------|-------|
+| **Signal** | Proportion of `Operation.status == "error"` among operations started in the last 10 min |
+| **Threshold (SEV2)** | >10 % error rate, sustained 10 min |
+| **Threshold (SEV1)** | >80 % error rate, sustained 5 min |
+| **Severity** | SEV2 → escalate to SEV1 |
+| **Runbook** | [sev2-runbook.md](sev2-runbook.md) §3a → [sev1-runbook.md](sev1-runbook.md) |
+| **Sample query** | `sqlite3 vecna.db "SELECT status, count(*) FROM operations WHERE created_at > datetime('now','-10 minutes') GROUP BY status;"` |
+
+**Do not page on:** a single failed operation caused by an invalid user-supplied URL (these produce `error` status legitimately). Alert only on bulk failure.
+
+---
+
+## A04 — Operations Stuck in Running
+
+| Field | Value |
+|-------|-------|
+| **Signal** | Count of operations with `status == "running"` AND `updated_at < now - 10 min` |
+| **Threshold** | ≥ 3 such operations |
+| **Severity** | SEV2 |
+| **Runbook** | [sev2-runbook.md](sev2-runbook.md) §3a → [tier0/agent-runner-rollback.md](tier0/agent-runner-rollback.md) |
+| **Sample query** | `sqlite3 vecna.db "SELECT count(*) FROM operations WHERE status='running' AND updated_at < datetime('now','-10 minutes');"` |
+
+---
+
+## A05 — Database Integrity
+
+| Field | Value |
+|-------|-------|
+| **Signal** | `sqlite3 vecna.db "PRAGMA integrity_check;"` returns anything other than `ok` |
+| **Probe interval** | 15 min |
+| **Threshold** | Any single failure |
+| **Severity** | SEV1 |
+| **Runbook** | [sev1-runbook.md](sev1-runbook.md) → [tier0/database-rollback.md](tier0/database-rollback.md) |
+| **Sample check** | `sqlite3 /path/to/vecna.db "PRAGMA integrity_check;"` |
+
+---
+
+## A06 — Metrics Endpoint Unavailable
+
+| Field | Value |
+|-------|-------|
+| **Signal** | `GET /api/metrics` returns non-200 or times out |
+| **Probe interval** | 60 s |
+| **Threshold** | 5 consecutive failures (5 min) |
+| **Severity** | SEV2 |
+| **Runbook** | [sev2-runbook.md](sev2-runbook.md) → [tier0/api-server-rollback.md](tier0/api-server-rollback.md) |
+
+---
+
+## A07 — High Operation Latency
+
+| Field | Value |
+|-------|-------|
+| **Signal** | Median time from operation creation to `done` status exceeds 3× rolling 7-day median |
+| **Threshold** | Sustained >30 min |
+| **Severity** | SEV2 |
+| **Runbook** | [sev2-runbook.md](sev2-runbook.md) §3a (check LLM provider status first before rolling back) |
+
+**Do not page on:** provider-wide slowness confirmed on the LLM provider's status page — this is not a Vecna Ops incident.
+
+---
+
+## Wiring alerts to this file
+
+When configuring a monitoring tool (Prometheus Alertmanager, Grafana, PagerDuty, UptimeRobot, etc.) for each alert above:
+
+1. Set the alert `name` to the `A0X` identifier (e.g. `vecna_A01_api_liveness`).
+2. Add an annotation/label `runbook_url` pointing to the absolute GitHub URL of the linked runbook file.
+3. Include the threshold and severity in the alert body so the on-call engineer sees them in the notification without needing to open this doc.
+
+Example Prometheus alert annotations block:
+
+```yaml
+annotations:
+  summary: "A01 — Vecna API server liveness failure"
+  description: "GET /health has returned non-200 for 90 s. Immediate action required."
+  runbook_url: "https://github.com/bdevierno1/Vecna-Operations-Console/blob/main/docs/runbooks/sev1-runbook.md"
+  severity: "SEV1"
+```
diff --git a/docs/runbooks/sev1-runbook.md b/docs/runbooks/sev1-runbook.md
@@ -0,0 +1,126 @@
+# SEV1 Incident Runbook — Vecna Operations Console
+
+**Severity:** SEV1 — Complete service outage or critical data loss  
+**Response SLA:** Acknowledge within 5 min · Mitigate within 30 min · Resolve within 2 h  
+**Postmortem SLA:** Blameless postmortem within 3 business days  
+**Template:** [docs/postmortem-template.md](../postmortem-template.md)
+
+---
+
+## SEV1 definition
+
+A SEV1 exists when **any** of the following is true:
+
+- All operations fail to start (API returns 5xx on `POST /api/operations`)
+- `GET /health` returns non-200 or is unreachable
+- Database is corrupted or unreadable
+- All in-flight operations are silently stuck in `running` with no events
+- Data loss confirmed (operations deleted or findings lost outside normal flow)
+- Security breach — unauthorized access to operations data or the SQLite file
+
+---
+
+## Step 1 — Acknowledge (< 5 min)
+
+1. Claim the incident in the alert channel: "@here I'm IC on this."
+2. Open an incident channel: `#inc-YYYYMMDD-<short-description>`.
+3. Post initial status: "SEV1 declared. Service: Vecna Ops Console. Symptom: `<one line>`. IC: `<name>`."
+4. Page backup if this is outside business hours.
+
+---
+
+## Step 2 — Assess (< 10 min from acknowledge)
+
+Run these checks in order. Stop at the first failure — that is your blast radius.
+
+```bash
+HOST=http://127.0.0.1:8000   # adjust to your deployed hostname
+
+# Is the process alive?
+pgrep -a uvicorn
+
+# Basic liveness
+curl -s -o /dev/null -w "%{http_code}" $HOST/health
+
+# Deep health: DB + model
+curl -s $HOST/api/health/deep | jq .
+
+# Recent errors in logs
+journalctl -u vecna-api -n 100 --no-pager | grep -E "ERROR|CRITICAL|Traceback" | tail -20
+# or: tail -100 /path/to/uvicorn.log | grep -E "ERROR|CRITICAL"
+
+# Database integrity
+sqlite3 /path/to/vecna.db "PRAGMA integrity_check;" 2>/dev/null || echo "DB UNREACHABLE"
+
+# Active operation count
+curl -s $HOST/api/operations | jq '[.[] | select(.status == "running")] | length'
+```
+
+Record findings in the incident channel before acting.
+
+---
+
+## Step 3 — Contain
+
+Apply the appropriate Tier-0 rollback runbook based on your assessment:
+
+| Component failing | Runbook |
+|-------------------|---------|
+| API server (FastAPI/Uvicorn process) | [tier0/api-server-rollback.md](tier0/api-server-rollback.md) |
+| Agent runner (operations stuck/erroring) | [tier0/agent-runner-rollback.md](tier0/agent-runner-rollback.md) |
+| Database (integrity check failure, schema error) | [tier0/database-rollback.md](tier0/database-rollback.md) |
+| Frontend (UI blank, JS crash) | [tier0/frontend-rollback.md](tier0/frontend-rollback.md) |
+
+If multiple components are failing, address in this order: **Database → API Server → Agent Runner → Frontend**.
+
+---
+
+## Step 4 — Communicate (every 15 min until resolved)
+
+Post in `#inc-YYYYMMDD-<short-description>`:
+
+```
+[HH:MM UTC] Status update
+- Symptom: <what is failing>
+- Blast radius: <who is affected, how many operations impacted>
+- Action taken: <what you did in last 15 min>
+- Next action: <what you are doing now>
+- ETA to resolution: <estimate or "unknown">
+```
+
+---
+
+## Step 5 — Resolve
+
+Resolution criteria — **all** must be true:
+
+- [ ] `GET /health` returns 200
+- [ ] `GET /api/health/deep` shows `database: true` and `model: <non-empty>`
+- [ ] A smoke-test operation on `https://example.com` reaches `done` status
+- [ ] No `ERROR` or `CRITICAL` entries in logs for 10 consecutive minutes
+
+Post in incident channel: "SEV1 resolved at [HH:MM UTC]. All checks pass."
+
+---
+
+## Step 6 — Learn
+
+- Open a postmortem issue within 3 business days.
+- Use template: [docs/postmortem-template.md](../postmortem-template.md).
+- Schedule a 30-min review meeting with the team.
+- Update this runbook if any step was missing or wrong.
+
+---
+
+## Quick reference
+
+| Resource | Location |
+|----------|----------|
+| Health endpoint | `GET /health` |
+| Deep health | `GET /api/health/deep` |
+| Metrics | `GET /api/metrics` |
+| API server rollback | [tier0/api-server-rollback.md](tier0/api-server-rollback.md) |
+| Agent runner rollback | [tier0/agent-runner-rollback.md](tier0/agent-runner-rollback.md) |
+| Database rollback | [tier0/database-rollback.md](tier0/database-rollback.md) |
+| Frontend rollback | [tier0/frontend-rollback.md](tier0/frontend-rollback.md) |
+| Postmortem template | [../postmortem-template.md](../postmortem-template.md) |