From 4f2c4944509fbb8a9bc73b9fdaf7b3faa5c2b0d9 Mon Sep 17 00:00:00 2001 From: Benjamin De Vierno <54540257+bdevierno1@users.noreply.github.com> Date: Thu, 18 Jun 2026 20:38:37 -0400 Subject: [PATCH] Add incident-response runbooks, SEV1/SEV2 runbooks, and postmortem template MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - docs/runbooks/tier0/: documented + tested rollback paths for all four Tier-0 services (API server, agent runner, database, frontend) — each includes decision gate, artifact identification, step-by-step rollback commands, verification smoke tests, and a test log table. - docs/runbooks/sev1-runbook.md: full SEV1 response procedure (ack <5 min, mitigate <30 min) with assess commands and rollback dispatch table. - docs/runbooks/sev2-runbook.md: SEV2 response procedure (ack <15 min) with per-symptom diagnosis steps. - docs/runbooks/alert-definitions.md: seven named alerts (A01–A07) mapping signal → threshold → severity → runbook link; Prometheus annotation example included. - docs/postmortem-template.md: blameless postmortem template covering summary, timeline, root cause, contributing factors, what went well/wrong, where we got lucky, action items, and detection gap analysis. - docs/postmortems/: empty directory seeded with .gitkeep to hold future filed postmortems. Co-Authored-By: Vecna --- docs/postmortem-template.md | 133 ++++++++++++++ docs/postmortems/.gitkeep | 0 docs/runbooks/alert-definitions.md | 120 +++++++++++++ docs/runbooks/sev1-runbook.md | 126 ++++++++++++++ docs/runbooks/sev2-runbook.md | 174 +++++++++++++++++++ docs/runbooks/tier0/agent-runner-rollback.md | 115 ++++++++++++ docs/runbooks/tier0/api-server-rollback.md | 141 +++++++++++++++ docs/runbooks/tier0/database-rollback.md | 157 +++++++++++++++++ docs/runbooks/tier0/frontend-rollback.md | 137 +++++++++++++++ 9 files changed, 1103 insertions(+) create mode 100644 docs/postmortem-template.md create mode 100644 docs/postmortems/.gitkeep create mode 100644 docs/runbooks/alert-definitions.md create mode 100644 docs/runbooks/sev1-runbook.md create mode 100644 docs/runbooks/sev2-runbook.md create mode 100644 docs/runbooks/tier0/agent-runner-rollback.md create mode 100644 docs/runbooks/tier0/api-server-rollback.md create mode 100644 docs/runbooks/tier0/database-rollback.md create mode 100644 docs/runbooks/tier0/frontend-rollback.md diff --git a/docs/postmortem-template.md b/docs/postmortem-template.md new file mode 100644 index 0000000..b211064 --- /dev/null +++ b/docs/postmortem-template.md @@ -0,0 +1,133 @@ +# Blameless Postmortem Template + +> **How to use this template:** Copy this file to `docs/postmortems/YYYY-MM-DD-.md`, fill every section, and open it as a PR within 3 business days of a SEV1 or SEV2 incident. Systems fail; people do their best with the information available. This document focuses on what happened and how to prevent recurrence — not on who made a mistake. + +--- + +## Incident summary + +| Field | Value | +|-------|-------| +| **Title** | `` | +| **Date / time (UTC)** | `YYYY-MM-DD HH:MM – HH:MM UTC` | +| **Duration** | `` | +| **Severity** | `SEV1` / `SEV2` | +| **Incident commander** | `` | +| **Scribe** | `` | +| **Participants** | `` | +| **Affected service(s)** | `` | +| **Ticket / incident channel** | `#inc-YYYYMMDD-` | + +--- + +## Impact + +_Quantify the impact. Be precise._ + +- **Operations affected:** `` operations failed to start / complete during the window. +- **Users affected:** `` users experienced errors or degraded experience. +- **Data lost:** `` +- **External visibility:** `` + +--- + +## Timeline + +_List events in UTC. Include detection, escalation, key actions, and resolution. One line per event._ + +| Time (UTC) | Event | +|------------|-------| +| HH:MM | Alert fired / user reported issue | +| HH:MM | On-call acknowledged | +| HH:MM | SEV declared | +| HH:MM | Diagnosis: root cause identified | +| HH:MM | Mitigation applied (rollback / config change / restart) | +| HH:MM | Service restored; verification checks passed | +| HH:MM | Incident closed | + +--- + +## Root cause + +_One to three sentences. What was the direct technical cause? Avoid blame language._ + +Example: "A schema migration in `schema_migrate.py` dropped the `findings_json` column instead of adding it, causing all reads of `Operation.findings_json` to raise `OperationalError`." + +``` + +``` + +--- + +## Contributing factors + +_What conditions allowed the root cause to have this impact? Use "5 Whys" or similar._ + +1. **Why did this happen?** `` +2. **Why was it not caught before deploy?** `` +3. **Why did detection take N minutes?** `` +4. (Add more as needed) + +--- + +## What went well + +_Genuinely good practices that limited impact or aided recovery._ + +- `` +- `` + +--- + +## What went wrong + +_Things that made this harder than it needed to be._ + +- `` +- `` + +--- + +## Where we got lucky + +_Things that could have made this worse but didn't._ + +- `` +- `` + +--- + +## Action items + +_Each item has an owner and a due date. Track these as issues in the repo._ + +| # | Action | Owner | Due date | Issue | +|---|--------|-------|----------|-------| +| 1 | `` | `` | `YYYY-MM-DD` | `#` | +| 2 | `` | `` | `YYYY-MM-DD` | `#` | +| 3 | `` | `` | `YYYY-MM-DD` | `#` | + +--- + +## Detection gap + +_How long did it take to detect? What would have caught this faster?_ + +- **Time to detect:** `` +- **Alert that fired (or didn't):** `` +- **Missing alert:** `` + +--- + +## Appendix: supporting data + +_Paste relevant log snippets, queries, and metrics here._ + +``` + +``` + +--- + +_Filed by:_ `` on `YYYY-MM-DD` +_Reviewed by:_ `` on `YYYY-MM-DD` diff --git a/docs/postmortems/.gitkeep b/docs/postmortems/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/docs/runbooks/alert-definitions.md b/docs/runbooks/alert-definitions.md new file mode 100644 index 0000000..e5daee3 --- /dev/null +++ b/docs/runbooks/alert-definitions.md @@ -0,0 +1,120 @@ +# Alert Definitions — Vecna Operations Console + +This file is the single source of truth for alert conditions and their linked runbooks. +Every alert entry maps a **signal** to a **severity**, a **runbook link**, and a **threshold**. + +> **Rule:** No alert may be wired to monitoring without an entry here. Each entry links to a runbook with restart/backfill commands. Do not page on minor wobbles. + +--- + +## A01 — API Server Liveness + +| Field | Value | +|-------|-------| +| **Signal** | `GET /health` returns non-200 or connection refused | +| **Probe interval** | 30 s | +| **Threshold** | 3 consecutive failures (90 s) | +| **Severity** | SEV1 | +| **Runbook** | [sev1-runbook.md](sev1-runbook.md) → [tier0/api-server-rollback.md](tier0/api-server-rollback.md) | +| **Sample check** | `curl -sf http://127.0.0.1:8000/health` | + +**Do not page on:** a single transient 502 from a load balancer restart. + +--- + +## A02 — Deep Health Degradation + +| Field | Value | +|-------|-------| +| **Signal** | `GET /api/health/deep` returns `database: false` or `model: ""` | +| **Probe interval** | 60 s | +| **Threshold** | 2 consecutive failures (2 min) | +| **Severity** | SEV1 (database false) / SEV2 (model empty) | +| **Runbook — database** | [sev1-runbook.md](sev1-runbook.md) → [tier0/database-rollback.md](tier0/database-rollback.md) | +| **Runbook — model** | [sev2-runbook.md](sev2-runbook.md) §3a | +| **Sample check** | `curl -s http://127.0.0.1:8000/api/health/deep \| jq '{db:.database,model:.model}'` | + +--- + +## A03 — Operation Error Rate + +| Field | Value | +|-------|-------| +| **Signal** | Proportion of `Operation.status == "error"` among operations started in the last 10 min | +| **Threshold (SEV2)** | >10 % error rate, sustained 10 min | +| **Threshold (SEV1)** | >80 % error rate, sustained 5 min | +| **Severity** | SEV2 → escalate to SEV1 | +| **Runbook** | [sev2-runbook.md](sev2-runbook.md) §3a → [sev1-runbook.md](sev1-runbook.md) | +| **Sample query** | `sqlite3 vecna.db "SELECT status, count(*) FROM operations WHERE created_at > datetime('now','-10 minutes') GROUP BY status;"` | + +**Do not page on:** a single failed operation caused by an invalid user-supplied URL (these produce `error` status legitimately). Alert only on bulk failure. + +--- + +## A04 — Operations Stuck in Running + +| Field | Value | +|-------|-------| +| **Signal** | Count of operations with `status == "running"` AND `updated_at < now - 10 min` | +| **Threshold** | ≥ 3 such operations | +| **Severity** | SEV2 | +| **Runbook** | [sev2-runbook.md](sev2-runbook.md) §3a → [tier0/agent-runner-rollback.md](tier0/agent-runner-rollback.md) | +| **Sample query** | `sqlite3 vecna.db "SELECT count(*) FROM operations WHERE status='running' AND updated_at < datetime('now','-10 minutes');"` | + +--- + +## A05 — Database Integrity + +| Field | Value | +|-------|-------| +| **Signal** | `sqlite3 vecna.db "PRAGMA integrity_check;"` returns anything other than `ok` | +| **Probe interval** | 15 min | +| **Threshold** | Any single failure | +| **Severity** | SEV1 | +| **Runbook** | [sev1-runbook.md](sev1-runbook.md) → [tier0/database-rollback.md](tier0/database-rollback.md) | +| **Sample check** | `sqlite3 /path/to/vecna.db "PRAGMA integrity_check;"` | + +--- + +## A06 — Metrics Endpoint Unavailable + +| Field | Value | +|-------|-------| +| **Signal** | `GET /api/metrics` returns non-200 or times out | +| **Probe interval** | 60 s | +| **Threshold** | 5 consecutive failures (5 min) | +| **Severity** | SEV2 | +| **Runbook** | [sev2-runbook.md](sev2-runbook.md) → [tier0/api-server-rollback.md](tier0/api-server-rollback.md) | + +--- + +## A07 — High Operation Latency + +| Field | Value | +|-------|-------| +| **Signal** | Median time from operation creation to `done` status exceeds 3× rolling 7-day median | +| **Threshold** | Sustained >30 min | +| **Severity** | SEV2 | +| **Runbook** | [sev2-runbook.md](sev2-runbook.md) §3a (check LLM provider status first before rolling back) | + +**Do not page on:** provider-wide slowness confirmed on the LLM provider's status page — this is not a Vecna Ops incident. + +--- + +## Wiring alerts to this file + +When configuring a monitoring tool (Prometheus Alertmanager, Grafana, PagerDuty, UptimeRobot, etc.) for each alert above: + +1. Set the alert `name` to the `A0X` identifier (e.g. `vecna_A01_api_liveness`). +2. Add an annotation/label `runbook_url` pointing to the absolute GitHub URL of the linked runbook file. +3. Include the threshold and severity in the alert body so the on-call engineer sees them in the notification without needing to open this doc. + +Example Prometheus alert annotations block: + +```yaml +annotations: + summary: "A01 — Vecna API server liveness failure" + description: "GET /health has returned non-200 for 90 s. Immediate action required." + runbook_url: "https://github.com/bdevierno1/Vecna-Operations-Console/blob/main/docs/runbooks/sev1-runbook.md" + severity: "SEV1" +``` diff --git a/docs/runbooks/sev1-runbook.md b/docs/runbooks/sev1-runbook.md new file mode 100644 index 0000000..08dfe8c --- /dev/null +++ b/docs/runbooks/sev1-runbook.md @@ -0,0 +1,126 @@ +# SEV1 Incident Runbook — Vecna Operations Console + +**Severity:** SEV1 — Complete service outage or critical data loss +**Response SLA:** Acknowledge within 5 min · Mitigate within 30 min · Resolve within 2 h +**Postmortem SLA:** Blameless postmortem within 3 business days +**Template:** [docs/postmortem-template.md](../postmortem-template.md) + +--- + +## SEV1 definition + +A SEV1 exists when **any** of the following is true: + +- All operations fail to start (API returns 5xx on `POST /api/operations`) +- `GET /health` returns non-200 or is unreachable +- Database is corrupted or unreadable +- All in-flight operations are silently stuck in `running` with no events +- Data loss confirmed (operations deleted or findings lost outside normal flow) +- Security breach — unauthorized access to operations data or the SQLite file + +--- + +## Step 1 — Acknowledge (< 5 min) + +1. Claim the incident in the alert channel: "@here I'm IC on this." +2. Open an incident channel: `#inc-YYYYMMDD-`. +3. Post initial status: "SEV1 declared. Service: Vecna Ops Console. Symptom: ``. IC: ``." +4. Page backup if this is outside business hours. + +--- + +## Step 2 — Assess (< 10 min from acknowledge) + +Run these checks in order. Stop at the first failure — that is your blast radius. + +```bash +HOST=http://127.0.0.1:8000 # adjust to your deployed hostname + +# Is the process alive? +pgrep -a uvicorn + +# Basic liveness +curl -s -o /dev/null -w "%{http_code}" $HOST/health + +# Deep health: DB + model +curl -s $HOST/api/health/deep | jq . + +# Recent errors in logs +journalctl -u vecna-api -n 100 --no-pager | grep -E "ERROR|CRITICAL|Traceback" | tail -20 +# or: tail -100 /path/to/uvicorn.log | grep -E "ERROR|CRITICAL" + +# Database integrity +sqlite3 /path/to/vecna.db "PRAGMA integrity_check;" 2>/dev/null || echo "DB UNREACHABLE" + +# Active operation count +curl -s $HOST/api/operations | jq '[.[] | select(.status == "running")] | length' +``` + +Record findings in the incident channel before acting. + +--- + +## Step 3 — Contain + +Apply the appropriate Tier-0 rollback runbook based on your assessment: + +| Component failing | Runbook | +|-------------------|---------| +| API server (FastAPI/Uvicorn process) | [tier0/api-server-rollback.md](tier0/api-server-rollback.md) | +| Agent runner (operations stuck/erroring) | [tier0/agent-runner-rollback.md](tier0/agent-runner-rollback.md) | +| Database (integrity check failure, schema error) | [tier0/database-rollback.md](tier0/database-rollback.md) | +| Frontend (UI blank, JS crash) | [tier0/frontend-rollback.md](tier0/frontend-rollback.md) | + +If multiple components are failing, address in this order: **Database → API Server → Agent Runner → Frontend**. + +--- + +## Step 4 — Communicate (every 15 min until resolved) + +Post in `#inc-YYYYMMDD-`: + +``` +[HH:MM UTC] Status update +- Symptom: +- Blast radius: +- Action taken: +- Next action: +- ETA to resolution: +``` + +--- + +## Step 5 — Resolve + +Resolution criteria — **all** must be true: + +- [ ] `GET /health` returns 200 +- [ ] `GET /api/health/deep` shows `database: true` and `model: ` +- [ ] A smoke-test operation on `https://example.com` reaches `done` status +- [ ] No `ERROR` or `CRITICAL` entries in logs for 10 consecutive minutes + +Post in incident channel: "SEV1 resolved at [HH:MM UTC]. All checks pass." + +--- + +## Step 6 — Learn + +- Open a postmortem issue within 3 business days. +- Use template: [docs/postmortem-template.md](../postmortem-template.md). +- Schedule a 30-min review meeting with the team. +- Update this runbook if any step was missing or wrong. + +--- + +## Quick reference + +| Resource | Location | +|----------|----------| +| Health endpoint | `GET /health` | +| Deep health | `GET /api/health/deep` | +| Metrics | `GET /api/metrics` | +| API server rollback | [tier0/api-server-rollback.md](tier0/api-server-rollback.md) | +| Agent runner rollback | [tier0/agent-runner-rollback.md](tier0/agent-runner-rollback.md) | +| Database rollback | [tier0/database-rollback.md](tier0/database-rollback.md) | +| Frontend rollback | [tier0/frontend-rollback.md](tier0/frontend-rollback.md) | +| Postmortem template | [../postmortem-template.md](../postmortem-template.md) | diff --git a/docs/runbooks/sev2-runbook.md b/docs/runbooks/sev2-runbook.md new file mode 100644 index 0000000..18d04ae --- /dev/null +++ b/docs/runbooks/sev2-runbook.md @@ -0,0 +1,174 @@ +# SEV2 Incident Runbook — Vecna Operations Console + +**Severity:** SEV2 — Significant degradation; service partially available +**Response SLA:** Acknowledge within 15 min · Mitigate within 2 h · Resolve within 8 h +**Postmortem SLA:** Blameless postmortem within 3 business days +**Template:** [docs/postmortem-template.md](../postmortem-template.md) + +--- + +## SEV2 definition + +A SEV2 exists when **any** of the following is true and the service is partially available: + +- Operation error rate >10 % but <100 % (some operations completing) +- WebSocket streaming broken for a subset of users +- LLM authentication failures causing intermittent operation failures +- Rate limiting incorrectly blocking legitimate requests +- Frontend degraded (some panels not rendering, findings not loading) but page loads +- Billing session data incorrect or missing for recent operations +- Schema migration partially applied causing some columns to be missing +- Elevated latency (operations taking >3× normal time to complete) + +> If the issue escalates to complete outage, re-declare as [SEV1](sev1-runbook.md). + +--- + +## Step 1 — Acknowledge (< 15 min) + +1. Respond in the alert channel: "I'm looking at this — ." +2. Open an incident thread or channel: `#inc-YYYYMMDD-`. +3. Post initial triage: "SEV2 declared. Symptom: ``. IC: ``. Investigating." + +--- + +## Step 2 — Assess (< 30 min from acknowledge) + +```bash +HOST=http://127.0.0.1:8000 # adjust to your deployed hostname + +# Service liveness +curl -s -o /dev/null -w "%{http_code}" $HOST/health + +# Deep health +curl -s $HOST/api/health/deep | jq . + +# Recent error log sample +journalctl -u vecna-api -n 200 --no-pager | grep -E "ERROR|WARNING|Traceback" | tail -30 +# or: tail -200 /path/to/uvicorn.log | grep -E "ERROR|WARNING" + +# Operation status breakdown +curl -s $HOST/api/operations | jq 'group_by(.status) | map({status: .[0].status, count: length})' + +# Rate limit state (if rate limiting is suspected) +curl -s $HOST/api/health/deep | jq '.rate_limits' + +# LLM connectivity test (via health deep) +curl -s $HOST/api/health/deep | jq '{model_id: .model_id, model_ok: .model}' + +# Check recent schema migration log +grep -i "migration\|migrate\|schema" /path/to/uvicorn.log | tail -20 +``` + +--- + +## Step 3 — Diagnose and contain + +### 3a. Elevated error rate on operations + +```bash +# Find the failing operations and get their error messages +curl -s $HOST/api/operations | jq '[.[] | select(.status == "error") | {id:.id, created_at:.created_at}]' | head -10 + +# Check events_json on a failing operation for the error frame +OP_ID="" +curl -s $HOST/api/operations/$OP_ID | jq '.events_json | fromjson | .[] | select(.type == "error")' +``` + +If the error is a **LiteLLM auth failure** → check `.env` for key/model mismatch; fix config and restart. +If the error is a **tool timeout** → check network egress and `httpx` timeout settings in `http_client.py`. +If the error is a **runner crash** → roll back to previous commit via [tier0/agent-runner-rollback.md](tier0/agent-runner-rollback.md). + +### 3b. WebSocket streaming broken + +```bash +# Confirm WebSocket route is registered +curl -s $HOST/api/health/deep | jq . + +# Test WebSocket manually (requires wscat) +wscat -c ws://127.0.0.1:8000/ws/ +``` + +If WebSocket is broken after a code change → roll back via [tier0/api-server-rollback.md](tier0/api-server-rollback.md). + +### 3c. Rate limiting blocking legitimate requests + +```bash +# Check current limits configuration +curl -s $HOST/api/health/deep | jq '.rate_limits' +``` + +If limits are misconfigured → update `RATE_LIMIT_*` env vars in `.env` and restart the server. +If a per-IP limit is legitimately triggering → no rollback needed; document as expected behavior. + +### 3d. Schema migration partially applied + +```bash +sqlite3 /path/to/vecna.db "PRAGMA table_info(operations);" +sqlite3 /path/to/vecna.db "PRAGMA table_info(billing_sessions);" +``` + +If columns are missing → follow [tier0/database-rollback.md](tier0/database-rollback.md) §4b (schema-only rollback). + +### 3e. Billing session data incorrect + +```bash +# Inspect a specific billing session +SESSION_ID="" +curl -s $HOST/api/billing/sessions/$SESSION_ID | jq . +``` + +Billing data is computed client-side with `localStorage` and aggregated server-side. Incorrect totals are usually a display bug, not data loss. Roll back the frontend if a recent frontend change is suspected: [tier0/frontend-rollback.md](tier0/frontend-rollback.md). + +--- + +## Step 4 — Communicate (every 30 min until resolved) + +Post in incident thread: + +``` +[HH:MM UTC] SEV2 update +- Symptom: +- Blast radius: <% of users/operations affected> +- Root cause hypothesis: +- Actions taken: +- Next step: +- ETA: +``` + +--- + +## Step 5 — Resolve + +Resolution criteria — **all** must be true: + +- [ ] `GET /health` returns 200 +- [ ] `GET /api/health/deep` shows `database: true` +- [ ] Operation error rate back to baseline (<2 %) for 15 consecutive minutes +- [ ] WebSocket events stream correctly for a test operation +- [ ] No anomalous `ERROR` log entries for 15 consecutive minutes + +Post in incident channel: "SEV2 resolved at [HH:MM UTC]. Monitoring for 30 min." + +--- + +## Step 6 — Learn + +- Open a postmortem issue within 3 business days. +- Use template: [docs/postmortem-template.md](../postmortem-template.md). +- Update contributing runbooks if any steps were missing or incorrect. + +--- + +## Quick reference + +| Resource | Location | +|----------|----------| +| Health endpoint | `GET /health` | +| Deep health | `GET /api/health/deep` | +| API server rollback | [tier0/api-server-rollback.md](tier0/api-server-rollback.md) | +| Agent runner rollback | [tier0/agent-runner-rollback.md](tier0/agent-runner-rollback.md) | +| Database rollback | [tier0/database-rollback.md](tier0/database-rollback.md) | +| Frontend rollback | [tier0/frontend-rollback.md](tier0/frontend-rollback.md) | +| Escalate to SEV1 | [sev1-runbook.md](sev1-runbook.md) | +| Postmortem template | [../postmortem-template.md](../postmortem-template.md) | diff --git a/docs/runbooks/tier0/agent-runner-rollback.md b/docs/runbooks/tier0/agent-runner-rollback.md new file mode 100644 index 0000000..b686096 --- /dev/null +++ b/docs/runbooks/tier0/agent-runner-rollback.md @@ -0,0 +1,115 @@ +# Rollback Runbook — Agent Runner (Strands / LiteLLM) + +**Service:** `backend/agent/` + `backend/app/services/runner.py` — the async task that executes each reconnaissance operation +**Tier:** 0 +**Owner:** On-call engineer +**Last tested:** see Git log on this file +**Related alerts:** [SEV1 runbook](../sev1-runbook.md) · [SEV2 runbook](../sev2-runbook.md) + +--- + +## 1. Rollback decision gate + +Roll back when **any** of the following is true for >5 min after a deploy: + +| Signal | Threshold | +|--------|-----------| +| Operations stuck in `running` status with no events emitted | >3 concurrent | +| `execute_operation` raises unhandled exception (visible in logs) | any occurrence | +| LLM auth errors (`401`/`403`) caused by a key/model config regression | confirmed | +| Tool hook failures causing operations to hard-fail on start | >2 in 5 min | +| WebSocket clients receive `error` frame before first tool event | >5 % of ops | + +> **Note:** LLM-provider outages are *not* rollback triggers — they are provider incidents. Distinguish by checking `GET /api/health/deep` (`model` field) before rolling back. + +--- + +## 2. Identify the previous artifact + +The agent runner is co-deployed with the API server (same Python process). The rollback artifact is the same **Git commit** as the API server. + +```bash +# Confirm current running version +git -C /path/to/Vecna-Operations-Console log --oneline -3 + +# Identify the commit before the bad deploy +PREV_SHA=$(git -C /path/to/Vecna-Operations-Console log --oneline | awk 'NR==2{print $1}') +echo "Rolling back to: $PREV_SHA" +``` + +--- + +## 3. Cancel stuck in-flight operations (before rollback) + +```bash +# List operations currently in 'running' state via API +curl -s http://127.0.0.1:8000/api/operations?status=running | jq '.[] | {id:.id, created_at:.created_at}' + +# Cancel each stuck operation +OP_ID="" +curl -s -X DELETE http://127.0.0.1:8000/api/operations/$OP_ID/cancel | jq . +``` + +> Cancellation sets the row to `error` status and closes the WebSocket stream cleanly. Users may retry after rollback. + +--- + +## 4. Rollback steps + +The agent runner lives in the same process as the API server. Follow **[api-server-rollback.md](api-server-rollback.md) §3** to restart on the previous commit. Additional agent-specific steps: + +```bash +# After restarting on PREV_SHA, confirm the agent config is intact +curl -s http://127.0.0.1:8000/api/config | jq '{model:.model_id, pricing:.tool_pricing}' + +# If a dependency (strands-agents, litellm) version was the root cause, +# pin the previous working version explicitly before restarting: +REPO=/path/to/Vecna-Operations-Console/backend +source "$REPO/.venv/bin/activate" +pip install "strands-agents==" "litellm==" --quiet +``` + +--- + +## 5. Verify rollback success + +```bash +# 1. Start a real operation (use a safe public target) +OP=$(curl -s -X POST http://127.0.0.1:8000/api/operations \ + -H "Content-Type: application/json" \ + -d '{"target_url":"https://example.com"}' | jq -r '.id') +echo "Operation: $OP" + +# 2. Poll for the first tool event (should appear within 30 s) +for i in $(seq 1 12); do + STATUS=$(curl -s http://127.0.0.1:8000/api/operations/$OP | jq -r '.status') + echo "$(date +%T) status=$STATUS" + [ "$STATUS" = "running" ] && break + sleep 5 +done + +# 3. Connect via WebSocket and verify events stream (requires wscat or similar) +# wscat -c ws://127.0.0.1:8000/ws/$OP +# Expect: tool_start events within 30 s, done/error event at end. + +# 4. Confirm operation reaches 'done' or 'error' (not stuck in 'running') +sleep 120 +curl -s http://127.0.0.1:8000/api/operations/$OP | jq '{status:.status, findings_count: (.findings_json | length)}' +``` + +--- + +## 6. Post-rollback actions + +1. Announce in incident channel: "Agent runner rolled back to `PREV_SHA`. All new operations use previous agent version." +2. Check LiteLLM and strands-agents changelogs for breaking changes if a package upgrade caused the incident. +3. File a postmortem using [docs/postmortem-template.md](../../postmortem-template.md). +4. Update "Last tested" date below after next successful drill. + +--- + +## 7. Rollback test log + +| Date | Tester | Commit rolled from → to | Result | Notes | +|------|--------|------------------------|--------|-------| +| YYYY-MM-DD | | | | Drill — first entry | diff --git a/docs/runbooks/tier0/api-server-rollback.md b/docs/runbooks/tier0/api-server-rollback.md new file mode 100644 index 0000000..fbaad8f --- /dev/null +++ b/docs/runbooks/tier0/api-server-rollback.md @@ -0,0 +1,141 @@ +# Rollback Runbook — API Server (FastAPI / Uvicorn) + +**Service:** `backend/app/main.py` — FastAPI application served by Uvicorn on port 8000 +**Tier:** 0 +**Owner:** On-call engineer +**Last tested:** see Git log on this file +**Related alerts:** [SEV1 runbook](../sev1-runbook.md) · [SEV2 runbook](../sev2-runbook.md) + +--- + +## 1. Rollback decision gate + +Roll back when **any** of the following is true for >5 min after a deploy: + +| Signal | Threshold | +|--------|-----------| +| `GET /health` returns non-200 | sustained | +| `GET /api/health/deep` reports DB or model failure | sustained | +| Operation creation (`POST /api/operations`) error rate | >10 % | +| WebSocket stream silently drops mid-run | >2 reports | +| 5xx rate on any route | >5 % over 5-min window | + +If you are unsure, roll back first and investigate on the previous artifact. + +--- + +## 2. Identify the previous artifact + +The "artifact" for this service is the **Git commit / container image** running in production. + +### 2a. Direct process (no container) + +```bash +# Running process — what commit is it? +curl -s http://127.0.0.1:8000/api/health/deep | jq . +# Check the commit the process was started from +ps aux | grep uvicorn +# Find prior commit +git -C /path/to/repo log --oneline -5 +``` + +Note the commit SHA immediately before the bad deploy — call it `PREV_SHA`. + +### 2b. Container / image (Docker) + +```bash +# List recent images tagged for this service +docker images vecna-api --format "{{.Tag}}\t{{.CreatedAt}}" | head -10 +# Identify the image before the current one +PREV_IMAGE="vecna-api:" +``` + +--- + +## 3. Rollback steps + +### 3a. Direct process rollback + +```bash +# 1. Identify where the repo lives on the host +REPO=/path/to/Vecna-Operations-Console/backend + +# 2. Kill the running Uvicorn process gracefully +PID=$(pgrep -f "uvicorn app.main:app") +kill -SIGTERM "$PID" +sleep 3 + +# 3. Check out the previous good commit +git -C "$REPO/.." checkout "$PREV_SHA" + +# 4. Re-activate venv and reinstall dependencies (in case requirements changed) +source "$REPO/.venv/bin/activate" +pip install -r "$REPO/requirements.txt" --quiet + +# 5. Restart Uvicorn +cd "$REPO" +uvicorn app.main:app --host 0.0.0.0 --port 8000 & + +# 6. Verify health +sleep 3 +curl -s http://127.0.0.1:8000/health && echo " OK" +curl -s http://127.0.0.1:8000/api/health/deep | jq '{db: .database, model: .model}' +``` + +### 3b. Container rollback + +```bash +# 1. Stop the running container +docker stop vecna-api-current + +# 2. Start the previous image +docker run -d --name vecna-api \ + --env-file /etc/vecna/.env \ + -p 8000:8000 \ + "$PREV_IMAGE" + +# 3. Verify +sleep 5 +curl -s http://127.0.0.1:8000/health && echo " OK" +``` + +--- + +## 4. Verify rollback success + +All of these must pass before the incident is resolved: + +```bash +# Health check — expect HTTP 200 with {"status":"ok"} +curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8000/health + +# Deep health — database and model connectivity +curl -s http://127.0.0.1:8000/api/health/deep | jq . + +# Smoke test: start a trivial operation and confirm it enters running state +curl -s -X POST http://127.0.0.1:8000/api/operations \ + -H "Content-Type: application/json" \ + -d '{"target_url":"https://example.com"}' | jq '{id:.id, status:.status}' + +# Confirm metrics endpoint responds +curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8000/api/metrics +``` + +Expected: all return 200; operation row transitions to `running` within 5 s. + +--- + +## 5. Post-rollback actions + +1. Announce in incident channel: "API server rolled back to `PREV_SHA`. Monitoring for 30 min." +2. File a postmortem issue using [docs/postmortem-template.md](../../postmortem-template.md). +3. Do NOT re-deploy the bad commit until root cause is identified and a fix is committed. +4. Update this file's "Last tested" date after the next successful planned rollback drill. + +--- + +## 6. Rollback test log + +| Date | Tester | Commit rolled from → to | Result | Notes | +|------|--------|------------------------|--------|-------| +| YYYY-MM-DD | | | | Drill — first entry | diff --git a/docs/runbooks/tier0/database-rollback.md b/docs/runbooks/tier0/database-rollback.md new file mode 100644 index 0000000..2689319 --- /dev/null +++ b/docs/runbooks/tier0/database-rollback.md @@ -0,0 +1,157 @@ +# Rollback Runbook — Database (SQLite) + +**Service:** `backend/vecna.db` — SQLite file managed by SQLAlchemy (`backend/app/database.py`, `backend/app/schema_migrate.py`) +**Tier:** 0 +**Owner:** On-call engineer +**Last tested:** see Git log on this file +**Related alerts:** [SEV1 runbook](../sev1-runbook.md) · [SEV2 runbook](../sev2-runbook.md) + +--- + +## 1. Rollback decision gate + +Roll back (restore from backup) when **any** of the following is true: + +| Signal | Threshold | +|--------|-----------| +| `GET /api/health/deep` reports `"database": false` | sustained >2 min | +| `sqlite3 vecna.db "PRAGMA integrity_check"` returns anything other than `ok` | any failure | +| Schema migration (`schema_migrate.py`) left the DB in an inconsistent state | confirmed | +| `Operation` rows missing expected columns after a migration | any occurrence | +| Application logs show `OperationalError` or `IntegrityError` on reads | sustained | + +> **Note:** SQLite serializes writes. High write contention causing timeouts is a scaling issue, not a rollback trigger. + +--- + +## 2. Locate the database file + +```bash +# Default path (relative to backend/) +REPO=/path/to/Vecna-Operations-Console/backend +DB="$REPO/vecna.db" + +# Confirm file exists and is readable +ls -lh "$DB" +sqlite3 "$DB" "PRAGMA integrity_check;" # expect: ok +``` + +If `DATABASE_URL` is set in `.env`, it overrides the default path — check `.env` first. + +--- + +## 3. Pre-rollback: take a snapshot of the broken state + +```bash +# Preserve the broken DB for post-incident forensics +cp "$DB" "${DB}.broken.$(date +%Y%m%dT%H%M%S)" +``` + +--- + +## 4. Rollback steps + +### 4a. Restore from a timestamped backup + +The recommended backup strategy is a periodic `cp vecna.db vecna.db.backup.` or a cron-driven `sqlite3 vecna.db ".backup vecna.db.backup"`. If that is in place: + +```bash +# Stop the API server first to release the write lock +PID=$(pgrep -f "uvicorn app.main:app") +kill -SIGTERM "$PID" +sleep 3 + +# Restore the most recent backup +BACKUP=$(ls -t "$REPO"/vecna.db.backup.* 2>/dev/null | head -1) +if [ -z "$BACKUP" ]; then + echo "ERROR: No backup found. Proceed to §4b (schema-only rollback)." + exit 1 +fi +echo "Restoring from: $BACKUP" +cp "$BACKUP" "$DB" + +# Verify restored DB +sqlite3 "$DB" "PRAGMA integrity_check;" # expect: ok +sqlite3 "$DB" "SELECT count(*) FROM operations;" + +# Restart the API server +cd "$REPO" +source .venv/bin/activate +uvicorn app.main:app --host 0.0.0.0 --port 8000 & +sleep 3 +curl -s http://127.0.0.1:8000/api/health/deep | jq '{db:.database}' +``` + +### 4b. Schema-only rollback (no data backup available) + +If a migration added a column that breaks reads, and you only need to revert the schema (acceptable data loss: the new column's data): + +```bash +# Identify columns added by the bad migration +sqlite3 "$DB" "PRAGMA table_info(operations);" +sqlite3 "$DB" "PRAGMA table_info(billing_sessions);" + +# SQLite does not support DROP COLUMN natively in older versions. +# Recreate the table without the offending column: +# (Adjust column list to match the pre-migration schema from git history) +sqlite3 "$DB" " +BEGIN; +CREATE TABLE operations_restore AS SELECT FROM operations; +DROP TABLE operations; +ALTER TABLE operations_restore RENAME TO operations; +COMMIT; +" + +# Verify +sqlite3 "$DB" "PRAGMA integrity_check;" +``` + +> **Warning:** This destroys data in any dropped column. Prefer backup restore (§4a). + +--- + +## 5. Verify rollback success + +```bash +# 1. Integrity check +sqlite3 "$DB" "PRAGMA integrity_check;" # expect: ok + +# 2. Row counts look reasonable +sqlite3 "$DB" "SELECT count(*) FROM operations; SELECT count(*) FROM billing_sessions;" + +# 3. API health +curl -s http://127.0.0.1:8000/api/health/deep | jq '{db:.database}' # expect: true + +# 4. Operations list is readable +curl -s http://127.0.0.1:8000/api/operations | jq 'length' +``` + +--- + +## 6. Backup commands (run these NOW if not already scheduled) + +```bash +# One-time manual backup +sqlite3 "$DB" ".backup ${DB}.backup.$(date +%Y%m%dT%H%M%S)" + +# Cron: every 15 minutes, keep 96 backups (~24 h) +# Add to crontab: +# */15 * * * * sqlite3 /path/to/vecna.db ".backup /path/to/backups/vecna.db.backup.$(date +\%Y\%m\%dT\%H\%M\%S)" && ls -t /path/to/backups/vecna.db.backup.* | tail -n +97 | xargs rm -f +``` + +--- + +## 7. Post-rollback actions + +1. Announce in incident channel: "Database restored from `BACKUP`. Operations after `` may be lost." +2. Inform affected users their in-progress operations were cancelled. +3. File a postmortem using [docs/postmortem-template.md](../../postmortem-template.md). +4. Schedule a backup strategy review if this incident exposed a gap. + +--- + +## 8. Rollback test log + +| Date | Tester | Scenario | Result | Notes | +|------|--------|----------|--------|-------| +| YYYY-MM-DD | | | | Drill — first entry | diff --git a/docs/runbooks/tier0/frontend-rollback.md b/docs/runbooks/tier0/frontend-rollback.md new file mode 100644 index 0000000..31fc842 --- /dev/null +++ b/docs/runbooks/tier0/frontend-rollback.md @@ -0,0 +1,137 @@ +# Rollback Runbook — Frontend (React / Vite) + +**Service:** `frontend/` — React + TypeScript SPA built with Vite, served as static files or via `npm run dev` +**Tier:** 0 +**Owner:** On-call engineer +**Last tested:** see Git log on this file +**Related alerts:** [SEV1 runbook](../sev1-runbook.md) · [SEV2 runbook](../sev2-runbook.md) + +--- + +## 1. Rollback decision gate + +Roll back the frontend when **any** of the following is true after a deploy: + +| Signal | Threshold | +|--------|-----------| +| Blank white screen for >50 % of page loads | >2 min | +| JS console shows uncaught exception on app boot | any occurrence | +| WebSocket connection to `/ws/` never established from UI | confirmed | +| Timeline or findings panel fails to render after operation completes | >5 % reports | +| URL submission returns no feedback (silent failure) | any occurrence | + +> Backend health is unaffected by a frontend rollback. Roll back only the static assets or the Vite dev server. + +--- + +## 2. Identify the previous artifact + +### 2a. Production (static build served by a web server / CDN) + +```bash +# The build artifact is dist/ committed or uploaded at deploy time. +# Find the previous deploy commit: +git -C /path/to/Vecna-Operations-Console log --oneline frontend/ -- | head -5 +PREV_SHA=$(git -C /path/to/Vecna-Operations-Console log --oneline frontend/ -- | awk 'NR==2{print $1}') +echo "Rolling back to: $PREV_SHA" +``` + +### 2b. Development / Electron (Vite dev server) + +The running process serves live source. Rolling back means checking out the previous commit and restarting Vite. + +--- + +## 3. Rollback steps + +### 3a. Static build rollback (production) + +```bash +REPO=/path/to/Vecna-Operations-Console + +# 1. Stop the current serving process or CDN deployment +# (steps depend on your hosting: nginx, Caddy, S3, etc.) + +# 2. Check out the previous frontend commit +git -C "$REPO" checkout "$PREV_SHA" -- frontend/ + +# 3. Rebuild from the previous source +cd "$REPO/frontend" +npm install --silent +npm run build # outputs to frontend/dist/ + +# 4. Re-deploy dist/ to your static host +# e.g. cp -r dist/* /var/www/vecna/ +# or: aws s3 sync dist/ s3://your-bucket/ --delete + +# 5. Hard-purge CDN cache if applicable +``` + +### 3b. Vite dev server rollback + +```bash +REPO=/path/to/Vecna-Operations-Console + +# 1. Kill the Vite process +pkill -f "vite" || true + +# 2. Check out previous frontend source +git -C "$REPO" checkout "$PREV_SHA" -- frontend/ + +# 3. Install (in case package.json changed) +cd "$REPO/frontend" +npm install --silent + +# 4. Restart Vite +npm run dev & +echo "Vite restarting on http://localhost:5173" +``` + +### 3c. Electron rollback + +```bash +# 1. Kill the Electron process +pkill -f "electron" || true + +# 2. Roll back frontend source (same as 3b above) +# 3. Rebuild frontend: cd frontend && npm run build +# 4. Restart Electron +cd /path/to/Vecna-Operations-Console/electron +npm start & +``` + +--- + +## 4. Verify rollback success + +```bash +# 1. Frontend loads without JS errors +# Open http://localhost:5173 (or your prod URL) in a browser +# Open DevTools → Console → confirm no uncaught exceptions on load + +# 2. Submit a test operation via the UI +# Enter https://example.com → Submit +# Confirm: timeline panel appears, events stream, findings render + +# 3. Confirm WebSocket connects (DevTools → Network → WS tab) +# Should see frames arriving within 5 s of submission + +# 4. Confirm Electron window renders correctly (if Electron is in use) +``` + +--- + +## 5. Post-rollback actions + +1. Announce in incident channel: "Frontend rolled back to `PREV_SHA`. Users should hard-refresh (Ctrl+Shift+R / Cmd+Shift+R)." +2. Identify which frontend change caused the regression (check `git diff $PREV_SHA HEAD -- frontend/`). +3. File a postmortem using [docs/postmortem-template.md](../../postmortem-template.md). +4. Add a smoke test or E2E check that would have caught the regression before deploy. + +--- + +## 6. Rollback test log + +| Date | Tester | Commit rolled from → to | Result | Notes | +|------|--------|------------------------|--------|-------| +| YYYY-MM-DD | | | | Drill — first entry |