Add incident-response runbooks, SEV1/SEV2 runbooks, and postmortem template

[bdevierno1]

Summary

• Tier-0 rollback runbooks (docs/runbooks/tier0/) — one per service (API server, agent runner, database, frontend). Each includes: decision gate (when to roll back), artifact identification, step-by-step rollback commands, verification smoke tests, and a test-log table for drill tracking.
• SEV1 / SEV2 runbooks (docs/runbooks/sev1-runbook.md, sev2-runbook.md) — five-step response procedures (acknowledge → assess → contain → communicate → resolve) with SLA targets and rollback dispatch tables linking to the Tier-0 runbooks above.
• Alert definitions (docs/runbooks/alert-definitions.md) — seven named alerts (A01–A07) mapping signal/threshold/severity to runbook URLs; Prometheus annotation example included so each alert can carry runbook_url pointing here.
• Blameless postmortem template (docs/postmortem-template.md) + docs/postmortems/ directory — covers summary, timeline, root cause, contributing factors, what went well/wrong, where we got lucky, action items with owners, and detection gap analysis.

Why

The release-readiness checklist (rollback path requirement) and the incident-severity-and-response runbook both require that every Tier-0 service has a documented, testable rollback path and that SEV1/SEV2 alerts link to runbooks. Neither existed before this PR.

How to verify

1. Runbook links are internally consistent — follow every cross-reference in sev1-runbook.md and sev2-runbook.md; they should resolve to the correct tier0/ files.
2. Rollback steps are executable — on a local dev copy, run the smoke-test commands in each tier0/*.md §4 Verify section against http://127.0.0.1:8000 with uvicorn running.
3. Postmortem template renders — open docs/postmortem-template.md in the GitHub UI; all table columns and code blocks should render cleanly.
4. Alert definitions are complete — confirm each of A01–A07 in alert-definitions.md references a real endpoint (/health, /api/health/deep, etc.) that exists in backend/app/main.py.

Rollback

This PR adds documentation only — no code changes. It is safe to revert the entire branch with git revert --no-commit HEAD if the runbook structure needs to change.

Out of scope

• Automation of the rollback commands (deploy pipeline integration) — separate task.
• Wiring alerts to an actual monitoring tool (Prometheus / PagerDuty) — depends on infra decisions outside this repo.
• Rollback drill scheduling — recommend adding a calendar event after merge.

---

🤖 Generated with Vecna

Co-Authored-By: Vecna noreply@vecna.ai