Feature/846 prometheus alerting rules

[dot-enny]

Closes #846

Summary of Changes

1. New Git Branch Created:

   • Created and switched to feature/846-prometheus-alerting-rules.

2. PrometheusRule Custom Resource (charts/teachlink-backend/templates/prometheus-rules.yaml):

   • Created Helm template defining PrometheusRule (monitoring.coreos.com/v1) with SLA alerts:
     • HighErrorRate: Triggers when 5xx HTTP error rate exceeds 1% over 5 minutes.
     • HighP99Latency: Triggers when P99 HTTP request response latency exceeds 1 second over 10 minutes.
     • QueueDepthHigh: Triggers when queue depth exceeds 1,000 waiting jobs over 10 minutes.
     • DLQDepthHigh: Triggers when dead-letter queue contains failed jobs over 5 minutes.
   • Included direct runbook URL annotations pointing to the corresponding sections in docs/RUNBOOKS.md.

3. Helm Values Configuration (charts/teachlink-backend/values.yaml):

   • Added prometheusRule configuration block enabling the alerting rules template, severities, and runbook base URL.
   • Added alertmanager configuration block with webhook endpoint and Slack notification routing options.

4. Alert Runbook Documentation (docs/RUNBOOKS.md):

   • Documented sections 5–8 for HighErrorRate, HighP99Latency, QueueDepthHigh, and DLQDepthHigh detailing descriptions, SLA impacts, and step-by-step mitigation procedures.

5. Automated Unit Testing (src/monitoring/prometheus-rules.spec.ts):

   • Added unit test suite validating the existence, PromQL expression thresholds, duration rules, runbook link formatting, and Helm values configuration.
   • All 7 tests passed cleanly.

All tests passed cleanly. Resolution is complete and verified.

[drips-wave]

@dot-enny Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits