Repository navigation
[operations] docs: add runbook for DocsApiHighErrorRate/Latency alerts - #6753
Conversation
Both alerts in cluster-objects/prometheusrule.yaml (added in #6746) had no runbook_url annotation and no dedicated diagnosis/remediation doc, unlike every other alerting mechanism in this repo (generate-leaderboard, generate-acmm-history, scorecard/stale-workflow monitoring), which all link a runbook. Add runbooks/api-error-rate-latency.md covering detection, likely root causes for the two instrumented routes (search, docs-image), and recovery steps, link it via runbook_url on both alerts, and index it from README.md alongside the other operational runbooks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: copilot-operations <223556219+Copilot@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
✅ Deploy Preview for kubestellar-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Hi @kubestellar-hive[bot]. Thanks for your PR. I'm waiting for a kubestellar member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Thank you for your contribution! Your PR has been merged. Check out what's new:
Stay connected: Slack #kubestellar-dev | Multi-Cluster Survey |
Summary
cluster-objects/prometheusrule.yaml(added in #6746) defines twoalerts —
DocsApiHighErrorRateandDocsApiHighRequestLatency— overthe docs site's instrumented
search/docs-imageAPI routes, butneither had a
runbook_urlannotation or a dedicated diagnosis doc.Every other alerting mechanism in this repo (
generate-leaderboard.yml,generate-acmm-history.yml, and the scorecard/stale-workflow monitoringrunbooks) links a runbook — these two alerts were the outlier.
runbooks/api-error-rate-latency.md: detection/diagnosis stepsspecific to the two instrumented routes (what a 500 vs. elevated
latency means for
searchvs.docs-image, and when to insteadfollow
runbooks/deploy-rollback.mdfor a shared root cause), plusrecovery guidance.
runbook_urlannotation to both alerts pointing at the new doc.Why this is operations scope
This is documentation and an alert annotation only — no alert threshold,
condition, or severity is changed or weakened, and no exporter/external
data flow is added. Matches the allowed "user-impact alert rules with
runbook links" and "runbooks/*.md" scope.
Validation
python3 -c "import yaml; yaml.safe_load(open('cluster-objects/prometheusrule.yaml'))"passes.src/app/api/search/route.tsandsrc/app/api/docs-image/[...path]/route.tsto confirm thedocumented 500-status causes match the actual
catchhandling ineach route.
I will not merge this PR myself.
— hive: agent=operations backend=copilot model=claude-sonnet-4-6