Repository navigation
Consolidate duplicate PrometheusRule alert definitions into alerts.yaml - #7317
Conversation
Three separate PrometheusRule manifests in cluster-objects/ (alerts.yaml, prometheusrule.yaml, prometheusrule-docs-api.yaml) alerted on the same docs_api_requests_total/docs_api_request_duration_seconds metrics with drifted names and thresholds. This was already confirmed in #6884 but that issue was closed with only runbook cross-links added (#6885, #6936), not an actual fix, so the duplication was still live. - alerts.yaml is now the single canonical PrometheusRule for all docs_api_*/docs_i18n_* app-metric alerts. Every distinct alert from the other two files was merged in (renamed only to avoid an exact alert-name collision); no threshold was loosened. One alert (DocsApiHighLatencyP95, p95>2s aggregate) was dropped because it was strictly subsumed by the merged-in 1s aggregate alert, which always fires at least as early. - prometheusrule-docs-api.yaml is removed entirely. - prometheusrule.yaml now holds only the infra/platform alerts (DocsApiMetricsTargetDown, the rollout-checker CronJob alerts) that reference metrics outside src/lib/metrics.ts (up{}/kube_cronjob_status_last_successful_time), which scripts/lint-dashboard.mjs's known-metric check cannot validate if moved into alerts.yaml. - Updated runbooks/api-error-rate-latency.md, runbooks/README.md, README.md, and cluster-objects/grafana-dashboard.json's panel titles/ description to match the new canonical file and alert names. Verified: npm run lint:dashboard passes, scripts/lint-dashboard-helpers vitest suite passes (26/26), both remaining PrometheusRule YAML files parse and list the expected non-colliding alert names. The dashboard-JSON side of the same drift class (#6891, #6928) is a separate, still partly-open gap — #6928's uid collision was fixed in #6967, but #6891's panel/aggregation drift was not, and dashboard JSON edits are outside this PR's scope. Signed-off-by: operations <operations@hive.kubestellar.io>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
✅ Deploy Preview for kubestellar-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Hi @hivecommons-hive[bot]. Thanks for your PR. I'm waiting for a kubestellar member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Thank you for your contribution! Your PR has been merged. Check out what's new:
Stay connected: Slack #kubestellar-dev | Multi-Cluster Survey |
Adds .jscpd.json (1.0% threshold, ignoring test/CSS/vendor files) and a 'npm run duplication-check' script so the duplication scan this agent has been running manually each session (19+ merged dedupe/extract-shared PRs, e.g. #7478, #7453, #7451, #7449, #7442, #7435, #7427, #7421, #7411, #7384, #7382, #7368, #7352, #7337, #7325, #7317, #7313, #7488) can be run the same way locally and in CI. Wiring this into a .github/workflows/*.yml job is left to the tracking issue: this agent's token lacks the workflows permission needed to push workflow-file changes. Signed-off-by: architect <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: architect <280983584+hivecommons-hive[bot]@users.noreply.github.com>
What changed
Three separate
PrometheusRulemanifests incluster-objects/(alerts.yaml,prometheusrule.yaml,prometheusrule-docs-api.yaml) alerted on the samedocs_api_requests_total/docs_api_request_duration_secondsmetrics with drifted alert names and thresholds — including an exactalert:name collision (DocsApiHighErrorRatedefined twice, with different semantics, in two different PrometheusRule objects). This drift was already confirmed in #6884, but that issue was closed with onlyrunbook_urlcross-links added (#6885, #6936) — the actual duplication was never fixed and is still present onmaintoday.cluster-objects/alerts.yamlis now the single canonical PrometheusRule for alldocs_api_*/docs_i18n_*app-metric alerts. Every distinct alert from the other two files was merged in, renamed only where needed to avoid a name collision. No threshold was loosened — the one alert dropped (DocsApiHighLatencyP95, p95>2s aggregate) was strictly subsumed by the merged-in 1s aggregate alert, which always fires at least as early.cluster-objects/prometheusrule-docs-api.yamlis removed entirely (fully folded intoalerts.yaml).cluster-objects/prometheusrule.yamlnow holds only the infra/platform alerts (DocsApiMetricsTargetDown, the two rollout-checker CronJob alerts) that reference metrics outsidesrc/lib/metrics.ts(up{}/kube_cronjob_status_last_successful_time) —scripts/lint-dashboard.mjs's known-metric check only validatesalerts.yamlagainst that file's declareddocs_*metrics, so these can't be validated if moved there.runbooks/api-error-rate-latency.md,runbooks/README.md,README.md, andcluster-objects/grafana-dashboard.json's panel titles/description to match the new canonical file and alert names.Why
On-call visibility, not a cosmetic cleanup: if more than one of these manifests is ever applied to the same cluster, a single underlying condition fires under two or three different alert names at two or three different thresholds, with no single source of truth for "is the docs API actually degraded." The exact
DocsApiHighErrorRatename collision betweenalerts.yaml(per-route) andprometheusrule.yaml(aggregate) was the sharpest version of this.Verification
npm run lint:dashboardpasses (validatescluster-objects/alerts.yamlagainstsrc/lib/metrics.ts's declared metrics).npx vitest run scripts/lint-dashboard-helpers.test.mjs— 26/26 passed.PrometheusRuleYAML files parse and list the expected, non-colliding alert names (verified withpython3 -c "import yaml; ...").npx markdownlint-cli2on the changed runbook/README files — 0 issues.Known limitation / follow-up
The dashboard-JSON side of the same drift class (#6891, #6928) is a separate, still partly-open gap: #6928's
uidcollision was fixed in #6967, but #6891's panel/aggregation drift acrossdashboard.json/dashboard-docs-api.json/grafana-dashboard.jsonwas not, and editing those dashboards is outside this PR's scope (grafana-dashboard.json's panel titles/description were updated only because this PR's alert renames made them stale). Filing a separate issue for that.— hive: agent=operations backend=copilot model=claude-sonnet-4-6 copilot=1.0.88