Skip to content

Consolidate duplicate PrometheusRule alert definitions into alerts.yaml - #7317

Merged
hivecommons-hive[bot] merged 1 commit into
mainfrom
ops/consolidate-prometheusrule-duplication
Oct 8, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
mainfrom
ops/consolidate-prometheusrule-duplication

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

What changed

Three separate PrometheusRule manifests in cluster-objects/ (alerts.yaml, prometheusrule.yaml, prometheusrule-docs-api.yaml) alerted on the same docs_api_requests_total/docs_api_request_duration_seconds metrics with drifted alert names and thresholds — including an exact alert: name collision (DocsApiHighErrorRate defined twice, with different semantics, in two different PrometheusRule objects). This drift was already confirmed in #6884, but that issue was closed with only runbook_url cross-links added (#6885, #6936) — the actual duplication was never fixed and is still present on main today.

  • cluster-objects/alerts.yaml is now the single canonical PrometheusRule for all docs_api_*/docs_i18n_* app-metric alerts. Every distinct alert from the other two files was merged in, renamed only where needed to avoid a name collision. No threshold was loosened — the one alert dropped (DocsApiHighLatencyP95, p95>2s aggregate) was strictly subsumed by the merged-in 1s aggregate alert, which always fires at least as early.
  • cluster-objects/prometheusrule-docs-api.yaml is removed entirely (fully folded into alerts.yaml).
  • cluster-objects/prometheusrule.yaml now holds only the infra/platform alerts (DocsApiMetricsTargetDown, the two rollout-checker CronJob alerts) that reference metrics outside src/lib/metrics.ts (up{} / kube_cronjob_status_last_successful_time) — scripts/lint-dashboard.mjs's known-metric check only validates alerts.yaml against that file's declared docs_* metrics, so these can't be validated if moved there.
  • Updated runbooks/api-error-rate-latency.md, runbooks/README.md, README.md, and cluster-objects/grafana-dashboard.json's panel titles/description to match the new canonical file and alert names.

Why

On-call visibility, not a cosmetic cleanup: if more than one of these manifests is ever applied to the same cluster, a single underlying condition fires under two or three different alert names at two or three different thresholds, with no single source of truth for "is the docs API actually degraded." The exact DocsApiHighErrorRate name collision between alerts.yaml (per-route) and prometheusrule.yaml (aggregate) was the sharpest version of this.

Verification

  • npm run lint:dashboard passes (validates cluster-objects/alerts.yaml against src/lib/metrics.ts's declared metrics).
  • npx vitest run scripts/lint-dashboard-helpers.test.mjs — 26/26 passed.
  • Both remaining PrometheusRule YAML files parse and list the expected, non-colliding alert names (verified with python3 -c "import yaml; ...").
  • npx markdownlint-cli2 on the changed runbook/README files — 0 issues.

Known limitation / follow-up

The dashboard-JSON side of the same drift class (#6891, #6928) is a separate, still partly-open gap: #6928's uid collision was fixed in #6967, but #6891's panel/aggregation drift across dashboard.json/dashboard-docs-api.json/grafana-dashboard.json was not, and editing those dashboards is outside this PR's scope (grafana-dashboard.json's panel titles/description were updated only because this PR's alert renames made them stale). Filing a separate issue for that.

— hive: agent=operations backend=copilot model=claude-sonnet-4-6 copilot=1.0.88

Three separate PrometheusRule manifests in cluster-objects/ (alerts.yaml,
prometheusrule.yaml, prometheusrule-docs-api.yaml) alerted on the same
docs_api_requests_total/docs_api_request_duration_seconds metrics with
drifted names and thresholds. This was already confirmed in #6884 but
that issue was closed with only runbook cross-links added (#6885,
#6936), not an actual fix, so the duplication was still live.

- alerts.yaml is now the single canonical PrometheusRule for all
  docs_api_*/docs_i18n_* app-metric alerts. Every distinct alert from
  the other two files was merged in (renamed only to avoid an exact
  alert-name collision); no threshold was loosened. One alert
  (DocsApiHighLatencyP95, p95>2s aggregate) was dropped because it was
  strictly subsumed by the merged-in 1s aggregate alert, which always
  fires at least as early.
- prometheusrule-docs-api.yaml is removed entirely.
- prometheusrule.yaml now holds only the infra/platform alerts
  (DocsApiMetricsTargetDown, the rollout-checker CronJob alerts) that
  reference metrics outside src/lib/metrics.ts
  (up{}/kube_cronjob_status_last_successful_time), which
  scripts/lint-dashboard.mjs's known-metric check cannot validate if
  moved into alerts.yaml.
- Updated runbooks/api-error-rate-latency.md, runbooks/README.md,
  README.md, and cluster-objects/grafana-dashboard.json's panel titles/
  description to match the new canonical file and alert names.

Verified: npm run lint:dashboard passes, scripts/lint-dashboard-helpers
vitest suite passes (26/26), both remaining PrometheusRule YAML files
parse and list the expected non-colliding alert names.

The dashboard-JSON side of the same drift class (#6891, #6928) is a
separate, still partly-open gap — #6928's uid collision was fixed in
#6967, but #6891's panel/aggregation drift was not, and dashboard JSON
edits are outside this PR's scope.

Signed-off-by: operations <operations@hive.kubestellar.io>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 8, 2026
@kubestellar-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 8, 2026
@netlify

netlify Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for kubestellar-docs ready!

Name Link
🔨 Latest commit e001588
🔍 Latest deploy log https://app.netlify.com/projects/kubestellar-docs/deploys/6ac750a2634a800008018a62
😎 Deploy Preview https://deploy-preview-7317--kubestellar-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@kubestellar-prow

Copy link
Copy Markdown

Hi @hivecommons-hive[bot]. Thanks for your PR.

I'm waiting for a kubestellar member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@github-actions github-actions Bot added copilot documentation Improvements or additions to documentation json yaml labels Oct 8, 2026
@hivecommons-hive
hivecommons-hive Bot merged commit 495efc4 into main Oct 8, 2026
16 of 19 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the ops/consolidate-prometheusrule-duplication branch October 8, 2026 08:17
@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

Check out what's new:

Stay connected: Slack #kubestellar-dev | Multi-Cluster Survey

hivecommons-hive Bot added a commit that referenced this pull request Oct 22, 2026
Adds .jscpd.json (1.0% threshold, ignoring test/CSS/vendor files) and a
'npm run duplication-check' script so the duplication scan this agent has
been running manually each session (19+ merged dedupe/extract-shared PRs,
e.g. #7478, #7453, #7451, #7449, #7442, #7435, #7427, #7421, #7411, #7384,
#7382, #7368, #7352, #7337, #7325, #7317, #7313, #7488) can be run the same
way locally and in CI.

Wiring this into a .github/workflows/*.yml job is left to the tracking
issue: this agent's token lacks the workflows permission needed to push
workflow-file changes.

Signed-off-by: architect <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: architect <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

copilot dco-signoff: yes Indicates the PR's author has signed the DCO. documentation Improvements or additions to documentation json needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. yaml

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants