feat(obs): AUTH-004 synthetic prober alert + dashboard + catalog - #36
Merged
Merged
Conversation
Adds the rule-25 observability surface for the worker AUTH-004 synthetic prober (worker repo PR #68): - NR alert (P0): newrelic/alerts/auth-probe-fail.json fires CRITICAL on any instant_auth_probe_outcome_total{result="fail"} over a 10m window (two consecutive 5-min ticks of fail). - Prom rule: k8s/prometheus-rules.yaml::AuthProbeFail (same shape). - Dashboard tiles: 3 new tiles on instanode-reliability.json — per-leg outcomes line, fails billboard (CRITICAL threshold ≥1), P95 latency per leg. - Catalog rows: instant_auth_probe_outcome_total + instant_auth_probe_latency_seconds in observability/METRICS-CATALOG.md. Lands in lockstep with worker PR #68 per CLAUDE.md rule 25 ("every new metric ships with its alert + dashboard tile in the same PR"). The two repos auto-deploy on separate workflows so they ship as two PRs against their respective masters. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
newrelic/alerts/auth-probe-fail.json— CRITICAL on anyinstant_auth_probe_outcome_total{result=\"fail\"}over 10m (two consecutive 5-min ticks of fail).k8s/prometheus-rules.yaml::AuthProbeFail.instant_auth_probe_outcome_total+instant_auth_probe_latency_secondsinobservability/METRICS-CATALOG.md.Lands in lockstep with worker PR #68 per CLAUDE.md rule 25 ("every new metric ships with its alert + dashboard tile in the same PR"). The two repos auto-deploy on separate workflows so they ship as two PRs against their respective masters.
Test plan
/metricsshowsinstant_auth_probe_outcome_totalbefore this PR is applied.kubectl apply -f infra/k8s/prometheus-rules.yaml -n monitoring(or operator reconciliation) loadsAuthProbeFail.kubectl get prometheusrule -n instant instant-platform -o yaml | grep AuthProbeFailreturns the rule.newrelic/apply.shrerun: dashboardinstanode — reliabilityshows the 3 new tiles; alertinstant-worker — auth_probe_failedis active and enabled.AUTH_PROBE_BEARER_TOKENk8s secret so leg 3 stops reportingdegraded.🤖 Generated with Claude Code