Repository navigation
🌱 Add bounded Prometheus metrics to auth rate-limit and revocation - #24276
Conversation
The login brute-force FailureTracker (progressive escalate/soft-lock/ hard-lock tiers, #8676) and the JWT revokedTokenCache (#4759/#6577/ #6578) previously emitted only slog lines, so there was no way to see failure pressure trending toward a hard-lock, or revocation-cache growth approaching its cap, without grepping logs. Adds pkg/api/middleware/metrics.go with six bounded metrics covering both: failure counts by tier, tracked-key/cache-size gauges, and purge/eviction counters by fixed reason. No raw IP, user ID, or JTI is ever used as a label. Fixes #24275 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: telemetry <telemetry@hive.kubestellar.io>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
✅ Deploy Preview for kubestellarconsole canceled.
|
|
👋 Hey @hivecommons-hive[bot] — thanks for opening this PR!
This is an automated message. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: telemetry <telemetry@hive.kubestellar.io>
There was a problem hiding this comment.
Read the diff against the tree at head 6c31169 — this looks correct to me.
Verified:
TierName(rec.Count)is called after the increment inRecordFailure(pkg/api/middleware/ratelimit.go:76), so thetierlabel reflects the resulting tier, matching the metric's Help text. The new-record path correctly labelsTierName(1)= "normal".- Every gauge update is inside the owning lock:
authRateLimitTrackedKeys.Setunderft.muinpurgeStale(ratelimit.go:195-211), and bothauthRevokedTokensCacheSize.Setcalls beforec.Unlock()inRevoke/cleanup(auth_revocation.go:149, :258) — gauges can't go stale relative to the map they mirror. - Label cardinality is bounded:
tieris one of four fixed strings fromTierName,reasonone of three literals. No IP/user/JTI labels, as the body claims. - Registration is
sync.Once-guarded and hooked into both construction paths (NewFailureTracker,InitTokenRevocation), so repeated test construction can't panicMustRegister. No name collision with existingconsole_*metrics elsewhere in the tree. - Metrics are visible in production: the console serves the default registry via
promhttp.Handler()(pkg/api/metrics/metrics.go:488). pkg/api/middlewarecarries a 90.0 floor in.github/go-package-coverage-ratchet.txt:83; the PR adds tests alongside the new code and claims 91.2%.
One minor note (not a defect): TestPurgeStale_UpdatesGaugeAndCounter reads ft.failures without holding ft.mu while the tracker's cleanup goroutine is still alive. The ticker fires every 5 minutes so it can't realistically interleave in a test, but taking the lock for that read would make -race runs unconditionally safe.
Approval is a human's signature, so just flagging for a maintainer: metrics-only change to auth middleware, no behavioral change to the auth paths themselves.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.88
|
Thank you for your contribution! Your PR has been merged. Check out what's new:
Stay connected: Slack #kubestellar-dev | Multi-Cluster Survey |
|
Post-merge build verification passed ✅ Both Go and frontend builds compiled successfully against merge commit |
✅ Post-Merge Verification: passedCommit: |
Fixes #24275
Summary
pkg/api/middleware'sFailureTracker(login brute-force rate limiting) andrevokedTokenCache(JWT revocation) have security-relevant progressive logic — escalate/soft-lock/hard-lock tiers, bounded LRU-style eviction, background cleanup goroutines — but previously emitted zero Prometheus metrics, onlysloglines. Addspkg/api/middleware/metrics.gowith six bounded metrics:console_auth_rate_limit_failures_total{tier}— incremented on everyRecordFailure, labeled by the resulting tier (normal/escalate/soft-lock/hard-lock)console_auth_rate_limit_tracked_keys— gauge, refreshed after each cleanup sweepconsole_auth_rate_limit_purged_total— counter of stale records purgedconsole_auth_token_revocations_total— counter ofRevoke()callsconsole_auth_revoked_tokens_cache_size— gauge, refreshed onRevoke()/cleanup()console_auth_revoked_tokens_evicted_total{reason}— labeled by fixed reason (expired/capacity/stale-backfill)No raw IP, user ID, or JTI is ever used as a label — all labels are fixed, small enum values already computed by existing code (
TierName, the two eviction passes inRevoke(), the two passes incleanup()).Testing
go build ./...go test ./pkg/api/middleware/...(newmetrics_test.gocovers tier-labeled counting, purge/gauge updates, revocation counting, and eviction-reason labeling) — coverage 91.2%, above the 90% package floorgo vet ./pkg/api/middleware/...No web/frontend files were touched by this change.
— hive: agent=telemetry backend=copilot model=claude-sonnet-4-6