Skip to content

🌱 Add bounded Prometheus metrics to auth rate-limit and revocation - #24276

Merged
hivecommons-hive[bot] merged 2 commits into
mainfrom
telemetry/auth-middleware-metrics
Oct 13, 2026
Merged

hivecommons-hive[bot] merged 2 commits into
mainfrom
telemetry/auth-middleware-metrics

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Fixes #24275

Summary

pkg/api/middleware's FailureTracker (login brute-force rate limiting) and revokedTokenCache (JWT revocation) have security-relevant progressive logic — escalate/soft-lock/hard-lock tiers, bounded LRU-style eviction, background cleanup goroutines — but previously emitted zero Prometheus metrics, only slog lines. Adds pkg/api/middleware/metrics.go with six bounded metrics:

  • console_auth_rate_limit_failures_total{tier} — incremented on every RecordFailure, labeled by the resulting tier (normal/escalate/soft-lock/hard-lock)
  • console_auth_rate_limit_tracked_keys — gauge, refreshed after each cleanup sweep
  • console_auth_rate_limit_purged_total — counter of stale records purged
  • console_auth_token_revocations_total — counter of Revoke() calls
  • console_auth_revoked_tokens_cache_size — gauge, refreshed on Revoke()/cleanup()
  • console_auth_revoked_tokens_evicted_total{reason} — labeled by fixed reason (expired/capacity/stale-backfill)

No raw IP, user ID, or JTI is ever used as a label — all labels are fixed, small enum values already computed by existing code (TierName, the two eviction passes in Revoke(), the two passes in cleanup()).

Testing

  • go build ./...
  • go test ./pkg/api/middleware/... (new metrics_test.go covers tier-labeled counting, purge/gauge updates, revocation counting, and eviction-reason labeling) — coverage 91.2%, above the 90% package floor
  • go vet ./pkg/api/middleware/...

No web/frontend files were touched by this change.

— hive: agent=telemetry backend=copilot model=claude-sonnet-4-6

The login brute-force FailureTracker (progressive escalate/soft-lock/
hard-lock tiers, #8676) and the JWT revokedTokenCache (#4759/#6577/
#6578) previously emitted only slog lines, so there was no way to see
failure pressure trending toward a hard-lock, or revocation-cache
growth approaching its cap, without grepping logs.

Adds pkg/api/middleware/metrics.go with six bounded metrics covering
both: failure counts by tier, tracked-key/cache-size gauges, and
purge/eviction counters by fixed reason. No raw IP, user ID, or JTI
is ever used as a label.

Fixes #24275

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: telemetry <telemetry@hive.kubestellar.io>
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign mikespreitzer for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 13, 2026
@netlify

netlify Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for kubestellarconsole canceled.

Name Link
🔨 Latest commit 6c31169
🔍 Latest deploy log https://app.netlify.com/projects/kubestellarconsole/deploys/6acdd3da2c8778501cd42ebc

@github-actions

Copy link
Copy Markdown
Contributor

👋 Hey @hivecommons-hive[bot] — thanks for opening this PR!

🤖 This project is developed exclusively using AI coding assistants.

Please do not attempt to code anything for this project manually.
All contributions should be authored using an AI coding tool such as:

This ensures consistency in code style, architecture patterns, test coverage,
and commit quality across the entire codebase.


This is an automated message.

@github-actions github-actions Bot added ai-generated Pull request generated by AI tier/3-restricted labels Oct 13, 2026
@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 13, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: telemetry <telemetry@hive.kubestellar.io>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against the tree at head 6c31169 — this looks correct to me.

Verified:

  • TierName(rec.Count) is called after the increment in RecordFailure (pkg/api/middleware/ratelimit.go:76), so the tier label reflects the resulting tier, matching the metric's Help text. The new-record path correctly labels TierName(1) = "normal".
  • Every gauge update is inside the owning lock: authRateLimitTrackedKeys.Set under ft.mu in purgeStale (ratelimit.go:195-211), and both authRevokedTokensCacheSize.Set calls before c.Unlock() in Revoke/cleanup (auth_revocation.go:149, :258) — gauges can't go stale relative to the map they mirror.
  • Label cardinality is bounded: tier is one of four fixed strings from TierName, reason one of three literals. No IP/user/JTI labels, as the body claims.
  • Registration is sync.Once-guarded and hooked into both construction paths (NewFailureTracker, InitTokenRevocation), so repeated test construction can't panic MustRegister. No name collision with existing console_* metrics elsewhere in the tree.
  • Metrics are visible in production: the console serves the default registry via promhttp.Handler() (pkg/api/metrics/metrics.go:488).
  • pkg/api/middleware carries a 90.0 floor in .github/go-package-coverage-ratchet.txt:83; the PR adds tests alongside the new code and claims 91.2%.

One minor note (not a defect): TestPurgeStale_UpdatesGaugeAndCounter reads ft.failures without holding ft.mu while the tracker's cleanup goroutine is still alive. The ticker fires every 5 minutes so it can't realistically interleave in a test, but taking the lock for that read would make -race runs unconditionally safe.

Approval is a human's signature, so just flagging for a maintainer: metrics-only change to auth middleware, no behavioral change to the auth paths themselves.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit e562dc7 into main Oct 13, 2026
33 of 34 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the telemetry/auth-middleware-metrics branch October 13, 2026 07:33
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author
Evidence bundle `#242766c31169` sha256:050f4bf16767cde846d76d865122f1fcda5c53ef423def815866088bb6f36979 — download from the dashboard or `hivectl review evidence #24276`.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

Check out what's new:

Stay connected: Slack #kubestellar-dev | Multi-Cluster Survey

@github-actions

Copy link
Copy Markdown
Contributor

Post-merge build verification passed ✅

Both Go and frontend builds compiled successfully against merge commit e562dc79a554c3dd84331f5b1aae07da5bc0e652.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Post-Merge Verification: passed

Commit: e562dc79a554c3dd84331f5b1aae07da5bc0e652
Specs run: smoke.spec.ts
Report: https://github.com/kubestellar/console/actions/runs/38363422166

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-generated Pull request generated by AI dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. tier/3-restricted

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[telemetry] auth rate-limit tracker and token-revocation cache have rich logic but zero metrics

0 participants