feat: export Prometheus metrics for access request usage

[dnguy078]

Problem

Oz exports only controller-runtime's built-in metrics today. Those tell you how hard the operator is working (reconcile counts, webhook latencies, workqueue depths) but nothing about how it is being used — there is no way to answer "who is requesting access, through which template, and how long did it take to become usable".

Metrics

Metric	Type	Labels
oz_access_request_created_total	counter	kind, namespace, template, user
oz_access_request_requested_duration_seconds	histogram	kind, namespace, template
oz_access_request_ready_seconds	histogram	kind, namespace, template
oz_access_request_terminated_total	counter	kind, namespace, template, reason
oz_access_request_condition_errors_total	counter	kind, namespace, template, condition
oz_access_requests	gauge	kind, namespace, template, user, ready
oz_access_templates	gauge	kind, namespace, name, ready
oz_pod_exec_total	counter	namespace, subresource, user, interactive

oz_access_request_ready_seconds measures creation → Status.Ready, so for a PodAccessRequest it covers the whole spin-up: reading the target workload's PodSpec, applying the mutation config, creating the Pod/Role/RoleBinding, and waiting for the Pod to reach Running + PodReady. AccessResourcesAreReady polls the Pod every second inside a single reconcile, so resolution is ~1–2s (1s poll, plus CreationTimestamp being second-truncated) rather than the 5s requeue interval. Past 30s the blocking loop times out and resolution degrades to roughly ±5s.

oz_access_request_condition_errors_total is per reconcile attempt, keyed on the existing RequestConditionTypes. A request wedged on a missing template keeps incrementing it — deliberately, since it turns an otherwise silent requeue loop into a rate you can alert on.

Design notes

Persisting the requester. The identity is only visible inside an admission webhook; by the time a request reaches the reconciler the "who" is gone. The mutating defaulter now stamps crds.wizardofoz.co/requested-by:

• CREATE only. The reconciler issues Updates to set owner references. Stamping on update would rewrite every request to the controller's own service account.
• Always overwrites any client-supplied value, so a user cannot attribute their request to somebody else.
• Immutable on update, enforced in ValidateUpdate. Otherwise anyone with update access could disown or misattribute their request, making the annotation worthless for reporting.

This also makes access auditable without Prometheus at all:

kubectl get podaccessrequests -A \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,TEMPLATE:.spec.templateName,USER:'.metadata.annotations.crds\.wizardofoz\.co/requested-by'

Gauges are collected, not maintained. A controller never reliably observes a resource disappearing, so an incrementally-updated GaugeVec accumulates stale label sets forever. internal/metrics/collector instead implements prometheus.Collector and lists from the manager's informer cache at scrape time — no API calls, and the series cannot go stale. A listing failure surfaces as a scrape error rather than a silent zero, which would otherwise look identical to "nobody is using Oz". It is a separate package purely to break an import cycle (v1alpha1 imports metrics; the collector imports v1alpha1).

Dry-run is not counted. The API server runs the full webhook chain for kubectl apply --dry-run=server but persists nothing, so counting those would inflate usage with grants that never happened.

The user label is opt-in (--metrics-include-user-label, chart metrics.includeUserLabel), reported as redacted otherwise. Usernames are frequently email addresses, and metrics are typically far more widely readable than the cluster itself; they also multiply the series count of the affected metrics by the number of distinct people using Oz. The annotation is populated regardless of this setting, so nothing durable is lost by leaving it off.

Chart

Adds the ServiceMonitor, which previously existed only under config/prometheus/ — chart installs had a live metrics endpoint with nothing scraping it. Off by default (metrics.serviceMonitor.create) since it needs the Prometheus Operator CRDs. The scraping ServiceAccount needs the -metrics-reader ClusterRole bound to it or every scrape returns 403; that is called out in the template and the README.

Behavior change worth reviewing

Two existing specs did request.SetAnnotations(map[string]string{"foo": "bar"}), replacing the map wholesale and wiping requested-by. The new immutability check rejects that, so they now add to the existing map, plus new specs assert the wipe is refused.

Normal read-modify-write flows (kubectl edit, kubectl annotate, controller-runtime patches) preserve annotations and are unaffected. Pre-existing requests have no annotation, so old == new == "" and they update fine. But tooling that reconstructs metadata wholesale will now get a 403.

Verification

golangci-lint, revive, gofumpt, go vet and helm lint all clean. New specs cover the label gate, Kind(), annotation set/get/immutability, the dry-run guard, and the collector (aggregation, per-label separation, redaction collapsing distinct users into one series, both kinds, and read-failure surfacing as a scrape error).

A real HTTP scrape of the registry after driving the webhook and collector paths returns 200 / 157 metric families, 8 of them oz_*:

oz_access_request_created_total{kind="PodAccessRequest",namespace="prod",template="debug-web",user="alice@example.com"} 1
oz_access_request_created_total{kind="PodAccessRequest",namespace="prod",template="debug-api",user="alice@example.com"} 1
oz_access_requests{kind="PodAccessRequest",namespace="prod",template="debug-web",user="alice@example.com",ready="true"} 1
oz_access_templates{kind="PodAccessTemplate",namespace="prod",name="never-used",ready="false"} 1
oz_pod_exec_total{namespace="prod",subresource="exec",user="alice@example.com",interactive="true"} 1

Two gaps I would rather name than paper over:

• The reconciler-side observations (ready_seconds, terminated_total, condition_errors_total) are wired by inspection and the scrape above, not by assertion. Covering them properly means driving envtest through a full grant-then-expire cycle. The webhook and collector paths are asserted.
• internal/testing/e2e was not run — it needs bin/kustomize and a live cluster, and fails identically on unmodified main in this environment. The TLS/authn layer and the ServiceMonitor actually reaching the pod are consequently unverified.

Note for operators

Both gauges are computed per replica, so with controllerManager.replicas > 1 aggregate them with max by (...) rather than sum by (...). Counters and histograms are unaffected — only the leader reconciles, and each webhook call is handled once. Documented in the README.

🤖 Generated with Claude Code

[coveralls]

Coverage Report for CI Build 30595958740

Coverage increased (+5.0%) to 40.873%

Details

• Coverage increased (+5.0%) from the base build.
• Patch coverage: 21 uncovered changes across 2 files (234 of 255 lines covered, 91.76%).
• No coverage regressions found.

Uncovered Changes

File	Changed	Covered	%
internal/cmd/manager/main.go	16	0	0.0%
internal/metrics/collector/collector.go	86	81	94.19%
Total (16 files)	255	234	91.76%

Coverage Regressions

No coverage regressions found.

---

Coverage Stats

Relevant Lines:	3139
Covered Lines:	1283
Line Coverage:	40.87%
Coverage Strength:	2.03 hits per line

---

💛 - Coveralls