feat: export Prometheus metrics for access request usage

README.md

@@ -451,6 +451,92 @@ sequenceDiagram
     end
 ```
 
+## Metrics
+
+The controller serves Prometheus metrics on `:8443/metrics`. The endpoint is
+protected by [controller-runtime][cr]'s built-in authentication and
+authorization, so scrapers must present a bearer token belonging to a
+ServiceAccount that has the `<release>-metrics-reader` ClusterRole bound to it.
+
+Alongside the standard controller-runtime metrics (reconcile counts, webhook
+latencies, workqueue depths), **Oz** exports metrics describing how it is being
+*used*:
+
+| Metric | Type | Labels | Description |
+| ------ | ---- | ------ | ----------- |
+| `oz_access_request_created_total` | counter | `kind`, `namespace`, `template`, `user` | Access requests admitted by the validating webhook. |
+| `oz_access_request_requested_duration_seconds` | histogram | `kind`, `namespace`, `template` | Duration explicitly asked for in `spec.duration`. Requests that inherit the template default are not observed. |
+| `oz_access_request_ready_seconds` | histogram | `kind`, `namespace`, `template` | Time from creation to `Status.Ready`, ie how long the user waited before they could use their access. |
+| `oz_access_request_terminated_total` | counter | `kind`, `namespace`, `template`, `reason` | Requests that reached a terminal state. `reason` is `expired` (the happy path), `duration_invalid` or `duration_too_long`. |
+| `oz_access_request_condition_errors_total` | counter | `kind`, `namespace`, `template`, `condition` | Reconcile *attempts* that failed a verification step, labelled with the `Status.Condition` set to `False`. A permanently wedged request keeps incrementing this. |
+| `oz_access_requests` | gauge | `kind`, `namespace`, `template`, `user`, `ready` | Access requests that exist right now. |
+| `oz_access_templates` | gauge | `kind`, `namespace`, `name`, `ready` | Access templates that exist right now. |
+| `oz_pod_exec_total` | counter | `namespace`, `subresource`, `user`, `interactive` | `exec`/`attach` operations seen by the Pod watcher webhook. Covers all such traffic in the cluster, not just Oz-managed Pods. |
+
+Some examples:
+
+```promql
+# Which templates are being used, and by whom, over the last week?
+sum by (template, user) (increase(oz_access_request_created_total[7d]))
+
+# Who currently holds access? Note the `max` rather than `sum` - see the
+# caveat about replicas below.
+max by (user, template) (oz_access_requests{ready="true"})
+
+# Templates that nobody has requested access through in the last 30 days.
+# Note the join is on (namespace, name) only - the `kind` label differs between
+# the two metrics ("PodAccessTemplate" vs "PodAccessRequest").
+oz_access_templates
+  unless on (namespace, name)
+  label_replace(
+    sum by (namespace, template) (increase(oz_access_request_created_total[30d])) > 0,
+    "name", "$1", "template", "(.*)"
+  )
+
+# 95th percentile wait for a Pod-based grant to become usable
+histogram_quantile(0.95, sum by (le, template) (rate(oz_access_request_ready_seconds_bucket[1h])))
+```
+
+The two gauges (`oz_access_requests` and `oz_access_templates`) are computed by
+listing from the controller's cache at scrape time, so every replica reports the
+same inventory. If you run `controllerManager.replicas > 1`, aggregate them with
+`max by (...)` rather than `sum by (...)`, or the counts will be multiplied by
+the replica count. The counters and histograms are not affected - only the
+elected leader reconciles, and each webhook call is handled once.
+
+### Attributing usage to users
+
+The requesting user's identity is only visible inside an admission webhook, so
+**Oz** stamps it onto each request as the `crds.wizardofoz.co/requested-by`
+annotation during mutation. The annotation is set from authoritative API server
+data on create (overwriting any client-supplied value) and is rejected from
+being modified on update, so it is trustworthy for reporting - and it means you
+can also audit access with plain `kubectl`:
+
+```bash
+kubectl get podaccessrequests -A \
+  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,TEMPLATE:.spec.templateName,USER:'.metadata.annotations.crds\.wizardofoz\.co/requested-by'
+```
+
+The `user` **metric label** is opt-in, and reported as `redacted` until you turn
+it on:
+
+```yaml
+# values.yaml
+metrics:
+  includeUserLabel: true
+  serviceMonitor:
+    create: true
+```
+
+It defaults to off for two reasons: usernames are frequently email addresses
+(ie personally identifying, and metrics tend to be far more widely readable than
+the cluster itself), and they multiply the series count of the affected metrics
+by the number of distinct people using **Oz**. The annotation is always
+populated regardless of this setting.
+
+[cr]: https://github.com/kubernetes-sigs/controller-runtime
+
 ## License
 
 Copyright 2022 Matt Wise.

charts/oz/README.md

@@ -42,6 +42,12 @@ Kubernetes: `>=1.26.0-0`
 | controllerManager.replicas | `int` | `1` | Number of Oz Controllers to run. If more than one is used, leader-election is used to ensure only one controller is operating at a time. |
 | controllerManager.tolerations | `[]map]` | `[]` | A list of Tolerations that will be applied to the controller-manager pods. See https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/. |
 | kubernetesClusterDomain | `string` | `"cluster.local"` | Configures the KUBERNETES_CLUSTER_DOMAIN environment variable. |
+| metrics.includeUserLabel | `bool` | `false` | Report real Kubernetes usernames in the `user` label of the `oz_*` usage metrics. Without this, per-user attribution is unavailable and the label is reported as `redacted`. This is opt-in because usernames are frequently email addresses (ie personally identifying), and because they are considerably higher cardinality than the rest of the label set - the series count of the affected metrics is multiplied by the number of distinct people using Oz. |
+| metrics.serviceMonitor.create | `bool` | `false` | Whether or not to create a Prometheus Operator `ServiceMonitor` resource pointing at the metrics service. Requires the Prometheus Operator CRDs to be installed in the cluster. If you scrape via some other mechanism, leave this off and point your scraper at the `-controller-manager-metrics-service` Service on port 8443. |
+| metrics.serviceMonitor.interval | `string` | `nil` | How often Prometheus should scrape the controller. Defaults to the Prometheus global scrape interval when unset. |
+| metrics.serviceMonitor.labels | `map` | `{}` | Additional labels to apply to the `ServiceMonitor`. Most Prometheus Operator installations use a label selector to decide which ServiceMonitors to act on (eg `release: kube-prometheus-stack`), in which case you must set the matching label here. |
+| metrics.serviceMonitor.metricRelabelings | `[]map` | `[]` | Optional metric relabeling rules, applied by Prometheus after scraping. Useful for dropping high cardinality series you do not want to keep. |
+| metrics.serviceMonitor.scrapeTimeout | `string` | `nil` | Per-scrape timeout. Defaults to the Prometheus global scrape timeout when unset. |
 | metricsService.ports[0].name | string | `"https"` |  |
 | metricsService.ports[0].port | int | `8443` |  |
 | metricsService.ports[0].protocol | string | `"TCP"` |  |

charts/oz/templates/deployment.yaml

@@ -62,6 +62,9 @@ spec:
           - --health-probe-bind-address=:8081
           - --metrics-bind-address=:8443
           - --leader-elect
+          {{- if .Values.metrics.includeUserLabel }}
+          - --metrics-include-user-label
+          {{- end }}
         securityContext:
           allowPrivilegeEscalation: false
           capabilities:

charts/oz/templates/servicemonitor.yaml

@@ -0,0 +1,56 @@
+{{- if .Values.metrics.serviceMonitor.create }}
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: {{ include "oz.fullname" . }}-controller-manager-metrics-monitor
+  namespace: {{ .Release.Namespace }}
+  {{- /*
+    Only the labels that "oz.labels" does not already provide are listed here.
+    The sibling templates repeat several of them, but since "oz.labels" is
+    included last it wins, so the rendered output is unchanged and this avoids
+    emitting duplicate YAML keys.
+  */}}
+  labels:
+    app.kubernetes.io/component: metrics
+    control-plane: controller-manager
+  {{- include "oz.labels" . | nindent 4 }}
+  {{- with .Values.metrics.serviceMonitor.labels }}
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+spec:
+  namespaceSelector:
+    matchNames:
+      - {{ .Release.Namespace }}
+  selector:
+    matchLabels:
+      control-plane: controller-manager
+      {{- include "oz.selectorLabels" . | nindent 6 }}
+  endpoints:
+    - path: /metrics
+      port: https
+      scheme: https
+      {{- with .Values.metrics.serviceMonitor.interval }}
+      interval: {{ . }}
+      {{- end }}
+      {{- with .Values.metrics.serviceMonitor.scrapeTimeout }}
+      scrapeTimeout: {{ . }}
+      {{- end }}
+      {{- /*
+        The metrics endpoint is protected by controller-runtime's built-in
+        authn/authz (SecureServing + WithAuthenticationAndAuthorization), so the
+        scraper must present a bearer token. The ServiceAccount Prometheus runs
+        as needs the `-metrics-reader` ClusterRole bound to it, otherwise every
+        scrape returns 403.
+      */}}
+      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
+      {{- /*
+        controller-runtime generates a self-signed serving certificate on
+        startup, so there is no CA for Prometheus to verify against.
+      */}}
+      tlsConfig:
+        insecureSkipVerify: true
+      {{- with .Values.metrics.serviceMonitor.metricRelabelings }}
+      metricRelabelings:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+{{- end }}

charts/oz/values.yaml

@@ -40,6 +40,47 @@ metricsService:
       protocol: TCP
       targetPort: https
 
+# Configuration for the Prometheus metrics that the Oz controller exports. In
+# addition to the standard controller-runtime metrics (reconcile counts,
+# webhook latencies, workqueue depths), Oz exports usage metrics that attribute
+# access requests to templates and users - see the `oz_*` metrics.
+metrics:
+  # -- (`bool`) Report real Kubernetes usernames in the `user` label of the
+  # `oz_*` usage metrics. Without this, per-user attribution is unavailable and
+  # the label is reported as `redacted`. This is opt-in because usernames are
+  # frequently email addresses (ie personally identifying), and because they
+  # are considerably higher cardinality than the rest of the label set - the
+  # series count of the affected metrics is multiplied by the number of
+  # distinct people using Oz.
+  includeUserLabel: false
+
+  serviceMonitor:
+    # -- (`bool`) Whether or not to create a Prometheus Operator
+    # `ServiceMonitor` resource pointing at the metrics service. Requires the
+    # Prometheus Operator CRDs to be installed in the cluster. If you scrape
+    # via some other mechanism, leave this off and point your scraper at the
+    # `-controller-manager-metrics-service` Service on port 8443.
+    create: false
+
+    # -- (`map`) Additional labels to apply to the `ServiceMonitor`. Most
+    # Prometheus Operator installations use a label selector to decide which
+    # ServiceMonitors to act on (eg `release: kube-prometheus-stack`), in which
+    # case you must set the matching label here.
+    labels: {}
+
+    # -- (`string`) How often Prometheus should scrape the controller. Defaults
+    # to the Prometheus global scrape interval when unset.
+    interval:
+
+    # -- (`string`) Per-scrape timeout. Defaults to the Prometheus global
+    # scrape timeout when unset.
+    scrapeTimeout:
+
+    # -- (`[]map`) Optional metric relabeling rules, applied by Prometheus
+    # after scraping. Useful for dropping high cardinality series you do not
+    # want to keep.
+    metricRelabelings: []
+
 # Configuration for the oz-controller-manager-webhook-service, used for
 # handling ValidatingWebhookConfiguration and MutatingWebhookConfiguration
 # calls from the Kubernetes API.

go.mod

@@ -12,6 +12,7 @@ require (
 	github.com/ivanpirog/coloredcobra v1.0.1
 	github.com/onsi/ginkgo/v2 v2.28.1
 	github.com/onsi/gomega v1.39.1
+	github.com/prometheus/client_golang v1.23.2
 	github.com/spf13/cobra v1.10.2
 	go.uber.org/zap v1.27.1
 	k8s.io/api v0.35.4
@@ -56,6 +57,7 @@ require (
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/josharian/intern v1.0.0 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
+	github.com/kylelemons/godebug v1.1.0 // indirect
 	github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect
 	github.com/mailru/easyjson v0.9.0 // indirect
 	github.com/mattn/go-colorable v0.1.14 // indirect
@@ -67,7 +69,6 @@ require (
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
 	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
-	github.com/prometheus/client_golang v1.23.2 // indirect
 	github.com/prometheus/client_model v0.6.2 // indirect
 	github.com/prometheus/common v0.66.1 // indirect
 	github.com/prometheus/procfs v0.16.1 // indirect

internal/api/v1alpha1/exec_access_request_test.go

@@ -83,11 +83,41 @@ var _ = Describe("ExecAccessRequest", Ordered, func() {
 			}, request)
 			Expect(err).To(Not(HaveOccurred()))
 
-			// Update it and push it
-			request.SetAnnotations(map[string]string{"foo": "bar"})
+			// Update it and push it. Note that we add an annotation rather
+			// than replacing the map wholesale - the requested-by annotation is
+			// immutable, so wiping it out is rejected (see below).
+			annotations := request.GetAnnotations()
+			annotations["foo"] = "bar"
+			request.SetAnnotations(annotations)
 			err = k8sClient.Update(ctx, request)
 			Expect(err).To(Not(HaveOccurred()))
 		})
+
+		It("ExecAccessRequest records the requesting user", func() {
+			request := &ExecAccessRequest{}
+			err := k8sClient.Get(ctx, types.NamespacedName{
+				Name:      requestName,
+				Namespace: template.Namespace,
+			}, request)
+			Expect(err).To(Not(HaveOccurred()))
+			Expect(GetRequestedBy(request)).To(Not(BeEmpty()))
+		})
+
+		It("ExecAccessRequest Update - Rejects a change to the requested-by annotation", func() {
+			request := &ExecAccessRequest{}
+			err := k8sClient.Get(ctx, types.NamespacedName{
+				Name:      requestName,
+				Namespace: template.Namespace,
+			}, request)
+			Expect(err).To(Not(HaveOccurred()))
+
+			// Wiping the annotations (or editing just this one) would let a
+			// user disown or misattribute their own access request.
+			request.SetAnnotations(map[string]string{"foo": "bar"})
+			err = k8sClient.Update(ctx, request)
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring(AnnotationRequestedBy))
+		})
 		It("ExecAccessRequest Delete - Passes Webhook ValidateUpdate() Call", func() {
 			// Get the request first
 			request := &ExecAccessRequest{

internal/api/v1alpha1/exec_access_request_webhook.go

@@ -51,7 +51,10 @@ func (r *ExecAccessRequest) SetupWebhookWithManager(mgr ctrl.Manager) error {
 var _ webhook.IContextuallyDefaultableObject = &ExecAccessRequest{}
 
 // Default implements webhook.Defaulter so a webhook will be registered for the type
-func (r *ExecAccessRequest) Default(_ admission.Request) error {
+func (r *ExecAccessRequest) Default(req admission.Request) error {
+	// Record who asked for this access, so that the reconciler and the metrics
+	// collector can attribute it later.
+	SetRequestedBy(r, req)
 	return nil
 }
 
@@ -72,11 +75,15 @@ func (r *ExecAccessRequest) ValidateCreate(req admission.Request) (admission.War
 		warnings = append(warnings, w)
 		execaccessrequestlog.Info(w)
 	}
+	recordAccessRequestCreated(req, r)
 	return warnings, nil
 }
 
 // ValidateUpdate prevents immutable updates to the ExecAccessRequest.
-func (r *ExecAccessRequest) ValidateUpdate(_ admission.Request, old runtime.Object) (admission.Warnings, error) {
+func (r *ExecAccessRequest) ValidateUpdate(
+	_ admission.Request,
+	old runtime.Object,
+) (admission.Warnings, error) {
 	execaccessrequestlog.Info("validate update", "name", r.Name)
 
 	// https://stackoverflow.com/questions/70650677/manage-immutable-fields-in-kubebuilder-validating-webhook
@@ -86,6 +93,13 @@ func (r *ExecAccessRequest) ValidateUpdate(_ admission.Request, old runtime.Obje
 			"error - Spec.TargetPod is an immutable field, create a new PodAccessRequest instead",
 		)
 	}
+
+	// The requested-by annotation is the basis for all per-user reporting, so
+	// it must not be editable after creation.
+	if err := ValidateRequestedByUnchanged(oldRequest, r); err != nil {
+		return nil, err
+	}
+
 	return nil, nil
 }
 

internal/api/v1alpha1/pod_access_request_test.go

@@ -83,11 +83,41 @@ var _ = Describe("PodAccessRequest", Ordered, func() {
 			}, request)
 			Expect(err).To(Not(HaveOccurred()))
 
-			// Update it and push it
-			request.SetAnnotations(map[string]string{"foo": "bar"})
+			// Update it and push it. Note that we add an annotation rather
+			// than replacing the map wholesale - the requested-by annotation is
+			// immutable, so wiping it out is rejected (see below).
+			annotations := request.GetAnnotations()
+			annotations["foo"] = "bar"
+			request.SetAnnotations(annotations)
 			err = k8sClient.Update(ctx, request)
 			Expect(err).To(Not(HaveOccurred()))
 		})
+
+		It("PodAccessRequest records the requesting user", func() {
+			request := &PodAccessRequest{}
+			err := k8sClient.Get(ctx, types.NamespacedName{
+				Name:      requestName,
+				Namespace: template.Namespace,
+			}, request)
+			Expect(err).To(Not(HaveOccurred()))
+			Expect(GetRequestedBy(request)).To(Not(BeEmpty()))
+		})
+
+		It("PodAccessRequest Update - Rejects a change to the requested-by annotation", func() {
+			request := &PodAccessRequest{}
+			err := k8sClient.Get(ctx, types.NamespacedName{
+				Name:      requestName,
+				Namespace: template.Namespace,
+			}, request)
+			Expect(err).To(Not(HaveOccurred()))
+
+			// Wiping the annotations (or editing just this one) would let a
+			// user disown or misattribute their own access request.
+			request.SetAnnotations(map[string]string{"foo": "bar"})
+			err = k8sClient.Update(ctx, request)
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring(AnnotationRequestedBy))
+		})
 		It("PodAccessRequest Delete - Passes Webhook ValidateUpdate() Call", func() {
 			// Get the request first
 			request := &PodAccessRequest{