Skip to content

feat(alloy): add node, instance and cluster labels and kubelet volume stats - #97

Merged
jtslear merged 1 commit into
mainfrom
skarbek/k8s-dashboard-labels
Nov 2, 2026
Merged

jtslear merged 1 commit into
mainfrom
skarbek/k8s-dashboard-labels

Conversation

@jtslear

@jtslear jtslear commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

What changed

Makes the collector emit the labels common Kubernetes dashboards select on (for example dotdc/grafana-dashboards-kubernetes):

  • cAdvisor node label. container_* and machine_* series now say which node they came from.

  • kube-state-metrics node label. The pod target's node label is the node kube-state-metrics itself runs on, which pushed the real node into exported_node, and that was then dropped. It is now restored on kube_* series, mirroring the existing exported_pod/exported_namespace handling.

  • instance on node_* series. instance is still dropped on everything else, but node-exporter dashboards tell nodes apart by it, so it is kept for node_*.

  • global.telemetry.metrics.externalLabels adds labels (for example cluster) to every series written. Empty by default, so no change unless set.

  • alloy.kubeletVolumeStats scrapes kubelet_volume_stats_* (PVC capacity and usage) from each kubelet's /metrics. Enabled by default, clustered like the cAdvisor scrape, and alloy.kubeletVolumeStats.enabled: false turns it off.

  • alloy.kubeletTLS makes TLS verification for the cAdvisor and kubelet scrapes configurable (insecure_skip_verify, ca_file). Defaults keep the existing insecure_skip_verify: true. Kubelets usually serve a self-signed certificate (including on EKS unless serverTLSBootstrap is enabled and the CSRs are approved), so verification is opt-in.

README, CHANGELOG and compiled/resources.yaml are regenerated.

Why

Importing the dotdc dashboards into our dogfood Grafana left the cluster dropdown empty, Nodes CPU/RAM Total at 0, CPU and memory by pod empty, and every Storage panel empty. All of it traced back to labels or series the collector did not produce. Dogfood currently works around this by patching a full copy of the Alloy config into its own repo (coder/dogfood#780). With this change dogfood only needs to set externalLabels.cluster and the patch can be dropped.

Reviewer notes

  • Behavior change for existing users: new node label on cAdvisor series, a different node on kube_* series, instance retained on node_* series, and a new kubelet scrape. Series identity changes for those metrics, so queries that aggregate across them are unaffected but anything matching exact label sets is worth a look.
  • instance on node_* is the node-exporter pod ip:port, so it changes when a node-exporter pod restarts.
  • The rendered config passes alloy validate with defaults, with externalLabels set, and with TLS verification enabled.

🤖 Generated with Claude Code

@jtslear
jtslear requested a review from dannykopping as a code owner November 2, 2026 19:51

@dannykopping dannykopping left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, that was quick

… stats

Common Kubernetes dashboards (e.g. dotdc/grafana-dashboards-kubernetes) select
on labels this chart's collector did not produce:

- cAdvisor series get a node label.
- kube-state-metrics' own node label is restored on kube_* series instead of
  being overwritten by the node its pod runs on.
- node_* series keep instance; it is still dropped on everything else.
- global.telemetry.metrics.externalLabels adds labels such as cluster to every
  series (empty by default).
- alloy.kubeletVolumeStats scrapes kubelet_volume_stats_* from each kubelet
  (enabled by default).
- alloy.kubeletTLS makes TLS verification for the cAdvisor and kubelet scrapes
  configurable. Defaults keep the previous insecure_skip_verify behaviour.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@jtslear
jtslear force-pushed the skarbek/k8s-dashboard-labels branch from f7f2b28 to 7826d71 Compare November 2, 2026 20:17
@jtslear
jtslear merged commit 078ad30 into main Nov 2, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants