Repository navigation
feat(alloy): add node, instance and cluster labels and kubelet volume stats - #97
Merged
Merged
Conversation
dannykopping
approved these changes
Nov 2, 2026
dannykopping
left a comment
Collaborator
There was a problem hiding this comment.
Wow, that was quick
… stats Common Kubernetes dashboards (e.g. dotdc/grafana-dashboards-kubernetes) select on labels this chart's collector did not produce: - cAdvisor series get a node label. - kube-state-metrics' own node label is restored on kube_* series instead of being overwritten by the node its pod runs on. - node_* series keep instance; it is still dropped on everything else. - global.telemetry.metrics.externalLabels adds labels such as cluster to every series (empty by default). - alloy.kubeletVolumeStats scrapes kubelet_volume_stats_* from each kubelet (enabled by default). - alloy.kubeletTLS makes TLS verification for the cAdvisor and kubelet scrapes configurable. Defaults keep the previous insecure_skip_verify behaviour. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
jtslear
force-pushed
the
skarbek/k8s-dashboard-labels
branch
from
November 2, 2026 20:17
f7f2b28 to
7826d71
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Makes the collector emit the labels common Kubernetes dashboards select on (for example dotdc/grafana-dashboards-kubernetes):
cAdvisor
nodelabel.container_*andmachine_*series now say which node they came from.kube-state-metrics
nodelabel. The pod target'snodelabel is the node kube-state-metrics itself runs on, which pushed the real node intoexported_node, and that was then dropped. It is now restored onkube_*series, mirroring the existingexported_pod/exported_namespacehandling.instanceonnode_*series.instanceis still dropped on everything else, but node-exporter dashboards tell nodes apart by it, so it is kept fornode_*.global.telemetry.metrics.externalLabelsadds labels (for examplecluster) to every series written. Empty by default, so no change unless set.alloy.kubeletVolumeStatsscrapeskubelet_volume_stats_*(PVC capacity and usage) from each kubelet's/metrics. Enabled by default, clustered like the cAdvisor scrape, andalloy.kubeletVolumeStats.enabled: falseturns it off.alloy.kubeletTLSmakes TLS verification for the cAdvisor and kubelet scrapes configurable (insecure_skip_verify,ca_file). Defaults keep the existinginsecure_skip_verify: true. Kubelets usually serve a self-signed certificate (including on EKS unlessserverTLSBootstrapis enabled and the CSRs are approved), so verification is opt-in.README, CHANGELOG and
compiled/resources.yamlare regenerated.Why
Importing the dotdc dashboards into our dogfood Grafana left the cluster dropdown empty, Nodes CPU/RAM Total at 0, CPU and memory by pod empty, and every Storage panel empty. All of it traced back to labels or series the collector did not produce. Dogfood currently works around this by patching a full copy of the Alloy config into its own repo (coder/dogfood#780). With this change dogfood only needs to set
externalLabels.clusterand the patch can be dropped.Reviewer notes
nodelabel on cAdvisor series, a differentnodeonkube_*series,instanceretained onnode_*series, and a new kubelet scrape. Series identity changes for those metrics, so queries that aggregate across them are unaffected but anything matching exact label sets is worth a look.instanceonnode_*is the node-exporter podip:port, so it changes when a node-exporter pod restarts.alloy validatewith defaults, withexternalLabelsset, and with TLS verification enabled.🤖 Generated with Claude Code