Skip to content

fix(validator): stop copying Service annotations onto the metrics Services - #340

Merged
devkoriel merged 1 commit into
mainfrom
fix/validator-metrics-service-annotations
Nov 3, 2026
Merged

devkoriel merged 1 commit into
mainfrom
fix/validator-metrics-service-annotations

Conversation

@devkoriel

@devkoriel devkoriel commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

The validator chart's -metrics and -metrics-vao Services copied ghost.service.annotations and vao.service.annotations. Those values usually carry load balancer annotations, so on AWS the metrics Services ended up with their own internet-facing load balancers.

Why

The AWS Load Balancer Controller decides whether a Service is its job from annotations (aws-load-balancer-type: external plus a target type), not from spec.type. That is IsServiceSupported in pkg/service/service_utils.go at v3.3.0, with the ServiceTypeLoadBalancerOnly feature gate off by default. So a ClusterIP Service with those annotations still gets an NLB, and the controller keeps it running.

With the copied annotations, every release that set the libp2p NLB annotations also got an NLB per metrics Service, publishing the metrics port. The copied external-dns.alpha.kubernetes.io/hostname annotation was wrong on these Services too.

What

  • The two metrics Service templates now read annotations from a new value, global.metrics.service.annotations (default {}).
  • ghost.service.annotations and vao.service.annotations apply only to the libp2p Services, as before.
  • Chart version 0.9.0 -> 0.10.0. README regenerated with helm-docs.

Upgrade note

Releases that relied on the copy, for example a prometheus.io/scrape annotation set on ghost.service, need to move it to global.metrics.service.annotations.

On AWS, once the annotations are gone, the controller cleans up the metrics load balancer, target group and security group through its Service finalizer: the Service still has the finalizer, is no longer supported, so reconcile builds a nil LB and runs cleanupLoadBalancerResources. Releases with load balancer deletion protection enabled on that NLB will see the cleanup refused until it is turned off.

Testing

  • helm template with ghost and vao Services set to internet-facing NLB annotations, before and after. The only diff is those four annotations gone from ghost-metrics and ghost-metrics-vao. The libp2p Services render identically.
  • --set-string global.metrics.service.annotations.prometheus\.io/scrape=true lands on both metrics Services. global.metrics.service=null still renders.
  • helm lint with defaults and with both ci/ values files: pass.
  • ct lint --charts charts/validator: pass.

Summary by CodeRabbit

  • New Features
    • Added a dedicated annotations setting for the metrics and VAO metrics services, so they can be configured independently of the Ghost and VAO services.
    • Metrics services no longer inherit annotations from those services, avoiding unintended load balancer provisioning from inherited settings.
  • Release
    • Updated the Helm chart version to 0.10.0.

…vices

The -metrics and -metrics-vao ClusterIP Services copied
ghost.service.annotations and vao.service.annotations. Those usually hold
load balancer annotations, and the AWS Load Balancer Controller builds a
load balancer for any Service carrying them, whatever its spec.type. Each
metrics Service therefore got its own load balancer exposing the metrics
port, and an external-dns hostname annotation was copied along too.

Read the metrics Service annotations from a new value,
global.metrics.service.annotations (default empty), instead. Bump the
chart to 0.10.0.
@devkoriel devkoriel added bug Something isn't working validator labels Nov 3, 2026
@devkoriel devkoriel self-assigned this Nov 3, 2026
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

The chart adds global.metrics.service.annotations for both metrics Services. Both templates use this setting, which defaults to an empty map. The README documents the setting, and the chart version changes from 0.9.0 to 0.10.0.

Changes: Metrics Service annotation configuration

Layer / File(s) Summary
Shared metrics Service annotations
charts/validator/values.yaml, charts/validator/templates/service-metrics.yaml, charts/validator/templates/service-metrics-vao.yaml, charts/validator/README.md, charts/validator/Chart.yaml
Both metrics Services now use global.metrics.service.annotations, which defaults to an empty map. The README documents the setting and its previous annotation sources. The chart version changes to 0.10.0.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: wesleycharlesblake, paulohpigatto

Merge Risk: Minimal · up to 44bf8

No actionable merge-blocking risk remains in this chart change.

Security architecture risk: Low · up to 44bf8

The change removes unintended annotation inheritance and keeps metrics Service identities and ports unchanged. No introduced security defect was established. However, retiring existing external load balancers depends on controller behavior and deployment state that were not verified, and rolling back can restore the previous exposure-triggering configuration.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The relevant exposure is the metrics endpoint of each enabled Ghost and VAO Service in an affected release. Where inherited annotations trigger an external load balancer, removing that inheritance reduces accidental exposure. Actual internet reachability and the retirement of existing infrastructure remain unverified; the inherited configuration predates this PR.

Trust Boundaries and Controls

  • observed — The change separates metrics infrastructure annotations from the component Service configuration; it is not an annotation allowlist. Authors of chart values can still explicitly supply arbitrary metrics Service annotations, and those annotations are emitted for both enabled metrics Services.

Hardening Proposals

  • proposed — For affected deployments, treat metrics load-balancer retirement as a separately verified rollout outcome. Check controller ownership, finalizers, deletion protection, and remaining external resources after upgrade, and account for rollback restoring inherited annotations. This is operational assurance guidance, not a verified PR defect.
Pre-merge checks: Passed 5
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: metrics Services no longer copy annotations from other Services.
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (5 skipped: 5 unsupported.)

Finishing Touches
Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.
Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 4ec03e0e-6be9-4cfb-9990-1753ed5382d4
Commits

Reviewing files that changed from the base of the PR and between 1feb7b7 and 44bf82a.

Files selected for processing: 5
  • charts/validator/Chart.yaml
  • charts/validator/README.md
  • charts/validator/templates/service-metrics-vao.yaml
  • charts/validator/templates/service-metrics.yaml
  • charts/validator/values.yaml

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.

@devkoriel
devkoriel merged commit 4c2016b into main Nov 3, 2026
3 of 4 checks passed
@devkoriel
devkoriel deleted the fix/validator-metrics-service-annotations branch November 3, 2026 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working validator

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants