bug: exit consul-dataplane when Envoy version is rejected by Consul server

[himanshu-8693]

Problem

When the Consul server closes the ADS gRPC stream because the connected Envoy version is too old, consul-dataplane previously kept running indefinitely. This is a permanent, non-retriable condition — no amount of retrying will allow the process to serve traffic.

The zombie container blocks rolling deployments in orchestrated environments (ECS, Kubernetes) because the orchestrator treats the task/pod as healthy since the process is still running, and new tasks wait for the old ones to drain.

Example Envoy log line emitted when this happens:

[warning] envoy.config(44) DeltaAggregatedResources gRPC config stream to consul-dataplane closed: 3, Envoy 1.33.6 is too old and is not supported by Consul

Related upstream issue: #1245

Root Cause Analysis

The Consul server closes the ADS stream using HTTP/2 trailers (grpc-status: 3 UNAVAILABLE, grpc-message: Envoy X.Y.Z is too old and is not supported by Consul). The grpc-proxy transparent handler forwards those trailers to the Envoy-facing side and returns io.EOF — not a gRPC status error — to the stream interceptor chain. This means a StreamServerInterceptor cannot observe the rejection as an actionable error value.

Envoy itself always logs the full rejection message to its stderr stream (cmd.Stderr = p.cfg.EnvoyErrorStream in pkg/envoy/proxy.go). This is the reliable, version-agnostic interception point.

Fix

Added an envoyLogScanner — a lightweight io.Writer wrapper — that is set as the EnvoyErrorStream for the Envoy subprocess. It:

1. Forwards every Write() call transparently to the underlying writer (os.Stderr), so logging is unaffected.
2. Checks each write for the substring "is too old and is not supported by Consul" (version-number-agnostic).
3. On match: logs a clear ERROR-level message and calls xdsServer.closeExitedCh(), which feeds into the existing doneCh-based shutdown path — killing the Envoy subprocess and returning a non-zero exit code.

closeExitedCh() uses sync.Once to prevent double-close panics if multiple concurrent writes carry the same message.

Files changed

File	Change
pkg/consuldp/xds.go	Added envoyVersionUnsupportedMsg constant; newEnvoyLogScanner() constructor; envoyLogScanner struct and Write() method; xdsServer.closeExitedCh() using sync.Once
pkg/consuldp/consul_dataplane.go	Added closeOnce sync.Once to xdsServer struct; envoyProxyConfig() sets EnvoyErrorStream: cdp.newEnvoyLogScanner(os.Stderr); startXDSServer uses closeExitedCh()
pkg/consuldp/xds_test.go	TestEnvoyLogScannerFatalExit (table-driven, 4 cases); TestEnvoyLogScannerCloseOnce (concurrent double-write panic guard)

Testing

Unit tests

go test ./pkg/... -run TestEnvoyLogScanner -v

All tests pass. Coverage:

• Exact rejection message → exitedCh closed ✅
• Different Envoy version number in message → exitedCh closed ✅
• Unrelated log line → exitedCh stays open ✅
• Empty write → exitedCh stays open ✅
• 5 concurrent writes of fatal message → no panic (sync.Once guard) ✅

Live cluster validation (KIND + Consul 2.0.0+ent)

Tested two images built with Envoy 1.33.4 (rejected by Consul ≥2.0):

Image	Consul-dataplane binary	Result
stock	upstream main	Pod stays Running indefinitely — zombie
fixed	this branch	Pod reaches Error state in <5 seconds

Fixed binary log sequence on exit:

[ERROR] consul-dataplane: Envoy version is not supported by Consul server — this is a permanent error, consul-dataplane will now exit. Please upgrade Envoy to a supported version.
[INFO]  consul-dataplane: xds server exited. triggering quit
xDS server exited unexpectedly

Changelog

bug fixes:
* Fixed consul-dataplane looping forever when Envoy version is rejected by Consul server as unsupported. consul-dataplane now exits immediately with a non-zero code and a clear error message, unblocking rolling deployments.

[github-actions]

Go Test Coverage: 67.0%

Patch coverage: 67.9% (19/28 changed lines covered)

See the workflow run for the full per-package breakdown and downloadable HTML report.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 64.00000% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.67%. Comparing base (28a963a) to head (83d4526).
⚠️ Report is 18 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/consuldp/consul_dataplane.go	0.00%	8 Missing ⚠️
pkg/consuldp/xds.go	94.11%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1252      +/-   ##
==========================================
+ Coverage   68.28%   68.67%   +0.39%     
==========================================
  Files          19       19              
  Lines        2188     2203      +15     
==========================================
+ Hits         1494     1513      +19     
+ Misses        593      588       -5     
- Partials      101      102       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.