Openrca bench overlay - #14
Closed
kylehounslow wants to merge 7 commits into
Closed
kylehounslow wants to merge 7 commits into
kylehounslow wants to merge 7 commits into
Conversation
The values.yaml `ports:` list declared `metrics: 4900`, but the data-prepper subchart already declares port 4900 as `server` (its admin / Prometheus metrics endpoint). On Kubernetes server-side apply, the resulting Deployment fails with: duplicate entries for key [containerPort=4900,protocol="TCP"] Drop the duplicate entry. The subchart's `server` port is the canonical 4900 listener and is what `/metrics/prometheus` is exposed on.
Adds 7 new variables and the corresponding helm-release `set` wiring so
high-volume deployments can override sizing without forking values.yaml:
opensearch_replicas (default 3)
opensearch_storage_size (default 100Gi)
opensearch_storage_class (default gp2)
opensearch_node_memory (default 4Gi)
opensearch_jvm_heap (default 2g)
cortex_storage_size (default 50Gi)
cortex_storage_class (default gp2)
Defaults preserve existing behavior. The JVM heap variable is rendered
into the chart's expected `-Xms${heap} -Xmx${heap}` form via string
interpolation; chart values that take a single string (replicas, sizes,
classes) pass through unchanged.
Motivation: a 290GB OpenRCA benchmark deploy needs 16Gi RAM / 8g heap /
500Gi gp3 per node — possible to pass via `--set` on every install but
brittle. Variables make it a per-environment overlay.
Adds three TF variables and the corresponding helm wiring so high-volume deployments can size Data Prepper without forking values.yaml or the pipeline secret: data_prepper_memory (default 1Gi, subchart default) data_prepper_jvm_heap (default ""; empty = no JAVA_OPTS override) data_prepper_trace_flush_interval (default 180s, upstream behavior) Memory passes through `data-prepper.resources.requests.memory` + `limits.memory`. JVM heap is wired via `data-prepper.extraEnvs[0]` as JAVA_OPTS, dynamically emitted only when an override is given so the JVM picks heap from MaxRAMPercentage when unset. Flush interval is templated into the pipeline secret's `otel_traces` processor block via a new top-level `dataPrepperTraceFlushInterval` chart value. Motivation: at sustained ingest the otel_traces processor's 180s trace_flush_interval buffers all in-flight spans for traceGroup inference. At 30K spans/sec that's ~5M spans (~10 GB heap) at peak, exceeding the subchart's default 1G memory limit and OOM'ing the container. Lowering flush interval (e.g. 90s for trace shapes whose p99 is <30s) plus matching heap sizing keeps DP stable under load.
NOT for upstream — the OpenRCA-specific deployment overlay on top of the
configurable-sizing wiring branch:
- data-prepper-pipeline-secret.yaml: route by openrca_dataset/cloudbed to
eval-otel-v1-{traces,logs}-openrca-{system} indices (add_entries surfaces
the dotted attr as a flat routable key; management_disabled sinks write
literal index names; catch-all preserves non-OpenRCA traffic).
- init-opensearch-dashboards.py: create eval-* index patterns + point
Trace Analytics at the eval-* prefix (global + workspace scope, verified).
- rca-bench.tfvars: TB-scale overlay (m5.2xlarge x3, OS 3x16Gi/8g/500Gi gp3,
Cortex 50Gi, DP 12Gi/8g heap, trace_flush_interval 90s, demo off).
- docker-compose mirrors kept in sync per SYNC.md.
The data-prepper subchart serves Micrometer Prometheus metrics on its
admin server port (4900, /metrics/prometheus) but renders its Service
only from .Values.ports, so 4900 lands on the pod and never on the
Service. The collector's prometheus/data-prepper scrape targets
data-prepper:4900 and gets connection-refused, leaving the pipeline-health
dashboard's Data Prepper panels empty.
Adding 4900 to .Values.ports would duplicate the subchart's hardcoded
server containerPort and the API server rejects the Deployment. Add a
separate ClusterIP Service (data-prepper-metrics) that targets the
existing server port by name instead, gated on dataPrepperMetricsService.
Repoint both collector scrape jobs at it.
Validated on kind with the full stack: collector
otelcol_receiver_accepted_metric_points_total{receiver=prometheus/data-prepper}
climbs with 0 failed/refused.
rca-bench.tfvars held deployment-specific VALUES (cluster sizing, DP heap, storage classes) for one benchmark deployment. Those belong with the consumer, not the chart. The TF variable definitions that make these tunable stay here as wiring; the values move to the agent-eval-benchmark repo and are applied via -var-file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Describe what this change achieves.
Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.