Skip to content

feat(api): add an optional stateless odbingest node group #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tdakkota wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat(api): add an optional stateless odbingest node group #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dce7e88
feat(api): add an optional stateless odbingest node group
tdakkota Aug 17, 2026
014944a
chore(crd): regenerate manifests for the ingest node group
tdakkota Aug 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 38 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,30 @@ The operator models this as **one StatefulSet** of oteldb pods (stable identity
FQDN → peer address, per-pod PVC), a **headless Service** for peer DNS, and a **client Service** for
the query/ingest APIs.

### Optional role node groups

Symmetric nodes couple ingest capacity to storage capacity: the only way to absorb a write spike is
to add a node, and that node immediately takes ownership of a share of the shards. `spec.ingest`
breaks the coupling by deploying a pool of stateless **`odbingest`** write nodes as a separate
Deployment. It is opt-in — omit the stanza and the cluster stays symmetric, exactly as before.

Unlike VictoriaMetrics' `vminsert`, an `odbingest` pod is **not** handed a list of storage nodes. It
watches etcd for membership and computes shard owners locally from the rendezvous-hash ring, so
`spec.replicas` and `spec.ingest.replicas` are genuinely independent: scaling storage triggers no
config re-render and no restart of the ingest pool.

What the pool *does* need is the ring's shape — `replicationFactor`, `shardsPerTenant`, `etcdPrefix`
— and a mismatch there **fails silently**: the pool resolves a different owner set than the nodes
do, and writes land where no read will look for them. The operator renders both roles from the same
`spec.cluster`, so the mismatch cannot be expressed.

> **`odbingest` is not in the released oteldb image yet.** It is absent from oteldb's goreleaser
> builds and `release.Dockerfile`, so no published `ghcr.io/oteldb/oteldb` tag contains the binary.
> Set `spec.ingest.image` to a build that ships it until that is fixed upstream.

A stateless `odbselect` query pool (`spec.query`) is the matching read half; it is not implemented
yet, pending [oteldb/oteldb#1266](https://github.com/oteldb/oteldb/pull/1266).

### etcd is bring-your-own

The operator **does not manage etcd**. etcd's real operational needs — backups, disaster recovery,
Expand Down Expand Up @@ -60,7 +84,8 @@ for a fuller example including the S3 backend.

| Field | Purpose |
|---|---|
| `replicas` | Number of oteldb nodes (StatefulSet size). Use `>= cluster.replicationFactor`. |
| `replicas` | Number of oteldb storage nodes (StatefulSet size). Use `>= cluster.replicationFactor`. |
| `ingest` | Optional stateless `odbingest` write pool (Deployment). Absent ⇒ symmetric nodes handle ingest. Takes `replicas` (default 2), `image`, `service`, `extraConfig` and the standard scheduling/security knobs. |
| `image` / `imagePullPolicy` / `imagePullSecrets` | oteldb container image (default `ghcr.io/oteldb/oteldb:v0.46.0`). |
| `etcd.endpoints` | **Required.** External etcd endpoint list. |
| `storage.backend` | `file` (default, per-node PVC) or `s3` (shared object store). |
Expand Down Expand Up @@ -116,11 +141,16 @@ spec field to use instead.

`storage.policy` is now modelled in full, so the whole block is reserved.

`spec.ingest.extraConfig` is merged over the generated `odbingest.yml` under the same rules, with
the whole `cluster` block reserved for `spec.cluster` and `spec.etcd.endpoints`.

### Status

`status` reports `phase` (`Pending`/`Progressing`/`Ready`/`Degraded`), `replicas`,
`readyReplicas`, the resolved `etcdEndpoints`, and standard `Available`/`Progressing`/`Degraded`
conditions. `kubectl get oteldbcluster` prints replicas, ready count, and phase.
`readyReplicas`, `ingestReplicas`, `ingestReadyReplicas`, the resolved `etcdEndpoints`, and standard
`Available`/`Progressing`/`Degraded` conditions. `kubectl get oteldbcluster` prints replicas, ready
count, and phase. The phase tracks the storage nodes: they hold the data, and an ingest pool still
rolling out does not make the cluster unavailable.

## How per-pod identity works

Expand All @@ -138,6 +168,11 @@ Client APIs (exposed by the client Service): `4317` OTLP gRPC, `4318` OTLP HTTP,
remote-write, `9090` PromQL, `3200` TraceQL (Tempo), `3100` LogQL (Loki), `4040` Pyroscope, `8090`
self-metrics, `13133` health. Peer replication: `7946` (headless Service).

The ingest Service (`<name>-ingest`) exposes `4317` OTLP gRPC, `4318` OTLP HTTP, `19291` Prometheus
remote-write and `8090` self-metrics. `odbingest` serves OTLP/HTTP, remote write and its health
endpoints on **one** listener (`19291`), so the Service publishes `4318` — the port stock OTLP/HTTP
exporters target — and remaps it onto that listener.

## Getting Started

Prerequisites: Go 1.26+, a Kubernetes cluster, `kubectl`, and (for building the image) Docker.
Expand Down
103 changes: 103 additions & 0 deletions api/v1alpha1/oteldbcluster_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,25 @@ type OtelDBClusterSpec struct {
// Replicas is the number of oteldb storage nodes in the cluster. Each node is a symmetric
// StatefulSet pod that ingests, queries, stores, and replicates. For real redundancy this
// should be at least the replication factor (see Cluster.ReplicationFactor).
//
// This is the storage dimension only. Ingest capacity scales separately via spec.ingest, and
// scaling either has no effect on the other: membership lives in etcd and the ring is computed
// from it, so an ingest pod needs no knowledge of how many storage nodes there are.
// +kubebuilder:default=3
// +kubebuilder:validation:Minimum=1
// +optional
Replicas *int32 `json:"replicas,omitempty"`

// Ingest optionally deploys a pool of stateless odbingest write nodes, so ingest capacity
// scales independently of storage. Absent (the default) means no ingest pool: the storage
// nodes are symmetric and accept writes themselves.
//
// The pool holds no data and is not a ring member. It follows etcd membership read-only and
// routes each shard's write to that shard's primary, so scaling spec.replicas needs no change
// to it at all.
// +optional
Ingest *IngestSpec `json:"ingest,omitempty"`

// Image is the oteldb container image. Defaults to the operator's pinned image when empty.
// +optional
Image string `json:"image,omitempty"`
Expand Down Expand Up @@ -157,6 +171,86 @@ type EtcdSpec struct {
Endpoints []string `json:"endpoints"`
}

// IngestSpec configures the stateless odbingest write pool: a Deployment of nodes that accept
// OTLP and Prometheus remote write and route each shard to its ring primary.
//
// The pool's ring parameters (etcd endpoints, replication factor, shards per tenant, key prefix)
// are rendered from spec.etcd and spec.cluster, the same source the storage nodes render from. A
// mismatch there does not fail — it resolves a different owner set than the nodes do, and writes
// land where no read will look for them — so it is deliberately not configurable per pool.
//
// The odbingest binary is not yet shipped in the released oteldb image (it is absent from oteldb's
// goreleaser builds and release Dockerfile). Until it is, set Image to a build that contains it.
type IngestSpec struct {
// Replicas is the number of odbingest pods. They hold no data, so this is purely a throughput
// and availability knob and can be changed freely.
// +kubebuilder:default=2
// +kubebuilder:validation:Minimum=0
// +optional
Replicas *int32 `json:"replicas,omitempty"`

// Image is the container image running odbingest. Defaults to the cluster image
// (spec.image, else the operator's pinned image).
// +optional
Image string `json:"image,omitempty"`

// Resources are the compute resources for each odbingest container.
// +optional
Resources corev1.ResourceRequirements `json:"resources,omitempty"`

// Service configures the client-facing Service that exposes the ingest endpoints.
// +optional
Service ServiceSpec `json:"service,omitempty"`

// PodAnnotations are added to every odbingest pod.
// +optional
PodAnnotations map[string]string `json:"podAnnotations,omitempty"`

// PodLabels are added to every odbingest pod.
// +optional
PodLabels map[string]string `json:"podLabels,omitempty"`

// ServiceAccountName is the ServiceAccount for the odbingest pods. Empty falls back to
// spec.serviceAccountName.
// +optional
ServiceAccountName string `json:"serviceAccountName,omitempty"`

// NodeSelector constrains odbingest pods to nodes with matching labels.
// +optional
NodeSelector map[string]string `json:"nodeSelector,omitempty"`

// Affinity for odbingest pods. When empty, the operator applies a soft anti-affinity that
// spreads the pool across nodes.
// +optional
Affinity *corev1.Affinity `json:"affinity,omitempty"`

// Tolerations for odbingest pods.
// +optional
Tolerations []corev1.Toleration `json:"tolerations,omitempty"`

// TopologySpreadConstraints for odbingest pods.
// +optional
TopologySpreadConstraints []corev1.TopologySpreadConstraint `json:"topologySpreadConstraints,omitempty"`

// PodSecurityContext for odbingest pods.
// +optional
PodSecurityContext *corev1.PodSecurityContext `json:"podSecurityContext,omitempty"`

// SecurityContext for the odbingest container.
// +optional
SecurityContext *corev1.SecurityContext `json:"securityContext,omitempty"`

// ExtraConfig is arbitrary additional odbingest config deeply merged over the generated
// config, as a top-level YAML/JSON object. Use it to set fields the CRD does not model, such as
// prometheus_remote_write.time_threshold or the OTLP body-size limits.
//
// The cluster block is reserved and rejected instead of merged: it must stay in step with the
// storage nodes. Configure it through spec.etcd and spec.cluster.
// +optional
// +kubebuilder:pruning:PreserveUnknownFields
ExtraConfig *runtime.RawExtension `json:"extraConfig,omitempty"`
}

// StorageBackend selects the embedded storage engine's durable backend.
// +kubebuilder:validation:Enum=file;s3
type StorageBackend string
Expand Down Expand Up @@ -519,6 +613,15 @@ type OtelDBClusterStatus struct {
// +optional
ReadyReplicas int32 `json:"readyReplicas,omitempty"`

// IngestReplicas is the desired number of odbingest nodes. Zero when no ingest pool is
// configured.
// +optional
IngestReplicas int32 `json:"ingestReplicas,omitempty"`

// IngestReadyReplicas is the number of ready odbingest nodes.
// +optional
IngestReadyReplicas int32 `json:"ingestReadyReplicas,omitempty"`

// EtcdEndpoints is the resolved etcd endpoint list the cluster is using.
// +optional
EtcdEndpoints []string `json:"etcdEndpoints,omitempty"`
Expand Down
82 changes: 82 additions & 0 deletions api/v1alpha1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading