diff --git a/README.md b/README.md index fb90b58..e1ba01d 100644 --- a/README.md +++ b/README.md @@ -24,27 +24,47 @@ the query/ingest APIs. ### Optional role node groups -Symmetric nodes couple ingest capacity to storage capacity: the only way to absorb a write spike is -to add a node, and that node immediately takes ownership of a share of the shards. `spec.ingest` -breaks the coupling by deploying a pool of stateless **`odbingest`** write nodes as a separate -Deployment. It is opt-in — omit the stanza and the cluster stays symmetric, exactly as before. - -Unlike VictoriaMetrics' `vminsert`, an `odbingest` pod is **not** handed a list of storage nodes. It -watches etcd for membership and computes shard owners locally from the rendezvous-hash ring, so -`spec.replicas` and `spec.ingest.replicas` are genuinely independent: scaling storage triggers no -config re-render and no restart of the ingest pool. - -What the pool *does* need is the ring's shape — `replicationFactor`, `shardsPerTenant`, `etcdPrefix` +Symmetric nodes couple ingest and query capacity to storage capacity: the only way to absorb a write +spike or a dashboard reload is to add a node, and that node immediately takes ownership of a share +of the shards. `spec.ingest` and `spec.query` break the coupling by deploying pools of stateless +**`odbingest`** write nodes and **`odbselect`** query nodes as separate Deployments. Both are opt-in +— omit the stanzas and the cluster stays symmetric, exactly as before. + +Unlike VictoriaMetrics' `vminsert`/`vmselect`, a stateless oteldb pod is **not** handed a list of +storage nodes. It watches etcd for membership and computes shard owners locally from the +rendezvous-hash ring, so `spec.replicas`, `spec.ingest.replicas` and `spec.query.replicas` are +genuinely independent: scaling storage triggers no config re-render and no restart of either pool. + +What a pool *does* need is the ring's shape — `replicationFactor`, `shardsPerTenant`, `etcdPrefix` — and a mismatch there **fails silently**: the pool resolves a different owner set than the nodes -do, and writes land where no read will look for them. The operator renders both roles from the same -`spec.cluster`, so the mismatch cannot be expressed. - -> **`odbingest` is not in the released oteldb image yet.** It is absent from oteldb's goreleaser -> builds and `release.Dockerfile`, so no published `ghcr.io/oteldb/oteldb` tag contains the binary. -> Set `spec.ingest.image` to a build that ships it until that is fixed upstream. - -A stateless `odbselect` query pool (`spec.query`) is the matching read half; it is not implemented -yet, pending [oteldb/oteldb#1266](https://github.com/oteldb/oteldb/pull/1266). +do, so writes land where no read will look for them and reads look where the data is not. The +operator renders all three roles from the same `spec.cluster`, so the mismatch cannot be expressed. + +> **Neither binary is in the released oteldb image yet.** `odbingest` is absent from oteldb's +> goreleaser builds and `release.Dockerfile`, and `odbselect`'s release packaging ships with +> [oteldb/oteldb#1266](https://github.com/oteldb/oteldb/pull/1266). No published +> `ghcr.io/oteldb/oteldb` tag contains either one. Set `spec.ingest.image` / `spec.query.image` to a +> build that ships them until that is fixed upstream. + +#### The query pool's shape + +`odbingest` serves OTLP/HTTP, remote write and its health endpoints on one listener. `odbselect` +does not: it serves **four query APIs on four listeners** (`9090` PromQL, `3100` LogQL, `3200` +TraceQL, `4040` Pyroscope) plus health on `13133`. Three consequences: + +- **One Service, several ports.** `-query` publishes every enabled API. The four APIs are one + pod set behind one selector and each already has its own port, so a Service per API would add DNS + names — and, at `type: LoadBalancer`, four load balancers — without making anything addressable + that is not addressable now. +- **`spec.signals` turns an API off, not a per-API knob.** The four APIs map 1:1 onto the four + signals the cluster already toggles, and serving an API for a signal the cluster does not store + can only answer empty. A disabled signal renders `bind: "-"` (odbselect's disable convention) and + drops the port from the pool and its Service. Note the `"-"` is *required*: odbselect defaults + every API block it does not find, so simply omitting one would serve it anyway. +- **Probes hit the health listener**, not a data port — it is the one listener that is served + whatever `spec.signals` leaves enabled. `/readyz` answers 503 until the ring has a member, so a + starting pod stays out of the load balancer while a query could only return an empty result that + looks like an answer. ### etcd is bring-your-own @@ -86,6 +106,7 @@ for a fuller example including the S3 backend. |---|---| | `replicas` | Number of oteldb storage nodes (StatefulSet size). Use `>= cluster.replicationFactor`. | | `ingest` | Optional stateless `odbingest` write pool (Deployment). Absent ⇒ symmetric nodes handle ingest. Takes `replicas` (default 2), `image`, `service`, `extraConfig` and the standard scheduling/security knobs. | +| `query` | Optional stateless `odbselect` query pool (Deployment). Absent ⇒ symmetric nodes answer queries. Same knobs as `ingest`. Which APIs it serves follows `spec.signals`. | | `image` / `imagePullPolicy` / `imagePullSecrets` | oteldb container image (default `ghcr.io/oteldb/oteldb:v0.46.0`). | | `etcd.endpoints` | **Required.** External etcd endpoint list. | | `storage.backend` | `file` (default, per-node PVC) or `s3` (shared object store). | @@ -144,13 +165,21 @@ spec field to use instead. `spec.ingest.extraConfig` is merged over the generated `odbingest.yml` under the same rules, with the whole `cluster` block reserved for `spec.cluster` and `spec.etcd.endpoints`. +`spec.query.extraConfig` is merged over the generated `odbselect.yml`, reserving `cluster` and the +five listener addresses (`prometheus.bind`, `loki.bind`, `tempo.bind`, `pyroscope.bind`, +`health.bind`). The binds are reserved because odbselect keeps serving its other listeners if one +moves: the Service port the operator published would point at nothing while the pod stayed `Ready`. +Everything else in a block is free — `prometheus.max_samples`, `loki.max_sample_rows`, per-listener +`auth`, `shutdown_timeout`. Use `spec.signals` to turn an API off. + ### Status `status` reports `phase` (`Pending`/`Progressing`/`Ready`/`Degraded`), `replicas`, -`readyReplicas`, `ingestReplicas`, `ingestReadyReplicas`, the resolved `etcdEndpoints`, and standard -`Available`/`Progressing`/`Degraded` conditions. `kubectl get oteldbcluster` prints replicas, ready -count, and phase. The phase tracks the storage nodes: they hold the data, and an ingest pool still -rolling out does not make the cluster unavailable. +`readyReplicas`, `ingestReplicas`, `ingestReadyReplicas`, `queryReplicas`, `queryReadyReplicas`, the +resolved `etcdEndpoints`, and standard `Available`/`Progressing`/`Degraded` conditions. +`kubectl get oteldbcluster` prints replicas, ready count, and phase. The phase tracks the storage +nodes: they hold the data, and a stateless pool still rolling out does not make the cluster +unavailable. ## How per-pod identity works @@ -173,6 +202,11 @@ remote-write and `8090` self-metrics. `odbingest` serves OTLP/HTTP, remote write endpoints on **one** listener (`19291`), so the Service publishes `4318` — the port stock OTLP/HTTP exporters target — and remaps it onto that listener. +The query Service (`-query`) exposes `9090` PromQL, `3200` TraceQL, `3100` LogQL, `4040` +Pyroscope and `8090` self-metrics, dropping any whose signal is disabled. `odbselect` serves each on +its own listener, so these are published one-to-one; its health listener (`13133`) is probed, not +published. + ## Getting Started Prerequisites: Go 1.26+, a Kubernetes cluster, `kubectl`, and (for building the image) Docker. @@ -209,4 +243,6 @@ Controller layout under `internal/controller/`: - `oteldbcluster_controller.go` — reconcile loop, apply/owner-ref helper, status. - `config.go` — renders the oteldb `config.yml` from the spec. - `resources.go` — builds the ConfigMap, headless + client Services, and StatefulSet. +- `ingest.go` / `query.go` — the stateless `odbingest` and `odbselect` pools: config, Service, + Deployment. - `naming.go` — names, labels, ports. diff --git a/api/v1alpha1/oteldbcluster_types.go b/api/v1alpha1/oteldbcluster_types.go index b9dbafa..474b7b4 100644 --- a/api/v1alpha1/oteldbcluster_types.go +++ b/api/v1alpha1/oteldbcluster_types.go @@ -31,9 +31,10 @@ type OtelDBClusterSpec struct { // StatefulSet pod that ingests, queries, stores, and replicates. For real redundancy this // should be at least the replication factor (see Cluster.ReplicationFactor). // - // This is the storage dimension only. Ingest capacity scales separately via spec.ingest, and - // scaling either has no effect on the other: membership lives in etcd and the ring is computed - // from it, so an ingest pod needs no knowledge of how many storage nodes there are. + // This is the storage dimension only. Ingest and query capacity scale separately via + // spec.ingest and spec.query, and scaling any of them has no effect on the others: membership + // lives in etcd and the ring is computed from it, so a stateless pod needs no knowledge of how + // many storage nodes there are. // +kubebuilder:default=3 // +kubebuilder:validation:Minimum=1 // +optional @@ -49,6 +50,15 @@ type OtelDBClusterSpec struct { // +optional Ingest *IngestSpec `json:"ingest,omitempty"` + // Query optionally deploys a pool of stateless odbselect query nodes, so query capacity scales + // independently of storage. Absent (the default) means no query pool: the storage nodes are + // symmetric and answer queries themselves. + // + // The pool holds no data and is not a ring member. It follows etcd membership read-only and + // fans each read out to the shard's owners, so scaling spec.replicas needs no change to it. + // +optional + Query *QuerySpec `json:"query,omitempty"` + // Image is the oteldb container image. Defaults to the operator's pinned image when empty. // +optional Image string `json:"image,omitempty"` @@ -251,6 +261,97 @@ type IngestSpec struct { ExtraConfig *runtime.RawExtension `json:"extraConfig,omitempty"` } +// QuerySpec configures the stateless odbselect query pool: a Deployment of nodes that serve the +// PromQL, LogQL, TraceQL and Pyroscope APIs by reading through the ring instead of local storage. +// +// The pool's ring parameters (etcd endpoints, replication factor, shards per tenant, key prefix) +// are rendered from spec.etcd and spec.cluster, the same source the storage nodes render from. A +// mismatch there does not fail — it resolves a different owner set than the nodes do, so reads look +// where the data is not — so it is deliberately not configurable per pool. +// +// Which APIs the pool serves follows spec.signals, the same toggle that decides what the storage +// nodes serve and store: a disabled signal's API is switched off on the pool and its port is +// dropped from the query Service. There is no separate per-API toggle, because serving an API for +// a signal the cluster does not store can only answer empty. +// +// The odbselect binary is not yet shipped in the released oteldb image (it lands with +// oteldb/oteldb#1266). Until it is, set Image to a build that contains it. +type QuerySpec struct { + // Replicas is the number of odbselect pods. They hold no data, so this is purely a throughput + // and availability knob and can be changed freely. + // +kubebuilder:default=2 + // +kubebuilder:validation:Minimum=0 + // +optional + Replicas *int32 `json:"replicas,omitempty"` + + // Image is the container image running odbselect. Defaults to the cluster image + // (spec.image, else the operator's pinned image). + // +optional + Image string `json:"image,omitempty"` + + // Resources are the compute resources for each odbselect container. + // +optional + Resources corev1.ResourceRequirements `json:"resources,omitempty"` + + // Service configures the client-facing Service that exposes the query APIs. One Service carries + // every enabled API, each on its own port; they share a pod set, so splitting them across four + // Services would only add DNS names. + // +optional + Service ServiceSpec `json:"service,omitempty"` + + // PodAnnotations are added to every odbselect pod. + // +optional + PodAnnotations map[string]string `json:"podAnnotations,omitempty"` + + // PodLabels are added to every odbselect pod. + // +optional + PodLabels map[string]string `json:"podLabels,omitempty"` + + // ServiceAccountName is the ServiceAccount for the odbselect pods. Empty falls back to + // spec.serviceAccountName. + // +optional + ServiceAccountName string `json:"serviceAccountName,omitempty"` + + // NodeSelector constrains odbselect pods to nodes with matching labels. + // +optional + NodeSelector map[string]string `json:"nodeSelector,omitempty"` + + // Affinity for odbselect pods. When empty, the operator applies a soft anti-affinity that + // spreads the pool across nodes. + // +optional + Affinity *corev1.Affinity `json:"affinity,omitempty"` + + // Tolerations for odbselect pods. + // +optional + Tolerations []corev1.Toleration `json:"tolerations,omitempty"` + + // TopologySpreadConstraints for odbselect pods. + // +optional + TopologySpreadConstraints []corev1.TopologySpreadConstraint `json:"topologySpreadConstraints,omitempty"` + + // PodSecurityContext for odbselect pods. + // +optional + PodSecurityContext *corev1.PodSecurityContext `json:"podSecurityContext,omitempty"` + + // SecurityContext for the odbselect container. + // +optional + SecurityContext *corev1.SecurityContext `json:"securityContext,omitempty"` + + // ExtraConfig is arbitrary additional odbselect config deeply merged over the generated config, + // as a top-level YAML/JSON object. Use it to set fields the CRD does not model, such as + // prometheus.max_samples, loki.max_sample_rows, per-listener auth or shutdown_timeout. + // + // The cluster block is reserved and rejected instead of merged: it must stay in step with the + // storage nodes. Configure it through spec.etcd and spec.cluster. The listener addresses + // (prometheus.bind, loki.bind, tempo.bind, pyroscope.bind, health.bind) are reserved too: the + // operator publishes them as Service ports and probes them, and odbselect keeps serving its + // other listeners if one moves, so a rebind would fail silently rather than crash the pod. Use + // spec.signals to turn an API off. + // +optional + // +kubebuilder:pruning:PreserveUnknownFields + ExtraConfig *runtime.RawExtension `json:"extraConfig,omitempty"` +} + // StorageBackend selects the embedded storage engine's durable backend. // +kubebuilder:validation:Enum=file;s3 type StorageBackend string @@ -622,6 +723,15 @@ type OtelDBClusterStatus struct { // +optional IngestReadyReplicas int32 `json:"ingestReadyReplicas,omitempty"` + // QueryReplicas is the desired number of odbselect nodes. Zero when no query pool is + // configured. + // +optional + QueryReplicas int32 `json:"queryReplicas,omitempty"` + + // QueryReadyReplicas is the number of ready odbselect nodes. + // +optional + QueryReadyReplicas int32 `json:"queryReadyReplicas,omitempty"` + // EtcdEndpoints is the resolved etcd endpoint list the cluster is using. // +optional EtcdEndpoints []string `json:"etcdEndpoints,omitempty"` diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index 1e164d4..ec26f00 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -317,6 +317,11 @@ func (in *OtelDBClusterSpec) DeepCopyInto(out *OtelDBClusterSpec) { *out = new(IngestSpec) (*in).DeepCopyInto(*out) } + if in.Query != nil { + in, out := &in.Query, &out.Query + *out = new(QuerySpec) + (*in).DeepCopyInto(*out) + } if in.ImagePullSecrets != nil { in, out := &in.ImagePullSecrets, &out.ImagePullSecrets *out = make([]v1.LocalObjectReference, len(*in)) @@ -472,6 +477,83 @@ func (in *PrecisionTierSpec) DeepCopy() *PrecisionTierSpec { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *QuerySpec) DeepCopyInto(out *QuerySpec) { + *out = *in + if in.Replicas != nil { + in, out := &in.Replicas, &out.Replicas + *out = new(int32) + **out = **in + } + in.Resources.DeepCopyInto(&out.Resources) + in.Service.DeepCopyInto(&out.Service) + if in.PodAnnotations != nil { + in, out := &in.PodAnnotations, &out.PodAnnotations + *out = make(map[string]string, len(*in)) + for key, val := range *in { + (*out)[key] = val + } + } + if in.PodLabels != nil { + in, out := &in.PodLabels, &out.PodLabels + *out = make(map[string]string, len(*in)) + for key, val := range *in { + (*out)[key] = val + } + } + if in.NodeSelector != nil { + in, out := &in.NodeSelector, &out.NodeSelector + *out = make(map[string]string, len(*in)) + for key, val := range *in { + (*out)[key] = val + } + } + if in.Affinity != nil { + in, out := &in.Affinity, &out.Affinity + *out = new(v1.Affinity) + (*in).DeepCopyInto(*out) + } + if in.Tolerations != nil { + in, out := &in.Tolerations, &out.Tolerations + *out = make([]v1.Toleration, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.TopologySpreadConstraints != nil { + in, out := &in.TopologySpreadConstraints, &out.TopologySpreadConstraints + *out = make([]v1.TopologySpreadConstraint, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.PodSecurityContext != nil { + in, out := &in.PodSecurityContext, &out.PodSecurityContext + *out = new(v1.PodSecurityContext) + (*in).DeepCopyInto(*out) + } + if in.SecurityContext != nil { + in, out := &in.SecurityContext, &out.SecurityContext + *out = new(v1.SecurityContext) + (*in).DeepCopyInto(*out) + } + if in.ExtraConfig != nil { + in, out := &in.ExtraConfig, &out.ExtraConfig + *out = new(runtime.RawExtension) + (*in).DeepCopyInto(*out) + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new QuerySpec. +func (in *QuerySpec) DeepCopy() *QuerySpec { + if in == nil { + return nil + } + out := new(QuerySpec) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *RecompressSpec) DeepCopyInto(out *RecompressSpec) { *out = *in diff --git a/config/crd/bases/db.oteldb.io_oteldbclusters.yaml b/config/crd/bases/db.oteldb.io_oteldbclusters.yaml index 6580a70..19688c5 100644 --- a/config/crd/bases/db.oteldb.io_oteldbclusters.yaml +++ b/config/crd/bases/db.oteldb.io_oteldbclusters.yaml @@ -3223,6 +3223,1702 @@ spec: x-kubernetes-int-or-string: true type: object type: object + query: + description: |- + Query optionally deploys a pool of stateless odbselect query nodes, so query capacity scales + independently of storage. Absent (the default) means no query pool: the storage nodes are + symmetric and answer queries themselves. + + The pool holds no data and is not a ring member. It follows etcd membership read-only and + fans each read out to the shard's owners, so scaling spec.replicas needs no change to it. + properties: + affinity: + description: |- + Affinity for odbselect pods. When empty, the operator applies a soft anti-affinity that + spreads the pool across nodes. + properties: + nodeAffinity: + description: Describes node affinity scheduling rules for + the pod. + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: |- + The scheduler will prefer to schedule pods to nodes that satisfy + the affinity expressions specified by this field, but it may choose + a node that violates one or more of the expressions. The node that is + most preferred is the one with the greatest sum of weights, i.e. + for each node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, etc.), + compute a sum by iterating through the elements of this field and adding + "weight" to the sum if the node matches the corresponding matchExpressions; the + node(s) with the highest sum are the most preferred. + items: + description: |- + An empty preferred scheduling term matches all objects with implicit weight 0 + (i.e. it's a no-op). A null preferred scheduling term matches no objects (i.e. is also a no-op). + properties: + preference: + description: A node selector term, associated with + the corresponding weight. + properties: + matchExpressions: + description: A list of node selector requirements + by node's labels. + items: + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. + type: string + values: + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchFields: + description: A list of node selector requirements + by node's fields. + items: + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. + type: string + values: + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + type: object + x-kubernetes-map-type: atomic + weight: + description: Weight associated with matching the + corresponding nodeSelectorTerm, in the range 1-100. + format: int32 + type: integer + required: + - preference + - weight + type: object + type: array + x-kubernetes-list-type: atomic + requiredDuringSchedulingIgnoredDuringExecution: + description: |- + If the affinity requirements specified by this field are not met at + scheduling time, the pod will not be scheduled onto the node. + If the affinity requirements specified by this field cease to be met + at some point during pod execution (e.g. due to an update), the system + may or may not try to eventually evict the pod from its node. + properties: + nodeSelectorTerms: + description: Required. A list of node selector terms. + The terms are ORed. + items: + description: |- + A null or empty node selector term matches no objects. The requirements of + them are ANDed. + The TopologySelectorTerm type implements a subset of the NodeSelectorTerm. + properties: + matchExpressions: + description: A list of node selector requirements + by node's labels. + items: + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. + type: string + values: + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchFields: + description: A list of node selector requirements + by node's fields. + items: + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. + type: string + values: + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + type: object + x-kubernetes-map-type: atomic + type: array + x-kubernetes-list-type: atomic + required: + - nodeSelectorTerms + type: object + x-kubernetes-map-type: atomic + type: object + podAffinity: + description: Describes pod affinity scheduling rules (e.g. + co-locate this pod in the same node, zone, etc. as some + other pod(s)). + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: |- + The scheduler will prefer to schedule pods to nodes that satisfy + the affinity expressions specified by this field, but it may choose + a node that violates one or more of the expressions. The node that is + most preferred is the one with the greatest sum of weights, i.e. + for each node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, etc.), + compute a sum by iterating through the elements of this field and adding + "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the + node(s) with the highest sum are the most preferred. + items: + description: The weights of all of the matched WeightedPodAffinityTerm + fields are added per-node to find the most preferred + node(s) + properties: + podAffinityTerm: + description: Required. A pod affinity term, associated + with the corresponding weight. + properties: + labelSelector: + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. + properties: + matchExpressions: + description: matchExpressions is a list + of label selector requirements. The requirements + are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key + that the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both matchLabelKeys and labelSelector. + Also, matchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. + Also, mismatchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + namespaceSelector: + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list + of label selector requirements. The requirements + are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key + that the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + x-kubernetes-list-type: atomic + topologyKey: + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + weight: + description: |- + weight associated with matching the corresponding podAffinityTerm, + in the range 1-100. + format: int32 + type: integer + required: + - podAffinityTerm + - weight + type: object + type: array + x-kubernetes-list-type: atomic + requiredDuringSchedulingIgnoredDuringExecution: + description: |- + If the affinity requirements specified by this field are not met at + scheduling time, the pod will not be scheduled onto the node. + If the affinity requirements specified by this field cease to be met + at some point during pod execution (e.g. due to a pod label update), the + system may or may not try to eventually evict the pod from its node. + When there are multiple elements, the lists of nodes corresponding to each + podAffinityTerm are intersected, i.e. all terms must be satisfied. + items: + description: |- + Defines a set of pods (namely those matching the labelSelector + relative to the given namespace(s)) that this pod should be + co-located (affinity) or not co-located (anti-affinity) with, + where co-located is defined as running on a node whose value of + the label with key matches that of any node on which + a pod of the set of pods is running + properties: + labelSelector: + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both matchLabelKeys and labelSelector. + Also, matchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. + Also, mismatchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + namespaceSelector: + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + x-kubernetes-list-type: atomic + topologyKey: + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + type: array + x-kubernetes-list-type: atomic + type: object + podAntiAffinity: + description: Describes pod anti-affinity scheduling rules + (e.g. avoid putting this pod in the same node, zone, etc. + as some other pod(s)). + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: |- + The scheduler will prefer to schedule pods to nodes that satisfy + the anti-affinity expressions specified by this field, but it may choose + a node that violates one or more of the expressions. The node that is + most preferred is the one with the greatest sum of weights, i.e. + for each node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling anti-affinity expressions, etc.), + compute a sum by iterating through the elements of this field and subtracting + "weight" from the sum if the node has pods which matches the corresponding podAffinityTerm; the + node(s) with the highest sum are the most preferred. + items: + description: The weights of all of the matched WeightedPodAffinityTerm + fields are added per-node to find the most preferred + node(s) + properties: + podAffinityTerm: + description: Required. A pod affinity term, associated + with the corresponding weight. + properties: + labelSelector: + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. + properties: + matchExpressions: + description: matchExpressions is a list + of label selector requirements. The requirements + are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key + that the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both matchLabelKeys and labelSelector. + Also, matchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. + Also, mismatchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + namespaceSelector: + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list + of label selector requirements. The requirements + are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key + that the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + x-kubernetes-list-type: atomic + topologyKey: + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + weight: + description: |- + weight associated with matching the corresponding podAffinityTerm, + in the range 1-100. + format: int32 + type: integer + required: + - podAffinityTerm + - weight + type: object + type: array + x-kubernetes-list-type: atomic + requiredDuringSchedulingIgnoredDuringExecution: + description: |- + If the anti-affinity requirements specified by this field are not met at + scheduling time, the pod will not be scheduled onto the node. + If the anti-affinity requirements specified by this field cease to be met + at some point during pod execution (e.g. due to a pod label update), the + system may or may not try to eventually evict the pod from its node. + When there are multiple elements, the lists of nodes corresponding to each + podAffinityTerm are intersected, i.e. all terms must be satisfied. + items: + description: |- + Defines a set of pods (namely those matching the labelSelector + relative to the given namespace(s)) that this pod should be + co-located (affinity) or not co-located (anti-affinity) with, + where co-located is defined as running on a node whose value of + the label with key matches that of any node on which + a pod of the set of pods is running + properties: + labelSelector: + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both matchLabelKeys and labelSelector. + Also, matchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. + Also, mismatchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + namespaceSelector: + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + x-kubernetes-list-type: atomic + topologyKey: + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + type: array + x-kubernetes-list-type: atomic + type: object + type: object + extraConfig: + description: |- + ExtraConfig is arbitrary additional odbselect config deeply merged over the generated config, + as a top-level YAML/JSON object. Use it to set fields the CRD does not model, such as + prometheus.max_samples, loki.max_sample_rows, per-listener auth or shutdown_timeout. + + The cluster block is reserved and rejected instead of merged: it must stay in step with the + storage nodes. Configure it through spec.etcd and spec.cluster. The listener addresses + (prometheus.bind, loki.bind, tempo.bind, pyroscope.bind, health.bind) are reserved too: the + operator publishes them as Service ports and probes them, and odbselect keeps serving its + other listeners if one moves, so a rebind would fail silently rather than crash the pod. Use + spec.signals to turn an API off. + type: object + x-kubernetes-preserve-unknown-fields: true + image: + description: |- + Image is the container image running odbselect. Defaults to the cluster image + (spec.image, else the operator's pinned image). + type: string + nodeSelector: + additionalProperties: + type: string + description: NodeSelector constrains odbselect pods to nodes with + matching labels. + type: object + podAnnotations: + additionalProperties: + type: string + description: PodAnnotations are added to every odbselect pod. + type: object + podLabels: + additionalProperties: + type: string + description: PodLabels are added to every odbselect pod. + type: object + podSecurityContext: + description: PodSecurityContext for odbselect pods. + properties: + appArmorProfile: + description: |- + appArmorProfile is the AppArmor options to use by the containers in this pod. + Note that this field cannot be set when spec.os.name is windows. + properties: + localhostProfile: + description: |- + localhostProfile indicates a profile loaded on the node that should be used. + The profile must be preconfigured on the node to work. + Must match the loaded name of the profile. + Must be set if and only if type is "Localhost". + type: string + type: + description: |- + type indicates which kind of AppArmor profile will be applied. + Valid options are: + Localhost - a profile pre-loaded on the node. + RuntimeDefault - the container runtime's default profile. + Unconfined - no AppArmor enforcement. + type: string + required: + - type + type: object + fsGroup: + description: |- + A special supplemental group that applies to all containers in a pod. + Some volume types allow the Kubelet to change the ownership of that volume + to be owned by the pod: + + 1. The owning GID will be the FSGroup + 2. The setgid bit is set (new files created in the volume will be owned by FSGroup) + 3. The permission bits are OR'd with rw-rw---- + + If unset, the Kubelet will not modify the ownership and permissions of any volume. + Note that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + fsGroupChangePolicy: + description: |- + fsGroupChangePolicy defines behavior of changing ownership and permission of the volume + before being exposed inside Pod. This field will only apply to + volume types which support fsGroup based ownership(and permissions). + It will have no effect on ephemeral volume types such as: secret, configmaps + and emptydir. + Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. + Note that this field cannot be set when spec.os.name is windows. + type: string + runAsGroup: + description: |- + The GID to run the entrypoint of the container process. + Uses runtime default if unset. + May also be set in SecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence + for that container. + Note that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + runAsNonRoot: + description: |- + Indicates that the container must run as a non-root user. + If true, the Kubelet will validate the image at runtime to ensure that it + does not run as UID 0 (root) and fail to start the container if it does. + If unset or false, no such validation will be performed. + May also be set in SecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + type: boolean + runAsUser: + description: |- + The UID to run the entrypoint of the container process. + Defaults to user specified in image metadata if unspecified. + May also be set in SecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence + for that container. + Note that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + seLinuxChangePolicy: + description: |- + seLinuxChangePolicy defines how the container's SELinux label is applied to all volumes used by the Pod. + It has no effect on nodes that do not support SELinux or to volumes does not support SELinux. + Valid values are "MountOption" and "Recursive". + + "Recursive" means relabeling of all files on all Pod volumes by the container runtime. + This may be slow for large volumes, but allows mixing privileged and unprivileged Pods sharing the same volume on the same node. + + "MountOption" mounts all eligible Pod volumes with `-o context` mount option. + This requires all Pods that share the same volume to use the same SELinux label. + It is not possible to share the same volume among privileged and unprivileged Pods. + Eligible volumes are in-tree FibreChannel and iSCSI volumes, and all CSI volumes + whose CSI driver announces SELinux support by setting spec.seLinuxMount: true in their + CSIDriver instance. Other volumes are always re-labelled recursively. + "MountOption" value is allowed only when SELinuxMount feature gate is enabled. + + If not specified and SELinuxMount feature gate is enabled, "MountOption" is used. + If not specified and SELinuxMount feature gate is disabled, "MountOption" is used for ReadWriteOncePod volumes + and "Recursive" for all other volumes. + + This field affects only Pods that have SELinux label set, either in PodSecurityContext or in SecurityContext of all containers. + + All Pods that use the same volume should use the same seLinuxChangePolicy, otherwise some pods can get stuck in ContainerCreating state. + Note that this field cannot be set when spec.os.name is windows. + type: string + seLinuxOptions: + description: |- + The SELinux context to be applied to all containers. + If unspecified, the container runtime will allocate a random SELinux context for each + container. May also be set in SecurityContext. If set in + both SecurityContext and PodSecurityContext, the value specified in SecurityContext + takes precedence for that container. + Note that this field cannot be set when spec.os.name is windows. + properties: + level: + description: Level is SELinux level label that applies + to the container. + type: string + role: + description: Role is a SELinux role label that applies + to the container. + type: string + type: + description: Type is a SELinux type label that applies + to the container. + type: string + user: + description: User is a SELinux user label that applies + to the container. + type: string + type: object + seccompProfile: + description: |- + The seccomp options to use by the containers in this pod. + Note that this field cannot be set when spec.os.name is windows. + properties: + localhostProfile: + description: |- + localhostProfile indicates a profile defined in a file on the node should be used. + The profile must be preconfigured on the node to work. + Must be a descending path, relative to the kubelet's configured seccomp profile location. + Must be set if type is "Localhost". Must NOT be set for any other type. + type: string + type: + description: |- + type indicates which kind of seccomp profile will be applied. + Valid options are: + + Localhost - a profile defined in a file on the node should be used. + RuntimeDefault - the container runtime default profile should be used. + Unconfined - no profile should be applied. + type: string + required: + - type + type: object + supplementalGroups: + description: |- + A list of groups applied to the first process run in each container, in + addition to the container's primary GID and fsGroup (if specified). If + the SupplementalGroupsPolicy feature is enabled, the + supplementalGroupsPolicy field determines whether these are in addition + to or instead of any group memberships defined in the container image. + If unspecified, no additional groups are added, though group memberships + defined in the container image may still be used, depending on the + supplementalGroupsPolicy field. + Note that this field cannot be set when spec.os.name is windows. + items: + format: int64 + type: integer + type: array + x-kubernetes-list-type: atomic + supplementalGroupsPolicy: + description: |- + Defines how supplemental groups of the first container processes are calculated. + Valid values are "Merge" and "Strict". If not specified, "Merge" is used. + (Alpha) Using the field requires the SupplementalGroupsPolicy feature gate to be enabled + and the container runtime must implement support for this feature. + Note that this field cannot be set when spec.os.name is windows. + type: string + sysctls: + description: |- + Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported + sysctls (by the container runtime) might fail to launch. + Note that this field cannot be set when spec.os.name is windows. + items: + description: Sysctl defines a kernel parameter to be set + properties: + name: + description: Name of a property to set + type: string + value: + description: Value of a property to set + type: string + required: + - name + - value + type: object + type: array + x-kubernetes-list-type: atomic + windowsOptions: + description: |- + The Windows specific settings applied to all containers. + If unspecified, the options within a container's SecurityContext will be used. + If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is linux. + properties: + gmsaCredentialSpec: + description: |- + GMSACredentialSpec is where the GMSA admission webhook + (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the + GMSA credential spec named by the GMSACredentialSpecName field. + type: string + gmsaCredentialSpecName: + description: GMSACredentialSpecName is the name of the + GMSA credential spec to use. + type: string + hostProcess: + description: |- + HostProcess determines if a container should be run as a 'Host Process' container. + All of a Pod's containers must have the same effective HostProcess value + (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). + In addition, if HostProcess is true then HostNetwork must also be set to true. + type: boolean + runAsUserName: + description: |- + The UserName in Windows to run the entrypoint of the container process. + Defaults to the user specified in image metadata if unspecified. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + type: string + type: object + type: object + replicas: + default: 2 + description: |- + Replicas is the number of odbselect pods. They hold no data, so this is purely a throughput + and availability knob and can be changed freely. + format: int32 + minimum: 0 + type: integer + resources: + description: Resources are the compute resources for each odbselect + container. + properties: + claims: + description: |- + Claims lists the names of resources, defined in spec.resourceClaims, + that are used by this container. + + This field depends on the + DynamicResourceAllocation feature gate. + + This field is immutable. It can only be set for containers. + items: + description: ResourceClaim references one entry in PodSpec.ResourceClaims. + properties: + name: + description: |- + Name must match the name of one entry in pod.spec.resourceClaims of + the Pod where this field is used. It makes that resource available + inside a container. + type: string + request: + description: |- + Request is the name chosen for a request in the referenced claim. + If empty, everything from the claim is made available, otherwise + only the result of this request. + type: string + required: + - name + type: object + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + limits: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: |- + Limits describes the maximum amount of compute resources allowed. + More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ + type: object + requests: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: |- + Requests describes the minimum amount of compute resources required. + If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, + otherwise to an implementation-defined value. Requests cannot exceed Limits. + More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ + type: object + type: object + securityContext: + description: SecurityContext for the odbselect container. + properties: + allowPrivilegeEscalation: + description: |- + AllowPrivilegeEscalation controls whether a process can gain more + privileges than its parent process. This bool directly controls if + the no_new_privs flag will be set on the container process. + AllowPrivilegeEscalation is true always when the container is: + 1) run as Privileged + 2) has CAP_SYS_ADMIN + Note that this field cannot be set when spec.os.name is windows. + type: boolean + appArmorProfile: + description: |- + appArmorProfile is the AppArmor options to use by this container. If set, this profile + overrides the pod's appArmorProfile. + Note that this field cannot be set when spec.os.name is windows. + properties: + localhostProfile: + description: |- + localhostProfile indicates a profile loaded on the node that should be used. + The profile must be preconfigured on the node to work. + Must match the loaded name of the profile. + Must be set if and only if type is "Localhost". + type: string + type: + description: |- + type indicates which kind of AppArmor profile will be applied. + Valid options are: + Localhost - a profile pre-loaded on the node. + RuntimeDefault - the container runtime's default profile. + Unconfined - no AppArmor enforcement. + type: string + required: + - type + type: object + capabilities: + description: |- + The capabilities to add/drop when running containers. + Defaults to the default set of capabilities granted by the container runtime. + Note that this field cannot be set when spec.os.name is windows. + properties: + add: + description: Added capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + x-kubernetes-list-type: atomic + drop: + description: Removed capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + x-kubernetes-list-type: atomic + type: object + privileged: + description: |- + Run container in privileged mode. + Processes in privileged containers are essentially equivalent to root on the host. + Defaults to false. + Note that this field cannot be set when spec.os.name is windows. + type: boolean + procMount: + description: |- + procMount denotes the type of proc mount to use for the containers. + The default value is Default which uses the container runtime defaults for + readonly paths and masked paths. + Note that this field cannot be set when spec.os.name is windows. + type: string + readOnlyRootFilesystem: + description: |- + Whether this container has a read-only root filesystem. + Default is false. + Note that this field cannot be set when spec.os.name is windows. + type: boolean + runAsGroup: + description: |- + The GID to run the entrypoint of the container process. + Uses runtime default if unset. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + runAsNonRoot: + description: |- + Indicates that the container must run as a non-root user. + If true, the Kubelet will validate the image at runtime to ensure that it + does not run as UID 0 (root) and fail to start the container if it does. + If unset or false, no such validation will be performed. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + type: boolean + runAsUser: + description: |- + The UID to run the entrypoint of the container process. + Defaults to user specified in image metadata if unspecified. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + seLinuxOptions: + description: |- + The SELinux context to be applied to the container. + If unspecified, the container runtime will allocate a random SELinux context for each + container. May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is windows. + properties: + level: + description: Level is SELinux level label that applies + to the container. + type: string + role: + description: Role is a SELinux role label that applies + to the container. + type: string + type: + description: Type is a SELinux type label that applies + to the container. + type: string + user: + description: User is a SELinux user label that applies + to the container. + type: string + type: object + seccompProfile: + description: |- + The seccomp options to use by this container. If seccomp options are + provided at both the pod & container level, the container options + override the pod options. + Note that this field cannot be set when spec.os.name is windows. + properties: + localhostProfile: + description: |- + localhostProfile indicates a profile defined in a file on the node should be used. + The profile must be preconfigured on the node to work. + Must be a descending path, relative to the kubelet's configured seccomp profile location. + Must be set if type is "Localhost". Must NOT be set for any other type. + type: string + type: + description: |- + type indicates which kind of seccomp profile will be applied. + Valid options are: + + Localhost - a profile defined in a file on the node should be used. + RuntimeDefault - the container runtime default profile should be used. + Unconfined - no profile should be applied. + type: string + required: + - type + type: object + windowsOptions: + description: |- + The Windows specific settings applied to all containers. + If unspecified, the options from the PodSecurityContext will be used. + If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is linux. + properties: + gmsaCredentialSpec: + description: |- + GMSACredentialSpec is where the GMSA admission webhook + (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the + GMSA credential spec named by the GMSACredentialSpecName field. + type: string + gmsaCredentialSpecName: + description: GMSACredentialSpecName is the name of the + GMSA credential spec to use. + type: string + hostProcess: + description: |- + HostProcess determines if a container should be run as a 'Host Process' container. + All of a Pod's containers must have the same effective HostProcess value + (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). + In addition, if HostProcess is true then HostNetwork must also be set to true. + type: boolean + runAsUserName: + description: |- + The UserName in Windows to run the entrypoint of the container process. + Defaults to the user specified in image metadata if unspecified. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + type: string + type: object + type: object + service: + description: |- + Service configures the client-facing Service that exposes the query APIs. One Service carries + every enabled API, each on its own port; they share a pod set, so splitting them across four + Services would only add DNS names. + properties: + annotations: + additionalProperties: + type: string + description: Annotations added to the client Service. + type: object + type: + default: ClusterIP + description: Type of the client Service. + enum: + - ClusterIP + - NodePort + - LoadBalancer + type: string + type: object + serviceAccountName: + description: |- + ServiceAccountName is the ServiceAccount for the odbselect pods. Empty falls back to + spec.serviceAccountName. + type: string + tolerations: + description: Tolerations for odbselect pods. + items: + description: |- + The pod this Toleration is attached to tolerates any taint that matches + the triple using the matching operator . + properties: + effect: + description: |- + Effect indicates the taint effect to match. Empty means match all taint effects. + When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute. + type: string + key: + description: |- + Key is the taint key that the toleration applies to. Empty means match all taint keys. + If the key is empty, operator must be Exists; this combination means to match all values and all keys. + type: string + operator: + description: |- + Operator represents a key's relationship to the value. + Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal. + Exists is equivalent to wildcard for value, so that a pod can + tolerate all taints of a particular category. + Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators). + type: string + tolerationSeconds: + description: |- + TolerationSeconds represents the period of time the toleration (which must be + of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, + it is not set, which means tolerate the taint forever (do not evict). Zero and + negative values will be treated as 0 (evict immediately) by the system. + format: int64 + type: integer + value: + description: |- + Value is the taint value the toleration matches to. + If the operator is Exists, the value should be empty, otherwise just a regular string. + type: string + type: object + type: array + topologySpreadConstraints: + description: TopologySpreadConstraints for odbselect pods. + items: + description: TopologySpreadConstraint specifies how to spread + matching pods among the given topology. + properties: + labelSelector: + description: |- + LabelSelector is used to find matching pods. + Pods that match this label selector are counted to determine the number of pods + in their corresponding topology domain. + properties: + matchExpressions: + description: matchExpressions is a list of label selector + requirements. The requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that the selector + applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select the pods over which + spreading will be calculated. The keys are used to lookup values from the + incoming pod labels, those key-value labels are ANDed with labelSelector + to select the group of existing pods over which spreading will be calculated + for the incoming pod. The same key is forbidden to exist in both MatchLabelKeys and LabelSelector. + MatchLabelKeys cannot be set when LabelSelector isn't set. + Keys that don't exist in the incoming pod labels will + be ignored. A null or empty list means only match against labelSelector. + + This is a beta field and requires the MatchLabelKeysInPodTopologySpread feature gate to be enabled (enabled by default). + items: + type: string + type: array + x-kubernetes-list-type: atomic + maxSkew: + description: |- + MaxSkew describes the degree to which pods may be unevenly distributed. + When `whenUnsatisfiable=DoNotSchedule`, it is the maximum permitted difference + between the number of matching pods in the target topology and the global minimum. + The global minimum is the minimum number of matching pods in an eligible domain + or zero if the number of eligible domains is less than MinDomains. + For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same + labelSelector spread as 2/2/1: + In this case, the global minimum is 1. + | zone1 | zone2 | zone3 | + | P P | P P | P | + - if MaxSkew is 1, incoming pod can only be scheduled to zone3 to become 2/2/2; + scheduling it onto zone1(zone2) would make the ActualSkew(3-1) on zone1(zone2) + violate MaxSkew(1). + - if MaxSkew is 2, incoming pod can be scheduled onto any zone. + When `whenUnsatisfiable=ScheduleAnyway`, it is used to give higher precedence + to topologies that satisfy it. + It's a required field. Default value is 1 and 0 is not allowed. + format: int32 + type: integer + minDomains: + description: |- + MinDomains indicates a minimum number of eligible domains. + When the number of eligible domains with matching topology keys is less than minDomains, + Pod Topology Spread treats "global minimum" as 0, and then the calculation of Skew is performed. + And when the number of eligible domains with matching topology keys equals or greater than minDomains, + this value has no effect on scheduling. + As a result, when the number of eligible domains is less than minDomains, + scheduler won't schedule more than maxSkew Pods to those domains. + If value is nil, the constraint behaves as if MinDomains is equal to 1. + Valid values are integers greater than 0. + When value is not nil, WhenUnsatisfiable must be DoNotSchedule. + + For example, in a 3-zone cluster, MaxSkew is set to 2, MinDomains is set to 5 and pods with the same + labelSelector spread as 2/2/2: + | zone1 | zone2 | zone3 | + | P P | P P | P P | + The number of domains is less than 5(MinDomains), so "global minimum" is treated as 0. + In this situation, new pod with the same labelSelector cannot be scheduled, + because computed skew will be 3(3 - 0) if new Pod is scheduled to any of the three zones, + it will violate MaxSkew. + format: int32 + type: integer + nodeAffinityPolicy: + description: |- + NodeAffinityPolicy indicates how we will treat Pod's nodeAffinity/nodeSelector + when calculating pod topology spread skew. Options are: + - Honor: only nodes matching nodeAffinity/nodeSelector are included in the calculations. + - Ignore: nodeAffinity/nodeSelector are ignored. All nodes are included in the calculations. + + If this value is nil, the behavior is equivalent to the Honor policy. + type: string + nodeTaintsPolicy: + description: |- + NodeTaintsPolicy indicates how we will treat node taints when calculating + pod topology spread skew. Options are: + - Honor: nodes without taints, along with tainted nodes for which the incoming pod + has a toleration, are included. + - Ignore: node taints are ignored. All nodes are included. + + If this value is nil, the behavior is equivalent to the Ignore policy. + type: string + topologyKey: + description: |- + TopologyKey is the key of node labels. Nodes that have a label with this key + and identical values are considered to be in the same topology. + We consider each as a "bucket", and try to put balanced number + of pods into each bucket. + We define a domain as a particular instance of a topology. + Also, we define an eligible domain as a domain whose nodes meet the requirements of + nodeAffinityPolicy and nodeTaintsPolicy. + e.g. If TopologyKey is "kubernetes.io/hostname", each Node is a domain of that topology. + And, if TopologyKey is "topology.kubernetes.io/zone", each zone is a domain of that topology. + It's a required field. + type: string + whenUnsatisfiable: + description: |- + WhenUnsatisfiable indicates how to deal with a pod if it doesn't satisfy + the spread constraint. + - DoNotSchedule (default) tells the scheduler not to schedule it. + - ScheduleAnyway tells the scheduler to schedule the pod in any location, + but giving higher precedence to topologies that would help reduce the + skew. + A constraint is considered "Unsatisfiable" for an incoming pod + if and only if every possible node assignment for that pod would violate + "MaxSkew" on some topology. + For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same + labelSelector spread as 3/1/1: + | zone1 | zone2 | zone3 | + | P P P | P | P | + If WhenUnsatisfiable is set to DoNotSchedule, incoming pod can only be scheduled + to zone2(zone3) to become 3/2/1(3/1/2) as ActualSkew(2-1) on zone2(zone3) satisfies + MaxSkew(1). In other words, the cluster can still be imbalanced, but scheduler + won't make it *more* imbalanced. + It's a required field. + type: string + required: + - maxSkew + - topologyKey + - whenUnsatisfiable + type: object + type: array + type: object replicas: default: 3 description: |- @@ -3230,9 +4926,10 @@ spec: StatefulSet pod that ingests, queries, stores, and replicates. For real redundancy this should be at least the replication factor (see Cluster.ReplicationFactor). - This is the storage dimension only. Ingest capacity scales separately via spec.ingest, and - scaling either has no effect on the other: membership lives in etcd and the ring is computed - from it, so an ingest pod needs no knowledge of how many storage nodes there are. + This is the storage dimension only. Ingest and query capacity scale separately via + spec.ingest and spec.query, and scaling any of them has no effect on the others: membership + lives in etcd and the ring is computed from it, so a stateless pod needs no knowledge of how + many storage nodes there are. format: int32 minimum: 1 type: integer @@ -3933,6 +5630,16 @@ spec: - Ready - Degraded type: string + queryReadyReplicas: + description: QueryReadyReplicas is the number of ready odbselect nodes. + format: int32 + type: integer + queryReplicas: + description: |- + QueryReplicas is the desired number of odbselect nodes. Zero when no query pool is + configured. + format: int32 + type: integer readyReplicas: description: ReadyReplicas is the number of ready oteldb nodes. format: int32 diff --git a/config/samples/db_v1alpha1_oteldbcluster.yaml b/config/samples/db_v1alpha1_oteldbcluster.yaml index 904e8ad..0907a18 100644 --- a/config/samples/db_v1alpha1_oteldbcluster.yaml +++ b/config/samples/db_v1alpha1_oteldbcluster.yaml @@ -111,15 +111,17 @@ spec: cluster: replicationFactor: 2 --- -# Example splitting ingest off the storage nodes: three storage nodes plus a stateless odbingest -# pool that scales on its own. The pool holds no data and is not a ring member — it watches etcd -# for membership and routes each shard to its primary, so scaling `replicas` needs no change here. +# Example splitting ingest and query off the storage nodes: three storage nodes plus a stateless +# odbingest pool and a stateless odbselect pool, each scaling on its own. Neither pool holds data or +# is a ring member — both watch etcd for membership and resolve shard owners locally, so scaling +# `replicas` needs no change here. # -# NOTE: odbingest is not shipped in the released oteldb image yet, hence the explicit image below. +# NOTE: neither odbingest nor odbselect is shipped in the released oteldb image yet, hence the +# explicit images below. apiVersion: db.oteldb.io/v1alpha1 kind: OtelDBCluster metadata: - name: oteldbcluster-ingest-sample + name: oteldbcluster-roles-sample spec: replicas: 3 etcd: @@ -144,3 +146,24 @@ spec: time_threshold: 1h service: type: ClusterIP + + # The read half. odbselect serves the four query APIs on four separate listeners, published as + # four ports on one `-query` Service; `spec.signals` decides which of them are served. + query: + replicas: 4 + image: "ghcr.io/oteldb/oteldb:with-odbselect" + resources: + requests: + cpu: "2" + memory: 2Gi + limits: + memory: 4Gi + # `cluster` and every `*.bind` are reserved: the binds are what the Service publishes and the + # probes hit. Everything else in a block is free. + extraConfig: + prometheus: + max_samples: 20000000 + loki: + max_sample_rows: 500000 + service: + type: ClusterIP diff --git a/internal/controller/extraconfig.go b/internal/controller/extraconfig.go index 86d3537..19eae57 100644 --- a/internal/controller/extraconfig.go +++ b/internal/controller/extraconfig.go @@ -40,7 +40,7 @@ var reservedConfigPaths = map[string]string{ "storage.dir": "use spec.storage.dir", "storage.wal_dir": "use spec.storage.dir", "storage.s3": "use spec.storage.s3", - "storage.cluster": "use spec.cluster and spec.etcd.endpoints", + "storage.cluster": hintRingFromSpec, "storage.flush_interval": "use spec.engine.flushInterval", "storage.read_cache_bytes": "use spec.engine.readCacheSize", "storage.decode_cache_bytes": "use spec.engine.decodeCacheSize", diff --git a/internal/controller/ingest.go b/internal/controller/ingest.go index 8d6628b..e7cccc8 100644 --- a/internal/controller/ingest.go +++ b/internal/controller/ingest.go @@ -34,7 +34,7 @@ import ( // must stay in step with the storage nodes, and a mismatch is silent: it resolves a different owner // set, so writes land where no read will look for them. var ingestReservedConfigPaths = map[string]string{ - keyCluster: "use spec.cluster and spec.etcd.endpoints", + keyCluster: hintRingFromSpec, } func ingestEnabled(cr *dbv1alpha1.OtelDBCluster) bool { return cr.Spec.Ingest != nil } @@ -55,10 +55,10 @@ func renderIngestConfig(cr *dbv1alpha1.OtelDBCluster, etcdEndpoints []string) (s cfg := map[string]any{ keyCluster: ringConfig(cr, etcdEndpoints), "prometheus_remote_write": map[string]any{ - keyBind: fmt.Sprintf("0.0.0.0:%d", portPromRW), + keyBind: bindAll(portPromRW), }, "otlp": map[string]any{ - "grpc_bind": fmt.Sprintf("0.0.0.0:%d", portOTLPGRPC), + "grpc_bind": bindAll(portOTLPGRPC), }, } @@ -170,7 +170,7 @@ func buildIngestDeployment(cr *dbv1alpha1.OtelDBCluster, configHash string) *app } podLabels := mergeLabels(ingestCommonLabels(cr), spec.PodLabels) - annotations := mergeLabels(map[string]string{"oteldb.io/config-hash": configHash}, spec.PodAnnotations) + annotations := mergeLabels(map[string]string{annConfigHash: configHash}, spec.PodAnnotations) container := corev1.Container{ Name: ingestAppName, @@ -178,7 +178,7 @@ func buildIngestDeployment(cr *dbv1alpha1.OtelDBCluster, configHash string) *app ImagePullPolicy: cr.Spec.ImagePullPolicy, Command: []string{ingestBinPath}, Args: []string{"--config=" + configMountPath + "/" + ingestConfigFileName}, - Env: ingestPodEnv(cr), + Env: statelessPodEnv(cr), Ports: []corev1.ContainerPort{ {Name: portNameOTLPGRPC, ContainerPort: portOTLPGRPC, Protocol: corev1.ProtocolTCP}, {Name: portNameIngestHTTP, ContainerPort: portPromRW, Protocol: corev1.ProtocolTCP}, @@ -232,26 +232,4 @@ func buildIngestDeployment(cr *dbv1alpha1.OtelDBCluster, configHash string) *app } } -// ingestPodEnv is the odbingest pod environment. Unlike a storage pod it carries no ring identity: -// odbingest joins nothing, so it has no id, address or zone to advertise. -func ingestPodEnv(cr *dbv1alpha1.OtelDBCluster) []corev1.EnvVar { - env := []corev1.EnvVar{ - {Name: "OTEL_EXPORTER_PROMETHEUS_HOST", Value: "0.0.0.0"}, - {Name: "OTEL_EXPORTER_PROMETHEUS_PORT", Value: fmt.Sprintf("%d", portSelfMetric)}, - } - if lvl := cr.Spec.LogLevel; lvl != "" { - env = append(env, corev1.EnvVar{Name: "OTEL_LOG_LEVEL", Value: lvl}) - } - return env -} - -func ingestProbe(path string) *corev1.Probe { - return &corev1.Probe{ - ProbeHandler: corev1.ProbeHandler{ - HTTPGet: &corev1.HTTPGetAction{Path: path, Port: intstr.FromString(portNameIngestHTTP)}, - }, - PeriodSeconds: 10, - TimeoutSeconds: 3, - FailureThreshold: 3, - } -} +func ingestProbe(path string) *corev1.Probe { return roleProbe(path, portNameIngestHTTP) } diff --git a/internal/controller/ingest_test.go b/internal/controller/ingest_test.go index 8cf185f..a5567d2 100644 --- a/internal/controller/ingest_test.go +++ b/internal/controller/ingest_test.go @@ -79,24 +79,7 @@ func TestRenderIngestConfig(t *testing.T) { require.Equal(t, "0.0.0.0:4317", otlp["grpc_bind"]) } -// A ring parameter that differs between the roles does not fail: the pool resolves a different -// owner set than the storage nodes do, and writes land where no read will look for them. Both -// renderers must therefore agree key for key. -func TestIngestRingMatchesStorageRing(t *testing.T) { - cr := ingestCluster() - - storageOut, err := renderConfig(cr, cr.Spec.Etcd.Endpoints) - require.NoError(t, err) - var storageCfg map[string]any - require.NoError(t, yaml.Unmarshal([]byte(storageOut), &storageCfg)) - - storageRing := storageCfg["storage"].(map[string]any)["cluster"].(map[string]any) - ingestRing := renderedIngest(t, cr)["cluster"].(map[string]any) - - for _, key := range []string{"etcd", "rf", "shards_per_tenant", "root"} { - require.Equal(t, storageRing[key], ingestRing[key], "ring parameter %q differs between roles", key) - } -} +// The ring every role must agree on is covered for all three roles by TestRoleRingsMatch. func TestRenderIngestConfigExtraConfig(t *testing.T) { cr := ingestCluster() diff --git a/internal/controller/naming.go b/internal/controller/naming.go index ba81726..bf48604 100644 --- a/internal/controller/naming.go +++ b/internal/controller/naming.go @@ -29,13 +29,33 @@ const defaultImage = "ghcr.io/oteldb/oteldb:v0.46.0" const ( appName = "oteldb" // app label value and container name ingestAppName = "odbingest" // ingest app label value, container name and binary + queryAppName = "odbselect" // query app label value, container name and binary valStorage = "storage" // oteldb signal-backend value and component label valIngest = "ingest" // ingest component label + valQuery = "query" // query component label defaultDataDir = "/var/lib/oteldb" // default storage.dir / WAL dir keyBind = "bind" // oteldb per-API bind config key + // bindDisabled is odbselect's "do not serve this API" bind. An omitted block is not enough: + // odbselect applies its defaults to every API block unconditionally, so a missing one is served + // on its default port (see cmd/odbselect/config.go, setDefaults and enabled). + bindDisabled = "-" envAWSAccessKeyID = "AWS_ACCESS_KEY_ID" envAWSSecretAccessKey = "AWS_SECRET_ACCESS_KEY" + envPrometheusHost = "OTEL_EXPORTER_PROMETHEUS_HOST" + envPrometheusPort = "OTEL_EXPORTER_PROMETHEUS_PORT" + envLogLevel = "OTEL_LOG_LEVEL" + + // annConfigHash carries the rendered config's digest on a pod template, so a config change rolls + // the workload. + annConfigHash = "oteldb.io/config-hash" + + // hintRingFromSpec is the remediation for a reserved ring path: every role renders it from the + // same two spec fields, which is what keeps them from resolving different owner sets. + hintRingFromSpec = "use spec.cluster and spec.etcd.endpoints" + + // bindAllHost is the address every listener binds, leaving reachability to the Service. + bindAllHost = "0.0.0.0" ) // oteldb config keys shared by the renderer and the reserved-path guard. @@ -52,6 +72,17 @@ const ( keyPort = "port" ) +// Query API config block keys, shared by cmd/oteldb and cmd/odbselect. +const ( + keyPrometheus = "prometheus" + keyTempo = "tempo" + keyLoki = "loki" + keyPyroscope = "pyroscope" + // keyHealth is odbselect's health listener block. cmd/oteldb spells the same thing + // "health_check". + keyHealth = "health" +) + // Container ports exposed by every oteldb node. Names must be <= 15 chars (k8s port-name limit). const ( portOTLPGRPC = 4317 @@ -91,9 +122,12 @@ const ( configMountPath = "/etc/otel" configFileName = "config.yml" ingestConfigFileName = "odbingest.yml" + queryConfigFileName = "odbselect.yml" // ingestBinPath is the odbingest entrypoint inside the oteldb image, which defaults to the // oteldb binary. ingestBinPath = "/usr/local/bin/odbingest" + // queryBinPath is the odbselect entrypoint inside the oteldb image. + queryBinPath = "/usr/local/bin/odbselect" ) // resourceNames centralizes the derived names for a cluster's child objects. @@ -110,6 +144,9 @@ func (n resourceNames) clientService() string { return n.base } func (n resourceNames) ingestDeployment() string { return n.base + "-ingest" } func (n resourceNames) ingestConfigMap() string { return n.base + "-ingest-config" } func (n resourceNames) ingestService() string { return n.base + "-ingest" } +func (n resourceNames) queryDeployment() string { return n.base + "-query" } +func (n resourceNames) queryConfigMap() string { return n.base + "-query-config" } +func (n resourceNames) queryService() string { return n.base + "-query" } // roleSelectorLabels are the immutable pod-selector labels for one role's pods. The role's binary // is the app name, which is what keeps the storage and ingest pod sets from selecting each other: @@ -148,6 +185,14 @@ func ingestCommonLabels(cr *dbv1alpha1.OtelDBCluster) map[string]string { return roleCommonLabels(cr, ingestAppName, valIngest) } +func querySelectorLabels(cr *dbv1alpha1.OtelDBCluster) map[string]string { + return roleSelectorLabels(cr, queryAppName) +} + +func queryCommonLabels(cr *dbv1alpha1.OtelDBCluster) map[string]string { + return roleCommonLabels(cr, queryAppName, valQuery) +} + // mergeLabels returns a new map combining base with extra (extra wins). func mergeLabels(base, extra map[string]string) map[string]string { out := make(map[string]string, len(base)+len(extra)) diff --git a/internal/controller/oteldbcluster_controller.go b/internal/controller/oteldbcluster_controller.go index f5f58c7..83356dd 100644 --- a/internal/controller/oteldbcluster_controller.go +++ b/internal/controller/oteldbcluster_controller.go @@ -104,7 +104,10 @@ func (r *OtelDBClusterReconciler) reconcile(ctx context.Context, cr *dbv1alpha1. if err := r.apply(ctx, cr, buildStatefulSet(cr, hash)); err != nil { return fmt.Errorf("statefulset: %w", err) } - return r.reconcileIngest(ctx, cr, endpoints) + if err := r.reconcileIngest(ctx, cr, endpoints); err != nil { + return err + } + return r.reconcileQuery(ctx, cr, endpoints) } // reconcileIngest applies the stateless odbingest pool, or removes it when spec.ingest is absent. @@ -148,6 +151,47 @@ func (r *OtelDBClusterReconciler) pruneIngest(ctx context.Context, cr *dbv1alpha return nil } +// reconcileQuery applies the stateless odbselect pool, or removes it when spec.query is absent. +func (r *OtelDBClusterReconciler) reconcileQuery(ctx context.Context, cr *dbv1alpha1.OtelDBCluster, endpoints []string) error { + if !queryEnabled(cr) { + return r.pruneQuery(ctx, cr) + } + + cm, err := buildQueryConfigMap(cr, endpoints) + if err != nil { + return err + } + if err := r.apply(ctx, cr, cm); err != nil { + return fmt.Errorf("query configmap: %w", err) + } + hash := configHash(cm.Data[queryConfigFileName]) + + if err := r.apply(ctx, cr, buildQueryService(cr)); err != nil { + return fmt.Errorf("query service: %w", err) + } + if err := r.apply(ctx, cr, buildQueryDeployment(cr, hash)); err != nil { + return fmt.Errorf("query deployment: %w", err) + } + return nil +} + +// pruneQuery deletes the query pool's objects. Owner references only collect them when the whole +// cluster goes away, so dropping spec.query has to remove them explicitly. +func (r *OtelDBClusterReconciler) pruneQuery(ctx context.Context, cr *dbv1alpha1.OtelDBCluster) error { + n := namesFor(cr) + objs := []client.Object{ + &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: n.queryDeployment(), Namespace: cr.Namespace}}, + &corev1.Service{ObjectMeta: metav1.ObjectMeta{Name: n.queryService(), Namespace: cr.Namespace}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: n.queryConfigMap(), Namespace: cr.Namespace}}, + } + for _, obj := range objs { + if err := r.Delete(ctx, obj); client.IgnoreNotFound(err) != nil { + return fmt.Errorf("delete query %T: %w", obj, err) + } + } + return nil +} + // apply creates or updates obj as a child of cr, preserving server-managed fields. func (r *OtelDBClusterReconciler) apply(ctx context.Context, cr *dbv1alpha1.OtelDBCluster, obj client.Object) error { // Snapshot the desired spec so we can restore it after CreateOrUpdate fetches the live object. @@ -213,27 +257,27 @@ func (r *OtelDBClusterReconciler) updateStatus(ctx context.Context, cr *dbv1alph return err } - ingestDesired := ingestReplicasOf(cr) - ingestReady := int32(0) - if ingestEnabled(cr) { - deploy := &appsv1.Deployment{} - key := client.ObjectKey{Namespace: cr.Namespace, Name: namesFor(cr).ingestDeployment()} - if err := r.Get(ctx, key, deploy); err == nil { - ingestReady = deploy.Status.ReadyReplicas - } else if !apierrors.IsNotFound(err) { - return err - } + n := namesFor(cr) + ingestReady, err := r.deploymentReady(ctx, cr, ingestEnabled(cr), n.ingestDeployment()) + if err != nil { + return err + } + queryReady, err := r.deploymentReady(ctx, cr, queryEnabled(cr), n.queryDeployment()) + if err != nil { + return err } cr.Status.Replicas = desired cr.Status.ReadyReplicas = ready - cr.Status.IngestReplicas = ingestDesired + cr.Status.IngestReplicas = ingestReplicasOf(cr) cr.Status.IngestReadyReplicas = ingestReady + cr.Status.QueryReplicas = queryReplicasOf(cr) + cr.Status.QueryReadyReplicas = queryReady cr.Status.EtcdEndpoints = cr.Spec.Etcd.Endpoints cr.Status.ObservedGeneration = cr.Generation - // The phase tracks the storage nodes: they are what holds the data, and an ingest pool that is - // still rolling out does not make the cluster unavailable. + // The phase tracks the storage nodes: they are what holds the data, and a stateless pool that + // is still rolling out does not make the cluster unavailable. switch { case ready == 0: cr.Status.Phase = dbv1alpha1.PhasePending @@ -252,6 +296,22 @@ func (r *OtelDBClusterReconciler) updateStatus(ctx context.Context, cr *dbv1alph return r.Status().Update(ctx, cr) } +// deploymentReady reads a stateless role's ready replica count, or zero when the role is disabled +// or its Deployment does not exist yet. +func (r *OtelDBClusterReconciler) deploymentReady(ctx context.Context, cr *dbv1alpha1.OtelDBCluster, enabled bool, name string) (int32, error) { + if !enabled { + return 0, nil + } + deploy := &appsv1.Deployment{} + if err := r.Get(ctx, client.ObjectKey{Namespace: cr.Namespace, Name: name}, deploy); err != nil { + if apierrors.IsNotFound(err) { + return 0, nil + } + return 0, err + } + return deploy.Status.ReadyReplicas, nil +} + func (r *OtelDBClusterReconciler) setDegraded(ctx context.Context, cr *dbv1alpha1.OtelDBCluster, reason string, cause error) { cr.Status.Phase = dbv1alpha1.PhaseDegraded setCondition(cr, dbv1alpha1.ConditionDegraded, metav1.ConditionTrue, reason, cause.Error()) diff --git a/internal/controller/query.go b/internal/controller/query.go new file mode 100644 index 0000000..575b4cc --- /dev/null +++ b/internal/controller/query.go @@ -0,0 +1,271 @@ +/* +Copyright 2026. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package controller + +import ( + "encoding/json" + "fmt" + + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/yaml" + + dbv1alpha1 "github.com/oteldb/operator/api/v1alpha1" +) + +// queryReservedConfigPaths are the odbselect config paths the operator owns. +// +// The ring parameters must stay in step with the storage nodes, and a mismatch is silent: it +// resolves a different owner set, so reads look where the data is not. +// +// The listener addresses are reserved for the same reason. odbselect serves each API on its own +// listener and its health endpoints on a fifth, so moving one bind does not stop the process or +// fail a probe — it leaves the Service port the operator published pointing at nothing while the +// pod stays Ready. spec.signals is how an API is turned off. +var queryReservedConfigPaths = map[string]string{ + keyCluster: hintRingFromSpec, + + keyPrometheus + "." + keyBind: "use spec.signals.metrics", + keyLoki + "." + keyBind: "use spec.signals.logs", + keyTempo + "." + keyBind: "use spec.signals.traces", + keyPyroscope + "." + keyBind: "use spec.signals.profiles", + keyHealth + "." + keyBind: "the operator probes this listener", +} + +func queryEnabled(cr *dbv1alpha1.OtelDBCluster) bool { return cr.Spec.Query != nil } + +func queryReplicasOf(cr *dbv1alpha1.OtelDBCluster) int32 { + if !queryEnabled(cr) { + return 0 + } + if r := cr.Spec.Query.Replicas; r != nil { + return *r + } + return 2 +} + +// queryAPIs are the four query APIs odbselect serves, each on its own listener, paired with the +// signal that decides whether the cluster stores anything for it. +func queryAPIs(cr *dbv1alpha1.OtelDBCluster) []queryAPI { + return []queryAPI{ + {key: keyPrometheus, port: namedPort{portNamePromHTTP, portPromHTTP}, enabled: metricsEnabled(cr)}, + {key: keyLoki, port: namedPort{portNameLokiHTTP, portLokiHTTP}, enabled: logsEnabled(cr)}, + {key: keyTempo, port: namedPort{portNameTempoHTTP, portTempoHTTP}, enabled: tracesEnabled(cr)}, + {key: keyPyroscope, port: namedPort{portNamePyroscope, portPyroscope}, enabled: profilesEnabled(cr)}, + } +} + +// queryAPI is one odbselect query API: its config block, its port, and whether it is served. +type queryAPI struct { + key string + port namedPort + enabled bool +} + +// renderQueryConfig builds odbselect.yml. odbselect holds no data and joins nothing, so the whole +// config is the ring it reads through plus the listeners it answers on. +func renderQueryConfig(cr *dbv1alpha1.OtelDBCluster, etcdEndpoints []string) (string, error) { + if err := validateSignals(cr); err != nil { + return "", err + } + + cfg := map[string]any{ + keyCluster: ringConfig(cr, etcdEndpoints), + keyHealth: map[string]any{keyBind: bindAll(portHealth)}, + } + + // A disabled signal is switched off explicitly rather than by dropping its block: odbselect + // defaults every API block it does not find, so an omitted one is served on its default port. + for _, api := range queryAPIs(cr) { + bind := bindDisabled + if api.enabled { + bind = bindAll(api.port.port) + } + cfg[api.key] = map[string]any{keyBind: bind} + } + + if raw := cr.Spec.Query.ExtraConfig; raw != nil && len(raw.Raw) > 0 { + var extra map[string]any + if err := json.Unmarshal(raw.Raw, &extra); err != nil { + return "", invalidSpec("parse spec.query.extraConfig: %s", err) + } + if err := validateExtraConfigPaths("spec.query.extraConfig", extra, queryReservedConfigPaths); err != nil { + return "", err + } + deepMerge(cfg, extra) + } + + out, err := yaml.Marshal(cfg) + if err != nil { + return "", fmt.Errorf("marshal query config: %w", err) + } + return string(out), nil +} + +// buildQueryConfigMap renders the shared odbselect config into a ConfigMap. +func buildQueryConfigMap(cr *dbv1alpha1.OtelDBCluster, etcdEndpoints []string) (*corev1.ConfigMap, error) { + data, err := renderQueryConfig(cr, etcdEndpoints) + if err != nil { + return nil, err + } + n := namesFor(cr) + return &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: n.queryConfigMap(), + Namespace: cr.Namespace, + Labels: queryCommonLabels(cr), + }, + Data: map[string]string{queryConfigFileName: data}, + }, nil +} + +// buildQueryService exposes every enabled query API across the odbselect pool. This is where +// Grafana points; the cluster's client Service keeps fronting the storage nodes. +// +// One Service carries all four APIs. They are one pod set behind one selector and each API already +// has its own port, so a Service per API would add DNS names and, for a LoadBalancer type, four +// load balancers, without making anything addressable that is not addressable now. +func buildQueryService(cr *dbv1alpha1.OtelDBCluster) *corev1.Service { + n := namesFor(cr) + svcType := cr.Spec.Query.Service.Type + if svcType == "" { + svcType = corev1.ServiceTypeClusterIP + } + + ports := make([]corev1.ServicePort, 0, len(queryAPIs(cr))+1) + for _, p := range queryPorts(cr) { + ports = append(ports, corev1.ServicePort{ + Name: p.name, + Port: p.port, + TargetPort: intstr.FromString(p.name), + Protocol: corev1.ProtocolTCP, + }) + } + + return &corev1.Service{ + ObjectMeta: metav1.ObjectMeta{ + Name: n.queryService(), + Namespace: cr.Namespace, + Labels: queryCommonLabels(cr), + Annotations: cr.Spec.Query.Service.Annotations, + }, + Spec: corev1.ServiceSpec{ + Type: svcType, + Selector: querySelectorLabels(cr), + Ports: ports, + }, + } +} + +// queryPorts are the ports an odbselect pod listens on: one per enabled API plus self-metrics. The +// health listener is deliberately not published — it is probed, not served to clients. +func queryPorts(cr *dbv1alpha1.OtelDBCluster) []namedPort { + apis := queryAPIs(cr) + ports := make([]namedPort, 0, len(apis)+1) + for _, api := range apis { + if api.enabled { + ports = append(ports, api.port) + } + } + return append(ports, namedPort{portNameSelfMetric, portSelfMetric}) +} + +// buildQueryDeployment builds the odbselect pool. It is a Deployment, not a StatefulSet: the pods +// hold no data, are not ring members, and need neither a stable identity nor a volume. +func buildQueryDeployment(cr *dbv1alpha1.OtelDBCluster, configHash string) *appsv1.Deployment { + n := namesFor(cr) + spec := cr.Spec.Query + + image := spec.Image + if image == "" { + image = imageOf(cr) + } + + serviceAccount := spec.ServiceAccountName + if serviceAccount == "" { + serviceAccount = cr.Spec.ServiceAccountName + } + + podLabels := mergeLabels(queryCommonLabels(cr), spec.PodLabels) + annotations := mergeLabels(map[string]string{annConfigHash: configHash}, spec.PodAnnotations) + + ports := make([]corev1.ContainerPort, 0, len(queryPorts(cr))+1) + for _, p := range queryPorts(cr) { + ports = append(ports, corev1.ContainerPort{Name: p.name, ContainerPort: p.port, Protocol: corev1.ProtocolTCP}) + } + ports = append(ports, corev1.ContainerPort{Name: portNameHealth, ContainerPort: portHealth, Protocol: corev1.ProtocolTCP}) + + container := corev1.Container{ + Name: queryAppName, + Image: image, + ImagePullPolicy: cr.Spec.ImagePullPolicy, + Command: []string{queryBinPath}, + Args: []string{"--config=" + configMountPath + "/" + queryConfigFileName}, + Env: statelessPodEnv(cr), + Ports: ports, + VolumeMounts: []corev1.VolumeMount{ + {Name: configVolumeName, MountPath: configMountPath}, + }, + Resources: spec.Resources, + SecurityContext: spec.SecurityContext, + // The probes hit the health listener, not a query API: it is the one listener odbselect + // always serves, whatever spec.signals leaves enabled. + LivenessProbe: roleProbe("/healthz", portNameHealth), + // /readyz answers 503 until the ring has a member, keeping a starting pod out of the load + // balancer while a query could only return an empty result that looks like an answer. + ReadinessProbe: roleProbe("/readyz", portNameHealth), + } + + return &appsv1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: n.queryDeployment(), + Namespace: cr.Namespace, + Labels: queryCommonLabels(cr), + }, + Spec: appsv1.DeploymentSpec{ + Replicas: ptr.To(queryReplicasOf(cr)), + Selector: &metav1.LabelSelector{MatchLabels: querySelectorLabels(cr)}, + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: podLabels, + Annotations: annotations, + }, + Spec: corev1.PodSpec{ + ServiceAccountName: serviceAccount, + ImagePullSecrets: cr.Spec.ImagePullSecrets, + SecurityContext: spec.PodSecurityContext, + NodeSelector: spec.NodeSelector, + Affinity: affinityOr(spec.Affinity, querySelectorLabels(cr)), + Tolerations: spec.Tolerations, + TopologySpreadConstraints: spec.TopologySpreadConstraints, + Containers: []corev1.Container{container}, + Volumes: []corev1.Volume{{ + Name: configVolumeName, + VolumeSource: corev1.VolumeSource{ + ConfigMap: &corev1.ConfigMapVolumeSource{ + LocalObjectReference: corev1.LocalObjectReference{Name: n.queryConfigMap()}, + }, + }, + }}, + }, + }, + }, + } +} diff --git a/internal/controller/query_test.go b/internal/controller/query_test.go new file mode 100644 index 0000000..6713d5f --- /dev/null +++ b/internal/controller/query_test.go @@ -0,0 +1,267 @@ +/* +Copyright 2026. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package controller + +import ( + "errors" + "testing" + + "github.com/stretchr/testify/require" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/labels" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/utils/ptr" + "sigs.k8s.io/yaml" + + dbv1alpha1 "github.com/oteldb/operator/api/v1alpha1" +) + +// queryCluster is ingestCluster with a query pool too, so every role renders from the same +// non-default ring shape. +func queryCluster() *dbv1alpha1.OtelDBCluster { + cr := ingestCluster() + cr.Spec.Query = &dbv1alpha1.QuerySpec{Replicas: ptr.To[int32](3)} + return cr +} + +func renderedQuery(t *testing.T, cr *dbv1alpha1.OtelDBCluster) map[string]any { + t.Helper() + + out, err := renderQueryConfig(cr, cr.Spec.Etcd.Endpoints) + require.NoError(t, err) + + var cfg map[string]any + require.NoError(t, yaml.Unmarshal([]byte(out), &cfg)) + return cfg +} + +func TestQueryDisabledByDefault(t *testing.T) { + cr := testCluster() + require.False(t, queryEnabled(cr), "an unchanged cluster must stay symmetric") + require.Zero(t, queryReplicasOf(cr)) +} + +func TestRenderQueryConfig(t *testing.T) { + cfg := renderedQuery(t, queryCluster()) + + cluster, ok := cfg["cluster"].(map[string]any) + require.True(t, ok, "cluster block missing") + require.Equal(t, []any{"http://etcd:2379"}, cluster["etcd"]) + require.EqualValues(t, 2, cluster["rf"]) + require.EqualValues(t, 8, cluster["shards_per_tenant"]) + require.Equal(t, "/tenants/a", cluster["root"]) + // odbselect is not a ring member, so it has no peer port to advertise. + require.NotContains(t, cluster, "port") + + // Each query API is its own listener, unlike odbingest's single shared one. + for key, want := range map[string]string{ + "prometheus": "0.0.0.0:9090", + "loki": "0.0.0.0:3100", + "tempo": "0.0.0.0:3200", + "pyroscope": "0.0.0.0:4040", + "health": "0.0.0.0:13133", + } { + block, ok := cfg[key].(map[string]any) + require.True(t, ok, "%s block missing", key) + require.Equal(t, want, block["bind"]) + } +} + +// A disabled signal must be switched off explicitly: odbselect defaults every API block it does not +// find, so an omitted block would be served on its default port. +func TestRenderQueryConfigDisabledSignal(t *testing.T) { + cr := queryCluster() + cr.Spec.Signals.Logs = ptr.To(false) + + cfg := renderedQuery(t, cr) + require.Equal(t, bindDisabled, cfg["loki"].(map[string]any)["bind"]) + require.Equal(t, "0.0.0.0:9090", cfg["prometheus"].(map[string]any)["bind"]) +} + +func TestRenderQueryConfigRejectsEverySignalDisabled(t *testing.T) { + cr := queryCluster() + cr.Spec.Signals = dbv1alpha1.SignalsSpec{ + Metrics: ptr.To(false), Logs: ptr.To(false), Traces: ptr.To(false), Profiles: ptr.To(false), + } + + _, err := renderQueryConfig(cr, cr.Spec.Etcd.Endpoints) + require.Error(t, err, "odbselect would serve nothing") +} + +// A ring parameter that differs between the roles does not fail: a role resolves a different owner +// set than the storage nodes do, so writes land where no read will look and reads look where the +// data is not. All three renderers must therefore agree key for key. +func TestRoleRingsMatch(t *testing.T) { + cr := queryCluster() + + storageOut, err := renderConfig(cr, cr.Spec.Etcd.Endpoints) + require.NoError(t, err) + var storageCfg map[string]any + require.NoError(t, yaml.Unmarshal([]byte(storageOut), &storageCfg)) + + storageRing := storageCfg["storage"].(map[string]any)["cluster"].(map[string]any) + rings := map[string]map[string]any{ + "ingest": renderedIngest(t, cr)["cluster"].(map[string]any), + "query": renderedQuery(t, cr)["cluster"].(map[string]any), + } + + for role, ring := range rings { + for _, key := range []string{"etcd", "rf", "shards_per_tenant", "root"} { + require.Equal(t, storageRing[key], ring[key], + "ring parameter %q differs between storage and %s", key, role) + } + } +} + +func TestRenderQueryConfigExtraConfig(t *testing.T) { + cr := queryCluster() + cr.Spec.Query.ExtraConfig = &runtime.RawExtension{ + Raw: []byte(`{"prometheus":{"max_samples":1000},"shutdown_timeout":"1m"}`), + } + + cfg := renderedQuery(t, cr) + prom := cfg["prometheus"].(map[string]any) + require.EqualValues(t, 1000, prom["max_samples"]) + require.Equal(t, "0.0.0.0:9090", prom["bind"], "extraConfig must merge into the block, not replace it") + require.Equal(t, "1m", cfg["shutdown_timeout"]) +} + +func TestRenderQueryConfigReservedPaths(t *testing.T) { + for name, raw := range map[string]string{ + "cluster": `{"cluster":{"rf":9}}`, + "prometheus bind": `{"prometheus":{"bind":"0.0.0.0:9999"}}`, + "health bind": `{"health":{"bind":"0.0.0.0:9999"}}`, + } { + t.Run(name, func(t *testing.T) { + cr := queryCluster() + cr.Spec.Query.ExtraConfig = &runtime.RawExtension{Raw: []byte(raw)} + + _, err := renderQueryConfig(cr, cr.Spec.Etcd.Endpoints) + require.ErrorContains(t, err, "spec.query.extraConfig") + + var invalid validationError + require.True(t, errors.As(err, &invalid), "reserved paths must fail as a spec validation error") + }) + } +} + +func TestBuildQueryDeployment(t *testing.T) { + cr := queryCluster() + deploy := buildQueryDeployment(cr, "cafebabe") + + require.Equal(t, "obs-query", deploy.Name) + require.EqualValues(t, 3, *deploy.Spec.Replicas) + require.Equal(t, "cafebabe", deploy.Spec.Template.Annotations["oteldb.io/config-hash"]) + + container := deploy.Spec.Template.Spec.Containers[0] + require.Equal(t, []string{queryBinPath}, container.Command) + require.Equal(t, []string{"--config=/etc/otel/odbselect.yml"}, container.Args) + require.Equal(t, defaultImage, container.Image, "the query pool defaults to the cluster image") + require.Empty(t, container.VolumeMounts[1:], "odbselect holds no data, so it mounts only its config") + + for _, e := range container.Env { + require.NotContains(t, e.Name, "OTELDB_CLUSTER_", + "odbselect joins nothing and must carry no ring identity") + } + + // The probes target the health listener, not a query API: it is the one listener odbselect + // always serves. + require.Equal(t, "/healthz", container.LivenessProbe.HTTPGet.Path) + require.Equal(t, "/readyz", container.ReadinessProbe.HTTPGet.Path) + require.Equal(t, portNameHealth, container.ReadinessProbe.HTTPGet.Port.StrVal) + + byName := map[string]int32{} + for _, p := range container.Ports { + byName[p.Name] = p.ContainerPort + } + require.EqualValues(t, portHealth, byName[portNameHealth]) + require.EqualValues(t, portPromHTTP, byName[portNamePromHTTP]) + require.NotContains(t, byName, portNameOTLPGRPC, "odbselect does not ingest") +} + +func TestBuildQueryDeploymentImageOverride(t *testing.T) { + cr := queryCluster() + cr.Spec.Query.Image = "ghcr.io/oteldb/oteldb:with-odbselect" + + deploy := buildQueryDeployment(cr, "x") + require.Equal(t, "ghcr.io/oteldb/oteldb:with-odbselect", deploy.Spec.Template.Spec.Containers[0].Image) +} + +// The storage StatefulSet's selector is immutable, so the roles must be told apart by the app name +// alone. No selector may match another role's pods. +func TestQuerySelectorIsDisjointFromOtherRoles(t *testing.T) { + cr := queryCluster() + + queryPods := labels.Set(mergeLabels(queryCommonLabels(cr), cr.Spec.Query.PodLabels)) + querySelector := labels.SelectorFromSet(querySelectorLabels(cr)) + + require.True(t, querySelector.Matches(queryPods)) + require.False(t, querySelector.Matches(labels.Set(commonLabels(cr))), + "the query Service must not front storage pods") + require.False(t, querySelector.Matches(labels.Set(ingestCommonLabels(cr))), + "the query Service must not front ingest pods") + require.False(t, labels.SelectorFromSet(selectorLabels(cr)).Matches(queryPods), + "the client Service must not front query pods") + require.False(t, labels.SelectorFromSet(ingestSelectorLabels(cr)).Matches(queryPods), + "the ingest Service must not front query pods") +} + +// One Service carries every enabled API, each on its own port: odbselect serves four listeners, not +// one shared mux like odbingest. +func TestBuildQueryServicePorts(t *testing.T) { + svc := buildQueryService(queryCluster()) + + require.Equal(t, "obs-query", svc.Name) + require.Equal(t, corev1.ServiceTypeClusterIP, svc.Spec.Type) + + byName := map[string]corev1.ServicePort{} + for _, p := range svc.Spec.Ports { + byName[p.Name] = p + } + + for name, port := range map[string]int32{ + portNamePromHTTP: portPromHTTP, + portNameLokiHTTP: portLokiHTTP, + portNameTempoHTTP: portTempoHTTP, + portNamePyroscope: portPyroscope, + portNameSelfMetric: portSelfMetric, + } { + require.EqualValues(t, port, byName[name].Port, "port %s", name) + require.Equal(t, name, byName[name].TargetPort.StrVal, "port %s", name) + } + require.NotContains(t, byName, portNameHealth, "the health listener is probed, not published") + require.Len(t, svc.Spec.Ports, 5) +} + +func TestBuildQueryServiceDropsDisabledSignals(t *testing.T) { + cr := queryCluster() + cr.Spec.Signals.Traces = ptr.To(false) + + svc := buildQueryService(cr) + for _, p := range svc.Spec.Ports { + require.NotEqual(t, portNameTempoHTTP, p.Name) + } +} + +// The existing client Service keeps fronting the storage nodes: its selector is immutable, and +// repointing an in-use query endpoint at a new workload is a data-path change. +func TestClientServiceUnchangedByQueryPool(t *testing.T) { + cr := queryCluster() + without := testCluster() + + require.Equal(t, buildClientService(without).Spec.Selector, buildClientService(cr).Spec.Selector) +} diff --git a/internal/controller/resources.go b/internal/controller/resources.go index 90a6367..332da9b 100644 --- a/internal/controller/resources.go +++ b/internal/controller/resources.go @@ -144,7 +144,7 @@ func buildStatefulSet(cr *dbv1alpha1.OtelDBCluster, configHash string) *appsv1.S replicas := replicasOf(cr) podLabels := mergeLabels(commonLabels(cr), cr.Spec.PodLabels) - annotations := mergeLabels(map[string]string{"oteldb.io/config-hash": configHash}, cr.Spec.PodAnnotations) + annotations := mergeLabels(map[string]string{annConfigHash: configHash}, cr.Spec.PodAnnotations) container := corev1.Container{ Name: appName, @@ -212,8 +212,8 @@ func podEnv(cr *dbv1alpha1.OtelDBCluster) []corev1.EnvVar { n := namesFor(cr) fqdnSuffix := fmt.Sprintf(".%s.%s.svc.cluster.local:%d", n.peerService(), cr.Namespace, peerPortOf(cr)) env := []corev1.EnvVar{ - {Name: "OTEL_EXPORTER_PROMETHEUS_HOST", Value: "0.0.0.0"}, - {Name: "OTEL_EXPORTER_PROMETHEUS_PORT", Value: fmt.Sprintf("%d", portSelfMetric)}, + {Name: envPrometheusHost, Value: bindAllHost}, + {Name: envPrometheusPort, Value: fmt.Sprintf("%d", portSelfMetric)}, {Name: "POD_NAME", ValueFrom: &corev1.EnvVarSource{ FieldRef: &corev1.ObjectFieldSelector{FieldPath: "metadata.name"}, }}, @@ -221,7 +221,7 @@ func podEnv(cr *dbv1alpha1.OtelDBCluster) []corev1.EnvVar { {Name: "OTELDB_CLUSTER_ADDR", Value: "$(POD_NAME)" + fqdnSuffix}, } if lvl := cr.Spec.LogLevel; lvl != "" { - env = append(env, corev1.EnvVar{Name: "OTEL_LOG_LEVEL", Value: lvl}) + env = append(env, corev1.EnvVar{Name: envLogLevel, Value: lvl}) } if z := cr.Spec.Cluster.StaticZone; z != "" { env = append(env, corev1.EnvVar{Name: "OTELDB_CLUSTER_ZONE", Value: z}) @@ -265,10 +265,12 @@ func containerPorts(cr *dbv1alpha1.OtelDBCluster) []corev1.ContainerPort { return ports } -func httpProbe(path string) *corev1.Probe { +// roleProbe is the HTTP probe shape every role shares: which port carries the health endpoints is +// the only thing that differs between them. +func roleProbe(path, portName string) *corev1.Probe { return &corev1.Probe{ ProbeHandler: corev1.ProbeHandler{ - HTTPGet: &corev1.HTTPGetAction{Path: path, Port: intstr.FromString(portNameHealth)}, + HTTPGet: &corev1.HTTPGetAction{Path: path, Port: intstr.FromString(portName)}, }, PeriodSeconds: 10, TimeoutSeconds: 3, @@ -276,6 +278,25 @@ func httpProbe(path string) *corev1.Probe { } } +func httpProbe(path string) *corev1.Probe { return roleProbe(path, portNameHealth) } + +// bindAll is the listen address for port: reachability is the Service's job. +func bindAll(port int32) string { return fmt.Sprintf("%s:%d", bindAllHost, port) } + +// statelessPodEnv is the environment of a pod in either stateless pool. Unlike a storage pod it +// carries no ring identity: neither odbingest nor odbselect joins the ring, so neither has an id, +// address or zone to advertise. +func statelessPodEnv(cr *dbv1alpha1.OtelDBCluster) []corev1.EnvVar { + env := []corev1.EnvVar{ + {Name: envPrometheusHost, Value: bindAllHost}, + {Name: envPrometheusPort, Value: fmt.Sprintf("%d", portSelfMetric)}, + } + if lvl := cr.Spec.LogLevel; lvl != "" { + env = append(env, corev1.EnvVar{Name: envLogLevel, Value: lvl}) + } + return env +} + func startupProbe(path string) *corev1.Probe { p := httpProbe(path) p.FailureThreshold = 30 // allow up to ~5m for recovery/replay on startup