From ed47cd51d37c747d00b26943f88083209423d25b Mon Sep 17 00:00:00 2001 From: tdakkota Date: Tue, 18 Aug 2026 00:44:21 +0300 Subject: [PATCH 1/2] feat(api): add an optional stateless odbselect query node group MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit spec.query is the read half of spec.ingest: an optional *QuerySpec that deploys a Deployment of stateless odbselect pods reading through the ring. Absent means no pool and byte-identical rendering to today; the storage StatefulSet's selector and the client Service are untouched. Unlike odbingest's single shared listener, odbselect serves four query APIs on four listeners plus health on 13133. One -query Service publishes them all — one pod set, one selector, a port each — and the probes hit the health listener rather than a data port. spec.signals decides which APIs are served: a disabled signal renders odbselect's bind: "-", which is required because odbselect defaults any API block it does not find. The ring comes from the shared ringConfig() helper with the cluster block reserved from extraConfig, and the listener binds are reserved too: odbselect keeps serving its other listeners if one moves, so a rebind leaves a published Service port pointing at nothing while the pod stays Ready. TestRoleRingsMatch now pins that all three roles render the same ring key for key. odbselect is not in any released image yet; its packaging ships with oteldb/oteldb#1266. --- README.md | 84 ++++-- api/v1alpha1/oteldbcluster_types.go | 116 +++++++- config/samples/db_v1alpha1_oteldbcluster.yaml | 33 ++- internal/controller/extraconfig.go | 2 +- internal/controller/ingest.go | 34 +-- internal/controller/ingest_test.go | 19 +- internal/controller/naming.go | 45 +++ .../controller/oteldbcluster_controller.go | 88 +++++- internal/controller/query.go | 271 ++++++++++++++++++ internal/controller/query_test.go | 267 +++++++++++++++++ internal/controller/resources.go | 33 ++- 11 files changed, 893 insertions(+), 99 deletions(-) create mode 100644 internal/controller/query.go create mode 100644 internal/controller/query_test.go diff --git a/README.md b/README.md index fb90b58..e1ba01d 100644 --- a/README.md +++ b/README.md @@ -24,27 +24,47 @@ the query/ingest APIs. ### Optional role node groups -Symmetric nodes couple ingest capacity to storage capacity: the only way to absorb a write spike is -to add a node, and that node immediately takes ownership of a share of the shards. `spec.ingest` -breaks the coupling by deploying a pool of stateless **`odbingest`** write nodes as a separate -Deployment. It is opt-in — omit the stanza and the cluster stays symmetric, exactly as before. - -Unlike VictoriaMetrics' `vminsert`, an `odbingest` pod is **not** handed a list of storage nodes. It -watches etcd for membership and computes shard owners locally from the rendezvous-hash ring, so -`spec.replicas` and `spec.ingest.replicas` are genuinely independent: scaling storage triggers no -config re-render and no restart of the ingest pool. - -What the pool *does* need is the ring's shape — `replicationFactor`, `shardsPerTenant`, `etcdPrefix` +Symmetric nodes couple ingest and query capacity to storage capacity: the only way to absorb a write +spike or a dashboard reload is to add a node, and that node immediately takes ownership of a share +of the shards. `spec.ingest` and `spec.query` break the coupling by deploying pools of stateless +**`odbingest`** write nodes and **`odbselect`** query nodes as separate Deployments. Both are opt-in +— omit the stanzas and the cluster stays symmetric, exactly as before. + +Unlike VictoriaMetrics' `vminsert`/`vmselect`, a stateless oteldb pod is **not** handed a list of +storage nodes. It watches etcd for membership and computes shard owners locally from the +rendezvous-hash ring, so `spec.replicas`, `spec.ingest.replicas` and `spec.query.replicas` are +genuinely independent: scaling storage triggers no config re-render and no restart of either pool. + +What a pool *does* need is the ring's shape — `replicationFactor`, `shardsPerTenant`, `etcdPrefix` — and a mismatch there **fails silently**: the pool resolves a different owner set than the nodes -do, and writes land where no read will look for them. The operator renders both roles from the same -`spec.cluster`, so the mismatch cannot be expressed. - -> **`odbingest` is not in the released oteldb image yet.** It is absent from oteldb's goreleaser -> builds and `release.Dockerfile`, so no published `ghcr.io/oteldb/oteldb` tag contains the binary. -> Set `spec.ingest.image` to a build that ships it until that is fixed upstream. - -A stateless `odbselect` query pool (`spec.query`) is the matching read half; it is not implemented -yet, pending [oteldb/oteldb#1266](https://github.com/oteldb/oteldb/pull/1266). +do, so writes land where no read will look for them and reads look where the data is not. The +operator renders all three roles from the same `spec.cluster`, so the mismatch cannot be expressed. + +> **Neither binary is in the released oteldb image yet.** `odbingest` is absent from oteldb's +> goreleaser builds and `release.Dockerfile`, and `odbselect`'s release packaging ships with +> [oteldb/oteldb#1266](https://github.com/oteldb/oteldb/pull/1266). No published +> `ghcr.io/oteldb/oteldb` tag contains either one. Set `spec.ingest.image` / `spec.query.image` to a +> build that ships them until that is fixed upstream. + +#### The query pool's shape + +`odbingest` serves OTLP/HTTP, remote write and its health endpoints on one listener. `odbselect` +does not: it serves **four query APIs on four listeners** (`9090` PromQL, `3100` LogQL, `3200` +TraceQL, `4040` Pyroscope) plus health on `13133`. Three consequences: + +- **One Service, several ports.** `-query` publishes every enabled API. The four APIs are one + pod set behind one selector and each already has its own port, so a Service per API would add DNS + names — and, at `type: LoadBalancer`, four load balancers — without making anything addressable + that is not addressable now. +- **`spec.signals` turns an API off, not a per-API knob.** The four APIs map 1:1 onto the four + signals the cluster already toggles, and serving an API for a signal the cluster does not store + can only answer empty. A disabled signal renders `bind: "-"` (odbselect's disable convention) and + drops the port from the pool and its Service. Note the `"-"` is *required*: odbselect defaults + every API block it does not find, so simply omitting one would serve it anyway. +- **Probes hit the health listener**, not a data port — it is the one listener that is served + whatever `spec.signals` leaves enabled. `/readyz` answers 503 until the ring has a member, so a + starting pod stays out of the load balancer while a query could only return an empty result that + looks like an answer. ### etcd is bring-your-own @@ -86,6 +106,7 @@ for a fuller example including the S3 backend. |---|---| | `replicas` | Number of oteldb storage nodes (StatefulSet size). Use `>= cluster.replicationFactor`. | | `ingest` | Optional stateless `odbingest` write pool (Deployment). Absent ⇒ symmetric nodes handle ingest. Takes `replicas` (default 2), `image`, `service`, `extraConfig` and the standard scheduling/security knobs. | +| `query` | Optional stateless `odbselect` query pool (Deployment). Absent ⇒ symmetric nodes answer queries. Same knobs as `ingest`. Which APIs it serves follows `spec.signals`. | | `image` / `imagePullPolicy` / `imagePullSecrets` | oteldb container image (default `ghcr.io/oteldb/oteldb:v0.46.0`). | | `etcd.endpoints` | **Required.** External etcd endpoint list. | | `storage.backend` | `file` (default, per-node PVC) or `s3` (shared object store). | @@ -144,13 +165,21 @@ spec field to use instead. `spec.ingest.extraConfig` is merged over the generated `odbingest.yml` under the same rules, with the whole `cluster` block reserved for `spec.cluster` and `spec.etcd.endpoints`. +`spec.query.extraConfig` is merged over the generated `odbselect.yml`, reserving `cluster` and the +five listener addresses (`prometheus.bind`, `loki.bind`, `tempo.bind`, `pyroscope.bind`, +`health.bind`). The binds are reserved because odbselect keeps serving its other listeners if one +moves: the Service port the operator published would point at nothing while the pod stayed `Ready`. +Everything else in a block is free — `prometheus.max_samples`, `loki.max_sample_rows`, per-listener +`auth`, `shutdown_timeout`. Use `spec.signals` to turn an API off. + ### Status `status` reports `phase` (`Pending`/`Progressing`/`Ready`/`Degraded`), `replicas`, -`readyReplicas`, `ingestReplicas`, `ingestReadyReplicas`, the resolved `etcdEndpoints`, and standard -`Available`/`Progressing`/`Degraded` conditions. `kubectl get oteldbcluster` prints replicas, ready -count, and phase. The phase tracks the storage nodes: they hold the data, and an ingest pool still -rolling out does not make the cluster unavailable. +`readyReplicas`, `ingestReplicas`, `ingestReadyReplicas`, `queryReplicas`, `queryReadyReplicas`, the +resolved `etcdEndpoints`, and standard `Available`/`Progressing`/`Degraded` conditions. +`kubectl get oteldbcluster` prints replicas, ready count, and phase. The phase tracks the storage +nodes: they hold the data, and a stateless pool still rolling out does not make the cluster +unavailable. ## How per-pod identity works @@ -173,6 +202,11 @@ remote-write and `8090` self-metrics. `odbingest` serves OTLP/HTTP, remote write endpoints on **one** listener (`19291`), so the Service publishes `4318` — the port stock OTLP/HTTP exporters target — and remaps it onto that listener. +The query Service (`-query`) exposes `9090` PromQL, `3200` TraceQL, `3100` LogQL, `4040` +Pyroscope and `8090` self-metrics, dropping any whose signal is disabled. `odbselect` serves each on +its own listener, so these are published one-to-one; its health listener (`13133`) is probed, not +published. + ## Getting Started Prerequisites: Go 1.26+, a Kubernetes cluster, `kubectl`, and (for building the image) Docker. @@ -209,4 +243,6 @@ Controller layout under `internal/controller/`: - `oteldbcluster_controller.go` — reconcile loop, apply/owner-ref helper, status. - `config.go` — renders the oteldb `config.yml` from the spec. - `resources.go` — builds the ConfigMap, headless + client Services, and StatefulSet. +- `ingest.go` / `query.go` — the stateless `odbingest` and `odbselect` pools: config, Service, + Deployment. - `naming.go` — names, labels, ports. diff --git a/api/v1alpha1/oteldbcluster_types.go b/api/v1alpha1/oteldbcluster_types.go index b9dbafa..474b7b4 100644 --- a/api/v1alpha1/oteldbcluster_types.go +++ b/api/v1alpha1/oteldbcluster_types.go @@ -31,9 +31,10 @@ type OtelDBClusterSpec struct { // StatefulSet pod that ingests, queries, stores, and replicates. For real redundancy this // should be at least the replication factor (see Cluster.ReplicationFactor). // - // This is the storage dimension only. Ingest capacity scales separately via spec.ingest, and - // scaling either has no effect on the other: membership lives in etcd and the ring is computed - // from it, so an ingest pod needs no knowledge of how many storage nodes there are. + // This is the storage dimension only. Ingest and query capacity scale separately via + // spec.ingest and spec.query, and scaling any of them has no effect on the others: membership + // lives in etcd and the ring is computed from it, so a stateless pod needs no knowledge of how + // many storage nodes there are. // +kubebuilder:default=3 // +kubebuilder:validation:Minimum=1 // +optional @@ -49,6 +50,15 @@ type OtelDBClusterSpec struct { // +optional Ingest *IngestSpec `json:"ingest,omitempty"` + // Query optionally deploys a pool of stateless odbselect query nodes, so query capacity scales + // independently of storage. Absent (the default) means no query pool: the storage nodes are + // symmetric and answer queries themselves. + // + // The pool holds no data and is not a ring member. It follows etcd membership read-only and + // fans each read out to the shard's owners, so scaling spec.replicas needs no change to it. + // +optional + Query *QuerySpec `json:"query,omitempty"` + // Image is the oteldb container image. Defaults to the operator's pinned image when empty. // +optional Image string `json:"image,omitempty"` @@ -251,6 +261,97 @@ type IngestSpec struct { ExtraConfig *runtime.RawExtension `json:"extraConfig,omitempty"` } +// QuerySpec configures the stateless odbselect query pool: a Deployment of nodes that serve the +// PromQL, LogQL, TraceQL and Pyroscope APIs by reading through the ring instead of local storage. +// +// The pool's ring parameters (etcd endpoints, replication factor, shards per tenant, key prefix) +// are rendered from spec.etcd and spec.cluster, the same source the storage nodes render from. A +// mismatch there does not fail — it resolves a different owner set than the nodes do, so reads look +// where the data is not — so it is deliberately not configurable per pool. +// +// Which APIs the pool serves follows spec.signals, the same toggle that decides what the storage +// nodes serve and store: a disabled signal's API is switched off on the pool and its port is +// dropped from the query Service. There is no separate per-API toggle, because serving an API for +// a signal the cluster does not store can only answer empty. +// +// The odbselect binary is not yet shipped in the released oteldb image (it lands with +// oteldb/oteldb#1266). Until it is, set Image to a build that contains it. +type QuerySpec struct { + // Replicas is the number of odbselect pods. They hold no data, so this is purely a throughput + // and availability knob and can be changed freely. + // +kubebuilder:default=2 + // +kubebuilder:validation:Minimum=0 + // +optional + Replicas *int32 `json:"replicas,omitempty"` + + // Image is the container image running odbselect. Defaults to the cluster image + // (spec.image, else the operator's pinned image). + // +optional + Image string `json:"image,omitempty"` + + // Resources are the compute resources for each odbselect container. + // +optional + Resources corev1.ResourceRequirements `json:"resources,omitempty"` + + // Service configures the client-facing Service that exposes the query APIs. One Service carries + // every enabled API, each on its own port; they share a pod set, so splitting them across four + // Services would only add DNS names. + // +optional + Service ServiceSpec `json:"service,omitempty"` + + // PodAnnotations are added to every odbselect pod. + // +optional + PodAnnotations map[string]string `json:"podAnnotations,omitempty"` + + // PodLabels are added to every odbselect pod. + // +optional + PodLabels map[string]string `json:"podLabels,omitempty"` + + // ServiceAccountName is the ServiceAccount for the odbselect pods. Empty falls back to + // spec.serviceAccountName. + // +optional + ServiceAccountName string `json:"serviceAccountName,omitempty"` + + // NodeSelector constrains odbselect pods to nodes with matching labels. + // +optional + NodeSelector map[string]string `json:"nodeSelector,omitempty"` + + // Affinity for odbselect pods. When empty, the operator applies a soft anti-affinity that + // spreads the pool across nodes. + // +optional + Affinity *corev1.Affinity `json:"affinity,omitempty"` + + // Tolerations for odbselect pods. + // +optional + Tolerations []corev1.Toleration `json:"tolerations,omitempty"` + + // TopologySpreadConstraints for odbselect pods. + // +optional + TopologySpreadConstraints []corev1.TopologySpreadConstraint `json:"topologySpreadConstraints,omitempty"` + + // PodSecurityContext for odbselect pods. + // +optional + PodSecurityContext *corev1.PodSecurityContext `json:"podSecurityContext,omitempty"` + + // SecurityContext for the odbselect container. + // +optional + SecurityContext *corev1.SecurityContext `json:"securityContext,omitempty"` + + // ExtraConfig is arbitrary additional odbselect config deeply merged over the generated config, + // as a top-level YAML/JSON object. Use it to set fields the CRD does not model, such as + // prometheus.max_samples, loki.max_sample_rows, per-listener auth or shutdown_timeout. + // + // The cluster block is reserved and rejected instead of merged: it must stay in step with the + // storage nodes. Configure it through spec.etcd and spec.cluster. The listener addresses + // (prometheus.bind, loki.bind, tempo.bind, pyroscope.bind, health.bind) are reserved too: the + // operator publishes them as Service ports and probes them, and odbselect keeps serving its + // other listeners if one moves, so a rebind would fail silently rather than crash the pod. Use + // spec.signals to turn an API off. + // +optional + // +kubebuilder:pruning:PreserveUnknownFields + ExtraConfig *runtime.RawExtension `json:"extraConfig,omitempty"` +} + // StorageBackend selects the embedded storage engine's durable backend. // +kubebuilder:validation:Enum=file;s3 type StorageBackend string @@ -622,6 +723,15 @@ type OtelDBClusterStatus struct { // +optional IngestReadyReplicas int32 `json:"ingestReadyReplicas,omitempty"` + // QueryReplicas is the desired number of odbselect nodes. Zero when no query pool is + // configured. + // +optional + QueryReplicas int32 `json:"queryReplicas,omitempty"` + + // QueryReadyReplicas is the number of ready odbselect nodes. + // +optional + QueryReadyReplicas int32 `json:"queryReadyReplicas,omitempty"` + // EtcdEndpoints is the resolved etcd endpoint list the cluster is using. // +optional EtcdEndpoints []string `json:"etcdEndpoints,omitempty"` diff --git a/config/samples/db_v1alpha1_oteldbcluster.yaml b/config/samples/db_v1alpha1_oteldbcluster.yaml index 904e8ad..0907a18 100644 --- a/config/samples/db_v1alpha1_oteldbcluster.yaml +++ b/config/samples/db_v1alpha1_oteldbcluster.yaml @@ -111,15 +111,17 @@ spec: cluster: replicationFactor: 2 --- -# Example splitting ingest off the storage nodes: three storage nodes plus a stateless odbingest -# pool that scales on its own. The pool holds no data and is not a ring member — it watches etcd -# for membership and routes each shard to its primary, so scaling `replicas` needs no change here. +# Example splitting ingest and query off the storage nodes: three storage nodes plus a stateless +# odbingest pool and a stateless odbselect pool, each scaling on its own. Neither pool holds data or +# is a ring member — both watch etcd for membership and resolve shard owners locally, so scaling +# `replicas` needs no change here. # -# NOTE: odbingest is not shipped in the released oteldb image yet, hence the explicit image below. +# NOTE: neither odbingest nor odbselect is shipped in the released oteldb image yet, hence the +# explicit images below. apiVersion: db.oteldb.io/v1alpha1 kind: OtelDBCluster metadata: - name: oteldbcluster-ingest-sample + name: oteldbcluster-roles-sample spec: replicas: 3 etcd: @@ -144,3 +146,24 @@ spec: time_threshold: 1h service: type: ClusterIP + + # The read half. odbselect serves the four query APIs on four separate listeners, published as + # four ports on one `-query` Service; `spec.signals` decides which of them are served. + query: + replicas: 4 + image: "ghcr.io/oteldb/oteldb:with-odbselect" + resources: + requests: + cpu: "2" + memory: 2Gi + limits: + memory: 4Gi + # `cluster` and every `*.bind` are reserved: the binds are what the Service publishes and the + # probes hit. Everything else in a block is free. + extraConfig: + prometheus: + max_samples: 20000000 + loki: + max_sample_rows: 500000 + service: + type: ClusterIP diff --git a/internal/controller/extraconfig.go b/internal/controller/extraconfig.go index 86d3537..19eae57 100644 --- a/internal/controller/extraconfig.go +++ b/internal/controller/extraconfig.go @@ -40,7 +40,7 @@ var reservedConfigPaths = map[string]string{ "storage.dir": "use spec.storage.dir", "storage.wal_dir": "use spec.storage.dir", "storage.s3": "use spec.storage.s3", - "storage.cluster": "use spec.cluster and spec.etcd.endpoints", + "storage.cluster": hintRingFromSpec, "storage.flush_interval": "use spec.engine.flushInterval", "storage.read_cache_bytes": "use spec.engine.readCacheSize", "storage.decode_cache_bytes": "use spec.engine.decodeCacheSize", diff --git a/internal/controller/ingest.go b/internal/controller/ingest.go index 8d6628b..e7cccc8 100644 --- a/internal/controller/ingest.go +++ b/internal/controller/ingest.go @@ -34,7 +34,7 @@ import ( // must stay in step with the storage nodes, and a mismatch is silent: it resolves a different owner // set, so writes land where no read will look for them. var ingestReservedConfigPaths = map[string]string{ - keyCluster: "use spec.cluster and spec.etcd.endpoints", + keyCluster: hintRingFromSpec, } func ingestEnabled(cr *dbv1alpha1.OtelDBCluster) bool { return cr.Spec.Ingest != nil } @@ -55,10 +55,10 @@ func renderIngestConfig(cr *dbv1alpha1.OtelDBCluster, etcdEndpoints []string) (s cfg := map[string]any{ keyCluster: ringConfig(cr, etcdEndpoints), "prometheus_remote_write": map[string]any{ - keyBind: fmt.Sprintf("0.0.0.0:%d", portPromRW), + keyBind: bindAll(portPromRW), }, "otlp": map[string]any{ - "grpc_bind": fmt.Sprintf("0.0.0.0:%d", portOTLPGRPC), + "grpc_bind": bindAll(portOTLPGRPC), }, } @@ -170,7 +170,7 @@ func buildIngestDeployment(cr *dbv1alpha1.OtelDBCluster, configHash string) *app } podLabels := mergeLabels(ingestCommonLabels(cr), spec.PodLabels) - annotations := mergeLabels(map[string]string{"oteldb.io/config-hash": configHash}, spec.PodAnnotations) + annotations := mergeLabels(map[string]string{annConfigHash: configHash}, spec.PodAnnotations) container := corev1.Container{ Name: ingestAppName, @@ -178,7 +178,7 @@ func buildIngestDeployment(cr *dbv1alpha1.OtelDBCluster, configHash string) *app ImagePullPolicy: cr.Spec.ImagePullPolicy, Command: []string{ingestBinPath}, Args: []string{"--config=" + configMountPath + "/" + ingestConfigFileName}, - Env: ingestPodEnv(cr), + Env: statelessPodEnv(cr), Ports: []corev1.ContainerPort{ {Name: portNameOTLPGRPC, ContainerPort: portOTLPGRPC, Protocol: corev1.ProtocolTCP}, {Name: portNameIngestHTTP, ContainerPort: portPromRW, Protocol: corev1.ProtocolTCP}, @@ -232,26 +232,4 @@ func buildIngestDeployment(cr *dbv1alpha1.OtelDBCluster, configHash string) *app } } -// ingestPodEnv is the odbingest pod environment. Unlike a storage pod it carries no ring identity: -// odbingest joins nothing, so it has no id, address or zone to advertise. -func ingestPodEnv(cr *dbv1alpha1.OtelDBCluster) []corev1.EnvVar { - env := []corev1.EnvVar{ - {Name: "OTEL_EXPORTER_PROMETHEUS_HOST", Value: "0.0.0.0"}, - {Name: "OTEL_EXPORTER_PROMETHEUS_PORT", Value: fmt.Sprintf("%d", portSelfMetric)}, - } - if lvl := cr.Spec.LogLevel; lvl != "" { - env = append(env, corev1.EnvVar{Name: "OTEL_LOG_LEVEL", Value: lvl}) - } - return env -} - -func ingestProbe(path string) *corev1.Probe { - return &corev1.Probe{ - ProbeHandler: corev1.ProbeHandler{ - HTTPGet: &corev1.HTTPGetAction{Path: path, Port: intstr.FromString(portNameIngestHTTP)}, - }, - PeriodSeconds: 10, - TimeoutSeconds: 3, - FailureThreshold: 3, - } -} +func ingestProbe(path string) *corev1.Probe { return roleProbe(path, portNameIngestHTTP) } diff --git a/internal/controller/ingest_test.go b/internal/controller/ingest_test.go index 8cf185f..a5567d2 100644 --- a/internal/controller/ingest_test.go +++ b/internal/controller/ingest_test.go @@ -79,24 +79,7 @@ func TestRenderIngestConfig(t *testing.T) { require.Equal(t, "0.0.0.0:4317", otlp["grpc_bind"]) } -// A ring parameter that differs between the roles does not fail: the pool resolves a different -// owner set than the storage nodes do, and writes land where no read will look for them. Both -// renderers must therefore agree key for key. -func TestIngestRingMatchesStorageRing(t *testing.T) { - cr := ingestCluster() - - storageOut, err := renderConfig(cr, cr.Spec.Etcd.Endpoints) - require.NoError(t, err) - var storageCfg map[string]any - require.NoError(t, yaml.Unmarshal([]byte(storageOut), &storageCfg)) - - storageRing := storageCfg["storage"].(map[string]any)["cluster"].(map[string]any) - ingestRing := renderedIngest(t, cr)["cluster"].(map[string]any) - - for _, key := range []string{"etcd", "rf", "shards_per_tenant", "root"} { - require.Equal(t, storageRing[key], ingestRing[key], "ring parameter %q differs between roles", key) - } -} +// The ring every role must agree on is covered for all three roles by TestRoleRingsMatch. func TestRenderIngestConfigExtraConfig(t *testing.T) { cr := ingestCluster() diff --git a/internal/controller/naming.go b/internal/controller/naming.go index ba81726..bf48604 100644 --- a/internal/controller/naming.go +++ b/internal/controller/naming.go @@ -29,13 +29,33 @@ const defaultImage = "ghcr.io/oteldb/oteldb:v0.46.0" const ( appName = "oteldb" // app label value and container name ingestAppName = "odbingest" // ingest app label value, container name and binary + queryAppName = "odbselect" // query app label value, container name and binary valStorage = "storage" // oteldb signal-backend value and component label valIngest = "ingest" // ingest component label + valQuery = "query" // query component label defaultDataDir = "/var/lib/oteldb" // default storage.dir / WAL dir keyBind = "bind" // oteldb per-API bind config key + // bindDisabled is odbselect's "do not serve this API" bind. An omitted block is not enough: + // odbselect applies its defaults to every API block unconditionally, so a missing one is served + // on its default port (see cmd/odbselect/config.go, setDefaults and enabled). + bindDisabled = "-" envAWSAccessKeyID = "AWS_ACCESS_KEY_ID" envAWSSecretAccessKey = "AWS_SECRET_ACCESS_KEY" + envPrometheusHost = "OTEL_EXPORTER_PROMETHEUS_HOST" + envPrometheusPort = "OTEL_EXPORTER_PROMETHEUS_PORT" + envLogLevel = "OTEL_LOG_LEVEL" + + // annConfigHash carries the rendered config's digest on a pod template, so a config change rolls + // the workload. + annConfigHash = "oteldb.io/config-hash" + + // hintRingFromSpec is the remediation for a reserved ring path: every role renders it from the + // same two spec fields, which is what keeps them from resolving different owner sets. + hintRingFromSpec = "use spec.cluster and spec.etcd.endpoints" + + // bindAllHost is the address every listener binds, leaving reachability to the Service. + bindAllHost = "0.0.0.0" ) // oteldb config keys shared by the renderer and the reserved-path guard. @@ -52,6 +72,17 @@ const ( keyPort = "port" ) +// Query API config block keys, shared by cmd/oteldb and cmd/odbselect. +const ( + keyPrometheus = "prometheus" + keyTempo = "tempo" + keyLoki = "loki" + keyPyroscope = "pyroscope" + // keyHealth is odbselect's health listener block. cmd/oteldb spells the same thing + // "health_check". + keyHealth = "health" +) + // Container ports exposed by every oteldb node. Names must be <= 15 chars (k8s port-name limit). const ( portOTLPGRPC = 4317 @@ -91,9 +122,12 @@ const ( configMountPath = "/etc/otel" configFileName = "config.yml" ingestConfigFileName = "odbingest.yml" + queryConfigFileName = "odbselect.yml" // ingestBinPath is the odbingest entrypoint inside the oteldb image, which defaults to the // oteldb binary. ingestBinPath = "/usr/local/bin/odbingest" + // queryBinPath is the odbselect entrypoint inside the oteldb image. + queryBinPath = "/usr/local/bin/odbselect" ) // resourceNames centralizes the derived names for a cluster's child objects. @@ -110,6 +144,9 @@ func (n resourceNames) clientService() string { return n.base } func (n resourceNames) ingestDeployment() string { return n.base + "-ingest" } func (n resourceNames) ingestConfigMap() string { return n.base + "-ingest-config" } func (n resourceNames) ingestService() string { return n.base + "-ingest" } +func (n resourceNames) queryDeployment() string { return n.base + "-query" } +func (n resourceNames) queryConfigMap() string { return n.base + "-query-config" } +func (n resourceNames) queryService() string { return n.base + "-query" } // roleSelectorLabels are the immutable pod-selector labels for one role's pods. The role's binary // is the app name, which is what keeps the storage and ingest pod sets from selecting each other: @@ -148,6 +185,14 @@ func ingestCommonLabels(cr *dbv1alpha1.OtelDBCluster) map[string]string { return roleCommonLabels(cr, ingestAppName, valIngest) } +func querySelectorLabels(cr *dbv1alpha1.OtelDBCluster) map[string]string { + return roleSelectorLabels(cr, queryAppName) +} + +func queryCommonLabels(cr *dbv1alpha1.OtelDBCluster) map[string]string { + return roleCommonLabels(cr, queryAppName, valQuery) +} + // mergeLabels returns a new map combining base with extra (extra wins). func mergeLabels(base, extra map[string]string) map[string]string { out := make(map[string]string, len(base)+len(extra)) diff --git a/internal/controller/oteldbcluster_controller.go b/internal/controller/oteldbcluster_controller.go index f5f58c7..83356dd 100644 --- a/internal/controller/oteldbcluster_controller.go +++ b/internal/controller/oteldbcluster_controller.go @@ -104,7 +104,10 @@ func (r *OtelDBClusterReconciler) reconcile(ctx context.Context, cr *dbv1alpha1. if err := r.apply(ctx, cr, buildStatefulSet(cr, hash)); err != nil { return fmt.Errorf("statefulset: %w", err) } - return r.reconcileIngest(ctx, cr, endpoints) + if err := r.reconcileIngest(ctx, cr, endpoints); err != nil { + return err + } + return r.reconcileQuery(ctx, cr, endpoints) } // reconcileIngest applies the stateless odbingest pool, or removes it when spec.ingest is absent. @@ -148,6 +151,47 @@ func (r *OtelDBClusterReconciler) pruneIngest(ctx context.Context, cr *dbv1alpha return nil } +// reconcileQuery applies the stateless odbselect pool, or removes it when spec.query is absent. +func (r *OtelDBClusterReconciler) reconcileQuery(ctx context.Context, cr *dbv1alpha1.OtelDBCluster, endpoints []string) error { + if !queryEnabled(cr) { + return r.pruneQuery(ctx, cr) + } + + cm, err := buildQueryConfigMap(cr, endpoints) + if err != nil { + return err + } + if err := r.apply(ctx, cr, cm); err != nil { + return fmt.Errorf("query configmap: %w", err) + } + hash := configHash(cm.Data[queryConfigFileName]) + + if err := r.apply(ctx, cr, buildQueryService(cr)); err != nil { + return fmt.Errorf("query service: %w", err) + } + if err := r.apply(ctx, cr, buildQueryDeployment(cr, hash)); err != nil { + return fmt.Errorf("query deployment: %w", err) + } + return nil +} + +// pruneQuery deletes the query pool's objects. Owner references only collect them when the whole +// cluster goes away, so dropping spec.query has to remove them explicitly. +func (r *OtelDBClusterReconciler) pruneQuery(ctx context.Context, cr *dbv1alpha1.OtelDBCluster) error { + n := namesFor(cr) + objs := []client.Object{ + &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: n.queryDeployment(), Namespace: cr.Namespace}}, + &corev1.Service{ObjectMeta: metav1.ObjectMeta{Name: n.queryService(), Namespace: cr.Namespace}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: n.queryConfigMap(), Namespace: cr.Namespace}}, + } + for _, obj := range objs { + if err := r.Delete(ctx, obj); client.IgnoreNotFound(err) != nil { + return fmt.Errorf("delete query %T: %w", obj, err) + } + } + return nil +} + // apply creates or updates obj as a child of cr, preserving server-managed fields. func (r *OtelDBClusterReconciler) apply(ctx context.Context, cr *dbv1alpha1.OtelDBCluster, obj client.Object) error { // Snapshot the desired spec so we can restore it after CreateOrUpdate fetches the live object. @@ -213,27 +257,27 @@ func (r *OtelDBClusterReconciler) updateStatus(ctx context.Context, cr *dbv1alph return err } - ingestDesired := ingestReplicasOf(cr) - ingestReady := int32(0) - if ingestEnabled(cr) { - deploy := &appsv1.Deployment{} - key := client.ObjectKey{Namespace: cr.Namespace, Name: namesFor(cr).ingestDeployment()} - if err := r.Get(ctx, key, deploy); err == nil { - ingestReady = deploy.Status.ReadyReplicas - } else if !apierrors.IsNotFound(err) { - return err - } + n := namesFor(cr) + ingestReady, err := r.deploymentReady(ctx, cr, ingestEnabled(cr), n.ingestDeployment()) + if err != nil { + return err + } + queryReady, err := r.deploymentReady(ctx, cr, queryEnabled(cr), n.queryDeployment()) + if err != nil { + return err } cr.Status.Replicas = desired cr.Status.ReadyReplicas = ready - cr.Status.IngestReplicas = ingestDesired + cr.Status.IngestReplicas = ingestReplicasOf(cr) cr.Status.IngestReadyReplicas = ingestReady + cr.Status.QueryReplicas = queryReplicasOf(cr) + cr.Status.QueryReadyReplicas = queryReady cr.Status.EtcdEndpoints = cr.Spec.Etcd.Endpoints cr.Status.ObservedGeneration = cr.Generation - // The phase tracks the storage nodes: they are what holds the data, and an ingest pool that is - // still rolling out does not make the cluster unavailable. + // The phase tracks the storage nodes: they are what holds the data, and a stateless pool that + // is still rolling out does not make the cluster unavailable. switch { case ready == 0: cr.Status.Phase = dbv1alpha1.PhasePending @@ -252,6 +296,22 @@ func (r *OtelDBClusterReconciler) updateStatus(ctx context.Context, cr *dbv1alph return r.Status().Update(ctx, cr) } +// deploymentReady reads a stateless role's ready replica count, or zero when the role is disabled +// or its Deployment does not exist yet. +func (r *OtelDBClusterReconciler) deploymentReady(ctx context.Context, cr *dbv1alpha1.OtelDBCluster, enabled bool, name string) (int32, error) { + if !enabled { + return 0, nil + } + deploy := &appsv1.Deployment{} + if err := r.Get(ctx, client.ObjectKey{Namespace: cr.Namespace, Name: name}, deploy); err != nil { + if apierrors.IsNotFound(err) { + return 0, nil + } + return 0, err + } + return deploy.Status.ReadyReplicas, nil +} + func (r *OtelDBClusterReconciler) setDegraded(ctx context.Context, cr *dbv1alpha1.OtelDBCluster, reason string, cause error) { cr.Status.Phase = dbv1alpha1.PhaseDegraded setCondition(cr, dbv1alpha1.ConditionDegraded, metav1.ConditionTrue, reason, cause.Error()) diff --git a/internal/controller/query.go b/internal/controller/query.go new file mode 100644 index 0000000..575b4cc --- /dev/null +++ b/internal/controller/query.go @@ -0,0 +1,271 @@ +/* +Copyright 2026. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package controller + +import ( + "encoding/json" + "fmt" + + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/yaml" + + dbv1alpha1 "github.com/oteldb/operator/api/v1alpha1" +) + +// queryReservedConfigPaths are the odbselect config paths the operator owns. +// +// The ring parameters must stay in step with the storage nodes, and a mismatch is silent: it +// resolves a different owner set, so reads look where the data is not. +// +// The listener addresses are reserved for the same reason. odbselect serves each API on its own +// listener and its health endpoints on a fifth, so moving one bind does not stop the process or +// fail a probe — it leaves the Service port the operator published pointing at nothing while the +// pod stays Ready. spec.signals is how an API is turned off. +var queryReservedConfigPaths = map[string]string{ + keyCluster: hintRingFromSpec, + + keyPrometheus + "." + keyBind: "use spec.signals.metrics", + keyLoki + "." + keyBind: "use spec.signals.logs", + keyTempo + "." + keyBind: "use spec.signals.traces", + keyPyroscope + "." + keyBind: "use spec.signals.profiles", + keyHealth + "." + keyBind: "the operator probes this listener", +} + +func queryEnabled(cr *dbv1alpha1.OtelDBCluster) bool { return cr.Spec.Query != nil } + +func queryReplicasOf(cr *dbv1alpha1.OtelDBCluster) int32 { + if !queryEnabled(cr) { + return 0 + } + if r := cr.Spec.Query.Replicas; r != nil { + return *r + } + return 2 +} + +// queryAPIs are the four query APIs odbselect serves, each on its own listener, paired with the +// signal that decides whether the cluster stores anything for it. +func queryAPIs(cr *dbv1alpha1.OtelDBCluster) []queryAPI { + return []queryAPI{ + {key: keyPrometheus, port: namedPort{portNamePromHTTP, portPromHTTP}, enabled: metricsEnabled(cr)}, + {key: keyLoki, port: namedPort{portNameLokiHTTP, portLokiHTTP}, enabled: logsEnabled(cr)}, + {key: keyTempo, port: namedPort{portNameTempoHTTP, portTempoHTTP}, enabled: tracesEnabled(cr)}, + {key: keyPyroscope, port: namedPort{portNamePyroscope, portPyroscope}, enabled: profilesEnabled(cr)}, + } +} + +// queryAPI is one odbselect query API: its config block, its port, and whether it is served. +type queryAPI struct { + key string + port namedPort + enabled bool +} + +// renderQueryConfig builds odbselect.yml. odbselect holds no data and joins nothing, so the whole +// config is the ring it reads through plus the listeners it answers on. +func renderQueryConfig(cr *dbv1alpha1.OtelDBCluster, etcdEndpoints []string) (string, error) { + if err := validateSignals(cr); err != nil { + return "", err + } + + cfg := map[string]any{ + keyCluster: ringConfig(cr, etcdEndpoints), + keyHealth: map[string]any{keyBind: bindAll(portHealth)}, + } + + // A disabled signal is switched off explicitly rather than by dropping its block: odbselect + // defaults every API block it does not find, so an omitted one is served on its default port. + for _, api := range queryAPIs(cr) { + bind := bindDisabled + if api.enabled { + bind = bindAll(api.port.port) + } + cfg[api.key] = map[string]any{keyBind: bind} + } + + if raw := cr.Spec.Query.ExtraConfig; raw != nil && len(raw.Raw) > 0 { + var extra map[string]any + if err := json.Unmarshal(raw.Raw, &extra); err != nil { + return "", invalidSpec("parse spec.query.extraConfig: %s", err) + } + if err := validateExtraConfigPaths("spec.query.extraConfig", extra, queryReservedConfigPaths); err != nil { + return "", err + } + deepMerge(cfg, extra) + } + + out, err := yaml.Marshal(cfg) + if err != nil { + return "", fmt.Errorf("marshal query config: %w", err) + } + return string(out), nil +} + +// buildQueryConfigMap renders the shared odbselect config into a ConfigMap. +func buildQueryConfigMap(cr *dbv1alpha1.OtelDBCluster, etcdEndpoints []string) (*corev1.ConfigMap, error) { + data, err := renderQueryConfig(cr, etcdEndpoints) + if err != nil { + return nil, err + } + n := namesFor(cr) + return &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: n.queryConfigMap(), + Namespace: cr.Namespace, + Labels: queryCommonLabels(cr), + }, + Data: map[string]string{queryConfigFileName: data}, + }, nil +} + +// buildQueryService exposes every enabled query API across the odbselect pool. This is where +// Grafana points; the cluster's client Service keeps fronting the storage nodes. +// +// One Service carries all four APIs. They are one pod set behind one selector and each API already +// has its own port, so a Service per API would add DNS names and, for a LoadBalancer type, four +// load balancers, without making anything addressable that is not addressable now. +func buildQueryService(cr *dbv1alpha1.OtelDBCluster) *corev1.Service { + n := namesFor(cr) + svcType := cr.Spec.Query.Service.Type + if svcType == "" { + svcType = corev1.ServiceTypeClusterIP + } + + ports := make([]corev1.ServicePort, 0, len(queryAPIs(cr))+1) + for _, p := range queryPorts(cr) { + ports = append(ports, corev1.ServicePort{ + Name: p.name, + Port: p.port, + TargetPort: intstr.FromString(p.name), + Protocol: corev1.ProtocolTCP, + }) + } + + return &corev1.Service{ + ObjectMeta: metav1.ObjectMeta{ + Name: n.queryService(), + Namespace: cr.Namespace, + Labels: queryCommonLabels(cr), + Annotations: cr.Spec.Query.Service.Annotations, + }, + Spec: corev1.ServiceSpec{ + Type: svcType, + Selector: querySelectorLabels(cr), + Ports: ports, + }, + } +} + +// queryPorts are the ports an odbselect pod listens on: one per enabled API plus self-metrics. The +// health listener is deliberately not published — it is probed, not served to clients. +func queryPorts(cr *dbv1alpha1.OtelDBCluster) []namedPort { + apis := queryAPIs(cr) + ports := make([]namedPort, 0, len(apis)+1) + for _, api := range apis { + if api.enabled { + ports = append(ports, api.port) + } + } + return append(ports, namedPort{portNameSelfMetric, portSelfMetric}) +} + +// buildQueryDeployment builds the odbselect pool. It is a Deployment, not a StatefulSet: the pods +// hold no data, are not ring members, and need neither a stable identity nor a volume. +func buildQueryDeployment(cr *dbv1alpha1.OtelDBCluster, configHash string) *appsv1.Deployment { + n := namesFor(cr) + spec := cr.Spec.Query + + image := spec.Image + if image == "" { + image = imageOf(cr) + } + + serviceAccount := spec.ServiceAccountName + if serviceAccount == "" { + serviceAccount = cr.Spec.ServiceAccountName + } + + podLabels := mergeLabels(queryCommonLabels(cr), spec.PodLabels) + annotations := mergeLabels(map[string]string{annConfigHash: configHash}, spec.PodAnnotations) + + ports := make([]corev1.ContainerPort, 0, len(queryPorts(cr))+1) + for _, p := range queryPorts(cr) { + ports = append(ports, corev1.ContainerPort{Name: p.name, ContainerPort: p.port, Protocol: corev1.ProtocolTCP}) + } + ports = append(ports, corev1.ContainerPort{Name: portNameHealth, ContainerPort: portHealth, Protocol: corev1.ProtocolTCP}) + + container := corev1.Container{ + Name: queryAppName, + Image: image, + ImagePullPolicy: cr.Spec.ImagePullPolicy, + Command: []string{queryBinPath}, + Args: []string{"--config=" + configMountPath + "/" + queryConfigFileName}, + Env: statelessPodEnv(cr), + Ports: ports, + VolumeMounts: []corev1.VolumeMount{ + {Name: configVolumeName, MountPath: configMountPath}, + }, + Resources: spec.Resources, + SecurityContext: spec.SecurityContext, + // The probes hit the health listener, not a query API: it is the one listener odbselect + // always serves, whatever spec.signals leaves enabled. + LivenessProbe: roleProbe("/healthz", portNameHealth), + // /readyz answers 503 until the ring has a member, keeping a starting pod out of the load + // balancer while a query could only return an empty result that looks like an answer. + ReadinessProbe: roleProbe("/readyz", portNameHealth), + } + + return &appsv1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: n.queryDeployment(), + Namespace: cr.Namespace, + Labels: queryCommonLabels(cr), + }, + Spec: appsv1.DeploymentSpec{ + Replicas: ptr.To(queryReplicasOf(cr)), + Selector: &metav1.LabelSelector{MatchLabels: querySelectorLabels(cr)}, + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: podLabels, + Annotations: annotations, + }, + Spec: corev1.PodSpec{ + ServiceAccountName: serviceAccount, + ImagePullSecrets: cr.Spec.ImagePullSecrets, + SecurityContext: spec.PodSecurityContext, + NodeSelector: spec.NodeSelector, + Affinity: affinityOr(spec.Affinity, querySelectorLabels(cr)), + Tolerations: spec.Tolerations, + TopologySpreadConstraints: spec.TopologySpreadConstraints, + Containers: []corev1.Container{container}, + Volumes: []corev1.Volume{{ + Name: configVolumeName, + VolumeSource: corev1.VolumeSource{ + ConfigMap: &corev1.ConfigMapVolumeSource{ + LocalObjectReference: corev1.LocalObjectReference{Name: n.queryConfigMap()}, + }, + }, + }}, + }, + }, + }, + } +} diff --git a/internal/controller/query_test.go b/internal/controller/query_test.go new file mode 100644 index 0000000..6713d5f --- /dev/null +++ b/internal/controller/query_test.go @@ -0,0 +1,267 @@ +/* +Copyright 2026. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package controller + +import ( + "errors" + "testing" + + "github.com/stretchr/testify/require" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/labels" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/utils/ptr" + "sigs.k8s.io/yaml" + + dbv1alpha1 "github.com/oteldb/operator/api/v1alpha1" +) + +// queryCluster is ingestCluster with a query pool too, so every role renders from the same +// non-default ring shape. +func queryCluster() *dbv1alpha1.OtelDBCluster { + cr := ingestCluster() + cr.Spec.Query = &dbv1alpha1.QuerySpec{Replicas: ptr.To[int32](3)} + return cr +} + +func renderedQuery(t *testing.T, cr *dbv1alpha1.OtelDBCluster) map[string]any { + t.Helper() + + out, err := renderQueryConfig(cr, cr.Spec.Etcd.Endpoints) + require.NoError(t, err) + + var cfg map[string]any + require.NoError(t, yaml.Unmarshal([]byte(out), &cfg)) + return cfg +} + +func TestQueryDisabledByDefault(t *testing.T) { + cr := testCluster() + require.False(t, queryEnabled(cr), "an unchanged cluster must stay symmetric") + require.Zero(t, queryReplicasOf(cr)) +} + +func TestRenderQueryConfig(t *testing.T) { + cfg := renderedQuery(t, queryCluster()) + + cluster, ok := cfg["cluster"].(map[string]any) + require.True(t, ok, "cluster block missing") + require.Equal(t, []any{"http://etcd:2379"}, cluster["etcd"]) + require.EqualValues(t, 2, cluster["rf"]) + require.EqualValues(t, 8, cluster["shards_per_tenant"]) + require.Equal(t, "/tenants/a", cluster["root"]) + // odbselect is not a ring member, so it has no peer port to advertise. + require.NotContains(t, cluster, "port") + + // Each query API is its own listener, unlike odbingest's single shared one. + for key, want := range map[string]string{ + "prometheus": "0.0.0.0:9090", + "loki": "0.0.0.0:3100", + "tempo": "0.0.0.0:3200", + "pyroscope": "0.0.0.0:4040", + "health": "0.0.0.0:13133", + } { + block, ok := cfg[key].(map[string]any) + require.True(t, ok, "%s block missing", key) + require.Equal(t, want, block["bind"]) + } +} + +// A disabled signal must be switched off explicitly: odbselect defaults every API block it does not +// find, so an omitted block would be served on its default port. +func TestRenderQueryConfigDisabledSignal(t *testing.T) { + cr := queryCluster() + cr.Spec.Signals.Logs = ptr.To(false) + + cfg := renderedQuery(t, cr) + require.Equal(t, bindDisabled, cfg["loki"].(map[string]any)["bind"]) + require.Equal(t, "0.0.0.0:9090", cfg["prometheus"].(map[string]any)["bind"]) +} + +func TestRenderQueryConfigRejectsEverySignalDisabled(t *testing.T) { + cr := queryCluster() + cr.Spec.Signals = dbv1alpha1.SignalsSpec{ + Metrics: ptr.To(false), Logs: ptr.To(false), Traces: ptr.To(false), Profiles: ptr.To(false), + } + + _, err := renderQueryConfig(cr, cr.Spec.Etcd.Endpoints) + require.Error(t, err, "odbselect would serve nothing") +} + +// A ring parameter that differs between the roles does not fail: a role resolves a different owner +// set than the storage nodes do, so writes land where no read will look and reads look where the +// data is not. All three renderers must therefore agree key for key. +func TestRoleRingsMatch(t *testing.T) { + cr := queryCluster() + + storageOut, err := renderConfig(cr, cr.Spec.Etcd.Endpoints) + require.NoError(t, err) + var storageCfg map[string]any + require.NoError(t, yaml.Unmarshal([]byte(storageOut), &storageCfg)) + + storageRing := storageCfg["storage"].(map[string]any)["cluster"].(map[string]any) + rings := map[string]map[string]any{ + "ingest": renderedIngest(t, cr)["cluster"].(map[string]any), + "query": renderedQuery(t, cr)["cluster"].(map[string]any), + } + + for role, ring := range rings { + for _, key := range []string{"etcd", "rf", "shards_per_tenant", "root"} { + require.Equal(t, storageRing[key], ring[key], + "ring parameter %q differs between storage and %s", key, role) + } + } +} + +func TestRenderQueryConfigExtraConfig(t *testing.T) { + cr := queryCluster() + cr.Spec.Query.ExtraConfig = &runtime.RawExtension{ + Raw: []byte(`{"prometheus":{"max_samples":1000},"shutdown_timeout":"1m"}`), + } + + cfg := renderedQuery(t, cr) + prom := cfg["prometheus"].(map[string]any) + require.EqualValues(t, 1000, prom["max_samples"]) + require.Equal(t, "0.0.0.0:9090", prom["bind"], "extraConfig must merge into the block, not replace it") + require.Equal(t, "1m", cfg["shutdown_timeout"]) +} + +func TestRenderQueryConfigReservedPaths(t *testing.T) { + for name, raw := range map[string]string{ + "cluster": `{"cluster":{"rf":9}}`, + "prometheus bind": `{"prometheus":{"bind":"0.0.0.0:9999"}}`, + "health bind": `{"health":{"bind":"0.0.0.0:9999"}}`, + } { + t.Run(name, func(t *testing.T) { + cr := queryCluster() + cr.Spec.Query.ExtraConfig = &runtime.RawExtension{Raw: []byte(raw)} + + _, err := renderQueryConfig(cr, cr.Spec.Etcd.Endpoints) + require.ErrorContains(t, err, "spec.query.extraConfig") + + var invalid validationError + require.True(t, errors.As(err, &invalid), "reserved paths must fail as a spec validation error") + }) + } +} + +func TestBuildQueryDeployment(t *testing.T) { + cr := queryCluster() + deploy := buildQueryDeployment(cr, "cafebabe") + + require.Equal(t, "obs-query", deploy.Name) + require.EqualValues(t, 3, *deploy.Spec.Replicas) + require.Equal(t, "cafebabe", deploy.Spec.Template.Annotations["oteldb.io/config-hash"]) + + container := deploy.Spec.Template.Spec.Containers[0] + require.Equal(t, []string{queryBinPath}, container.Command) + require.Equal(t, []string{"--config=/etc/otel/odbselect.yml"}, container.Args) + require.Equal(t, defaultImage, container.Image, "the query pool defaults to the cluster image") + require.Empty(t, container.VolumeMounts[1:], "odbselect holds no data, so it mounts only its config") + + for _, e := range container.Env { + require.NotContains(t, e.Name, "OTELDB_CLUSTER_", + "odbselect joins nothing and must carry no ring identity") + } + + // The probes target the health listener, not a query API: it is the one listener odbselect + // always serves. + require.Equal(t, "/healthz", container.LivenessProbe.HTTPGet.Path) + require.Equal(t, "/readyz", container.ReadinessProbe.HTTPGet.Path) + require.Equal(t, portNameHealth, container.ReadinessProbe.HTTPGet.Port.StrVal) + + byName := map[string]int32{} + for _, p := range container.Ports { + byName[p.Name] = p.ContainerPort + } + require.EqualValues(t, portHealth, byName[portNameHealth]) + require.EqualValues(t, portPromHTTP, byName[portNamePromHTTP]) + require.NotContains(t, byName, portNameOTLPGRPC, "odbselect does not ingest") +} + +func TestBuildQueryDeploymentImageOverride(t *testing.T) { + cr := queryCluster() + cr.Spec.Query.Image = "ghcr.io/oteldb/oteldb:with-odbselect" + + deploy := buildQueryDeployment(cr, "x") + require.Equal(t, "ghcr.io/oteldb/oteldb:with-odbselect", deploy.Spec.Template.Spec.Containers[0].Image) +} + +// The storage StatefulSet's selector is immutable, so the roles must be told apart by the app name +// alone. No selector may match another role's pods. +func TestQuerySelectorIsDisjointFromOtherRoles(t *testing.T) { + cr := queryCluster() + + queryPods := labels.Set(mergeLabels(queryCommonLabels(cr), cr.Spec.Query.PodLabels)) + querySelector := labels.SelectorFromSet(querySelectorLabels(cr)) + + require.True(t, querySelector.Matches(queryPods)) + require.False(t, querySelector.Matches(labels.Set(commonLabels(cr))), + "the query Service must not front storage pods") + require.False(t, querySelector.Matches(labels.Set(ingestCommonLabels(cr))), + "the query Service must not front ingest pods") + require.False(t, labels.SelectorFromSet(selectorLabels(cr)).Matches(queryPods), + "the client Service must not front query pods") + require.False(t, labels.SelectorFromSet(ingestSelectorLabels(cr)).Matches(queryPods), + "the ingest Service must not front query pods") +} + +// One Service carries every enabled API, each on its own port: odbselect serves four listeners, not +// one shared mux like odbingest. +func TestBuildQueryServicePorts(t *testing.T) { + svc := buildQueryService(queryCluster()) + + require.Equal(t, "obs-query", svc.Name) + require.Equal(t, corev1.ServiceTypeClusterIP, svc.Spec.Type) + + byName := map[string]corev1.ServicePort{} + for _, p := range svc.Spec.Ports { + byName[p.Name] = p + } + + for name, port := range map[string]int32{ + portNamePromHTTP: portPromHTTP, + portNameLokiHTTP: portLokiHTTP, + portNameTempoHTTP: portTempoHTTP, + portNamePyroscope: portPyroscope, + portNameSelfMetric: portSelfMetric, + } { + require.EqualValues(t, port, byName[name].Port, "port %s", name) + require.Equal(t, name, byName[name].TargetPort.StrVal, "port %s", name) + } + require.NotContains(t, byName, portNameHealth, "the health listener is probed, not published") + require.Len(t, svc.Spec.Ports, 5) +} + +func TestBuildQueryServiceDropsDisabledSignals(t *testing.T) { + cr := queryCluster() + cr.Spec.Signals.Traces = ptr.To(false) + + svc := buildQueryService(cr) + for _, p := range svc.Spec.Ports { + require.NotEqual(t, portNameTempoHTTP, p.Name) + } +} + +// The existing client Service keeps fronting the storage nodes: its selector is immutable, and +// repointing an in-use query endpoint at a new workload is a data-path change. +func TestClientServiceUnchangedByQueryPool(t *testing.T) { + cr := queryCluster() + without := testCluster() + + require.Equal(t, buildClientService(without).Spec.Selector, buildClientService(cr).Spec.Selector) +} diff --git a/internal/controller/resources.go b/internal/controller/resources.go index 90a6367..332da9b 100644 --- a/internal/controller/resources.go +++ b/internal/controller/resources.go @@ -144,7 +144,7 @@ func buildStatefulSet(cr *dbv1alpha1.OtelDBCluster, configHash string) *appsv1.S replicas := replicasOf(cr) podLabels := mergeLabels(commonLabels(cr), cr.Spec.PodLabels) - annotations := mergeLabels(map[string]string{"oteldb.io/config-hash": configHash}, cr.Spec.PodAnnotations) + annotations := mergeLabels(map[string]string{annConfigHash: configHash}, cr.Spec.PodAnnotations) container := corev1.Container{ Name: appName, @@ -212,8 +212,8 @@ func podEnv(cr *dbv1alpha1.OtelDBCluster) []corev1.EnvVar { n := namesFor(cr) fqdnSuffix := fmt.Sprintf(".%s.%s.svc.cluster.local:%d", n.peerService(), cr.Namespace, peerPortOf(cr)) env := []corev1.EnvVar{ - {Name: "OTEL_EXPORTER_PROMETHEUS_HOST", Value: "0.0.0.0"}, - {Name: "OTEL_EXPORTER_PROMETHEUS_PORT", Value: fmt.Sprintf("%d", portSelfMetric)}, + {Name: envPrometheusHost, Value: bindAllHost}, + {Name: envPrometheusPort, Value: fmt.Sprintf("%d", portSelfMetric)}, {Name: "POD_NAME", ValueFrom: &corev1.EnvVarSource{ FieldRef: &corev1.ObjectFieldSelector{FieldPath: "metadata.name"}, }}, @@ -221,7 +221,7 @@ func podEnv(cr *dbv1alpha1.OtelDBCluster) []corev1.EnvVar { {Name: "OTELDB_CLUSTER_ADDR", Value: "$(POD_NAME)" + fqdnSuffix}, } if lvl := cr.Spec.LogLevel; lvl != "" { - env = append(env, corev1.EnvVar{Name: "OTEL_LOG_LEVEL", Value: lvl}) + env = append(env, corev1.EnvVar{Name: envLogLevel, Value: lvl}) } if z := cr.Spec.Cluster.StaticZone; z != "" { env = append(env, corev1.EnvVar{Name: "OTELDB_CLUSTER_ZONE", Value: z}) @@ -265,10 +265,12 @@ func containerPorts(cr *dbv1alpha1.OtelDBCluster) []corev1.ContainerPort { return ports } -func httpProbe(path string) *corev1.Probe { +// roleProbe is the HTTP probe shape every role shares: which port carries the health endpoints is +// the only thing that differs between them. +func roleProbe(path, portName string) *corev1.Probe { return &corev1.Probe{ ProbeHandler: corev1.ProbeHandler{ - HTTPGet: &corev1.HTTPGetAction{Path: path, Port: intstr.FromString(portNameHealth)}, + HTTPGet: &corev1.HTTPGetAction{Path: path, Port: intstr.FromString(portName)}, }, PeriodSeconds: 10, TimeoutSeconds: 3, @@ -276,6 +278,25 @@ func httpProbe(path string) *corev1.Probe { } } +func httpProbe(path string) *corev1.Probe { return roleProbe(path, portNameHealth) } + +// bindAll is the listen address for port: reachability is the Service's job. +func bindAll(port int32) string { return fmt.Sprintf("%s:%d", bindAllHost, port) } + +// statelessPodEnv is the environment of a pod in either stateless pool. Unlike a storage pod it +// carries no ring identity: neither odbingest nor odbselect joins the ring, so neither has an id, +// address or zone to advertise. +func statelessPodEnv(cr *dbv1alpha1.OtelDBCluster) []corev1.EnvVar { + env := []corev1.EnvVar{ + {Name: envPrometheusHost, Value: bindAllHost}, + {Name: envPrometheusPort, Value: fmt.Sprintf("%d", portSelfMetric)}, + } + if lvl := cr.Spec.LogLevel; lvl != "" { + env = append(env, corev1.EnvVar{Name: envLogLevel, Value: lvl}) + } + return env +} + func startupProbe(path string) *corev1.Probe { p := httpProbe(path) p.FailureThreshold = 30 // allow up to ~5m for recovery/replay on startup From b62a51a6645e71f8c6a2e0cd47f9703fc5b18fc3 Mon Sep 17 00:00:00 2001 From: tdakkota Date: Tue, 18 Aug 2026 00:44:26 +0300 Subject: [PATCH 2/2] chore(crd): regenerate manifests for the query node group --- api/v1alpha1/zz_generated.deepcopy.go | 82 + .../bases/db.oteldb.io_oteldbclusters.yaml | 1713 ++++++++++++++++- 2 files changed, 1792 insertions(+), 3 deletions(-) diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index 1e164d4..ec26f00 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -317,6 +317,11 @@ func (in *OtelDBClusterSpec) DeepCopyInto(out *OtelDBClusterSpec) { *out = new(IngestSpec) (*in).DeepCopyInto(*out) } + if in.Query != nil { + in, out := &in.Query, &out.Query + *out = new(QuerySpec) + (*in).DeepCopyInto(*out) + } if in.ImagePullSecrets != nil { in, out := &in.ImagePullSecrets, &out.ImagePullSecrets *out = make([]v1.LocalObjectReference, len(*in)) @@ -472,6 +477,83 @@ func (in *PrecisionTierSpec) DeepCopy() *PrecisionTierSpec { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *QuerySpec) DeepCopyInto(out *QuerySpec) { + *out = *in + if in.Replicas != nil { + in, out := &in.Replicas, &out.Replicas + *out = new(int32) + **out = **in + } + in.Resources.DeepCopyInto(&out.Resources) + in.Service.DeepCopyInto(&out.Service) + if in.PodAnnotations != nil { + in, out := &in.PodAnnotations, &out.PodAnnotations + *out = make(map[string]string, len(*in)) + for key, val := range *in { + (*out)[key] = val + } + } + if in.PodLabels != nil { + in, out := &in.PodLabels, &out.PodLabels + *out = make(map[string]string, len(*in)) + for key, val := range *in { + (*out)[key] = val + } + } + if in.NodeSelector != nil { + in, out := &in.NodeSelector, &out.NodeSelector + *out = make(map[string]string, len(*in)) + for key, val := range *in { + (*out)[key] = val + } + } + if in.Affinity != nil { + in, out := &in.Affinity, &out.Affinity + *out = new(v1.Affinity) + (*in).DeepCopyInto(*out) + } + if in.Tolerations != nil { + in, out := &in.Tolerations, &out.Tolerations + *out = make([]v1.Toleration, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.TopologySpreadConstraints != nil { + in, out := &in.TopologySpreadConstraints, &out.TopologySpreadConstraints + *out = make([]v1.TopologySpreadConstraint, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.PodSecurityContext != nil { + in, out := &in.PodSecurityContext, &out.PodSecurityContext + *out = new(v1.PodSecurityContext) + (*in).DeepCopyInto(*out) + } + if in.SecurityContext != nil { + in, out := &in.SecurityContext, &out.SecurityContext + *out = new(v1.SecurityContext) + (*in).DeepCopyInto(*out) + } + if in.ExtraConfig != nil { + in, out := &in.ExtraConfig, &out.ExtraConfig + *out = new(runtime.RawExtension) + (*in).DeepCopyInto(*out) + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new QuerySpec. +func (in *QuerySpec) DeepCopy() *QuerySpec { + if in == nil { + return nil + } + out := new(QuerySpec) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *RecompressSpec) DeepCopyInto(out *RecompressSpec) { *out = *in diff --git a/config/crd/bases/db.oteldb.io_oteldbclusters.yaml b/config/crd/bases/db.oteldb.io_oteldbclusters.yaml index 6580a70..19688c5 100644 --- a/config/crd/bases/db.oteldb.io_oteldbclusters.yaml +++ b/config/crd/bases/db.oteldb.io_oteldbclusters.yaml @@ -3223,6 +3223,1702 @@ spec: x-kubernetes-int-or-string: true type: object type: object + query: + description: |- + Query optionally deploys a pool of stateless odbselect query nodes, so query capacity scales + independently of storage. Absent (the default) means no query pool: the storage nodes are + symmetric and answer queries themselves. + + The pool holds no data and is not a ring member. It follows etcd membership read-only and + fans each read out to the shard's owners, so scaling spec.replicas needs no change to it. + properties: + affinity: + description: |- + Affinity for odbselect pods. When empty, the operator applies a soft anti-affinity that + spreads the pool across nodes. + properties: + nodeAffinity: + description: Describes node affinity scheduling rules for + the pod. + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: |- + The scheduler will prefer to schedule pods to nodes that satisfy + the affinity expressions specified by this field, but it may choose + a node that violates one or more of the expressions. The node that is + most preferred is the one with the greatest sum of weights, i.e. + for each node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, etc.), + compute a sum by iterating through the elements of this field and adding + "weight" to the sum if the node matches the corresponding matchExpressions; the + node(s) with the highest sum are the most preferred. + items: + description: |- + An empty preferred scheduling term matches all objects with implicit weight 0 + (i.e. it's a no-op). A null preferred scheduling term matches no objects (i.e. is also a no-op). + properties: + preference: + description: A node selector term, associated with + the corresponding weight. + properties: + matchExpressions: + description: A list of node selector requirements + by node's labels. + items: + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. + type: string + values: + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchFields: + description: A list of node selector requirements + by node's fields. + items: + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. + type: string + values: + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + type: object + x-kubernetes-map-type: atomic + weight: + description: Weight associated with matching the + corresponding nodeSelectorTerm, in the range 1-100. + format: int32 + type: integer + required: + - preference + - weight + type: object + type: array + x-kubernetes-list-type: atomic + requiredDuringSchedulingIgnoredDuringExecution: + description: |- + If the affinity requirements specified by this field are not met at + scheduling time, the pod will not be scheduled onto the node. + If the affinity requirements specified by this field cease to be met + at some point during pod execution (e.g. due to an update), the system + may or may not try to eventually evict the pod from its node. + properties: + nodeSelectorTerms: + description: Required. A list of node selector terms. + The terms are ORed. + items: + description: |- + A null or empty node selector term matches no objects. The requirements of + them are ANDed. + The TopologySelectorTerm type implements a subset of the NodeSelectorTerm. + properties: + matchExpressions: + description: A list of node selector requirements + by node's labels. + items: + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. + type: string + values: + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchFields: + description: A list of node selector requirements + by node's fields. + items: + description: |- + A node selector requirement is a selector that contains values, a key, and an operator + that relates the key and values. + properties: + key: + description: The label key that the selector + applies to. + type: string + operator: + description: |- + Represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. + type: string + values: + description: |- + An array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. If the operator is Gt or Lt, the values + array must have a single element, which will be interpreted as an integer. + This array is replaced during a strategic merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + type: object + x-kubernetes-map-type: atomic + type: array + x-kubernetes-list-type: atomic + required: + - nodeSelectorTerms + type: object + x-kubernetes-map-type: atomic + type: object + podAffinity: + description: Describes pod affinity scheduling rules (e.g. + co-locate this pod in the same node, zone, etc. as some + other pod(s)). + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: |- + The scheduler will prefer to schedule pods to nodes that satisfy + the affinity expressions specified by this field, but it may choose + a node that violates one or more of the expressions. The node that is + most preferred is the one with the greatest sum of weights, i.e. + for each node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling affinity expressions, etc.), + compute a sum by iterating through the elements of this field and adding + "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the + node(s) with the highest sum are the most preferred. + items: + description: The weights of all of the matched WeightedPodAffinityTerm + fields are added per-node to find the most preferred + node(s) + properties: + podAffinityTerm: + description: Required. A pod affinity term, associated + with the corresponding weight. + properties: + labelSelector: + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. + properties: + matchExpressions: + description: matchExpressions is a list + of label selector requirements. The requirements + are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key + that the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both matchLabelKeys and labelSelector. + Also, matchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. + Also, mismatchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + namespaceSelector: + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list + of label selector requirements. The requirements + are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key + that the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + x-kubernetes-list-type: atomic + topologyKey: + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + weight: + description: |- + weight associated with matching the corresponding podAffinityTerm, + in the range 1-100. + format: int32 + type: integer + required: + - podAffinityTerm + - weight + type: object + type: array + x-kubernetes-list-type: atomic + requiredDuringSchedulingIgnoredDuringExecution: + description: |- + If the affinity requirements specified by this field are not met at + scheduling time, the pod will not be scheduled onto the node. + If the affinity requirements specified by this field cease to be met + at some point during pod execution (e.g. due to a pod label update), the + system may or may not try to eventually evict the pod from its node. + When there are multiple elements, the lists of nodes corresponding to each + podAffinityTerm are intersected, i.e. all terms must be satisfied. + items: + description: |- + Defines a set of pods (namely those matching the labelSelector + relative to the given namespace(s)) that this pod should be + co-located (affinity) or not co-located (anti-affinity) with, + where co-located is defined as running on a node whose value of + the label with key matches that of any node on which + a pod of the set of pods is running + properties: + labelSelector: + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both matchLabelKeys and labelSelector. + Also, matchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. + Also, mismatchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + namespaceSelector: + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + x-kubernetes-list-type: atomic + topologyKey: + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + type: array + x-kubernetes-list-type: atomic + type: object + podAntiAffinity: + description: Describes pod anti-affinity scheduling rules + (e.g. avoid putting this pod in the same node, zone, etc. + as some other pod(s)). + properties: + preferredDuringSchedulingIgnoredDuringExecution: + description: |- + The scheduler will prefer to schedule pods to nodes that satisfy + the anti-affinity expressions specified by this field, but it may choose + a node that violates one or more of the expressions. The node that is + most preferred is the one with the greatest sum of weights, i.e. + for each node that meets all of the scheduling requirements (resource + request, requiredDuringScheduling anti-affinity expressions, etc.), + compute a sum by iterating through the elements of this field and subtracting + "weight" from the sum if the node has pods which matches the corresponding podAffinityTerm; the + node(s) with the highest sum are the most preferred. + items: + description: The weights of all of the matched WeightedPodAffinityTerm + fields are added per-node to find the most preferred + node(s) + properties: + podAffinityTerm: + description: Required. A pod affinity term, associated + with the corresponding weight. + properties: + labelSelector: + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. + properties: + matchExpressions: + description: matchExpressions is a list + of label selector requirements. The requirements + are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key + that the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both matchLabelKeys and labelSelector. + Also, matchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. + Also, mismatchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + namespaceSelector: + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list + of label selector requirements. The requirements + are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key + that the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + x-kubernetes-list-type: atomic + topologyKey: + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + weight: + description: |- + weight associated with matching the corresponding podAffinityTerm, + in the range 1-100. + format: int32 + type: integer + required: + - podAffinityTerm + - weight + type: object + type: array + x-kubernetes-list-type: atomic + requiredDuringSchedulingIgnoredDuringExecution: + description: |- + If the anti-affinity requirements specified by this field are not met at + scheduling time, the pod will not be scheduled onto the node. + If the anti-affinity requirements specified by this field cease to be met + at some point during pod execution (e.g. due to a pod label update), the + system may or may not try to eventually evict the pod from its node. + When there are multiple elements, the lists of nodes corresponding to each + podAffinityTerm are intersected, i.e. all terms must be satisfied. + items: + description: |- + Defines a set of pods (namely those matching the labelSelector + relative to the given namespace(s)) that this pod should be + co-located (affinity) or not co-located (anti-affinity) with, + where co-located is defined as running on a node whose value of + the label with key matches that of any node on which + a pod of the set of pods is running + properties: + labelSelector: + description: |- + A label query over a set of resources, in this case pods. + If it's null, this PodAffinityTerm matches with no Pods. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key in (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both matchLabelKeys and labelSelector. + Also, matchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + mismatchLabelKeys: + description: |- + MismatchLabelKeys is a set of pod label keys to select which pods will + be taken into consideration. The keys are used to lookup values from the + incoming pod labels, those key-value labels are merged with `labelSelector` as `key notin (value)` + to select the group of existing pods which pods will be taken into consideration + for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming + pod labels will be ignored. The default value is empty. + The same key is forbidden to exist in both mismatchLabelKeys and labelSelector. + Also, mismatchLabelKeys cannot be set when labelSelector isn't set. + items: + type: string + type: array + x-kubernetes-list-type: atomic + namespaceSelector: + description: |- + A label query over the set of namespaces that the term applies to. + The term is applied to the union of the namespaces selected by this field + and the ones listed in the namespaces field. + null selector and null or empty namespaces list means "this pod's namespace". + An empty selector ({}) matches all namespaces. + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are + ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that + the selector applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + namespaces: + description: |- + namespaces specifies a static list of namespace names that the term applies to. + The term is applied to the union of the namespaces listed in this field + and the ones selected by namespaceSelector. + null or empty namespaces list and null namespaceSelector means "this pod's namespace". + items: + type: string + type: array + x-kubernetes-list-type: atomic + topologyKey: + description: |- + This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching + the labelSelector in the specified namespaces, where co-located is defined as running on a node + whose value of the label with key topologyKey matches that of any node on which any of the + selected pods is running. + Empty topologyKey is not allowed. + type: string + required: + - topologyKey + type: object + type: array + x-kubernetes-list-type: atomic + type: object + type: object + extraConfig: + description: |- + ExtraConfig is arbitrary additional odbselect config deeply merged over the generated config, + as a top-level YAML/JSON object. Use it to set fields the CRD does not model, such as + prometheus.max_samples, loki.max_sample_rows, per-listener auth or shutdown_timeout. + + The cluster block is reserved and rejected instead of merged: it must stay in step with the + storage nodes. Configure it through spec.etcd and spec.cluster. The listener addresses + (prometheus.bind, loki.bind, tempo.bind, pyroscope.bind, health.bind) are reserved too: the + operator publishes them as Service ports and probes them, and odbselect keeps serving its + other listeners if one moves, so a rebind would fail silently rather than crash the pod. Use + spec.signals to turn an API off. + type: object + x-kubernetes-preserve-unknown-fields: true + image: + description: |- + Image is the container image running odbselect. Defaults to the cluster image + (spec.image, else the operator's pinned image). + type: string + nodeSelector: + additionalProperties: + type: string + description: NodeSelector constrains odbselect pods to nodes with + matching labels. + type: object + podAnnotations: + additionalProperties: + type: string + description: PodAnnotations are added to every odbselect pod. + type: object + podLabels: + additionalProperties: + type: string + description: PodLabels are added to every odbselect pod. + type: object + podSecurityContext: + description: PodSecurityContext for odbselect pods. + properties: + appArmorProfile: + description: |- + appArmorProfile is the AppArmor options to use by the containers in this pod. + Note that this field cannot be set when spec.os.name is windows. + properties: + localhostProfile: + description: |- + localhostProfile indicates a profile loaded on the node that should be used. + The profile must be preconfigured on the node to work. + Must match the loaded name of the profile. + Must be set if and only if type is "Localhost". + type: string + type: + description: |- + type indicates which kind of AppArmor profile will be applied. + Valid options are: + Localhost - a profile pre-loaded on the node. + RuntimeDefault - the container runtime's default profile. + Unconfined - no AppArmor enforcement. + type: string + required: + - type + type: object + fsGroup: + description: |- + A special supplemental group that applies to all containers in a pod. + Some volume types allow the Kubelet to change the ownership of that volume + to be owned by the pod: + + 1. The owning GID will be the FSGroup + 2. The setgid bit is set (new files created in the volume will be owned by FSGroup) + 3. The permission bits are OR'd with rw-rw---- + + If unset, the Kubelet will not modify the ownership and permissions of any volume. + Note that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + fsGroupChangePolicy: + description: |- + fsGroupChangePolicy defines behavior of changing ownership and permission of the volume + before being exposed inside Pod. This field will only apply to + volume types which support fsGroup based ownership(and permissions). + It will have no effect on ephemeral volume types such as: secret, configmaps + and emptydir. + Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. + Note that this field cannot be set when spec.os.name is windows. + type: string + runAsGroup: + description: |- + The GID to run the entrypoint of the container process. + Uses runtime default if unset. + May also be set in SecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence + for that container. + Note that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + runAsNonRoot: + description: |- + Indicates that the container must run as a non-root user. + If true, the Kubelet will validate the image at runtime to ensure that it + does not run as UID 0 (root) and fail to start the container if it does. + If unset or false, no such validation will be performed. + May also be set in SecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + type: boolean + runAsUser: + description: |- + The UID to run the entrypoint of the container process. + Defaults to user specified in image metadata if unspecified. + May also be set in SecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence + for that container. + Note that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + seLinuxChangePolicy: + description: |- + seLinuxChangePolicy defines how the container's SELinux label is applied to all volumes used by the Pod. + It has no effect on nodes that do not support SELinux or to volumes does not support SELinux. + Valid values are "MountOption" and "Recursive". + + "Recursive" means relabeling of all files on all Pod volumes by the container runtime. + This may be slow for large volumes, but allows mixing privileged and unprivileged Pods sharing the same volume on the same node. + + "MountOption" mounts all eligible Pod volumes with `-o context` mount option. + This requires all Pods that share the same volume to use the same SELinux label. + It is not possible to share the same volume among privileged and unprivileged Pods. + Eligible volumes are in-tree FibreChannel and iSCSI volumes, and all CSI volumes + whose CSI driver announces SELinux support by setting spec.seLinuxMount: true in their + CSIDriver instance. Other volumes are always re-labelled recursively. + "MountOption" value is allowed only when SELinuxMount feature gate is enabled. + + If not specified and SELinuxMount feature gate is enabled, "MountOption" is used. + If not specified and SELinuxMount feature gate is disabled, "MountOption" is used for ReadWriteOncePod volumes + and "Recursive" for all other volumes. + + This field affects only Pods that have SELinux label set, either in PodSecurityContext or in SecurityContext of all containers. + + All Pods that use the same volume should use the same seLinuxChangePolicy, otherwise some pods can get stuck in ContainerCreating state. + Note that this field cannot be set when spec.os.name is windows. + type: string + seLinuxOptions: + description: |- + The SELinux context to be applied to all containers. + If unspecified, the container runtime will allocate a random SELinux context for each + container. May also be set in SecurityContext. If set in + both SecurityContext and PodSecurityContext, the value specified in SecurityContext + takes precedence for that container. + Note that this field cannot be set when spec.os.name is windows. + properties: + level: + description: Level is SELinux level label that applies + to the container. + type: string + role: + description: Role is a SELinux role label that applies + to the container. + type: string + type: + description: Type is a SELinux type label that applies + to the container. + type: string + user: + description: User is a SELinux user label that applies + to the container. + type: string + type: object + seccompProfile: + description: |- + The seccomp options to use by the containers in this pod. + Note that this field cannot be set when spec.os.name is windows. + properties: + localhostProfile: + description: |- + localhostProfile indicates a profile defined in a file on the node should be used. + The profile must be preconfigured on the node to work. + Must be a descending path, relative to the kubelet's configured seccomp profile location. + Must be set if type is "Localhost". Must NOT be set for any other type. + type: string + type: + description: |- + type indicates which kind of seccomp profile will be applied. + Valid options are: + + Localhost - a profile defined in a file on the node should be used. + RuntimeDefault - the container runtime default profile should be used. + Unconfined - no profile should be applied. + type: string + required: + - type + type: object + supplementalGroups: + description: |- + A list of groups applied to the first process run in each container, in + addition to the container's primary GID and fsGroup (if specified). If + the SupplementalGroupsPolicy feature is enabled, the + supplementalGroupsPolicy field determines whether these are in addition + to or instead of any group memberships defined in the container image. + If unspecified, no additional groups are added, though group memberships + defined in the container image may still be used, depending on the + supplementalGroupsPolicy field. + Note that this field cannot be set when spec.os.name is windows. + items: + format: int64 + type: integer + type: array + x-kubernetes-list-type: atomic + supplementalGroupsPolicy: + description: |- + Defines how supplemental groups of the first container processes are calculated. + Valid values are "Merge" and "Strict". If not specified, "Merge" is used. + (Alpha) Using the field requires the SupplementalGroupsPolicy feature gate to be enabled + and the container runtime must implement support for this feature. + Note that this field cannot be set when spec.os.name is windows. + type: string + sysctls: + description: |- + Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported + sysctls (by the container runtime) might fail to launch. + Note that this field cannot be set when spec.os.name is windows. + items: + description: Sysctl defines a kernel parameter to be set + properties: + name: + description: Name of a property to set + type: string + value: + description: Value of a property to set + type: string + required: + - name + - value + type: object + type: array + x-kubernetes-list-type: atomic + windowsOptions: + description: |- + The Windows specific settings applied to all containers. + If unspecified, the options within a container's SecurityContext will be used. + If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is linux. + properties: + gmsaCredentialSpec: + description: |- + GMSACredentialSpec is where the GMSA admission webhook + (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the + GMSA credential spec named by the GMSACredentialSpecName field. + type: string + gmsaCredentialSpecName: + description: GMSACredentialSpecName is the name of the + GMSA credential spec to use. + type: string + hostProcess: + description: |- + HostProcess determines if a container should be run as a 'Host Process' container. + All of a Pod's containers must have the same effective HostProcess value + (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). + In addition, if HostProcess is true then HostNetwork must also be set to true. + type: boolean + runAsUserName: + description: |- + The UserName in Windows to run the entrypoint of the container process. + Defaults to the user specified in image metadata if unspecified. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + type: string + type: object + type: object + replicas: + default: 2 + description: |- + Replicas is the number of odbselect pods. They hold no data, so this is purely a throughput + and availability knob and can be changed freely. + format: int32 + minimum: 0 + type: integer + resources: + description: Resources are the compute resources for each odbselect + container. + properties: + claims: + description: |- + Claims lists the names of resources, defined in spec.resourceClaims, + that are used by this container. + + This field depends on the + DynamicResourceAllocation feature gate. + + This field is immutable. It can only be set for containers. + items: + description: ResourceClaim references one entry in PodSpec.ResourceClaims. + properties: + name: + description: |- + Name must match the name of one entry in pod.spec.resourceClaims of + the Pod where this field is used. It makes that resource available + inside a container. + type: string + request: + description: |- + Request is the name chosen for a request in the referenced claim. + If empty, everything from the claim is made available, otherwise + only the result of this request. + type: string + required: + - name + type: object + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + limits: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: |- + Limits describes the maximum amount of compute resources allowed. + More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ + type: object + requests: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: |- + Requests describes the minimum amount of compute resources required. + If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, + otherwise to an implementation-defined value. Requests cannot exceed Limits. + More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ + type: object + type: object + securityContext: + description: SecurityContext for the odbselect container. + properties: + allowPrivilegeEscalation: + description: |- + AllowPrivilegeEscalation controls whether a process can gain more + privileges than its parent process. This bool directly controls if + the no_new_privs flag will be set on the container process. + AllowPrivilegeEscalation is true always when the container is: + 1) run as Privileged + 2) has CAP_SYS_ADMIN + Note that this field cannot be set when spec.os.name is windows. + type: boolean + appArmorProfile: + description: |- + appArmorProfile is the AppArmor options to use by this container. If set, this profile + overrides the pod's appArmorProfile. + Note that this field cannot be set when spec.os.name is windows. + properties: + localhostProfile: + description: |- + localhostProfile indicates a profile loaded on the node that should be used. + The profile must be preconfigured on the node to work. + Must match the loaded name of the profile. + Must be set if and only if type is "Localhost". + type: string + type: + description: |- + type indicates which kind of AppArmor profile will be applied. + Valid options are: + Localhost - a profile pre-loaded on the node. + RuntimeDefault - the container runtime's default profile. + Unconfined - no AppArmor enforcement. + type: string + required: + - type + type: object + capabilities: + description: |- + The capabilities to add/drop when running containers. + Defaults to the default set of capabilities granted by the container runtime. + Note that this field cannot be set when spec.os.name is windows. + properties: + add: + description: Added capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + x-kubernetes-list-type: atomic + drop: + description: Removed capabilities + items: + description: Capability represent POSIX capabilities + type + type: string + type: array + x-kubernetes-list-type: atomic + type: object + privileged: + description: |- + Run container in privileged mode. + Processes in privileged containers are essentially equivalent to root on the host. + Defaults to false. + Note that this field cannot be set when spec.os.name is windows. + type: boolean + procMount: + description: |- + procMount denotes the type of proc mount to use for the containers. + The default value is Default which uses the container runtime defaults for + readonly paths and masked paths. + Note that this field cannot be set when spec.os.name is windows. + type: string + readOnlyRootFilesystem: + description: |- + Whether this container has a read-only root filesystem. + Default is false. + Note that this field cannot be set when spec.os.name is windows. + type: boolean + runAsGroup: + description: |- + The GID to run the entrypoint of the container process. + Uses runtime default if unset. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + runAsNonRoot: + description: |- + Indicates that the container must run as a non-root user. + If true, the Kubelet will validate the image at runtime to ensure that it + does not run as UID 0 (root) and fail to start the container if it does. + If unset or false, no such validation will be performed. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + type: boolean + runAsUser: + description: |- + The UID to run the entrypoint of the container process. + Defaults to user specified in image metadata if unspecified. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is windows. + format: int64 + type: integer + seLinuxOptions: + description: |- + The SELinux context to be applied to the container. + If unspecified, the container runtime will allocate a random SELinux context for each + container. May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is windows. + properties: + level: + description: Level is SELinux level label that applies + to the container. + type: string + role: + description: Role is a SELinux role label that applies + to the container. + type: string + type: + description: Type is a SELinux type label that applies + to the container. + type: string + user: + description: User is a SELinux user label that applies + to the container. + type: string + type: object + seccompProfile: + description: |- + The seccomp options to use by this container. If seccomp options are + provided at both the pod & container level, the container options + override the pod options. + Note that this field cannot be set when spec.os.name is windows. + properties: + localhostProfile: + description: |- + localhostProfile indicates a profile defined in a file on the node should be used. + The profile must be preconfigured on the node to work. + Must be a descending path, relative to the kubelet's configured seccomp profile location. + Must be set if type is "Localhost". Must NOT be set for any other type. + type: string + type: + description: |- + type indicates which kind of seccomp profile will be applied. + Valid options are: + + Localhost - a profile defined in a file on the node should be used. + RuntimeDefault - the container runtime default profile should be used. + Unconfined - no profile should be applied. + type: string + required: + - type + type: object + windowsOptions: + description: |- + The Windows specific settings applied to all containers. + If unspecified, the options from the PodSecurityContext will be used. + If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. + Note that this field cannot be set when spec.os.name is linux. + properties: + gmsaCredentialSpec: + description: |- + GMSACredentialSpec is where the GMSA admission webhook + (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the + GMSA credential spec named by the GMSACredentialSpecName field. + type: string + gmsaCredentialSpecName: + description: GMSACredentialSpecName is the name of the + GMSA credential spec to use. + type: string + hostProcess: + description: |- + HostProcess determines if a container should be run as a 'Host Process' container. + All of a Pod's containers must have the same effective HostProcess value + (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). + In addition, if HostProcess is true then HostNetwork must also be set to true. + type: boolean + runAsUserName: + description: |- + The UserName in Windows to run the entrypoint of the container process. + Defaults to the user specified in image metadata if unspecified. + May also be set in PodSecurityContext. If set in both SecurityContext and + PodSecurityContext, the value specified in SecurityContext takes precedence. + type: string + type: object + type: object + service: + description: |- + Service configures the client-facing Service that exposes the query APIs. One Service carries + every enabled API, each on its own port; they share a pod set, so splitting them across four + Services would only add DNS names. + properties: + annotations: + additionalProperties: + type: string + description: Annotations added to the client Service. + type: object + type: + default: ClusterIP + description: Type of the client Service. + enum: + - ClusterIP + - NodePort + - LoadBalancer + type: string + type: object + serviceAccountName: + description: |- + ServiceAccountName is the ServiceAccount for the odbselect pods. Empty falls back to + spec.serviceAccountName. + type: string + tolerations: + description: Tolerations for odbselect pods. + items: + description: |- + The pod this Toleration is attached to tolerates any taint that matches + the triple using the matching operator . + properties: + effect: + description: |- + Effect indicates the taint effect to match. Empty means match all taint effects. + When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute. + type: string + key: + description: |- + Key is the taint key that the toleration applies to. Empty means match all taint keys. + If the key is empty, operator must be Exists; this combination means to match all values and all keys. + type: string + operator: + description: |- + Operator represents a key's relationship to the value. + Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal. + Exists is equivalent to wildcard for value, so that a pod can + tolerate all taints of a particular category. + Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators). + type: string + tolerationSeconds: + description: |- + TolerationSeconds represents the period of time the toleration (which must be + of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, + it is not set, which means tolerate the taint forever (do not evict). Zero and + negative values will be treated as 0 (evict immediately) by the system. + format: int64 + type: integer + value: + description: |- + Value is the taint value the toleration matches to. + If the operator is Exists, the value should be empty, otherwise just a regular string. + type: string + type: object + type: array + topologySpreadConstraints: + description: TopologySpreadConstraints for odbselect pods. + items: + description: TopologySpreadConstraint specifies how to spread + matching pods among the given topology. + properties: + labelSelector: + description: |- + LabelSelector is used to find matching pods. + Pods that match this label selector are counted to determine the number of pods + in their corresponding topology domain. + properties: + matchExpressions: + description: matchExpressions is a list of label selector + requirements. The requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that the selector + applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + matchLabelKeys: + description: |- + MatchLabelKeys is a set of pod label keys to select the pods over which + spreading will be calculated. The keys are used to lookup values from the + incoming pod labels, those key-value labels are ANDed with labelSelector + to select the group of existing pods over which spreading will be calculated + for the incoming pod. The same key is forbidden to exist in both MatchLabelKeys and LabelSelector. + MatchLabelKeys cannot be set when LabelSelector isn't set. + Keys that don't exist in the incoming pod labels will + be ignored. A null or empty list means only match against labelSelector. + + This is a beta field and requires the MatchLabelKeysInPodTopologySpread feature gate to be enabled (enabled by default). + items: + type: string + type: array + x-kubernetes-list-type: atomic + maxSkew: + description: |- + MaxSkew describes the degree to which pods may be unevenly distributed. + When `whenUnsatisfiable=DoNotSchedule`, it is the maximum permitted difference + between the number of matching pods in the target topology and the global minimum. + The global minimum is the minimum number of matching pods in an eligible domain + or zero if the number of eligible domains is less than MinDomains. + For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same + labelSelector spread as 2/2/1: + In this case, the global minimum is 1. + | zone1 | zone2 | zone3 | + | P P | P P | P | + - if MaxSkew is 1, incoming pod can only be scheduled to zone3 to become 2/2/2; + scheduling it onto zone1(zone2) would make the ActualSkew(3-1) on zone1(zone2) + violate MaxSkew(1). + - if MaxSkew is 2, incoming pod can be scheduled onto any zone. + When `whenUnsatisfiable=ScheduleAnyway`, it is used to give higher precedence + to topologies that satisfy it. + It's a required field. Default value is 1 and 0 is not allowed. + format: int32 + type: integer + minDomains: + description: |- + MinDomains indicates a minimum number of eligible domains. + When the number of eligible domains with matching topology keys is less than minDomains, + Pod Topology Spread treats "global minimum" as 0, and then the calculation of Skew is performed. + And when the number of eligible domains with matching topology keys equals or greater than minDomains, + this value has no effect on scheduling. + As a result, when the number of eligible domains is less than minDomains, + scheduler won't schedule more than maxSkew Pods to those domains. + If value is nil, the constraint behaves as if MinDomains is equal to 1. + Valid values are integers greater than 0. + When value is not nil, WhenUnsatisfiable must be DoNotSchedule. + + For example, in a 3-zone cluster, MaxSkew is set to 2, MinDomains is set to 5 and pods with the same + labelSelector spread as 2/2/2: + | zone1 | zone2 | zone3 | + | P P | P P | P P | + The number of domains is less than 5(MinDomains), so "global minimum" is treated as 0. + In this situation, new pod with the same labelSelector cannot be scheduled, + because computed skew will be 3(3 - 0) if new Pod is scheduled to any of the three zones, + it will violate MaxSkew. + format: int32 + type: integer + nodeAffinityPolicy: + description: |- + NodeAffinityPolicy indicates how we will treat Pod's nodeAffinity/nodeSelector + when calculating pod topology spread skew. Options are: + - Honor: only nodes matching nodeAffinity/nodeSelector are included in the calculations. + - Ignore: nodeAffinity/nodeSelector are ignored. All nodes are included in the calculations. + + If this value is nil, the behavior is equivalent to the Honor policy. + type: string + nodeTaintsPolicy: + description: |- + NodeTaintsPolicy indicates how we will treat node taints when calculating + pod topology spread skew. Options are: + - Honor: nodes without taints, along with tainted nodes for which the incoming pod + has a toleration, are included. + - Ignore: node taints are ignored. All nodes are included. + + If this value is nil, the behavior is equivalent to the Ignore policy. + type: string + topologyKey: + description: |- + TopologyKey is the key of node labels. Nodes that have a label with this key + and identical values are considered to be in the same topology. + We consider each as a "bucket", and try to put balanced number + of pods into each bucket. + We define a domain as a particular instance of a topology. + Also, we define an eligible domain as a domain whose nodes meet the requirements of + nodeAffinityPolicy and nodeTaintsPolicy. + e.g. If TopologyKey is "kubernetes.io/hostname", each Node is a domain of that topology. + And, if TopologyKey is "topology.kubernetes.io/zone", each zone is a domain of that topology. + It's a required field. + type: string + whenUnsatisfiable: + description: |- + WhenUnsatisfiable indicates how to deal with a pod if it doesn't satisfy + the spread constraint. + - DoNotSchedule (default) tells the scheduler not to schedule it. + - ScheduleAnyway tells the scheduler to schedule the pod in any location, + but giving higher precedence to topologies that would help reduce the + skew. + A constraint is considered "Unsatisfiable" for an incoming pod + if and only if every possible node assignment for that pod would violate + "MaxSkew" on some topology. + For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same + labelSelector spread as 3/1/1: + | zone1 | zone2 | zone3 | + | P P P | P | P | + If WhenUnsatisfiable is set to DoNotSchedule, incoming pod can only be scheduled + to zone2(zone3) to become 3/2/1(3/1/2) as ActualSkew(2-1) on zone2(zone3) satisfies + MaxSkew(1). In other words, the cluster can still be imbalanced, but scheduler + won't make it *more* imbalanced. + It's a required field. + type: string + required: + - maxSkew + - topologyKey + - whenUnsatisfiable + type: object + type: array + type: object replicas: default: 3 description: |- @@ -3230,9 +4926,10 @@ spec: StatefulSet pod that ingests, queries, stores, and replicates. For real redundancy this should be at least the replication factor (see Cluster.ReplicationFactor). - This is the storage dimension only. Ingest capacity scales separately via spec.ingest, and - scaling either has no effect on the other: membership lives in etcd and the ring is computed - from it, so an ingest pod needs no knowledge of how many storage nodes there are. + This is the storage dimension only. Ingest and query capacity scale separately via + spec.ingest and spec.query, and scaling any of them has no effect on the others: membership + lives in etcd and the ring is computed from it, so a stateless pod needs no knowledge of how + many storage nodes there are. format: int32 minimum: 1 type: integer @@ -3933,6 +5630,16 @@ spec: - Ready - Degraded type: string + queryReadyReplicas: + description: QueryReadyReplicas is the number of ready odbselect nodes. + format: int32 + type: integer + queryReplicas: + description: |- + QueryReplicas is the desired number of odbselect nodes. Zero when no query pool is + configured. + format: int32 + type: integer readyReplicas: description: ReadyReplicas is the number of ready oteldb nodes. format: int32