feat(api): add opt-in spec.ingest.tenant write routing

README.md

@@ -46,6 +46,39 @@ operator renders all three roles from the same `spec.cluster`, so the mismatch c
 > `ghcr.io/oteldb/oteldb` tag contains either one. Set `spec.ingest.image` / `spec.query.image` to a
 > build that ships them until that is fixed upstream.
 
+#### Ingest tenancy
+
+`spec.ingest.tenant` maps onto odbingest's `tenant` block
+([oteldb/oteldb#1280](https://github.com/oteldb/oteldb/pull/1280)) and resolves the tenant a write
+routes to. The sources compose narrowest-first — `header` (a whole request) beats
+`resourceAttributes` (one resource within it) beats `default`:
+
+```yaml
+spec:
+  ingest:
+    tenant:
+      header: X-Scope-OrgID          # what Grafana-stack senders use
+      resourceAttributes: [service.namespace]
+      default: shared
+      require: false                 # 400 a request with no header instead of using default
+```
+
+> **Leave it absent unless you are running a deliberate migration.** oteldb's *read* path is still
+> pinned to one tenant: the query `Backend`'s tenant id is never assigned, so every PromQL, LogQL,
+> TraceQL and Pyroscope query resolves `"default"`
+> ([oteldb/oteldb#820](https://github.com/oteldb/oteldb/issues/820)). Turning tenant resolution on
+> at ingest today routes writes into tenants **nothing can query** — the data is stored, under a
+> different shard key, and is invisible until the read path catches up. The field exists so the
+> deployment is configurable ahead of that, not because it is ready.
+
+Absent renders no `tenant` block at all, which is byte-identical to the config an ingest pool got
+before this field existed: odbingest installs no resolver and every write lands in `"default"`.
+
+Because a tenant id picks the shard key, which picks the ring owners, changing how tenants resolve
+moves where data lands — a migration, not a config tweak. The operator validates what odbingest
+would otherwise crash-loop on (`require` without `header`, an empty attribute key, a `default` that
+is not a legal tenant id, a block naming no source at all) and reports it as `Degraded`/`InvalidSpec`.
+
 #### The query pool's shape
 
 `odbingest` serves OTLP/HTTP, remote write and its health endpoints on one listener. `odbselect`
@@ -105,7 +138,8 @@ for a fuller example including the S3 backend.
 | Field | Purpose |
 |---|---|
 | `replicas` | Number of oteldb storage nodes (StatefulSet size). Use `>= cluster.replicationFactor`. |
-| `ingest` | Optional stateless `odbingest` write pool (Deployment). Absent ⇒ symmetric nodes handle ingest. Takes `replicas` (default 2), `image`, `service`, `extraConfig` and the standard scheduling/security knobs. |
+| `ingest` | Optional stateless `odbingest` write pool (Deployment). Absent ⇒ symmetric nodes handle ingest. Takes `replicas` (default 2), `image`, `service`, `tenant`, `extraConfig` and the standard scheduling/security knobs. |
+| `ingest.tenant` | Opt-in multi-tenant write routing: `{header, resourceAttributes, default, require}`. **Absent by default, and that is the recommended setting** — see [Ingest tenancy](#ingest-tenancy). |
 | `query` | Optional stateless `odbselect` query pool (Deployment). Absent ⇒ symmetric nodes answer queries. Same knobs as `ingest`. Which APIs it serves follows `spec.signals`. |
 | `image` / `imagePullPolicy` / `imagePullSecrets` | oteldb container image (default `ghcr.io/oteldb/oteldb:v0.46.0`). |
 | `etcd.endpoints` | **Required.** External etcd endpoint list. |
@@ -199,7 +233,8 @@ spec field to use instead.
 `storage.policy` is now modelled in full, so the whole block is reserved.
 
 `spec.ingest.extraConfig` is merged over the generated `odbingest.yml` under the same rules, with
-the whole `cluster` block reserved for `spec.cluster` and `spec.etcd.endpoints`.
+the whole `cluster` block reserved for `spec.cluster` and `spec.etcd.endpoints`, and the whole
+`tenant` block reserved for `spec.ingest.tenant`.
 
 `spec.query.extraConfig` is merged over the generated `odbselect.yml`, reserving `cluster` and the
 five listener addresses (`prometheus.bind`, `loki.bind`, `tempo.bind`, `pyroscope.bind`,

api/v1alpha1/oteldbcluster_types.go

@@ -190,6 +190,9 @@ type EtcdSpec struct {
 // mismatch there does not fail — it resolves a different owner set than the nodes do, and writes
 // land where no read will look for them — so it is deliberately not configurable per pool.
 //
+// Writes route to the "default" tenant unless Tenant opts the pool into tenant resolution; see the
+// warning on that field before doing so.
+//
 // The odbingest binary is not yet shipped in the released oteldb image (it is absent from oteldb's
 // goreleaser builds and release Dockerfile). Until it is, set Image to a build that contains it.
 type IngestSpec struct {
@@ -251,17 +254,71 @@ type IngestSpec struct {
 	// +optional
 	SecurityContext *corev1.SecurityContext `json:"securityContext,omitempty"`
 
+	// Tenant opts the ingest pool into multi-tenant write routing. Absent — the default — routes
+	// every write to the "default" tenant, exactly as an unconfigured odbingest does, and renders
+	// no tenant block at all.
+	//
+	// Leave it absent unless you are running a deliberate migration. oteldb's *read* path is still
+	// pinned to a single tenant: the query Backend's tenant id is never assigned, so every PromQL,
+	// LogQL, TraceQL and Pyroscope query resolves "default" (oteldb/oteldb#820). Enabling tenant
+	// resolution here therefore routes writes into tenants that nothing can currently query — the
+	// data is stored, under a different shard key, and is invisible until the read path catches up.
+	// The field exists so the deployment is configurable ahead of that, not because it is ready.
+	// +optional
+	Tenant *TenantSpec `json:"tenant,omitempty"`
+
 	// ExtraConfig is arbitrary additional odbingest config deeply merged over the generated
 	// config, as a top-level YAML/JSON object. Use it to set fields the CRD does not model, such as
 	// prometheus_remote_write.time_threshold or the OTLP body-size limits.
 	//
-	// The cluster block is reserved and rejected instead of merged: it must stay in step with the
-	// storage nodes. Configure it through spec.etcd and spec.cluster.
+	// The cluster and tenant blocks are reserved and rejected instead of merged: cluster must stay
+	// in step with the storage nodes, and tenant is modelled in full. Configure them through
+	// spec.etcd, spec.cluster and spec.ingest.tenant.
 	// +optional
 	// +kubebuilder:pruning:PreserveUnknownFields
 	ExtraConfig *runtime.RawExtension `json:"extraConfig,omitempty"`
 }
 
+// TenantSpec configures which tenant an ingested write routes to, mapping onto odbingest's tenant
+// config block.
+//
+// The sources compose narrowest-first: Header names the tenant of a whole request and wins, then
+// ResourceAttributes names the tenant of one resource within it, and Default backs both. A header
+// beats an attribute because a gateway that authenticated the sender sets the header, whereas an
+// attribute is whatever the sender put in its own payload.
+//
+// A tenant id picks the shard key, which picks the ring owners, so changing how tenants resolve
+// moves where data lands. That makes enabling this a migration rather than a config tweak — and,
+// while oteldb's read path is single-tenant (see IngestSpec.Tenant), a one-way one.
+type TenantSpec struct {
+	// Header is the request header — and OTLP/gRPC metadata key — carrying the tenant. Empty does
+	// not read a header. "X-Scope-OrgID" is what Grafana-stack senders (Loki, Mimir, and their
+	// Grafana datasources) put it in.
+	// +optional
+	Header string `json:"header,omitempty"`
+
+	// ResourceAttributes are OTLP resource attribute keys, read in order until one holds a usable
+	// tenant. Empty does not read resource attributes. "service.namespace" is the OTel-native
+	// candidate.
+	//
+	// A missing, non-string or malformed value falls back to Default rather than failing the batch:
+	// by the time a resource is framed, the request can no longer be answered with an error.
+	// +optional
+	// +listType=atomic
+	ResourceAttributes []string `json:"resourceAttributes,omitempty"`
+
+	// Default is the tenant a write routes to when no source names one. Empty uses odbingest's own
+	// default tenant, "default", which is where every write goes today.
+	// +optional
+	Default string `json:"default,omitempty"`
+
+	// Require refuses a request that does not carry Header with 400 (InvalidArgument over gRPC),
+	// instead of routing it to Default. Use it when two senders must not be able to land in one
+	// shared tenant by omitting the header. It needs Header to be set.
+	// +optional
+	Require bool `json:"require,omitempty"`
+}
+
 // QuerySpec configures the stateless odbselect query pool: a Deployment of nodes that serve the
 // PromQL, LogQL, TraceQL and Pyroscope APIs by reading through the ring instead of local storage.
 //

config/crd/bases/db.oteldb.io_oteldbclusters.yaml

@@ -2062,8 +2062,9 @@ spec:
                       config, as a top-level YAML/JSON object. Use it to set fields the CRD does not model, such as
                       prometheus_remote_write.time_threshold or the OTLP body-size limits.
 
-                      The cluster block is reserved and rejected instead of merged: it must stay in step with the
-                      storage nodes. Configure it through spec.etcd and spec.cluster.
+                      The cluster and tenant blocks are reserved and rejected instead of merged: cluster must stay
+                      in step with the storage nodes, and tenant is modelled in full. Configure them through
+                      spec.etcd, spec.cluster and spec.ingest.tenant.
                     type: object
                     x-kubernetes-preserve-unknown-fields: true
                   image:
@@ -2601,6 +2602,49 @@ spec:
                       ServiceAccountName is the ServiceAccount for the odbingest pods. Empty falls back to
                       spec.serviceAccountName.
                     type: string
+                  tenant:
+                    description: |-
+                      Tenant opts the ingest pool into multi-tenant write routing. Absent — the default — routes
+                      every write to the "default" tenant, exactly as an unconfigured odbingest does, and renders
+                      no tenant block at all.
+
+                      Leave it absent unless you are running a deliberate migration. oteldb's *read* path is still
+                      pinned to a single tenant: the query Backend's tenant id is never assigned, so every PromQL,
+                      LogQL, TraceQL and Pyroscope query resolves "default" (oteldb/oteldb#820). Enabling tenant
+                      resolution here therefore routes writes into tenants that nothing can currently query — the
+                      data is stored, under a different shard key, and is invisible until the read path catches up.
+                      The field exists so the deployment is configurable ahead of that, not because it is ready.
+                    properties:
+                      default:
+                        description: |-
+                          Default is the tenant a write routes to when no source names one. Empty uses odbingest's own
+                          default tenant, "default", which is where every write goes today.
+                        type: string
+                      header:
+                        description: |-
+                          Header is the request header — and OTLP/gRPC metadata key — carrying the tenant. Empty does
+                          not read a header. "X-Scope-OrgID" is what Grafana-stack senders (Loki, Mimir, and their
+                          Grafana datasources) put it in.
+                        type: string
+                      require:
+                        description: |-
+                          Require refuses a request that does not carry Header with 400 (InvalidArgument over gRPC),
+                          instead of routing it to Default. Use it when two senders must not be able to land in one
+                          shared tenant by omitting the header. It needs Header to be set.
+                        type: boolean
+                      resourceAttributes:
+                        description: |-
+                          ResourceAttributes are OTLP resource attribute keys, read in order until one holds a usable
+                          tenant. Empty does not read resource attributes. "service.namespace" is the OTel-native
+                          candidate.
+
+                          A missing, non-string or malformed value falls back to Default rather than failing the batch:
+                          by the time a resource is framed, the request can no longer be answered with an error.
+                        items:
+                          type: string
+                        type: array
+                        x-kubernetes-list-type: atomic
+                    type: object
                   tolerations:
                     description: Tolerations for odbingest pods.
                     items:

internal/controller/ingest.go

@@ -35,6 +35,7 @@ import (
 // set, so writes land where no read will look for them.
 var ingestReservedConfigPaths = map[string]string{
 	keyCluster: hintRingFromSpec,
+	keyTenant:  "use spec.ingest.tenant",
 }
 
 func ingestEnabled(cr *dbv1alpha1.OtelDBCluster) bool { return cr.Spec.Ingest != nil }
@@ -52,6 +53,10 @@ func ingestReplicasOf(cr *dbv1alpha1.OtelDBCluster) int32 {
 // renderIngestConfig builds odbingest.yml. odbingest holds no data and joins nothing, so the whole
 // config is the ring it routes into plus the two listeners it serves.
 func renderIngestConfig(cr *dbv1alpha1.OtelDBCluster, etcdEndpoints []string) (string, error) {
+	if err := validateIngestTenant(cr); err != nil {
+		return "", err
+	}
+
 	cfg := map[string]any{
 		keyCluster: ringConfig(cr, etcdEndpoints),
 		"prometheus_remote_write": map[string]any{
@@ -62,6 +67,10 @@ func renderIngestConfig(cr *dbv1alpha1.OtelDBCluster, etcdEndpoints []string) (s
 		},
 	}
 
+	if tenant := renderTenant(cr.Spec.Ingest.Tenant); tenant != nil {
+		cfg[keyTenant] = tenant
+	}
+
 	if raw := cr.Spec.Ingest.ExtraConfig; raw != nil && len(raw.Raw) > 0 {
 		var extra map[string]any
 		if err := json.Unmarshal(raw.Raw, &extra); err != nil {
@@ -233,3 +242,100 @@ func buildIngestDeployment(cr *dbv1alpha1.OtelDBCluster, configHash string) *app
 }
 
 func ingestProbe(path string) *corev1.Probe { return roleProbe(path, portNameIngestHTTP) }
+
+// renderTenant builds odbingest's tenant block, or nil when the spec configures nothing. odbingest
+// installs no resolver for an empty block, which is the unconfigured behaviour: everything routes
+// to "default".
+func renderTenant(spec *dbv1alpha1.TenantSpec) map[string]any {
+	if spec == nil {
+		return nil
+	}
+
+	tenant := map[string]any{}
+	if spec.Header != "" {
+		tenant[keyTenantHeader] = spec.Header
+	}
+	if len(spec.ResourceAttributes) > 0 {
+		attrs := make([]any, 0, len(spec.ResourceAttributes))
+		for _, k := range spec.ResourceAttributes {
+			attrs = append(attrs, k)
+		}
+		tenant[keyTenantResourceAttr] = attrs
+	}
+	if spec.Default != "" {
+		tenant[keyTenantDefault] = spec.Default
+	}
+	// require is only meaningful alongside a header, which validateIngestTenant enforces, so an
+	// unset one is left out rather than rendered as an explicit false.
+	if spec.Require {
+		tenant[keyTenantRequire] = true
+	}
+
+	if len(tenant) == 0 {
+		return nil
+	}
+	return tenant
+}
+
+// maxTenantLen bounds a tenant id, mirroring cmd/odbingest. A tenant id becomes a shard key, which
+// becomes a backend path segment and an etcd key component.
+const maxTenantLen = 150
+
+// validateIngestTenant rejects a tenant block odbingest would refuse at startup, or that would
+// resolve to nothing. Every check mirrors cmd/odbingest's newTenantResolver and parseTenantID: a
+// pod that crash-loops on its config is a worse report than a Degraded condition naming the field.
+func validateIngestTenant(cr *dbv1alpha1.OtelDBCluster) error {
+	if !ingestEnabled(cr) {
+		return nil
+	}
+	spec := cr.Spec.Ingest.Tenant
+	if spec == nil {
+		return nil
+	}
+
+	if spec.Require && spec.Header == "" {
+		return invalidSpec("spec.ingest.tenant.require needs spec.ingest.tenant.header: " +
+			"there is no header to require")
+	}
+	for i, key := range spec.ResourceAttributes {
+		if key == "" {
+			return invalidSpec("spec.ingest.tenant.resourceAttributes[%d] must not be empty", i)
+		}
+	}
+	if spec.Default != "" {
+		if err := validateTenantID(spec.Default); err != nil {
+			return invalidSpec("spec.ingest.tenant.default: %s", err)
+		}
+	}
+
+	// A block that names no source resolves nothing: odbingest builds no resolver and routes to
+	// "default", so the stanza is present and inert.
+	if spec.Header == "" && spec.Default == "" && len(spec.ResourceAttributes) == 0 {
+		return invalidSpec("spec.ingest.tenant sets no source: give it a header, " +
+			"resourceAttributes or a default, or remove the block")
+	}
+
+	return nil
+}
+
+// validateTenantID mirrors cmd/odbingest's parseTenantID. The character set is what stays safe as a
+// backend path segment and an etcd key.
+func validateTenantID(s string) error {
+	switch {
+	case len(s) > maxTenantLen:
+		return fmt.Errorf("tenant id longer than %d bytes", maxTenantLen)
+	case s == ".", s == "..":
+		return fmt.Errorf("invalid tenant id %q", s)
+	}
+	for _, c := range []byte(s) {
+		switch {
+		case c >= 'a' && c <= 'z',
+			c >= 'A' && c <= 'Z',
+			c >= '0' && c <= '9',
+			c == '-', c == '_', c == '.':
+		default:
+			return fmt.Errorf("invalid character %q in tenant id", string(c))
+		}
+	}
+	return nil
+}