From 434a8ff762a021c6db303023bc476cf7ada9f0a4 Mon Sep 17 00:00:00 2001 From: Konstantinos Kopanidis Date: Sun, 6 Sep 2026 20:37:14 +0300 Subject: [PATCH 1/4] feat(helm): add disabled-by-default embeddings module Wire optional embeddings onto generic service templates so it can be enabled after a compatible image exists, without deleting vector data on rollback. --- charts/conduit/Chart.yaml | 2 +- charts/conduit/README.md | 62 +++++++++++++++++-- charts/conduit/README.md.gotmpl | 60 +++++++++++++++++- charts/conduit/templates/NOTES.txt | 4 +- .../templates/services/deployment.yaml | 21 ++++--- .../templates/services/service.headless.yaml | 8 +-- .../conduit/templates/services/service.yaml | 4 +- charts/conduit/values.dev.yaml | 2 + charts/conduit/values.yaml | 36 +++++++++++ 9 files changed, 175 insertions(+), 24 deletions(-) diff --git a/charts/conduit/Chart.yaml b/charts/conduit/Chart.yaml index e5aee45..60df582 100644 --- a/charts/conduit/Chart.yaml +++ b/charts/conduit/Chart.yaml @@ -15,7 +15,7 @@ type: application # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. # Versions are expected to follow Semantic Versioning (https://semver.org/) -version: 0.2.9 +version: 0.2.10 # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. Versions are not expected to diff --git a/charts/conduit/README.md b/charts/conduit/README.md index 3ed7e87..457729b 100644 --- a/charts/conduit/README.md +++ b/charts/conduit/README.md @@ -1,6 +1,6 @@ # Conduit -![Version: 0.1.0](https://img.shields.io/badge/Version-0.1.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 0.14.7](https://img.shields.io/badge/AppVersion-0.14.7-informational?style=flat-square) +![Version: 0.2.10](https://img.shields.io/badge/Version-0.2.10-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v0.16.17](https://img.shields.io/badge/AppVersion-v0.16.17-informational?style=flat-square) Conduit is a NodeJS-based Self-Hosted backend, that aims to cut down development times by providing ready-made modules that offer common functionality out of the box, and allowing maximum flexibility to add custom functionality. @@ -81,10 +81,6 @@ a) Default Mongo image. Set `.Values.mongodb.enabled` to `true`, for this option b) External Database, Mongo or Postgres. Set the `.Values.externalDatabase.url` along with its type, for this option. -### MongoDB replicas (Atlas) - -For deployments with replica sets, connector URI options (for example `readPreference` query parameters on `externalDatabase.url`) define driver-level defaults. After install, operators can tune routing via **Admin UI → Database → Settings** (`readPreference`, `writeConcern`, `readConcern`) or the admin config API; these merge with optional **per-schema** overrides on declared schemas and **per-query** overrides from modules. Defaults match the MongoDB driver (`primary` / `w: 1` / `local`) so upgrades stay non-breaking until you opt in to secondary reads. - 3. Loki (optional) a) Loki chart dependency. Set `.Values.loki.setup` to `true`, for this option. @@ -97,6 +93,60 @@ a) Prometheus chart dependency. Set `.Values.prometheus.setup` to `true`, for th b) External Prometheus. Set `.Values.externalPrometheus.url` to `true`, for this option. +## Embeddings (disabled by default) + +Embeddings is an optional module. Keep `install.embeddings.enabled` at `false` until a compatible embeddings image is published and pinned. Chart `appVersion` still tracks the last published Conduit release and does not imply that `conduitplatform/embeddings:` exists. + +### Enablement + +1. Confirm Core, Database, and grpc-sdk images are embeddings-compatible, and that an embeddings image exists at the selected tag (`install.embeddings.image.name` is `embeddings`; tag defaults to `global.image.tag`). +2. Leave `global.secret.grpc_enable` at `true` (the chart default). Production embeddings requires `GRPC_KEY` from `conduit-secret`, the same Secret used by Core, Database, Router, and other modules. +3. Enable the workload: + +```yaml +install: + embeddings: + enabled: true +``` + +4. Deploy while embeddings config workers stay disabled in module config. Kubernetes gRPC probes only check that the process is serving; they are not a substitute for capability or index readiness. + +### Capability and index readiness + +Live Atlas, pgvector, Redis, and provider checks are not part of the Helm or unit-test gate. Before activating generation or search: + +1. Confirm the embeddings pod is Ready and registered with Core. +2. Call Database `getVectorCapabilities` (or Admin `GET /embeddings/capabilities`) and confirm the backend can store and query vectors. +3. Create the embedding config with `enabled: false` if needed. Saving a disabled config may succeed with a capability warning; activation must not. +4. Create or wait for the vector index until its status is queryable. Do not enable search against a pending or failed index. + +### Staged activation + +Recommended order: + +1. Release compatible Core / Database / grpc-sdk and the embeddings image. +2. Deploy with `install.embeddings.enabled: true` and module workers still off. +3. Verify gRPC peer health and vector capabilities. +4. Configure the HTTPS provider (`allowedHosts`, API key) in embeddings module config. +5. Create the vector index and wait until it is ready. +6. Run a bounded backfill (`onlyMissing` where possible). +7. Execute a scoped canary semantic search. +8. Enable normal workers and search only after the canary succeeds. + +### Resource guidance + +Default embeddings requests are `256Mi` / `100m` with limits `1Gi` / `1000m`, gRPC `55165`, and metrics `9192`. Start with `replicas: 1`. Extra replicas multiply BullMQ workers against the shared Redis; raise CPU/memory before replica count if backfills or provider latency saturate the pod. Provider calls, index builds, and backfills are the usual memory drivers. + +### Rollback without deleting data + +To roll back: + +1. Disable embeddings workers / config activation in module config (stop generation and search). +2. Set `install.embeddings.enabled: false` or scale replicas to `0`, and roll back the embeddings image or chart revision if needed. +3. Do **not** delete vector fields, indexes, embedding configs, backfill documents, or Redis queue state as part of rollback. + +Data and index removal is a separate, explicit operator action after you have confirmed you no longer need them. + ## Custom Resource Definition By default this chart will install the Custom Resource Definition for Prometheus Service Monitors, located in crds/crd-servicemonitor.yaml. @@ -143,7 +193,7 @@ Due to the use of the lookup function the random secret values will not be re-ge ## Install object | Key | Type | Default | Description | |-----|------|---------|-------------| -| install | object | `{"authentication":{"enabled":true,"image":{"name":"authentication"},"metrics":{"enabled":true}},"chat":{"enabled":true,"image":{"name":"chat"},"metrics":{"enabled":true}},"email":{"enabled":true,"image":{"name":"email"},"metrics":{"enabled":true}},"forms":{"enabled":true,"image":{"name":"forms"},"metrics":{"enabled":true}},"notification":{"enabled":true,"image":{"name":"push-notifications"},"metrics":{"enabled":true}},"sms":{"enabled":true,"image":{"name":"sms"},"metrics":{"enabled":true}},"storage":{"enabled":true,"image":{"name":"storage"},"metrics":{"enabled":true}}}` | Choosing which microservices you want deployed (except for Admin-UI, Core, Database and Router) | +| install | object | See `values.yaml` | Choosing which microservices you want deployed (except for Admin-UI, Core, Database and Router). `install.embeddings` is disabled by default. | ## Module settings to apply to all services of the install object | Key | Type | Default | Description | diff --git a/charts/conduit/README.md.gotmpl b/charts/conduit/README.md.gotmpl index f4c7ffd..525fe83 100644 --- a/charts/conduit/README.md.gotmpl +++ b/charts/conduit/README.md.gotmpl @@ -1,6 +1,6 @@ -# Conduit +# Conduit -![Version: 0.1.0](https://img.shields.io/badge/Version-0.1.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 0.14.7](https://img.shields.io/badge/AppVersion-0.14.7-informational?style=flat-square) +{{ template "chart.versionBadge" . }} {{ template "chart.typeBadge" . }} {{ template "chart.appVersionBadge" . }} Conduit is a NodeJS-based Self-Hosted backend, that aims to cut down development times by providing ready-made modules that offer common functionality out of the box, and allowing maximum flexibility to add custom functionality. @@ -93,6 +93,60 @@ a) Prometheus chart dependency. Set `.Values.prometheus.setup` to `true`, for th b) External Prometheus. Set `.Values.externalPrometheus.url` to `true`, for this option. +## Embeddings (disabled by default) + +Embeddings is an optional module. Keep `install.embeddings.enabled` at `false` until a compatible embeddings image is published and pinned. Chart `appVersion` still tracks the last published Conduit release and does not imply that `conduitplatform/embeddings:` exists. + +### Enablement + +1. Confirm Core, Database, and grpc-sdk images are embeddings-compatible, and that an embeddings image exists at the selected tag (`install.embeddings.image.name` is `embeddings`; tag defaults to `global.image.tag`). +2. Leave `global.secret.grpc_enable` at `true` (the chart default). Production embeddings requires `GRPC_KEY` from `conduit-secret`, the same Secret used by Core, Database, Router, and other modules. +3. Enable the workload: + +```yaml +install: + embeddings: + enabled: true +``` + +4. Deploy while embeddings config workers stay disabled in module config. Kubernetes gRPC probes only check that the process is serving; they are not a substitute for capability or index readiness. + +### Capability and index readiness + +Live Atlas, pgvector, Redis, and provider checks are not part of the Helm or unit-test gate. Before activating generation or search: + +1. Confirm the embeddings pod is Ready and registered with Core. +2. Call Database `getVectorCapabilities` (or Admin `GET /embeddings/capabilities`) and confirm the backend can store and query vectors. +3. Create the embedding config with `enabled: false` if needed. Saving a disabled config may succeed with a capability warning; activation must not. +4. Create or wait for the vector index until its status is queryable. Do not enable search against a pending or failed index. + +### Staged activation + +Recommended order: + +1. Release compatible Core / Database / grpc-sdk and the embeddings image. +2. Deploy with `install.embeddings.enabled: true` and module workers still off. +3. Verify gRPC peer health and vector capabilities. +4. Configure the HTTPS provider (`allowedHosts`, API key) in embeddings module config. +5. Create the vector index and wait until it is ready. +6. Run a bounded backfill (`onlyMissing` where possible). +7. Execute a scoped canary semantic search. +8. Enable normal workers and search only after the canary succeeds. + +### Resource guidance + +Default embeddings requests are `256Mi` / `100m` with limits `1Gi` / `1000m`, gRPC `55165`, and metrics `9192`. Start with `replicas: 1`. Extra replicas multiply BullMQ workers against the shared Redis; raise CPU/memory before replica count if backfills or provider latency saturate the pod. Provider calls, index builds, and backfills are the usual memory drivers. + +### Rollback without deleting data + +To roll back: + +1. Disable embeddings workers / config activation in module config (stop generation and search). +2. Set `install.embeddings.enabled: false` or scale replicas to `0`, and roll back the embeddings image or chart revision if needed. +3. Do **not** delete vector fields, indexes, embedding configs, backfill documents, or Redis queue state as part of rollback. + +Data and index removal is a separate, explicit operator action after you have confirmed you no longer need them. + ## Custom Resource Definition By default this chart will install the Custom Resource Definition for Prometheus Service Monitors, located in crds/crd-servicemonitor.yaml. @@ -139,7 +193,7 @@ Due to the use of the lookup function the random secret values will not be re-ge ## Install object | Key | Type | Default | Description | |-----|------|---------|-------------| -| install | object | `{"authentication":{"enabled":true,"image":{"name":"authentication"},"metrics":{"enabled":true}},"chat":{"enabled":true,"image":{"name":"chat"},"metrics":{"enabled":true}},"email":{"enabled":true,"image":{"name":"email"},"metrics":{"enabled":true}},"forms":{"enabled":true,"image":{"name":"forms"},"metrics":{"enabled":true}},"notification":{"enabled":true,"image":{"name":"push-notifications"},"metrics":{"enabled":true}},"sms":{"enabled":true,"image":{"name":"sms"},"metrics":{"enabled":true}},"storage":{"enabled":true,"image":{"name":"storage"},"metrics":{"enabled":true}}}` | Choosing which microservices you want deployed (except for Admin-UI, Core, Database and Router) | +| install | object | See `values.yaml` | Choosing which microservices you want deployed (except for Admin-UI, Core, Database and Router). `install.embeddings` is disabled by default. | ## Module settings to apply to all services of the install object | Key | Type | Default | Description | diff --git a/charts/conduit/templates/NOTES.txt b/charts/conduit/templates/NOTES.txt index 5f92ba4..dd3e109 100644 --- a/charts/conduit/templates/NOTES.txt +++ b/charts/conduit/templates/NOTES.txt @@ -27,4 +27,6 @@ b) External Loki. Set .Values.externalLoki.url to true, for this option. 4. Prometheus Metrics (optional) a) Prometheus chart dependency. Set .Values.prometheus.setup to true, for this option. -b) External Prometheus. Set .Values.externalPrometheus.url to true, for this option. \ No newline at end of file +b) External Prometheus. Set .Values.externalPrometheus.url to true, for this option. + +Embeddings is disabled by default (install.embeddings.enabled). Do not enable it until a published embeddings image exists at the selected tag. Production requires GRPC_KEY (global.secret.grpc_enable, default true). Rolling back the workload must not delete vector fields, indexes, configs, or Redis state. \ No newline at end of file diff --git a/charts/conduit/templates/services/deployment.yaml b/charts/conduit/templates/services/deployment.yaml index 5672c12..af78a61 100644 --- a/charts/conduit/templates/services/deployment.yaml +++ b/charts/conduit/templates/services/deployment.yaml @@ -1,6 +1,8 @@ {{ $moduleSettings := index .Values "module-settings" }} {{- range $module, $spec := .Values.install -}} {{- $specService := $spec.service -}} +{{- $specMetrics := default (dict) $spec.metrics -}} +{{- $specMetricsService := default (dict) $specMetrics.service -}} {{- if $spec.enabled }} --- apiVersion: apps/v1 @@ -73,14 +75,15 @@ spec: securityContext: {{- toYaml $.Values.securityContext | nindent 12 }} resources: - {{- if $moduleSettings.resources }} - {{- toYaml $moduleSettings.resources | nindent 12 }} - {{- end }} - {{- if $spec.resources }} - {{- toYaml $spec.resources | nindent 12 }} - {{- end }} + {{- $moduleResources := default (dict) $moduleSettings.resources }} + {{- $specResources := default (dict) $spec.resources }} + {{- toYaml (mergeOverwrite (deepCopy $moduleResources) $specResources) | nindent 12 }} ports: - containerPort: {{ coalesce $spec.containerPort $moduleSettings.containerPort $.Values.global.config.grpc_port }} + {{- if $specMetrics.enabled }} + - name: metrics + containerPort: {{ coalesce $specMetricsService.targetPort $specMetricsService.port $moduleSettings.metrics.service.targetPort $.Values.global.config.metrics_port }} + {{- end }} {{- $specReadiness := default (dict) $spec.readinessProbe }} {{- $moduleReadiness := default (dict) $moduleSettings.readinessProbe }} {{- $specLiveness := default (dict) $spec.livenessProbe }} @@ -123,7 +126,11 @@ spec: key: GRPC_KEY {{- end }} - name: GRPC_PORT - value: {{ coalesce $specService.containerPort $moduleSettings.containerPort $.Values.global.config.grpc_port | quote }} + value: {{ coalesce $spec.containerPort $specService.grpc_port $specService.containerPort $moduleSettings.containerPort $.Values.global.config.grpc_port | quote }} + {{- if $specMetrics.enabled }} + - name: METRICS_PORT + value: {{ coalesce $specMetricsService.targetPort $specMetricsService.port $moduleSettings.metrics.service.targetPort $.Values.global.config.metrics_port | quote }} + {{- end }} {{- if $moduleSettings.env }} {{- toYaml $moduleSettings.env | nindent 12 }} {{- end }} diff --git a/charts/conduit/templates/services/service.headless.yaml b/charts/conduit/templates/services/service.headless.yaml index 71a0e0c..fa8b2a5 100644 --- a/charts/conduit/templates/services/service.headless.yaml +++ b/charts/conduit/templates/services/service.headless.yaml @@ -37,13 +37,13 @@ spec: app: {{ template "conduit-helm.fullname" $ }}-{{ $module }} ports: - protocol: TCP - port: {{ coalesce $specService.grpc_port $moduleSettings.containerPort $.Values.global.config.grpc_port }} + port: {{ coalesce $specService.grpc_port $spec.containerPort $moduleSettings.containerPort $.Values.global.config.grpc_port }} name: tcp - targetPort: {{ coalesce $specService.grpc_port $moduleSettings.containerPort $.Values.global.config.grpc_port }} + targetPort: {{ coalesce $specService.grpc_port $spec.containerPort $moduleSettings.containerPort $.Values.global.config.grpc_port }} {{- if $spec.metrics.enabled }} - - port: {{ $.Values.global.config.metrics_port }} + - port: {{ coalesce $specMetricsService.targetPort $specMetricsService.port $moduleSettings.metrics.service.targetPort $.Values.global.config.metrics_port }} name: {{ default $moduleSettings.metrics.service.name $specMetricsService.name }} - targetPort: {{ $.Values.global.config.metrics_port }} + targetPort: {{ coalesce $specMetricsService.targetPort $specMetricsService.port $moduleSettings.metrics.service.targetPort $.Values.global.config.metrics_port }} {{- end }} {{- end }} {{- end }} diff --git a/charts/conduit/templates/services/service.yaml b/charts/conduit/templates/services/service.yaml index 36558b4..c37f5c1 100644 --- a/charts/conduit/templates/services/service.yaml +++ b/charts/conduit/templates/services/service.yaml @@ -47,11 +47,11 @@ spec: - protocol: TCP port: {{ default $moduleSettings.service.tcp_port $specService.tcp_port }} name: tcp - targetPort: {{ coalesce $specService.grpc_port $moduleSettings.containerPort $.Values.global.config.grpc_port }} + targetPort: {{ coalesce $specService.grpc_port $spec.containerPort $moduleSettings.containerPort $.Values.global.config.grpc_port }} {{- if $spec.metrics.enabled }} - port: {{ default $moduleSettings.metrics.service.port $specMetricsService.port }} name: {{ default $moduleSettings.metrics.service.name $specMetricsService.name }} - targetPort: {{ $.Values.global.config.metrics_port }} + targetPort: {{ coalesce $specMetricsService.targetPort $specMetricsService.port $moduleSettings.metrics.service.targetPort $.Values.global.config.metrics_port }} {{- end }} type: ClusterIP {{- end }} diff --git a/charts/conduit/values.dev.yaml b/charts/conduit/values.dev.yaml index 643a3c6..1ed321e 100644 --- a/charts/conduit/values.dev.yaml +++ b/charts/conduit/values.dev.yaml @@ -27,6 +27,8 @@ install: enabled: false storage: enabled: false + embeddings: + enabled: false module-settings: # -- Resource limits and requests resources: diff --git a/charts/conduit/values.yaml b/charts/conduit/values.yaml index 6766967..78b8cbf 100644 --- a/charts/conduit/values.yaml +++ b/charts/conduit/values.yaml @@ -140,6 +140,42 @@ install: name: storage metrics: enabled: true + # -- Embeddings module. Disabled by default until a published embeddings image tag exists. + embeddings: + enabled: false + image: + name: embeddings + # -- gRPC listen port (compose-aligned). Also used for SERVICE_URL and probes. + containerPort: 55165 + # -- Keep a single replica unless BullMQ worker concurrency is tuned for horizontal scale. + replicas: 1 + resources: + requests: + memory: "256Mi" + cpu: "100m" + limits: + memory: "1Gi" + cpu: "1000m" + readinessProbe: + initialDelaySeconds: 15 + periodSeconds: 5 + timeoutSeconds: 1 + successThreshold: 1 + failureThreshold: 6 + livenessProbe: + initialDelaySeconds: 30 + periodSeconds: 10 + timeoutSeconds: 1 + successThreshold: 1 + failureThreshold: 5 + service: + tcp_port: 55165 + grpc_port: 55165 + metrics: + enabled: true + service: + port: 9192 + targetPort: 9192 # -- Default module settings for the microservices (except Admin-UI, Core, Database, Router) module-settings: From 7453410b7d174ec998e6fd28c517094998fcb5d9 Mon Sep 17 00:00:00 2001 From: Konstantinos Kopanidis Date: Sun, 6 Sep 2026 20:45:02 +0300 Subject: [PATCH 2/4] chore(helm): trim embeddings chart slop Remove redundant embeddings values/comments, defensive port coalesces, and the long operator runbook so the module matches surrounding chart conventions without changing rendered behavior. --- charts/conduit/README.md | 52 +------------------ charts/conduit/README.md.gotmpl | 52 +------------------ charts/conduit/templates/NOTES.txt | 4 +- .../templates/services/deployment.yaml | 10 ++-- .../templates/services/service.headless.yaml | 4 +- .../conduit/templates/services/service.yaml | 2 +- charts/conduit/values.yaml | 3 -- 7 files changed, 13 insertions(+), 114 deletions(-) diff --git a/charts/conduit/README.md b/charts/conduit/README.md index 457729b..727d08e 100644 --- a/charts/conduit/README.md +++ b/charts/conduit/README.md @@ -95,57 +95,7 @@ b) External Prometheus. Set `.Values.externalPrometheus.url` to `true`, for this ## Embeddings (disabled by default) -Embeddings is an optional module. Keep `install.embeddings.enabled` at `false` until a compatible embeddings image is published and pinned. Chart `appVersion` still tracks the last published Conduit release and does not imply that `conduitplatform/embeddings:` exists. - -### Enablement - -1. Confirm Core, Database, and grpc-sdk images are embeddings-compatible, and that an embeddings image exists at the selected tag (`install.embeddings.image.name` is `embeddings`; tag defaults to `global.image.tag`). -2. Leave `global.secret.grpc_enable` at `true` (the chart default). Production embeddings requires `GRPC_KEY` from `conduit-secret`, the same Secret used by Core, Database, Router, and other modules. -3. Enable the workload: - -```yaml -install: - embeddings: - enabled: true -``` - -4. Deploy while embeddings config workers stay disabled in module config. Kubernetes gRPC probes only check that the process is serving; they are not a substitute for capability or index readiness. - -### Capability and index readiness - -Live Atlas, pgvector, Redis, and provider checks are not part of the Helm or unit-test gate. Before activating generation or search: - -1. Confirm the embeddings pod is Ready and registered with Core. -2. Call Database `getVectorCapabilities` (or Admin `GET /embeddings/capabilities`) and confirm the backend can store and query vectors. -3. Create the embedding config with `enabled: false` if needed. Saving a disabled config may succeed with a capability warning; activation must not. -4. Create or wait for the vector index until its status is queryable. Do not enable search against a pending or failed index. - -### Staged activation - -Recommended order: - -1. Release compatible Core / Database / grpc-sdk and the embeddings image. -2. Deploy with `install.embeddings.enabled: true` and module workers still off. -3. Verify gRPC peer health and vector capabilities. -4. Configure the HTTPS provider (`allowedHosts`, API key) in embeddings module config. -5. Create the vector index and wait until it is ready. -6. Run a bounded backfill (`onlyMissing` where possible). -7. Execute a scoped canary semantic search. -8. Enable normal workers and search only after the canary succeeds. - -### Resource guidance - -Default embeddings requests are `256Mi` / `100m` with limits `1Gi` / `1000m`, gRPC `55165`, and metrics `9192`. Start with `replicas: 1`. Extra replicas multiply BullMQ workers against the shared Redis; raise CPU/memory before replica count if backfills or provider latency saturate the pod. Provider calls, index builds, and backfills are the usual memory drivers. - -### Rollback without deleting data - -To roll back: - -1. Disable embeddings workers / config activation in module config (stop generation and search). -2. Set `install.embeddings.enabled: false` or scale replicas to `0`, and roll back the embeddings image or chart revision if needed. -3. Do **not** delete vector fields, indexes, embedding configs, backfill documents, or Redis queue state as part of rollback. - -Data and index removal is a separate, explicit operator action after you have confirmed you no longer need them. +Keep `install.embeddings.enabled` at `false` until a published embeddings image exists at the selected tag. Chart `appVersion` does not imply that `conduitplatform/embeddings:` exists. Production requires `GRPC_KEY` (`global.secret.grpc_enable`, default `true`). Rolling back the workload must not delete vector fields, indexes, configs, or Redis state. ## Custom Resource Definition diff --git a/charts/conduit/README.md.gotmpl b/charts/conduit/README.md.gotmpl index 525fe83..3278b66 100644 --- a/charts/conduit/README.md.gotmpl +++ b/charts/conduit/README.md.gotmpl @@ -95,57 +95,7 @@ b) External Prometheus. Set `.Values.externalPrometheus.url` to `true`, for this ## Embeddings (disabled by default) -Embeddings is an optional module. Keep `install.embeddings.enabled` at `false` until a compatible embeddings image is published and pinned. Chart `appVersion` still tracks the last published Conduit release and does not imply that `conduitplatform/embeddings:` exists. - -### Enablement - -1. Confirm Core, Database, and grpc-sdk images are embeddings-compatible, and that an embeddings image exists at the selected tag (`install.embeddings.image.name` is `embeddings`; tag defaults to `global.image.tag`). -2. Leave `global.secret.grpc_enable` at `true` (the chart default). Production embeddings requires `GRPC_KEY` from `conduit-secret`, the same Secret used by Core, Database, Router, and other modules. -3. Enable the workload: - -```yaml -install: - embeddings: - enabled: true -``` - -4. Deploy while embeddings config workers stay disabled in module config. Kubernetes gRPC probes only check that the process is serving; they are not a substitute for capability or index readiness. - -### Capability and index readiness - -Live Atlas, pgvector, Redis, and provider checks are not part of the Helm or unit-test gate. Before activating generation or search: - -1. Confirm the embeddings pod is Ready and registered with Core. -2. Call Database `getVectorCapabilities` (or Admin `GET /embeddings/capabilities`) and confirm the backend can store and query vectors. -3. Create the embedding config with `enabled: false` if needed. Saving a disabled config may succeed with a capability warning; activation must not. -4. Create or wait for the vector index until its status is queryable. Do not enable search against a pending or failed index. - -### Staged activation - -Recommended order: - -1. Release compatible Core / Database / grpc-sdk and the embeddings image. -2. Deploy with `install.embeddings.enabled: true` and module workers still off. -3. Verify gRPC peer health and vector capabilities. -4. Configure the HTTPS provider (`allowedHosts`, API key) in embeddings module config. -5. Create the vector index and wait until it is ready. -6. Run a bounded backfill (`onlyMissing` where possible). -7. Execute a scoped canary semantic search. -8. Enable normal workers and search only after the canary succeeds. - -### Resource guidance - -Default embeddings requests are `256Mi` / `100m` with limits `1Gi` / `1000m`, gRPC `55165`, and metrics `9192`. Start with `replicas: 1`. Extra replicas multiply BullMQ workers against the shared Redis; raise CPU/memory before replica count if backfills or provider latency saturate the pod. Provider calls, index builds, and backfills are the usual memory drivers. - -### Rollback without deleting data - -To roll back: - -1. Disable embeddings workers / config activation in module config (stop generation and search). -2. Set `install.embeddings.enabled: false` or scale replicas to `0`, and roll back the embeddings image or chart revision if needed. -3. Do **not** delete vector fields, indexes, embedding configs, backfill documents, or Redis queue state as part of rollback. - -Data and index removal is a separate, explicit operator action after you have confirmed you no longer need them. +Keep `install.embeddings.enabled` at `false` until a published embeddings image exists at the selected tag. Chart `appVersion` does not imply that `conduitplatform/embeddings:` exists. Production requires `GRPC_KEY` (`global.secret.grpc_enable`, default `true`). Rolling back the workload must not delete vector fields, indexes, configs, or Redis state. ## Custom Resource Definition diff --git a/charts/conduit/templates/NOTES.txt b/charts/conduit/templates/NOTES.txt index dd3e109..4c35417 100644 --- a/charts/conduit/templates/NOTES.txt +++ b/charts/conduit/templates/NOTES.txt @@ -29,4 +29,6 @@ b) External Loki. Set .Values.externalLoki.url to true, for this option. a) Prometheus chart dependency. Set .Values.prometheus.setup to true, for this option. b) External Prometheus. Set .Values.externalPrometheus.url to true, for this option. -Embeddings is disabled by default (install.embeddings.enabled). Do not enable it until a published embeddings image exists at the selected tag. Production requires GRPC_KEY (global.secret.grpc_enable, default true). Rolling back the workload must not delete vector fields, indexes, configs, or Redis state. \ No newline at end of file +5. Embeddings (optional, disabled by default) + +Set .Values.install.embeddings.enabled to true only after a published embeddings image exists at the selected tag. Production requires GRPC_KEY (global.secret.grpc_enable). Do not delete vector fields, indexes, configs, or Redis state when rolling back the workload. \ No newline at end of file diff --git a/charts/conduit/templates/services/deployment.yaml b/charts/conduit/templates/services/deployment.yaml index af78a61..17726fe 100644 --- a/charts/conduit/templates/services/deployment.yaml +++ b/charts/conduit/templates/services/deployment.yaml @@ -1,8 +1,8 @@ {{ $moduleSettings := index .Values "module-settings" }} {{- range $module, $spec := .Values.install -}} {{- $specService := $spec.service -}} -{{- $specMetrics := default (dict) $spec.metrics -}} -{{- $specMetricsService := default (dict) $specMetrics.service -}} +{{- $specMetrics := $spec.metrics -}} +{{- $specMetricsService := $spec.metrics.service -}} {{- if $spec.enabled }} --- apiVersion: apps/v1 @@ -82,7 +82,7 @@ spec: - containerPort: {{ coalesce $spec.containerPort $moduleSettings.containerPort $.Values.global.config.grpc_port }} {{- if $specMetrics.enabled }} - name: metrics - containerPort: {{ coalesce $specMetricsService.targetPort $specMetricsService.port $moduleSettings.metrics.service.targetPort $.Values.global.config.metrics_port }} + containerPort: {{ default $moduleSettings.metrics.service.targetPort $specMetricsService.targetPort }} {{- end }} {{- $specReadiness := default (dict) $spec.readinessProbe }} {{- $moduleReadiness := default (dict) $moduleSettings.readinessProbe }} @@ -126,10 +126,10 @@ spec: key: GRPC_KEY {{- end }} - name: GRPC_PORT - value: {{ coalesce $spec.containerPort $specService.grpc_port $specService.containerPort $moduleSettings.containerPort $.Values.global.config.grpc_port | quote }} + value: {{ coalesce $spec.containerPort $moduleSettings.containerPort $.Values.global.config.grpc_port | quote }} {{- if $specMetrics.enabled }} - name: METRICS_PORT - value: {{ coalesce $specMetricsService.targetPort $specMetricsService.port $moduleSettings.metrics.service.targetPort $.Values.global.config.metrics_port | quote }} + value: {{ default $moduleSettings.metrics.service.targetPort $specMetricsService.targetPort | quote }} {{- end }} {{- if $moduleSettings.env }} {{- toYaml $moduleSettings.env | nindent 12 }} diff --git a/charts/conduit/templates/services/service.headless.yaml b/charts/conduit/templates/services/service.headless.yaml index fa8b2a5..74d4f49 100644 --- a/charts/conduit/templates/services/service.headless.yaml +++ b/charts/conduit/templates/services/service.headless.yaml @@ -41,9 +41,9 @@ spec: name: tcp targetPort: {{ coalesce $specService.grpc_port $spec.containerPort $moduleSettings.containerPort $.Values.global.config.grpc_port }} {{- if $spec.metrics.enabled }} - - port: {{ coalesce $specMetricsService.targetPort $specMetricsService.port $moduleSettings.metrics.service.targetPort $.Values.global.config.metrics_port }} + - port: {{ default $moduleSettings.metrics.service.port $specMetricsService.port }} name: {{ default $moduleSettings.metrics.service.name $specMetricsService.name }} - targetPort: {{ coalesce $specMetricsService.targetPort $specMetricsService.port $moduleSettings.metrics.service.targetPort $.Values.global.config.metrics_port }} + targetPort: {{ default $moduleSettings.metrics.service.targetPort $specMetricsService.targetPort }} {{- end }} {{- end }} {{- end }} diff --git a/charts/conduit/templates/services/service.yaml b/charts/conduit/templates/services/service.yaml index c37f5c1..83d46ac 100644 --- a/charts/conduit/templates/services/service.yaml +++ b/charts/conduit/templates/services/service.yaml @@ -51,7 +51,7 @@ spec: {{- if $spec.metrics.enabled }} - port: {{ default $moduleSettings.metrics.service.port $specMetricsService.port }} name: {{ default $moduleSettings.metrics.service.name $specMetricsService.name }} - targetPort: {{ coalesce $specMetricsService.targetPort $specMetricsService.port $moduleSettings.metrics.service.targetPort $.Values.global.config.metrics_port }} + targetPort: {{ default $moduleSettings.metrics.service.targetPort $specMetricsService.targetPort }} {{- end }} type: ClusterIP {{- end }} diff --git a/charts/conduit/values.yaml b/charts/conduit/values.yaml index 78b8cbf..31352b1 100644 --- a/charts/conduit/values.yaml +++ b/charts/conduit/values.yaml @@ -145,10 +145,7 @@ install: enabled: false image: name: embeddings - # -- gRPC listen port (compose-aligned). Also used for SERVICE_URL and probes. containerPort: 55165 - # -- Keep a single replica unless BullMQ worker concurrency is tuned for horizontal scale. - replicas: 1 resources: requests: memory: "256Mi" From 210213021996a62f28d9569ccf9484fba1c9a9db Mon Sep 17 00:00:00 2001 From: Konstantinos Kopanidis Date: Sun, 6 Sep 2026 21:34:51 +0300 Subject: [PATCH 3/4] fix(helm): gate embeddings on GRPC_KEY Reject embeddings when gRPC secrets are disabled, align default tag validation with Chart.AppVersion, and document workload versus module-config enablement plus rollback that retains vector state. --- charts/conduit/README.md | 8 +++++++- charts/conduit/README.md.gotmpl | 8 +++++++- charts/conduit/templates/NOTES.txt | 6 +++++- charts/conduit/templates/_helpers.tpl | 18 +++++++++++++++--- .../conduit/templates/validate-image-tag.yaml | 1 + charts/conduit/values.yaml | 2 +- 6 files changed, 36 insertions(+), 7 deletions(-) diff --git a/charts/conduit/README.md b/charts/conduit/README.md index 727d08e..e6bb9b3 100644 --- a/charts/conduit/README.md +++ b/charts/conduit/README.md @@ -95,7 +95,13 @@ b) External Prometheus. Set `.Values.externalPrometheus.url` to `true`, for this ## Embeddings (disabled by default) -Keep `install.embeddings.enabled` at `false` until a published embeddings image exists at the selected tag. Chart `appVersion` does not imply that `conduitplatform/embeddings:` exists. Production requires `GRPC_KEY` (`global.secret.grpc_enable`, default `true`). Rolling back the workload must not delete vector fields, indexes, configs, or Redis state. +`install.embeddings.enabled` deploys the embeddings Kubernetes workload only. It is not the embeddings module-config `enabled` flag that starts generation or search workers. + +Keep the workload at `false` until `conduitplatform/embeddings:` exists at the image tag you will run (`install.embeddings.image.tag`, defaulting to `global.image.tag`). Chart `appVersion` and the default global tag track the last published Conduit release and do **not** imply that an embeddings image exists at that tag. When you enable the workload, set the tag to the first published compatible embeddings release. + +Production requires `GRPC_KEY`. Leave `global.secret.grpc_enable` at `true` (the chart default). The chart rejects `install.embeddings.enabled=true` when gRPC secrets are disabled. + +Staged rollout: deploy with `install.embeddings.enabled: true` while module-config workers stay disabled, then enable workers only after the pod is running. Rollback the workload with `install.embeddings.enabled: false`. That removes the embeddings Deployment/Service only; vector fields, indexes, embedding configs, and Redis queue state are retained. ## Custom Resource Definition diff --git a/charts/conduit/README.md.gotmpl b/charts/conduit/README.md.gotmpl index 3278b66..294293b 100644 --- a/charts/conduit/README.md.gotmpl +++ b/charts/conduit/README.md.gotmpl @@ -95,7 +95,13 @@ b) External Prometheus. Set `.Values.externalPrometheus.url` to `true`, for this ## Embeddings (disabled by default) -Keep `install.embeddings.enabled` at `false` until a published embeddings image exists at the selected tag. Chart `appVersion` does not imply that `conduitplatform/embeddings:` exists. Production requires `GRPC_KEY` (`global.secret.grpc_enable`, default `true`). Rolling back the workload must not delete vector fields, indexes, configs, or Redis state. +`install.embeddings.enabled` deploys the embeddings Kubernetes workload only. It is not the embeddings module-config `enabled` flag that starts generation or search workers. + +Keep the workload at `false` until `conduitplatform/embeddings:` exists at the image tag you will run (`install.embeddings.image.tag`, defaulting to `global.image.tag`). Chart `appVersion` and the default global tag track the last published Conduit release and do **not** imply that an embeddings image exists at that tag. When you enable the workload, set the tag to the first published compatible embeddings release. + +Production requires `GRPC_KEY`. Leave `global.secret.grpc_enable` at `true` (the chart default). The chart rejects `install.embeddings.enabled=true` when gRPC secrets are disabled. + +Staged rollout: deploy with `install.embeddings.enabled: true` while module-config workers stay disabled, then enable workers only after the pod is running. Rollback the workload with `install.embeddings.enabled: false`. That removes the embeddings Deployment/Service only; vector fields, indexes, embedding configs, and Redis queue state are retained. ## Custom Resource Definition diff --git a/charts/conduit/templates/NOTES.txt b/charts/conduit/templates/NOTES.txt index 4c35417..26dc35b 100644 --- a/charts/conduit/templates/NOTES.txt +++ b/charts/conduit/templates/NOTES.txt @@ -31,4 +31,8 @@ b) External Prometheus. Set .Values.externalPrometheus.url to true, for this opt 5. Embeddings (optional, disabled by default) -Set .Values.install.embeddings.enabled to true only after a published embeddings image exists at the selected tag. Production requires GRPC_KEY (global.secret.grpc_enable). Do not delete vector fields, indexes, configs, or Redis state when rolling back the workload. \ No newline at end of file +install.embeddings.enabled deploys the Kubernetes workload only; it is not the embeddings module-config enabled flag that starts generation/search workers. + +Set install.embeddings.enabled=true only after conduitplatform/embeddings exists at the chosen image tag (install.embeddings.image.tag, defaulting to global.image.tag). Chart appVersion does not imply that image exists; when enabling, select the first published compatible tag. + +Requires global.secret.grpc_enable=true (GRPC_KEY). Roll out with module-config workers still disabled. Rollback with install.embeddings.enabled=false; vector fields, indexes, embedding configs, and Redis state are retained. \ No newline at end of file diff --git a/charts/conduit/templates/_helpers.tpl b/charts/conduit/templates/_helpers.tpl index 8349990..88e4545 100644 --- a/charts/conduit/templates/_helpers.tpl +++ b/charts/conduit/templates/_helpers.tpl @@ -233,11 +233,23 @@ app.kubernetes.io/instance: {{ .Release.Name }} {{- end }} {{/* -Validate global image tag version (must be 'latest' or >= v0.16.18) */}} +Validate global image tag version (must be 'latest', 'dev', 'next', or >= Chart.AppVersion) +*/}} {{- define "conduit-helm.validateImageTag" -}} {{- $tag := default "" .Values.global.image.tag -}} -{{- if and $tag (not (eq $tag "latest")) (not (eq $tag "dev")) (not (eq $tag "next")) (semverCompare " Date: Mon, 7 Sep 2026 10:36:33 +0300 Subject: [PATCH 4/4] chore(helm): tighten embeddings operator docs Drop leftover tutorial phrasing from embeddings README, NOTES, and values comments without changing chart behavior. --- charts/conduit/README.md | 6 +++--- charts/conduit/README.md.gotmpl | 6 +++--- charts/conduit/templates/NOTES.txt | 2 +- charts/conduit/values.yaml | 2 +- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/charts/conduit/README.md b/charts/conduit/README.md index e6bb9b3..fe78fac 100644 --- a/charts/conduit/README.md +++ b/charts/conduit/README.md @@ -97,11 +97,11 @@ b) External Prometheus. Set `.Values.externalPrometheus.url` to `true`, for this `install.embeddings.enabled` deploys the embeddings Kubernetes workload only. It is not the embeddings module-config `enabled` flag that starts generation or search workers. -Keep the workload at `false` until `conduitplatform/embeddings:` exists at the image tag you will run (`install.embeddings.image.tag`, defaulting to `global.image.tag`). Chart `appVersion` and the default global tag track the last published Conduit release and do **not** imply that an embeddings image exists at that tag. When you enable the workload, set the tag to the first published compatible embeddings release. +Set `install.embeddings.enabled=true` only after `conduitplatform/embeddings` exists at `install.embeddings.image.tag` (defaults to `global.image.tag`). Chart `appVersion` does not mean that image exists. -Production requires `GRPC_KEY`. Leave `global.secret.grpc_enable` at `true` (the chart default). The chart rejects `install.embeddings.enabled=true` when gRPC secrets are disabled. +Requires `global.secret.grpc_enable=true` (`GRPC_KEY`). The chart rejects `install.embeddings.enabled=true` when gRPC secrets are disabled. -Staged rollout: deploy with `install.embeddings.enabled: true` while module-config workers stay disabled, then enable workers only after the pod is running. Rollback the workload with `install.embeddings.enabled: false`. That removes the embeddings Deployment/Service only; vector fields, indexes, embedding configs, and Redis queue state are retained. +Deploy the workload with module-config workers still disabled, then enable workers after the pod is running. `install.embeddings.enabled: false` removes the Deployment/Service; vector fields, indexes, embedding configs, and Redis queue state are retained. ## Custom Resource Definition diff --git a/charts/conduit/README.md.gotmpl b/charts/conduit/README.md.gotmpl index 294293b..168dd3d 100644 --- a/charts/conduit/README.md.gotmpl +++ b/charts/conduit/README.md.gotmpl @@ -97,11 +97,11 @@ b) External Prometheus. Set `.Values.externalPrometheus.url` to `true`, for this `install.embeddings.enabled` deploys the embeddings Kubernetes workload only. It is not the embeddings module-config `enabled` flag that starts generation or search workers. -Keep the workload at `false` until `conduitplatform/embeddings:` exists at the image tag you will run (`install.embeddings.image.tag`, defaulting to `global.image.tag`). Chart `appVersion` and the default global tag track the last published Conduit release and do **not** imply that an embeddings image exists at that tag. When you enable the workload, set the tag to the first published compatible embeddings release. +Set `install.embeddings.enabled=true` only after `conduitplatform/embeddings` exists at `install.embeddings.image.tag` (defaults to `global.image.tag`). Chart `appVersion` does not mean that image exists. -Production requires `GRPC_KEY`. Leave `global.secret.grpc_enable` at `true` (the chart default). The chart rejects `install.embeddings.enabled=true` when gRPC secrets are disabled. +Requires `global.secret.grpc_enable=true` (`GRPC_KEY`). The chart rejects `install.embeddings.enabled=true` when gRPC secrets are disabled. -Staged rollout: deploy with `install.embeddings.enabled: true` while module-config workers stay disabled, then enable workers only after the pod is running. Rollback the workload with `install.embeddings.enabled: false`. That removes the embeddings Deployment/Service only; vector fields, indexes, embedding configs, and Redis queue state are retained. +Deploy the workload with module-config workers still disabled, then enable workers after the pod is running. `install.embeddings.enabled: false` removes the Deployment/Service; vector fields, indexes, embedding configs, and Redis queue state are retained. ## Custom Resource Definition diff --git a/charts/conduit/templates/NOTES.txt b/charts/conduit/templates/NOTES.txt index 26dc35b..e4fb7be 100644 --- a/charts/conduit/templates/NOTES.txt +++ b/charts/conduit/templates/NOTES.txt @@ -33,6 +33,6 @@ b) External Prometheus. Set .Values.externalPrometheus.url to true, for this opt install.embeddings.enabled deploys the Kubernetes workload only; it is not the embeddings module-config enabled flag that starts generation/search workers. -Set install.embeddings.enabled=true only after conduitplatform/embeddings exists at the chosen image tag (install.embeddings.image.tag, defaulting to global.image.tag). Chart appVersion does not imply that image exists; when enabling, select the first published compatible tag. +Set install.embeddings.enabled=true only after conduitplatform/embeddings exists at the chosen image tag (install.embeddings.image.tag, defaulting to global.image.tag). Chart appVersion does not mean that image exists. Requires global.secret.grpc_enable=true (GRPC_KEY). Roll out with module-config workers still disabled. Rollback with install.embeddings.enabled=false; vector fields, indexes, embedding configs, and Redis state are retained. \ No newline at end of file diff --git a/charts/conduit/values.yaml b/charts/conduit/values.yaml index ef59c19..07f664e 100644 --- a/charts/conduit/values.yaml +++ b/charts/conduit/values.yaml @@ -140,7 +140,7 @@ install: name: storage metrics: enabled: true - # -- Embeddings workload (not module-config workers). Disabled by default; enable only after a published embeddings image exists at the selected tag. Requires global.secret.grpc_enable=true. + # -- Embeddings Kubernetes workload, not module-config workers. Disabled by default. Requires global.secret.grpc_enable=true. embeddings: enabled: false image: