Skip to content

[Fix] TimescaleDB connection exhaustion by model monitoring - #305

Merged
royischoss merged 1 commit into
mlrun:developmentfrom
alxtkr77:ML-12857-timescaledb-max-connections
Jul 22, 2026
Merged

[Fix] TimescaleDB connection exhaustion by model monitoring#305
royischoss merged 1 commit into
mlrun:developmentfrom
alxtkr77:ML-12857-timescaledb-max-connections

Conversation

@alxtkr77

Copy link
Copy Markdown
Member

📝 Description

CE ships TimescaleDB with max_connections=25 — the value timescaledb-tune derives
at first boot for a 1Gi pod. Model monitoring holds ~21 standing connections per
monitored project (controller workers, writer/stream storey targets), so a single
busy project exhausts the server and queries fail with
FATAL: sorry, too many clients already.

The limit is now pinned explicitly via a server argument. This is deliberate:
timescaledb-tune only runs on an empty data directory, and the PVC is kept across
upgrades (helm.sh/resource-policy: keep) — a tune-side or memory-driven change
would never reach existing installs, while a -c argument overrides
postgresql.conf on every pod start.


🛠️ Changes Made

  • New timescaledb.maxConnections value (default 100, ~4 concurrent monitored projects)
  • Pass postgres -c max_connections=... as container args in the TimescaleDB StatefulSet
  • Raise TimescaleDB memory limit 1Gi → 2Gi (~3MB per open backend; keeps fresh-install
    headroom after timescaledb-tune sizes shared_buffers for the larger pod)
  • Chart version 0.12.0-rc.5 → 0.12.0-rc.6

✅ Checklist

  • I have tested the changes in this PR
  • I confirmed whether my changes require a change in documentation and if so, I created another PR in MLRun for the relevant documentation.
  • I confirmed whether my changes require a changes in QA tests, for example: credentials changes, resources naming change and if so, I updated the relevant Jira ticket for QA.
  • I increased the Chart version in charts/mlrun-ce/Chart.yaml.
  • I confirmed that the installation works both on a local Docker Desktop environment and on a real cluster when using the required prerequisites.
    • If installation issues were found, I updated the relevant Jira ticket with the issue and steps to reproduce, or updated the prerequisites documentation if the issue is related to missing or outdated prerequisites.
  • If needed, update https://github.com/mlrun/ce/blob/development/charts/mlrun-ce/README.md with the relevant installation instructions and version Matrix.
  • If needed, update the following values files for multi namespace support:

🧪 Testing

  • helm template render verified: args: [postgres, -c, max_connections=100] on the
    TimescaleDB container
  • Image config (timescale/timescaledb-ha:pg17.7-ts2.24.0) verified:
    Entrypoint: docker-entrypoint.sh, Cmd: [postgres] — the explicit args preserve the
    default startup path
  • Cluster install pending

Note: the multi-namespace values files need no sync — they only override
timescaledb.enabled / service type.


🔗 References


🚨 Breaking Changes?

  • Yes (explain below)
  • No

🔍️ Additional Notes

Existing installs pick up the new limit on the next pod restart (rollout on upgrade).
SDK-side hardening (shared pool per DSN, min_size=0) is a separate follow-up in
mlrun/mlrun — it reduces mlrun's connection footprint but would not have prevented
this failure mode on its own.

Set max_connections=100 as a server arg, overriding the too-low
timescaledb-tune default, and raise the pod memory limit to 2Gi
accordingly.
@github-actions github-actions Bot added the bug Something isn't working label Jul 22, 2026

@royischoss royischoss left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@royischoss
royischoss merged commit e4aa07a into mlrun:development Jul 22, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants