[Fix] TimescaleDB connection exhaustion by model monitoring - #305
Merged
royischoss merged 1 commit intoJul 22, 2026
Merged
Conversation
Set max_connections=100 as a server arg, overriding the too-low timescaledb-tune default, and raise the pod memory limit to 2Gi accordingly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 Description
CE ships TimescaleDB with
max_connections=25— the value timescaledb-tune derivesat first boot for a 1Gi pod. Model monitoring holds ~21 standing connections per
monitored project (controller workers, writer/stream storey targets), so a single
busy project exhausts the server and queries fail with
FATAL: sorry, too many clients already.The limit is now pinned explicitly via a server argument. This is deliberate:
timescaledb-tune only runs on an empty data directory, and the PVC is kept across
upgrades (
helm.sh/resource-policy: keep) — a tune-side or memory-driven changewould never reach existing installs, while a
-cargument overridespostgresql.confon every pod start.🛠️ Changes Made
timescaledb.maxConnectionsvalue (default 100, ~4 concurrent monitored projects)postgres -c max_connections=...as container args in the TimescaleDB StatefulSetheadroom after timescaledb-tune sizes shared_buffers for the larger pod)
✅ Checklist
charts/mlrun-ce/Chart.yaml.🧪 Testing
helm templaterender verified:args: [postgres, -c, max_connections=100]on theTimescaleDB container
timescale/timescaledb-ha:pg17.7-ts2.24.0) verified:Entrypoint: docker-entrypoint.sh,Cmd: [postgres]— the explicit args preserve thedefault startup path
Note: the multi-namespace values files need no sync — they only override
timescaledb.enabled/ service type.🔗 References
🚨 Breaking Changes?
🔍️ Additional Notes
Existing installs pick up the new limit on the next pod restart (rollout on upgrade).
SDK-side hardening (shared pool per DSN,
min_size=0) is a separate follow-up inmlrun/mlrun — it reduces mlrun's connection footprint but would not have prevented
this failure mode on its own.