feat(lake): bound ClickHouse connection pool (SC-9)#498
Merged
Conversation
getLakeClient() built the cached @clickhouse/client client via
createClient() with no pool configuration, so under load the lake opened
unbounded connections. Pass connection bounds sourced from the centralized
env module:
- max_open_connections from VF_LAKE_CH_POOL_MAX (default 10)
- request_timeout from VF_LAKE_CH_REQUEST_TIMEOUT_MS (default 30000ms)
- keep_alive: { enabled: true } (explicit; @clickhouse/client@1.x default)
The pool is per-client, so the single cached-client pattern
(globalForLake.__vfLakeClient) is kept unchanged. New env vars follow the
existing DATABASE_POOL_MAX / DATABASE_CONNECTION_TIMEOUT_MS pattern in
src/lib/env.ts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SC-9 — ClickHouse connection pooling
Problem
getLakeClient()(src/server/services/lake/clickhouse.ts) built the cached@clickhouse/clientclient viacreateClient()with no pool configuration, so under load the lake opened unbounded connections.Change
createClient()now receives bounded connection settings, sourced from the centralized env module (mirrors theprisma.ts→env.DATABASE_POOL_MAXconvention):max_open_connections←VF_LAKE_CH_POOL_MAX(conservative default 10)request_timeout←VF_LAKE_CH_REQUEST_TIMEOUT_MS(default 30000ms) — fail slow lake requests instead of hanging a held connectionkeep_alive: { enabled: true }— set explicitly (it is the@clickhouse/client@1.xdefault) to document intent / socket reusesrc/lib/env.tsfollowing the existingDATABASE_POOL_MAX/DATABASE_CONNECTION_TIMEOUT_MSpattern (z.coerce.number().int().positive().default(...)).@clickhouse/client@1.20.0(not invented):max_open_connections,request_timeout,keep_alive.globalForLake.__vfLakeClient) is kept unchanged — the pool is the process-wide ceiling.Acceptance
The lake ClickHouse client is created with a bounded connection pool. ✅
Tests
src/server/services/lake/__tests__/clickhouse-pool.test.ts(new):getLakeClient()callscreateClientwithmax_open_connectionsfrom env — default (10) and overridden (25).request_timeout+keep_alive: { enabled: true }are passed.createClientruns once across twogetLakeClient()calls.@clickhouse/clientis mocked (not installed in OSS dev);@/lib/envis mocked with a mutable hoisted object so default + override are exercised under static imports (no dynamic import / module reset, per repo rules).Notes
@clickhouse/clientmodule-not-found on the existing import (package absent in OSS dev) — to be ignored per the local-gap convention.@cloud/*imports.