Skip to content

feat(lake): bound ClickHouse connection pool (SC-9)#498

Merged
TerrifiedBug merged 1 commit into
mainfrom
feat/sc-9-clickhouse-pool
Jun 8, 2026
Merged

feat(lake): bound ClickHouse connection pool (SC-9)#498
TerrifiedBug merged 1 commit into
mainfrom
feat/sc-9-clickhouse-pool

Conversation

@TerrifiedBug

Copy link
Copy Markdown
Owner

SC-9 — ClickHouse connection pooling

Problem

getLakeClient() (src/server/services/lake/clickhouse.ts) built the cached @clickhouse/client client via createClient() with no pool configuration, so under load the lake opened unbounded connections.

Change

  • createClient() now receives bounded connection settings, sourced from the centralized env module (mirrors the prisma.tsenv.DATABASE_POOL_MAX convention):
    • max_open_connectionsVF_LAKE_CH_POOL_MAX (conservative default 10)
    • request_timeoutVF_LAKE_CH_REQUEST_TIMEOUT_MS (default 30000ms) — fail slow lake requests instead of hanging a held connection
    • keep_alive: { enabled: true } — set explicitly (it is the @clickhouse/client@1.x default) to document intent / socket reuse
  • New env vars added to src/lib/env.ts following the existing DATABASE_POOL_MAX / DATABASE_CONNECTION_TIMEOUT_MS pattern (z.coerce.number().int().positive().default(...)).
  • Option names verified against @clickhouse/client@1.20.0 (not invented): max_open_connections, request_timeout, keep_alive.
  • The pool is per-client, so the single cached-client pattern (globalForLake.__vfLakeClient) is kept unchanged — the pool is the process-wide ceiling.

Acceptance

The lake ClickHouse client is created with a bounded connection pool. ✅

Tests

src/server/services/lake/__tests__/clickhouse-pool.test.ts (new):

  • getLakeClient() calls createClient with max_open_connections from env — default (10) and overridden (25).
  • request_timeout + keep_alive: { enabled: true } are passed.
  • Client is cachedcreateClient runs once across two getLakeClient() calls.

@clickhouse/client is mocked (not installed in OSS dev); @/lib/env is mocked with a mutable hoisted object so default + override are exercised under static imports (no dynamic import / module reset, per repo rules).

pnpm exec vitest run src/server/services/lake/__tests__/clickhouse-pool.test.ts   # 3 passed
pnpm exec vitest run src/server/services/lake/__tests__/clickhouse.test.ts        # 11 passed (unaffected)
pnpm exec vitest run src/lib/__tests__/env.test.ts                                # 21 passed (unaffected)

Notes

  • No migration.
  • Filtered typecheck: clean except the known, pre-existing @clickhouse/client module-not-found on the existing import (package absent in OSS dev) — to be ignored per the local-gap convention.
  • ClickHouse/lake runtime paths aren't locally/visually verifiable; unit tests + types are the bar.
  • License boundary respected: no @cloud/* imports.

getLakeClient() built the cached @clickhouse/client client via
createClient() with no pool configuration, so under load the lake opened
unbounded connections. Pass connection bounds sourced from the centralized
env module:

- max_open_connections from VF_LAKE_CH_POOL_MAX (default 10)
- request_timeout from VF_LAKE_CH_REQUEST_TIMEOUT_MS (default 30000ms)
- keep_alive: { enabled: true } (explicit; @clickhouse/client@1.x default)

The pool is per-client, so the single cached-client pattern
(globalForLake.__vfLakeClient) is kept unchanged. New env vars follow the
existing DATABASE_POOL_MAX / DATABASE_CONNECTION_TIMEOUT_MS pattern in
src/lib/env.ts.
@TerrifiedBug TerrifiedBug merged commit 194264a into main Jun 8, 2026
17 checks passed
@TerrifiedBug TerrifiedBug deleted the feat/sc-9-clickhouse-pool branch June 8, 2026 12:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant