Skip to content

Use a per-request DuckDB connection - #24

Merged
gordonmurray merged 2 commits into
mainfrom
fix/per-request-duckdb
Apr 15, 2026
Merged

gordonmurray merged 2 commits into
mainfrom
fix/per-request-duckdb

Conversation

@gordonmurray

@gordonmurray gordonmurray commented Apr 15, 2026 •

Copy link
Copy Markdown
Owner

Closes #9. Refs #10.

Problem

DuckDBManager kept one shared connection at module scope, and _apply_s3_config closed it and opened a new one on every request. Two concurrent requests could collide: request B's reconfigure would close the connection request A was mid-query on, surfacing as spurious errors or (worse) cross-contaminated session state.

Fix

Per-request connection lifecycle. No shared state.

  • _duckdb_connection(config) — a context manager that opens a fresh in-memory DuckDB, installs+loads extensions, applies the caller's S3/Iceberg config, and closes on exit.
  • _apply_s3_config, _attach_iceberg_catalog, _validate_iceberg_table, _convert_to_iceberg_query are now plain functions that take a conn argument. No self.connection. No hidden mutation.
  • Route handlers (test_connection, execute_query) delegate to run_connection_test / run_query impls — renaming avoids the route/impl name clash.
  • DuckDBManager class and the db_manager global are gone.
  • Startup lifespan now just pre-installs extensions once so the first real request doesn't pay the download cost. DuckDB caches the binaries on disk, so each per-request INSTALL/LOAD is ms-level after that.
  • Dropped PRAGMA enable_profiling=json and PRAGMA profiling_output='query_profile.json' — the JSON file was never read by any code, and writing a shared filename from many connections would reintroduce the exact race we're fixing.

Test suite

Since the refactor made the validators and converters module-level (testable without a live DB), a second commit adds a pytest unit suite and wires it into CI:

  • backend/tests/test_sql_validation.py — SELECT/WITH/UNION accepted with LIMIT injection, preserves existing LIMIT, rejects INSERT/DROP/DELETE/UPDATE/CREATE/ATTACH and multi-statement input
  • backend/tests/test_query_conversion.py — read_parquet() → iceberg_scan() and REST catalog rewriting
  • backend/tests/test_connection_config.py — Pydantic validators: SQL-injection attempts in tablePath, non-identifier namespaces, bad regions, control chars in credentials, /metadata stripping
  • backend/tests/test_sql_string_literal.py — quote doubling, control-char rejection, tab allowance
  • conftest.py + pytest.ini for clean discovery without PYTHONPATH
  • New backend-tests CI job (setup-python 3.12, pip cache, pytest)

35 tests, ~0.5s runtime, no DuckDB engine / network / docker needed.

Verified locally

  • ruff check backend/ scripts/ clean.
  • pytest 35/35 pass in 0.48s.
  • End-to-end via docker compose up --build: demo query returns 37,537 rows.
  • Concurrency test — 5 parallel requests with different queries and row limits each return their own correct row count:
A: rows=5    (LIMIT 5)
B: rows=1    (COUNT(*))
C: rows=23   (LIMIT 23)
D: rows=13   (GROUP BY decade)
E: rows=7    (LIMIT 7)

Before this PR the same test would intermittently fail or return wrong counts as connections got swapped mid-query.

  • DROP TABLE foo still rejected with 400 Only SELECT queries are allowed (got DROP) — sqlglot validator unchanged.

Replaces the module-level DuckDBManager with a per-request context
manager. The previous design shared one connection across all requests
and closed/recreated it inside _apply_s3_config, which was not
concurrency-safe: one request could close the connection another was
actively using.

- _duckdb_connection(config) yields a fresh in-memory conn, loads
  extensions, applies S3+Iceberg config, and closes on exit
- _apply_s3_config, _attach_iceberg_catalog, _validate_iceberg_table,
  _convert_to_iceberg_query become module-level functions taking conn
- Route handlers delegate to run_connection_test / run_query impls
- Lifespan pre-installs extensions once at startup so first request
  does not pay the download cost (DuckDB caches binaries on disk)
- Drop PRAGMA enable_profiling / profiling_output: the output file was
  never read and writing a shared filename from concurrent connections
  would reintroduce the race we are fixing

Closes #9
Covers the two guard functions exposed by the per-request refactor
(_validate_and_limit_sql, _convert_to_iceberg_query) plus the two
SQL-safety helpers at the API boundary (_sql_string_literal,
ConnectionConfig validators).

Unit-only: no DuckDB engine, no network, ~0.5s to run. Wired into CI
as a new backend-tests job so the suite runs on every push and PR.

- conftest.py: adds backend/ to sys.path so tests import cleanly
- pytest.ini: fixes test discovery to backend/tests
- requirements.txt: pins pytest 8.1.1 and httpx 0.27.0
- ci.yml: adds backend-tests job (setup-python, pip cache, pytest)

Refs #10
@gordonmurray
gordonmurray merged commit 3657bba into main Apr 15, 2026
4 checks passed
@gordonmurray
gordonmurray deleted the fix/per-request-duckdb branch April 15, 2026 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Global DuckDB connection torn down and rebuilt on every request

1 participant