Repository navigation
Use a per-request DuckDB connection - #24
Merged
Merged
Conversation
Replaces the module-level DuckDBManager with a per-request context manager. The previous design shared one connection across all requests and closed/recreated it inside _apply_s3_config, which was not concurrency-safe: one request could close the connection another was actively using. - _duckdb_connection(config) yields a fresh in-memory conn, loads extensions, applies S3+Iceberg config, and closes on exit - _apply_s3_config, _attach_iceberg_catalog, _validate_iceberg_table, _convert_to_iceberg_query become module-level functions taking conn - Route handlers delegate to run_connection_test / run_query impls - Lifespan pre-installs extensions once at startup so first request does not pay the download cost (DuckDB caches binaries on disk) - Drop PRAGMA enable_profiling / profiling_output: the output file was never read and writing a shared filename from concurrent connections would reintroduce the race we are fixing Closes #9
Covers the two guard functions exposed by the per-request refactor (_validate_and_limit_sql, _convert_to_iceberg_query) plus the two SQL-safety helpers at the API boundary (_sql_string_literal, ConnectionConfig validators). Unit-only: no DuckDB engine, no network, ~0.5s to run. Wired into CI as a new backend-tests job so the suite runs on every push and PR. - conftest.py: adds backend/ to sys.path so tests import cleanly - pytest.ini: fixes test discovery to backend/tests - requirements.txt: pins pytest 8.1.1 and httpx 0.27.0 - ci.yml: adds backend-tests job (setup-python, pip cache, pytest) Refs #10
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9. Refs #10.
Problem
DuckDBManagerkept one shared connection at module scope, and_apply_s3_configclosed it and opened a new one on every request. Two concurrent requests could collide: request B's reconfigure would close the connection request A was mid-query on, surfacing as spurious errors or (worse) cross-contaminated session state.Fix
Per-request connection lifecycle. No shared state.
_duckdb_connection(config)— a context manager that opens a fresh in-memory DuckDB, installs+loads extensions, applies the caller's S3/Iceberg config, and closes on exit._apply_s3_config,_attach_iceberg_catalog,_validate_iceberg_table,_convert_to_iceberg_queryare now plain functions that take aconnargument. Noself.connection. No hidden mutation.test_connection,execute_query) delegate torun_connection_test/run_queryimpls — renaming avoids the route/impl name clash.DuckDBManagerclass and thedb_managerglobal are gone.INSTALL/LOADis ms-level after that.PRAGMA enable_profiling=jsonandPRAGMA profiling_output='query_profile.json'— the JSON file was never read by any code, and writing a shared filename from many connections would reintroduce the exact race we're fixing.Test suite
Since the refactor made the validators and converters module-level (testable without a live DB), a second commit adds a pytest unit suite and wires it into CI:
backend/tests/test_sql_validation.py— SELECT/WITH/UNION accepted with LIMIT injection, preserves existing LIMIT, rejects INSERT/DROP/DELETE/UPDATE/CREATE/ATTACH and multi-statement inputbackend/tests/test_query_conversion.py—read_parquet()→iceberg_scan()and REST catalog rewritingbackend/tests/test_connection_config.py— Pydantic validators: SQL-injection attempts intablePath, non-identifier namespaces, bad regions, control chars in credentials,/metadatastrippingbackend/tests/test_sql_string_literal.py— quote doubling, control-char rejection, tab allowanceconftest.py+pytest.inifor clean discovery withoutPYTHONPATHbackend-testsCI job (setup-python 3.12, pip cache, pytest)35 tests, ~0.5s runtime, no DuckDB engine / network / docker needed.
Verified locally
ruff check backend/ scripts/clean.pytest35/35 pass in 0.48s.docker compose up --build: demo query returns 37,537 rows.Before this PR the same test would intermittently fail or return wrong counts as connections got swapped mid-query.
DROP TABLE foostill rejected with400 Only SELECT queries are allowed (got DROP)— sqlglot validator unchanged.