Repository navigation
Conversation
CI's build/release libduckdb statically links quackapi and json, so the POST contract tests could exercise the built-in copy instead of the artifact under test. Build a clean pinned core for those tests, assert quackapi and the HTTP providers are not preloaded before explicit LOAD, run conformance with an empty HOME, and report the loopback status and body when the same-database check fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The clean test core has no built-in extensions, and CI has no installed json, so serving failed with "QuackAPI requires the DuckDB JSON extension". Pass the release build's json artifact to the HTTP and TLS contracts and load it before QuackAPI in every test database, so they never depend on autoload from the user's extension directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The database runs with quackapi_post_concurrency = 1 here, and the interrupted 900 ms 'wide' POST keeps draining in that sole worker after its caller returns. The loopback route's POST queued behind it and spent its 1000 ms budget waiting, returning 504 on slower Linux runners. Wait for the fixture to finish that request first, as the deadline check above already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture-side 'active' counter hits 0 when the fixture finishes its 900ms sleep, which races the POST worker actually releasing its concurrency-1 slot after an interrupted drain. On a loaded runner the loopback POST can queue behind the drain and burn the 1s budget -> 504 flake. The loopback check is functional (same-database HTTP loopback works); deadline behavior is still covered by the 100ms /post-wait and /post-deadline checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
QuackAPI now focuses on SQL-backed HTTP routes and bounded outbound POST. Route SQL runs through DuckDB; route paths no longer select a separate executor. This breaking cleanup removes over 10,000 net production-source lines.
Removes GraphQL, SSE/streams, durable queues, policy rewriting, source/table generators, native libpq execution, fetch/pool diagnostics, automatic extension installation/tuning, and custom telemetry storage. Authentication, middleware, OpenAPI, HTTP limits, static files, compression, and lifecycle remain. Migration notes explicitly identify the lost queue and row-filtering semantics.
Response encoding consumes result chunks using DuckDB JSON/CSV facilities, preserves raw BLOBs, and limits encoded uncompressed bytes. Arrow/Parquet HTTP exporters are removed; typed transport through existing Flight SQL extensions is documented separately, including current type/concurrency restrictions.
Scalar POST preserves its overloads, result fields, volatility, and row alignment. It uses a database-owned bounded I/O executor and HTTPUtil providers, with shared configurable concurrency, admission/deadline/cancellation checks, and no automatic POST retry. The additive quackapi_post(TABLE) form emits batches sized to effective concurrency, allowing downstream processing before all responses in a DuckDB input vector accumulate. It retains input columns and appends the same response STRUCT. Individual HTTP responses remain buffered; this does not introduce HTTP streaming or a global byte budget.
On the same 256-file AST workload (541,619 nodes, 293,703,282 complete response bytes), the table form at concurrency four completed with 259,010,560 bytes of profiled peak buffer memory and no spill; the matched scalar form exhausted the 1 GB limit. At concurrency 32, the table form completed but reported 1,078,173,696 bytes of peak buffer memory. Full-value comparison against direct readers passed with a separately documented 4 GB verifier. The table-form result uses stock DuckDB JSON.
Validation includes real HTTP/TLS and provider replacement tests against both pinned providers, 85 HTTP conformance checks, and real-file self-dispatch. Tests verify 33/4,097-row alignment and exact side-effect counts, low-memory downstream JSON parsing, cancellation, shared concurrency, typed bodies and headers. Deliberate whole-vector batching and cursor mutations fail the relevant tests. Reproducible SQL and measurement limits are in bench/reader_self_dispatch_README.md. The POST contract tests run on a pinned DuckDB core with no built-in extensions and load the quackapi, json and provider artifacts explicitly.
Provider dependency
The POST contract needs provider fixes that are not in any released provider. CI builds them from pinned fork commits:
curl_httpfs: asubbarao-ifr/duckdb-curl-filesystem@a48f83c — upstream f253956 plus subsecond timeouts (d80c08e), caller Content-Type on POST (59d3f4a), and MultiCurl connection reuse (a48f83c).httpfs(Curl client): asubbarao-ifr/duckdb-httpfs@748f5ac — fractional Curl timeouts.The earlier upstream PRs for these fixes (dentiny/duckdb-curl-filesystem#75–#77, duckdb/duckdb-httpfs#490) and the DuckDB JSON parser PR (duckdb/duckdb#27268) were opened by an agent without approval and have been closed. Any upstream submission is a separate decision.
This PR is not ready to merge until the provider dependency is resolved (released providers, or an agreed fork distribution) and CI is green. The same-database loopback contract currently fails on Linux CI (504 at its 1000 ms deadline) and is under investigation.
🤖 Generated with Claude Code