[grokspice]

Summary

Under on_conflict: upsert, the DuckDB sink wrote a whole stream as one INSERT INTO t SELECT * FROM <arrow scan view> ON CONFLICT (k) DO UPDATE SET …. DuckDB resolves a conflict between two rows of the same statement by keeping whichever row it processes first, and its parallel scan of the statement's input decides that order. So a key the write repeated across record batches kept a copy chosen by scheduling: the first copy in 9 of 10 runs on the reporter's host, in 1 of 10 runs on mine (spiceai/spiceai#14643).

A conflict between a row and a row an earlier statement of the same transaction inserted is resolved as DO UPDATE says. This change cuts an upsert write into statements at the first row whose key an earlier row of the same statement carried (core/src/duckdb/upsert_groups.rs): no key repeats within a statement, and across statements the last copy wins, in arrival order. A write that repeats no key is still one statement.

• Keys are compared as u64 hashes of their arrow row encoding; a collision only ends a statement early (one more statement, same rows).
• Keys are compared as DuckDB stores them, so two arrow values the table holds as one key hash alike (all checked on DuckDB 1.4.4 through CREATE TABLE … AS SELECT * FROM <arrow view>, which is how the sink creates its tables): -0.0 as 0.0 and every NaN as one NaN (INSERT … VALUES (-0.0, …) ON CONFLICT (id) DO UPDATE updates a stored 0.0); a Date64 as a DATE (ms / 86_400_000, truncated toward zero: -1 ms is 1970-01-01), a nanosecond Time64 as a microsecond TIME, a nanosecond timestamp with a time zone as a microsecond TIMESTAMP WITH TIME ZONE (-1 ns is epoch 0 µs), and a dictionary as its values; an INTERVAL (a duration or interval column) and a Float16 cannot be index keys in DuckDB at all (Invalid type for index key / Unsupported Internal Arrow Type e), so they are never conflict targets. A nanosecond timestamp without a time zone is a TIMESTAMP_NS and keeps every nanosecond. The rows written still carry the values as given.
• A NULL key is hashed like any value. Within one statement DuckDB resolves two NULL unique keys as one conflict and drops the later row (INSERT (1,NULL,'x'),(1,NULL,'y') … ON CONFLICT (a,b) DO UPDATE stores one row), where across statements, as for a plain insert, it keeps both; so a repeated NULL key starts a new statement and every row is kept. Through the sink, repeated NULL keys within one batch are refused by the per-batch constraint check (its DISTINCT counts NULLs as one), so the one-statement-per-NULL cost applies only to write_to_table without constraints; across batches it is one statement per batch, as for any key.
• A statement holds at most 2^20 distinct keys (about 16 MiB of hashes) before the next one starts, so the set is bounded for a write that repeats no key.
• The insert is positional (INSERT INTO table SELECT * FROM <view>), so the conflict target is resolved against the table's schema and read from the written data at that column position; the written data may name its columns differently, and a key column written as another type is compared as the table's type when arrow's cast is an exact conversion the insert performs the same way (a widened integer or float, a string or binary spelling, a dictionary or run-end encoding, a widened Date32); a key written as any other type (a float into an integer key, which arrow truncates and DuckDB rounds; text into a number or date; a timestamp given a time zone, which DuckDB reads in its session's zone) puts every row in a statement of its own, which is slow and right.
• write_to_table is the one entry point for the overwrite, append, and file-swap overwrite paths, so all three get the behavior.

Refs spiceai/spiceai#14643 (the spiceai pin bump, with its accelerator regression tests and the docs/dev/fork_patches.md row, closes it).

Reproduction — spiceai/spiceai#14643

Shipped runtime, before. Spice v2.4.0-rc.1 (spiced_linux_x86_64, whose datafusion-table-providers pin 465926a3 carries the same core/src/duckdb/write.rs as spiceai-55's head; git diff 465926a3 ef2e590 -- core/src/duckdb/write.rs is empty), the issue's Spicepod (engine: duckdb, mode: file, refresh_mode: full, primary_key: id, on_conflict: { id: upsert }, target_partitions: 1) over across.csv (ids 0..8191 with v = first, then one row 0,last), ten fresh starts, each asked:

SELECT (SELECT COUNT(*) FROM t) AS rows, (SELECT string_agg(v, ',') FROM t WHERE id = 0) AS key_0

      1 rc1 upsert across.csv: [{"rows":8192,"key_0":"first"}]
      9 rc1 upsert across.csv: [{"rows":8192,"key_0":"last"}]

Engine, before (why one statement cannot be fixed by ordering). DuckDB 1.4.4 via Python, a RecordBatchReader yielding the same two batches, one INSERT … SELECT * FROM src ON CONFLICT (id) DO UPDATE SET v = excluded.v, 20 runs each:

== one statement over the whole stream (trunk's shape)      20 rows=8192 key_0=last
== one statement, threads=1                                  20 rows=8192 key_0=first
== repeat inside ONE 8193-row batch (spans DuckDB chunks)    20 rows=8192 key_0=first
== repeat inside ONE 101-row batch (one DuckDB chunk)        20 rows=100  key_0=first
== per-batch statements, in one transaction                  20 rows=8192 key_0=last
== per-batch statements, in one transaction, threads=1       20 rows=8192 key_0=last

With one thread the rows are processed in arrival order and DuckDB still keeps the first copy: within a statement the first-processed copy wins, and only a later statement's DO UPDATE replaces a stored row.

Fork tests, before and after, on a CI sandbox (grokspice's fork; make test runs the duckdb-federation lib tests in the Tests job):

• the final regression tests alone on the unchanged spiceai-55 base (CI sandbox (red run): upsert regression tests alone on the unchanged base (spiceai/spiceai#14643) grokspice/datafusion-table-providers#2, commit a1dd93e): https://github.com/grokspice/datafusion-table-providers/actions/runs/37894009849 (Tests job 113701090024, make test):

  test duckdb::write::test::upsert_overwrite_keeps_the_last_copy_of_a_key_repeated_across_batches ... FAILED
    left: (8192, Some("first"))   right: (8192, Some("last"))
  test duckdb::write::test::upsert_overwrite_keeps_the_last_of_many_copies_under_a_parallel_scan ... FAILED
    left: (51, Some("copy 5"))    right: (51, Some("copy 49"))
  test duckdb::write::test::upsert_append_keeps_the_last_copy_over_stored_and_earlier_rows ... FAILED
    left: (3, Some("first"))      right: (3, Some("last"))
  test duckdb::write::test::upsert_keeps_every_row_whose_unique_key_is_null ... FAILED
    left: (2, Some("a"))          right: (3, Some("a,b"))
  test duckdb::write::test::upsert_keeps_the_last_copy_of_a_date64_key_duckdb_stores_as_a_date ... FAILED
    left: (2, Some("first second"))  right: (2, Some("second second"))
  test duckdb::write::test::upsert_keeps_the_last_copy_of_a_float_key_duckdb_holds_as_one ... FAILED
    left: (2, Some("zero"))       right: (2, Some("negative zero"))
  test duckdb::write::test::upsert_keeps_the_last_copy_of_a_key_repeated_within_a_batch ... FAILED
    left: (2, Some("a"))          right: (2, Some("c"))
  test duckdb::write::test::upsert_resolves_the_conflict_target_against_the_table_not_the_written_names ... FAILED
    left: (2, Some("first"))      right: (2, Some("last"))
  test duckdb::write::test::upsert_keeps_the_last_copy_of_a_key_written_as_text_into_a_bigint ... FAILED
    to write: ... Constraint Error: PRIMARY KEY or UNIQUE constraint violation: duplicate key "1"
  test result: FAILED. 483 passed; 9 failed; 2 ignored
  

• the same tests with the fix (CI sandbox: DuckDB upsert keeps the last copy of a repeated key (spiceai/spiceai#14643) grokspice/datafusion-table-providers#1, commit fed05b3): https://github.com/grokspice/datafusion-table-providers/actions/runs/37893972430 (Tests job 113700969462): the 9 duckdb::write::test::upsert_* tests and the 27 duckdb::upsert_groups::tests::* tests all ok; test result: ok. 519 passed; 0 failed; 2 ignored; finished in 64.15s. (Clippy green on both runs; DuckDB Tests (release) runs only make test-integration-duckdb, which dies on the unrelated upsert_retention_memory soak crash, DuckDB: upsert_retention_memory fails with engine state corruption (checksum-0 blocks, upsert PK violation, allocator bitmask) across unrelated branches spiceai#14183, on every branch of this fork today; Python Tests hangs on every branch, so both sandbox runs were cancelled after their Tests job finished.)

(The first tests-only push, be156d8, did not compile on CI: its tests called connect_sync on a borrowed pool. That is why the red run is a separate branch carrying the final tests.)

Changes

• core/src/duckdb/upsert_groups.rs (new): write_statements drives one insert per statement over UpsertGroups, which hands out a write's rows statement by statement, and StatementReader, the arrow stream DuckDB scans for one statement.
• core/src/duckdb/write.rs: write_to_table routes an OnConflict::Upsert write through write_statements; insert_from_reader is the one-statement body it and every other write use.
• Tests: upsert_groups::tests drive the production loop (statement boundaries across batches and within a batch, every copy in its own statement, a repeated NULL key, empty write and empty batches, the key cap, composite keys, -0.0/NaN for f32 and f64, Date64, nanosecond Time64, zoned and plain nanosecond timestamps, dictionary keys, case-insensitive target, missing and ambiguous target, an insert error); write::test through the real sink and DuckDB (upsert_overwrite_keeps_the_last_copy_of_a_key_repeated_across_batches, upsert_append_keeps_the_last_copy_over_stored_and_earlier_rows, upsert_overwrite_keeps_the_last_of_many_copies_under_a_parallel_scan, upsert_keeps_the_last_copy_of_a_key_repeated_within_a_batch, upsert_keeps_the_last_copy_of_a_float_key_duckdb_holds_as_one, upsert_keeps_the_last_copy_of_a_date64_key_duckdb_stores_as_a_date, upsert_keeps_every_row_whose_unique_key_is_null, upsert_resolves_the_conflict_target_against_the_table_not_the_written_names, upsert_keeps_the_last_copy_of_a_key_written_as_text_into_a_bigint). The module's tests also cover the conflict target read by the table's column position from differently named written data, written data too narrow for that position (an error), and a run-end-encoded key. The sink tests set SET threads = 1 on the test database so that, before the fix, one statement keeps the first copy on every run rather than at random.

Performance impact

• A write that repeats no key is still one statement; the added work is hashing each row's key once. Grouping 1,007,616 distinct Int64 keys in 123 batches of 8,192 costs 82.3 ms (release build, best of 5, arrow-only probe of the module on this host, final revision), against 1,156 ms for the single-statement INSERT … ON CONFLICT DO UPDATE of the same rows into an empty table on DuckDB 1.4.4 (Python, best of 3), i.e. about 6%.
• A write that repeats keys pays one statement per run of non-repeating rows. Through the sink, a key repeated within a record batch is refused by the per-batch constraint check before the write (unchanged: spiceai turns primary_key and every unique index into a constraint, and upsert_dedup_by_row_id resolves the batch to one row per key above the sink), so the ceiling there is one statement per record batch: 7,539 ms vs 1,156 ms for 1,000,000 rows in 123 batches into an empty table when every batch repeats a key of the previous one, 1,082 ms vs 172 ms into a table already holding them (same Python probe). write_to_table called without constraints can reach one statement per repeated row; measured in Python at about 4.7–7.7 ms P99 per upsert statement (6,000 sequential statements, per 1,000-statement block), so one 8,192-row batch over 2 distinct keys is 4,096 statements and about 18 s where one statement took 0.14 s. That is the cost of the answer being right: before, those writes were one statement that kept a random copy, and on a NULL unique key dropped rows.
• The row count a write reports is now the rows inserted or updated across its statements ((1,a),(2,b),(1,c) reports 3); as one statement DuckDB reported 2, not counting the copy it discarded. on_data_written receives that count.

Test plan

• make test (cargo test --features …duckdb-federation… -p datafusion-table-providers --lib) — fork CI Tests job: 9 failed on the tests-only branch, 519 passed with the fix (runs and output above).
• cargo clippy --all-features -- -D warnings and cargo fmt --all -- --check — fork CI Clippy job green on run 37893972430 (fed05b3); the module also passes -D clippy::pedantic -D clippy::unwrap_used in an arrow-only probe on this host.
• Module tests watched failing: with first_repeat forced to None (no grouping) in the probe, 9 of the then 13 tests fail (a_key_repeated_in_a_later_batch_starts_a_new_statement_at_that_batch, every_copy_of_a_key_lands_in_its_own_statement_in_arrival_order, … FAILED; test result: FAILED. 4 passed; 9 failed).

Review gate

• /simplify: applied before the reviews (c691366): the statement loop moved into upsert_groups::write_statements, the source trait and generics dropped, each batch's keys hashed once and a held batch kept with its hashes and offset, test fixtures deduplicated; the module's tests and the verify script re-run afterwards (20 passed; batch_verify.py base run exit 1 at 8a1e2a4c7).
• Adversarial review: codex, two completed jobs. review-mv0k13lr-1vsvln on d8045a6: needs-attention, one medium finding — the sink's per-batch constraint check (UpsertOptions::default()) rejects a key repeated within one batch before the grouping runs, so [(0,a),(1,b),(0,c)] through write_all fails rather than keeping c. Refuted as out of scope: that is the documented upsert contract (constraints.md: "a key repeated within one record batch fails the write unless upsert_dedup_by_row_id is set"), unchanged by this PR; #14643 is the across-batch case, and relaxing the check is a surface change for its own Enhancement. The within-batch test is deliberately at the write_to_table level and says so. review-mv0k6u3d-ixapo5 on 182d6e6: needs-attention, two medium findings, both fixed in 1bdb838: the conflict target was resolved against the written data's column names although the insert is positional (now resolved against the table's schema and read by position; tests the_conflict_target_is_read_from_the_written_data_by_the_tables_position, written_data_with_fewer_columns_than_the_tables_key_position_is_an_error, and upsert_resolves_the_conflict_target_against_the_table_not_the_written_names through DuckDB), and a run-end-encoded key column skipped canonicalisation (now decoded like a dictionary; test a_run_end_encoded_key_repeats_by_its_decoded_value). review-mv0kgjoz-b81wes on 1bdb838: needs-attention, one medium finding, fixed in 8721463: a key written as another type than the table's ("01"/"1" into BIGINT) collides after DuckDB's cast but hashed apart; the key column is now cast to the table's type first. review-mv0kow3p-yrwb6p on 8721463: needs-attention: arrow's cast disagrees with DuckDB's for a float written to an integer key (truncate vs round) → fixed in 19c9716: only exact conversions are trusted (converts_exactly, test only_exact_conversions_are_trusted; the time-zone arm was later removed, next round), any other type change isolates every row in its own statement (tests a_key_written_as_a_type_the_insert_converts_inexactly_isolates_every_row, a_key_written_as_a_narrower_integer_is_compared_as_the_tables_integer, and upsert_keeps_the_last_copy_of_a_key_written_as_text_into_a_bigint through DuckDB); the within-batch finding repeated and stands refuted as above. review-mv0kx53s-sex12p on 19c9716: needs-attention, two medium findings, both fixed in fed05b3: the isolate path still ran arrow's cast (a struct written into a VARCHAR key has no arrow cast but DuckDB's insert converts it) → an isolated write hashes nothing (test an_isolated_write_converts_nothing_itself); attaching a time zone is not exact, since DuckDB reads a zone-less timestamp in its session's zone → removed from the trusted conversions (only_exact_conversions_are_trusted). Final HEAD fed05b3: review-mv0l8gzv-0cc65u, needs-attention with that same within-batch finding only (refuted as above); receipt engine=codex exit=0.
  Before codex had credits (Your workspace is out of credits on c691366, receipt attempt=codex result=exit-1 engine=none), a same-family stand-in (a fresh-context Claude agent told to run probes, not cross-model) reviewed c691366 and found, with Python DuckDB 1.4.4 artifacts: (1) a later NULL unique key within one statement is dropped by DuckDB → fixed in d8045a6 (NULL keys hashed like values; test upsert_keeps_every_row_whose_unique_key_is_null); (2) Date64/Time64(ns)/zoned ns timestamps compared above DuckDB's stored precision → fixed in d8045a6 (canonical_type/canonical_key_column; unit tests per type and upsert_keeps_the_last_copy_of_a_date64_key_duckdb_stores_as_a_date); (3) one statement per repeated key within a batch costs ~4.7–7.7 ms P99 each → accepted with numbers above (unreachable through the sink, which refuses within-batch repeats); (4) the reported row count now counts updates → noted above.
• /code-review high (sub-agent on the branch diff; the skill diffs the main checkout): no confirmed defect; raised the precision gap (fixed, above), the dictionary interning hypothesis (arrow 59's row format no longer interns; the dictionary is cast to its values before encoding), the statement-count cost (accepted, above), and the new key-resolution error wording (kept: DuckDB previously raised a binder error for the same input). Round 2 on the c691366..d8045a6 delta: one vacuous test (the zoned-timestamp case could not tell truncation from floor) → fixed in 182d6e6, watched failing under div_euclid (left: [2, 3] right: [2, 2, 1]); the INTERVAL gap → moot, DuckDB refuses INTERVAL as an index key, arm removed and documented; unordered string_agg → ORDER BY name.
• /security-review (sub-agent, same diff): no critical or high finding; no new SQL path (the conflict-target names only reach schema.index_of), the per-statement view name is dropped before the next registration and duckdb_arrow_scan replaces on a clash, memory bounded by the key cap, no panic reachable from data, an error rolls the transaction back. One pre-existing identifier-quoting observation in on_conflict.rs, unchanged by this PR, is being reported privately rather than in this public thread.

Follow-ups

• spiceai/spiceai: move the datafusion-table-providers pin, add the docs/dev/fork_patches.md row with the accelerator-duckdb guard tests, and close DuckDB acceleration: on_conflict: upsert keeps the first copy of a key a refresh repeats across record batches, varying run to run spiceai#14643.
• spiceai/docs website/docs/features/data-acceleration/constraints.md line 98 ("which copy is kept can vary from run to run") describes the behavior this change removes; it changes with the pin bump.

[Copilot]

🟡 Changes recommended

Key grouping uses the configured schema rather than the live positional DuckDB schema, allowing incorrect grouping on accepted append schemas.

2 open findings

• Resolve key positions from the live table schema · New
• Document statement splitting for writes exceeding the key limit · New

What changed in this PR

Fixes nondeterministic DuckDB upserts so repeated keys retain their last-arriving row.

Changes:

• Splits upserts into bounded statements without duplicate keys.
• Canonicalizes DuckDB key representations.
• Adds regression and edge-case coverage.

File	Description
core/​src/​duckdb.rs	Registers the grouping module.
core/​src/​duckdb/​upsert_groups.rs	Implements grouping, canonicalization, and unit tests.
core/​src/​duckdb/​write.rs	Integrates grouped inserts and sink tests.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[grokspice]

The red Python Tests on this PR is a repo-wide break, not this change. Recording the measurement so a reviewer does not have to re-derive it; this PR's own checks (Clippy, Tests, DuckDB Tests (release), every Build (*)) are green.

Python Tests has failed on every PR run in this repository since 2026-09-26

Job conclusions for Python Tests, read per run over the last 45 runs of the pr workflow (gh run list --workflow pr → gh api .../actions/runs/<id>/jobs):

last success  2026-09-20T00:39:33Z
first failure 2026-09-26T16:05:44Z
since then    27 runs, 27 failures, 0 successes

Identical signature on four independently authored branches, including maintainers' own:

run	branch	result
37897037853	fix/duckdb-upsert-keep-last (this PR)	17 passed, 2 errors
37874989216	lukim/federation-correlated-binding	17 passed, 2 errors
37871863462	fix/sqlite-interval-datetime	17 passed, 2 errors
37870489732	fix/sqlite-rows-error	17 passed, 2 errors

ERROR test_clickhouse.py::TestClickHouseParameterized::test_get_tables - Exception: ClickHouse connection failed. decompression error: incorrect magic number
ERROR test_clickhouse.py::TestClickHouseParameterized::test_parameterized_view - Exception: ClickHouse connection failed. decompression error: incorrect magic number
============ 17 passed, 16 warnings, 2 errors in 146.89s (0:02:26) =============

Only the two ClickHouse tests fail, on all four. Everything else in the suite passes.

Nothing in this repository changed when it broke

$ gh api "repos/spiceai/datafusion-table-providers/commits?sha=main&since=2026-09-19T00:00:00Z&until=2026-09-27T00:00:00Z"
[]

No commit landed on main between the last green run and the first red one.

The probable cause, labelled as a hypothesis

python/python/tests/test_clickhouse.py:12 starts the server from an unpinned tag:

["docker", "run", "--name", "clickhouse", ..., "clickhouse/clickhouse-server:latest"]

clickhouse/clickhouse-server:latest today resolves to 26.9.13.15, republished 2026-10-08 (Docker Hub tags API), while the client is pinned at clickhouse = "0.13.3" (core/Cargo.toml:31). A moving server image against a pinned client is consistent with both the timing and a decompression failure during the HTTP handshake.

This is not verified. I could not reproduce it: this host has no Docker available, so I have not run the two tests against any specific server version and have not identified which release first breaks the 0.13.3 client. The correlation above is the whole of the evidence; the causal claim is a hypothesis. What is established is the first list: the failure predates this branch, is identical across four unrelated branches, and no repository change coincides with it.

What this needs

Pinning the image to a known-good tag is the obvious shape of a fix, but choosing the tag requires a run that I cannot perform, and I have read-only access here (gh api repos/spiceai/datafusion-table-providers → permissions.push=false), so I can neither push such a fix nor re-run the job. A maintainer with write access is required — both for that and for this PR's review.

[Copilot]

🔵 Needs a closer look

Correctness depends on extensive Arrow-to-DuckDB type and conflict semantics that warrant final human validation.

0 open findings

2 resolved since last review

• Resolve key positions from the live table schema
• Document statement splitting for writes exceeding the key limit

🧠 Review effort: Balanced

[grokspice]

Status update, not a new escalation. While re-verifying the red checks on #87 and #88, I measured something that makes this PR more urgent than its own review queue suggests: the DuckDB Tests (release) failure blocking both of those PRs correlates exactly with the absence of this change.

The measurement

One test fails, and it is the same one every time:

duckdb::upsert_retention_memory::upsert_with_retention_under_query_load_does_not_grow_memory::case_1_single_column

Job conclusions for DuckDB Tests (release), read per run via gh api .../actions/runs/<id>/jobs:

run	PR	branch	carries this change?	result
37870489732	#87	fix/sqlite-rows-error	no	failure
37871863462	#88	fix/sqlite-interval-datetime	no	failure
37927461670	#88	fix/sqlite-interval-datetime	no	failure
37897037853	#89	fix/duckdb-upsert-keep-last	yes	success
37917557707	#89	fix/duckdb-upsert-keep-last	yes	success

3/3 fail without it, 2/2 pass with it.

Why the obvious confounders don't explain it

• The failing PRs cannot reach DuckDB. fix(sqlite): return the error SQLite raises mid-query instead of ending the rows early #87 touches core/src/sql/arrow_sql_gen/sqlite.rs and core/src/sqlite/federation.rs; fix(sqlite): shift a timestamp by an interval through datetime(), choosing date() by the operand's type #88 touches core/src/sqlite/federation.rs and core/src/sqlite/sqlite_interval.rs. Neither has a DuckDB file in its diff.
• This PR did not edit the test into passing. Its diff is core/src/duckdb.rs, core/src/duckdb/upsert_groups.rs, core/src/duckdb/write.rs — production only. core/tests/duckdb/upsert_retention_memory.rs is untouched here.
• It is not a newer base. fix(sqlite): return the error SQLite raises mid-query instead of ending the rows early #87/fix(sqlite): shift a timestamp by an interval through datetime(), choosing date() by the operand's type #88 sit on b4a86350d5, this branch on ef2e590539. gh api .../compare/b4a86350d5...ef2e590539 → ahead by exactly 1 commit, "build: pin datafusion-federation at fix(sqlite): store every nested column as the JSON text Arrow encodes, and accept a list of any item type #90 (build: pin datafusion-federation at spiceai/datafusion-federation#90 #85)", changing Cargo.toml +1/-1 and Cargo.lock +1/-1. A dependency pin, not a write-path change.
• The test did run in both passing runs — job 113777339566 logs running 26 tests and upsert_retention_memory: starting 60 cycles (50000 rows/cycle, …), then completes. The 3m45s wall time is a warm dependency cache, not a skipped step.

The failure mode matches what this PR describes

#87's run panics at core/tests/duckdb/upsert_retention_memory.rs:694 with:

refresh write failed on cycle 25 of 60 with 262005 rows in the table (key space 300000, memory_limit 256MB).
Error: External error: External error: Unable to insert into duckdb table:
Constraint Error: Duplicate key "id: 263916" violates primary key constraint.

A duplicate key surviving an ON CONFLICT … DO UPDATE is precisely the single-statement conflict-resolution defect this PR cuts apart into per-key statements. The test writes 50,000 rows/cycle into a 300,000 key space, so repeated keys within one statement are near-certain by cycle 25.

Calibration — what this is and is not

This is a strong correlation, not a proof. n is small (5 runs), and this PR's own description establishes the underlying ordering is scheduling-dependent ("the first copy in 9 of 10 runs on the reporter's host, in 1 of 10 runs on mine"), so 2 green runs cannot rule out luck. The error text also varies between the failing runs — #88's 37871863462 aborted with IO Error: Corrupt database file: computed checksum 5381 does not match stored checksum 0 rather than the duplicate-key constraint error, though it failed in the same test and case.

I have pull only on this repository, so I could not re-run any job to extend the sample — every number above comes from runs that already existed. A maintainer re-running DuckDB Tests (release) a few times on #87's head would settle it cheaply.

What this means for the queue

If the correlation holds, merging this PR clears the DuckDB Tests (release) red on both #87 and #88 without either of them changing a line. That is three PRs unblocked by one merge. Noting it here rather than on each so there is one place to argue with the data.

[grokspice]

Retracting my previous comment (6095044070). Its central claim — that the DuckDB Tests (release) failure on #87 and #88 tracks the absence of this change — is wrong, and this PR's own description already said so. Correcting the record before anyone acts on it.

The artifact that refutes it

I sampled only the runs in this repository and missed a sixth run on the fork. The same push produced two runs two seconds apart, on the same commit:

run	repo	head_sha	DuckDB Tests (release)
37917557707	spiceai/datafusion-table-providers	ecf886a0d66e9f8d5a3b7afb042f4c68a8c2850d	success
37917555233	grokspice/datafusion-table-providers	ecf886a0d66e9f8d5a3b7afb042f4c68a8c2850d	failure

Identical SHA, opposite outcomes. The fork run died in the same test with:

what():  {"exception_type":"FATAL","exception_message":"IO Error: Corrupt database file:
computed checksum 5381 does not match stored checksum 0 in block at location 6004326…

A same-commit pass/fail pair is the artifact for flaky, and it is the one piece of evidence my correlation table could not survive. 2 passes out of 3 runs on this commit is not a fix.

It is already tracked, and this PR already said so

spiceai/spiceai#14183 — open since 2026-09-18 — is titled "DuckDB: upsert_retention_memory fails with engine state corruption (checksum-0 blocks, upsert PK violation, allocator bitmask) across unrelated branches". It names both symptoms I treated as one causal story: the checksum-0 block (#88's run and the fork run) and the upsert PK violation (#87's run). This PR's description cites it directly — "dies on the unrelated upsert_retention_memory soak crash, spiceai/spiceai#14183, on every branch of this fork today".

What the record should say

• DuckDB Tests (release) red on fix(sqlite): return the error SQLite raises mid-query instead of ending the rows early #87, fix(sqlite): shift a timestamp by an interval through datetime(), choosing date() by the operand's type #88 and here is #14183, a pre-existing soak-test instability in the integration suite, unrelated to all three diffs.
• Merging this PR will not clear that check on fix(sqlite): return the error SQLite raises mid-query instead of ending the rows early #87 or fix(sqlite): shift a timestamp by an interval through datetime(), choosing date() by the operand's type #88. My "three PRs unblocked by one merge" line was unfounded; please disregard it.
• This PR's actual evidence is the Tests job before/after in the sandbox runs its description cites, which I did not touch and am not disputing.

The process failure was mine and is specific: I built a causal claim from a 5-run sample without first searching for an existing issue covering the failure signature, and without looking for a same-commit rerun — the one check that would have refuted it in a single query. Both were available before I posted.