[grokspice]

Summary

A SQLite table stores a nested column as JSON text (CreateTableBuilder::build_sqlite gives every nested type a JSON column), but only a list of primitives, strings or booleans was accepted and written as JSON:

• SqliteConnection::is_data_type_supported refused a list of anything else, so a dataset with a List(Struct) column — the github connector's comments — failed at load with The field 'comments' has an unsupported data type: List(Struct(...)) (GitHub issues connector doesn't work with duckdb and sqlite accelerators spiceai#3551).
• The prepared-statement path (insert_batch_prepared, the path Spice's accelerator uses) serialized a list item by item with array.value(i), so a null item was written as its type's zero value — ["bug","","sqlite"] for ["bug", NULL, "sqlite"] — and fell back to Arrow's display text for any other item type and for every struct, text the read cast cannot decode.
• The inline-SQL path (InsertBuilder) hit unimplemented!("List to json conversion is not implemented for …") for the same items, and wrote a struct with implicit nulls and a trailing newline, unlike the other path.

Every nested column is now written as the JSON text Arrow's own JSON encoder produces for the row (make_encoder, built once per column per batch), in both paths, and a list or struct is accepted whenever its item type is — except a Duration, which the encoder writes as an ISO 8601 period the reader does not parse, so it is refused at create instead of stored unreadably. Reading the text back is the federation cast's job; spiceai/datafusion-federation#92 teaches it every item type.

Stacked on spiceai/datafusion-federation#92. is_data_type_supported now accepts lists the pinned federation revision cannot decode (its cast_string_to_list stops at primitives), so the nested-column tests below are red on the pinned 2fc4617 and this PR stays a draft until the pin moves to that PR's merge commit.

Changes

• core/src/sql/db_connection_pool/dbconnection/sqliteconn.rs: a list or struct is supported when its items are (is_json_item_supported: anything the table accepts except Duration).
• core/src/sqlite.rs: stores_as_json and nested_json_encoders replace serialize_list_to_json; the per-row match encodes a nested value with the column's encoder (null for a null row) and keeps Arrow's display text only for Union, Dictionary and RunEndEncoded, whose storage this change does not touch.
• core/src/sql/arrow_sql_gen/statement.rs: insert_list_into_row_values_json and insert_struct_into_row_values_json encode through the same encoder with explicit nulls; the serialize_list_values! macro and the unimplemented! arm are gone. (This path also serves MySQL's JSON columns.)
• core/src/sqlite/write.rs: test module nested_columns (lib tests, so make test runs them in CI).

Test plan

• cargo fmt --all -- --check — clean (EXIT=0)
• cargo test -p datafusion-table-providers --features sqlite-bundled,sqlite-federation --lib on the final HEAD, with federation at build: pin datafusion-federation at spiceai/datafusion-federation#94 #92's head — 276 passed; 0 failed, EXIT=0
• cargo test -p datafusion-table-providers --features sqlite-bundled,sqlite-federation --test integration sqlite:: — 21 passed; 0 failed; 2 ignored, EXIT=0
• cargo clippy -p datafusion-table-providers --features sqlite-bundled,sqlite-federation --lib -- -D warnings — clean (EXIT=0)
• CI on ef71560: every Build (*) arm passes; Tests is red exactly as stacked — 486 passed; 4 failed, the 4 being the nested-column round trips failing at the cast with Unsupported list item type: Struct(…) on the pinned federation 2fc4617 (run 37934086334); it turns green when the pin moves to build: pin datafusion-federation at spiceai/datafusion-federation#94 #92. DuckDB Tests (release) died with SIGSEGV inside duckdb::upsert_retention_memory at cycle 40 — the soak flake tracked as DuckDB: upsert_retention_memory fails with engine state corruption (checksum-0 blocks, upsert PK violation, allocator bitmask) across unrelated branches spiceai#14183, which this diff does not touch (no DuckDB code changed)

Red before the fix

nested_columns_round_trip_through_the_sink::case_1_prepared_statements (the same test as nested_columns_round_trip_through_prepared_statements here, then in core/tests) on the base write path (core/src at ef2e590) with the federation fix in place:

thread '...case_1_prepared_statements' panicked at core\tests\sqlite\nested_columns.rs:222:10:
a table with nested columns should be created: External(UnsupportedDataType { data_type: "List(Struct(\"author\": Struct(\"login\": Utf8), \"body\": Utf8))", field_name: "comments" })
test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 21 filtered out

The same tests with this write fix and the base read cast (federation 2fc4617):

the JSON text should decode back into every nested column: UnableToCastColumn { source: CastError("Unsupported list item type: Struct(\"author\": Struct(\"login\": Utf8), \"body\": Utf8)"), column_index: 1, column_name: "comments", from_type: Utf8, to_type: List(...) }
test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 21 filtered out

a_null_item_in_a_list_of_strings_reads_back_null on the base write path:

thread 'sqlite::write::tests::nested_columns::a_null_item_in_a_list_of_strings_reads_back_null' panicked at core\src\sqlite\write.rs:1543:13:
assertion `left == right` failed
  left: "[\"bug\",\"\",\"sqlite\"]"
 right: "[\"bug\",null,\"sqlite\"]"
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 271 filtered out

Tests added (sqlite::write::tests::nested_columns)

• nested_columns_round_trip_through_prepared_statements / …_through_inline_sql — one batch with comments: List(Struct{author: Struct{login}, body}) (two comments, one null body; an empty list; a null list; a body needing JSON escaping), large_comments: LargeList(Struct), tags: List(Utf8) with a null item, matrix: List(List(Int32)), dates: List(Date32), pairs: FixedSizeList(Int32, 2) with a null row and a null item, meta: Struct{n, label} with a null row; asserts the raw read is the JSON text, then exact batch equality after the cast.
• every_leaf_type_round_trips_inside_a_struct_and_a_list_through_prepared_statements / …_inline_sql — a struct with one field per scalar the type check admits (all integer widths, Float32/64, Utf8/LargeUtf8/Utf8View, Boolean, Binary/LargeBinary/BinaryView/FixedSizeBinary, Date32/64, Time32/64 in every unit, Timestamp in every unit with and without a time zone, Decimal128/256) with a value row, an all-null row and a null struct row, and a list of that struct; exact column equality after the cast.
• a_null_item_in_a_list_of_strings_reads_back_null — the stored text is ["bug",null,"sqlite"] / [null] and the cast equals the original.
• a_duration_inside_a_list_or_a_struct_is_refused_at_create — List(Duration) and Struct{Duration} fail create naming the field and its type.
• both_insert_paths_store_the_same_json_text_for_a_struct — {"n":1,"label":"x"}, {"n":null,"label":"y"}, NULL on both paths.

Review gate

• Adversarial review (hygiene §1a companion script, codex): job review-mv0xz00z-shs0t8 on the first HEAD a96a8e9, verdict needs-attention, 1 finding: the newly accepted lists cannot be read back with the committed federation pin 2fc4617 — valid, and the reason this PR is stacked (above); the pin moves before it leaves draft. On the final HEAD ef71560 codex was out of credits: declared skip (receipt attempt=codex result=exit-1).
• /code-review high (read-only sub-agent on the diff): 1 confirmed — List(Duration) / Struct{Duration} were accepted but written as PT1S, which the reader cannot parse → refused at create (is_json_item_supported, test above); 1 low — the inline path stored a struct with implicit nulls and a trailing newline → both paths share the encoder (test above); the stores_as_json doc over-claimed Union/RunEndEncoded → corrected; its leaf-by-leaf trace of the encoder/reader pair is pinned by the leaf-type matrix tests.
• /security-review (read-only sub-agent): no SQL injection on the inline path (sea-query quotes the literal; the JSON encoder escapes ", \ and control bytes including NUL); no new unsafe, dependency or panic path.
• /simplify: applied before the reviews (shared test setup, let … else in the encoder lookup, to_owned); fmt, clippy and the lib tests re-run after it and after each review fix.

Refs spiceai/spiceai#3551

[Copilot]

🟡 Changes recommended

Nested validation still admits incompatible types, and inline struct encoding can introduce quadratic work.

2 open issues

• Avoid rebuilding full-column encoders for each row · New
• Reject unsupported nested unions and wrapper value types · New

What changed in this PR

Updates SQLite nested-column writes to use Arrow’s JSON encoding consistently, preserving nulls and supporting more nested item types.

Changes:

• Expands list and struct validation while rejecting nested Duration values.
• Replaces manual serialization in prepared and inline insert paths.
• Adds nested-type, null-preservation, and round-trip regression tests.

File	Description
core/​src/​sqlite/​write.rs	Adds nested-column regression tests.
core/​src/​sqlite.rs	Uses per-batch JSON encoders for prepared inserts.
core/​src/​sql/​db_connection_pool/​dbconnection/​sqliteconn.rs	Expands nested-type validation.
core/​src/​sql/​arrow_sql_gen/​statement.rs	Uses Arrow JSON encoding for inline inserts.

_{Review effort: Balanced}