fix: preserve SQLITE_NULL type for NULL values in BOOLEAN / DATETIME columns by antoineleclair · Pull Request #883 · canonical/dqlite

antoineleclair · 2026-04-25T13:32:02Z

Fixes #882.

value_type() in src/query.c was remapping NULL values to
DQLITE_BOOLEAN / DQLITE_ISO8601 based on the column's declared type,
so a NULL in a BOOLEAN column was sent on the wire as BOOLEAN(11)
with value 0 (indistinguishable from FALSE) and a NULL in a DATETIME
/ DATE / TIMESTAMP column was sent as ISO8601(10) with "". The
fix returns early when sqlite3_column_type() reports SQLITE_NULL,
matching plain SQLite's behaviour.

Adds a regression test
(test/integration/test_client.c::client/nullPreservedAcrossDeclaredTypes)
that asserts every column of a row of NULLs comes back with
type == SQLITE_NULL, covering BOOLEAN, DATETIME, DATE, TIMESTAMP
as well as the unaffected types as a sanity check.

Running the same code samples that reproduced the issue in #882
shows that all types appear correctly as NULL.

C (null_types_repro.c):

    === SQLite (sqlite3_column_type for each column) ===
      int_col       : NULL                              ok
      text_col      : NULL                              ok
      real_col      : NULL                              ok
      blob_col      : NULL                              ok
      bool_col      : NULL                              ok
      datetime_col  : NULL                              ok
      date_col      : NULL                              ok
      timestamp_col : NULL                              ok

    === dqlite (wire-level type tag from server) ===
      int_col       : NULL                              ok
      text_col      : NULL                              ok
      real_col      : NULL                              ok
      blob_col      : NULL                              ok
      bool_col      : NULL                              ok
      datetime_col  : NULL                              ok
      date_col      : NULL                              ok
      timestamp_col : NULL                              ok

Go (main.go, via go-dqlite):

    === SQLite ===
      int_col       : NULL                       ok
      text_col      : NULL                       ok
      real_col      : NULL                       ok
      blob_col      : NULL                       ok
      bool_col      : NULL                       ok
      datetime_col  : NULL                       ok
      date_col      : NULL                       ok
      timestamp_col : NULL                       ok

    === dqlite ===
      int_col       : NULL                       ok
      text_col      : NULL                       ok
      real_col      : NULL                       ok
      blob_col      : NULL                       ok
      bool_col      : NULL                       ok
      datetime_col  : NULL                       ok
      date_col      : NULL                       ok
      timestamp_col : NULL                       ok

Python (null_types_repro.py, via dqlite-client / dqlite-wire):

    === SQLite (stdlib sqlite3) ===
      int_col       : NULL                              ok
      text_col      : NULL                              ok
      real_col      : NULL                              ok
      blob_col      : NULL                              ok
      bool_col      : NULL                              ok
      datetime_col  : NULL                              ok
      date_col      : NULL                              ok
      timestamp_col : NULL                              ok

    === dqlite (wire-level type tag from server) ===
      int_col       : NULL                              ok
      text_col      : NULL                              ok
      real_col      : NULL                              ok
      blob_col      : NULL                              ok
      bool_col      : NULL                              ok
      datetime_col  : NULL                              ok
      date_col      : NULL                              ok
      timestamp_col : NULL                              ok

When encoding query results, value_type() remaps certain column types (BOOLEAN, DATETIME) to dqlite-specific type codes. However, it was also doing this for NULL values, causing them to be encoded as their "zero" values (0 for BOOLEAN, "" for DATETIME) instead of NULL. This makes it impossible for clients to distinguish: - BOOLEAN NULL vs FALSE - DATETIME NULL vs empty string SQLite's sqlite3_column_type() correctly returns SQLITE_NULL for NULL values regardless of declared column type. This change preserves that behavior by returning early when the value is NULL. Example demonstrating SQLite's behavior: CREATE TABLE t (b BOOLEAN, d DATETIME); INSERT INTO t VALUES (NULL, NULL); INSERT INTO t VALUES (0, ''); -- sqlite3_column_type() returns: -- Row 1: NULL (5), NULL (5) <- both are SQLITE_NULL -- Row 2: INTEGER (1), TEXT (3) <- actual types Before this fix, dqlite would return: -- Row 1: BOOLEAN (11) with 0, ISO8601 (10) with "" -- Row 2: BOOLEAN (11) with 0, ISO8601 (10) with "" After this fix: -- Row 1: NULL (5), NULL (5) -- Row 2: BOOLEAN (11) with 0, ISO8601 (10) with ""

Export null markers for nullable Boolean and date/time columns so values changed by dqlite can be restored to nil before serialization. The original reason that forces us to do this is a bug on dqlite (PR to address it): canonical/dqlite#883

#23283 _Note/TL;DR: The real fix is upstrem [canonical/dqlite#883](canonical/dqlite#883), which preserves `SQLITE_NULL` before applying declared-type conversions for Boolean and date/time columns. Landing and consuming that fix requires new dqlite and go-dqlite releases. This change provides a Juju-side workaround in the meantime by exporting companion `IS NULL` markers and using them to restore the original nil values before serialization._ Dqlite determines the wire type of `BOOLEAN`, `DATETIME`, `DATE`, and `TIMESTAMP` results from the column's declared type, even when the stored value is SQL `NULL`. For nullable Booleans, this causes `database/sql` to receive `false` instead of `nil`. Model export then serializes `*bool(false)` rather than a nil pointer. For alive filesystems and volumes, importing that value violates the following constraint: ```sql CHECK (obliterate_on_cleanup IS NULL OR life_id <> 0) ``` Update the generated export queries to select both the column value and a companion null marker: ```sql SELECT &StorageFilesystem.*, sf.obliterate_on_cleanup IS NULL AS &nullableStorageFilesystem.obliterate_on_cleanup_is_null FROM storage_filesystem AS sf ``` SQLair reads the exported rows and their null markers together. Before returning the payload, the exporter restores the corresponding pointer to nil whenever the marker is true. The generator applies this handling to every nullable BOOLEAN, DATETIME, DATE, and TIMESTAMP column. The current model schema contains 41 such fields. Null markers remain internal to the export state and are not included in the serialized payload. Other column types and tables without affected nullable columns retain their existing export queries. The generated exporter also clears its result before each transaction attempt so a transaction retry cannot retain rows from an earlier attempt. ## QA steps The commands below assume: - juju_40 is built from this 4.0 branch. - juju_41 is a 4.1 client. - LXD is available as the localhost cloud. Bootstrap the patched 4.0 source controller and a 4.1 target controller: ``` juju_40 bootstrap localhost src40 --build-agent juju_41 bootstrap localhost dst41 ``` Create the source model and an LXD storage pool: ``` juju_40 add-model -c src40 fs-export localhost juju_40 create-storage-pool -m src40:fs-export qa-pool lxd ``` Deploy a charm with filesystem storage: ``` juju_40 deploy -m src40:fs-export \n juju-qa-dummy-storage storage \n --base ubuntu@24.04 \n --storage single-fs=qa-pool,1G ``` Wait until storage/0 is active and idle and single-fs/0 is attached: ``` juju_40 status -m src40:fs-export --storage ``` Write a sentinel file to the attached filesystem: ``` juju_40 exec -m src40:fs-export --unit storage/0 -- \n sh -c 'printf nullable-export-qa | sudo tee /srv/single-fs/sentinel' ``` Confirm the file before migration: ``` juju_40 exec -m src40:fs-export --unit storage/0 -- \n cat /srv/single-fs/sentinel ``` Migrate the model: ``` juju_40 migrate src40:fs-export dst41 ``` Wait for the model, unit, and filesystem to become available on the target: ``` juju_41 status -m dst41:fs-export --storage ``` Verify the filesystem data survived: ``` juju_41 exec -m dst41:fs-export --unit storage/0 -- \n cat /srv/single-fs/sentinel ``` Expected output: ``` nullable-export-qa ``` Confirm that the model moved between controllers: ``` juju_40 models -c src40 juju_41 models -c dst41 ``` Inspect the controller and model logs: ``` juju_40 debug-log -m src40:controller --replay juju_41 debug-log -m dst41:controller --replay juju_41 debug-log -m dst41:fs-export --replay ``` The migration should complete without an obliterate_on_cleanup constraint failure. The migrated filesystem should remain attached and contain the sentinel file. ## Links **Issue:** Fixes #23259. **Jira card:** [JUJU-10342](https://warthogs.atlassian.net/browse/JUJU-10342) [JUJU-10342]: https://warthogs.atlassian.net/browse/JUJU-10342?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

marco6

Thanks for this. Approved.

Just a note which is very generic: SQLite has no support for anything like BOOLEAN or DATE/DATETIME. It is a type that simply doesn't exist in SQLite.

The original dqlite author decided to model this as he was basically modelling everything from the Go driver. From my PoV this was a mistake.

Still, it's better if the NULLability is preserved even in those weird cases as we really can't change the format now.

nvinuesa mentioned this pull request Sep 11, 2026

fix(modelmigration): preserve null values in model exports juju/juju#23283

Merged

marco6 approved these changes Sep 29, 2026

View reviewed changes

gregsadetsky mentioned this pull request Oct 5, 2026

SELECTing a non-TEXT value from a TIMESTAMP column aborts the server + cascades. #912

Open

Navigation Menu

Repository navigation

fix: preserve SQLITE_NULL type for NULL values in BOOLEAN / DATETIME columns - #883

fix: preserve SQLITE_NULL type for NULL values in BOOLEAN / DATETIME columns#883
antoineleclair wants to merge 1 commit into
canonical:mainfrom
antoineleclair:main

antoineleclair commented Apr 25, 2026

Uh oh!

marco6 left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

antoineleclair commented Apr 25, 2026

Uh oh!

marco6 left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants