Skip to content

Add SQLite backend - #864

Open
phdoerfler wants to merge 7 commits into
typelevel:mainfrom
phdoerfler:topic/sqlite-backend
Open

Add SQLite backend#864
phdoerfler wants to merge 7 commits into
typelevel:mainfrom
phdoerfler:topic/sqlite-backend

Conversation

@phdoerfler

@phdoerfler phdoerfler commented Jul 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Adds SQLite as backend. SQLite does not support lateral joins so SqlMapping had to be modified accordingly. This work was partially done by Claude but it was kept on a tight leash and critically reviewed both from Claude itself and me. Care has been taken not to deviate too much from the other backend implementations despite needing to work around the lateral join limitation.
This also opens the door for a H2 implementation which suffers from the same limitation as SQLite.

⚠️ This changes the API and thus affects backends as a result of the lateral join business (new abstract members, a rename):
It adds two abstract members to SqlMappingLike (normalizeOffsetLimit, unionBranchToFragment) and removes the superseded defaultOffsetForSubquery, which MiMa flags as three binary-compatibility problems — hence the tlBaseVersion bump. If that is an issue, a source- and binary-compatible variant is also possible which defaults on the new hooks plus a deprecated no-op shim for the old one.

⚠️ A bigger change was necessary in the MS SQL backend specifically to avoid an if-else in SqlMapping that would have been necessary for sqlite. This syntactically changes the generated SQL but it should be semantically identical:

  • OFFSET 0 ROWS after an ORDER BY at the root should return the same rows in the same order. It's also the same clause MSSQL already emits for every ordered subquery, so it's not entirely new. And of course the tests being green indicate that it does not break anything covered by the tests.
  • The alternative designs are uglier. To keep MSSQL's root SQL byte-identical, the mapping would need to know "am I being rendered as a subquery or the root?" — that means, for instance, a boolean context parameter (normalizeOffsetLimit(query, inSubquery)). I wanted to avoid having a flag-driven API.

Personally, I think that tradeoff is worth it but I could also see how we would not want to change the SQL generated by existing backends so let me know what you think.

I also increased the timeout for munit because on my machine it did sporadically run into the default 30s timeout and that became old quickly.

@phdoerfler
phdoerfler force-pushed the topic/sqlite-backend branch 5 times, most recently from 021ba0d to bd80fa9 Compare July 12, 2026 19:49
@phdoerfler
phdoerfler marked this pull request as ready for review July 12, 2026 19:58
@phdoerfler phdoerfler mentioned this pull request Jul 13, 2026
Open
@phdoerfler
phdoerfler force-pushed the topic/sqlite-backend branch from 5236471 to 8ebaad9 Compare July 14, 2026 12:30
This was referenced Jul 14, 2026
Open
Open
@phdoerfler
Bump tlBaseVersion to 0.30
a5de057 
Latest tag is v0.29.0; the SQLite backend work also changes the
SqlMappingLike dialect API (new abstract members, defaultOffsetForSubquery
renamed to normalizeOffsetLimit), so the next release must start a new
binary-incompatible minor line for MiMa.
@phdoerfler
Pin forked test JVMs to UTC
4fab3d6 
Datetime tests in the MSSQL suite (MovieSuite's `moviesShownBetween` and
computed-field cases) fail on a developer machine in a non-UTC time zone,
though they pass in CI. The MSSQL test codec encodes an OffsetDateTime
argument via a zone-naive `java.sql.Timestamp`, and mssql-jdbc binds it
into a DATETIMEOFFSET parameter using the ambient JVM time zone. On a
UTC+2 host the filter bound shifts by two hours, admitting an extra row.
Postgres is immune (native `Meta[OffsetDateTime]`); Oracle uses the same
codec but ojdbc binds it without the shift; CI runs in UTC so it never
surfaces there.
@phdoerfler
Add SQLite backend (grackle-doobie-sqlite)
8153847 
A new Doobie-based SQL backend for SQLite, mirroring the doobie-oracle/
doobie-mssql module shape: a single dialect mapping supplying the
SqlMappingLike dialect hooks, plus a test harness wiring up the shared
sql-core suites against a fresh temp-file SQLite database per test
suite (no docker service needed, since SQLite runs in-process).

Two dialect-specific choices, documented inline: LIMIT/OFFSET uses
SQLite's legacy comma form (`LIMIT offset, limit`), since the shared
query builder's fixed offset-then-limit render order is incompatible
with SQLite's OFFSET-must-follow-LIMIT grammar; and LIKE
case-sensitivity relies on a connection-level `case_sensitive_like`
pragma paired with `UPPER()`-normalization for case-insensitive matches.

SQLite has two grammar gaps no pre-existing dialect hook could route
around, addressed in sql-core with existing dialects' behavior
unchanged:

- No correlated FROM-clause subqueries (no LATERAL equivalent).
  `SqlSelect.nest()` embeds a parent-constraint equality predicate into
  a nested subquery's own WHERE clause - redundant with the enclosing
  JOIN ... ON clause, and only useful as an optimization for genuinely
  lateral-evaluated subqueries (Postgres/Oracle LATERAL, MSSQL
  CROSS/OUTER APPLY). On a dialect with no lateral join at all, that
  predicate references a column out of scope and fails at runtime with
  "no such column". A derived capability, `supportsLateralJoin` (false
  exactly when `mkLateral` has no lateral form to render and answers
  `NotLateral`, as SQLite's does), makes `addFilterOrderByOffsetLimit`
  skip embedding it.

- No parenthesized UNION branches. `SqlUnion.toFragment` used to wrap
  each UNION ALL branch in parentheses unconditionally; SQLite's
  compound-select grammar forbids that. Branch rendering is now a
  dialect hook, `unionBranchToFragment`, which the existing dialects
  implement with the previous parenthesizing behavior and SQLite
  implements as the identity - safe for any number of branches since
  grackle only ever uses UNION ALL, a purely associative bag union.
  `DoobieSqliteMappingLike.encapsulateUnionBranch` is also extended to
  wrap branches carrying their own offset/limit (not just order) into
  a derived-table subquery, since a branch-level LIMIT is rejected by
  SQLite regardless of parenthesization.

Fixing the LATERAL gap exposed a real correctness bug, also fixed
here: `addFilterOrderByOffsetLimit` has two "Case 1" fast paths that
apply LIMIT/OFFSET directly to a query built from pre-existing joins,
trusting `oneToOne && predIsOneToOne` to mean one row per key. That's
only guaranteed when a lateral-evaluated subquery produced those joins
(at most one contribution per outer row); a join chain containing a
nested one-to-many hop (e.g. a windowed/limited grandchild list) can
produce both a real match row and a NULL-extended LEFT JOIN row
sharing the same key, and an ORDER BY/LIMIT on top with no
de-duplication picks whichever the database's tie-breaking favors.
Both fast paths are now gated behind `supportsLateralJoin`, safe to
skip only when the dialect genuinely evaluates subqueries laterally,
or when there's provably no risk (no joins, or no offset/limit at all).

The offsetDateTime test codec normalizes to UTC on decode: SQLite has
no native timestamptz, so it stores and returns whatever literal
offset is written in the seed data, whereas Postgres/Oracle/MSSQL's
drivers hand back a UTC-normalized value regardless of storage;
without normalizing, otherwise-correct results failed pure
string-equality against the shared UTC ("Z")-formatted expected-JSON
fixtures.

Also applies the shared utcTestSettings (pinning the forked test JVM to
UTC) to doobiesqlite for consistency with the other backends, even
though SQLite's own datetime handling is UTC-normalized independent of
host time zone.
@phdoerfler
Raise munit timeout for database-backed test suites
9344084 
munit-cats-effect's 30s default is not enough headroom for the
database-backed suites on a constrained machine: sbt runs each
backend's tests in its own forked JVM, several of which contend for
one docker daemon, and under that load individual tests have been
observed to exceed 30s and fail with spurious TimeoutExceptions
unrelated to what they test.

Introduce a shared SqlDatabaseSuite base trait in sql-core's test
scope, extended by DoobieDatabaseSuite and SqlPgDatabaseSuite (a
single base rather than per-trait overrides, since
DoobiePgDatabaseSuite mixes in both and separate overrides of
munitIOTimeout would conflict), raising the timeout to 2 minutes for
every backend suite.
@phdoerfler
Normalize offset/limit at render time for every select
6e1d0e8 
defaultOffsetForSubquery supplied dialect-required offset/limit
defaults only for selects rendered as subqueries, leaving the true
query root unnormalized. That gap was invisible for Postgres and
Oracle (identity implementations) and harmless for MSSQL (ORDER BY
without OFFSET is legal at the root, unlike inside a derived table),
but fatal for SQLite's comma-form LIMIT clause: a flat root query with
an offset and no limit - the one shape whose offset survives to the
top level unwrapped - rendered a dangling "LIMIT n," and failed at
runtime with an incomplete-input error.

Rename the hook to normalizeOffsetLimit and apply it once in
SqlSelect.toFragment, so every rendered select is normalized wherever
it appears; SubqueryRef no longer applies it separately. The only
rendering change for existing dialects is that MSSQL root-level
ordered queries now carry a redundant-but-legal "OFFSET 0 ROWS".

Adds a shared regression test covering the flat root-offset shape,
run by all backends.
@phdoerfler
Fix NULL placement in SQLite ORDER BY rendering
74fb76a 
SQLite sorts NULLs low by default (first in ASC, last in DESC), but
orderToFragment carried the Postgres conditional, which only emits an
explicit NULLS FIRST/LAST clause on the cases that deviate from a
nulls-high default. On SQLite that renders the explicit clause exactly
where the engine default already matches and stays silent on the two
cases that need correcting, so orderings requesting the non-default
placement (ascending nulls-last, descending nulls-first) put NULLs at
the wrong end. Emit the clause on the mirror-image cases instead - the
same polarity correction the MSSQL dialect makes, expressed with
SQLite's native syntax.

No existing fixture could observe this: grackle core re-sorts fetched
rows in memory (stripCompiled preserves OrderBy), so the SQL-side
placement only matters when it decides which rows survive a LIMIT cut,
and no shared fixture orders a NULL-containing nullable column. The new
NullOrderingSuite pins all four ascending/nullsLast combinations
through a limit for exactly that reason.
@phdoerfler
phdoerfler force-pushed the topic/sqlite-backend branch from 8ebaad9 to 74fb76a Compare July 17, 2026 13:42
@phdoerfler
Remove now-redundant local NullOrderingSuite
11f4169 
Superseded by the shared SqlNullOrderingMapping/SqlNullOrderingSuite
in sql-core, landing via typelevel#870. This branch will pick
that up once it rebases past typelevel#870's merge.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@phdoerfler