Skip to content

Fix numeric Datalog range indexing for actor due-work - #36

Merged
bjacobso merged 1 commit into
mainfrom
conductor/portable-app-indexes-for-triplex
Sep 27, 2026
Merged

bjacobso merged 1 commit into
mainfrom
conductor/portable-app-indexes-for-triplex

Conversation

@bjacobso

@bjacobso bjacobso commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Runfold's actor due-work query filters ?due <= now, orders by due, and limits the result to 128. Triplex compiles that comparison through COALESCE(value_number, value_datetime), which cannot use the existing number-only range index. Current Triples.query also pins a snapshot, making Runfold's live-only runfold_actor_due workaround ineligible for that read path.

This PR adds the shared idx_attr_numeric expression index, including numeric history, through SQL migration v2 and Cloudflare migration v3. Baseline migrations stay unchanged; the current index lists include it for bulk-load rebuilding. Number/datetime comparison, projection distinctness, ordering, and temporal visibility remain unchanged. No compiler rewrite or actor/query-time DDL is needed.

Scope

Takes the smaller core-index option: Runfold's concrete requirement needs no consumer index declaration. The backend-portable query and existing provisioning/migration APIs remain the consumer contract. Arbitrary portable consumer index declarations are explicitly deferred in the docs, including their remaining use cases and required backend/migration semantics. KV gets conformance coverage, not a new range execution strategy.

The documented reproduction also distinguishes the logical limit: 128 from the current default page size of 100; { pageSize: 128 } requests all 128 rows.

Actual plan evidence

Fixture: 10,000 pending items (160 due, 9,840 future), alternating number/datetime storage, 80 retracted numeric facts, and 10,000 pending markers. Both databases were analyzed; no planner hints or forced index settings were used. Tests explain actual public page SQL as well as live queryAll SQL.

Backend Before After
SQLite snapshot page idx_attribute_history attribute/position search idx_attr_numeric (attribute=? AND <expr><?)
PostgreSQL snapshot page 10,080 due-attribute candidates; 9,920 removed by filters Expression bitmap index range reads 240 candidates; history filter removes 80

Both backends still sort/deduplicate. PostgreSQL may scan pending markers for its join. This bounds future-due candidate lookup, not total query work or the cost of a dense due/history population. Reproduction commands and limitations are documented in docs/datalog-performance.md.

Validation

  • pnpm check passed: formatting, lint, types, checked docs, unit/integration tests, builds.
  • pnpm pack:check passed: 10 tarballs and clean external consumer install.
  • PG_TEST_URL=… pnpm test:postgres:integration passed all 20 tests on PostgreSQL 17.10.
  • Shared KV/SQLite/PostgreSQL regression covers mixed numeric aliases, all ordered comparison boundaries, limits, retractions, validity windows, and continuation after retraction.
  • SQLite/PostgreSQL plan tests assert the actual numeric range index condition; existing-v1 upgrade tests verify data preservation and repeat application. Cloudflare upgrade/conformance tests also pass.
  • Final focused conformance and checked-doc tests passed after tightening boundary assertions.

Rollout

Apply SQL v2 (Cloudflare v3) once per database/schema before removing Runfold's workaround DDL. Triplex does not drop host-owned indexes; existing runfold_actor_due removal remains a separate host migration. Convenience layers apply pending migrations at startup; host-owned/unmigrated runtimes require deployment tooling to apply them.

Ordinary index creation scans existing facts and can block writes. It adds index storage/write cost for numeric history. The docs cover serialized migration execution, PostgreSQL concurrent prebuilding, name/validity checks, and compatibility with existing supported baseline databases. No Runfold files are changed.


Open workspace in Conductor

@bjacobso
bjacobso merged commit 20bc2a4 into main Sep 27, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant