Fix numeric Datalog range indexing for actor due-work - #36
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runfold's actor due-work query filters
?due <= now, orders by due, and limits the result to 128. Triplex compiles that comparison throughCOALESCE(value_number, value_datetime), which cannot use the existing number-only range index. CurrentTriples.queryalso pins a snapshot, making Runfold's live-onlyrunfold_actor_dueworkaround ineligible for that read path.This PR adds the shared
idx_attr_numericexpression index, including numeric history, through SQL migration v2 and Cloudflare migration v3. Baseline migrations stay unchanged; the current index lists include it for bulk-load rebuilding. Number/datetime comparison, projection distinctness, ordering, and temporal visibility remain unchanged. No compiler rewrite or actor/query-time DDL is needed.Scope
Takes the smaller core-index option: Runfold's concrete requirement needs no consumer index declaration. The backend-portable query and existing provisioning/migration APIs remain the consumer contract. Arbitrary portable consumer index declarations are explicitly deferred in the docs, including their remaining use cases and required backend/migration semantics. KV gets conformance coverage, not a new range execution strategy.
The documented reproduction also distinguishes the logical
limit: 128from the current default page size of 100;{ pageSize: 128 }requests all 128 rows.Actual plan evidence
Fixture: 10,000 pending items (160 due, 9,840 future), alternating number/datetime storage, 80 retracted numeric facts, and 10,000 pending markers. Both databases were analyzed; no planner hints or forced index settings were used. Tests explain actual public page SQL as well as live
queryAllSQL.idx_attribute_historyattribute/position searchidx_attr_numeric (attribute=? AND <expr><?)Both backends still sort/deduplicate. PostgreSQL may scan pending markers for its join. This bounds future-due candidate lookup, not total query work or the cost of a dense due/history population. Reproduction commands and limitations are documented in
docs/datalog-performance.md.Validation
pnpm checkpassed: formatting, lint, types, checked docs, unit/integration tests, builds.pnpm pack:checkpassed: 10 tarballs and clean external consumer install.PG_TEST_URL=… pnpm test:postgres:integrationpassed all 20 tests on PostgreSQL 17.10.Rollout
Apply SQL v2 (Cloudflare v3) once per database/schema before removing Runfold's workaround DDL. Triplex does not drop host-owned indexes; existing
runfold_actor_dueremoval remains a separate host migration. Convenience layers apply pending migrations at startup; host-owned/unmigrated runtimes require deployment tooling to apply them.Ordinary index creation scans existing facts and can block writes. It adds index storage/write cost for numeric history. The docs cover serialized migration execution, PostgreSQL concurrent prebuilding, name/validity checks, and compatibility with existing supported baseline databases. No Runfold files are changed.
Open workspace in Conductor