Skip to content

fix: keep owned WAL merges catching up between bounded passes - #297

Draft
beinan wants to merge 3 commits into
lance-format:mainfrom
beinan:codex/continuous-wal-catchup
Draft

beinan wants to merge 3 commits into
lance-format:mainfrom
beinan:codex/continuous-wal-catchup

Conversation

@beinan

@beinan beinan commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator

Byte-bounded WAL merges could complete a successful pass while leaving historical generations behind, then wait for the next stats sweep or an unrelated new-write trigger. Owned-target triggering also shared the legacy self-merge threshold, making incremental production activation difficult without enabling uncoordinated merges on other tables.

Persist a coalesced continuation request atomically with successful owned execution release. The normal scheduler queues another bounded pass after the current task ends; an empty pass stops continuation. Worker fan-out stays serial, other queued tables get a turn, and failures retain their per-shard backoff. Add OWNED_MERGE_AFTER_GENERATIONS (default 32, 0 disables), sampled from the local shard manifest on the existing flush timer only for explicitly owned targets. Legacy self-merge can remain disabled.

Validation:

  • 17 merge-coordinator tests passed against isolated local etcd, including lost-claim rejection, durable continuation after reconnect, stale request acknowledgement, unchanged failure budgets, and no continuation for empty/failed/maintenance work.
  • Formatting and diff checks passed.
  • Scheduler fairness/backlog regression and server sweeper tests are running.
  • CI and native service staging are pending; production activation is not included in the source change.

Rollout requires compatible masters/workers and completing each table's legacy drain before selecting it. The new count trigger needs an enabled flush timer. It does not remove an existing migration lock or retroactively fence unguarded legacy writes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant