Repository navigation
feat(master): pure demand fold over per-shard watermarks - #344
Merged
Merged
Conversation
First P1 code, implementing docs/design/scheduler-p1-data-model.md §1. `demand.rs` defines the `DemandEvent` and `TableDemand` records and `TableDemand::fold`, a pure function with no etcd dependency. Nothing calls it yet; later PRs wire writers, executors and the scanner to emit events and the planner to fold them. Fold rules: a strictly higher writer epoch replaces the shard but never discards a merged watermark an executor already recorded; a lower epoch is ignored; at equal epoch sealed and merged only move forward; a scan may raise sealed but never lower it (a stale scan must not hide a fresh flush); ignored events do not advance the observed revision. Validation: five unit tests. The order-independence test folds every permutation of a five-event set, with and without each event duplicated, and replays the final state onto itself, asserting identical records each time (spec invariant 5a). Others cover the scan epoch rule, epoch precedence with merged-watermark preservation, aggregates and revision tracking, and wire-format stability with defaults for optional fields. Clippy -D warnings, fmt and typos clean. Refs lance-format#333. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Oct 8, 2026
beinan
added a commit
that referenced
this pull request
Oct 8, 2026
Part of #333 (P1.2). Implements spec §1.1/§1.3 (#343) on top of the fold from #344. ## What Writers publish their shard watermark to `P/demand-events/<hex>/<shard>` on **every** flush, beside the existing `merge-requests` put. A planner can then see demand below the count threshold, on tables this master does not own, and for low-count tails — without any sweep. - **core**: `own_shard_watermark()` reads the writer's own MemWAL manifest → `(shard_id, writer_epoch, max flushed generation, pending count)`. Metadata only. - **merge**: `demand.rs` moves here (shared by server and master). `Coordinator::publish_demand_event` is a monotonic put ordered by `(writer_epoch, sealed_through, merged_through)` under a mod-revision CAS; unchanged watermarks are not rewritten; never deletes. - **server**: `route_merge` publishes after the drain check and before the ownership gate; publish failure never blocks the merge request. New metric `rollout_wal_demand_events_total{result=written|unchanged|failed}`. **Additive.** `merge-requests`, count trigger and self-merge paths are untouched; older masters ignore the key family (review point 5). ## Validation - `publish_is_monotonic_by_epoch_then_sealed_then_merged` (etcd): duplicate / lower sealed / lower epoch / merged regress rejected; sealed, merged, epoch advances and separate shards accepted. - `flush_publishes_shard_demand_below_threshold_and_without_ownership` (etcd): threshold 8, 2 pending, non-owned target → no merge request, exactly one event per shard, folds to the right pending count, unchanged flush does not bump `mod_revision`, new generation advances `sealed_through` at the same epoch. - All 8 sweeper tests, 6 demand tests green against real etcd; clippy `-D warnings`, fmt, typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This was referenced Oct 8, 2026
beinan
added a commit
that referenced
this pull request
Oct 8, 2026
#350) Part of #333 (P1.3). Spec §1.1/§2.2 (#343), on top of #344/#345. ## What A successful WAL catch-up writes `merged_through` per shard **in the same etcd txn that releases ownership**, so demand and ownership change together and a planner never sees a released table whose pending count still includes the generations just merged. - **core**: `rollout_append::merged_watermarks(uri)` — per-shard merged generation from the base table's MemWAL index. - **merge**: `Coordinator::release_with_demand(claim, execution, &[ShardMerged])`. `demand_release_changes` only advances `merged_through`, keeps the writer's `sealed`/`epoch`/`bytes`, creates a fresh executor-sourced record for an unknown shard, and emits no compare/op for a shard already at or past the watermark. `release` is unchanged (delegates with `&[]`). - **master**: `run_as` reads watermarks after a successful `Catchup` and releases with them; read failure logs and releases without demand. Other kinds unchanged. Metric `master_demand_merged_watermarks_total`. Additive; no key family removed; old masters ignore `demand-events/`. ## Validation - `release_changes_only_advance_merged_and_keep_writer_fields` (etcd): advance 15 keeps sealed 20 / epoch 3 / bytes; lower 12 emits nothing; unknown shard → `sealed == merged`; fold → expected pending. - `resident_replicas_merge_from_queue_with_no_workers_or_jobs` extended: after both masters merge, each table has exactly two executor-sourced events whose `merged_through` equals `merged_watermarks(uri)` and which fold to `pending_generations == 0`. - 7 demand tests, 9 master etcd tests (resident + maintenance_execution) green; clippy `-D warnings`, fmt, typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #333 (P1.1). Implements
docs/design/scheduler-p1-data-model.md§1 (#343).What
New
demand.rswithDemandEvent,ShardDemand,TableDemandandTableDemand::fold. Pure, no etcd. Nothing calls it yet — writers/executors/scanner emission and planner folding come in follow-up PRs; this lands the schema and the invariant tests first.Fold rules (spec §1.1, review point 3):
writer_epochreplaces the shard but keeps anymergedwatermark already recorded.Ignored::StaleEpoch.sealed/mergedonly move forward; aScanmay raisesealedbut lowering →Ignored::ScanCannotLower.observed_revision.Validation
fold_is_order_independent_and_idempotent: all 120 permutations of a five-event set, each also with every event duplicated, plus replay-onto-final-state, all yield the identical record (spec test feat: add initial lance-context python/rust scaffold and uv workflow #1 / invariant 5a).scan_may_raise_but_not_lower_without_higher_epoch(spec test ci: add GitHub Actions for python and rust tests #2).-D warnings, fmt, typos clean.🤖 Generated with Claude Code