Repository navigation
feat(rollout): expose flush-on-add in SDKs - #4
Draft
KaiqiJinWow wants to merge 128 commits into
Draft
KaiqiJinWow wants to merge 128 commits into
KaiqiJinWow wants to merge 128 commits into
Conversation
KaiqiJinWow
force-pushed
the
feat/rollout-client-flush
branch
2 times, most recently
from
October 8, 2026 20:21
640a335 to
9f32b9e
Compare
…pressure (lance-format#349) Part of lance-format#333. Second-round review clarifications (credit: production review). | Point | Change | |---|---| | Promotion to class 1 does not guarantee service (class 1 still score-sorted; hot tables' critical merges are class 1 too) | Class 1 sorts **oldest-first** by `wait_since_ms`; score applies to classes 2–5 only. Invariant 4 restated. Starvation is then only possible when no executor is ever feasible — a capacity fault with its own metric. | | etcd revision vs manifest version are not comparable | Watermark ordering uses only the shard's own `(writer_epoch, sealed_through)`. Scan may raise at equal epoch, replace only with higher epoch, never lower at equal epoch regardless of write time. `observed_revision` is freshness bookkeeping only. Design §4.1 and P1 spec §1.1. | | Stretching flush interval converts backlog into memory pressure / un-flushed windows | §4.7 rewritten: memtable budget and flush interval remain hard caps; the `critical` lever is proportional admission delay. Policy gate `backpressure: off \| metric \| delay_admission`. Backlog C7a corrected. | Each is now a required P1 test (spec §7 tests 6–8). Docs only; `typos` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…/403 (lance-format#347) Part of lance-format#333. Fixes two defects in lance-format#338 found in review (credit: production review). ## Problems 1. **Mixed-version regression (P1).** lance-format#338 changed the exact string old masters match on. A stale preparation produced by a new executor and classified by an old master → `DataOrConfiguration`, one-hour cooldown — the bug lance-format#338 set out to fix. Reproduced with old vs new classifier. 2. **Code outranks a 403 (P2).** `classify` checked the code before HTTP status, so `HTTP 403: … [LC_STALE_PREPARATION]` → `Retryable`. The lance-format#338 description claimed a test for this; there was none. `corrected()` also had its own rule instead of sharing `classify`'s. ## Fixes - Emission **off by default**; `MAINTENANCE_ERROR_CODES=1` enables it after every master/executor runs a code-aware reader. `tag()` returns the exact legacy string when off; `tag_forced()` for tests. - Code moved to a **suffix**; the legacy text matcher tolerates the suffix (defence in depth for a reader with only the text path). - `classify` order: ownership → explicit 401/403 → code → text. `corrected()` delegates to `classify` and only softens a stored `DataOrConfiguration`. - `docs/concurrent-indexing.md` documents the rollout order. ## Validation - New: 401/403 wrapping the code in either position → `DataOrConfiguration`, and `corrected()` leaves it; emission off by default; byte-exact legacy string still matched; tagged string recognised by text path alone. - `lance-context-api` 1/1, `lance-context-merge` failure 7/7, `lance-context-core` prepared/stale 9/9; clippy `-D warnings`, fmt, typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…rom running (lance-format#348) Part of lance-format#333. Fixes a defect in lance-format#339 found in review (credit: production review). ## Problem `master_task_pool_in_use` was set once per poll iteration before any permit was taken. A claim blocked on etcd for minutes — the case the gauge is for — kept showing the pre-claim value, possibly zero. ## Fix Gauge updated at the permit boundaries, labelled by holder: `state="claiming"` during the etcd claim, `state="running"` from task spawn until permit drop. Runbook row updated: running at cap ⇒ capacity; claiming high ⇒ slow/blocked admission scan; both low ⇒ nothing eligible. ## Validation Clippy `-D warnings`, fmt, typos clean; `reserved_resident_merge_finishes_while_legacy_pool_is_saturated` passes locally against etcd (exercises the poller path). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
lance-format#350) Part of lance-format#333 (P1.3). Spec §1.1/§2.2 (lance-format#343), on top of lance-format#344/lance-format#345. ## What A successful WAL catch-up writes `merged_through` per shard **in the same etcd txn that releases ownership**, so demand and ownership change together and a planner never sees a released table whose pending count still includes the generations just merged. - **core**: `rollout_append::merged_watermarks(uri)` — per-shard merged generation from the base table's MemWAL index. - **merge**: `Coordinator::release_with_demand(claim, execution, &[ShardMerged])`. `demand_release_changes` only advances `merged_through`, keeps the writer's `sealed`/`epoch`/`bytes`, creates a fresh executor-sourced record for an unknown shard, and emits no compare/op for a shard already at or past the watermark. `release` is unchanged (delegates with `&[]`). - **master**: `run_as` reads watermarks after a successful `Catchup` and releases with them; read failure logs and releases without demand. Other kinds unchanged. Metric `master_demand_merged_watermarks_total`. Additive; no key family removed; old masters ignore `demand-events/`. ## Validation - `release_changes_only_advance_merged_and_keep_writer_fields` (etcd): advance 15 keeps sealed 20 / epoch 3 / bytes; lower 12 emits nothing; unknown shard → `sealed == merged`; fold → expected pending. - `resident_replicas_merge_from_queue_with_no_workers_or_jobs` extended: after both masters merge, each table has exactly two executor-sourced events whose `merged_through` equals `merged_watermarks(uri)` and which fold to `pending_generations == 0`. - 7 demand tests, 9 master etcd tests (resident + maintenance_execution) green; clippy `-D warnings`, fmt, typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
lance-format#354) Part of lance-format#333. Fixes the runbook finding from the third review round (credit: production review). §5 told operators to "upgrade or **stop** that contender" when compaction cannot get its commit turn. A continuous merger that keeps winning on a hot table is working as designed and progressing; stopping it trades a bounded fairness gap for WAL backlog and a fenced recovery. Rewritten: states first that this is not a merger fault and never to stop/drain/restart a progressing merge; remedies are scheduling levers only (upgrade at next rollout, raise commit wait, remove from prepare targets and build in a quiet window); names `MERGE_DRAIN_TARGETS` as the wrong tool. §3: long **and** progressing is never a reason to act. Docs only; typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ath (lance-format#351) Part of lance-format#333. Fixes review finding **1 [P1]** on lance-format#345 (credit: production review). ## Problem lance-format#345 awaited the manifest read + etcd put inline in `route_merge`, *before* the merge request. A hung publish ate the flush pass timeout, skipped that table's merge request, and delayed every later table in the pass — up to 150 s/table with defaults. Telemetry was on the scheduling path, which P1 forbids. ## Fix Publish is spawned and never awaited: 5 s timeout, process-wide semaphore of 4 in-flight publishes, drop-and-count when full (the next flush republishes an absolute watermark, so nothing is lost). All outcomes on `rollout_wal_demand_events_total{result=written|unchanged|no_manifest|failed|timeout|dropped}`. `Sweepable: Clone` (all implementors are `Arc<RwLock<_>>`). ## Validation - **New** `hung_demand_publish_never_delays_flush_or_merge_request`: a store whose watermark read is `pending()` forever → flush pass returns in < 2 s, merge request still issued, 8 further passes while still hung return in < 2 s total. - `flush_publishes_shard_demand_…` adapted to poll for the detached write (10 s bound). - 9/9 sweeper tests against real etcd; clippy `-D warnings`, fmt, typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…mat#353) Part of lance-format#333. Fixes review finding **5 (panic path)** on lance-format#348 (credit: production review). ## Problem `running.increment` before spawn, `running.decrement` after `run_task` returns. A panic in `run_task` unwinds past the decrement: permits are released by `Drop`, the gauge is not, so `master_task_pool_in_use{state="running"}` reads as consumed forever. ## Fix `GaugeHold(Gauge)` with `Drop` → decrement, held across `run_task`. Three lines. ## Validation Type-level: the decrement is now in `Drop`, which runs on both normal return and unwind. No new test — asserting on a `metrics::Gauge` value needs a recorder harness this crate's tests do not have; the existing poller test (`reserved_resident_merge_finishes_while_legacy_pool_is_saturated`) passes against etcd, clippy `-D warnings` and fmt clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…t IO (lance-format#355) Part of lance-format#333. Fixes review finding **6** on lance-format#340 (credit: production review): the earlier test used hand-placed `checkpoint()`s, a one-row real build, and never asserted the build outlived the idle timeout. ## Scenarios (real BTree preparation, 64 rows, 3 s idle timeout) | | Storage | Expected | Guard against vacuity | |---|---|---|---| | (a) | every data/index op sleeps 1.2 s | completes | asserts ≥ 3 ops intercepted **and** total time > idle timeout | | (b) | parked after first op | cancelled with exact stall error in `[IDLE, IDLE+4)` | asserts shim saw IO | | (c) | healthy table concurrent with (b) | completes; not cancelled; does not keep (b) alive | — | ## Support code - `MergeWriteScope::with_preparation_authorizer_and_io_shim`: layers a shim **beneath** the progress wrapper, so the watchdog counts the shimmed store's completed IO exactly as real storage. - `preparation_io::test_support::{Throttle, ThrottleWrapper}` (`#[doc(hidden)]`). - Uses Lance's `file-object-store://` scheme: local directory through the ObjectStore API so wrappers apply as on cloud stores (plain `file` paths take the direct IO fast path and bypass them — which is why the first attempt saw 0 ops). ## Mutation check Removing read-chunk + `get_ranges` progress accounting from `ProgressStore` fails (a) with `slow-but-alive real build was cancelled` after ~4 s. Restored. ## Validation New test + `slow_but_alive_…` + `prepared_index_task_…` pass against etcd (12 s); clippy `-D warnings`, fmt, typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ance-format#358) Part of lance-format#333. Fixes review finding **3 [P1]** (round 4) on lance-format#354 (credit: production review). §5 advised removing a hot table from `COMPACTION_PREPARE_TARGETS` / `INDEX_PREPARE_TARGETS` to defer its maintenance. Verified against `eligibility::claim_shape` and `run_task`: a table not in the prepare list takes the **legacy path**, which acquires the table write lock *before* compacting/indexing — the next task blocks WAL publication for the whole build. Strictly worse than repeated preparation expiry. Rewritten: no side-effect-free per-table lever exists today; both tempting levers named with why they are wrong; fleet-wide levers that do exist (`COMPACTION_INTERVAL_SECS`, `quiet_hours`, `MIN_FRAGMENTS`, manual enqueue in a quiet window); per-table `kinds_allowed` is design §4.8, not yet built. Docs only; typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ted (lance-format#360) Part of lance-format#333. Closes the `claiming` cancellation gap noted in review round 4 (credit: production review). The `claiming` gauge was incremented before the etcd claim and decremented after. A poller aborted mid-claim (drain, shutdown, test teardown) skipped the decrement. Same `GaugeHold` Drop guard as lance-format#353, held across the await. Clippy `-D warnings`, fmt clean; poller test passes against etcd. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ut (lance-format#357) Part of lance-format#333. Fixes review finding **2 [P1]** (round 4) on lance-format#350 (credit: production review). Independent of lance-format#352. ## Problems in lance-format#350 1. Watermark read after the watchdog exited and before `release`, with **no deadline** → a stalled storage read holds a completed merge's ownership. 2. Release txn compared every shard's demand key by `mod_revision` → **a normal writer flush on any shard fails the whole release**. Reproduced. 3. 129+ shards → etcd default `--max-txn-ops=128` exceeded. ## Fix - `release` is the plain ownership txn again; `release_with_demand` / `demand_release_changes` removed. - After a successful `Catchup` release: `demand_publish::spawn_merged_demand` — detached, 15 s timeout, 4 slots, drop-and-count, **one monotonic put per shard**, bounded CAS retry on a lost race. Loss/lateness cannot misstate demand (per-dimension join; scan reconciles). - Both publish paths share `join_events` + retry, so the writer path no longer silently returns `Ok(false)` after losing a race either. - `master_demand_merged_watermarks_total{result}`. ## Validation - New etcd test: writer `sealed=25` and executor `merged=18` published **concurrently** on one shard → both succeed, stored `(25, Some(18))`; lower merged no-op; unknown shard → fresh epoch-0 record. - Resident two-master merge test polls for the detached publish; both shards fold to zero pending. - 7 demand, 9 maintenance/resident, 9 sweeper tests green against etcd; clippy `-D warnings`, fmt, typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…wer schema (lance-format#359) Part of lance-format#333. **Supersedes lance-format#352** (which I will close). Fixes review findings **1 [P1], 4, 5 [P2]** (round 4) (credit: production review). Stacked on lance-format#357. ## What lance-format#352 got wrong It ordered marks by `(epoch, value)` — treating a writer epoch change as a numbering restart. Lance keeps generation numbering on a new writer claim (`ShardManifest { writer_epoch: next, ..base }`). So "merged to 40, new writer flushes to 60" is **20 pending**; lance-format#352 said 60 in both orders. ## Fix - `Mark` orders by `(value, epoch)`; epoch is a fence/tiebreak. `pending = sealed − merged`, plain. - Stored record is the per-dimension join; `fold` applies the same join → **rebuild from etcd equals in-memory fold**, now including the age clock, which the stored record carries (`sealed_times`, bounded to 64 lowest unmerged, min per generation). Finding 4: in-memory 10 vs rebuilt 20 before. - `merged_epoch` keeps provenance when merged was reported under a different epoch. - Publishers **refuse to overwrite a stored record from a newer schema** (finding 5: `v=999` was overwritten). Both publish paths share one read-join-CAS with bounded retry. - Spec §1.1 rewritten. ## Validation (15 tests, all green against real etcd) - Named counterexamples: round-3 (both), round-4 (merged 40 / epoch-2 flush 60 ⇒ 20, in every order, and after rebuild). - Writer never forgets merged; lower value from any source is a no-op; marks order by generation with epoch tiebreak. - Age clock: stable across newer flushes, moves correctly on merge, **identical after rebuild**; beyond-cap still exact for counts. - Schema: enforced on fold; stored `v=999` untouched by both publishers (mod_revision unchanged). - Concurrent writer + executor publishes both succeed and both survive. - **Randomized**: 200 × 5-event streams (every permutation) and 50 × 200-event (reverse, shuffle) over 2 shards / 3 epochs / 3 sources — fold order-independent **and** `rebuild(events) == fold(events)`. - Resident two-master merge and 9 sweeper tests pass; clippy `-D warnings`, fmt, typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
lance-format#356) Part of lance-format#333 (P1.4). Spec §1.3, source `scan`. **Stacked on lance-format#359** (rebased from the superseded lance-format#352); merge after lance-format#357 and lance-format#359. ## What - **core**: `all_shard_watermarks()` — every shard's `(shard_id, writer_epoch, max flushed generation, pending)` from its latest manifest. Does **not** enforce the read cap (the scheduler must see an over-cap shard, not be refused). - **master**: `demand_publish::spawn_scan_demand` after each successful observation. Detached and bounded (15 s, 4 slots, drop-and-count), one `scan`-sourced event per shard via `publish_demand_event`'s per-dimension join, so a scan can only raise and never lowers a `merged` it does not know. Metric `master_scan_demand_events_total{result}`. Existing enqueue/retire behaviour untouched (review point 5 / P1 rule). ## Validation - `scan_publishes_every_shard_and_never_lowers_a_watermark` (etcd): two writer shards, no server flush path → one event per shard, fold = 6 pending; executor reports `merged=2` on a shard; a second scan leaves `merged` intact and does not bump `mod_revision`. - 15 scanner unit tests; clippy `-D warnings`, fmt, typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…lance-format#361) Part of lance-format#333 (P1.5). Spec §1.2/§3. The gate (lance-format#359) is merged. ## What Behind `PLANNER_ENABLED` (default **off**): one master wins `P/leader` under a lease and every `PLANNER_RECONCILE_SECS` folds all `demand-events/` into a per-table `P/demand/<hex>` cache via `TableDemand::fold`. Rows for tables with no events are dropped. The cache is a pure function of stored events — never a source of truth. **Shadow only. Places nothing, removes nothing; every existing loop keeps running** (review point 5). Outputs: the cache; `GET /api/v1/scheduler/demand[?target=]` (leader record + per-table pending/bytes/age/shards + full record); metrics `planner_is_leader`, `planner_tables`, `planner_pending_generations_total`, `planner_reconcile_seconds`, `planner_events_skipped_total`, `planner_reconcile_errors_total`. ## Leadership Campaign txn on `Version(P/leader)==0` writes the record **and** a bare `P/leader-token` under one lease. Every cache-write txn compares the token; every reconcile reads it first even with nothing to write, so a deposed leader notices on the next tick. Writes chunked under etcd's 128-op limit. Losers return promptly and re-campaign after 2 s. ## Validation - Unit: `fold_events` groups by table and skips bad key / bad body / future version (counted); hex round-trip. - etcd: two masters campaign → **exactly one leads** (the test detects a dual leader; mutation that ignores the campaign result fails it with `finished=[false, false]`); cache == direct fold; HTTP view shows leader and 42 pending on a two-shard table; forged token → `leadership lost`; deleting a table's events drops its row next tick. - Clippy `-D warnings`, fmt, typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ons (lance-format#362) Part of lance-format#333 (P1.6). Spec §2. ## What Behind `PLANNER_ENABLED`: every master publishes a **leased** `P/executors/<id>` heartbeat — kind, slots per task kind mirrored from the real dispatch pools (general / compact / merge + resident lane), byte budget from the local append budget, a diagnostic `bytes_free_sample`, `draining`. Drain stops the heartbeat and revokes the lease. **Headroom** is a pure fold of heartbeats + `P/assignments/` records: `slots_total − Σ reserved_slots`, `bytes_total − Σ reserved_bytes` over every non-`Released` assignment **including unbound ones**, never the sample (review point 4). An assignment whose executor has no heartbeat still reserves (shown as draining, zero totals) so capacity cannot be double-booked across a failover. Planner loads headroom on every reconcile; `GET /api/v1/scheduler/executors` exposes it. **No assignments are written yet** — `Shadow` state is defined for the next PR. `TaskKind` gains `PartialOrd/Ord/Hash` for the slot maps. ## Validation - Spec test 3: sample says 2 GiB free, 2 GiB reserved (Assigned + Shadow) → `fits` false; Released doesn't count. - Spec test 4: assignment outlives a missing heartbeat. - Draining takes nothing; zero byte budget defers to slots; fold order-independent. - etcd: leased heartbeat with configured slots (merge 3, index 2); shadow assignment drops merge headroom 3→2; `begin_drain` ends loop, key gone, assignment still reserves. - Clippy `-D warnings`, fmt, typos clean; api crate tests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…format#363) Part of lance-format#333 (P1.7 — last shadow-phase piece). Design §4.2, spec §2.3. ## What - **`scoring.rs`** (pure): `merge_score = max(pending/min_gens, bytes/min_bytes) × (1 + wait/max_age)`. Classes: **Critical** (≥256 pending, or promoted for age ≥ `max_turn_wait_secs`), **Normal** (score ≥ 1), **Tail**. `planner_order`: class asc; **class 1 oldest-first, never by score** (review round 2); score desc elsewhere; name tiebreak → total order. - **Shadow pass** in the planner, every reconcile: score → order → skip tails (real sweeps do too) → place on fittest executor by headroom → write `Shadow` assignments (replaced wholesale per pass, leader-token guarded). Then compare with reality (real `MergeWal` task active?) → `scheduler_shadow_disagreements_total{kind}` + per-pass gauge. **This is the number that must approach zero before P2.** - Gauges: `planner_merge_score{target}`, `planner_units{class}`, `planner_shadow_placements`, `planner_shadow_unplaceable`. - **Spec test 5** as a source-level test: no file outside `planner.rs`/`executors.rs`/operator route references assignments. - Deferred to P2: compaction/index scoring, class 2 commit-ready, `max_consecutive_turns`. ## Validation - Scoring: thresholds; age doubles score at `max_age`; promotion for age; **class 1 oldest-first beats a 100× score** (spec test 6); class before score; total stable order (20 items, forward == reverse). - Planner etcd test extended: exactly one Shadow assignment for the 42-pending table on the live heartbeat, none for the empty table, headroom shows it, enqueuing a real task → next pass replaces not accumulates. - Isolation test passes; clippy `-D warnings`, fmt, typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Part of lance-format#333. Fixes all six findings of review round 5 (credit: production review), which brought real numbers: **13,405 tables, 354 with pending, 318 generic**. | # | Finding | Fix | |---|---|---| | 1 [P1] | Full prefix read of ~27k records exceeds the 4 MiB etcd client cap | Every planner read paged (512 keys) **at one revision**. Reconcile detached from the leader loop → keepalive cannot starve; one pass in flight. | | 2 [P1] | Scan read only `sealed`; a lost completion event left a merged shard pending forever; old table's first contact → gen 1001 = 1001 pending | `all_shard_watermarks` reads the base MemWAL index's merged watermark per shard; scan events carry it. | | 3 | `generic:generic:<name>` | `entry.name` already has the prefix. | | 4 | 1-byte reservations when `pending_bytes == 0` (all real publishers) | Unknown reserves the **full per-pass cost** (`2 × ROLLOUT_APPEND_MAX_BYTES`); counted. | | 5 | "disagreement ≈ 0" is not a takeover gate | `planner_shadow_comparison{outcome ∈ discovery_miss, placement_gap, phantom_task, agree}`, each a precise question; documented as observations. Old counter kept as the sum. | | 6 | 128 pending, merge 64 → age unknown / wrong after rebuild; slow reconcile starves keepalive | Times map bounded by cap only (lowest 63 + newest), filtered by merged on read; answer carries `Exact`/`Upper`; **only Exact promotes**. Executor events no longer seed flush times. Keepalive: see #1. | ## Validation - **Production-shape test**: 13,405 × 2 shards seeded; asserts the **unpaged read fails** (so the test cannot pass vacuously), paged read returns 26,810 at one revision, fold counts them, leader completes a reconcile and holds its lease 12 s past it, cleanup verified. Gated on `PLANNER_SCALE_TEST=1` (run alone; it starves a shared single-node etcd). - Demand: 128/64 case → `Upper` bound, same after rebuild and in reverse; randomized `rebuild == fold` still holds; scan-with-merged folds. - Scoring: a bound never promotes or boosts. - 16 demand, 5 scoring, 17 etcd (planner/executor/scan/resident), 9 sweeper green on clean etcd; clippy `-D warnings`, fmt, typos clean. Spec §1.1/§1.2a/§1.3/§2.3 updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…rmat#365) Part of lance-format#333. Follow-up to lance-format#364. Review round 5 measured production: **13,405 tables, ~354 with pending, 318 generic, 6 masters**. The design assumed "tens → low hundreds active" — wrong by two orders of magnitude — and that assumption directly produced the unpaged etcd reads and 1-byte reservations fixed in lance-format#364. Corrects §2 (workload table, with the error acknowledged), §4.5 (per-cycle cost now stated with paging and measured at production shape), §6 (when the single leader would need watch-driven folding or sharding), and marks backlog open question 1 answered. Docs only; typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…oors (lance-format#366) Part of lance-format#333. Fixes all five findings of review round 6 (credit: production review). | # | Finding | Fix | |---|---|---| | 1 [P1] | 512-key page bounds count, not bytes; 512×8 with full time maps → 5.6 MB → fails and loops | Adaptive pages: start 512, halve on >1 MiB or message-size error, grow back; one revision | | 2 [P1] | Writes chunked by op count only; 100×20-shard records ≈ 2.7 MB > 1.5 MiB request cap | Chunk by ops (100) **and** bytes (768 KiB), sized per op | | 3 [P2] | Long-lived tail never promotes: first 63 timestamps pin the map; Upper bounds excluded | Upper bound on flush time = **floor on wait**; promotion uses the floor from either bound (sound); score aging uses exact only | | 4 [P2] | Comparison conflated queued/running, skipped unplaceable tables, wrote zeros on read failure | Separate paged queued/running sets; per-due-table `discovery_miss / queued_only / running / placement_gap / phantom_task`; skip-and-count on read failure | | 5 [P2] | Estimate capped at 2×max_bytes but a whole generation is read first | `2 × max(max_bytes, largest generation seen)`; whole local budget when unknown | ## Validation - `chunk_ops`: count-bound, byte-bound, oversized-op-alone. - Scale test at the reviewer's shapes (`PLANNER_SCALE_TEST=1`, run alone, drop-guard cleanup): 512×8 full maps — fixture asserted **> 4 MiB total and unpaged read fails**, paged read returns 4,096; 612-table reconcile incl. 100×20-shard records writes within limits. - Demand: 70 merged-then-2-pending tail → usable day-long wait floor, identical after rebuild. - Scoring: a bound promotes on wait, adds no age boost. - 16 demand, 5 scoring, 19 etcd planner/executor/scan/resident, 39 merge-crate tests green on clean etcd; clippy `-D warnings`, fmt, typos clean. Spec §1.1/§1.2a/§2.3 updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Part of lance-format#333. Review round 7 cleared a single-master shadow canary in production; this is the runbook for it. Covers: what `PLANNER_ENABLED=1` does on one master (observe and simulate only — no merge behaviour changes), how to confirm it is leading, what to watch and in what order (master RSS, etcd latency, reconcile completion, leadership flaps, real merge progress, pending/503), what each `planner_shadow_comparison{outcome}` means and does **not** mean, what the canary cannot tell you (scheduling correctness, parallel capacity, compaction/index), how to disable, and when to widen to a second master. Explicit that P2 does not follow from canary data alone. Docs only; typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ax (lance-format#367) Part of lance-format#333. Fixes the two code findings of review round 7 (credit: production review), which otherwise cleared a **single-master shadow canary in production**. | # | Finding | Fix | |---|---|---| | 1 | `GET /scheduler/demand` list mode still one unpaged read → fails at 13k tables | Paged read reduced to a bounded summary: pending-sorted, `limit` (default 200, max 2000), `include_idle`, `total_tables`/`tables_with_pending`. Full record only for `?target=`. Response size independent of cache size. | | 2 | Byte estimate used `pending_bytes / pending_generations` (an average) as the max | Every shadow reservation is the executor's whole local budget — the only honest value for an unknown max. Spec now says the shadow capacity model answers "can one unit fit", **not** parallelism; `placement_gap` must not be read as such. `planner_shadow_reservation_basis_total{basis}`, `planner_shadow_average_generation_bytes` record the basis. | ## Validation Planner etcd test extended: list view with `limit=1` returns only the pending table, record omitted, `total_tables=2`, `tables_with_pending=1`. 18 planner/routes tests green; clippy `-D warnings`, fmt, typos clean. Spec §2.3 and observability list updated. Finding 3 (real-traffic observation: master RSS, etcd latency, reconcile completion, merge progress, pending/503) is the canary itself, not code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
lance-format#371) Part of lance-format#333. Closes the two test gaps left open across review rounds 5 and 7. Test-only; no behaviour change. ## Leader failover (spec test 4) Master A leads, builds cache + shadow placements over three tables, is **aborted without releasing** (lease expires naturally). Its token is then refused. Master B campaigns with only etcd and must rebuild **identical** demand cache, shadow placements (target/executor/bytes), executor totals, and keep a durable `Bound` reservation written by the old leader. Mutations caught: second leader placing differently → placements assertion; rebuild dropping `Bound` → durable-reservation assertion. ## Sustained concurrent isolation Four real index preparations at once over several idle windows: throttled-alive ×2 (different rates), frozen-from-start, frozen-midway after real progress. Asserts: alive pair finishes past the idle window with real IO and the slower one takes longer; frozen-start cancelled in `[IDLE, IDLE+4)`; frozen-mid cancelled in that window **measured from its own freeze**; alive builds outlive both cancellations. Whole scenario bounded at 60 s so a never-cancelled build fails fast. Mutation caught: a shared progress counter across watchdogs fails the test. ## Validation Both green against real etcd (9 s, 16 s); planner/watchdog/isolation suites green; clippy `-D warnings`, fmt, typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…at#372) Recovery health checks currently run Kubernetes exec on every worker/master to read container memory and OOM state. Expose a bounded read-only `/api/v1/internal/runtime-resources` endpoint through their existing shared router so managed recovery can obtain the same evidence directly. Each request reads the three cgroup-v2 memory files, with an 8 KiB limit per file, and returns usage, nullable limit, raw OOM counters, sample time and an optional caller nonce. Responses are not cached; missing/malformed/unsupported cgroups return 503 rather than stale data or invented zero values. It does not access stores or alter writer ownership. Callers must still validate Pod/container identity, readiness, expected configuration, freshness and their existing admission thresholds. Validation: `cargo test -p lance-context-metrics --locked` (4 tests passed) and `cargo clippy -p lance-context-metrics --all-targets --locked -- -D warnings` pass. Tests cover fresh changes, failed reads after previous success, unlimited budgets, retained OOM evidence, malformed/oversized files, and shared-router wiring. Only the existing serde_json dependency is added to this crate's lock entry. Co-authored-by: Beinan Wang <beinanwang@microsoft.com>
…rmat#373) Recovery currently depends on ingress log streams to identify hot tables and table-specific read failures. Add opt-in bounded application request hints to the existing HTTP middleware and an internal nonce-bound JSON endpoint, so an observer can discover demand even while a handler is still waiting. Requests never wait for the cache. Capacity, TTL and reply bytes are bounded; dropped/truncated observations remain explicitly incomplete and recover when loss ages out. Maintenance routes do not generate their own demand. Hints retain no payload/query/record identifiers and cannot authorize merge replacement or prove progress. The endpoint does not observe ingress-only or streaming-body failures. Validation: metrics crate unit and actual Axum middleware integration tests pass for enabled/disabled processes; Clippy passes with warnings denied. An uncontended release microbenchmark measured 1M parse+record operations in 410 ms (not a production latency claim). See the crate documentation for rollout and coverage semantics. Co-authored-by: Beinan Wang <beinanwang@microsoft.com>
## Summary Upgrade Rust Lance and optional Python pylance dependencies from 9 to 13.0.0, including the SSTable metadata, explicit manifest refresh, SealFence, and object-store API adaptations. Preserve exact-version merge authorization across Lance's internal commit retries; a regression test races a concurrent manifest commit with authorization revocation. Before deploying to existing MemWAL datasets, flush and merge all old pending generations using the old binaries, then upgrade readers and writers together. The upstream field-ID change prevents safely mixing old flushed generations with new readers; see `docs/design/lance-13-upgrade.md`. ## Testing - Passed the repository's full `run_ci_checks.sh`: workspace format, all-target Clippy and tests; Python pytest (229 passed, 2 skipped, 1 expected failure), Ruff format/lint, and Pyright (0 errors/warnings). - Built `lance-context-server` and the Python extension against the upgraded dependencies. - Concurrent-manifest/revocation regression passed with the core suite. GitHub's Rust/etcd HA, coverage, Python test-wheel matrix and Python test jobs also passed.
## Summary Add a feature-gated, unpublished Rust WAL experiment crate to compare Arrow IPC decoding/row-copy costs and local durable log writes before integrating the distributed WAL. The codec benchmark verifies complete Arrow equality and actual payload buffer sharing, including unaligned input; the raft-engine benchmark performs synced writes across groups, reopens storage, and compares every payload. Results are JSON with allocation counts and separate write/recovery/verification timing. The README documents measurement limits, and CI builds/lints the optional binary and runs tiny correctness smoke cases. ## Testing - Passed the repository's full `run_ci_checks.sh`: workspace format, all-target Clippy and tests; Python pytest (229 passed, 2 skipped, 1 expected failure), Ruff format/lint, and Pyright (0 errors/warnings). - Component check and all-target Clippy with `--features lab` passed. Codec smoke passed all five paths and shared-buffer checks; disk smoke reopened and verified 64 entries. - Spelling and diff checks passed. Rebasing after lance-format#374's squash merge preserved the exact source tree tested locally.
…at#376) ## Summary Add the optional `placement` module to the independent WAL crate. It routes a tenant/table/session key to a stable virtual bucket, validates an immutable RF3 layout, and scopes receipts by cluster, layout fingerprint, group and log index. Invalid or ambiguous ownership is rejected before routing; changing a validated layout requires migration. The hash formats match the existing Rust/PHX experiments. This PR supplies placement types only; it does not connect a distributed writer to production ingestion. ## Testing - Full `.codex/skills/ci-pr-helper/scripts/run_ci_checks.sh`: workspace Rust formatting, Clippy and tests; Python 229 passed, 2 skipped, 1 xfailed; Ruff and Pyright passed. - `cargo test --release -p lance-context-wal --features placement`: three tests, including the recorded PHX layout fingerprint and an independently checked session hash vector. - `cargo clippy --release -p lance-context-wal --features lab,placement --all-targets -- -D warnings`. - Spelling and diff whitespace checks passed. CI explicitly exercises the optional module.
## Summary Multiple WAL groups on one node need a shared proposal budget so concurrent producers cannot each consume the full disk allowance. Add an opt-in `admission` module with atomic reservations, drop-based release, encoded-write accounting, and a reserved control window. Refresh uses real file lengths and allocated blocks; successful admission does not scan the directory. This module is not connected to production ingestion. It does not reserve filesystem extents or cap previously accepted replication traffic; replica-pressure coordination and snapshot collection remain necessary. ## Testing - Release admission/placement tests: five passed, including racing reservations and reclamation while proposals remain pending. - Release Clippy with `lab,placement,admission`, formatting, spelling and diff checks passed. - Full local CI script passed: workspace Rust Clippy/tests, Python pytest, Ruff format/lint and Pyright. - CI explicitly tests and lints the optional module. Refs lance-format#378.
…e-format#458) A hot table discovered early in a stats scan currently waits for every remaining table to finish before the merge sweep can see its pending WAL. On one production master, completed scans averaged about 92 minutes; this is a discovery delay independent of the separate legacy-worker admission blocker. Consume each completed observation immediately and submit eligible targets to the existing durable merge queue. Incremental observations also qualify because they recount pending WAL. Preserve automatic-merge disablement, canonical deduplication, target encoding and all existing execution/ownership rules. Each scan can admit at most 256 new tasks and spend at most 10 seconds on queue admission, with a 2-second individual bound. An ambiguous queue error stops early admission for that round while observation and snapshot publication continue. Validation: formatting and diff checks pass. Three real isolated-etcd regressions cover enqueue before a blocked observation completes, disablement/threshold/deduplication/budgets, and queue failure preserving the entire scan result. The final committed source passed all 18 scanner tests, including the three isolated-etcd regressions (0 failed or ignored); its commit and diff fingerprint stayed unchanged throughout the run. CI is running. This adds no executor pool or rollout mode. It does not resolve historical unfenced requests, authorize replacement of an old writer, or bypass `core_not_ready` in the external recovery controller. It is separate from the already-building lance-format#454/lance-format#456 release. Co-authored-by: Beinan Wang <beinanwang@microsoft.com>
…at#449) ## Summary Add an independent rollout-ingestion adapter using the durable WAL's TLS client interface. It verifies the authorized table generation and rollout contract at quorum, reserves shared memory before buffering, validates Arrow IPC and session-bucket boundaries, and retains the original bytes for exact retries. Durable ACKs produce authenticated receipts that can be restored after restart and checked against compact publication visibility. The adapter lives in the unpublished `lance-context-ingest` crate so the existing publishable core does not acquire an unpublished WAL dependency. Applications must persist producer-open and append retry state before sending; HTTP routing, that journal wiring and the materializer loop remain separate integration work. Refs lance-format#378. ## Testing - Full repository CI-equivalent checks, including Rust, Python, Ruff and Pyright. - Real TLS three-replica test with a real Lance genesis manifest, all original replica directories restarted, exact retries, and complete metadata/payload reads. - IPC validation, shared-memory pressure, virtual buckets sharing a Raft group, receipt tampering/key rotation/scope, producer fencing, and compact quorum visibility bounds. Built on merged lance-format#448. The change is now based on main with an identical feature diff; all checks must pass for the new commit before merging. CI correction: registry packaging excludes the private ingress adapter; its Rust feature tests remain required. Linux wheel setup uses bounded downloads from the official AlmaLinux repositories, with TLS and package signature checks preserved. APT wheel setup now uses only signed official distribution repositories in an isolated metadata directory. This avoids unrelated PPA outages; bounded network retries, HTTPS and archive signature verification remain enabled. Bash/ShellCheck and a local Debian metadata/install simulation passed; the Ubuntu cross-build installation is validated by remote CI. CI runs the existing workspace, WAL, Lance, and etcd suites in separate jobs so cold feature builds do not exhaust one shared 60-minute timeout. Every original check command and the coverage jobs are preserved. Checkpoint reopen cases are serialized around subprocess creation to prevent inherited file locks; corruption cases must report the expected validation error. Minimal-feature and all-feature checkpoint tests and Clippy pass; the workflow passes actionlint. Final-commit CI is required before merging.
) ## Summary An RPC connection can remain in the client pool after the WAL server closes it for inactivity. The next catalog read then fails with a TLS EOF or broken pipe, even when the cluster is healthy. Recover Query, Fetch, and Status requests once on a fresh, authenticated connection when a reused stream disconnects. The reconnect stays within the original request deadline and holds the existing connection and byte credits. Append and Commit are never automatically replayed; TLS authentication, response scope, correlation, and payload validation failures remain errors. Refs lance-format#378. ## Testing - Reproduced the failure against the unchanged client with real three-node Raft runtimes and TLS listeners; the regression passes with the fix. - `cargo test -p lance-context-wal --features peer-transport --lib`: 130 passed, 1 existing ignored test, including process restart tests. - `cargo clippy -p lance-context-wal --features peer-transport --all-targets -- -D warnings`: passed. - `cargo fmt --all -- --check`: passed. - Added coverage for direct and routed reads, fetch payloads, mutation uncertainty, a second disconnect, stalled reconnect, invalid scope/correlation, and released admission credits. - Full workspace checks are running separately; no throughput improvement is claimed by this correctness fix.
## Summary A bounded runtime can reject materializer queries because response-byte credits are exhausted even while request slots remain available. Expose local admission observations for data, control, and incoming/outgoing peer budgets, including capacities, usage, and separate request/byte rejection counters. Counters change only on rejected admission; successful requests keep the existing accounting path. Fields are sampled independently and describe local pressure, not quorum state. Refs lance-format#378. ## Testing - 127 WAL library tests passed with `peer-transport` enabled; one existing ignored test. - A real three-node quorum test retains 16 Publisher query results: the 4 MiB control budget admits 2 and rejects 14 for byte pressure; 32 MiB admits all 16. A separate two-request limit reports request pressure, and consuming results restores admission. - Feature-enabled Clippy and workspace formatting passed. This exposes diagnostics; it does not change budget defaults or claim an end-to-end throughput improvement.
…ormat#450) ## Summary Durable rollout ACKs need a recoverable path into queryable Lance files. The materializer selects bounded prefixes across session-sharded WAL groups, fetches their Arrow IPC, and writes Lance fragments with bounded parallelism. It records prepared descriptors before publication, reconciles uncertain outcomes, and retires source payloads with catalog-issued closure proofs. Takeover reuses prepared files or completes retirement without republishing rows. Refs lance-format#378. Built on merged lance-format#449; the feature diff is unchanged when moved to main. ## Testing - Real TLS replicas and complete rollout schemas: restart original directories after preparation and publication, verify descriptors and every row/column, reject stale owners, and preserve exact append retries. - Bounded manifest history, multi-round publication, and client/catalog/Lance reader regressions. - Previous feature revision passed repository Rust and Python checks (229 passed, 2 skipped, 1 xfailed), formatting, Clippy, Ruff and Pyright. CI must pass again for the new commit before merge. These checks establish correctness; they do not establish sustained throughput.
…#451) ## Summary Add `context-wal-node`, an independently deployable Rust WAL replica service. Validated, persisted configuration starts durable replication, mutually authenticated peer and snapshot transport, separate client data/control listeners, automatic maintenance, and joined shutdown. Local diagnostics expose runtime and disk state; the executable does not depend on Lance. Refs lance-format#378. Built on merged lance-format#449; the feature diff is unchanged when moved to main. ## Testing - Six-process service test with two disjoint three-voter groups: checkpointing, compacted-follower snapshot catch-up, leader SIGKILL, complete restart, payload verification, and exact retries. - Library/process regressions, service-feature Clippy, and standalone build/dependency validation. - Previous feature revision passed repository Rust and Python checks (229 passed, 2 skipped, 1 xfailed), formatting, Clippy, Ruff and Pyright. CI must pass again for the new commit before merge. These checks establish correctness; they do not establish sustained throughput.
…#461) ## Summary Node diagnostics currently expose disk admission but cannot show whether runtime requests are rejected by request slots or byte capacity. Include the existing bounded admission observations in node status and readiness responses for data, control, incoming peers, and outgoing peers. The values are sampled independently, exclude transport budgets, and do not change readiness decisions. Refs lance-format#378. Built on merged lance-format#451 and lance-format#460. ## Testing - Formatting and service-feature Clippy passed. - All three node-service tests passed, including six actual replica processes across two groups, snapshot catch-up, leader termination, and restart. - The real-quorum publisher admission regression passed: byte pressure, request pressure, and credit release remain distinguishable. - The same feature also passed a three-process release smoke check with separately configured control budgets before promotion onto main. - Repository CI-equivalent checks passed: workspace Rust tests, formatting, Clippy, Python tests, Ruff, and Pyright. Nine Python fixtures initially could not find the server in the shared-target build environment; all nine passed after building and placing the correct executable. Final-commit CI is still required before merge.
…format#453) ## Summary Add an HTTP gateway for durable rollout ingestion and a Rust HTTP producer that journals exact retry state. Gateways accept the original Arrow IPC buffer, reserve memory before buffering, separate data/control admission, and return 202 only after the WAL quorum has persisted the append. Signed producer credentials work across gateways without another producer-allocation RPC on each append. Producer lanes share one bounded group-commit journal and HTTP pool. Original bytes and sequence are durable before transmission; the ACK is durable locally before the body is released or the sequence advances. An uncertain network result preserves the original intent, a different body cannot replace it, and local journal ambiguity fences the lane until reopen. Large digest/byte comparisons run off the async executor. The gateway takes authorized fixed table bindings and must sit behind application access control. These are application library interfaces; deployment, dynamic namespace integration and sustained performance qualification remain follow-up work. Refs lance-format#378. Built on merged lance-format#450. The feature diff is unchanged when moved to main; all CI must pass on the new commit before merging. ## Testing - Complete repository CI-equivalent script passed: Rust, Python (229 passed, 2 skipped, 1 xfailed), formatting, Clippy, Ruff and Pyright. - HTTP and HTTP-client/materializer feature checks passed, along with all five integration tests. - Real HTTP to TLS replicas to Lance: inject an intermediary error after a durable ACK, reopen the client journal/gateway/all original replica directories, recover the exact ACK, reuse prepared Lance files after another restart, and verify every row/column and subsequent producer sequences. - Data-lane saturation keeps control requests available; draining joins a disconnected partial-body reader without advancing its sequence. Producer/receipt domain separation and tampering checks remain required. - Native client-journal recovery regressions passed (10 tests, with two child fixtures invoked by their parent tests). The new application restart tests use joined shutdown/reopen, not application SIGKILL or throughput measurements. APT wheel setup now uses only signed official distribution repositories in an isolated metadata directory. This avoids unrelated PPA outages; bounded network retries, HTTPS and archive signature verification remain enabled. Bash/ShellCheck and a local Debian metadata/install simulation passed; the Ubuntu cross-build installation is validated by remote CI. CI runs the existing workspace, WAL, Lance, and etcd suites in separate jobs so cold feature builds do not exhaust one shared 60-minute timeout. Every original check command and the coverage jobs are preserved. Checkpoint reopen cases are serialized around subprocess creation to prevent inherited file locks; corruption cases must report the expected validation error. Minimal-feature and all-feature checkpoint tests and Clippy pass; the workflow passes actionlint. Final-commit CI is required before merging.
lance-format#455) ## Summary Coordinate materializer processes with bounded etcd claims and durable WAL publisher epochs. A worker reads the WAL epoch before validating its claim, so an expired owner cannot steal a successor's newer epoch. Prepared Lance files survive ownership changes and can be reused exactly. Workers share memory and WAL client pools, bound concurrent tables and publication rounds, and join admitted file work and lease renewal on shutdown. Busy-table probes use a local, keys-only hint before attempting a lease, avoiding quorum barriers and replicated grant/revoke writes on occupied-table polls. Ownership still requires the linearizable create-if-absent transaction, and claim validation remains linearizable. Managed readers can open the selected Lance version without acquiring publisher ownership. Assignments are an application-authorized fixed set. Dynamic namespace discovery, application process startup, and sustained throughput qualification are follow-up work. Refs lance-format#378. Built on merged lance-format#453. The feature diff is unchanged when moved to main; all final-commit CI checks must pass before merging. ## Testing - The complete repository CI-equivalent script passed: Rust, Python (229 passed, 2 skipped, 1 xfailed), formatting, Clippy, Ruff and Pyright. - The final busy-claim optimization was qualified separately with real etcd tests and worker-only / worker plus HTTP-client feature Clippy checks. - Revoke a claim after real Lance preparation, acquire a successor epoch, reject the old claimant, reuse the exact prepared result, and publish through two competing automatic workers with full-column readback. - Abandon a claim and require actual TTL expiry before takeover; all RPC and expiration deadlines remain enforced. These are correctness checks. They do not establish process-kill recovery or sustained throughput for the application. CI runs the existing workspace, WAL, Lance, and etcd suites in separate jobs so cold feature builds do not exhaust one shared 60-minute timeout. Every original check command and the coverage jobs are preserved. Checkpoint reopen cases are serialized around subprocess creation to prevent inherited file locks; corruption cases must report the expected validation error. Minimal-feature and all-feature checkpoint tests and Clippy pass; the workflow passes actionlint. Final-commit CI is required before merging. Independent Raft-log recovery cases exclude sibling subprocess-creation windows, preventing inherited file locks from racing immediate reopen. The two-group concurrent-write case and real SIGKILL, rotation, and write-error assertions remain unchanged. All nine Raft-log cases pass with minimal and all features, and Clippy passes. This adjustment changes only test scheduling. TLS process startup failures include the child stdout/stderr and identify peer/image listener binding errors. The unchanged quorum, snapshot-interruption, SIGKILL and full-restart assertions pass in five complete peer-transport library runs (126 tests per run), plus six process tests and feature Clippy. The intermittent CI startup failure has not been reproduced locally; these diagnostics do not establish its root cause. Final-head CI remains required.
…rmat#457) Initialize a managed rollout generation in Lance and the durable WAL catalog using persisted operation identities. Retrying after a crash reuses the original genesis manifest; initialization rejects an existing foreign table instead of adopting or overwriting it. The regression tests use real Lance files and replicated WALs, including a restart after the genesis file exists but before catalog selection. Focused ingestion/bootstrap tests, feature Clippy, and the complete repository checks pass (Python: 229 passed, 2 skipped, 1 xfailed). Refs lance-format#378. Built on merged lance-format#455. The feature diff is unchanged when moved to main; all final-commit CI checks must pass before merging. Independent Raft-log recovery cases exclude sibling subprocess-creation windows, preventing inherited file locks from racing immediate reopen. The two-group concurrent-write case and real SIGKILL, rotation, and write-error assertions remain unchanged. All nine Raft-log cases pass with minimal and all features, and Clippy passes. This adjustment changes only test scheduling. TLS process startup failures include the child stdout/stderr and identify peer/image listener binding errors. The unchanged quorum, snapshot-interruption, SIGKILL and full-restart assertions pass in five complete peer-transport library runs (126 tests per run), plus six process tests and feature Clippy. The intermittent CI startup failure has not been reproduced locally; these diagnostics do not establish its root cause. Final-head CI remains required.
…at#462) ## Summary Compose the durable ingestion components into a Rust executable with bootstrap, HTTP ingress, materializer worker, and managed-reader inspection roles. A fixed deployment-authorized registry shares WAL connections and memory admission across tables; startup validates existing generations, and shutdown joins admitted work before releasing claims. WAL connections use mutual TLS and pinned endpoints. HTTP requests require a workload token and expect TLS termination at the edge. Configuration parsing rejects unknown fields and bounds file reads. Dynamic namespace discovery, legacy-table migration and sustained scaling remain follow-up work. Refs lance-format#378. Built on merged lance-format#457. The feature patch is unchanged after moving to main; all final-commit CI checks must pass before merging. ## Testing - Cargo formatting and service/http-client Clippy passed. - All 7 ordinary ingestion feature tests passed, including exact retry after replica restart and full Lance publication recovery. - Built and executed this branch's application binary against real TLS WAL nodes and isolated etcd: repeated bootstrap, SIGKILL/restarted ingress, original-receipt retry, two workers, full Arrow readback and joined shutdown passed. - Both etcd worker ownership/expiry tests passed. - CI runs the executable recovery test and retains process logs on failure. These tests establish the exercised recovery behavior, not sustained throughput.
…t#465) ## Summary Closing a WAL engine could leave its directory locked when a duplicated file descriptor remained open, including a descriptor inherited during concurrent process creation before exec. Explicitly unlock the directory when the engine's final pipe owner closes, instead of depending solely on descriptor closure. Record the acquiring process ID so a forked child's destructor cannot release its parent's live lock. The vendored raft-engine patch preserves exclusion while the engine is live and changes no journal, Raft log, snapshot or wire format. It also retains the normal shutdown ordering: pipe owners finish before the directory lock is released. ## Testing - A deterministic Linux regression duplicates the actual journal lock descriptor. It fails before this change and passes after it, without sleeps or retries. It also checks that closing the old duplicate cannot unlock a replacement engine. - Client-journal Clippy and tests: 11 passed, including SIGKILL/rewrite/sync-failure recovery; two subprocess helpers are invoked by their parent tests. - All-feature WAL library tests: 158 passed, 3 ignored; all-feature all-target Clippy passed. - The interrupted-file admission regression measures actual allocated file bytes after sync rather than assuming physical allocation equals logical length. It retains the unfinished-reservation, reopen and collection assertions. - The existing pre-exec inheritance test is updated to verify that a replacement engine can acquire the lock before the child executes, while both the original live owner and replacement owner still exclude concurrent opens. The final explicit all-feature/all-target WAL run passed, including this process regression and the interrupted-file accounting regression. The vendored engine was explicitly rebuilt and its artifact hash verified to exclude stale shared-target dependencies. - Unchanged non-WAL workspace tests, formatting and Clippy passed. Python pytest: 229 passed, 2 skipped, 1 xfailed; Ruff and Pyright passed, using source- and hash-verified unchanged native/server artifacts. Local checks are complete. All CI checks on the final commit must pass before merging. Refs lance-format#378.
…-format#467) A legacy worker can accept a merge POST and never return, occupying a master scheduler slot indefinitely. The master now retains the same task, renewing claim, durable execution, admission guard and HTTP future while allowing that response wait to yield its execution slot after a configurable observation interval. This does not cancel or retry the merge, declare the worker stalled, or fail dependent tasks. Additional waits are bounded per master (default 32, zero disables yielding). A late response first joins the fair execution queue, then releases its wait capacity before blocking for a slot; this preserves the concurrency bound without deadlocking full wait/execution pools. The original serial fan-out continues only after reacquiring a slot. Local merges, compaction, indexing and fenced execution retain their current budgets. `/executor` phases and dedicated wait/yield metrics expose the distinction. Validation: 34 targeted tests pass on the final commit, including 26 legacy/ownership/routing regressions with real local etcd, three response-capacity tests, three admission/drain tests, the existing executor-drain integration, and merge/compaction pool isolation. The one-slot scheduler regression proves another table completes while the original task remains Running; its dependent task stays Queued, then both original/dependent tasks complete after the delayed response. Full waiting/execution capacity also recovers without deadlocking. CI is still running. There is no persisted-schema change or worker-upgrade prerequisite. This does not transfer requests already held by old master processes, retire worker0/worker9, reset ownership, or solve permanent exhaustion of the bounded wait capacity. Those old executions still require verified handoff. No Kubernetes, ConfigMap, RBAC, worker-memory/slot, or separate merge-pool changes. --------- Co-authored-by: Beinan Wang <beinanwang@microsoft.com>
…-format#463) ## Summary Expose fixed cumulative counters for ten materializer phases, so slow publication can be separated into ownership acquisition, WAL reads, decoding, Lance file writes, commit, retirement and memory admission. Worker counters survive successive table claims; failed and cancelled work remain observable without per-table metric labels. Parent and concurrent phase durations overlap. The documentation explains each counter and why these values are neither CPU time nor latency percentiles. This adds measurement and does not claim a throughput improvement. Refs lance-format#378. Built on merged lance-format#462 and the engine directory-lock fix lance-format#465. The feature patch is unchanged after moving to main; all final-commit CI checks must pass before merging. ## Testing - On the final combined branch: formatting, service/http-client Clippy, all 8 ordinary ingestion feature tests, both isolated-etcd worker ownership tests and the actual application executable recovery test passed. - The executable test covers repeated bootstrap, killed/restarted ingress, original-receipt retry, two workers, full Arrow readback and joined shutdown using real TLS WAL nodes and etcd. - Before integrating lance-format#462, the phase-only branch also passed workspace Clippy and 814 Rust tests, 220 Python tests (11 skipped, 1 expected failure), Ruff and Pyright. Python checks resumed with the built native extension on PYTHONPATH after missing test dependencies were installed. Final combined workspace checks run in CI.
## Summary Materializers currently fetch and decode an entire pinned publication before starting Lance file writers. Add an experimental, default-off `worker.publication.pipeline_preparation` option that feeds bounded Arrow streams into Lance 13 writers as records become available, without payload reserialization. Both paths preserve the same immutable plan, whole-record partitioning and complete descriptor aggregation. Writers share fetch/decode permits; blocking decode retains its permit and source reservation through cancellation, and all admitted writers finish before a failure is returned. No partial descriptor set can be prepared or published. This distributes preparation within one worker only. Lance encoder workspace remains outside the source reservation, streamed `blob_write` timing includes source waiting, and no throughput improvement or end-to-end zero-copy is claimed. A matched short cloud comparison completed full payload and aggregate checks before and after restart in both modes. In the common 60–150 second window, buffered publication was 60.17–60.63 MiB/s and streamed publication 57.23–57.87 MiB/s. Including drain, the respective rates were 52.62–54.15 and 57.33–59.05 MiB/s. These are one-run sampling/clock bounds, not statistical confidence or sustained capacity; initial leaders differed and the streamed run experienced term changes. The results do not establish a steady-window throughput improvement or support enabling the option by default. ## Testing - Workspace formatting, all-target Clippy and Rust tests. - Python pytest: 229 passed, 2 skipped, 1 xfailed; Ruff formatting/lint and Pyright. The server executable was rebuilt from this source; the unchanged Python native extension was reused after checking its source inputs. - Service/http-client all-target Clippy and 10 feature tests, including real WAL/Lance recovery in both modes and injected stream failure with observer cancellation. - Both modes passed release-process qualification: 192 MiB ACKed but unpublished, all nine WAL processes killed, original directories reopened, then one Lance publication with complete payload, aggregate-row, read-only and negative-verifier checks; no producer replay. - A separate streaming-mode process test replaced one replica directory with an empty directory after the crash. Its group payloads were fetched exclusively from the rebuilt replica and all data checks passed. This is local directory-loss simulation, not a Kubernetes volume-loss qualification. Refs lance-format#378. Built on merged lance-format#463 and lance-format#465. The streamed materializer implementation matches the qualified candidate; the base additionally contains the engine directory-lock fix lance-format#465. A test helper is additionally gated to the features that use it; materializer-only and service/http-client all-target Clippy pass with that correction. Validation follow-up: the HTTP test publisher helper is compiled only with `http-client`, matching its callers. Local all-target Clippy passes with `worker`, `materializer`, and `http-client,materializer` separately; production code is unchanged by this test fix. Final-head CI must pass before merge.
) ## Summary A single managed table currently prepares all of its files inside one publishing worker, so adding workers does not parallelize that table's data preparation. Add an opt-in Rust `preparer` role that divides one immutable, pinned publication plan across bounded remote services. Each service reads its assigned records directly from WAL and passes Arrow handles into the existing Lance writers. The publisher exchanges only plan references and fragment metadata, joins every partition and records the complete aggregate through the existing durable Prepare and fenced publication protocol. The services resolve the deployment-authorized table registry, verify the exact plan and source pins through quorum reads, and use authenticated bounded requests, job slots, memory reservations and a volatile result cache. Admitted jobs survive disconnected observers. A missing or failed partition cannot publish a partial result; after durable Prepare, recovery needs no preparer cache. Losing a cache earlier may redo unpublished immutable files under the existing orphan-cleanup contract. The option is disabled when `preparation` is absent. This changes no WAL wire format or Lance OSS code. It does not provide orphan deletion, a hard bound on Lance encoder RSS, or a demonstrated throughput improvement. ## Testing - Integration coverage uses two authenticated HTTP services over real TLS WAL and Lance files: missing-service failure, healthy-partition cache reuse, disjoint source reads, complete aggregate recovery after both caches disappear, and full payload verification. - Independent release-process qualification passed: two producers acknowledge 192 MiB before publication, all nine WAL processes are killed and reopened, one preparer is missing, and its completed sibling is killed to lose its cache. Restarting both preparers publishes all 64 original records exactly once with complete payload, aggregate-count, read-only and negative-verifier checks and no producer replay. All source reads, decodes and file writes occur in preparers; publisher counters remain zero. This is local process recovery, not Kubernetes volume-loss qualification. - Materializer-only and service/http-client all-target Clippy passed. Feature tests: 11 passed, 3 external-process fixtures ignored by the ordinary run. Workspace formatting, all-target Clippy and Rust tests passed, as did Python pytest (229 passed, 2 skipped, 1 xfailed), Ruff and Pyright. Unchanged native/server artifacts were reused after verifying source inputs and hashes. Final-head CI and prerequisite merges are still required. - Kubernetes 1/2/4-preparer scaling is queued behind the current qualified-control experiment. No distributed throughput result or sustained-capacity claim yet. Refs lance-format#378. Built on merged lance-format#464 and lance-format#465. The distributed preparation implementation matches the independently qualified release; the base additionally contains the engine directory-lock fix lance-format#465 and prerequisite test-helper feature gate. Final-head CI validates that combined source. Feature-only validation also covers the test fixtures with `worker` and `service` enabled independently, including the actual preparer recovery test.
…#468) When multiple preparers read a WAL group, every payload currently starts at the first configured voter. Add the opt-in `source_read_policy: spread` setting to distribute initial reads by immutable group/index while retaining every other voter as a fallback. The default remains `first_voter`. This changes read placement only: exact source-pin matching, payload/schema validation, fetch concurrency, reservations and manifest authority stay in place. The selection hash touches two integers and does not hash or copy payload buffers. It applies to both buffered and streamed preparation, including remote preparers. Validation covers real Lance recovery, exact retries and retirement, plus a network failure test that leaves only one payload endpoint reachable at a time, exercising each replica as the sole source. An independent Rust process fixture also recovered 192 MiB of acknowledged/unpublished records after all nine WAL processes were killed and reopened, followed by a preparer crash/cache loss; all 64 records and columns were read back without producer replay. A controlled cloud comparison is queued; no throughput benefit has been established, so this option remains disabled by default. The qualified candidate includes the directory-lock fix from lance-format#465; final CI will validate the combined source after dependency promotion. Built on merged lance-format#466 and lance-format#465. Refs lance-format#378. ## Testing - Workspace formatting, all-target Clippy and Rust tests: 814 passed (200 optional tests ignored). - Service/http-client feature suite: 15 passed, 3 external-process tests ignored; separate single-reachable-replica fallback test passed. - Independent release-process crash/cache-loss recovery fixture passed. - Python: 229 passed, 2 skipped, 1 expected failure; Ruff formatting/lint and Pyright passed. Python native/server artifacts were reused after verifying unchanged source inputs and artifact hashes. An additional 10-minute local endurance run published 9,447 MiB / 3,149 rows through two preparers. All columns and the aggregate remained correct after killing and reopening all nine WAL processes, with no producer replay or restarted publisher. Each node completed 9–10 checkpoints and collected 16–18 obsolete files. The run also had 619 producer retries and repeated leader-term changes under shared-disk pressure; it establishes recovery after this finite workload, not a throughput gain or stable capacity.
…format#469) ## Summary The runtime currently waits for the leader's complete synchronous storage poll before sending replication packets, delaying the start of follower persistence. Add the default-off `runtime.pipeline_replication` option to dispatch ordinary, non-snapshot Raft `Ready.messages` through the existing bounded transport before local persistence. Follower persistence-required responses, all existing sync barriers, client completions, quorum reads, snapshots, and public synchronous `Node::poll()` semantics retain their ordering. Packets move shared payload buffers and retain node-wide byte/request credits; a full queue drops for Raft retransmission without blocking the owner. An early packet followed by failed leader persistence remains an uncertain attempt and never a successful ACK from that poll. This is an experimental option, with no established throughput improvement. It does not make storage asynchronous or allow the Raft owner to tick during a blocking filesystem operation. Part of lance-format#378. ## Testing - Full repository CI helper: workspace formatting, Clippy and tests; Python tests, Ruff and Pyright. All passed on this source. - All-feature/all-target WAL Clippy and tests: 161 library tests passed, 3 ignored; binary and integration suites passed. - Block the actual leader sync gate: a follower durably accepts an append while the leader remains blocked, without a client ACK. Assert payload pointer identity, bounded outgoing credits, injected leader/follower sync failures, shared-engine fencing, complete recovery and exact-index retry after reopening all original directories. - Snapshot catch-up and restart with both option values. - Independent release executables: 192 MiB acknowledged but unpublished; SIGKILL all nine WAL processes; reopen original directories; a missing preparer cannot partially publish; kill the completed preparer to discard its cache; restore both and verify all 64 original 3 MiB rows and columns, aggregate count, version, read-only and negative checks, without producer replay. This is retained-disk process recovery, not volume-loss or throughput evidence. A cloud comparison using identical binaries and changing only the node option is queued; no result is claimed here. Final-head CI must pass before merge.
When an uncompressed engine block misses the read cache, the decoder currently copies the newly read block into another allocation before parsing entries. Reuse the original allocation after verifying the complete block checksum, removing one block-sized allocation and copy. Compressed decoding, cache isolation, entry parsing, durable ACKs, and file formats retain their existing behavior. Refs lance-format#378. Built on merged lance-format#469. Validation: actual-file cache and rewrite regressions; compressed/uncompressed multi-entry reopen; corruption rejection and recovery after restoring the original byte; log/checkpoint/snapshot tests; all-feature WAL checks; full local `ci-pr-helper` checks. The candidate release also passes nine-process WAL crash/reopen and complete Arrow/Lance recovery with a missing preparer and lost preparation cache. Local allocation diagnostics confirm the removed allocation; no end-to-end throughput improvement is claimed.
Committed Fetch replies currently wait for the next Raft poll after their payload has already been validated. Complete those replies immediately, so an unrelated append sync cannot delay or discard them. Append ACKs and quorum queries still wait for durable Raft completion. Also give each server RPC its full processing timeout after the first request bytes arrive. Previously, time spent waiting on an idle pooled connection consumed that same deadline and could disconnect a newly admitted request during its durability wait. The remaining header, body, admission, execution, and reply remain bounded by one active deadline. Client deadlines and frame/credit limits are unchanged. Refs lance-format#378. Built on merged lance-format#470. Validation: real-file fault tests cover committed reads during an unrelated failed sync, without prematurely acknowledging new data. A real TLS test with blocked journal sync reproduces expiration under the old idle/request timer and passes with separate deadlines; partial headers and bodies still expire and release the connection. RPC regressions, the six-process node restart/snapshot test, all-feature WAL checks, and full local ci-pr-helper checks pass. The original committed-fetch change also passed a nine-process WAL/Arrow/Lance recovery fixture. No throughput benefit is claimed for the deadline fix.
…t#472) A bounded source scan can encounter only records from other tables. The worker previously released its claim and discarded the scan cursor, then started from the same durable checkpoint on its next claim. A table behind more than one scan budget of retained foreign records could remain unpublished indefinitely. Retain the empty-page scan hints per fixed worker assignment and move them through claim jobs. Hints are checked against the complete table binding, contain only quorum-observed ranges without that table's records, and never advance durable checkpoints. Scan limits, claim rotation and publisher fencing remain unchanged. Restart may rescan those ranges safely. Refs lance-format#378. Built on merged lance-format#466, lance-format#468, lance-format#469, lance-format#470 and lance-format#471. Validation: - Real three-replica WAL, etcd and Lance regression: a target behind 129 retained foreign records fails with the old worker and publishes its exact payload with the fix, while another table remains unpublished. - Materializer, worker, service and service/http-client feature Clippy checks; ingestion and etcd-backed worker recovery suites. - Full repository CI helper, including Rust/Python tests, formatting and lint/type checks. The first worker-suite run encountered an existing lease-revoke timeout with slow local etcd synchronization. The unchanged suite passed on a fresh run with the same storage and deadlines; the original failure was retained. This change fixes publication progress and does not claim a throughput improvement.
Queueing behind replication work and reading or validating an accepted record currently contribute to the same fetch latency. Expose separate cumulative queue and service timings in each group's local `/status` observation, alongside found, missing, failed, and returned-byte counts. Counters use fixed storage and saturating arithmetic. They cover serviced requests, including callers that disconnect; queued work discarded on shutdown is excluded. Queue timing starts after runtime admission, and service timing excludes reply transmission. These local wall-time observations do not establish quorum freshness or change durable acknowledgements. Refs lance-format#378. Built on merged lance-format#471 and lance-format#472. ## Testing - Real blocked and failed storage-sync regression with both replication modes: existing and missing-record fetches complete and are counted before the unrelated sync, while the new append remains unacknowledged. - Actual WAL service and Lance recovery fixture with full-column readback and retained process identities. - WAL all-feature/all-target Clippy and tests passed. - Complete local CI-equivalent script passed: workspace formatting, Clippy and all-target tests; Python tests (229 passed, 2 skipped, 1 expected failure), Ruff formatting/lint and Pyright.
lance-format#475) Rust CI jobs can exhaust their entire job deadline in `apt update` when a hosted Ubuntu image selects an unresponsive archive mirror, leaving the tests unexecuted. Use the official HTTPS archive for the known mirror-list entry, bound APT timeouts and retries, and limit each dependency-install step to ten minutes. Incomplete package-index updates fail explicitly; package selection and test commands stay unchanged. Refs lance-format#378. Validation: Actionlint 1.7.12, ShellCheck and Bash syntax checks for the installation commands, plus the repository CI helper (workspace formatting, Clippy and tests; Python tests, Ruff and Pyright). Hosted-runner execution is validated by this PR's CI.
…ance-format#479) A legacy worker can stop completing merges while remaining Ready and serving ingestion. Its protocol-0 execution outlives the HTTP request, so the master cannot safely retry merely because a lease expires or a process disappears. Bounded response-wait slots only postpone scheduler saturation. This adds an explicit master retirement API and one-shot native command. Before termination, capture the exact live Pod/container/worker identity against the existing execution in etcd. After a same-Pod container replacement, require the old container termination receipt and a fenced replacement reporting the same execution and storage namespace. Persist a base/shard manifest barrier plan, cross its immutable version bounds, then CAS the exact old execution to terminal. Canonical task retries complete prepared handoffs automatically after verified process retirement; normal scheduler claims still own lock release and retries. Missing shards and ambiguous identities remain protected. The master also negotiates fencing per worker in a mixed fleet. A fenced worker receives protocol 2 directly, avoiding the protocol-0 reservation that otherwise blocks its own background admission. Failed capability probes never fall back to legacy POSTs; unresolved admission mode does not admit work. Fan-out remains serial. `MERGE_FENCE_THIS_WORKER` persists activation against one stable writer ID, preventing a single-worker retirement from switching the startup mode of unrelated workers or masters; existing cluster-wide activation remains authoritative. Validation at `c3a2c7d`: workspace unit/integration tests and real-etcd HA CI pass. Local isolated-etcd checks pass for 13 retirement-focused cases, 23 routing/recovery regressions, scheduler dependency/capacity isolation, and persisted admission modes. The worker is a binary target: its replacement-status tests and the real-store merge-budget lock regression are verified in the successful CI binary test runs, not inferred from a `--lib` invocation. Formatting and diff checks pass; remaining CI is running. Exact-image AKS/Azure same-Pod retirement qualification is prepared and awaits the candidate image from services #9889. No production worker has been replaced and no production ownership record has been reset by this change. See `docs/legacy-merge-retirement.md` for the transition contract and explicit limits. This does not infer process death from age or add an automatic process killer, pool, ConfigMap, or RBAC. --------- Co-authored-by: Beinan Wang <beinanwang@microsoft.com>
…format#480) ## Summary Publication policies currently reject source-byte limits above 256 MiB, preventing larger publication batches even when a deployment provides sufficient memory and admission capacity. Allow an explicitly configured limit up to 1 GiB while keeping the 64 MiB default unchanged. This remains part of the immutable replicated policy. A higher limit requires consistent replica configuration and matching client/materializer budgets; this change does not migrate existing groups or raise their allocations. Refs lance-format#378. ## Testing - Repository CI helper: workspace formatting, Clippy and tests; Python tests, Ruff and Pyright. - WAL service-feature and ingestion-feature tests. - Real-process recovery of acknowledged, unpublished records after replica crashes and preparer cache loss, including complete payload readback and publication identity checks.
lance-format#481) ## Summary An acknowledged rollout append can remain in WAL while workers accumulate larger Lance writes. Add `Registry::read_session` and `GET /api/v1/ingest/{table}/sessions/{session}` so callers can read those rows before publication. The reader streams committed WAL records, then scans the latest managed Lance snapshot, removing overlap by immutable row ID. Source closure checks handle retirement during a read; missing unpublished payload fails the stream. Binary filtering uses Arrow slices, with payload workspace reserved per record. The HTTP response is an Arrow IPC stream under the existing workload authorization. This is a session read with unspecified order. It scans the session's bucket; captured metadata and deduplication IDs grow with the retained tail. Consumers must read through EOF and propagate errors. Published-version dataset reads retain their existing semantics. Refs lance-format#378. ## Testing - Real three-replica RPC tests: ACK visibility without a publisher, replica restart, session isolation, publication/retirement races, and unavailable unpublished payloads. - Actual authenticated HTTP append/read against WAL and Lance, including mixed published and unpublished rows. - A tail exceeding its per-record payload budget, cancellation release, and binary buffer sharing. - Ingestion all-feature Clippy and library tests: 22 passed; 4 existing etcd/executable tests require the dedicated CI environment. - Workspace formatting, Clippy, and all-target Rust tests passed. - Python suite: 220 passed, 11 skipped, 1 expected failure; Ruff and Pyright passed after installing the declared test dependencies. - Full GitHub CI is required before merge.
…ce-format#482) ## Summary A publication byte cap alone still allows every small polling result to become a Lance write. Add an optional `min_source_bytes` threshold and `max_batch_wait`: publish a full selection immediately, or flush a smaller tail after its wait on the next poll. The default minimum is zero, preserving existing behavior. Workers retain a valid scheduling claim while accumulating the batch and still respond to shutdown. Deadline state survives local claim rotation; durable pending publications resume immediately. No source checkpoint moves while a batch is waiting. This changes source-batch sizing, not exact encoded file sizes or the group's immutable WAL byte limit. Session WAL reads can provide visibility before the delayed Lance publication. Refs lance-format#378. ## Testing - All-feature ingestion Clippy. - Seven real WAL/Lance materializer regressions, including threshold flush, timed tail, record-cap flush, and deadline preservation across claim rotation. - Real etcd worker suite: all four passed, including competing workers flushing a small tail. An initial run hit an existing lease-expiry timeout; the unchanged isolated test and complete suite both passed on rerun. - Workspace formatting, Clippy, and all-target Rust tests passed. - Python: 229 passed, 2 skipped, 1 expected failure; Ruff and Pyright passed. The first Python invocation failed because the fixture could not find the shared-target server binary; rebuilding the server and linking the expected target path resolved all nine setup errors. - GitHub CI must pass before merge.
KaiqiJinWow
force-pushed
the
feat/rollout-client-flush
branch
from
October 22, 2026 23:22
9f32b9e to
f4a9edc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to lance-format#191.
Problem
The server already supports
POST /rollouts/{name}/records?flush=true, but the Rust and Python clients cannot request it. The standalone remoteflush()call is a no-op, so SDK callers that need read-your-write visibility must poll for the background sweeper.Change
add_with_flush/add_rollouts_with_flushmethods.flush=Falseto sync and async Pythonaddandadd_one.?flush=trueon the same append request.async-traitto 0.1.92 in the lockfile for Rust 1.99 Clippy compatibility.A flush can also publish earlier or concurrent pending appends. If the flush step fails, the append may already be durable, so retryable callers should supply and reuse stable record ids. Remote flush-on-add requires server v0.6.5 or newer.
Testing
cargo check --locked -p lance-context-client -p lance-context -p lance-context-python --all-featurescargo clippy --locked -p lance-context-client -p lance-context -p lance-context-python --all-features -- -D warningscargo +1.99.0 clippy --locked -p lance-context-core --lib -- -D warningscargo test --locked -p lance-context --all-featurescargo test --locked -p lance-context-server flush_true_gives_read_your_write_and_default_does_not