Skip to content

feat(rollout): expose flush-on-add in SDKs - #4

Draft
KaiqiJinWow wants to merge 128 commits into
mainfrom
feat/rollout-client-flush
Draft

KaiqiJinWow wants to merge 128 commits into
mainfrom
feat/rollout-client-flush

Conversation

@KaiqiJinWow

@KaiqiJinWow KaiqiJinWow commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

Follow-up to lance-format#191.

Problem

The server already supports POST /rollouts/{name}/records?flush=true, but the Rust and Python clients cannot request it. The standalone remote flush() call is a no-op, so SDK callers that need read-your-write visibility must poll for the background sweeper.

await store.add(record)
rows = await store.list()  # the new row may not be visible yet

Change

  • Add backward-compatible Rust add_with_flush / add_rollouts_with_flush methods.
  • Add keyword-only flush=False to sync and async Python add and add_one.
  • For remote stores, send ?flush=true on the same append request.
  • For local stores, append and then seal the local MemWAL memtable.
  • Keep the existing default path unchanged.
  • Document the latency and partial-success semantics.
  • Update async-trait to 0.1.92 in the lockfile for Rust 1.99 Clippy compatibility.
await store.add(record, flush=True)
rows = await store.list()  # the new row is readable

A flush can also publish earlier or concurrent pending appends. If the flush step fails, the append may already be durable, so retryable callers should supply and reuse stable record ids. Remote flush-on-add requires server v0.6.5 or newer.

Testing

  • cargo check --locked -p lance-context-client -p lance-context -p lance-context-python --all-features
  • cargo clippy --locked -p lance-context-client -p lance-context -p lance-context-python --all-features -- -D warnings
  • cargo +1.99.0 clippy --locked -p lance-context-core --lib -- -D warnings
  • cargo test --locked -p lance-context --all-features
  • cargo test --locked -p lance-context-server flush_true_gives_read_your_write_and_default_does_not
  • Ruff format and lint
  • Local public-API verification with default and flushing appends
  • Real-server JSON and multipart verification with both background sweepers disabled

@KaiqiJinWow
KaiqiJinWow force-pushed the feat/rollout-client-flush branch 2 times, most recently from 640a335 to 9f32b9e Compare October 8, 2026 20:21
beinan and others added 28 commits October 8, 2026 13:32
…pressure (lance-format#349)

Part of lance-format#333. Second-round review clarifications (credit: production
review).

| Point | Change |
|---|---|
| Promotion to class 1 does not guarantee service (class 1 still
score-sorted; hot tables' critical merges are class 1 too) | Class 1
sorts **oldest-first** by `wait_since_ms`; score applies to classes 2–5
only. Invariant 4 restated. Starvation is then only possible when no
executor is ever feasible — a capacity fault with its own metric. |
| etcd revision vs manifest version are not comparable | Watermark
ordering uses only the shard's own `(writer_epoch, sealed_through)`.
Scan may raise at equal epoch, replace only with higher epoch, never
lower at equal epoch regardless of write time. `observed_revision` is
freshness bookkeeping only. Design §4.1 and P1 spec §1.1. |
| Stretching flush interval converts backlog into memory pressure /
un-flushed windows | §4.7 rewritten: memtable budget and flush interval
remain hard caps; the `critical` lever is proportional admission delay.
Policy gate `backpressure: off \| metric \| delay_admission`. Backlog
C7a corrected. |

Each is now a required P1 test (spec §7 tests 6–8). Docs only; `typos`
clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…/403 (lance-format#347)

Part of lance-format#333. Fixes two defects in lance-format#338 found in review (credit:
production review).

## Problems

1. **Mixed-version regression (P1).** lance-format#338 changed the exact string old
masters match on. A stale preparation produced by a new executor and
classified by an old master → `DataOrConfiguration`, one-hour cooldown —
the bug lance-format#338 set out to fix. Reproduced with old vs new classifier.
2. **Code outranks a 403 (P2).** `classify` checked the code before HTTP
status, so `HTTP 403: … [LC_STALE_PREPARATION]` → `Retryable`. The lance-format#338
description claimed a test for this; there was none. `corrected()` also
had its own rule instead of sharing `classify`'s.

## Fixes

- Emission **off by default**; `MAINTENANCE_ERROR_CODES=1` enables it
after every master/executor runs a code-aware reader. `tag()` returns
the exact legacy string when off; `tag_forced()` for tests.
- Code moved to a **suffix**; the legacy text matcher tolerates the
suffix (defence in depth for a reader with only the text path).
- `classify` order: ownership → explicit 401/403 → code → text.
`corrected()` delegates to `classify` and only softens a stored
`DataOrConfiguration`.
- `docs/concurrent-indexing.md` documents the rollout order.

## Validation

- New: 401/403 wrapping the code in either position →
`DataOrConfiguration`, and `corrected()` leaves it; emission off by
default; byte-exact legacy string still matched; tagged string
recognised by text path alone.
- `lance-context-api` 1/1, `lance-context-merge` failure 7/7,
`lance-context-core` prepared/stale 9/9; clippy `-D warnings`, fmt,
typos clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…rom running (lance-format#348)

Part of lance-format#333. Fixes a defect in lance-format#339 found in review (credit: production
review).

## Problem

`master_task_pool_in_use` was set once per poll iteration before any
permit was taken. A claim blocked on etcd for minutes — the case the
gauge is for — kept showing the pre-claim value, possibly zero.

## Fix

Gauge updated at the permit boundaries, labelled by holder:
`state="claiming"` during the etcd claim, `state="running"` from task
spawn until permit drop. Runbook row updated: running at cap ⇒ capacity;
claiming high ⇒ slow/blocked admission scan; both low ⇒ nothing
eligible.

## Validation

Clippy `-D warnings`, fmt, typos clean;
`reserved_resident_merge_finishes_while_legacy_pool_is_saturated` passes
locally against etcd (exercises the poller path).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
lance-format#350)

Part of lance-format#333 (P1.3). Spec §1.1/§2.2 (lance-format#343), on top of lance-format#344/lance-format#345.

## What

A successful WAL catch-up writes `merged_through` per shard **in the
same etcd txn that releases ownership**, so demand and ownership change
together and a planner never sees a released table whose pending count
still includes the generations just merged.

- **core**: `rollout_append::merged_watermarks(uri)` — per-shard merged
generation from the base table's MemWAL index.
- **merge**: `Coordinator::release_with_demand(claim, execution,
&[ShardMerged])`. `demand_release_changes` only advances
`merged_through`, keeps the writer's `sealed`/`epoch`/`bytes`, creates a
fresh executor-sourced record for an unknown shard, and emits no
compare/op for a shard already at or past the watermark. `release` is
unchanged (delegates with `&[]`).
- **master**: `run_as` reads watermarks after a successful `Catchup` and
releases with them; read failure logs and releases without demand. Other
kinds unchanged. Metric `master_demand_merged_watermarks_total`.

Additive; no key family removed; old masters ignore `demand-events/`.

## Validation

- `release_changes_only_advance_merged_and_keep_writer_fields` (etcd):
advance 15 keeps sealed 20 / epoch 3 / bytes; lower 12 emits nothing;
unknown shard → `sealed == merged`; fold → expected pending.
- `resident_replicas_merge_from_queue_with_no_workers_or_jobs` extended:
after both masters merge, each table has exactly two executor-sourced
events whose `merged_through` equals `merged_watermarks(uri)` and which
fold to `pending_generations == 0`.
- 7 demand tests, 9 master etcd tests (resident + maintenance_execution)
green; clippy `-D warnings`, fmt, typos clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
lance-format#354)

Part of lance-format#333. Fixes the runbook finding from the third review round
(credit: production review).

§5 told operators to "upgrade or **stop** that contender" when
compaction cannot get its commit turn. A continuous merger that keeps
winning on a hot table is working as designed and progressing; stopping
it trades a bounded fairness gap for WAL backlog and a fenced recovery.

Rewritten: states first that this is not a merger fault and never to
stop/drain/restart a progressing merge; remedies are scheduling levers
only (upgrade at next rollout, raise commit wait, remove from prepare
targets and build in a quiet window); names `MERGE_DRAIN_TARGETS` as the
wrong tool. §3: long **and** progressing is never a reason to act.

Docs only; typos clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ath (lance-format#351)

Part of lance-format#333. Fixes review finding **1 [P1]** on lance-format#345 (credit:
production review).

## Problem

lance-format#345 awaited the manifest read + etcd put inline in `route_merge`,
*before* the merge request. A hung publish ate the flush pass timeout,
skipped that table's merge request, and delayed every later table in the
pass — up to 150 s/table with defaults. Telemetry was on the scheduling
path, which P1 forbids.

## Fix

Publish is spawned and never awaited: 5 s timeout, process-wide
semaphore of 4 in-flight publishes, drop-and-count when full (the next
flush republishes an absolute watermark, so nothing is lost). All
outcomes on
`rollout_wal_demand_events_total{result=written|unchanged|no_manifest|failed|timeout|dropped}`.
`Sweepable: Clone` (all implementors are `Arc<RwLock<_>>`).

## Validation

- **New** `hung_demand_publish_never_delays_flush_or_merge_request`: a
store whose watermark read is `pending()` forever → flush pass returns
in < 2 s, merge request still issued, 8 further passes while still hung
return in < 2 s total.
- `flush_publishes_shard_demand_…` adapted to poll for the detached
write (10 s bound).
- 9/9 sweeper tests against real etcd; clippy `-D warnings`, fmt, typos
clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…mat#353)

Part of lance-format#333. Fixes review finding **5 (panic path)** on lance-format#348 (credit:
production review).

## Problem

`running.increment` before spawn, `running.decrement` after `run_task`
returns. A panic in `run_task` unwinds past the decrement: permits are
released by `Drop`, the gauge is not, so
`master_task_pool_in_use{state="running"}` reads as consumed forever.

## Fix

`GaugeHold(Gauge)` with `Drop` → decrement, held across `run_task`.
Three lines.

## Validation

Type-level: the decrement is now in `Drop`, which runs on both normal
return and unwind. No new test — asserting on a `metrics::Gauge` value
needs a recorder harness this crate's tests do not have; the existing
poller test
(`reserved_resident_merge_finishes_while_legacy_pool_is_saturated`)
passes against etcd, clippy `-D warnings` and fmt clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…t IO (lance-format#355)

Part of lance-format#333. Fixes review finding **6** on lance-format#340 (credit: production
review): the earlier test used hand-placed `checkpoint()`s, a one-row
real build, and never asserted the build outlived the idle timeout.

## Scenarios (real BTree preparation, 64 rows, 3 s idle timeout)

| | Storage | Expected | Guard against vacuity |
|---|---|---|---|
| (a) | every data/index op sleeps 1.2 s | completes | asserts ≥ 3 ops
intercepted **and** total time > idle timeout |
| (b) | parked after first op | cancelled with exact stall error in
`[IDLE, IDLE+4)` | asserts shim saw IO |
| (c) | healthy table concurrent with (b) | completes; not cancelled;
does not keep (b) alive | — |

## Support code

- `MergeWriteScope::with_preparation_authorizer_and_io_shim`: layers a
shim **beneath** the progress wrapper, so the watchdog counts the
shimmed store's completed IO exactly as real storage.
- `preparation_io::test_support::{Throttle, ThrottleWrapper}`
(`#[doc(hidden)]`).
- Uses Lance's `file-object-store://` scheme: local directory through
the ObjectStore API so wrappers apply as on cloud stores (plain `file`
paths take the direct IO fast path and bypass them — which is why the
first attempt saw 0 ops).

## Mutation check

Removing read-chunk + `get_ranges` progress accounting from
`ProgressStore` fails (a) with `slow-but-alive real build was cancelled`
after ~4 s. Restored.

## Validation

New test + `slow_but_alive_…` + `prepared_index_task_…` pass against
etcd (12 s); clippy `-D warnings`, fmt, typos clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ance-format#358)

Part of lance-format#333. Fixes review finding **3 [P1]** (round 4) on lance-format#354 (credit:
production review).

§5 advised removing a hot table from `COMPACTION_PREPARE_TARGETS` /
`INDEX_PREPARE_TARGETS` to defer its maintenance. Verified against
`eligibility::claim_shape` and `run_task`: a table not in the prepare
list takes the **legacy path**, which acquires the table write lock
*before* compacting/indexing — the next task blocks WAL publication for
the whole build. Strictly worse than repeated preparation expiry.

Rewritten: no side-effect-free per-table lever exists today; both
tempting levers named with why they are wrong; fleet-wide levers that do
exist (`COMPACTION_INTERVAL_SECS`, `quiet_hours`, `MIN_FRAGMENTS`,
manual enqueue in a quiet window); per-table `kinds_allowed` is design
§4.8, not yet built.

Docs only; typos clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ted (lance-format#360)

Part of lance-format#333. Closes the `claiming` cancellation gap noted in review
round 4 (credit: production review).

The `claiming` gauge was incremented before the etcd claim and
decremented after. A poller aborted mid-claim (drain, shutdown, test
teardown) skipped the decrement. Same `GaugeHold` Drop guard as lance-format#353,
held across the await.

Clippy `-D warnings`, fmt clean; poller test passes against etcd.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ut (lance-format#357)

Part of lance-format#333. Fixes review finding **2 [P1]** (round 4) on lance-format#350 (credit:
production review). Independent of lance-format#352.

## Problems in lance-format#350

1. Watermark read after the watchdog exited and before `release`, with
**no deadline** → a stalled storage read holds a completed merge's
ownership.
2. Release txn compared every shard's demand key by `mod_revision` → **a
normal writer flush on any shard fails the whole release**. Reproduced.
3. 129+ shards → etcd default `--max-txn-ops=128` exceeded.

## Fix

- `release` is the plain ownership txn again; `release_with_demand` /
`demand_release_changes` removed.
- After a successful `Catchup` release:
`demand_publish::spawn_merged_demand` — detached, 15 s timeout, 4 slots,
drop-and-count, **one monotonic put per shard**, bounded CAS retry on a
lost race. Loss/lateness cannot misstate demand (per-dimension join;
scan reconciles).
- Both publish paths share `join_events` + retry, so the writer path no
longer silently returns `Ok(false)` after losing a race either.
- `master_demand_merged_watermarks_total{result}`.

## Validation

- New etcd test: writer `sealed=25` and executor `merged=18` published
**concurrently** on one shard → both succeed, stored `(25, Some(18))`;
lower merged no-op; unknown shard → fresh epoch-0 record.
- Resident two-master merge test polls for the detached publish; both
shards fold to zero pending.
- 7 demand, 9 maintenance/resident, 9 sweeper tests green against etcd;
clippy `-D warnings`, fmt, typos clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…wer schema (lance-format#359)

Part of lance-format#333. **Supersedes lance-format#352** (which I will close). Fixes review
findings **1 [P1], 4, 5 [P2]** (round 4) (credit: production review).
Stacked on lance-format#357.

## What lance-format#352 got wrong

It ordered marks by `(epoch, value)` — treating a writer epoch change as
a numbering restart. Lance keeps generation numbering on a new writer
claim (`ShardManifest { writer_epoch: next, ..base }`). So "merged to
40, new writer flushes to 60" is **20 pending**; lance-format#352 said 60 in both
orders.

## Fix

- `Mark` orders by `(value, epoch)`; epoch is a fence/tiebreak. `pending
= sealed − merged`, plain.
- Stored record is the per-dimension join; `fold` applies the same join
→ **rebuild from etcd equals in-memory fold**, now including the age
clock, which the stored record carries (`sealed_times`, bounded to 64
lowest unmerged, min per generation). Finding 4: in-memory 10 vs rebuilt
20 before.
- `merged_epoch` keeps provenance when merged was reported under a
different epoch.
- Publishers **refuse to overwrite a stored record from a newer schema**
(finding 5: `v=999` was overwritten). Both publish paths share one
read-join-CAS with bounded retry.
- Spec §1.1 rewritten.

## Validation (15 tests, all green against real etcd)

- Named counterexamples: round-3 (both), round-4 (merged 40 / epoch-2
flush 60 ⇒ 20, in every order, and after rebuild).
- Writer never forgets merged; lower value from any source is a no-op;
marks order by generation with epoch tiebreak.
- Age clock: stable across newer flushes, moves correctly on merge,
**identical after rebuild**; beyond-cap still exact for counts.
- Schema: enforced on fold; stored `v=999` untouched by both publishers
(mod_revision unchanged).
- Concurrent writer + executor publishes both succeed and both survive.
- **Randomized**: 200 × 5-event streams (every permutation) and 50 ×
200-event (reverse, shuffle) over 2 shards / 3 epochs / 3 sources — fold
order-independent **and** `rebuild(events) == fold(events)`.
- Resident two-master merge and 9 sweeper tests pass; clippy `-D
warnings`, fmt, typos clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
lance-format#356)

Part of lance-format#333 (P1.4). Spec §1.3, source `scan`. **Stacked on lance-format#359**
(rebased from the superseded lance-format#352); merge after lance-format#357 and lance-format#359.

## What

- **core**: `all_shard_watermarks()` — every shard's `(shard_id,
writer_epoch, max flushed generation, pending)` from its latest
manifest. Does **not** enforce the read cap (the scheduler must see an
over-cap shard, not be refused).
- **master**: `demand_publish::spawn_scan_demand` after each successful
observation. Detached and bounded (15 s, 4 slots, drop-and-count), one
`scan`-sourced event per shard via `publish_demand_event`'s
per-dimension join, so a scan can only raise and never lowers a `merged`
it does not know. Metric `master_scan_demand_events_total{result}`.
Existing enqueue/retire behaviour untouched (review point 5 / P1 rule).

## Validation

- `scan_publishes_every_shard_and_never_lowers_a_watermark` (etcd): two
writer shards, no server flush path → one event per shard, fold = 6
pending; executor reports `merged=2` on a shard; a second scan leaves
`merged` intact and does not bump `mod_revision`.
- 15 scanner unit tests; clippy `-D warnings`, fmt, typos clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…lance-format#361)

Part of lance-format#333 (P1.5). Spec §1.2/§3. The gate (lance-format#359) is merged.

## What

Behind `PLANNER_ENABLED` (default **off**): one master wins `P/leader`
under a lease and every `PLANNER_RECONCILE_SECS` folds all
`demand-events/` into a per-table `P/demand/<hex>` cache via
`TableDemand::fold`. Rows for tables with no events are dropped. The
cache is a pure function of stored events — never a source of truth.

**Shadow only. Places nothing, removes nothing; every existing loop
keeps running** (review point 5).

Outputs: the cache; `GET /api/v1/scheduler/demand[?target=]` (leader
record + per-table pending/bytes/age/shards + full record); metrics
`planner_is_leader`, `planner_tables`,
`planner_pending_generations_total`, `planner_reconcile_seconds`,
`planner_events_skipped_total`, `planner_reconcile_errors_total`.

## Leadership

Campaign txn on `Version(P/leader)==0` writes the record **and** a bare
`P/leader-token` under one lease. Every cache-write txn compares the
token; every reconcile reads it first even with nothing to write, so a
deposed leader notices on the next tick. Writes chunked under etcd's
128-op limit. Losers return promptly and re-campaign after 2 s.

## Validation

- Unit: `fold_events` groups by table and skips bad key / bad body /
future version (counted); hex round-trip.
- etcd: two masters campaign → **exactly one leads** (the test detects a
dual leader; mutation that ignores the campaign result fails it with
`finished=[false, false]`); cache == direct fold; HTTP view shows leader
and 42 pending on a two-shard table; forged token → `leadership lost`;
deleting a table's events drops its row next tick.
- Clippy `-D warnings`, fmt, typos clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ons (lance-format#362)

Part of lance-format#333 (P1.6). Spec §2.

## What

Behind `PLANNER_ENABLED`: every master publishes a **leased**
`P/executors/<id>` heartbeat — kind, slots per task kind mirrored from
the real dispatch pools (general / compact / merge + resident lane),
byte budget from the local append budget, a diagnostic
`bytes_free_sample`, `draining`. Drain stops the heartbeat and revokes
the lease.

**Headroom** is a pure fold of heartbeats + `P/assignments/` records:
`slots_total − Σ reserved_slots`, `bytes_total − Σ reserved_bytes` over
every non-`Released` assignment **including unbound ones**, never the
sample (review point 4). An assignment whose executor has no heartbeat
still reserves (shown as draining, zero totals) so capacity cannot be
double-booked across a failover.

Planner loads headroom on every reconcile; `GET
/api/v1/scheduler/executors` exposes it. **No assignments are written
yet** — `Shadow` state is defined for the next PR. `TaskKind` gains
`PartialOrd/Ord/Hash` for the slot maps.

## Validation

- Spec test 3: sample says 2 GiB free, 2 GiB reserved (Assigned +
Shadow) → `fits` false; Released doesn't count.
- Spec test 4: assignment outlives a missing heartbeat.
- Draining takes nothing; zero byte budget defers to slots; fold
order-independent.
- etcd: leased heartbeat with configured slots (merge 3, index 2);
shadow assignment drops merge headroom 3→2; `begin_drain` ends loop, key
gone, assignment still reserves.
- Clippy `-D warnings`, fmt, typos clean; api crate tests pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…format#363)

Part of lance-format#333 (P1.7 — last shadow-phase piece). Design §4.2, spec §2.3.

## What

- **`scoring.rs`** (pure): `merge_score = max(pending/min_gens,
bytes/min_bytes) × (1 + wait/max_age)`. Classes: **Critical** (≥256
pending, or promoted for age ≥ `max_turn_wait_secs`), **Normal** (score
≥ 1), **Tail**. `planner_order`: class asc; **class 1 oldest-first,
never by score** (review round 2); score desc elsewhere; name tiebreak →
total order.
- **Shadow pass** in the planner, every reconcile: score → order → skip
tails (real sweeps do too) → place on fittest executor by headroom →
write `Shadow` assignments (replaced wholesale per pass, leader-token
guarded). Then compare with reality (real `MergeWal` task active?) →
`scheduler_shadow_disagreements_total{kind}` + per-pass gauge. **This is
the number that must approach zero before P2.**
- Gauges: `planner_merge_score{target}`, `planner_units{class}`,
`planner_shadow_placements`, `planner_shadow_unplaceable`.
- **Spec test 5** as a source-level test: no file outside
`planner.rs`/`executors.rs`/operator route references assignments.
- Deferred to P2: compaction/index scoring, class 2 commit-ready,
`max_consecutive_turns`.

## Validation

- Scoring: thresholds; age doubles score at `max_age`; promotion for
age; **class 1 oldest-first beats a 100× score** (spec test 6); class
before score; total stable order (20 items, forward == reverse).
- Planner etcd test extended: exactly one Shadow assignment for the
42-pending table on the live heartbeat, none for the empty table,
headroom shows it, enqueuing a real task → next pass replaces not
accumulates.
- Isolation test passes; clippy `-D warnings`, fmt, typos clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Part of lance-format#333. Fixes all six findings of review round 5 (credit:
production review), which brought real numbers: **13,405 tables, 354
with pending, 318 generic**.

| # | Finding | Fix |
|---|---|---|
| 1 [P1] | Full prefix read of ~27k records exceeds the 4 MiB etcd
client cap | Every planner read paged (512 keys) **at one revision**.
Reconcile detached from the leader loop → keepalive cannot starve; one
pass in flight. |
| 2 [P1] | Scan read only `sealed`; a lost completion event left a
merged shard pending forever; old table's first contact → gen 1001 =
1001 pending | `all_shard_watermarks` reads the base MemWAL index's
merged watermark per shard; scan events carry it. |
| 3 | `generic:generic:<name>` | `entry.name` already has the prefix. |
| 4 | 1-byte reservations when `pending_bytes == 0` (all real
publishers) | Unknown reserves the **full per-pass cost** (`2 ×
ROLLOUT_APPEND_MAX_BYTES`); counted. |
| 5 | "disagreement ≈ 0" is not a takeover gate |
`planner_shadow_comparison{outcome ∈ discovery_miss, placement_gap,
phantom_task, agree}`, each a precise question; documented as
observations. Old counter kept as the sum. |
| 6 | 128 pending, merge 64 → age unknown / wrong after rebuild; slow
reconcile starves keepalive | Times map bounded by cap only (lowest 63 +
newest), filtered by merged on read; answer carries `Exact`/`Upper`;
**only Exact promotes**. Executor events no longer seed flush times.
Keepalive: see #1. |

## Validation

- **Production-shape test**: 13,405 × 2 shards seeded; asserts the
**unpaged read fails** (so the test cannot pass vacuously), paged read
returns 26,810 at one revision, fold counts them, leader completes a
reconcile and holds its lease 12 s past it, cleanup verified. Gated on
`PLANNER_SCALE_TEST=1` (run alone; it starves a shared single-node
etcd).
- Demand: 128/64 case → `Upper` bound, same after rebuild and in
reverse; randomized `rebuild == fold` still holds; scan-with-merged
folds.
- Scoring: a bound never promotes or boosts.
- 16 demand, 5 scoring, 17 etcd (planner/executor/scan/resident), 9
sweeper green on clean etcd; clippy `-D warnings`, fmt, typos clean.
Spec §1.1/§1.2a/§1.3/§2.3 updated.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…rmat#365)

Part of lance-format#333. Follow-up to lance-format#364.

Review round 5 measured production: **13,405 tables, ~354 with pending,
318 generic, 6 masters**. The design assumed "tens → low hundreds
active" — wrong by two orders of magnitude — and that assumption
directly produced the unpaged etcd reads and 1-byte reservations fixed
in lance-format#364.

Corrects §2 (workload table, with the error acknowledged), §4.5
(per-cycle cost now stated with paging and measured at production
shape), §6 (when the single leader would need watch-driven folding or
sharding), and marks backlog open question 1 answered.

Docs only; typos clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…oors (lance-format#366)

Part of lance-format#333. Fixes all five findings of review round 6 (credit:
production review).

| # | Finding | Fix |
|---|---|---|
| 1 [P1] | 512-key page bounds count, not bytes; 512×8 with full time
maps → 5.6 MB → fails and loops | Adaptive pages: start 512, halve on >1
MiB or message-size error, grow back; one revision |
| 2 [P1] | Writes chunked by op count only; 100×20-shard records ≈ 2.7
MB > 1.5 MiB request cap | Chunk by ops (100) **and** bytes (768 KiB),
sized per op |
| 3 [P2] | Long-lived tail never promotes: first 63 timestamps pin the
map; Upper bounds excluded | Upper bound on flush time = **floor on
wait**; promotion uses the floor from either bound (sound); score aging
uses exact only |
| 4 [P2] | Comparison conflated queued/running, skipped unplaceable
tables, wrote zeros on read failure | Separate paged queued/running
sets; per-due-table `discovery_miss / queued_only / running /
placement_gap / phantom_task`; skip-and-count on read failure |
| 5 [P2] | Estimate capped at 2×max_bytes but a whole generation is read
first | `2 × max(max_bytes, largest generation seen)`; whole local
budget when unknown |

## Validation

- `chunk_ops`: count-bound, byte-bound, oversized-op-alone.
- Scale test at the reviewer's shapes (`PLANNER_SCALE_TEST=1`, run
alone, drop-guard cleanup): 512×8 full maps — fixture asserted **> 4 MiB
total and unpaged read fails**, paged read returns 4,096; 612-table
reconcile incl. 100×20-shard records writes within limits.
- Demand: 70 merged-then-2-pending tail → usable day-long wait floor,
identical after rebuild.
- Scoring: a bound promotes on wait, adds no age boost.
- 16 demand, 5 scoring, 19 etcd planner/executor/scan/resident, 39
merge-crate tests green on clean etcd; clippy `-D warnings`, fmt, typos
clean. Spec §1.1/§1.2a/§2.3 updated.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Part of lance-format#333. Review round 7 cleared a single-master shadow canary in
production; this is the runbook for it.

Covers: what `PLANNER_ENABLED=1` does on one master (observe and
simulate only — no merge behaviour changes), how to confirm it is
leading, what to watch and in what order (master RSS, etcd latency,
reconcile completion, leadership flaps, real merge progress,
pending/503), what each `planner_shadow_comparison{outcome}` means and
does **not** mean, what the canary cannot tell you (scheduling
correctness, parallel capacity, compaction/index), how to disable, and
when to widen to a second master. Explicit that P2 does not follow from
canary data alone.

Docs only; typos clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ax (lance-format#367)

Part of lance-format#333. Fixes the two code findings of review round 7 (credit:
production review), which otherwise cleared a **single-master shadow
canary in production**.

| # | Finding | Fix |
|---|---|---|
| 1 | `GET /scheduler/demand` list mode still one unpaged read → fails
at 13k tables | Paged read reduced to a bounded summary: pending-sorted,
`limit` (default 200, max 2000), `include_idle`,
`total_tables`/`tables_with_pending`. Full record only for `?target=`.
Response size independent of cache size. |
| 2 | Byte estimate used `pending_bytes / pending_generations` (an
average) as the max | Every shadow reservation is the executor's whole
local budget — the only honest value for an unknown max. Spec now says
the shadow capacity model answers "can one unit fit", **not**
parallelism; `placement_gap` must not be read as such.
`planner_shadow_reservation_basis_total{basis}`,
`planner_shadow_average_generation_bytes` record the basis. |

## Validation

Planner etcd test extended: list view with `limit=1` returns only the
pending table, record omitted, `total_tables=2`,
`tables_with_pending=1`. 18 planner/routes tests green; clippy `-D
warnings`, fmt, typos clean. Spec §2.3 and observability list updated.

Finding 3 (real-traffic observation: master RSS, etcd latency, reconcile
completion, merge progress, pending/503) is the canary itself, not code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
lance-format#371)

Part of lance-format#333. Closes the two test gaps left open across review rounds 5
and 7. Test-only; no behaviour change.

## Leader failover (spec test 4)

Master A leads, builds cache + shadow placements over three tables, is
**aborted without releasing** (lease expires naturally). Its token is
then refused. Master B campaigns with only etcd and must rebuild
**identical** demand cache, shadow placements (target/executor/bytes),
executor totals, and keep a durable `Bound` reservation written by the
old leader.

Mutations caught: second leader placing differently → placements
assertion; rebuild dropping `Bound` → durable-reservation assertion.

## Sustained concurrent isolation

Four real index preparations at once over several idle windows:
throttled-alive ×2 (different rates), frozen-from-start, frozen-midway
after real progress. Asserts: alive pair finishes past the idle window
with real IO and the slower one takes longer; frozen-start cancelled in
`[IDLE, IDLE+4)`; frozen-mid cancelled in that window **measured from
its own freeze**; alive builds outlive both cancellations. Whole
scenario bounded at 60 s so a never-cancelled build fails fast.

Mutation caught: a shared progress counter across watchdogs fails the
test.

## Validation

Both green against real etcd (9 s, 16 s); planner/watchdog/isolation
suites green; clippy `-D warnings`, fmt, typos clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…at#372)

Recovery health checks currently run Kubernetes exec on every
worker/master to read container memory and OOM state. Expose a bounded
read-only `/api/v1/internal/runtime-resources` endpoint through their
existing shared router so managed recovery can obtain the same evidence
directly.

Each request reads the three cgroup-v2 memory files, with an 8 KiB limit
per file, and returns usage, nullable limit, raw OOM counters, sample
time and an optional caller nonce. Responses are not cached;
missing/malformed/unsupported cgroups return 503 rather than stale data
or invented zero values. It does not access stores or alter writer
ownership. Callers must still validate Pod/container identity,
readiness, expected configuration, freshness and their existing
admission thresholds.

Validation: `cargo test -p lance-context-metrics --locked` (4 tests
passed) and `cargo clippy -p lance-context-metrics --all-targets
--locked -- -D warnings` pass. Tests cover fresh changes, failed reads
after previous success, unlimited budgets, retained OOM evidence,
malformed/oversized files, and shared-router wiring. Only the existing
serde_json dependency is added to this crate's lock entry.

Co-authored-by: Beinan Wang <beinanwang@microsoft.com>
…rmat#373)

Recovery currently depends on ingress log streams to identify hot tables
and table-specific read failures. Add opt-in bounded application request
hints to the existing HTTP middleware and an internal nonce-bound JSON
endpoint, so an observer can discover demand even while a handler is
still waiting.

Requests never wait for the cache. Capacity, TTL and reply bytes are
bounded; dropped/truncated observations remain explicitly incomplete and
recover when loss ages out. Maintenance routes do not generate their own
demand. Hints retain no payload/query/record identifiers and cannot
authorize merge replacement or prove progress. The endpoint does not
observe ingress-only or streaming-body failures.

Validation: metrics crate unit and actual Axum middleware integration
tests pass for enabled/disabled processes; Clippy passes with warnings
denied. An uncontended release microbenchmark measured 1M parse+record
operations in 410 ms (not a production latency claim). See the crate
documentation for rollout and coverage semantics.

Co-authored-by: Beinan Wang <beinanwang@microsoft.com>
## Summary
Upgrade Rust Lance and optional Python pylance dependencies from 9 to
13.0.0, including the SSTable metadata, explicit manifest refresh,
SealFence, and object-store API adaptations. Preserve exact-version
merge authorization across Lance's internal commit retries; a regression
test races a concurrent manifest commit with authorization revocation.

Before deploying to existing MemWAL datasets, flush and merge all old
pending generations using the old binaries, then upgrade readers and
writers together. The upstream field-ID change prevents safely mixing
old flushed generations with new readers; see
`docs/design/lance-13-upgrade.md`.

## Testing
- Passed the repository's full `run_ci_checks.sh`: workspace format,
all-target Clippy and tests; Python pytest (229 passed, 2 skipped, 1
expected failure), Ruff format/lint, and Pyright (0 errors/warnings).
- Built `lance-context-server` and the Python extension against the
upgraded dependencies.
- Concurrent-manifest/revocation regression passed with the core suite.
GitHub's Rust/etcd HA, coverage, Python test-wheel matrix and Python
test jobs also passed.
## Summary
Add a feature-gated, unpublished Rust WAL experiment crate to compare
Arrow IPC decoding/row-copy costs and local durable log writes before
integrating the distributed WAL. The codec benchmark verifies complete
Arrow equality and actual payload buffer sharing, including unaligned
input; the raft-engine benchmark performs synced writes across groups,
reopens storage, and compares every payload.

Results are JSON with allocation counts and separate
write/recovery/verification timing. The README documents measurement
limits, and CI builds/lints the optional binary and runs tiny
correctness smoke cases.

## Testing
- Passed the repository's full `run_ci_checks.sh`: workspace format,
all-target Clippy and tests; Python pytest (229 passed, 2 skipped, 1
expected failure), Ruff format/lint, and Pyright (0 errors/warnings).
- Component check and all-target Clippy with `--features lab` passed.
Codec smoke passed all five paths and shared-buffer checks; disk smoke
reopened and verified 64 entries.
- Spelling and diff checks passed. Rebasing after lance-format#374's squash merge
preserved the exact source tree tested locally.
…at#376)

## Summary

Add the optional `placement` module to the independent WAL crate. It
routes a tenant/table/session key to a stable virtual bucket, validates
an immutable RF3 layout, and scopes receipts by cluster, layout
fingerprint, group and log index. Invalid or ambiguous ownership is
rejected before routing; changing a validated layout requires migration.

The hash formats match the existing Rust/PHX experiments. This PR
supplies placement types only; it does not connect a distributed writer
to production ingestion.

## Testing

- Full `.codex/skills/ci-pr-helper/scripts/run_ci_checks.sh`: workspace
Rust formatting, Clippy and tests; Python 229 passed, 2 skipped, 1
xfailed; Ruff and Pyright passed.
- `cargo test --release -p lance-context-wal --features placement`:
three tests, including the recorded PHX layout fingerprint and an
independently checked session hash vector.
- `cargo clippy --release -p lance-context-wal --features lab,placement
--all-targets -- -D warnings`.
- Spelling and diff whitespace checks passed. CI explicitly exercises
the optional module.
## Summary
Multiple WAL groups on one node need a shared proposal budget so
concurrent producers cannot each consume the full disk allowance. Add an
opt-in `admission` module with atomic reservations, drop-based release,
encoded-write accounting, and a reserved control window. Refresh uses
real file lengths and allocated blocks; successful admission does not
scan the directory.

This module is not connected to production ingestion. It does not
reserve filesystem extents or cap previously accepted replication
traffic; replica-pressure coordination and snapshot collection remain
necessary.

## Testing
- Release admission/placement tests: five passed, including racing
reservations and reclamation while proposals remain pending.
- Release Clippy with `lab,placement,admission`, formatting, spelling
and diff checks passed.
- Full local CI script passed: workspace Rust Clippy/tests, Python
pytest, Ruff format/lint and Pyright.
- CI explicitly tests and lints the optional module.

Refs lance-format#378.
beinan and others added 29 commits October 21, 2026 18:51
…e-format#458)

A hot table discovered early in a stats scan currently waits for every
remaining table to finish before the merge sweep can see its pending
WAL. On one production master, completed scans averaged about 92
minutes; this is a discovery delay independent of the separate
legacy-worker admission blocker.

Consume each completed observation immediately and submit eligible
targets to the existing durable merge queue. Incremental observations
also qualify because they recount pending WAL. Preserve automatic-merge
disablement, canonical deduplication, target encoding and all existing
execution/ownership rules. Each scan can admit at most 256 new tasks and
spend at most 10 seconds on queue admission, with a 2-second individual
bound. An ambiguous queue error stops early admission for that round
while observation and snapshot publication continue.

Validation: formatting and diff checks pass. Three real isolated-etcd
regressions cover enqueue before a blocked observation completes,
disablement/threshold/deduplication/budgets, and queue failure
preserving the entire scan result. The final committed source passed all
18 scanner tests, including the three isolated-etcd regressions (0
failed or ignored); its commit and diff fingerprint stayed unchanged
throughout the run. CI is running.

This adds no executor pool or rollout mode. It does not resolve
historical unfenced requests, authorize replacement of an old writer, or
bypass `core_not_ready` in the external recovery controller. It is
separate from the already-building lance-format#454/lance-format#456 release.

Co-authored-by: Beinan Wang <beinanwang@microsoft.com>
…at#449)

## Summary

Add an independent rollout-ingestion adapter using the durable WAL's TLS
client interface. It verifies the authorized table generation and
rollout contract at quorum, reserves shared memory before buffering,
validates Arrow IPC and session-bucket boundaries, and retains the
original bytes for exact retries. Durable ACKs produce authenticated
receipts that can be restored after restart and checked against compact
publication visibility.

The adapter lives in the unpublished `lance-context-ingest` crate so the
existing publishable core does not acquire an unpublished WAL
dependency. Applications must persist producer-open and append retry
state before sending; HTTP routing, that journal wiring and the
materializer loop remain separate integration work.

Refs lance-format#378.

## Testing

- Full repository CI-equivalent checks, including Rust, Python, Ruff and
Pyright.
- Real TLS three-replica test with a real Lance genesis manifest, all
original replica directories restarted, exact retries, and complete
metadata/payload reads.
- IPC validation, shared-memory pressure, virtual buckets sharing a Raft
group, receipt tampering/key rotation/scope, producer fencing, and
compact quorum visibility bounds.

Built on merged lance-format#448. The change is now based on main with an identical
feature diff; all checks must pass for the new commit before merging.

CI correction: registry packaging excludes the private ingress adapter;
its Rust feature tests remain required. Linux wheel setup uses bounded
downloads from the official AlmaLinux repositories, with TLS and package
signature checks preserved.

APT wheel setup now uses only signed official distribution repositories
in an isolated metadata directory. This avoids unrelated PPA outages;
bounded network retries, HTTPS and archive signature verification remain
enabled. Bash/ShellCheck and a local Debian metadata/install simulation
passed; the Ubuntu cross-build installation is validated by remote CI.


CI runs the existing workspace, WAL, Lance, and etcd suites in separate
jobs so cold feature builds do not exhaust one shared 60-minute timeout.
Every original check command and the coverage jobs are preserved.
Checkpoint reopen cases are serialized around subprocess creation to
prevent inherited file locks; corruption cases must report the expected
validation error. Minimal-feature and all-feature checkpoint tests and
Clippy pass; the workflow passes actionlint. Final-commit CI is required
before merging.
)

## Summary

An RPC connection can remain in the client pool after the WAL server
closes it for inactivity. The next catalog read then fails with a TLS
EOF or broken pipe, even when the cluster is healthy. Recover Query,
Fetch, and Status requests once on a fresh, authenticated connection
when a reused stream disconnects.

The reconnect stays within the original request deadline and holds the
existing connection and byte credits. Append and Commit are never
automatically replayed; TLS authentication, response scope, correlation,
and payload validation failures remain errors.

Refs lance-format#378.

## Testing

- Reproduced the failure against the unchanged client with real
three-node Raft runtimes and TLS listeners; the regression passes with
the fix.
- `cargo test -p lance-context-wal --features peer-transport --lib`: 130
passed, 1 existing ignored test, including process restart tests.
- `cargo clippy -p lance-context-wal --features peer-transport
--all-targets -- -D warnings`: passed.
- `cargo fmt --all -- --check`: passed.
- Added coverage for direct and routed reads, fetch payloads, mutation
uncertainty, a second disconnect, stalled reconnect, invalid
scope/correlation, and released admission credits.
- Full workspace checks are running separately; no throughput
improvement is claimed by this correctness fix.
## Summary

A bounded runtime can reject materializer queries because response-byte
credits are exhausted even while request slots remain available. Expose
local admission observations for data, control, and incoming/outgoing
peer budgets, including capacities, usage, and separate request/byte
rejection counters. Counters change only on rejected admission;
successful requests keep the existing accounting path.

Fields are sampled independently and describe local pressure, not quorum
state. Refs lance-format#378.

## Testing

- 127 WAL library tests passed with `peer-transport` enabled; one
existing ignored test.
- A real three-node quorum test retains 16 Publisher query results: the
4 MiB control budget admits 2 and rejects 14 for byte pressure; 32 MiB
admits all 16. A separate two-request limit reports request pressure,
and consuming results restores admission.
- Feature-enabled Clippy and workspace formatting passed.

This exposes diagnostics; it does not change budget defaults or claim an
end-to-end throughput improvement.
…ormat#450)

## Summary

Durable rollout ACKs need a recoverable path into queryable Lance files.
The materializer selects bounded prefixes across session-sharded WAL
groups, fetches their Arrow IPC, and writes Lance fragments with bounded
parallelism. It records prepared descriptors before publication,
reconciles uncertain outcomes, and retires source payloads with
catalog-issued closure proofs. Takeover reuses prepared files or
completes retirement without republishing rows.

Refs lance-format#378. Built on merged lance-format#449; the feature diff is unchanged when
moved to main.

## Testing

- Real TLS replicas and complete rollout schemas: restart original
directories after preparation and publication, verify descriptors and
every row/column, reject stale owners, and preserve exact append
retries.
- Bounded manifest history, multi-round publication, and
client/catalog/Lance reader regressions.
- Previous feature revision passed repository Rust and Python checks
(229 passed, 2 skipped, 1 xfailed), formatting, Clippy, Ruff and
Pyright. CI must pass again for the new commit before merge.

These checks establish correctness; they do not establish sustained
throughput.
…#451)

## Summary

Add `context-wal-node`, an independently deployable Rust WAL replica
service. Validated, persisted configuration starts durable replication,
mutually authenticated peer and snapshot transport, separate client
data/control listeners, automatic maintenance, and joined shutdown.
Local diagnostics expose runtime and disk state; the executable does not
depend on Lance.

Refs lance-format#378. Built on merged lance-format#449; the feature diff is unchanged when
moved to main.

## Testing

- Six-process service test with two disjoint three-voter groups:
checkpointing, compacted-follower snapshot catch-up, leader SIGKILL,
complete restart, payload verification, and exact retries.
- Library/process regressions, service-feature Clippy, and standalone
build/dependency validation.
- Previous feature revision passed repository Rust and Python checks
(229 passed, 2 skipped, 1 xfailed), formatting, Clippy, Ruff and
Pyright. CI must pass again for the new commit before merge.

These checks establish correctness; they do not establish sustained
throughput.
…#461)

## Summary

Node diagnostics currently expose disk admission but cannot show whether
runtime requests are rejected by request slots or byte capacity. Include
the existing bounded admission observations in node status and readiness
responses for data, control, incoming peers, and outgoing peers. The
values are sampled independently, exclude transport budgets, and do not
change readiness decisions.

Refs lance-format#378. Built on merged lance-format#451 and lance-format#460.

## Testing

- Formatting and service-feature Clippy passed.
- All three node-service tests passed, including six actual replica
processes across two groups, snapshot catch-up, leader termination, and
restart.
- The real-quorum publisher admission regression passed: byte pressure,
request pressure, and credit release remain distinguishable.
- The same feature also passed a three-process release smoke check with
separately configured control budgets before promotion onto main.
- Repository CI-equivalent checks passed: workspace Rust tests,
formatting, Clippy, Python tests, Ruff, and Pyright. Nine Python
fixtures initially could not find the server in the shared-target build
environment; all nine passed after building and placing the correct
executable. Final-commit CI is still required before merge.
…format#453)

## Summary

Add an HTTP gateway for durable rollout ingestion and a Rust HTTP
producer that journals exact retry state. Gateways accept the original
Arrow IPC buffer, reserve memory before buffering, separate data/control
admission, and return 202 only after the WAL quorum has persisted the
append. Signed producer credentials work across gateways without another
producer-allocation RPC on each append.

Producer lanes share one bounded group-commit journal and HTTP pool.
Original bytes and sequence are durable before transmission; the ACK is
durable locally before the body is released or the sequence advances. An
uncertain network result preserves the original intent, a different body
cannot replace it, and local journal ambiguity fences the lane until
reopen. Large digest/byte comparisons run off the async executor.

The gateway takes authorized fixed table bindings and must sit behind
application access control. These are application library interfaces;
deployment, dynamic namespace integration and sustained performance
qualification remain follow-up work.

Refs lance-format#378. Built on merged lance-format#450. The feature diff is unchanged when
moved to main; all CI must pass on the new commit before merging.

## Testing

- Complete repository CI-equivalent script passed: Rust, Python (229
passed, 2 skipped, 1 xfailed), formatting, Clippy, Ruff and Pyright.
- HTTP and HTTP-client/materializer feature checks passed, along with
all five integration tests.
- Real HTTP to TLS replicas to Lance: inject an intermediary error after
a durable ACK, reopen the client journal/gateway/all original replica
directories, recover the exact ACK, reuse prepared Lance files after
another restart, and verify every row/column and subsequent producer
sequences.
- Data-lane saturation keeps control requests available; draining joins
a disconnected partial-body reader without advancing its sequence.
Producer/receipt domain separation and tampering checks remain required.
- Native client-journal recovery regressions passed (10 tests, with two
child fixtures invoked by their parent tests).

The new application restart tests use joined shutdown/reopen, not
application SIGKILL or throughput measurements.

APT wheel setup now uses only signed official distribution repositories
in an isolated metadata directory. This avoids unrelated PPA outages;
bounded network retries, HTTPS and archive signature verification remain
enabled. Bash/ShellCheck and a local Debian metadata/install simulation
passed; the Ubuntu cross-build installation is validated by remote CI.


CI runs the existing workspace, WAL, Lance, and etcd suites in separate
jobs so cold feature builds do not exhaust one shared 60-minute timeout.
Every original check command and the coverage jobs are preserved.
Checkpoint reopen cases are serialized around subprocess creation to
prevent inherited file locks; corruption cases must report the expected
validation error. Minimal-feature and all-feature checkpoint tests and
Clippy pass; the workflow passes actionlint. Final-commit CI is required
before merging.
lance-format#455)

## Summary

Coordinate materializer processes with bounded etcd claims and durable
WAL publisher epochs. A worker reads the WAL epoch before validating its
claim, so an expired owner cannot steal a successor's newer epoch.
Prepared Lance files survive ownership changes and can be reused
exactly.

Workers share memory and WAL client pools, bound concurrent tables and
publication rounds, and join admitted file work and lease renewal on
shutdown. Busy-table probes use a local, keys-only hint before
attempting a lease, avoiding quorum barriers and replicated grant/revoke
writes on occupied-table polls. Ownership still requires the
linearizable create-if-absent transaction, and claim validation remains
linearizable. Managed readers can open the selected Lance version
without acquiring publisher ownership.

Assignments are an application-authorized fixed set. Dynamic namespace
discovery, application process startup, and sustained throughput
qualification are follow-up work.

Refs lance-format#378. Built on merged lance-format#453. The feature diff is unchanged when
moved to main; all final-commit CI checks must pass before merging.

## Testing

- The complete repository CI-equivalent script passed: Rust, Python (229
passed, 2 skipped, 1 xfailed), formatting, Clippy, Ruff and Pyright.
- The final busy-claim optimization was qualified separately with real
etcd tests and worker-only / worker plus HTTP-client feature Clippy
checks.
- Revoke a claim after real Lance preparation, acquire a successor
epoch, reject the old claimant, reuse the exact prepared result, and
publish through two competing automatic workers with full-column
readback.
- Abandon a claim and require actual TTL expiry before takeover; all RPC
and expiration deadlines remain enforced.

These are correctness checks. They do not establish process-kill
recovery or sustained throughput for the application.


CI runs the existing workspace, WAL, Lance, and etcd suites in separate
jobs so cold feature builds do not exhaust one shared 60-minute timeout.
Every original check command and the coverage jobs are preserved.
Checkpoint reopen cases are serialized around subprocess creation to
prevent inherited file locks; corruption cases must report the expected
validation error. Minimal-feature and all-feature checkpoint tests and
Clippy pass; the workflow passes actionlint. Final-commit CI is required
before merging.

Independent Raft-log recovery cases exclude sibling subprocess-creation
windows, preventing inherited file locks from racing immediate reopen.
The two-group concurrent-write case and real SIGKILL, rotation, and
write-error assertions remain unchanged. All nine Raft-log cases pass
with minimal and all features, and Clippy passes. This adjustment
changes only test scheduling.

TLS process startup failures include the child stdout/stderr and
identify peer/image listener binding errors. The unchanged quorum,
snapshot-interruption, SIGKILL and full-restart assertions pass in five
complete peer-transport library runs (126 tests per run), plus six
process tests and feature Clippy. The intermittent CI startup failure
has not been reproduced locally; these diagnostics do not establish its
root cause. Final-head CI remains required.
…rmat#457)

Initialize a managed rollout generation in Lance and the durable WAL
catalog using persisted operation identities. Retrying after a crash
reuses the original genesis manifest; initialization rejects an existing
foreign table instead of adopting or overwriting it.

The regression tests use real Lance files and replicated WALs, including
a restart after the genesis file exists but before catalog selection.
Focused ingestion/bootstrap tests, feature Clippy, and the complete
repository checks pass (Python: 229 passed, 2 skipped, 1 xfailed).

Refs lance-format#378. Built on merged lance-format#455. The feature diff is unchanged when
moved to main; all final-commit CI checks must pass before merging.

Independent Raft-log recovery cases exclude sibling subprocess-creation
windows, preventing inherited file locks from racing immediate reopen.
The two-group concurrent-write case and real SIGKILL, rotation, and
write-error assertions remain unchanged. All nine Raft-log cases pass
with minimal and all features, and Clippy passes. This adjustment
changes only test scheduling.

TLS process startup failures include the child stdout/stderr and
identify peer/image listener binding errors. The unchanged quorum,
snapshot-interruption, SIGKILL and full-restart assertions pass in five
complete peer-transport library runs (126 tests per run), plus six
process tests and feature Clippy. The intermittent CI startup failure
has not been reproduced locally; these diagnostics do not establish its
root cause. Final-head CI remains required.
…at#462)

## Summary

Compose the durable ingestion components into a Rust executable with
bootstrap, HTTP ingress, materializer worker, and managed-reader
inspection roles. A fixed deployment-authorized registry shares WAL
connections and memory admission across tables; startup validates
existing generations, and shutdown joins admitted work before releasing
claims.

WAL connections use mutual TLS and pinned endpoints. HTTP requests
require a workload token and expect TLS termination at the edge.
Configuration parsing rejects unknown fields and bounds file reads.
Dynamic namespace discovery, legacy-table migration and sustained
scaling remain follow-up work.

Refs lance-format#378. Built on merged lance-format#457. The feature patch is unchanged after
moving to main; all final-commit CI checks must pass before merging.

## Testing

- Cargo formatting and service/http-client Clippy passed.
- All 7 ordinary ingestion feature tests passed, including exact retry
after replica restart and full Lance publication recovery.
- Built and executed this branch's application binary against real TLS
WAL nodes and isolated etcd: repeated bootstrap, SIGKILL/restarted
ingress, original-receipt retry, two workers, full Arrow readback and
joined shutdown passed.
- Both etcd worker ownership/expiry tests passed.
- CI runs the executable recovery test and retains process logs on
failure. These tests establish the exercised recovery behavior, not
sustained throughput.
…t#465)

## Summary

Closing a WAL engine could leave its directory locked when a duplicated
file descriptor remained open, including a descriptor inherited during
concurrent process creation before exec. Explicitly unlock the directory
when the engine's final pipe owner closes, instead of depending solely
on descriptor closure. Record the acquiring process ID so a forked
child's destructor cannot release its parent's live lock.

The vendored raft-engine patch preserves exclusion while the engine is
live and changes no journal, Raft log, snapshot or wire format. It also
retains the normal shutdown ordering: pipe owners finish before the
directory lock is released.

## Testing

- A deterministic Linux regression duplicates the actual journal lock
descriptor. It fails before this change and passes after it, without
sleeps or retries. It also checks that closing the old duplicate cannot
unlock a replacement engine.
- Client-journal Clippy and tests: 11 passed, including
SIGKILL/rewrite/sync-failure recovery; two subprocess helpers are
invoked by their parent tests.
- All-feature WAL library tests: 158 passed, 3 ignored; all-feature
all-target Clippy passed.
- The interrupted-file admission regression measures actual allocated
file bytes after sync rather than assuming physical allocation equals
logical length. It retains the unfinished-reservation, reopen and
collection assertions.
- The existing pre-exec inheritance test is updated to verify that a
replacement engine can acquire the lock before the child executes, while
both the original live owner and replacement owner still exclude
concurrent opens. The final explicit all-feature/all-target WAL run
passed, including this process regression and the interrupted-file
accounting regression. The vendored engine was explicitly rebuilt and
its artifact hash verified to exclude stale shared-target dependencies.
- Unchanged non-WAL workspace tests, formatting and Clippy passed.
Python pytest: 229 passed, 2 skipped, 1 xfailed; Ruff and Pyright
passed, using source- and hash-verified unchanged native/server
artifacts.

Local checks are complete. All CI checks on the final commit must pass
before merging.

Refs lance-format#378.
…-format#467)

A legacy worker can accept a merge POST and never return, occupying a
master scheduler slot indefinitely. The master now retains the same
task, renewing claim, durable execution, admission guard and HTTP future
while allowing that response wait to yield its execution slot after a
configurable observation interval. This does not cancel or retry the
merge, declare the worker stalled, or fail dependent tasks.

Additional waits are bounded per master (default 32, zero disables
yielding). A late response first joins the fair execution queue, then
releases its wait capacity before blocking for a slot; this preserves
the concurrency bound without deadlocking full wait/execution pools. The
original serial fan-out continues only after reacquiring a slot. Local
merges, compaction, indexing and fenced execution retain their current
budgets. `/executor` phases and dedicated wait/yield metrics expose the
distinction.

Validation: 34 targeted tests pass on the final commit, including 26
legacy/ownership/routing regressions with real local etcd, three
response-capacity tests, three admission/drain tests, the existing
executor-drain integration, and merge/compaction pool isolation. The
one-slot scheduler regression proves another table completes while the
original task remains Running; its dependent task stays Queued, then
both original/dependent tasks complete after the delayed response. Full
waiting/execution capacity also recovers without deadlocking. CI is
still running.

There is no persisted-schema change or worker-upgrade prerequisite. This
does not transfer requests already held by old master processes, retire
worker0/worker9, reset ownership, or solve permanent exhaustion of the
bounded wait capacity. Those old executions still require verified
handoff. No Kubernetes, ConfigMap, RBAC, worker-memory/slot, or separate
merge-pool changes.

---------

Co-authored-by: Beinan Wang <beinanwang@microsoft.com>
…-format#463)

## Summary

Expose fixed cumulative counters for ten materializer phases, so slow
publication can be separated into ownership acquisition, WAL reads,
decoding, Lance file writes, commit, retirement and memory admission.
Worker counters survive successive table claims; failed and cancelled
work remain observable without per-table metric labels.

Parent and concurrent phase durations overlap. The documentation
explains each counter and why these values are neither CPU time nor
latency percentiles. This adds measurement and does not claim a
throughput improvement.

Refs lance-format#378. Built on merged lance-format#462 and the engine directory-lock fix lance-format#465.
The feature patch is unchanged after moving to main; all final-commit CI
checks must pass before merging.

## Testing

- On the final combined branch: formatting, service/http-client Clippy,
all 8 ordinary ingestion feature tests, both isolated-etcd worker
ownership tests and the actual application executable recovery test
passed.
- The executable test covers repeated bootstrap, killed/restarted
ingress, original-receipt retry, two workers, full Arrow readback and
joined shutdown using real TLS WAL nodes and etcd.
- Before integrating lance-format#462, the phase-only branch also passed workspace
Clippy and 814 Rust tests, 220 Python tests (11 skipped, 1 expected
failure), Ruff and Pyright. Python checks resumed with the built native
extension on PYTHONPATH after missing test dependencies were installed.
Final combined workspace checks run in CI.
## Summary

Materializers currently fetch and decode an entire pinned publication
before starting Lance file writers. Add an experimental, default-off
`worker.publication.pipeline_preparation` option that feeds bounded
Arrow streams into Lance 13 writers as records become available, without
payload reserialization.

Both paths preserve the same immutable plan, whole-record partitioning
and complete descriptor aggregation. Writers share fetch/decode permits;
blocking decode retains its permit and source reservation through
cancellation, and all admitted writers finish before a failure is
returned. No partial descriptor set can be prepared or published.

This distributes preparation within one worker only. Lance encoder
workspace remains outside the source reservation, streamed `blob_write`
timing includes source waiting, and no throughput improvement or
end-to-end zero-copy is claimed. A matched short cloud comparison
completed full payload and aggregate checks before and after restart in
both modes. In the common 60–150 second window, buffered publication was
60.17–60.63 MiB/s and streamed publication 57.23–57.87 MiB/s. Including
drain, the respective rates were 52.62–54.15 and 57.33–59.05 MiB/s.
These are one-run sampling/clock bounds, not statistical confidence or
sustained capacity; initial leaders differed and the streamed run
experienced term changes. The results do not establish a steady-window
throughput improvement or support enabling the option by default.

## Testing

- Workspace formatting, all-target Clippy and Rust tests.
- Python pytest: 229 passed, 2 skipped, 1 xfailed; Ruff formatting/lint
and Pyright. The server executable was rebuilt from this source; the
unchanged Python native extension was reused after checking its source
inputs.
- Service/http-client all-target Clippy and 10 feature tests, including
real WAL/Lance recovery in both modes and injected stream failure with
observer cancellation.
- Both modes passed release-process qualification: 192 MiB ACKed but
unpublished, all nine WAL processes killed, original directories
reopened, then one Lance publication with complete payload,
aggregate-row, read-only and negative-verifier checks; no producer
replay.
- A separate streaming-mode process test replaced one replica directory
with an empty directory after the crash. Its group payloads were fetched
exclusively from the rebuilt replica and all data checks passed. This is
local directory-loss simulation, not a Kubernetes volume-loss
qualification.

Refs lance-format#378. Built on merged lance-format#463 and lance-format#465. The streamed materializer
implementation matches the qualified candidate; the base additionally
contains the engine directory-lock fix lance-format#465. A test helper is
additionally gated to the features that use it; materializer-only and
service/http-client all-target Clippy pass with that correction.



Validation follow-up: the HTTP test publisher helper is compiled only
with `http-client`, matching its callers. Local all-target Clippy passes
with `worker`, `materializer`, and `http-client,materializer`
separately; production code is unchanged by this test fix. Final-head CI
must pass before merge.
)

## Summary

A single managed table currently prepares all of its files inside one
publishing worker, so adding workers does not parallelize that table's
data preparation. Add an opt-in Rust `preparer` role that divides one
immutable, pinned publication plan across bounded remote services. Each
service reads its assigned records directly from WAL and passes Arrow
handles into the existing Lance writers. The publisher exchanges only
plan references and fragment metadata, joins every partition and records
the complete aggregate through the existing durable Prepare and fenced
publication protocol.

The services resolve the deployment-authorized table registry, verify
the exact plan and source pins through quorum reads, and use
authenticated bounded requests, job slots, memory reservations and a
volatile result cache. Admitted jobs survive disconnected observers. A
missing or failed partition cannot publish a partial result; after
durable Prepare, recovery needs no preparer cache. Losing a cache
earlier may redo unpublished immutable files under the existing
orphan-cleanup contract.

The option is disabled when `preparation` is absent. This changes no WAL
wire format or Lance OSS code. It does not provide orphan deletion, a
hard bound on Lance encoder RSS, or a demonstrated throughput
improvement.

## Testing

- Integration coverage uses two authenticated HTTP services over real
TLS WAL and Lance files: missing-service failure, healthy-partition
cache reuse, disjoint source reads, complete aggregate recovery after
both caches disappear, and full payload verification.
- Independent release-process qualification passed: two producers
acknowledge 192 MiB before publication, all nine WAL processes are
killed and reopened, one preparer is missing, and its completed sibling
is killed to lose its cache. Restarting both preparers publishes all 64
original records exactly once with complete payload, aggregate-count,
read-only and negative-verifier checks and no producer replay. All
source reads, decodes and file writes occur in preparers; publisher
counters remain zero. This is local process recovery, not Kubernetes
volume-loss qualification.
- Materializer-only and service/http-client all-target Clippy passed.
Feature tests: 11 passed, 3 external-process fixtures ignored by the
ordinary run. Workspace formatting, all-target Clippy and Rust tests
passed, as did Python pytest (229 passed, 2 skipped, 1 xfailed), Ruff
and Pyright. Unchanged native/server artifacts were reused after
verifying source inputs and hashes. Final-head CI and prerequisite
merges are still required.
- Kubernetes 1/2/4-preparer scaling is queued behind the current
qualified-control experiment. No distributed throughput result or
sustained-capacity claim yet.

Refs lance-format#378. Built on merged lance-format#464 and lance-format#465. The distributed preparation
implementation matches the independently qualified release; the base
additionally contains the engine directory-lock fix lance-format#465 and
prerequisite test-helper feature gate. Final-head CI validates that
combined source.

Feature-only validation also covers the test fixtures with `worker` and
`service` enabled independently, including the actual preparer recovery
test.
…#468)

When multiple preparers read a WAL group, every payload currently starts
at the first configured voter. Add the opt-in `source_read_policy:
spread` setting to distribute initial reads by immutable group/index
while retaining every other voter as a fallback. The default remains
`first_voter`.

This changes read placement only: exact source-pin matching,
payload/schema validation, fetch concurrency, reservations and manifest
authority stay in place. The selection hash touches two integers and
does not hash or copy payload buffers. It applies to both buffered and
streamed preparation, including remote preparers.

Validation covers real Lance recovery, exact retries and retirement,
plus a network failure test that leaves only one payload endpoint
reachable at a time, exercising each replica as the sole source. An
independent Rust process fixture also recovered 192 MiB of
acknowledged/unpublished records after all nine WAL processes were
killed and reopened, followed by a preparer crash/cache loss; all 64
records and columns were read back without producer replay.

A controlled cloud comparison is queued; no throughput benefit has been
established, so this option remains disabled by default. The qualified
candidate includes the directory-lock fix from lance-format#465; final CI will
validate the combined source after dependency promotion.

Built on merged lance-format#466 and lance-format#465. Refs lance-format#378.

## Testing

- Workspace formatting, all-target Clippy and Rust tests: 814 passed
(200 optional tests ignored).
- Service/http-client feature suite: 15 passed, 3 external-process tests
ignored; separate single-reachable-replica fallback test passed.
- Independent release-process crash/cache-loss recovery fixture passed.
- Python: 229 passed, 2 skipped, 1 expected failure; Ruff
formatting/lint and Pyright passed. Python native/server artifacts were
reused after verifying unchanged source inputs and artifact hashes.

An additional 10-minute local endurance run published 9,447 MiB / 3,149
rows through two preparers. All columns and the aggregate remained
correct after killing and reopening all nine WAL processes, with no
producer replay or restarted publisher. Each node completed 9–10
checkpoints and collected 16–18 obsolete files. The run also had 619
producer retries and repeated leader-term changes under shared-disk
pressure; it establishes recovery after this finite workload, not a
throughput gain or stable capacity.
…format#469)

## Summary

The runtime currently waits for the leader's complete synchronous
storage poll before sending replication packets, delaying the start of
follower persistence. Add the default-off `runtime.pipeline_replication`
option to dispatch ordinary, non-snapshot Raft `Ready.messages` through
the existing bounded transport before local persistence.

Follower persistence-required responses, all existing sync barriers,
client completions, quorum reads, snapshots, and public synchronous
`Node::poll()` semantics retain their ordering. Packets move shared
payload buffers and retain node-wide byte/request credits; a full queue
drops for Raft retransmission without blocking the owner. An early
packet followed by failed leader persistence remains an uncertain
attempt and never a successful ACK from that poll.

This is an experimental option, with no established throughput
improvement. It does not make storage asynchronous or allow the Raft
owner to tick during a blocking filesystem operation.

Part of lance-format#378.

## Testing

- Full repository CI helper: workspace formatting, Clippy and tests;
Python tests, Ruff and Pyright. All passed on this source.
- All-feature/all-target WAL Clippy and tests: 161 library tests passed,
3 ignored; binary and integration suites passed.
- Block the actual leader sync gate: a follower durably accepts an
append while the leader remains blocked, without a client ACK. Assert
payload pointer identity, bounded outgoing credits, injected
leader/follower sync failures, shared-engine fencing, complete recovery
and exact-index retry after reopening all original directories.
- Snapshot catch-up and restart with both option values.
- Independent release executables: 192 MiB acknowledged but unpublished;
SIGKILL all nine WAL processes; reopen original directories; a missing
preparer cannot partially publish; kill the completed preparer to
discard its cache; restore both and verify all 64 original 3 MiB rows
and columns, aggregate count, version, read-only and negative checks,
without producer replay. This is retained-disk process recovery, not
volume-loss or throughput evidence.

A cloud comparison using identical binaries and changing only the node
option is queued; no result is claimed here. Final-head CI must pass
before merge.
When an uncompressed engine block misses the read cache, the decoder
currently copies the newly read block into another allocation before
parsing entries. Reuse the original allocation after verifying the
complete block checksum, removing one block-sized allocation and copy.
Compressed decoding, cache isolation, entry parsing, durable ACKs, and
file formats retain their existing behavior.

Refs lance-format#378. Built on merged lance-format#469.

Validation: actual-file cache and rewrite regressions;
compressed/uncompressed multi-entry reopen; corruption rejection and
recovery after restoring the original byte; log/checkpoint/snapshot
tests; all-feature WAL checks; full local `ci-pr-helper` checks. The
candidate release also passes nine-process WAL crash/reopen and complete
Arrow/Lance recovery with a missing preparer and lost preparation cache.
Local allocation diagnostics confirm the removed allocation; no
end-to-end throughput improvement is claimed.
Committed Fetch replies currently wait for the next Raft poll after
their payload has already been validated. Complete those replies
immediately, so an unrelated append sync cannot delay or discard them.
Append ACKs and quorum queries still wait for durable Raft completion.

Also give each server RPC its full processing timeout after the first
request bytes arrive. Previously, time spent waiting on an idle pooled
connection consumed that same deadline and could disconnect a newly
admitted request during its durability wait. The remaining header, body,
admission, execution, and reply remain bounded by one active deadline.
Client deadlines and frame/credit limits are unchanged.

Refs lance-format#378. Built on merged lance-format#470.

Validation: real-file fault tests cover committed reads during an
unrelated failed sync, without prematurely acknowledging new data. A
real TLS test with blocked journal sync reproduces expiration under the
old idle/request timer and passes with separate deadlines; partial
headers and bodies still expire and release the connection. RPC
regressions, the six-process node restart/snapshot test, all-feature WAL
checks, and full local ci-pr-helper checks pass. The original
committed-fetch change also passed a nine-process WAL/Arrow/Lance
recovery fixture. No throughput benefit is claimed for the deadline fix.
…t#472)

A bounded source scan can encounter only records from other tables. The
worker previously released its claim and discarded the scan cursor, then
started from the same durable checkpoint on its next claim. A table
behind more than one scan budget of retained foreign records could
remain unpublished indefinitely.

Retain the empty-page scan hints per fixed worker assignment and move
them through claim jobs. Hints are checked against the complete table
binding, contain only quorum-observed ranges without that table's
records, and never advance durable checkpoints. Scan limits, claim
rotation and publisher fencing remain unchanged. Restart may rescan
those ranges safely.

Refs lance-format#378. Built on merged lance-format#466, lance-format#468, lance-format#469, lance-format#470 and lance-format#471.

Validation:
- Real three-replica WAL, etcd and Lance regression: a target behind 129
retained foreign records fails with the old worker and publishes its
exact payload with the fix, while another table remains unpublished.
- Materializer, worker, service and service/http-client feature Clippy
checks; ingestion and etcd-backed worker recovery suites.
- Full repository CI helper, including Rust/Python tests, formatting and
lint/type checks.

The first worker-suite run encountered an existing lease-revoke timeout
with slow local etcd synchronization. The unchanged suite passed on a
fresh run with the same storage and deadlines; the original failure was
retained. This change fixes publication progress and does not claim a
throughput improvement.
Queueing behind replication work and reading or validating an accepted
record currently contribute to the same fetch latency. Expose separate
cumulative queue and service timings in each group's local `/status`
observation, alongside found, missing, failed, and returned-byte counts.

Counters use fixed storage and saturating arithmetic. They cover
serviced requests, including callers that disconnect; queued work
discarded on shutdown is excluded. Queue timing starts after runtime
admission, and service timing excludes reply transmission. These local
wall-time observations do not establish quorum freshness or change
durable acknowledgements.

Refs lance-format#378. Built on merged lance-format#471 and lance-format#472.

## Testing

- Real blocked and failed storage-sync regression with both replication
modes: existing and missing-record fetches complete and are counted
before the unrelated sync, while the new append remains unacknowledged.
- Actual WAL service and Lance recovery fixture with full-column
readback and retained process identities.
- WAL all-feature/all-target Clippy and tests passed.
- Complete local CI-equivalent script passed: workspace formatting,
Clippy and all-target tests; Python tests (229 passed, 2 skipped, 1
expected failure), Ruff formatting/lint and Pyright.
lance-format#475)

Rust CI jobs can exhaust their entire job deadline in `apt update` when
a hosted Ubuntu image selects an unresponsive archive mirror, leaving
the tests unexecuted. Use the official HTTPS archive for the known
mirror-list entry, bound APT timeouts and retries, and limit each
dependency-install step to ten minutes. Incomplete package-index updates
fail explicitly; package selection and test commands stay unchanged.

Refs lance-format#378.

Validation: Actionlint 1.7.12, ShellCheck and Bash syntax checks for the
installation commands, plus the repository CI helper (workspace
formatting, Clippy and tests; Python tests, Ruff and Pyright).
Hosted-runner execution is validated by this PR's CI.
…ance-format#479)

A legacy worker can stop completing merges while remaining Ready and
serving ingestion. Its protocol-0 execution outlives the HTTP request,
so the master cannot safely retry merely because a lease expires or a
process disappears. Bounded response-wait slots only postpone scheduler
saturation.

This adds an explicit master retirement API and one-shot native command.
Before termination, capture the exact live Pod/container/worker identity
against the existing execution in etcd. After a same-Pod container
replacement, require the old container termination receipt and a fenced
replacement reporting the same execution and storage namespace. Persist
a base/shard manifest barrier plan, cross its immutable version bounds,
then CAS the exact old execution to terminal. Canonical task retries
complete prepared handoffs automatically after verified process
retirement; normal scheduler claims still own lock release and retries.
Missing shards and ambiguous identities remain protected.

The master also negotiates fencing per worker in a mixed fleet. A fenced
worker receives protocol 2 directly, avoiding the protocol-0 reservation
that otherwise blocks its own background admission. Failed capability
probes never fall back to legacy POSTs; unresolved admission mode does
not admit work. Fan-out remains serial. `MERGE_FENCE_THIS_WORKER`
persists activation against one stable writer ID, preventing a
single-worker retirement from switching the startup mode of unrelated
workers or masters; existing cluster-wide activation remains
authoritative.

Validation at `c3a2c7d`: workspace unit/integration tests and real-etcd
HA CI pass. Local isolated-etcd checks pass for 13 retirement-focused
cases, 23 routing/recovery regressions, scheduler dependency/capacity
isolation, and persisted admission modes. The worker is a binary target:
its replacement-status tests and the real-store merge-budget lock
regression are verified in the successful CI binary test runs, not
inferred from a `--lib` invocation. Formatting and diff checks pass;
remaining CI is running. Exact-image AKS/Azure same-Pod retirement
qualification is prepared and awaits the candidate image from services
#9889. No production worker has been replaced and no production
ownership record has been reset by this change.

See `docs/legacy-merge-retirement.md` for the transition contract and
explicit limits. This does not infer process death from age or add an
automatic process killer, pool, ConfigMap, or RBAC.

---------

Co-authored-by: Beinan Wang <beinanwang@microsoft.com>
…format#480)

## Summary

Publication policies currently reject source-byte limits above 256 MiB,
preventing larger publication batches even when a deployment provides
sufficient memory and admission capacity. Allow an explicitly configured
limit up to 1 GiB while keeping the 64 MiB default unchanged.

This remains part of the immutable replicated policy. A higher limit
requires consistent replica configuration and matching
client/materializer budgets; this change does not migrate existing
groups or raise their allocations.

Refs lance-format#378.

## Testing

- Repository CI helper: workspace formatting, Clippy and tests; Python
tests, Ruff and Pyright.
- WAL service-feature and ingestion-feature tests.
- Real-process recovery of acknowledged, unpublished records after
replica crashes and preparer cache loss, including complete payload
readback and publication identity checks.
lance-format#481)

## Summary

An acknowledged rollout append can remain in WAL while workers
accumulate larger Lance writes. Add `Registry::read_session` and `GET
/api/v1/ingest/{table}/sessions/{session}` so callers can read those
rows before publication.

The reader streams committed WAL records, then scans the latest managed
Lance snapshot, removing overlap by immutable row ID. Source closure
checks handle retirement during a read; missing unpublished payload
fails the stream. Binary filtering uses Arrow slices, with payload
workspace reserved per record. The HTTP response is an Arrow IPC stream
under the existing workload authorization.

This is a session read with unspecified order. It scans the session's
bucket; captured metadata and deduplication IDs grow with the retained
tail. Consumers must read through EOF and propagate errors.
Published-version dataset reads retain their existing semantics.

Refs lance-format#378.

## Testing

- Real three-replica RPC tests: ACK visibility without a publisher,
replica restart, session isolation, publication/retirement races, and
unavailable unpublished payloads.
- Actual authenticated HTTP append/read against WAL and Lance, including
mixed published and unpublished rows.
- A tail exceeding its per-record payload budget, cancellation release,
and binary buffer sharing.
- Ingestion all-feature Clippy and library tests: 22 passed; 4 existing
etcd/executable tests require the dedicated CI environment.
- Workspace formatting, Clippy, and all-target Rust tests passed.
- Python suite: 220 passed, 11 skipped, 1 expected failure; Ruff and
Pyright passed after installing the declared test dependencies.
- Full GitHub CI is required before merge.
…ce-format#482)

## Summary

A publication byte cap alone still allows every small polling result to
become a Lance write. Add an optional `min_source_bytes` threshold and
`max_batch_wait`: publish a full selection immediately, or flush a
smaller tail after its wait on the next poll. The default minimum is
zero, preserving existing behavior.

Workers retain a valid scheduling claim while accumulating the batch and
still respond to shutdown. Deadline state survives local claim rotation;
durable pending publications resume immediately. No source checkpoint
moves while a batch is waiting.

This changes source-batch sizing, not exact encoded file sizes or the
group's immutable WAL byte limit. Session WAL reads can provide
visibility before the delayed Lance publication.

Refs lance-format#378.

## Testing

- All-feature ingestion Clippy.
- Seven real WAL/Lance materializer regressions, including threshold
flush, timed tail, record-cap flush, and deadline preservation across
claim rotation.
- Real etcd worker suite: all four passed, including competing workers
flushing a small tail. An initial run hit an existing lease-expiry
timeout; the unchanged isolated test and complete suite both passed on
rerun.
- Workspace formatting, Clippy, and all-target Rust tests passed.
- Python: 229 passed, 2 skipped, 1 expected failure; Ruff and Pyright
passed. The first Python invocation failed because the fixture could not
find the shared-target server binary; rebuilding the server and linking
the expected target path resolved all nine setup errors.
- GitHub CI must pass before merge.
@KaiqiJinWow
KaiqiJinWow force-pushed the feat/rollout-client-flush branch from 9f32b9e to f4a9edc Compare October 22, 2026 23:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants