Repository navigation
feat: run owned WAL catch-up inside native masters - #390
Merged
Merged
Conversation
beinan
marked this pull request as ready for review
October 19, 2026 17:51
beinan
added a commit
that referenced
this pull request
Oct 19, 2026
Production recorded `/records` reads taking 217–241 seconds before returning 404. The recovery endpoint retained only 5xx outcomes, while route-level latency histograms could not identify the affected table. Retain the latest completed read taking at least five seconds in the existing bounded per-table hint cache, including its status and response-header duration. These additive schema-1 fields preserve the nonblocking recording path, five-minute expiry, 4,096-entry cap and 1 MiB response limit. Failures and slow reads precede ordinary activity when the response is truncated. Request parameters, record IDs and payloads are excluded. This supplies diagnostic hints; it does not cancel requests or infer that a slow read needs a merge. Validation: metrics unit tests passed (10, plus one ignored manual benchmark); enabled/disabled HTTP integration tests passed, including a delayed 404, in-flight demand and payload exclusion. Tests also cover threshold boundaries, independent expiry, preservation after successful requests, lock contention and bounded truncation. `cargo clippy -p lance-context-metrics --all-targets -- -D warnings`, formatting and diff checks passed. This is independent of #390 and does not block its release. Co-authored-by: Beinan Wang <beinanwang@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Owned generic/upsert tables currently need worker fan-out or a dedicated catch-up process even when resident master capacity is available. Add
MASTER_LOCAL_MERGE_TARGETSso native masters claim these tables from the existing durable queue and run bounded WAL catch-up themselves; additional master replicas can service different tables.The resident path reuses the existing ordered catch-up implementation, maintenance progress watchdog and storage fencing, while sharing one buffer budget per master. Both generic and ordinary rollout tables gain metadata-only demand discovery and durable continuation after bounded passes. Existing append-only preparation remains available.
*selects explicitly owned, non-draining targets only; this does not migrate legacy ownership or import external recovery ledgers, and requires no new Kubernetes pool or RBAC.Validation: local master unit suite passed (102 tests); real-etcd resident suite passed (10 tests), including generic upserts over a 16-pass boundary, successor claims on another master, concurrent ingestion, independent table execution under a saturated per-master budget, external-owner exclusion and reserved capacity during legacy RPC saturation. The existing dedicated-executor regression also passed with 1/2/3 MiB shared budgets (1 test, three configurations). The final drain fix allows resident-only masters with no worker endpoints to reconcile previously recorded executions while draining. On head
f4c28e1, the 102 master unit tests, 13 resident tests (including 10 real-etcd tests), and targeted no-worker drain integration test passed. The dedicated executor regression was also rerun successfully onf4c28e1; 16 legacy supervisor tests also passed. Formatting and diff checks passed. Full CI remains in progress.Deployment requires a mixed-version staging check and a qualified ownership handoff. This change does not retire existing external consumers by itself.