perf(0111): bound enrichment passes to monthly partitions - #242
Merged
Conversation
The partition-bounding mechanism already exists, is tested, and runs in production - ChEnrichConfig::time_window, threaded by window_pred(), set per month by the 0114 coarse-repair driver, with the candidate-side-only bound already designed so the pivot reference still forward-fills across month boundaries. The scheduled 1m pass sets it to None and main.rs:58 records that as intent. So the work is driving existing bounding from the scheduled pass, not building it. Design: a live pass bounded to two partitions, plus a frontier-driven historical sweep in the shape of CoarseRepairDriver. The frontier is a tiny ReplacingMergeTree state table - the fourth instance of a pattern already live in this schema - and is advisory, never authoritative: every month it selects is re-confirmed by a cheap partition-bounded count, so wrong state costs one query rather than silently skipping rows. That is the failure class 0215 just cost 26 days on, so it is designed out rather than documented around. The pre-2021 unpriceable floor then needs no hard-coded cutoff: those months make no progress and mark themselves exhausted on first visit. Flags one statement nobody has measured - watermark()'s SELECT max(timestamp) runs every pass, and every measurement in this task and in 0215 filtered query_log on INSERT INTO. Prepare only; nothing deployed.
…rtitions The scheduled enrichment pass re-scanned all 102 partitions of price_ohlcv_1m (736M rows / 18.4 GiB) on every statement, to serve a live window of 17M. Measured 2026-08-21 after the 0215 Caddy fix removed the 30s cut that had been masking it: an oracle batch costs 35.5s and a peg-pivot batch 82.3s, so an invocation budgeting 40 batches achieves 5.4 and is killed at the 300s Lambda timeout. That is what fired prices-production-enrichment-duration-near-timeout at 300,338 ms. ChEnrichConfig::time_window and window_pred() already bound every candidate scan and are covered by tests; the 0114 coarse-repair driver has used them per month since it landed. The scheduled Lambda simply never set one, and main.rs carried a comment stating that as intent. live_window::live_partition_window is a pure function of (now, partitions) so the pruning is testable without ClickHouse. The window is computed per invocation rather than at cold start: a warm container crossing a month boundary would otherwise keep scanning the old month and miss the new one entirely. ENRICH_LIVE_PARTITIONS defaults to 2 (current + previous month) and 0 restores the unbounded pass, so the bound can be widened or switched off by config alone. Also adds prices.enrichment_frontier to init.sql for the phase 2 historical sweep — a tiny ReplacingMergeTree state table matching ingest_cursor / backfill_progress / discovery_state. Advisory only: the sweep re-confirms each month with a bounded count before working it, so a stale row costs a cheap query and never skipped rows. Records one refutation in the task file and in the watermark() doc comment: watermark() is NOT a hidden third full scan. It reads 294 rows in 0.00s because PARTITION BY toYYYYMM(timestamp) gives every part a minmax_timestamp index. Do not add a window predicate there. Phase 2 (the frontier-driven historical sweep) is not in this commit. Phase 1 alone stops the historical drain, so the two should land together before deploy.
Phase 1 bounded the scheduled pass to the newest monthly partitions, which fixes the timeout but leaves the 646M unenriched rows below 202403 with nothing draining them. This is what drains them: one monthly partition per invocation, oldest-first, remembering the walk in prices.enrichment_frontier. The frontier is advisory, never authoritative. Every month is re-confirmed against the data before it is worked, and the confirmation is the pass's own candidates_before — a partition-bounded count_candidates — so the authoritative read and the work share one statement and cannot disagree. A wrong or stale frontier row costs one cheap query, never a skipped row. Skipped rows that read as healthy are the failure class that cost 26 days in 0215, and an authoritative cursor is how you get them. Termination needs no cutoff date. A month the pass enriches nothing in is marked exhausted and never revisited, which covers both empty calendar months and the ~5.34M rows predating the XLM/USDC reference market (first candle 2021-02) that are permanently unpriceable by this design. The month span comes from toYYYYMM(min|max(timestamp)) rather than system.parts: the system database is closed to prices_writer on prod and cannot be granted, so reading it would deploy green and fail every real run. Both aggregates are answered from part metadata via the minmax_timestamp index — measured at 294 rows / 0.00 s on 2026-08-21. Frontier position, pending-month count, rows enriched and whether the run stopped on its budget are published as metrics, so drain progress is a number rather than an archaeology dig. Rows-enriched is also the per-leg signal the pass cannot give: the XLM pivot writes exactly batch_size every batch and masks other legs' stalls in the aggregate (task 0219). Also factors the Lambda deadline arithmetic into one shared helper — two callers computing it slightly differently is how one ends up with no margin. Ships behind ENRICH_HISTORICAL_SWEEP and self-disables when the live pass is unbounded, since the two would then contend for the same partitions. prices.enrichment_frontier is created by hand (already applied to prod). Deploy remains out of scope for this branch.
The pure months_to_sweep tests cover the walk order and the exhausted-month skip, but not the half that can actually be wrong: the SQL. These exercise the Enum8-by-name insert, CAST(state AS String) on read, the server-side toUnixTimestamp64Milli version, and toYYYYMM(min|max(timestamp)) as the partition-span source. the_frontier_advances_exhausts_and_never_revisits walks three monthly partitions across four runs and asserts the sweep advances, marks an unpriceable month exhausted, never revisits it, and writes real USD values rather than just frontier rows. The 202101 fixture has no XLM/USDC reference before it, so it reproduces the pre-2021 permanently-unpriceable floor in miniature and shows it terminating with no cutoff date in the code. the_sweep_never_enters_the_live_window guards the partition split itself: if the sweep can reach a live-window partition the two contend every hour, which is the coupling this task exists to remove. Added to ch_enrich_it.rs to reuse its scratch-database harness rather than standing up a second one.
Both were written when the backlog was believed to be 8.4M rows. It is 656.69M, of which 5.34M predate the XLM/USDC reference market and are permanently unpriceable. As worded, neither could ever be satisfied — AC 1 demanded a full one_shot drain inside a 300s Lambda, which 556.78M XLM candidates will not do under any scan strategy, and AC 4 implied a zero residual the design cannot reach. A task whose ACs cannot be met is a task that keeps sliding, which is what happened here. AC 1 now asks that a bounded pass runs to completion, with an exact signal: count_remaining_at_volume_zero is issued once per pass at the very end, so its presence in query_log cannot be faked by a partial pass. It currently appears zero times in 2.5 hours. AC 4 now asks that the frontier demonstrably walks and declares the residual, recording the 5.34M floor here so a finished drain is not misread as an unfinished one. Adds a sixth AC covering live-window starvation, since the sweep is new work competing for the same invocation, and notes that EnrichmentRowsRemainingAtVolumeZero becomes window-scoped under a bounded pass — no alarm consumes it, but it will look like a dramatic fix that has not happened. Originals are preserved inline under each replacement. The baseline table now carries all three measurements, including the quiet-cluster one that must not be judged against.
…verdict The frontier integration test failed on first contact with a real engine, and it was the code that was wrong twice over. First: a month drained to zero candidates in one pass was marked `pending`, because the rule keyed only on rows_enriched > 0. It would then need a second visit purely to learn there was nothing left — doubling the walk across ~102 partitions. Terminal now on either candidates_after == 0 or rows_enriched == 0. Second, and worse: `exhausted` was permanent. A month is exhausted because nothing there could be priced at that moment, and two things falsify that later — a backfill writing into a historical partition (tasks 0088 and 0201 do exactly this), or a new reference making previously unpriceable candles priceable. Those rows would sit unenriched forever while the frontier read clean. That is the "skipped rows that look healthy" failure class this task keeps citing from 0215, reintroduced by the mechanism meant to avoid it. Exhausted marks now expire. Stale ones are re-confirmed against the data on a rotation, oldest-confirmation first so no month can go stale indefinitely, capped per invocation separately from max_months so drift correction can never starve the drain. A re-check is a pass with max_batches = 0 — which counts the month's bounded candidates and does no work, because max_batches keeps its literal meaning — so it reuses the pass's own tested counting path rather than a second candidate query that could drift from it. Staleness is judged against the ClickHouse clock, since swept_at is written server-side; comparing it to the Lambda wall clock would drift with host skew. a_backfill_into_an_exhausted_month_reopens_it asserts the hazard first (the backfilled row IS invisible while the mark is trusted) and then the fix, so the test cannot pass vacuously. 19 ignored tests green against ClickHouse 26.3.10.60, matching prod.
CI's `npx nx format:check --all` fails on docs/runbooks/0072-current-prices-mv-rollout.md, which this branch does not touch. The file is byte-identical on develop. It landed via a docs-only PR, where the TypeScript job — the one carrying `format:check --all` — is gated off by the Detect changes job, so nothing ever checked it. This is the first PR since that touches TypeScript, so it is the first to run the check and the first to see the violation. The change is `*italic*` to `_italic_` on five lines. Semantically identical markdown; no content is altered.
…ontier prices-clickhouse asserts init.sql parses into an exact number of statements, and adding prices.enrichment_frontier made it 31. My mistake: I verified with cargo test -p enrichment-worker throughout, so a schema change landing in a different crate's test was never exercised locally. The count is deliberately exact rather than a lower bound — it is what makes an accidental schema edit visible — so the fix is to update the tally and its inventory comment, not to loosen the assertion.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ENRICH_LIVE_PARTITIONS, default 2). It was re-scanning all 102 partitions — 736M rows / 18.4 GiB — on every statement to serve a live window of 17M.ChEnrichConfig::time_windowandwindow_pred()already did this and are tested; the scheduled Lambda simply never set one, andmain.rscarried a comment stating that as intent. The window is computed per invocation, not at cold start, so a warm container crossing a month boundary moves with the calendar.src/frontier.rs+prices.enrichment_frontier) to drain the 646M unenriched rows below202403, one monthly partition per invocation, oldest-first. The frontier is advisory: the re-confirmation is the pass's owncandidates_before, so the authoritative read and the work are the same statement and cannot disagree. Exhausted marks expire and are re-confirmed on a rotation, so a backfill writing into a finished partition is picked up rather than silently skipped.batch_sizeevery batch and masks other legs' stalls in the aggregate (task 0219).watermark()is not a hidden third full scan (294 rows / 0.00 s — the partition minmax index answers it).Context
prices-production-enrichment-duration-near-timeoutfired at 300,000 / 300,338 ms — at the timeout, not near it. Measured insystem.query_log: an oracle batch costs 35.5 s and a peg-pivot batch 82.3 s, so an invocation budgeting 40 batches achieves 5.4 and is killed. The accounting closes to the second.This is the flip side of #240 — with Caddy's 30 s cut removed, the 44.9 s XLM pivot survives, so the pass now proceeds through batches instead of aborting, and runs the Lambda clock out instead.
Verification
19 integration tests green against ClickHouse 26.3.10.60, matching prod; 64 unit tests; clippy clean.
prices.enrichment_frontieris already created on prod. Deploy is out of scope for this branch — the task file carries the runbook and the post-deploy measurement queries.