Repository navigation
osm lance: fix the total tile outage — verify the P-CACHE-4 anchors before serving - #124
Conversation
…efore serving Root cause of every /api/osm/geometry/tile-bin/* request returning 400 in production: WriteParams defaults max_rows_per_file to 1M, so Berlin's 2,525,052 rows were written as THREE Lance fragments. locate_row_column found row 0 at offset 0 of one fragment and reported the full 1.29 GB slab length against a file that only held ~40% of it; open_slab's .min() clamp then silently truncated the slice to the fragment's file size — footer bytes included — which is not a multiple of NODE_ROW_STRIDE, so RowSlab::new rejected every request. lance-graph's own soa_verbatim.rs warns about exactly this: "a split changes what an mmap offset means." The serving contract stays as directed: hydrate once from S3, sink into Lance ONCE, serve zero-copy lazily from the .lance bytes on disk (the mmap+offset mechanism soa_verbatim.rs proves). What changes is that every precondition that test asserts is now enforced at boot instead of assumed: - write_lance: max_rows_per_file = rows (+ raised max_bytes_per_file) — the dataset is always a single fragment, whatever the region size. - reopen_if_warm: also requires fragment count == 1, so the broken three-fragment Berlin dataset on volume01 is reconverted once and then stays warm. - locate_row_column: productionizes the verbatim test's assertions — sole data file, stride-aligned run start, head anchor, PLUS a tail anchor (the slab's last row must sit at offset + length - 512, in bounds). A coincidental head match cannot also anchor the tail at the computed distance; a partial fragment fails the bounds check outright. - open_slab: exact bounds instead of the .min() clamp — any doubt falls back to the raw .soa file rather than serving a truncated slice. - ensure_lance_local: digest via mmap instead of fs::read — the warm boot path no longer pays a 1.29 GiB heap allocation just to hash the slab. New falsifiers, each verified against the failure it kills: multi-fragment split (the production outage), truncated sole fragment, tail-anchor mismatch, unaligned run start. Tests: osm_lance 10/10, osm_features 18/18 passing (run for real this time — the earlier linker-SIGBUS environment issue is resolved). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NMeiLmtDKhomJNSo2ecbJw
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_09eb6b23-90ed-477f-8684-edd6a94f6815) |
…e in place Closes the gap the fragment-split fix (#124) left open: WriteMode::Overwrite only writes a new manifest VERSION — Lance keeps the previous version's fragment files on disk. Reconverting the pre-fix three-fragment Berlin dataset in place would therefore leave data/ holding FOUR .lance files, and locate_row_column's sole-data-file precondition would fall back to the raw .soa forever — the map would render, but Lance serving would silently never engage, with 1.3 GB of dead fragments squatting on volume01. The dataset is a derived cache of the slab; version history has no value and the slab itself is the recovery path. So a stale dataset is now removed whole (remove_stale_dataset) before write_lance runs, fail-closed: if the removal fails, nothing is written into the half-cleared directory (exactly how a 4-file data/ would happen) and the .soa keeps serving unchanged. Bonus: peak disk during reconversion drops from .soa + old + new (~3.9 GB) to .soa + new (~2.6 GB). Falsifier: remove_stale_dataset_clears_old_fragments_entirely lays out the pre-fix three-fragment shape plus Lance bookkeeping dirs and asserts the directory is GONE. Tests: osm_lance 11/11 passing; cargo check clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NMeiLmtDKhomJNSo2ecbJw
Summary
Hotfix for the live outage introduced by #123: every
/api/osm/geometry/tile-bin/*request returned 400.Root cause —
WriteParamsdefaultsmax_rows_per_fileto 1M, so Berlin's 2,525,052 rows were written as three Lance fragments.locate_row_columnfound row 0 at offset 0 of one fragment and reported the full 1.29 GB slab length against a file that only held ~40% of it;open_slab's.min()clamp then silently truncated the slice to the fragment's file size (footer bytes included), which is not a multiple ofNODE_ROW_STRIDE— soRowSlab::newrejected every request. lance-graph's ownsoa_verbatim.rswarns about exactly this: "a split changes what an mmap offset means."The serving contract stays as directed: hydrate once from S3 → sink into Lance ONCE → serve zero-copy lazily from the
.lancebytes on disk (the mmap+offset mechanismsoa_verbatim.rsproves). What changes is that every precondition that test asserts is now enforced at boot instead of assumed:write_lance:max_rows_per_file = rows(+ raisedmax_bytes_per_file) — always a single fragment.reopen_if_warm: also requires fragment count == 1, so the broken three-fragment Berlin dataset on volume01 reconverts once, then stays warm.locate_row_column: sole data file, stride-aligned run start, head anchor, plus a tail anchor (the slab's last row must sit atoffset + length − 512, in bounds). A coincidental head match cannot also anchor the tail; a partial fragment fails the bounds check outright.open_slab: exact bounds instead of the.min()clamp — any doubt falls back to the raw.soafile rather than serving a truncated slice.ensure_lance_local: digest via mmap instead offs::read— warm boots no longer pay a 1.29 GiB heap allocation just to hash the slab (relevant to the Railway RAM observation).Test plan
cargo check -p cockpit-servercleanosm_lancetests: 10/10 — including new falsifiers, each verified against the failure it kills: multi-fragment split (the production outage), truncated sole fragment, tail-anchor mismatch, unaligned run startosm_featurestests: 18/18 (run for real this time — the earlier linker-SIGBUS environment issue is resolved)reconverting(fragment count mismatch) once, thenrow column verified verbatim (sole fragment, aligned, head+tail anchors), and tiles should render againGenerated by Claude Code