Repository navigation
osm lance: remove a stale dataset before reconversion, never overwrite in place - #125
Conversation
…e in place Closes the gap the fragment-split fix (#124) left open: WriteMode::Overwrite only writes a new manifest VERSION — Lance keeps the previous version's fragment files on disk. Reconverting the pre-fix three-fragment Berlin dataset in place would therefore leave data/ holding FOUR .lance files, and locate_row_column's sole-data-file precondition would fall back to the raw .soa forever — the map would render, but Lance serving would silently never engage, with 1.3 GB of dead fragments squatting on volume01. The dataset is a derived cache of the slab; version history has no value and the slab itself is the recovery path. So a stale dataset is now removed whole (remove_stale_dataset) before write_lance runs, fail-closed: if the removal fails, nothing is written into the half-cleared directory (exactly how a 4-file data/ would happen) and the .soa keeps serving unchanged. Bonus: peak disk during reconversion drops from .soa + old + new (~3.9 GB) to .soa + new (~2.6 GB). Falsifier: remove_stale_dataset_clears_old_fragments_entirely lays out the pre-fix three-fragment shape plus Lance bookkeeping dirs and asserts the directory is GONE. Tests: osm_lance 11/11 passing; cargo check clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NMeiLmtDKhomJNSo2ecbJw
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughCold conversion now removes stale Lance datasets before reconversion. Removal failures stop conversion and preserve raw ChangesLance dataset cleanup
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: Poem
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_f148aaf4-7fde-4ef0-bf34-a8f78911611a) |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_a9ae37de-a098-4d55-8151-b589a16bf6c0) |
Summary
Closes the gap #124 left open.
WriteMode::Overwriteonly writes a new manifest version — Lance keeps the previous version's fragment files on disk. So on the first boot of the #124 build, reconverting the pre-fix three-fragment Berlin dataset in place would leavedata/holding four.lancefiles, andlocate_row_column's sole-data-file precondition would fall back to the raw.soaforever: the map renders, but Lance serving silently never engages, with 1.3 GB of dead fragments squatting on volume01.ensure_lance_localnow removes a stale dataset whole (remove_stale_dataset) beforewrite_lanceruns. The dataset is a derived cache of the slab — version history has no value; the slab is the recovery path.data/would happen) and the.soakeeps serving unchanged..soa+ old + new (~3.9 GB) to.soa+ new (~2.6 GB).With this, the first boot of the fixed build goes: stale detected (3 fragments) → old dataset removed → single-fragment rewrite →
row column verified verbatim … (sole fragment, aligned start, head + tail anchors)→ tiles served zero-copy lazily from the Lance data file, same boot.Test plan
remove_stale_dataset_clears_old_fragments_entirely— lays out the pre-fix three-fragment shape plus Lance bookkeeping dirs, asserts the directory is gonecargo test -p cockpit-server --bin q2-cockpit osm_lance— 11/11 passingcargo check -p cockpit-server— cleanGenerated by Claude Code
Summary by CodeRabbit