Skip to content

osm lance: fix the total tile outage — verify the P-CACHE-4 anchors before serving - #124

Merged
AdaWorldAPI merged 1 commit into
mainfrom
claude/q2-osm-map-reencoding-56p5e2
Aug 12, 2026
Merged

AdaWorldAPI merged 1 commit into
mainfrom
claude/q2-osm-map-reencoding-56p5e2

Conversation

@AdaWorldAPI

Copy link
Copy Markdown
Owner

Summary

Hotfix for the live outage introduced by #123: every /api/osm/geometry/tile-bin/* request returned 400.

Root cause — WriteParams defaults max_rows_per_file to 1M, so Berlin's 2,525,052 rows were written as three Lance fragments. locate_row_column found row 0 at offset 0 of one fragment and reported the full 1.29 GB slab length against a file that only held ~40% of it; open_slab's .min() clamp then silently truncated the slice to the fragment's file size (footer bytes included), which is not a multiple of NODE_ROW_STRIDE — so RowSlab::new rejected every request. lance-graph's own soa_verbatim.rs warns about exactly this: "a split changes what an mmap offset means."

The serving contract stays as directed: hydrate once from S3 → sink into Lance ONCE → serve zero-copy lazily from the .lance bytes on disk (the mmap+offset mechanism soa_verbatim.rs proves). What changes is that every precondition that test asserts is now enforced at boot instead of assumed:

  • write_lance: max_rows_per_file = rows (+ raised max_bytes_per_file) — always a single fragment.
  • reopen_if_warm: also requires fragment count == 1, so the broken three-fragment Berlin dataset on volume01 reconverts once, then stays warm.
  • locate_row_column: sole data file, stride-aligned run start, head anchor, plus a tail anchor (the slab's last row must sit at offset + length − 512, in bounds). A coincidental head match cannot also anchor the tail; a partial fragment fails the bounds check outright.
  • open_slab: exact bounds instead of the .min() clamp — any doubt falls back to the raw .soa file rather than serving a truncated slice.
  • ensure_lance_local: digest via mmap instead of fs::read — warm boots no longer pay a 1.29 GiB heap allocation just to hash the slab (relevant to the Railway RAM observation).

Test plan

  • cargo check -p cockpit-server clean
  • osm_lance tests: 10/10 — including new falsifiers, each verified against the failure it kills: multi-fragment split (the production outage), truncated sole fragment, tail-anchor mismatch, unaligned run start
  • osm_features tests: 18/18 (run for real this time — the earlier linker-SIGBUS environment issue is resolved)
  • Deploy verification: boot log should show reconverting (fragment count mismatch) once, then row column verified verbatim (sole fragment, aligned, head+tail anchors), and tiles should render again

Generated by Claude Code

…efore serving

Root cause of every /api/osm/geometry/tile-bin/* request returning 400 in
production: WriteParams defaults max_rows_per_file to 1M, so Berlin's
2,525,052 rows were written as THREE Lance fragments. locate_row_column
found row 0 at offset 0 of one fragment and reported the full 1.29 GB slab
length against a file that only held ~40% of it; open_slab's .min() clamp
then silently truncated the slice to the fragment's file size — footer
bytes included — which is not a multiple of NODE_ROW_STRIDE, so
RowSlab::new rejected every request. lance-graph's own soa_verbatim.rs
warns about exactly this: "a split changes what an mmap offset means."

The serving contract stays as directed: hydrate once from S3, sink into
Lance ONCE, serve zero-copy lazily from the .lance bytes on disk (the
mmap+offset mechanism soa_verbatim.rs proves). What changes is that every
precondition that test asserts is now enforced at boot instead of assumed:

- write_lance: max_rows_per_file = rows (+ raised max_bytes_per_file) — the
  dataset is always a single fragment, whatever the region size.
- reopen_if_warm: also requires fragment count == 1, so the broken
  three-fragment Berlin dataset on volume01 is reconverted once and then
  stays warm.
- locate_row_column: productionizes the verbatim test's assertions — sole
  data file, stride-aligned run start, head anchor, PLUS a tail anchor (the
  slab's last row must sit at offset + length - 512, in bounds). A
  coincidental head match cannot also anchor the tail at the computed
  distance; a partial fragment fails the bounds check outright.
- open_slab: exact bounds instead of the .min() clamp — any doubt falls
  back to the raw .soa file rather than serving a truncated slice.
- ensure_lance_local: digest via mmap instead of fs::read — the warm boot
  path no longer pays a 1.29 GiB heap allocation just to hash the slab.

New falsifiers, each verified against the failure it kills:
multi-fragment split (the production outage), truncated sole fragment,
tail-anchor mismatch, unaligned run start. Tests: osm_lance 10/10,
osm_features 18/18 passing (run for real this time — the earlier
linker-SIGBUS environment issue is resolved).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NMeiLmtDKhomJNSo2ecbJw
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d09d3407-5fe6-40ef-a2c0-006ba26d922a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cursor

cursor Bot commented Aug 12, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_09eb6b23-90ed-477f-8684-edd6a94f6815)

@AdaWorldAPI
AdaWorldAPI marked this pull request as ready for review August 12, 2026 14:04
@AdaWorldAPI
AdaWorldAPI merged commit c8e675a into main Aug 12, 2026
5 checks passed
AdaWorldAPI pushed a commit that referenced this pull request Aug 12, 2026
…e in place

Closes the gap the fragment-split fix (#124) left open: WriteMode::Overwrite
only writes a new manifest VERSION — Lance keeps the previous version's
fragment files on disk. Reconverting the pre-fix three-fragment Berlin
dataset in place would therefore leave data/ holding FOUR .lance files, and
locate_row_column's sole-data-file precondition would fall back to the raw
.soa forever — the map would render, but Lance serving would silently never
engage, with 1.3 GB of dead fragments squatting on volume01.

The dataset is a derived cache of the slab; version history has no value
and the slab itself is the recovery path. So a stale dataset is now removed
whole (remove_stale_dataset) before write_lance runs, fail-closed: if the
removal fails, nothing is written into the half-cleared directory (exactly
how a 4-file data/ would happen) and the .soa keeps serving unchanged.
Bonus: peak disk during reconversion drops from .soa + old + new (~3.9 GB)
to .soa + new (~2.6 GB).

Falsifier: remove_stale_dataset_clears_old_fragments_entirely lays out the
pre-fix three-fragment shape plus Lance bookkeeping dirs and asserts the
directory is GONE. Tests: osm_lance 11/11 passing; cargo check clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NMeiLmtDKhomJNSo2ecbJw
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants