Repository navigation
osm/lance lifecycle Phase A: three clocks, one lineage per region - #133
Conversation
notebook-query pinned `datafusion = "53"` directly even though nothing in its source references `datafusion::` at all (lance-graph itself already depends on it at 54, matching the lance-9 family: lance 9, lancedb 0.33, datafusion 54, arrow 58). The stale pin forced cargo to build an entire second datafusion-53 + arrow-57 dependency subtree (~26 duplicate datafusion-* crates) alongside the one already required transitively, for a dependency that was never used. Removed rather than bumped: if this crate ever needs datafusion types directly, it should track whatever lance-graph pins at that time, not a hardcoded major version that can silently drift again. Verified: Cargo.lock now carries a single datafusion (54.1.0) and a single arrow (58.3.0) across the whole workspace.
…gion
Pure decision layer for the OSM/Lance lifecycle redesign: a live Lance
dataset is a versioned lineage, not an immutable directory, so nothing
here hashes a whole dataset directory or compares its changing object
inventory to a static checksum. No I/O, no async, no Lance calls — that
is what makes "the S3 pointer is written last" and "a warm boot reads
no map data" testable as decisions before any wiring exists.
Three clocks, never conflated:
- semantic import identity (RegionId/SourceBakeId/SourceManifestSha256,
ImportIdentity, OriginSeal, ImportSeal, FNV-1a/64 ImportId as an
idempotency marker with tuple-level reconcile() behind it)
- physical Lance history (publication_version vs observed_head,
head-drift vs head-behind, HeadDecision/recover_behind)
- replica progress (StorageRoot resolution, ActivationPolicy)
Lifecycle is a PRODUCT of serving state and import-operation state
(Lifecycle{serving: Option<ServingSet>, operation: Operation}, fenced
by Generation) rather than one sum type — an earlier draft folded them
together and a failed import had nothing to restore to; a stale
completion for a superseded candidate is now structurally rejected
rather than able to activate or fail the wrong candidate.
Retention specifies intent only: GcAuthority::None in Phases A-C means
no code path here can return a non-empty deletion set, regardless of
grace period or inventory completeness, until Phase D's Lance-native
reachability mechanism lands.
62 tests (29 in osm_lifecycle + the pre-existing 120), each a Phase-A
CONTRACT PROBE proving a decision function, not yet a system falsifier
— the module doc and the plan file both track which system-level
falsifier each probe still owes to a later phase.
Plan: claude-notes/plans/2026-08-15-osm-lance-lifecycle.md
|
Important Review available on request
Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_d5cec63a-fcce-419d-b7bd-09c59e0230ea) |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Summary
Phase A of the OSM/Lance lifecycle redesign (plan:
claude-notes/plans/2026-08-15-osm-lance-lifecycle.md), correcting thecurrent code's failure mode: a full SHA-256 re-hash of every artifact on
every boot (even warm cache hits), which is the measured cause of RAM
climbing back to ~4 GB after a restart with no map traffic.
Core idea — three clocks, never conflated:
RegionId/SourceBakeId/SourceManifestSha256(all validated/canonical, path-traversal-safe),
ImportIdentity,OriginSeal,ImportSeal, FNV-1a/64ImportIdas a fast idempotencymarker with a full-tuple
reconcile()behind it (a marker match is acandidate, never proof).
publication_versionisSomeonly on afresh commit, never invented on a reconciled retry;
HeadDecisionseparates head-drift (later, unexplained) from head-behind
(replica truncated/stale) — two different failures, two different
recoveries (
recover_behind).StorageRoot(5-candidate ordered policy; onlyRAILWAY_VOLand a probed/volume01are intrinsically volume storage,everything else needs an explicit durability declaration),
ActivationPolicy.Lifecycleis a product, not a sum:serving: Option<ServingSet>besideoperation: Operation, fenced by a monotonicGenerationso a stalecompletion for a superseded import can never activate or fail the wrong
candidate. An earlier draft made this one enum and a failed import had
nothing to restore to — caught by a failing contract probe and fixed by
the product-state redesign (see the plan's "Correction pass" section for
the full nine-point review this went through).
Retention specifies intent only in Phases A–C:
GcAuthority::Nonemeansno code path here can return a non-empty deletion set, regardless of grace
period or inventory completeness, until Phase D ships a real Lance-native
reachability mechanism.
Also included:
crates/stubs/notebook-queryhad a stale directdatafusion = "53"dependency that nothing in its source actuallyreferenced — it forced cargo to build an entire duplicate
datafusion-53/arrow-57 subtree (~26 duplicate
datafusion-*crates)alongside the one
lance-graphalready requires at 54. Removed;Cargo.locknow carries a single
datafusion(54.1.0) andarrow(58.3.0) workspace-wide.What's NOT in this PR
Per the plan, this session is scoped to Phase A only (pure types, no
I/O/async/Lance calls). Phase B (
OsmArtifactManager,/api/osm/status,memory telemetry) and beyond are explicitly deferred — every test here is a
contract probe (proves a decision function), not yet a system
falsifier (proves the running system obeys it); the plan's falsifier table
tracks which phase owes each one.
Test plan
cargo test -p cockpit-server osm_lifecycle— 29/29 new tests passcargo test -p cockpit-server— 149/149 pass (0 regressions)cargo build --workspace— clean (warnings only, pre-existing)cargo clippy -p cockpit-server --no-deps --all-targets -- -D warnings— 52 pre-existing findings unchanged (verified via stash against
unmodified
origin/main); this PR's new code contributes zerocargo fmt—osm_lifecycle.rsand the one-linemain.rsdiff areclean (ran
rustfmton the leaf module directly to avoid theenvironmental full-crate reformat this repo's toolchain triggers —
see plan notes)
cargo nextest run --workspace/cargo xtask verify --skip-hub-build— blocked by pre-existing debt unrelated to this diff, not by
anything in this PR:
- 3 deterministic, reproducible failures in
crates/stubs/notebook-query/src/orchestrator.rs(
test_graph_sensorium_stagnant,test_restore_on_high_efficiency,test_temperature_rises_on_stagnation) — pure state-machine logic,no timing/randomness, no reference to
datafusionanywhere in thatcrate's source, so there is no causal path from this PR's dependency
cleanup to these results. Confirmed unrelated:
git difftoucheszero files under
crates/stubs/notebook-query/src/.- 3 pre-existing
clippy -D warningsfindings incrates/aiwar-ingest(collapsible-if / implicit-clone), also untouched by this diff.
Both are out of scope for this PR (different subsystems, "fix one bug at a
time"); flagging them here rather than silently working around them.
🤖 Generated with Claude Code
Generated by Claude Code