Pinned dependency: unmodified crates.io Lance 13.0.0, complete package family. OmniGraph does not vendor or patch Lance. Purpose: required upstream reading and current OmniGraph compatibility fences
Upgrade boundary: certified Lance 11 full-text indexes retain their analyzer compatibility under Lance 13. Uncertified or incompatible indexes still require an explicit rebuild before search. See full-text compatibility.
Lance is OmniGraph's storage substrate. Before changing a Lance-shaped behavior, read every full page in the matching domain below, plus every page that is even slightly relevant. The domains overlap: transactions affect indexes, compaction affects row IDs, and cleanup follows branch/tag references.
Fetch full pages, never summaries:
curl -sL <url> | pandoc -f html -t markdownSummaries routinely omit default flags, visibility restrictions, nested specs, and compatibility details. The index is a router, not a substitute for the pages.
| Topic | URL |
|---|---|
| Lance overview | https://lance.org/quickstart/ |
| Vector search | https://lance.org/quickstart/vector-search/ |
| Full-text search | https://lance.org/quickstart/full-text-search/ |
| Versioning and time travel | https://lance.org/quickstart/versioning/ |
| Lance's agent guide | https://lance.org/format/AGENTS/ |
Read the complete section for manifest/table work, staged writes, recovery, schema metadata, fragment lifecycle, or raw file assumptions.
| Topic | URL |
|---|---|
| Format overview | https://lance.org/format/ |
| File format | https://lance.org/format/file/ |
| File encoding | https://lance.org/format/file/encoding/ |
| File versioning | https://lance.org/format/file/versioning/ |
| Table layout | https://lance.org/format/table/layout/ |
| Table schema | https://lance.org/format/table/schema/ |
| Table versioning | https://lance.org/format/table/versioning/ |
| Transactions and conflicts | https://lance.org/format/table/transaction/ |
| Branch/tag format | https://lance.org/format/table/branch_tag/ |
| Row-ID lineage | https://lance.org/format/table/row_id_lineage/ |
| MemWAL format (upstream only; no OmniGraph consumer) | https://lance.org/format/table/mem_wal/ |
Read all four for graph branches, snapshots, merge ancestry, retention, or GC.
| Topic | URL |
|---|---|
| Branch/tag format | https://lance.org/format/table/branch_tag/ |
| Operational guide | https://lance.org/guide/tags_and_branches/ |
| Versioning quick start | https://lance.org/quickstart/versioning/ |
| Table versioning | https://lance.org/format/table/versioning/ |
Lance refs protect one dataset's files. OmniGraph's graph branch and manifest remain the authority that coordinates the corresponding refs across datasets.
Read the overview plus the concrete index and lifecycle pages involved.
| Topic | URL |
|---|---|
| Read/write guide | https://lance.org/guide/read_and_write/ |
| Distributed write | https://lance.org/guide/distributed_write/ |
| Rust write/read example | https://lance.org/examples/rust/write_read_dataset/ |
| Data evolution | https://lance.org/guide/data_evolution/ |
| Migration | https://lance.org/guide/migration/ |
Read both for local/S3/Azure behavior, retries, sessions, request accounting, credentials, or storage fault tests.
| Topic | URL |
|---|---|
| Object stores | https://lance.org/guide/object_store/ |
| Observability | https://lance.org/guide/observability/ |
| Topic | URL |
|---|---|
| Data types | https://lance.org/guide/data_types/ |
| Arrays/lists | https://lance.org/guide/arrays/ |
| Blob v2 | https://lance.org/guide/blob/ |
| JSON | https://lance.org/guide/json/ |
| Topic | URL |
|---|---|
| Performance and caches | https://lance.org/guide/performance/ |
| Read/write maintenance | https://lance.org/guide/read_and_write/ |
| Fragment reuse | https://lance.org/format/index/system/frag_reuse/ |
| Distributed indexing | https://lance.org/guide/distributed_indexing/ |
| DataFusion integration | https://lance.org/integrations/datafusion/ |
Lance 11's stemmer is not query-compatible with existing Lance 10 postings.
Every full-text segment used by a query must carry OmniGraph's artifact-scoped
analyzer certificate. The certificate is bound to the immutable UUID, index
details, and file inventory, not the current dataset writer or graph head.
Missing or unrecognized proof produces FullTextIndexRebuildRequired before
execution. Successful immutable proof uses Lance's bounded session metadata
cache; failures are not cached. Ordinary reads do not inspect these certificates.
The full builder writes proof before the existing CreateIndex publication.
The same builder with train(false) writes an untrained segment (an empty
fragment bitmap, no postings) with the same proof; Lance matches every row it
does not cover with its analyzer, and every writer declares one for a declared
column whose committed version holds no segment
(writes);
public object-store/base-path resolution and native garbage collection own the
bounded JSON file. Directory placement mirrors Lance's private helper and is
guarded against independently written native index files. Stable-ID
compaction preserves proof. Ordinary optimize excludes incremental full-text
folding, which lacks an uncommitted provenance hook; uncovered rows remain
searchable until an explicit rebuild. Planning and execution share the same
foldable-index predicate; pending full-text coverage alone is not recovery work.
Other index maintenance is unchanged.
The audited analyzer dependency set has exact pins and a resolved-lockfile guard. Any change needs a compatibility audit before keeping or changing the certificate generation. See RFC 0043 for the decision and the operator procedure.
These are the current OmniGraph deltas over stock Lance 13. They are not a history of dependency bumps.
| Surface | Current fence | Test owner |
|---|---|---|
| File format | Every production write explicitly selects stable V2_2; the graph stamp remains 16. Experimental V2_3, spilled row lineage and stable-row-ID fragment-reuse indexes are not enabled as graph capabilities. | integration/lifecycle.rs, write-site source guards |
| Graph keys | Schema-v9 node/edge tables carry the exact non-null __id unenforced primary key (edges also __src/__dst); v8 tables keep the id/src/dst spellings introduced in v6, and the explicit system-column upgrade (RFC 0040) renames them with one rename-only Operation::Project commit per table (TableStore::renamed_schema), staged detached and published as a pin (RFC 0067, detached-only), which must keep every fragment, field id, the primary-key marker and index attachments. Strict insert/upsert uses the sealed filter-bearing adapter; raw keyed Append is forbidden. |
lance_surface_guards.rs, staged-table tests, forbidden_apis.rs, system_column_upgrade.rs |
| Stable row IDs | Graph tables use stable row IDs; delete/update/index maintenance must retain their mapping. Overwrite allocates fresh IDs and restore retains the allocation high-water marks. Staged-view IDs are provisional, not committed identity. | lance_surface_guards.rs, staged-table tests, writes.rs |
| KNN result order | Lance 13 V2 row-stream payload hydration preserves ordering metadata, which WorkMemory uses to merge parallel results in rank order. Supported graph writers, including the supported upgrade sources, select V2_2; no Legacy V1 partition fence is needed. |
lance_surface_guards.rs::flat_knn_v2_late_payload_preserves_order, search.rs::nearest_large_k_preserves_global_order_through_payload_hydration |
| KNN probe budget | A nearest scan sets maximum_nprobes per index delta and widens it from Lance's execution summary (partitions_searched / partitions_ranked, read through scan_stats_callback); a prefilter admitting fewer rows than k makes Lance emit the unreached rows at _distance = +inf, which the engine resolves with a flat exact rescan (use_index(false)); count_rows(None) is the live row count (deletions excluded), read for the ladder's exhaustion stop and as the overfetch loop's exact-pass k. A renamed counter or marker turns the ladder fail-closed. The factor and both fallbacks are the plan's declared NearestPolicy. |
lance_surface_guards.rs, search.rs |
| Full-text eligibility placement | Lance scores fragments no full-text segment covers flat, counting documents and tokens over the rows its prefilter admits (plan_flat_match_query), so a prefilter changes those rows' BM25 scores; prefilter(false) scores the whole corpus and refines afterwards. A bm25 scan therefore filters before scoring only under the plan's recorded full coverage (Eligibility::BeforeScoring) and runs the postfilter otherwise. Coverage is the union of the column's inverted segments' fragment bitmaps, so an untrained segment (the analyzer declaration, an empty bitmap) adds none: a column with no postings is partial. Drop the placement rule at the bump whose flat tail scores from filter-independent statistics. |
lance_surface_guards.rs::fts_prefilter_changes_unindexed_scores_and_postfilter_keeps_them, cases/v2/bm25_score_ignores_a_filter_over_an_unindexed_fragment.gqt |
| Full-text analyzer | Every selected FTS segment needs artifact-scoped compatibility proof. Explicit branch rebuilds replace all segments from rows; old snapshots may refuse FTS. With no segment, Lance's flat path tokenizes with a bare SimpleTokenizer, so every committed version that holds no segment for a declared column gets an untrained one (TableStore::commit_staged_detached), and ensure_indices builds postings where no segment covers a fragment. |
staged-table tests, search.rs, integration/maintenance.rs, schema_apply.rs::full_text_analyzer_survives_schema_evolution_issue_904, search.rs::full_text_analyzer_holds_after_first_rows_and_overwrites_issue_904, lance_surface_guards.rs::fts_untrained_segment_applies_its_analyzer_to_rows_it_does_not_cover (the untrained segment carries its analyzer) |
| Blob v2 | Null, valid empty, non-empty, selector cardinality, neighboring bytes, and 3→1 compaction are pinned on Lance 13. A filtered scan that projects Blob-v2 descriptors is pinned before and after compaction: the single-cell read shape (read_blob_at: one descriptor column, an id equality filter through a BTREE plus an unindexed tail, stable row ids) and a predicate on the Blob column itself under every BlobHandling. Upstream fixed the projection assertion this shape once tripped (lance#7707, fixed by lance#7664 before Lance 9); OmniGraph carries no workaround or fallback, so a red guard blocks the bump. |
lance_surface_guards.rs (filtered_scan_projects_blob_descriptors_on_indexed_multi_fragment_table), integration/maintenance.rs |
| Blob batch reads | Every complete managed Blob read goes through TableStore::managed_blob_payloads, the one Dataset::read_blobs(column).with_row_ids(..).preserve_order(true) call, and consumes try_into_stream, never execute. Materializing rewrites (TableStore::rebuild_blob_column) take one batch's values; export and the change-feed baseline (BlobRowReader) open one call per Blob column per batch of rows and take its values row by row as they render; change-feed images, the change comparator's payload tie-break and entity reads open a one-row reader. Callers pass managed row ids only: given an external row, read_blobs opens and reads the referenced object. Every call sets with_io_buffer_size_bytes(BLOB_READ_IO_BUFFER_BYTES) (8 MiB); Lance's default is SchedulerConfig::max_bandwidth, 32 MiB times the store's I/O parallelism (2 GiB at the cloud default of 64). Lance plans the reads in request-order batches no larger than that buffer (a larger value alone) and reads a batch only when the stream reaches it, so a reader that holds a stream across rows keeps one batch per column resident. The stream yields one ReadBlob per requested id in order, with duplicates, None for null and empty bytes for a valid empty value; the engine checks each length against the descriptor. Only the single-cell read facade keeps take_blobs, for bounded ranges of one value. |
lance_surface_guards.rs::compact_files_succeeds_on_blob_columns (Guard 10: every managed placement, small-buffer stream equals execute, a taken value leaves the next batch unread, external rows are read), forbidden_apis.rs::lance_batched_blob_read_call_counts_are_pinned (one production read_blobs and one with_io_buffer_size_bytes call, both in table_store.rs; the helper's callers in table_store.rs and export.rs; take_blobs in blob.rs alone; no read_blob_ranges), integration/export.rs::export_reads_each_batch_of_managed_blobs_in_one_read_per_column |
| Hydration reads | HydrateExec reads deferred return columns through FileFragment::open(projection, FragReadConfig::with_row_address(true).with_scan_scheduler(..)) and FragmentReader::take(offsets, 1, ..): decode tasks of one row each (the reader merges overlays and applies the deletion vector as TakeBuilder does), each copied at its own size and charged before the next is awaited, because a decoded row can slice a buffer of the whole read and the pool charges a buffer by its allocation's capacity. The engine's ScanScheduler caps Lance's read-ahead at an I/O buffer reserved in the query pool; io_buffer_size is a soft limit, since the queue always admits its head request and a logical request keeps moving once admitted. Lance 13 still ignores the passed scheduler for a data file with a base_id (open_current_file_reader). Its optional BaseSchedulers cache can supply a scheduler for each base, but the current caller supplies none, so Lance opens a SchedulerConfig::max_bandwidth scheduler for each inherited file. A Lance branch reaches its parent's files through base_id; a graph branch stages detached commits in the table's own base and never holds such a file, but a table fork created before v11 (a Lance branch) does, so a fragment holding one is read one row per take, without I/O coalescing, and counted as single_row_reads. Drop that fallback at the bump that honors the passed scheduler for every base. |
lance_surface_guards.rs::fragment_reads_bypass_the_passed_scheduler_for_inherited_files (goes red at that bump), in-source operators::hydrate::tests::fragment_rows_reads_inherited_and_own_fragments_by_address, integration/engine_v2_memory.rs::a_hydrated_fetch_stops_at_the_chunk_bound_when_rows_widen, integration/engine_v2_memory.rs::hydration_on_a_graph_branch_reads_in_coalesced_requests, integration/engine_v2_memory.rs::hydrated_copies_of_a_joined_row_stay_within_the_chunk_bound |
| Schema evolution | Schema apply changes an existing table's columns without writing a data file (TableStore::stage_schema_evolution). Renames and drops commit as Operation::Project over the same field ids (Lance's own alter_columns rename and drop_columns); additions commit as Operation::Merge over the unchanged fragments with nullable fields whose ids exceed Manifest::max_field_id (Lance's own all-null add_columns). Lance 13 validates detached commits too, and its merge_schema_valid refuses a Merge that moves a field id to another path, so a table with renames or drops and additions chains the Merge on the Project. Lance 13 reads a field that no data file of a fragment holds as null (NullReader) and skips a data file holding no projected field; Project drops a data file whose every field is dropped; retain_relevant_indices keeps each index whose fields survive, with its fragment bitmap, and removes an index on a dropped field. Merge stamps no row version for fragments whose files are unchanged. Keyed writes send batches in catalog column order, and Lance 13's merge-insert matches a source to the full table schema only in the same field order (otherwise it takes its subschema path), so the evolved schema keeps the catalog order. A dropped column's values stay in the data files until optimize rewrites the fragment (dropped-column erasure, next row). |
lance_surface_guards.rs::metadata_only_schema_evolution_keeps_blob_files_ids_and_index_coverage, schema_apply.rs, the deferred benchmarks/deferred/crates/omnigraph/tests/compaction_memory.rs.disabled schema-apply instrument (not an active test) |
| Dropped-column erasure | A drop's Operation::Project keeps every data file that also holds a surviving column, and that file's DataFile::fields still lists the dropped field id. A Blob-v2 column is listed by its own id, never by its descriptor children's, so on a table without drops every listed id is a schema field id. Lance 13's planner also selects a fragment when a file contains only dropped user columns but remains necessary for row lineage. It does not select a lone fragment solely because a file mixes kept and dropped columns, so those bytes still need the engine's extra task. TableStore::plan_table_compaction, the one plan behind optimize's planning gate and stage_compaction, appends a single-fragment TaskData (public, as is CompactionPlan::tasks) for every other fragment whose data or overlay files list a non-negative id outside the schema. Lance's planner applies its per-run source budgets (max_source_fragments, max_source_rows, max_source_bytes, in limit_tasks_to_source_budget) to the tasks it returns, so the appended tasks would fall outside them; the engine sets no budget, and plan_table_compaction refuses options that set one. A single-fragment group is the shape Lance plans for deletion materialization, and Rewrite replaces it in place. Compaction re-encodes by default (CompactionMode::Reencode; binary copy would carry pages) and scans the current schema, so the new files list only current field ids and only the kept Blob values get new sidecars; the task is sized like any other, from the kept Blob columns alone. cleanup then sweeps the old files with their sidecars once no retained version names them. |
lance_surface_guards.rs::compacting_a_chosen_fragment_drops_the_bytes_of_dropped_columns, integration/maintenance.rs::optimize_then_cleanup_erases_dropped_property_values, staged_tests.rs::table_compaction_planning_refuses_source_budgets, the deferred benchmarks/deferred/crates/omnigraph/tests/compaction_memory.rs.disabled erasure instrument (not an active test) |
| Compaction of Blob tables | Lance 13 compaction cannot binary-copy a Blob table; it rewrites through a scanner and materializes every managed Blob-v2 payload of one scanner batch (external descriptors are carried unread). A batch never crosses a fragment, so at the default batch size one batch can hold a whole fragment's payloads. TableStore::stage_compaction executes the plan's tasks itself and sets each task's CompactionTask::options.batch_size from a descriptor-only scan of that task's fragments: the largest row's managed Blob bytes, summed over its Blob columns, with external descriptors counting nothing (compaction_blob_batch_rows: a 32 MiB budget, 1 to 8192 rows, so a row over 16 MiB puts its whole task on one-row batches; a nested or non-Blob-v2 Blob field fails closed to one row). Every task of a Blob table gets the derived size explicitly, capped by a caller's smaller one, so it takes precedence over LANCE_DEFAULT_BATCH_SIZE, including when the variable is smaller; all tasks are sized before any executes, so when a fragment being compacted holds a descriptor the decoder refuses, sizing can refuse the compaction with a Blob integrity error before that table is rewritten, and nothing is committed; a table with nothing to compact is not scanned. Lance reads a task's options into its scanner and ignores the batch size when planning. A set batch size beats LANCE_DEFAULT_BATCH_SIZE on the V2.x read path, where the variable is only the fallback. The fence is the scanner shape: a scanner built as compaction builds it honors an explicit batch size over the variable. No CI-run test observes Lance passing CompactionOptions::batch_size into that scanner; the historical compaction_memory.rs instrument was the only observer and is now deferred under benchmarks/deferred/ pending GQ optimization support. This bounds managed payload bytes per scanner batch, not total peak memory: decoder, I/O and copy buffers are additional, and one oversized value may exceed the target. Drop the derivation at the bump whose compaction streams Blob payloads. |
lance_surface_guards.rs::explicit_scanner_batch_size_beats_lance_default_batch_size_env (Guard 10b: the compaction scanner shape honors an explicit batch size over the variable), integration/maintenance.rs::optimize_compacts_large_blob_rows_in_bounded_batches, integration/maintenance.rs::optimize_sizes_a_blob_batch_by_the_sum_of_a_rows_blob_columns, integration/maintenance.rs::optimize_sizes_each_compaction_task_from_its_own_fragments, integration/maintenance.rs::optimize_does_not_size_a_blob_batch_by_external_references, integration/maintenance.rs::optimize_compacts_blob_fragments_with_deleted_rows, staged_tests.rs::compaction_blob_batch_rows_bounds_one_batch, the deferred benchmarks/deferred/crates/omnigraph/tests/compaction_memory.rs.disabled instrument (not an active test) |
| Index coverage | Indexes are derived. Rewrites and compaction may leave an uncovered tail; reads must combine indexed and scan paths until explicit reconciliation. | integration/scalar_indexes.rs, search.rs, integration/maintenance.rs |
| Index segments | Lance masks a logical index's results by the union of its segments' fragment coverage. On a stable-row-ID table an update keeps the row ID and moves the row to a new fragment; once a delta segment covers that fragment, the older segment's entry for the row passes the mask, so full-text search returns the row for a term it no longer holds (twice for a term both versions hold) and a BTREE equality returns it for its old value. Lance applies per-segment ownership to vector search only (lance#7371, lance#8351); the full-text and scalar paths are reported as lance#9817. OmniGraph never adds a delta segment: stage_index_fold rebuilds a lagging index whole and the full-text rebuild replaces every segment. An untrained full-text segment (an empty fragment bitmap) carries its analyzer to every row it does not cover, with rows and scores equal to a full build; with no segment, Lance's flat path tokenizes with a bare SimpleTokenizer. Drop the whole-rebuild rule at the bump whose scalar and full-text search apply segment ownership. |
lance_surface_guards.rs::index_delta_segment_serves_stale_entries_of_an_updated_row (goes red at that bump), lance_surface_guards.rs::fts_untrained_segment_applies_its_analyzer_to_rows_it_does_not_cover |
| Inherited index files | Lance 13 preserves inherited index bases during nested clones and resolves external fragment-reuse details at their original base. The adapter checks the exact source version and delegates clone metadata to Lance. Production branch creation clones only __manifest, which has no fragment-reuse index; graph tables are not cloned. Malformed external fragment-reuse payloads are outside this adapter's supported graph contract, and native branch creation need not read or reject them before publication. |
lance_clone tests; lance_surface_guards.rs::stock_lance_nested_clone_preserves_inherited_index_base_issue_7840 |
| Branches/tags | Each graph-branch lifetime is {logical}.{ULID}; bare names identify legacy lifetimes. Unretired native __manifest/_refs/branches/ entries define logical branches. Deletion writes identity-bound retirement metadata, archives the exact BranchContents inside its native tree, then unlinks the active ref. Needed native ancestry and historical merge-base providers remain readable through the archive. User __manifest tags block logical deletion; valid engine merge-input tags instead protect their exact snapshots until the target publication witness expires. |
integration/branching.rs, branch_control.rs tests, integration/merge_cleanup.rs |
| Branch ref reads | Stock Lance 13 reads a ref as head then get_range(0..size), so a replace_metadata landing between the two calls serves a prefix of the rewritten object and the read fails to parse instead of returning either version. branch_control::read_branch_refs rereads a torn read (Arrow, CorruptFile) or a raced delete (NotFound), at most three reads in all, around the engine's listed and exact authority reads (list_branches, branches().get, get_identifier). Resolving a logical branch name (branch_control::resolve_live_native_branch) lists _refs/branches/ once and reads only that branch's own incarnations, so another branch's retirement rewrite, or its malformed retirement record, no longer fails a branch open or a held-ref re-resolve; fresh creation narrows content reads to its source, target-related names and schema sentinel. Operations that require a complete inventory, including logical enumeration and cleanup, still read every active ref; maintenance and historical resolution also read retirement archives. Delete the retry arm and un-ignore the guard at the bump whose from_path reads a ref in one get. |
branch_control.rs stale-head tests (they go red at that bump), lance_surface_guards.rs::branch_ref_read_survives_concurrent_metadata_rewrite (run with --ignored) |
| Cleanup | Stock cleanup_old_versions is never called on a graph table. Lance 13 does not trace detached manifests, and since lance#8944 it lists data/ through max(retained-manifest floor, now − 7 days), making old files referenced only by detached versions eligible for deletion while their manifests remain. Lance 11 listed data only below the retained floor: the native guard puts the attached v1 manifest before the detached file, and the file before the seven-day cutoff, to distinguish the two behaviors. Files shared with retained attached versions stay protected; a recent detached snapshot can still lose an old file it inherits. An external Lance cleanup of graph tables or __manifest, whose proposals and unattached snapshots are detached versions, can therefore make retained snapshots, including the current graph, unreadable. The engine collector discovers canonical registrations, including tombstoned table lifetimes. Roots include retained graph snapshots, selected merge bases, exact tagged graph snapshots and exact tagged table versions. It marks base and overlay data files, deletion files, index directories, transaction files and Blob sidecars, preserving borrowed physical origins. Native tree candidates are frozen before graph capture; after table inventories, unchanged branches, identities, heads and tag inventories must validate before deletion. Valid staging witnesses are judged against captured heads and a complete post-list live-identity cut; ambiguous ownership stays. Unneeded retired trees and their archives are collectible after dependency closure. Foreign linear versions remain untouched; orphan objects require their own seven-day age. A failed trace keeps its table, and incomplete origin tracing prevents physical-file deletion across the run. |
lance_surface_guards.rs::stock_cleanup_sweeps_a_week_old_detached_data_file_lance_13 (the stock sweep; goes red at the bump that traces detached manifests), forbidden_apis.rs (cleanup_old_versions( is deny-listed in engine sources), integration/maintenance.rs, collector.rs tests, the DST collector oracle, integration/merge_cleanup.rs, integration/branching.rs |
| Bound branch create | Lance 13 Dataset::create_branch hands its CommitBuilder the dataset's object store and store parameters but not its session. The commit writes through that store, but its existence probe of the new branch location (DatasetBuilder::load, a _versions/ listing) opens on a default session, whose registry builds a client from the store parameters alone. A bound handle's datasets carry no provider options (its registry supplies them), so lance_clone::create_branch passes the captured binding's options as those parameters; without them the probe reaches the ambient provider configuration. Drop the parameters at the bump whose create_branch passes the dataset's session. |
bound_storage.rs (S3-configured decoy) |
| Table forks | No table fork is created for a graph branch: a branch write stages detached on the dataset and native ref its inherited registration names. Lance's create_branch accepts a detached id as the parent but leaves the branch headless (a linear commit on it is refused), which is why the engine forks nothing. Forks created before v11 (fork.{incarnation}.m{base_version}.{commit_ULID}, legacy in the incarnation position) stay valid: a registration naming one opens there, table_fork_owner records ownership, and cleanup may reclaim an unreferenced one after its parsed owner incarnation is gone and tag, ancestry and age protections allow it. Canonical ULIDs and version numbers are required for that name-based liveness check; generated forks with malformed or legacy owner components stay retained. Names never establish table ownership. |
lance_surface_guards.rs, integration/branching.rs, metadata tests |
| Detached commits | Every graph-table write is a Lance detached commit (_versions/d{version}.manifest), which never moves HEAD and needs no conflict pass, and it is the table's version for its whole life: nothing replays it linearly, no Restore is used, a graph table's linear HEAD stays at its v1 create, and the registration's omnigraph.last_linear_version records where the linear history stopped. Every manifest names its transaction file {read_version}-{uuid}.txn, which is how the engine matches a pin's transaction_uuid and follows a chain of detached commits by read_version links without a second request (the collector, walk_chain); the pure-insert merge proof follows the same links and also reads each link's transaction file to certify it. A detached commit never triggers Lance auto-cleanup: auto_cleanup_hook runs only from commit_transaction, and every engine commit passes skip_auto_cleanup: true. |
lance_surface_guards.rs (rfc_0067_*), collector.rs tests |
| Catalog publication | Every __manifest publish is an Operation::Overwrite committed through CommitBuilder::with_max_retries(0). Lance 13 treats that as a strict overwrite: it skips conflict resolution and writes _versions/{read_version + 1}.manifest only if absent, so a concurrent commit fails the publish with CommitConflict instead of being rebased over; the publisher maps that to RowLevelCasContention and retries from a fresh load. A Lance release that rebased overwrites under zero retries would break this version CAS. |
omnigraph-catalog stale_overwrite_loses_the_version_cas_without_replacing_the_winner (the stale publish itself), publisher_retry_vocabulary_is_the_version_cas (the error mapping) |
| Manifest request resolution | Lance 13 writes a version's manifest with one put-if-absent through the dataset's CommitHandler (ConditionalPutCommitHandler on local Unix, S3 and Azure roots). It writes the transaction file first and propagates that put's failure through a plain ?, so no manifest request was sent. On a failed manifest put it reads the version back (verify_commit_outcome), and because that handler calls a missing version definitive (is_version_not_found_definitive), an absent version ends the attempt at once, although an abandoned put can still land afterwards. The handler OmniGraph installs on every dataset (lance_clone) therefore re-issues an ambiguous manifest put with the same manifest and precondition, up to MANIFEST_REQUEST_REISSUES times, before Lance reads back, and records per commit whether a request was sent and how it ended (manifest_request). A held version goes back as CommitConflict to Lance's read-back, which compares the complete transaction at that version. Lance deletes the attempt's transaction file only when the handler calls a missing version definitive (is_version_not_found_definitive, read nowhere else), so only the commit's own request record answers it: the catalog's per-commit handler answers from its probe and keeps the inner answer unless the request may still land, and a dataset's shared handler, which holds no record of the attempt Lance reads back, never calls a missing version definitive. Lance then returns CommitStatusUnknown and keeps the file a late manifest would name; a table's orphaned transaction file goes to the collector's orphan sweep. The catalog publisher reads the record: no request sent, or one the store refused outright, is Unpublished; a request that may have applied is read back. A generic local-filesystem error is ambiguous too: the Windows rename handler can create the destination before temporary-file cleanup fails. Drop the re-issue at the bump whose commit path settles an ambiguous manifest put itself, and keep the record while Lance's errors do not say whether the manifest was sent. |
in-source lance_clone::tests::manifest_request_is_re_issued_until_the_store_answers_issue_1067, lance_clone::tests::a_shared_handler_never_calls_a_missing_version_definitive_issue_1067, manifest_request tests; omnigraph-catalog transaction_file_failure_before_the_manifest_is_unpublished_issue_1067, unresolved_manifest_create_keeps_its_transaction_file_issue_1067, lost_manifest_acknowledgement_resolves_by_re_issue_issue_1067, abandoned_manifest_create_that_lands_late_publishes_once_issue_1067, manifest_create_without_a_definite_answer_stays_in_doubt_issue_1067 |
| Catalog freshness probe | A read on a warm handle learns whether its branch's __manifest moved by asking for attached versions by name (ManifestNamingScheme::manifest_path, V1 or V2 paths): the held version v and v + 1 in two concurrent HEADs, then v + 2 when v + 1 exists; with v present, the first "not found" ends the history. Lance 13's latest_version_id lists _versions/ instead, and on a store with ordered listing (S3, GCS, Azure) that one request returns up to 1,000 keys, so a branch past 1,000 commits paid a full page on every read. The first absent name is the latest because Lance writes an attached version only with a put-if-absent over a read of the version below it, and nothing deletes a version of a live branch's __manifest (cleanup collects a retired branch's whole tree, which the named-branch probe's identifier read catches); a reclamation of old __manifest versions must keep every version above one a live handle holds. An absent v (a removed __manifest answers "not found" for every name), a gap of more than two versions, a detached handle or any error falls back to the listing, which reports a removed __manifest as NotFound. |
omnigraph-catalog freshness_probe_asks_for_the_next_version_by_name (V1 and V2 paths on an ordered store: HEADs and no listing, the listing past two versions, the listing's NotFound for a removed __manifest); engine lifecycle::write_preparation_manifest_read_failures_carry_before_effect_evidence (a write on a warm handle refuses before effects when __manifest is gone) |
| Local manifest identity | object_store 0.14 quotes its local inode-mtime-size ETag, while Lance 13 direct discovery (current_manifest_local, through object_writer::get_etag) returns the same token bare. The configured native-local commit handler quotes only that known three-part hexadecimal representation when the store exposes direct local paths, preserving warm manifest metadata reuse without extra I/O. Strict generation and identity checks remain; custom handlers, cloud tokens, absent tokens and unrecognized representations are unchanged. |
lance_clone::tests::local_discovery_only_quotes_the_native_generation_token, creation_preserves_configured_handler_session_and_failure_semantics, newly_created_dataset_can_fork_twice_without_reopening; catalog acknowledged_publisher_reuses_metadata_in_each_fresh_probe, publisher_reuses_only_its_exact_bounded_published_rows |
| Packed catalog record | Since stamp v12 the __manifest record fields are one Struct column with lance-encoding:packed = true, which Lance 2.1+ stores row-major in one column (PackedStructVariablePerValueEncoder), so a scan reads one column's pages for the record. Two Lance 13 limits still shape the layout: a packed struct child with a null is refused at write (Per-value compression not yet supported for block type Nullable, the row format has no per-child null bit), so every child is non-null and the present bitmask child carries the logical nulls; and projecting a fixed-width child alone fails to decode (invalid variable-width layout for UInt64), so every scan projects record whole (no scan filters on a record field). If packed struct children gain null support, removing the persisted present layout requires a separate graph-format decision. Remove the whole-record projection only when native child projection is qualified. |
omnigraph-catalog packed_record_round_trips_nulls_through_present_bits, lance_surface_guards.rs::packed_struct_refuses_null_values_and_lone_child_projection_lance_11 (goes red at either bump) |
| Ordered walks | Lance 13's HardCapBatchSizeExec deep-copies a one-row slice before measuring it, fixing the false rejection caused by its parent buffer's capacity (lance#9048). FilteredReadExec still re-slices tails without copying in rechunk_stream_by_size (lance#9692), and a truly wide row still exceeds the 37.5 MiB sorter cap. Every id-ordered walk (branch merge, change feed, export) therefore sorts only id, _rowid and _rowaddr and hydrates rows in bounded chunks (ordered_cursor.rs). |
integration/changes.rs, integration/export.rs and integration/merge_fast_forward.rs wide-row tests (tests/helpers/wide_rows.rs) |
| MemWAL | Upstream support exists, but OmniGraph's RFC 0018 and RFC 0026 experiments were removed. No stream profile, token ledger, hidden stream column, or _mem_wal path is current. |
integration/lifecycle.rs, cluster removed-field diagnostics |
The clone adapter is installed during normal dataset opens and initial dataset
creation, preserving configured commit handlers without an additional reopen.
Branch creation checks the exact source version. A task-local scope reuses the
known main-source ManifestLocation only for the matching native V2 path and
version, avoiding a listing; other locations retain native resolution. The
adapter delegates manifest writing, transactions and commit-handler capabilities
to the original handler. It neither captures an index inventory nor repairs
index bases or reads external fragment-reuse payloads.
Production branch creation clones only the catalog's __manifest dataset,
which has no fragment-reuse index. Graph branches stage detached table commits;
legacy table forks remain readable. Stable-row-ID graph compaction does not
produce external fragment-reuse details. Native Lance 13 preserves inherited
index origins, pinned by cold nested-clone queries. This is not a promise that
native branch creation validates malformed external details before publication:
such payloads can fail only when an index is later read. The remaining adapter
tests own exact source-version refusal, source-location reuse and fallback,
local ETag normalization, and configured-handler/session/failure semantics.
Schema v8 defines native-ref retirement metadata; schema v9 adds the
system-column namespace (RFC 0040); schema v10 lets a registration name a
detached table commit (RFC 0067); schema v11 makes that commit the table's
version for life and records omnigraph.last_linear_version on every
registration (RFC: Detached-only tables). Schema v12 packs the catalog record;
schema v13 stores the schema contract row in each branch's __manifest;
schema v14 keeps a buffer of a branch's latest commits there, as
settled_commit and replaced_table rows that store what differs from the
head and the current table rows, released on HISTORY_RELEASE_BYTES (or
the lower history_release_bytes a session set), and
older commits in Lance files under __history. Schema v15 adds released-run
and merge-edge registry rows. Schema v16 adds the reserved
unattached_<root>_<commit id> tags, each naming a detached __manifest
version that holds one unattached graph branch snapshot under the version 2
candidate property. A branch's highest-ID fragment holds its hot
rows; other fragments hold cold registry rows reused by the publisher.
Generic optimize preserves this layout. A v14 or v15 branch is stamped v16
by its first publication or by the stamp-only route of omnigraph upgrade,
with no offline conversion.
Normal open serves v14 to v16 only and refuses other stamps. The qualified
offline conversion and export/load rebuild routes are owned by
versioning. The system-column
respelling is a separate step on a served graph. Older binaries must not
expose retired refs as live branches.
Stock Branches::get and list include every physical ref. OmniGraph's logical
manifest helpers validate and filter omnigraph.retired_manifest_branch, a
version-1 JSON value binding the exact native name and identifier. An absent
marker means live; malformed, unsupported, or mismatched metadata fails closed.
The metadata update publishes retirement and preserves unrelated metadata.
Before unlinking the active ref, retirement archives the exact BranchContents
inside its native tree; retries validate the same identity in the ref or archive.
Native branch controls retain OmniGraph's existing process-local serialization.
Cold logical enumeration reads active refs; cached named-write admission checks
the existing ref lookup without an added request. Cleanup and historical
resolution read archives. Cleanup retains required native ancestors and exact
merge-base providers, then reclaims unneeded trees with their archives. Delayed
staging is judged from the complete post-list identity cut and final capture
validation, so its retirement history need not remain forever.
--keep N retains the newest N __manifest versions per live branch, which
is N graph commits, and every table version they pin; it never counts Lance
versions of a dataset. --older-than retains every __manifest version
newer than the cutoff and also gates pre-v11 fork collection using tree and
native ref object timestamps before closing over dependencies. Any recent
object retains such a fork, including a fresh retirement metadata update over
an old tree. Ref metadata listings for this age check occur only during
cleanup with an explicit age policy.
- Fetch every full page in every affected domain.
- Inspect the complete upstream release/source and dependency delta. Re-audit the public API assumptions behind every engine compatibility adapter; preserve unmodified crates.io dependencies.
- Run
lance_surface_guardsfirst; a red guard is a required design review, not a test to weaken. - Run focused write, merge, search, maintenance, Blob, branch, and recovery owners as applicable.
- Run local cost guards and configured RustFS/Azure tests for the affected backend.
- Run the canonical workspace test, GQT and both Clippy graphs, the genuine predecessor journey, and the release-profile guards described in testing.md.
- Update the pinned version and only the active compatibility table above.
- Record bump evidence in the release note or owning RFC. Git history is the archive; do not append a permanent audit diary to this live guide.
Namespace REST models and Spark/Trino/Databricks/Python integrations are deliberately absent because OmniGraph does not expose those surfaces. Add a domain only when code makes it reachable.