Skip to content

fix: prevent racing dataset creation from overwriting data - #385

Merged
beinan merged 2 commits into
lance-format:mainfrom
beinan:codex/create-first-manifest-only
Oct 19, 2026
Merged

beinan merged 2 commits into
lance-format:mainfrom
beinan:codex/create-first-manifest-only

Conversation

@beinan

@beinan beinan commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Lance checks whether a dataset exists before creating it, then reopens its URI when committing. If another writer creates the dataset and appends between these checks, the delayed creator's initial overwrite can erase that append. Restrict StorageBase::create_with_options to publishing version 1 and reject a successor manifest before committing it.

The restriction is scoped to the creation future, so the returned dataset remains usable for ordinary appends. This protection lives in lance-context and requires no Lance OSS change.

Refs #378.

Testing

  • A deterministic regression holds a creator's actual empty storage-listing result while a second creator creates and appends. Releasing that result verifies the stale create fails and the row/version survive. Removing the guard makes this test fail with zero rows instead of one; restoring it passes.
  • Repository checks passed: workspace Rust formatting, Clippy and all-target tests (594 passed); Python pytest (229 passed, 2 skipped, 1 xfailed), Ruff formatting/lint and Pyright.

@beinan
beinan merged commit e72297d into lance-format:main Oct 19, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant