Skip to content

Delete dataset URIs through object storage and propagate deletion failures #250

Closed

Description

@beinan

The rollout delete handler removes the registry entry and calls tokio::fs::remove_dir_all with the dataset URI. For a file://, s3://, or gs:// URI this is not a valid local filesystem deletion. Errors are only logged and the endpoint still returns HTTP 204.

Local reproduction using file:// (no cloud resources involved):

  1. Create a rollout store, write a row, and merge it.
  2. Delete it: the route returns 204.
  3. Create another store with the same name.
  4. The supposedly deleted row reappears.

Expected: successful deletion removes the dataset; failed deletion is reported and remains retryable.
Actual: the registry hides a dataset whose data still exists, and same-name recreation reopens it.

Relevant code: crates/lance-context-server/src/routes/rollouts.rs::delete_rollout_store. Other store-kind delete handlers use the same local-filesystem pattern and should be checked.

Please use the configured object-store backend, close resident writers before removal, preserve a retry path on failure, and add URI-backed deletion/recreation tests. Cloud behavior follows from the code path but was not tested against production storage.

Found while reviewing main at e292518; fixes will be based on refreshed main (currently 8fe27b2).

Activity

  1. beinan commented on Sep 10, 2026

    @beinan
    CollaboratorAuthor

    Review of PR #251 at 948b970 reproduced a remaining lifecycle regression: retain a rollout store Arc, evict its LRU entry, DELETE the store, then append through the retained handle. The append incorrectly succeeds (Ok(2)). Datagen and generic stores use the same deletion path.

    The follow-up tracks live handles with weak references independently of LRU residency. A per-name lock spans create/open/delete so reopening an evicted live store reuses its handle and a slow open cannot cross a local delete/recreate boundary. DELETE closes and fences the surviving handle before removing data. Tests cover all three store kinds, recreation, and reclamation of unused tracking entries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions