fix(gridfs): fence file-id ownership across uploads, finalize, abort and delete [WIP]

.github/workflows/gridfs-fix-validation.yml

@@ -0,0 +1,48 @@
+name: GridFS Ownership Safety Validation
+
+on:
+  pull_request:
+    branches: [main, 'fix/gridfs-id-ownership-v01']
+
+concurrency:
+  group: gridfs-safety-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  gridfs:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    strategy:
+      fail-fast: false
+      matrix:
+        mongo-version: ['7.0.37', '8.0.26']
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v5
+      - name: Setup Node 22
+        uses: actions/setup-node@v6
+        with:
+          node-version: '22.x'
+          cache: npm
+      - name: Install dependencies
+        run: npm ci
+      - name: Cache MongoDB binaries
+        uses: actions/cache@v5
+        with:
+          path: .cache/mongodb-memory-server/binaries
+          key: mongodb-memory-server-${{ runner.os }}-${{ runner.arch }}-gridfs
+      - name: Build
+        run: npm run build && npm run build:tests
+      - name: Execute GridFS safety and compatibility cases
+        env:
+          MONSQLIZE_MEMORY_MONGO_BINARY_VERSION: ${{ matrix.mongo-version }}
+        run: |
+          node --test --test-concurrency=1 \
+            .generated/test-dist/test/unit/gridfs/gridfs.test.js \
+            .generated/test-dist/test/integration/gridfs/gridfs.test.js \
+            .generated/test-dist/test/integration/gridfs/gridfs-scan-regressions.test.js \
+            .generated/test-dist/test/integration/gridfs/gridfs-ownership-regressions.test.js \
+            .generated/test-dist/test/integration/gridfs/gridfs-stale-recovery.test.js

.github/workflows/test.yml

@@ -4,11 +4,18 @@ on:
   push:
     branches: [main]
   pull_request:
-    branches: [main]
+    branches: [main, 'fix/gridfs-id-ownership-v01']
+
+# Superseded PR revisions must not accumulate long-running test jobs.
+# Cancelled prior SHAs are never accepted as evidence for the current HEAD.
+concurrency:
+  group: ci-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
 
 jobs:
   test:
     runs-on: ubuntu-latest
+    timeout-minutes: 60
     strategy:
       fail-fast: false
       matrix:
@@ -50,6 +57,7 @@ jobs:
 
   release-gate:
     runs-on: ubuntu-latest
+    timeout-minutes: 60
     steps:
       - name: Checkout
         uses: actions/checkout@v5

changelogs/unreleased.md

@@ -2,7 +2,7 @@
 
 ## Changes after v3.5.3
 
-No changes have been staged after 3.5.3. The GridFS and Search index fixes from the 2026-11-02 weekly scan are assigned to the [3.5.3 versioned notes](./v3.5.3.md); the vector search guidance and Search docs regrouping are in [3.5.2](./v3.5.2.md). Publication is established by the npm registry and immutable tag, not this file.
+Candidate work (not yet released): GridFS per-file ID ownership, durable append/finalize/abort state, deletion fencing, and expiresAt idle-TTL migration. These changes require a coordinated writer drain before upgrading and are pending full compatibility verification. No release version or publication is claimed. The GridFS and Search index fixes from the 2026-11-02 weekly scan are assigned to the [3.5.3 versioned notes](./v3.5.3.md); the vector search guidance and Search docs regrouping are in [3.5.2](./v3.5.2.md). Publication is established by the npm registry and immutable tag, not this file.
 
 ## Archived preparation notes for the 3.3.2 candidate
 

config/package-budget.json

@@ -1,8 +1,8 @@
 {
-  "comment": "GridFS 3.5.0 stays in the main bundle: sync gridfs() cannot be split out of the single-file build. CJS 881096 and ESM 878096 cover driver-error mapping, write-path iid matching, and the 2026-11-02 scan fixes (GridFS cleanup/cursor/session repairs and segmented Search index wait timers, idempotent finalize retries), plus 4 KiB of headroom each approved on 2026-11-02.",
+  "comment": "The 3.5.3 bundle ceilings were 881096 CJS and 878096 ESM. The GridFS cross-operation ownership, durable session CAS/abort fencing, delete protection and idle-TTL migration add ~4.4/4.7 KiB over that ceiling (CI build 2026-11-12: 885536 CJS / 882819 ESM). Raise to 890000 / 887000, keeping ~4 KiB reviewed headroom; no dependency change. Enforce these ceilings in release preflight.",
   "artifacts": {
-    "dist/cjs/index.cjs": 881096,
-    "dist/esm/index.mjs": 878096,
+    "dist/cjs/index.cjs": 890000,
+    "dist/esm/index.mjs": 887000,
     "dist/cjs/cli/data-task.cjs": 50000
   }
 }

docs/en/gridfs-recovery.md

@@ -0,0 +1,71 @@
+# GridFS crash recovery and no-data-loss upgrade runbook (V0.1 candidate)
+
+> **Operator-only.** This is a draft recovery design, not a completed or verified release. Never run recovery against a running, paused or network-partitioned GridFS writer. **Age alone is not proof that a writer is dead.** No automated TTL-based owner unlock is permitted.
+
+## Scope and invariant
+
+The ownership registry is the per-bucket `<bucket>.uploads` collection. A reservation excludes ordinary upload, uploadStream, resumable append/finalize/abort, delete, rename and orphan cleanup for the same ObjectId. A missing file document **does not** prove its chunks are orphaned: an ordinary upload may still be writing, or GridFS driver delete may have removed the files document before chunks. Recovering while an old writer can resume can destroy new data, even with a token-CAS. The maintenance process cannot prove cluster/process shutdown on its own.
+
+The script `scripts/maintenance/gridfs-recover.cjs` uses a default read-only preview, exact state/token/timestamp checks, a persisted `recovering` ownership claim, verified preservation of committed visible chunks, and an append-only-by-intent audit collection `<bucket>.ownership_recovery_audit`. It never performs a time-based lock steal.
+
+## Deployment and rollback gates (required)
+
+1. Stop all HTTP/worker/job/cron processes that can mutate this GridFS bucket, across **all** deployments, driver versions, replica instances, queue consumers and old binaries. Disable automatic restarts and new GridFS requests. Resolve in-flight operations, check for network-partitioned/paused workers, and ensure the old writer fleet is fully drained. A mixed old/new-writer deployment is unsupported because the old version ignores the new registry.
+2. Back up `<bucket>.files`, `<bucket>.chunks`, `<bucket>.uploads`, current indexes and relevant deployment version/worker inventory. Record a timestamp, operator, target database/bucket/id, intended delete/finalize outcome, and why no writer can resume. Retain backups and audit logs outside the cluster.
+3. Deploy the new version while writes remain disabled. Run `ensureIndexes()` after verifying the old `createdAt_1` TTL and any legacy pending sessions. The migration derives `expiresAt` from the last update for eligible idle sessions, removes the legacy TTL index, and creates `expiresAt_1` with `expireAfterSeconds: 0`. TTL can already have removed sessions before migration; examine orphan chunks separately. Never treat an expiring session as safe while a legacy writer may still run.
+4. Inspect every retained owner that must be recovered. Reconcile visible file and chunks, determine its original operation and the desired post-recovery state, and record the evidence. An owner without expiry is not automatically stale. Use the script only for an owner that has been conclusively quiesced.
+5. Run dry-run and compare the proposed action with the recorded intent. To apply, supply the exact observed `status`, `updatedAt` ISO timestamp, and `ownerToken`, `activeAppend.token`, or `recovery.token` when present. Any mismatch is a stop signal. An audit document is written before a change; if a step fails the state remains `recovering` and must be re-inspected.
+6. Recheck file visibility, stored chunk count, lengths, and any relevant SHA. Complete the latest-HEAD Node 18/20/22 CI, MongoDB 7.0.37/8.0.26 targeted suite, applicable Driver 6/7 compatibility checks and release preflight before unfreezing. Verify no `recovering` or unintended active reservations remain.
+7. Only then restart new writers. Rollback to an old binary is **not safe while a new-version writer or reservation is active**; first stop/drain all new writers, reconcile operations, and assess the old binary's inability to honor `<bucket>.uploads` ownership. Preserve the migrated index configuration; do not recreate `createdAt` TTL blindly.
+
+## Manual command
+
+Use a checkout with installed dependencies and a MongoDB driver. Never pass the connection string in CLI arguments or paste credentials into an issue.
+
+```sh
+export MONSQLIZE_RECOVERY_MONGODB_URI='<MongoDB URI from the secrets manager>'
+node scripts/maintenance/gridfs-recover.cjs \
+  --db=my_database --bucket=fs \
+  --id=0123456789abcdef01234567 \
+  --expected-status=aborting \
+  --expected-updated-at=2026-11-12T02:00:00.000Z
+```
+
+This previews but does not write. Retrieve the exact observed fields from the backed-up `<bucket>.uploads` collection first. If the document contains a token, append `--expected-token=...`. **Do not copy** the example ObjectId or timestamp. Apply only after external process quiescence is verified and logged:
+
+```sh
+node scripts/maintenance/gridfs-recover.cjs \
+  --db=my_database --bucket=fs \
+  --id=0123456789abcdef01234567 \
+  --expected-status=aborting \
+  --expected-updated-at=2026-11-12T02:00:00.000Z \
+  --operator=oncall-id --reason=verified-writer-shutdown \
+  --quiesced --apply
+```
+
+For an already `recovering` owner only, verify that the previous recovery executor has stopped and add `--prior-recovery-stopped` to the apply command; this is separate from stopping ordinary GridFS writers.
+
+Every apply attempt includes an audit entry. A crash can leave a `status: recovering` record with `recovery.token`; after proving **both all writers and the previous recovery executor have stopped**, re-inspect that record and use its new status, updatedAt and recovery token to retry. Supply the additional `--prior-recovery-stopped` flag. The restart acquires a fresh recovery token via atomic CAS. Two operators with the same observed snapshot cannot perform destructive steps simultaneously, but in-flight writes by an unquiesced prior executor still cannot be fenced by MongoDB alone. Never delete an ownership record by ID alone.
+
+## State-by-state recovery contract
+
+| Observed owner | After proven quiescence | Critical preservation rule |
+|---|---|---|
+| `kind:direct,status:uploading` | When a file is visible, verify every chunk, length and any claimed SHA before releasing ownership; otherwise delete orphan chunks and release | A mismatched caller-supplied metadata SHA blocks recovery; visible-file chunks are never deleted |
+| `kind:delete,status:deleting` | Complete the interrupted delete of file and chunks before releasing ownership | No new upload may claim the same ID until both steps finish |
+| `kind:cleanup,status:cleaning` | Check visible file; delete chunks only if no file exists; release | Never delete a visible file's chunks |
+| `kind:rename,status:renaming` | Verify the resulting visible file has complete chunks, then release; missing or corrupt files remain locked | The tool does not choose or replay a rename target; the operator must inspect the intended filename |
+| `status:pending,activeAppend` | Recompute the contiguous chunk prefix, clear only the observed append claim, restore seven-day idle expiry | Existing chunks are preserved; caller replays/checks the data |
+| `status:finalizing` without file | Restore pending with expiry; retry normal finalize, which revalidates bytes | Never discard chunks to unlock finalize |
+| `status:finalizing` with file | Re-hash the **actual** ordered chunks and check expected file length, chunk size and digest before releasing session | Forged metadata SHA is not sufficient |
+| `status:aborting` without file | Delete invisible chunks then remove session | No newly created file can take ownership during recovery |
+| `status:aborting` with file | **Blocked** for human forensic review | Never silently delete or detach a visible file |
+| `status:recovering` | Resume by exact recovery token and reverified quiescence | Keep claim if any destructive step is ambiguous |
+
+### Known limitations and release criteria
+
+- MongoDB cannot revoke an already-running Node.js process's in-memory chunk write. `--quiesced` is a mandatory **external operational assertion**, not a technical fencing guarantee. For non-quiesceable workloads, a future generation-aware writer fencing design and driver integration would be needed; do **not** automate unlocking based on age.
+- A direct uploadStream `metadata.sha256` is not authenticated at write time. The tool verifies visible chunk structure and length, but does not treat this metadata as a trusted digest for ordinary uploads.
+- The recovery code is a source-checkout maintenance utility. It is not an online public `GridFSBucket` API and must not be exposed to unprivileged application callers.
+- `drop()` is bucket-wide and ignores per-ID reservations; maintenance must globally stop writers before dropping the bucket.
+- Deterministic tests must show all state paths, wrong-token and wrong-timestamp rejection, fail-closed interrupted recovery, no deletion of visible chunks, and idempotent restart after an interrupted abort/delete. **No merge until the code and this runbook pass the latest-HEAD required CI matrices and independent review.**

docs/en/gridfs.md

@@ -1,6 +1,6 @@
 # GridFS
 
-Instance and Model helpers store and read MongoDB GridFS files. The published npm package is still 3.4.0; this API is staged in the source after that release and is not a published 3.5.0 package.
+Instance and Model helpers store and read MongoDB GridFS files. GridFS was introduced in monSQLize v3.5.0. This document describes the source checkout; verify the installed npm version and its changelog before relying on newer lifecycle behavior.
 
 GridFS is a client convention on ordinary collections (`<bucket>.files` and `<bucket>.chunks`). It does not require a Search or Vector Search deployment. Integration coverage uses `mongodb-memory-server` 7.0.37. The formal server matrix remains MongoDB 7.x and 8.x. Older servers, including 4.x, are unverified.
 
@@ -86,6 +86,7 @@ Chunk bytes must follow the session size:
 
 - `fromChunk` past the contiguous prefix throws `GRIDFS_RESUME_MISMATCH` with `expectedNextChunk`.
 - Replaying already stored chunks with the same bytes is idempotent and does not move `nextChunk` backward.
+- Concurrent appends to one uploadId are serialized by a durable session claim. A competing append, finalize, or abort fails with `GRIDFS_UPLOAD_BUSY` while another operation owns the session.
 - Replaying an earlier chunk with different bytes throws `GRIDFS_RESUME_MISMATCH`.
 - A payload that contains any new chunk must start at `nextChunk`. Starting earlier throws `GRIDFS_RESUME_MISMATCH` and writes nothing.
 - A duplicate `{ files_id, n }` insert is accepted only when the stored bytes match.
@@ -94,9 +95,11 @@ Chunk bytes must follow the session size:
 
 Retrying `finalizeResumableUpload` is idempotent while the session still exists: if a file with the same id is already visible and its `metadata.sha256` and length match the session, the call returns that file and deletes the session. A file with a different sha256 or length still throws `GRIDFS_FILE_EXISTS`. Once the session is gone (a finalize that fully succeeded, an abort, or the TTL), there is no digest to compare, so a retry throws `GRIDFS_FILE_NOT_FOUND`; use `getFile(uploadId)` to check the result.
 
-`abortResumableUpload` deletes the session and, when no files document is visible, that upload's chunks. A missing or already aborted session resolves, and any chunks left under that id without a visible file are still deleted. Aborting an id that already has a visible file does not delete that file's chunks.
+`abortResumableUpload` atomically claims a pending session before deleting chunks. It fails with `GRIDFS_UPLOAD_BUSY` while append or finalize owns the session. It deletes the session and, when no files document is visible, that upload's chunks. A missing or already aborted session resolves, and any chunks left under that id without a visible file are still deleted. Aborting an id that already has a visible file does not delete that file's chunks.
 
-`<bucket>.uploads` has a TTL index on `createdAt` with `expireAfterSeconds: 604800` (7 days). The TTL deletes upload session documents only. It does not delete chunks. Abort or finalize sessions that are no longer needed. After a session expired, `abortResumableUpload(uploadId)` removes its leftover chunks, and `beginResumableUpload` with the same explicit `id` clears them before the new session starts.
+`<bucket>.uploads` uses an `expiresAt` TTL index (`expireAfterSeconds: 0`). Pending resumable sessions expire after **7 days without a successful append**; a successful append refreshes their expiry. Active append/finalize/abort claims and direct upload/delete reservations have no TTL and must be recovered explicitly after a crash, otherwise a stale writer could resume after its lock expires. MongoDB TTL deletion is asynchronous and removes only the session document, not chunks. Known orphan chunks can be cleaned with `abortResumableUpload(uploadId)` after an exclusive cleanup claim, or by starting a new resumable upload with the same ID. Existing `createdAt` TTL indexes are migrated to `expiresAt`; legacy pending expiry is derived from their last update. The migration requires stopping/draining old GridFS writers; mixing old and new writer versions is unsafe.
+
+**Crash and migration recovery:** see [the operator runbook](./gridfs-recovery.md). Recovery is dry-run by default and requires verified process quiescence, an exact state/token check and an audit record; do not delete stale-looking reservations manually.
 
 ## Indexes
 
@@ -107,9 +110,9 @@ Retrying `finalizeResumableUpload` is idempotent while the session still exists:
 | `<bucket>.files` | `{ filename: 1, uploadDate: 1 }` | |
 | `<bucket>.files` | `{ 'metadata.sha256': 1 }` | not unique |
 | `<bucket>.chunks` | `{ files_id: 1, n: 1 }` | unique |
-| `<bucket>.uploads` | `{ createdAt: 1 }` | `expireAfterSeconds: 604800` |
+| `<bucket>.uploads` | `{ expiresAt: 1 }` | `expireAfterSeconds: 0`; pending resumable sessions set expiry for seven idle days |
 
-Reads do not create indexes. A write rejected by `writePathPolicy` does not create them either.
+Reads do not create indexes. A write rejected by `writePathPolicy` does not create them either. Every file ID is reserved atomically in `<bucket>.uploads` during upload, uploadStream, rename, and delete. A different operation using the same ID receives `GRIDFS_UPLOAD_BUSY`. `drop()` remains a bucket-wide destructive maintenance operation: stop/drain writers before calling it; the per-ID reservations do not coordinate bucket-wide drop.
 
 ## writePathPolicy
 
@@ -135,6 +138,7 @@ A blocked call throws `INVALID_OPERATION` before any GridFS write. `details[0]`
 | `GRIDFS_INTEGRITY_MISMATCH` | Bytes do not match `sha256`, or a finalized chunk has the wrong size |
 | `GRIDFS_UPLOAD_INCOMPLETE` | Finalize found gaps. `details[0].missingChunks` lists them |
 | `GRIDFS_RESUME_MISMATCH` | `fromChunk` is ahead of the contiguous prefix, or a replayed chunk differs |
+| `GRIDFS_UPLOAD_BUSY` | Another append/finalize/abort/upload/delete operation owns the ID; do not retry until it finishes or is explicitly recovered |
 | `GRIDFS_RANGE_INVALID` | `start` / `end` is not a legal half-open range |
 | `INVALID_ARGUMENT` | Filename, bucket name, chunk size, digest, metadata key, or chunk payload is invalid |
 | `INVALID_OPERATION` | `writePathPolicy` blocked the call |