fix(gridfs): operator-audited stale ownership recovery and crash matrix [WIP]

.github/workflows/gridfs-fix-validation.yml

@@ -2,14 +2,19 @@ name: GridFS Ownership Safety Validation
 
 on:
   pull_request:
-    branches: [main]
+    branches: [main, 'fix/gridfs-id-ownership-v01']
+
+concurrency:
+  group: gridfs-safety-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
 
 permissions:
   contents: read
 
 jobs:
   gridfs:
     runs-on: ubuntu-latest
+    timeout-minutes: 30
     strategy:
       fail-fast: false
       matrix:
@@ -39,4 +44,5 @@ jobs:
             .generated/test-dist/test/unit/gridfs/gridfs.test.js \
             .generated/test-dist/test/integration/gridfs/gridfs.test.js \
             .generated/test-dist/test/integration/gridfs/gridfs-scan-regressions.test.js \
-            .generated/test-dist/test/integration/gridfs/gridfs-ownership-regressions.test.js
+            .generated/test-dist/test/integration/gridfs/gridfs-ownership-regressions.test.js \
+            .generated/test-dist/test/integration/gridfs/gridfs-stale-recovery.test.js

.github/workflows/test.yml

@@ -4,11 +4,18 @@ on:
   push:
     branches: [main]
   pull_request:
-    branches: [main]
+    branches: [main, 'fix/gridfs-id-ownership-v01']
+
+# Superseded PR revisions must not accumulate long-running test jobs.
+# Cancelled prior SHAs are never accepted as evidence for the current HEAD.
+concurrency:
+  group: ci-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
 
 jobs:
   test:
     runs-on: ubuntu-latest
+    timeout-minutes: 60
     strategy:
       fail-fast: false
       matrix:
@@ -50,6 +57,7 @@ jobs:
 
   release-gate:
     runs-on: ubuntu-latest
+    timeout-minutes: 60
     steps:
       - name: Checkout
         uses: actions/checkout@v5

docs/en/gridfs-recovery.md

@@ -0,0 +1,71 @@
+# GridFS crash recovery and no-data-loss upgrade runbook (V0.1 candidate)
+
+> **Operator-only.** This is a draft recovery design, not a completed or verified release. Never run recovery against a running, paused or network-partitioned GridFS writer. **Age alone is not proof that a writer is dead.** No automated TTL-based owner unlock is permitted.
+
+## Scope and invariant
+
+The ownership registry is the per-bucket `<bucket>.uploads` collection. A reservation excludes ordinary upload, uploadStream, resumable append/finalize/abort, delete, rename and orphan cleanup for the same ObjectId. A missing file document **does not** prove its chunks are orphaned: an ordinary upload may still be writing, or GridFS driver delete may have removed the files document before chunks. Recovering while an old writer can resume can destroy new data, even with a token-CAS. The maintenance process cannot prove cluster/process shutdown on its own.
+
+The script `scripts/maintenance/gridfs-recover.cjs` uses a default read-only preview, exact state/token/timestamp checks, a persisted `recovering` ownership claim, verified preservation of committed visible chunks, and an append-only-by-intent audit collection `<bucket>.ownership_recovery_audit`. It never performs a time-based lock steal.
+
+## Deployment and rollback gates (required)
+
+1. Stop all HTTP/worker/job/cron processes that can mutate this GridFS bucket, across **all** deployments, driver versions, replica instances, queue consumers and old binaries. Disable automatic restarts and new GridFS requests. Resolve in-flight operations, check for network-partitioned/paused workers, and ensure the old writer fleet is fully drained. A mixed old/new-writer deployment is unsupported because the old version ignores the new registry.
+2. Back up `<bucket>.files`, `<bucket>.chunks`, `<bucket>.uploads`, current indexes and relevant deployment version/worker inventory. Record a timestamp, operator, target database/bucket/id, intended delete/finalize outcome, and why no writer can resume. Retain backups and audit logs outside the cluster.
+3. Deploy the new version while writes remain disabled. Run `ensureIndexes()` after verifying the old `createdAt_1` TTL and any legacy pending sessions. The migration derives `expiresAt` from the last update for eligible idle sessions, removes the legacy TTL index, and creates `expiresAt_1` with `expireAfterSeconds: 0`. TTL can already have removed sessions before migration; examine orphan chunks separately. Never treat an expiring session as safe while a legacy writer may still run.
+4. Inspect every retained owner that must be recovered. Reconcile visible file and chunks, determine its original operation and the desired post-recovery state, and record the evidence. An owner without expiry is not automatically stale. Use the script only for an owner that has been conclusively quiesced.
+5. Run dry-run and compare the proposed action with the recorded intent. To apply, supply the exact observed `status`, `updatedAt` ISO timestamp, and `ownerToken`, `activeAppend.token`, or `recovery.token` when present. Any mismatch is a stop signal. An audit document is written before a change; if a step fails the state remains `recovering` and must be re-inspected.
+6. Recheck file visibility, stored chunk count, lengths, and any relevant SHA. Complete the latest-HEAD Node 18/20/22 CI, MongoDB 7.0.37/8.0.26 targeted suite, applicable Driver 6/7 compatibility checks and release preflight before unfreezing. Verify no `recovering` or unintended active reservations remain.
+7. Only then restart new writers. Rollback to an old binary is **not safe while a new-version writer or reservation is active**; first stop/drain all new writers, reconcile operations, and assess the old binary's inability to honor `<bucket>.uploads` ownership. Preserve the migrated index configuration; do not recreate `createdAt` TTL blindly.
+
+## Manual command
+
+Use a checkout with installed dependencies and a MongoDB driver. Never pass the connection string in CLI arguments or paste credentials into an issue.
+
+```sh
+export MONSQLIZE_RECOVERY_MONGODB_URI='<MongoDB URI from the secrets manager>'
+node scripts/maintenance/gridfs-recover.cjs \
+  --db=my_database --bucket=fs \
+  --id=0123456789abcdef01234567 \
+  --expected-status=aborting \
+  --expected-updated-at=2026-11-12T02:00:00.000Z
+```
+
+This previews but does not write. Retrieve the exact observed fields from the backed-up `<bucket>.uploads` collection first. If the document contains a token, append `--expected-token=...`. **Do not copy** the example ObjectId or timestamp. Apply only after external process quiescence is verified and logged:
+
+```sh
+node scripts/maintenance/gridfs-recover.cjs \
+  --db=my_database --bucket=fs \
+  --id=0123456789abcdef01234567 \
+  --expected-status=aborting \
+  --expected-updated-at=2026-11-12T02:00:00.000Z \
+  --operator=oncall-id --reason=verified-writer-shutdown \
+  --quiesced --apply
+```
+
+For an already `recovering` owner only, verify that the previous recovery executor has stopped and add `--prior-recovery-stopped` to the apply command; this is separate from stopping ordinary GridFS writers.
+
+Every apply attempt includes an audit entry. A crash can leave a `status: recovering` record with `recovery.token`; after proving **both all writers and the previous recovery executor have stopped**, re-inspect that record and use its new status, updatedAt and recovery token to retry. Supply the additional `--prior-recovery-stopped` flag. The restart acquires a fresh recovery token via atomic CAS. Two operators with the same observed snapshot cannot perform destructive steps simultaneously, but in-flight writes by an unquiesced prior executor still cannot be fenced by MongoDB alone. Never delete an ownership record by ID alone.
+
+## State-by-state recovery contract
+
+| Observed owner | After proven quiescence | Critical preservation rule |
+|---|---|---|
+| `kind:direct,status:uploading` | When a file is visible, verify every chunk, length and any claimed SHA before releasing ownership; otherwise delete orphan chunks and release | A mismatched caller-supplied metadata SHA blocks recovery; visible-file chunks are never deleted |
+| `kind:delete,status:deleting` | Complete the interrupted delete of file and chunks before releasing ownership | No new upload may claim the same ID until both steps finish |
+| `kind:cleanup,status:cleaning` | Check visible file; delete chunks only if no file exists; release | Never delete a visible file's chunks |
+| `kind:rename,status:renaming` | Verify the resulting visible file has complete chunks, then release; missing or corrupt files remain locked | The tool does not choose or replay a rename target; the operator must inspect the intended filename |
+| `status:pending,activeAppend` | Recompute the contiguous chunk prefix, clear only the observed append claim, restore seven-day idle expiry | Existing chunks are preserved; caller replays/checks the data |
+| `status:finalizing` without file | Restore pending with expiry; retry normal finalize, which revalidates bytes | Never discard chunks to unlock finalize |
+| `status:finalizing` with file | Re-hash the **actual** ordered chunks and check expected file length, chunk size and digest before releasing session | Forged metadata SHA is not sufficient |
+| `status:aborting` without file | Delete invisible chunks then remove session | No newly created file can take ownership during recovery |
+| `status:aborting` with file | **Blocked** for human forensic review | Never silently delete or detach a visible file |
+| `status:recovering` | Resume by exact recovery token and reverified quiescence | Keep claim if any destructive step is ambiguous |
+
+### Known limitations and release criteria
+
+- MongoDB cannot revoke an already-running Node.js process's in-memory chunk write. `--quiesced` is a mandatory **external operational assertion**, not a technical fencing guarantee. For non-quiesceable workloads, a future generation-aware writer fencing design and driver integration would be needed; do **not** automate unlocking based on age.
+- A direct uploadStream `metadata.sha256` is not authenticated at write time. The tool verifies visible chunk structure and length, but does not treat this metadata as a trusted digest for ordinary uploads.
+- The recovery code is a source-checkout maintenance utility. It is not an online public `GridFSBucket` API and must not be exposed to unprivileged application callers.
+- `drop()` is bucket-wide and ignores per-ID reservations; maintenance must globally stop writers before dropping the bucket.
+- Deterministic tests must show all state paths, wrong-token and wrong-timestamp rejection, fail-closed interrupted recovery, no deletion of visible chunks, and idempotent restart after an interrupted abort/delete. **No merge until the code and this runbook pass the latest-HEAD required CI matrices and independent review.**

docs/en/gridfs.md

@@ -99,6 +99,8 @@ Retrying `finalizeResumableUpload` is idempotent while the session still exists:
 
 `<bucket>.uploads` uses an `expiresAt` TTL index (`expireAfterSeconds: 0`). Pending resumable sessions expire after **7 days without a successful append**; a successful append refreshes their expiry. Active append/finalize/abort claims and direct upload/delete reservations have no TTL and must be recovered explicitly after a crash, otherwise a stale writer could resume after its lock expires. MongoDB TTL deletion is asynchronous and removes only the session document, not chunks. Known orphan chunks can be cleaned with `abortResumableUpload(uploadId)` after an exclusive cleanup claim, or by starting a new resumable upload with the same ID. Existing `createdAt` TTL indexes are migrated to `expiresAt`; legacy pending expiry is derived from their last update. The migration requires stopping/draining old GridFS writers; mixing old and new writer versions is unsafe.
 
+**Crash and migration recovery:** see [the operator runbook](./gridfs-recovery.md). Recovery is dry-run by default and requires verified process quiescence, an exact state/token check and an audit record; do not delete stale-looking reservations manually.
+
 ## Indexes
 
 `upload`, `uploadStream` (before the driver stream is opened), `beginResumableUpload`, and `ensureIndexes` create:

docs/zh/gridfs-recovery.md

@@ -0,0 +1,62 @@
+# GridFS 故障恢复与安全升级操作规程（候选 V0.1）
+
+> **仅供运维管理员使用，当前未完成 CI 验收。** 不得对仍在运行、暂停或网络分区状态的写入进程执行恢复。所有权记录时间较久，不等于原写入进程已经终止。禁止按超时自动抢占所有权。
+
+## 不可破坏的安全约束
+
+`<bucket>.uploads` 持久化同一文件 ID 的上传、追加、完成、取消、删除、重命名和孤儿清理的互斥所有权。**文件记录不存在不代表其 chunks 可删**：上传过程中，files 文档尚未可见；删除过程中，MongoDB Driver 可能先删 files，再删 chunks。
+
+任何强制恢复必须先从集群所有节点停止并排空旧版和新版 GridFS 写入进程、队列消费者、定时任务与自动重启，确认没有仍可恢复的旧写入。恢复脚本无法在 MongoDB 层直接阻止旧进程已发出的 chunks 写入；只有外部进程停机隔离后才安全。不得混用忽略 ownership 的旧版与新版本。
+
+## 上线和回滚步骤
+
+1. 冻结所有 GridFS 写入，禁止新请求和自动重启，核查所有旧版 Writer 已排空；保留相关运行实例、进程、队列、版本和时间证据。
+2. 备份 `<bucket>.files`、`<bucket>.chunks`、`<bucket>.uploads`、索引及配置；备份应与目标集群隔离。
+3. 在冻结状态部署新版本并运行 `ensureIndexes()`。迁移需将 legacy `createdAt_1` TTL 改为 `expiresAt_1`，以最后活动时间计算普通待上传会话的七天有效期，活动所有权和无 TTL 状态不得被自动过期。迁移前就过期的旧会话可能已被 TTL 清除，需要额外核对孤儿 chunks。
+4. 对每一个需要恢复的 ID，核对上传会话状态、ownerToken/activeAppend.token/recovery.token、updatedAt、files 和 chunks 的一致性，确认目标动作并保留审计记录。
+5. **先 dry-run，后 apply**。apply 必须明确声明写入进程已停机、实际操作者、恢复原因，并提供精确原始状态、时间戳和令牌。脚本先写入审计记录，再使用条件更新取得 `recovering` 所有权，失败时保留该所有权，禁止直接删除锁。
+6. 核对实际 bytes、chunks 顺序/大小、可见性、恢复后状态，并完成当前 HEAD 的 Node 18/20/22、MongoDB 7.0.37/8.0.26、对应 Driver 6/7 和 Release Preflight 全矩阵后再恢复写入。
+7. **回滚到旧版并非直接安全**：旧版不会识别新 ownership，必须先停止并排空全部新版写入，审查所有未完成会话和残留锁，再评估可否回滚。不能简单恢复旧的 `createdAt` TTL。
+
+## 操作方式
+
+从已安装依赖的仓库源码目录执行，MongoDB URI 通过环境变量传递，不要在 shell 命令参数或 Issue 中公开凭证。
+
+```sh
+export MONSQLIZE_RECOVERY_MONGODB_URI='<从密钥管理系统读取的 MongoDB URI>'
+
+# 默认仅读取，不做变更；使用数据库里实际查询到的 ID、状态和时间
+node scripts/maintenance/gridfs-recover.cjs \
+  --db=database --bucket=fs --id=0123456789abcdef01234567 \
+  --expected-status=finalizing \
+  --expected-updated-at=2026-11-12T02:00:00.000Z
+
+# 完成所有进程隔离与审计后，才允许执行
+node scripts/maintenance/gridfs-recover.cjs \
+  --db=database --bucket=fs --id=0123456789abcdef01234567 \
+  --expected-status=finalizing \
+  --expected-updated-at=2026-11-12T02:00:00.000Z \
+  --operator=oncall-id --reason=writers-drained \
+  --quiesced --apply
+```
+
+以上 ID、时间戳仅为格式示例。如果记录存在 `ownerToken`、`activeAppend.token` 或 `recovery.token`，必须加 `--expected-token=实际令牌`。任何状态/令牌/时间戳不符都应中止，重新检查。**当 `status: recovering` 时，还必须先确认上一轮恢复执行器确实已停止，执行 apply 时加 `--prior-recovery-stopped`。**恢复续接会通过 CAS 原子轮换令牌，两个持同一旧快照的恢复进程不能同时执行删除；但仍无法隔离未停止的旧进程已经发出的 MongoDB 写入。
+
+## 按状态恢复
+
+| 状态 | 恢复动作 | 数据保护 |
+|---|---|---|
+| direct/uploading | 可见文件先验证 chunks 完整性及声明的 SHA 再解锁；没有 files 才清理孤儿 chunks | SHA 元数据不可信，但一旦与实际 bytes 不符就阻断；绝不删除可见文件 chunks |
+| delete/deleting | 继续删除 files 和 chunks，之后才释放同 ID 的所有权 | 删除未完成时禁止替换上传 |
+| cleanup/cleaning | 可见文件保留 chunks，否则清理孤儿 chunks | 先查 files |
+| rename/renaming | 先核验可见文件及完整 chunks，缺失或损坏时保留锁；运维核查文件名 | 工具不猜测目标或重新执行 rename |
+| pending + activeAppend | 根据 chunks 的连续前缀恢复 nextChunk，清除中断令牌，重新设置七天闲置过期 | 保留已写 chunks，后续需验证重传 |
+| finalizing，文件不可见 | 退回 pending，正常 finalize 重新校验 | 不删已有 chunks |
+| finalizing，文件可见 | 核对文件长度、chunk size、实际 SHA 和元数据，再清理遗留会话 | 不信任单独的元数据 SHA |
+| aborting，文件不可见 | 清理孤儿 chunks，移除会话 | 无新写入抢占 |
+| aborting，文件可见 | **阻断；人工分析** | 严禁删除可见文件 |
+| recovering | 必须确认上一轮恢复执行器也已停止，加 `--prior-recovery-stopped` 并通过 CAS 取得新恢复令牌 | 同一旧快照不能双重恢复，失败仍保留锁 |
+
+## 失败和验收边界
+
+恢复脚本位于源码 `scripts/maintenance/gridfs-recover.cjs`，不是对外公开 API。没有绝对可靠的在线“僵尸 Writer 自动判断”，因此不能靠超时解锁。必须以确定性测试覆盖异常中断后状态、错误令牌拒绝、错误时间戳拒绝、可见文件 chunks 绝不误删、finalize 重验 SHA、abort/delete 幂等续接、审计结果及两版 MongoDB。验证结果未全绿前不能把 PR #12 标记 Ready 或合入 main。

docs/zh/gridfs.md

@@ -99,6 +99,8 @@ await bucket.abortResumableUpload(session.uploadId);
 
 `<bucket>.uploads` 使用 `{ expiresAt: 1 }` TTL 索引，`expireAfterSeconds: 0`。pending 续传会话在**连续 7 天没有成功 append**时过期，每次成功 append 会刷新过期时间；正在执行的 append/finalize/abort，以及普通上传或删除的所有权记录不设置 TTL，进程异常退出后需确认旧进程停止再显式恢复，不能自动释放导致旧写入复活。MongoDB TTL 是异步清理，且只删除 session，不级联 chunks。已知 uploadId 的孤儿块由 `abortResumableUpload(uploadId)` 在取得独占清理权后回收，或复用同一 id 重新开始续传时清理。旧 `createdAt` TTL 索引会迁移为 `expiresAt`；迁移前必须停止/排空旧版 GridFS 写进程，新旧写入协议混跑不安全。
 
+**故障恢复与迁移：**参阅[运维恢复规程](./gridfs-recovery.md)。恢复默认只预览，必须在全部写入进程确实排空后，核对精确状态/令牌并留下审计记录；禁止直接删除看似过期的所有权。
+
 ## 索引
 
 `upload`、`uploadStream`（打开驱动流之前）、`beginResumableUpload` 和 `ensureIndexes` 会创建：