fix(gridfs): respect direct upload SHA provenance during recovery (MZ-19)

changelogs/unreleased.md

@@ -2,7 +2,9 @@
 
 ## Changes after v3.5.3
 
-Candidate work (not yet released): GridFS per-file ID ownership, durable append/finalize/abort state, deletion fencing, and expiresAt idle-TTL migration. These changes require a coordinated writer drain before upgrading and are pending full compatibility verification. No release version or publication is claimed. The GridFS and Search index fixes from the 2026-11-02 weekly scan are assigned to the [3.5.3 versioned notes](./v3.5.3.md); the vector search guidance and Search docs regrouping are in [3.5.2](./v3.5.2.md). Publication is established by the npm registry and immutable tag, not this file.
+Candidate work (not yet released): GridFS per-file ID ownership, durable append/finalize/abort state, deletion fencing, expiresAt idle-TTL migration, and recovery SHA provenance (trusted pre-verified upload() digest versus uploadStream metadata-only claims). These changes require a coordinated writer drain before upgrading and are pending full compatibility verification. No release version or publication is claimed. The GridFS and Search index fixes from the 2026-11-02 weekly scan are assigned to the [3.5.3 versioned notes](./v3.5.3.md); the vector search guidance and Search docs regrouping are in [3.5.2](./v3.5.2.md). Publication is established by the npm registry and immutable tag, not this file.
+
+- MZ-19 candidate fix: persist direct upload SHA provenance; recovery verifies physical chunks for every visible file, strictly checks only trusted upload()/resumable expected digests, and records metadata-only SHA mismatches as audit warnings.
 
 ## Archived preparation notes for the 3.3.2 candidate
 

docs/en/gridfs-recovery.md

@@ -45,13 +45,13 @@ node scripts/maintenance/gridfs-recover.cjs \
 
 For an already `recovering` owner only, verify that the previous recovery executor has stopped and add `--prior-recovery-stopped` to the apply command; this is separate from stopping ordinary GridFS writers.
 
-Every apply attempt includes an audit entry. A crash can leave a `status: recovering` record with `recovery.token`; after proving **both all writers and the previous recovery executor have stopped**, re-inspect that record and use its new status, updatedAt and recovery token to retry. Supply the additional `--prior-recovery-stopped` flag. The restart acquires a fresh recovery token via atomic CAS. Two operators with the same observed snapshot cannot perform destructive steps simultaneously, but in-flight writes by an unquiesced prior executor still cannot be fenced by MongoDB alone. Never delete an ownership record by ID alone.
+Every apply attempt includes an audit entry. A completed visible-file verification records actual SHA, claim provenance and warnings in that audit. A crash can leave a `status: recovering` record with `recovery.token`; after proving **both all writers and the previous recovery executor have stopped**, re-inspect that record and use its new status, updatedAt and recovery token to retry. Supply the additional `--prior-recovery-stopped` flag. The restart acquires a fresh recovery token via atomic CAS. Two operators with the same observed snapshot cannot perform destructive steps simultaneously, but in-flight writes by an unquiesced prior executor still cannot be fenced by MongoDB alone. Never delete an ownership record by ID alone.
 
 ## State-by-state recovery contract
 
 | Observed owner | After proven quiescence | Critical preservation rule |
 |---|---|---|
-| `kind:direct,status:uploading` | When a file is visible, verify every chunk, length and any claimed SHA before releasing ownership; otherwise delete orphan chunks and release | A mismatched caller-supplied metadata SHA blocks recovery; visible-file chunks are never deleted |
+| `kind:direct,status:uploading` | Verify chunk sequence, per-chunk length and total length. Enforce SHA only when the owner records a pre-verified `upload()` expected digest; otherwise record any metadata mismatch as an audit warning | `uploadStream()` and arbitrary `metadata.sha256` are untrusted claims; visible-file chunks are never deleted |
 | `kind:delete,status:deleting` | Complete the interrupted delete of file and chunks before releasing ownership | No new upload may claim the same ID until both steps finish |
 | `kind:cleanup,status:cleaning` | Check visible file; delete chunks only if no file exists; release | Never delete a visible file's chunks |
 | `kind:rename,status:renaming` | Verify the resulting visible file has complete chunks, then release; missing or corrupt files remain locked | The tool does not choose or replay a rename target; the operator must inspect the intended filename |
@@ -65,7 +65,7 @@ Every apply attempt includes an audit entry. A crash can leave a `status: recove
 ### Known limitations and release criteria
 
 - MongoDB cannot revoke an already-running Node.js process's in-memory chunk write. `--quiesced` is a mandatory **external operational assertion**, not a technical fencing guarantee. For non-quiesceable workloads, a future generation-aware writer fencing design and driver integration would be needed; do **not** automate unlocking based on age.
-- A direct uploadStream `metadata.sha256` is not authenticated at write time. The tool verifies visible chunk structure and length, but does not treat this metadata as a trusted digest for ordinary uploads.
+- Direct owner records preserve SHA provenance: `upload()` with `options.sha256` stores an expected digest that was verified before writing; `uploadStream()` and arbitrary `metadata.sha256` are explicitly unverified. Recovery always verifies visible chunk structure and length. It enforces the digest for resumable sessions and pre-verified direct `upload()` only; an unverified metadata mismatch is recorded in the audit result as a warning and does not strand a physically complete file.
 - The recovery code is a source-checkout maintenance utility. It is not an online public `GridFSBucket` API and must not be exposed to unprivileged application callers.
 - `drop()` is bucket-wide and ignores per-ID reservations; maintenance must globally stop writers before dropping the bucket.
 - Deterministic tests must show all state paths, wrong-token and wrong-timestamp rejection, fail-closed interrupted recovery, no deletion of visible chunks, and idempotent restart after an interrupted abort/delete. **No merge until the code and this runbook pass the latest-HEAD required CI matrices and independent review.**

docs/en/gridfs.md

@@ -48,7 +48,8 @@ Calling either method before `connect()` throws `NOT_CONNECTED`.
 - When `sha256` is set, the conflict key is `metadata.sha256`. The digest must be 64 lowercase hex characters. `Buffer` and `Uint8Array` payloads are hashed before any GridFS write. A mismatch throws `GRIDFS_INTEGRITY_MISMATCH` and leaves no visible file.
 - A `Readable` passed with `sha256` is buffered completely, then hashed, before any write. Large files should use the resumable API instead.
 - `skip` with `sha256` returns the newest visible file with that digest. `skip` without `sha256` returns the newest visible file with the same filename. `duplicate` always creates another file.
-- `uploadStream` stores `sha256` in metadata only. It does not check the bytes and it does not accept `ifExists`.
+- A `metadata.sha256` value supplied inside `metadata` is caller data and is not verified. To request byte verification with `upload()`, use the top-level `sha256` option.
+- `uploadStream` stores its top-level `sha256` option in metadata only. It does not check the bytes and it does not accept `ifExists`.
 
 `metadata.sha256` written by these helpers overwrites a caller-supplied `metadata.sha256`. A provided `contentType` is stored as `metadata.contentType` and overwrites a caller-supplied value of that field. `FileInfo.contentType` returns it. An older file that still has a top-level `contentType` is returned the same way.
 

docs/zh/gridfs-recovery.md

@@ -46,7 +46,7 @@ node scripts/maintenance/gridfs-recover.cjs \
 
 | 状态 | 恢复动作 | 数据保护 |
 |---|---|---|
-| direct/uploading | 可见文件先验证 chunks 完整性及声明的 SHA 再解锁；没有 files 才清理孤儿 chunks | SHA 元数据不可信，但一旦与实际 bytes 不符就阻断；绝不删除可见文件 chunks |
+| direct/uploading | 验证 chunks 连续性、每块长度和总长度；只有 owner 记录了 upload() 写入前验证过的 expected SHA 时才强制校验摘要，否则将 metadata 不一致记为审计 warning | uploadStream() 与任意 metadata.sha256 都是不可信声明；绝不删除可见文件 chunks |
 | delete/deleting | 继续删除 files 和 chunks，之后才释放同 ID 的所有权 | 删除未完成时禁止替换上传 |
 | cleanup/cleaning | 可见文件保留 chunks，否则清理孤儿 chunks | 先查 files |
 | rename/renaming | 先核验可见文件及完整 chunks，缺失或损坏时保留锁；运维核查文件名 | 工具不猜测目标或重新执行 rename |
@@ -57,6 +57,8 @@ node scripts/maintenance/gridfs-recover.cjs \
 | aborting，文件可见 | **阻断；人工分析** | 严禁删除可见文件 |
 | recovering | 必须确认上一轮恢复执行器也已停止，加 `--prior-recovery-stopped` 并通过 CAS 取得新恢复令牌 | 同一旧快照不能双重恢复，失败仍保留锁 |
 
+Direct owner 会保存摘要来源：`upload()` 的 `options.sha256` 在写入前已对完整 payload 校验，记录为可信 expected SHA；`uploadStream()` 和任意 `metadata.sha256` 明确标记为未验证。恢复时始终校验可见文件的 chunks 连续性、每块长度和总长度；只有 resumable session 或预校验的 direct upload 才强制比对 SHA。未验证 metadata 与实际内容不一致时只写入审计 warning，不会让完整文件永久卡在 recovering 状态。
+
 ## 失败和验收边界
 
 恢复脚本位于源码 `scripts/maintenance/gridfs-recover.cjs`，不是对外公开 API。没有绝对可靠的在线“僵尸 Writer 自动判断”，因此不能靠超时解锁。必须以确定性测试覆盖异常中断后状态、错误令牌拒绝、错误时间戳拒绝、可见文件 chunks 绝不误删、finalize 重验 SHA、abort/delete 幂等续接、审计结果及两版 MongoDB。验证结果未全绿前不能把 PR #12 标记 Ready 或合入 main。

docs/zh/gridfs.md

@@ -48,7 +48,8 @@ const modelBucket = model.gridfs(bucketName?: string, opts?: {
 - 提供 `sha256` 时，冲突键是 `metadata.sha256`。摘要必须是 64 位小写十六进制。`Buffer` 与 `Uint8Array` 会在任何 GridFS 写入之前完成哈希。不一致时抛出 `GRIDFS_INTEGRITY_MISMATCH`，并且不会留下可见文件。
 - 带 `sha256` 的 `Readable` 会先完整缓冲再哈希，然后才写入。大文件应改用可恢复上传。
 - 带 `sha256` 的 `skip` 返回该摘要最新的可见文件。不带 `sha256` 的 `skip` 返回同名最新可见文件。`duplicate` 总是再创建一份。
-- `uploadStream` 只把 `sha256` 写入 metadata。它不校验字节，也不接受 `ifExists`。
+- 通过 `metadata` 对象传入的 `metadata.sha256` 是调用方数据，不会被验证。`upload()` 如需校验字节，必须使用顶层 `sha256` 选项。
+- `uploadStream` 的顶层 `sha256` 只写入 metadata，不校验字节，也不接受 `ifExists`。
 
 这些助手写入的 `metadata.sha256` 会覆盖调用方自己放在 `metadata.sha256` 里的值。提供的 `contentType` 写入 `metadata.contentType`，并覆盖调用方在该字段上的值。`FileInfo.contentType` 会返回它。旧文件上残留的顶层 `contentType` 同样会返回。
 

scripts/maintenance/gridfs-recover.cjs

@@ -27,10 +27,10 @@ function toBuffer(data) {
     fail('Encountered a chunk with unsupported BSON binary data');
 }
 
-async function verifyVisibleFile(db, bucket, id, session) {
+async function verifyVisibleFile(db, bucket, id, session, owner) {
     const files = db.collection(bucket + '.files');
     const doc = await files.findOne({ _id: id });
-    if (!doc) return false;
+    if (!doc) return null;
     if (!Number.isSafeInteger(doc.length) || doc.length < 0 ||
         !Number.isSafeInteger(doc.chunkSize) || doc.chunkSize < 1) {
         fail('Visible file has invalid length/chunkSize; preserve ownership');
@@ -52,22 +52,50 @@ async function verifyVisibleFile(db, bucket, id, session) {
     if (length !== doc.length || n !== totalChunks) {
         fail('Visible file is incomplete; preserve ownership and chunks');
     }
-    const actualSha = digest.digest('hex');
+    const actualSha256 = digest.digest('hex');
+    let sha256Evidence;
     if (session) {
+        // Resumable session SHA is part of the upload protocol and is trusted
+        // only after the actual persisted chunks have been re-hashed.
         if (session.length !== doc.length || session.chunkSizeBytes !== doc.chunkSize ||
-            doc.metadata?.sha256 !== session.sha256 || actualSha !== session.sha256) {
+            doc.metadata?.sha256 !== session.sha256 || actualSha256 !== session.sha256) {
             fail('Finalizing session and visible bytes mismatch; preserve ownership and chunks');
         }
-    } else if (typeof doc.metadata?.sha256 === 'string') {
-        // UploadStream SHA is caller-supplied metadata, not proof of integrity.
-        // A mismatched claimed hash is nevertheless evidence of inconsistency:
-        // do not unlock an anomalous direct upload as if it had been verified.
-        if (!/^[0-9a-f]{64}$/i.test(doc.metadata.sha256) ||
-            actualSha !== doc.metadata.sha256.toLowerCase()) {
-            fail('Visible direct upload metadata SHA mismatch; preserve ownership and chunks');
+        sha256Evidence = {
+            sha256Policy: 'resumable-session-expected',
+            sha256Verified: true, expectedSha256: session.sha256, actualSha256, warnings: [],
+        };
+    } else {
+        const claimedSha256 = typeof owner?.claimedSha256 === 'string'
+            ? owner.claimedSha256
+            : typeof doc.metadata?.sha256 === 'string' ? doc.metadata.sha256 : undefined;
+        if (owner?.sha256Verified === true) {
+            // Only upload() records a digest that was checked against the full
+            // payload before any GridFS bytes were written.
+            if (owner.uploadMethod !== 'upload' || !/^[0-9a-f]{64}$/.test(owner.expectedSha256 || '') ||
+                owner.expectedSha256 !== doc.metadata?.sha256 || actualSha256 !== owner.expectedSha256) {
+                fail('Verified direct upload SHA provenance does not match visible bytes; preserve ownership and chunks');
+            }
+            sha256Evidence = {
+                sha256Policy: 'direct-upload-preverified',
+                sha256Verified: true, expectedSha256: owner.expectedSha256, actualSha256, warnings: [],
+            };
+        } else {
+            // uploadStream SHA and arbitrary metadata.sha256 are caller claims,
+            // not integrity guarantees. Keep the physical chunk checks above,
+            // but record a warning instead of permanently locking a complete file.
+            const warnings = [];
+            if (claimedSha256 !== undefined) {
+                if (!/^[0-9a-f]{64}$/i.test(claimedSha256)) warnings.push('metadata_sha256_invalid_unverified');
+                else if (actualSha256 !== claimedSha256.toLowerCase()) warnings.push('metadata_sha256_mismatch_unverified');
+            }
+            sha256Evidence = {
+                sha256Policy: owner?.uploadMethod === 'uploadStream' ? 'uploadStream-metadata-only' : 'unverified-metadata',
+                sha256Verified: false, claimedSha256: claimedSha256 ?? null, actualSha256, warnings,
+            };
         }
     }
-    return true;
+    return { fileLength: doc.length, chunkCount: n, ...sha256Evidence };
 }
 
 function snapshotToken(doc) {
@@ -211,6 +239,7 @@ async function recoverGridFSOwnership(input) {
         }
     }
     const held = { _id: id, status: 'recovering', 'recovery.token': recoveryToken };
+    let verification = null;
     try {
         const chunks = db.collection(bucket + '.chunks');
         const files = db.collection(bucket + '.files');
@@ -233,9 +262,9 @@ async function recoverGridFSOwnership(input) {
             // Verify before ANY deletion. Never delete chunks of a visible file
             // unless this is an explicitly quiesced, retained "deleting" owner.
             if (sourceStatus === 'finalizing') {
-                await verifyVisibleFile(db, bucket, id, doc);
+                verification = await verifyVisibleFile(db, bucket, id, doc, doc);
             } else if ((sourceStatus === 'uploading' || sourceStatus === 'renaming') && visible) {
-                await verifyVisibleFile(db, bucket, id, null);
+                verification = await verifyVisibleFile(db, bucket, id, null, doc);
             } else if (sourceStatus === 'renaming' && !visible) {
                 fail('Renaming owner has no visible file; preserve ownership for manual inspection');
             }
@@ -249,8 +278,8 @@ async function recoverGridFSOwnership(input) {
             const removed = await uploads.deleteOne(held);
             if (removed.deletedCount !== 1) fail('Lost recovery claim while releasing owner');
         }
-        await audit.updateOne({ _id: attemptId }, { $set: { outcome: 'completed', finishedAt: new Date() } });
-        return { outcome: 'completed', modified: true, plan, attemptId };
+        await audit.updateOne({ _id: attemptId }, { $set: { outcome: 'completed', finishedAt: new Date(), ...(verification ? { verification } : {}) } });
+        return { outcome: 'completed', modified: true, plan, attemptId, ...(verification ? { verification } : {}) };
     } catch (error) {
         // Deliberately retain recovering state; retry using its exact token and
         // updatedAt only after re-verifying process quiescence.

scripts/validation/doc-example-matrix.json

@@ -401,6 +401,13 @@
       "runner": "check:docs-examples",
       "status": "covered"
     },
+    {
+      "doc": "gridfs-recovery",
+      "kind": "doc-check",
+      "target": "docs/en/gridfs-recovery.md",
+      "runner": "check:docs-examples",
+      "status": "covered"
+    },
     {
       "doc": "hooks",
       "kind": "example",

src/capabilities/gridfs/operations.ts

@@ -86,7 +86,12 @@ export async function uploadFile(
     // The visible-file preflight above is not sufficient across collections or processes.
     let owner: Awaited<ReturnType<typeof claimOwner>>;
     try {
-        owner = await claimOwner(ctx, id, 'direct', 'upload');
+        owner = await claimOwner(ctx, id, 'direct', 'upload', {
+            uploadMethod: 'upload',
+            sha256Verified: options.sha256 !== undefined,
+            ...(options.sha256 !== undefined ? { expectedSha256: options.sha256 } : {}),
+            ...(typeof metadata?.sha256 === 'string' ? { claimedSha256: metadata.sha256 } : {}),
+        });
     } catch (error) {
         if (source) releaseReadable(source);
         throw error;
@@ -138,7 +143,12 @@ export function openUploadStream(ctx: GridFSContext, filename: string, opts?: Gr
         let target: GridFSDriverStream | undefined;
         let owner: Awaited<ReturnType<typeof claimOwner>> | undefined;
         try {
-            owner = await claimOwner(ctx, id, 'direct', 'uploadStream');
+            owner = await claimOwner(ctx, id, 'direct', 'uploadStream', {
+                uploadMethod: 'uploadStream',
+                // uploadStream's SHA is explicitly metadata-only; never mark it verified.
+                sha256Verified: false,
+                ...(typeof metadata?.sha256 === 'string' ? { claimedSha256: metadata.sha256 } : {}),
+            });
             // Check again after the ID is reserved, in case an older unreserved file exists.
             if (await getFile(ctx, id)) {
                 throw fileExists('uploadStream', ctx, { id, filename, sha256: options.sha256 });