diff --git a/changelogs/unreleased.md b/changelogs/unreleased.md index c0a6a01..837c6da 100644 --- a/changelogs/unreleased.md +++ b/changelogs/unreleased.md @@ -2,7 +2,9 @@ ## Changes after v3.5.3 -Candidate work (not yet released): GridFS per-file ID ownership, durable append/finalize/abort state, deletion fencing, and expiresAt idle-TTL migration. These changes require a coordinated writer drain before upgrading and are pending full compatibility verification. No release version or publication is claimed. The GridFS and Search index fixes from the 2026-11-02 weekly scan are assigned to the [3.5.3 versioned notes](./v3.5.3.md); the vector search guidance and Search docs regrouping are in [3.5.2](./v3.5.2.md). Publication is established by the npm registry and immutable tag, not this file. +Candidate work (not yet released): GridFS per-file ID ownership, durable append/finalize/abort state, deletion fencing, expiresAt idle-TTL migration, and recovery SHA provenance (trusted pre-verified upload() digest versus uploadStream metadata-only claims). These changes require a coordinated writer drain before upgrading and are pending full compatibility verification. No release version or publication is claimed. The GridFS and Search index fixes from the 2026-11-02 weekly scan are assigned to the [3.5.3 versioned notes](./v3.5.3.md); the vector search guidance and Search docs regrouping are in [3.5.2](./v3.5.2.md). Publication is established by the npm registry and immutable tag, not this file. + +- MZ-19 candidate fix: persist direct upload SHA provenance; recovery verifies physical chunks for every visible file, strictly checks only trusted upload()/resumable expected digests, and records metadata-only SHA mismatches as audit warnings. ## Archived preparation notes for the 3.3.2 candidate diff --git a/docs/en/gridfs-recovery.md b/docs/en/gridfs-recovery.md index 07915c8..5409cf5 100644 --- a/docs/en/gridfs-recovery.md +++ b/docs/en/gridfs-recovery.md @@ -45,13 +45,13 @@ node scripts/maintenance/gridfs-recover.cjs \ For an already `recovering` owner only, verify that the previous recovery executor has stopped and add `--prior-recovery-stopped` to the apply command; this is separate from stopping ordinary GridFS writers. -Every apply attempt includes an audit entry. A crash can leave a `status: recovering` record with `recovery.token`; after proving **both all writers and the previous recovery executor have stopped**, re-inspect that record and use its new status, updatedAt and recovery token to retry. Supply the additional `--prior-recovery-stopped` flag. The restart acquires a fresh recovery token via atomic CAS. Two operators with the same observed snapshot cannot perform destructive steps simultaneously, but in-flight writes by an unquiesced prior executor still cannot be fenced by MongoDB alone. Never delete an ownership record by ID alone. +Every apply attempt includes an audit entry. A completed visible-file verification records actual SHA, claim provenance and warnings in that audit. A crash can leave a `status: recovering` record with `recovery.token`; after proving **both all writers and the previous recovery executor have stopped**, re-inspect that record and use its new status, updatedAt and recovery token to retry. Supply the additional `--prior-recovery-stopped` flag. The restart acquires a fresh recovery token via atomic CAS. Two operators with the same observed snapshot cannot perform destructive steps simultaneously, but in-flight writes by an unquiesced prior executor still cannot be fenced by MongoDB alone. Never delete an ownership record by ID alone. ## State-by-state recovery contract | Observed owner | After proven quiescence | Critical preservation rule | |---|---|---| -| `kind:direct,status:uploading` | When a file is visible, verify every chunk, length and any claimed SHA before releasing ownership; otherwise delete orphan chunks and release | A mismatched caller-supplied metadata SHA blocks recovery; visible-file chunks are never deleted | +| `kind:direct,status:uploading` | Verify chunk sequence, per-chunk length and total length. Enforce SHA only when the owner records a pre-verified `upload()` expected digest; otherwise record any metadata mismatch as an audit warning | `uploadStream()` and arbitrary `metadata.sha256` are untrusted claims; visible-file chunks are never deleted | | `kind:delete,status:deleting` | Complete the interrupted delete of file and chunks before releasing ownership | No new upload may claim the same ID until both steps finish | | `kind:cleanup,status:cleaning` | Check visible file; delete chunks only if no file exists; release | Never delete a visible file's chunks | | `kind:rename,status:renaming` | Verify the resulting visible file has complete chunks, then release; missing or corrupt files remain locked | The tool does not choose or replay a rename target; the operator must inspect the intended filename | @@ -65,7 +65,7 @@ Every apply attempt includes an audit entry. A crash can leave a `status: recove ### Known limitations and release criteria - MongoDB cannot revoke an already-running Node.js process's in-memory chunk write. `--quiesced` is a mandatory **external operational assertion**, not a technical fencing guarantee. For non-quiesceable workloads, a future generation-aware writer fencing design and driver integration would be needed; do **not** automate unlocking based on age. -- A direct uploadStream `metadata.sha256` is not authenticated at write time. The tool verifies visible chunk structure and length, but does not treat this metadata as a trusted digest for ordinary uploads. +- Direct owner records preserve SHA provenance: `upload()` with `options.sha256` stores an expected digest that was verified before writing; `uploadStream()` and arbitrary `metadata.sha256` are explicitly unverified. Recovery always verifies visible chunk structure and length. It enforces the digest for resumable sessions and pre-verified direct `upload()` only; an unverified metadata mismatch is recorded in the audit result as a warning and does not strand a physically complete file. - The recovery code is a source-checkout maintenance utility. It is not an online public `GridFSBucket` API and must not be exposed to unprivileged application callers. - `drop()` is bucket-wide and ignores per-ID reservations; maintenance must globally stop writers before dropping the bucket. - Deterministic tests must show all state paths, wrong-token and wrong-timestamp rejection, fail-closed interrupted recovery, no deletion of visible chunks, and idempotent restart after an interrupted abort/delete. **No merge until the code and this runbook pass the latest-HEAD required CI matrices and independent review.** diff --git a/docs/en/gridfs.md b/docs/en/gridfs.md index d79980a..85d6d5f 100644 --- a/docs/en/gridfs.md +++ b/docs/en/gridfs.md @@ -48,7 +48,8 @@ Calling either method before `connect()` throws `NOT_CONNECTED`. - When `sha256` is set, the conflict key is `metadata.sha256`. The digest must be 64 lowercase hex characters. `Buffer` and `Uint8Array` payloads are hashed before any GridFS write. A mismatch throws `GRIDFS_INTEGRITY_MISMATCH` and leaves no visible file. - A `Readable` passed with `sha256` is buffered completely, then hashed, before any write. Large files should use the resumable API instead. - `skip` with `sha256` returns the newest visible file with that digest. `skip` without `sha256` returns the newest visible file with the same filename. `duplicate` always creates another file. -- `uploadStream` stores `sha256` in metadata only. It does not check the bytes and it does not accept `ifExists`. +- A `metadata.sha256` value supplied inside `metadata` is caller data and is not verified. To request byte verification with `upload()`, use the top-level `sha256` option. +- `uploadStream` stores its top-level `sha256` option in metadata only. It does not check the bytes and it does not accept `ifExists`. `metadata.sha256` written by these helpers overwrites a caller-supplied `metadata.sha256`. A provided `contentType` is stored as `metadata.contentType` and overwrites a caller-supplied value of that field. `FileInfo.contentType` returns it. An older file that still has a top-level `contentType` is returned the same way. diff --git a/docs/zh/gridfs-recovery.md b/docs/zh/gridfs-recovery.md index c5c6139..3ef801a 100644 --- a/docs/zh/gridfs-recovery.md +++ b/docs/zh/gridfs-recovery.md @@ -46,7 +46,7 @@ node scripts/maintenance/gridfs-recover.cjs \ | 状态 | 恢复动作 | 数据保护 | |---|---|---| -| direct/uploading | 可见文件先验证 chunks 完整性及声明的 SHA 再解锁；没有 files 才清理孤儿 chunks | SHA 元数据不可信，但一旦与实际 bytes 不符就阻断；绝不删除可见文件 chunks | +| direct/uploading | 验证 chunks 连续性、每块长度和总长度；只有 owner 记录了 upload() 写入前验证过的 expected SHA 时才强制校验摘要，否则将 metadata 不一致记为审计 warning | uploadStream() 与任意 metadata.sha256 都是不可信声明；绝不删除可见文件 chunks | | delete/deleting | 继续删除 files 和 chunks，之后才释放同 ID 的所有权 | 删除未完成时禁止替换上传 | | cleanup/cleaning | 可见文件保留 chunks，否则清理孤儿 chunks | 先查 files | | rename/renaming | 先核验可见文件及完整 chunks，缺失或损坏时保留锁；运维核查文件名 | 工具不猜测目标或重新执行 rename | @@ -57,6 +57,8 @@ node scripts/maintenance/gridfs-recover.cjs \ | aborting，文件可见 | **阻断；人工分析** | 严禁删除可见文件 | | recovering | 必须确认上一轮恢复执行器也已停止，加 `--prior-recovery-stopped` 并通过 CAS 取得新恢复令牌 | 同一旧快照不能双重恢复，失败仍保留锁 | +Direct owner 会保存摘要来源：`upload()` 的 `options.sha256` 在写入前已对完整 payload 校验，记录为可信 expected SHA；`uploadStream()` 和任意 `metadata.sha256` 明确标记为未验证。恢复时始终校验可见文件的 chunks 连续性、每块长度和总长度；只有 resumable session 或预校验的 direct upload 才强制比对 SHA。未验证 metadata 与实际内容不一致时只写入审计 warning，不会让完整文件永久卡在 recovering 状态。 + ## 失败和验收边界 恢复脚本位于源码 `scripts/maintenance/gridfs-recover.cjs`，不是对外公开 API。没有绝对可靠的在线“僵尸 Writer 自动判断”，因此不能靠超时解锁。必须以确定性测试覆盖异常中断后状态、错误令牌拒绝、错误时间戳拒绝、可见文件 chunks 绝不误删、finalize 重验 SHA、abort/delete 幂等续接、审计结果及两版 MongoDB。验证结果未全绿前不能把 PR #12 标记 Ready 或合入 main。 \ No newline at end of file diff --git a/docs/zh/gridfs.md b/docs/zh/gridfs.md index f82e31d..e08df52 100644 --- a/docs/zh/gridfs.md +++ b/docs/zh/gridfs.md @@ -48,7 +48,8 @@ const modelBucket = model.gridfs(bucketName?: string, opts?: { - 提供 `sha256` 时，冲突键是 `metadata.sha256`。摘要必须是 64 位小写十六进制。`Buffer` 与 `Uint8Array` 会在任何 GridFS 写入之前完成哈希。不一致时抛出 `GRIDFS_INTEGRITY_MISMATCH`，并且不会留下可见文件。 - 带 `sha256` 的 `Readable` 会先完整缓冲再哈希，然后才写入。大文件应改用可恢复上传。 - 带 `sha256` 的 `skip` 返回该摘要最新的可见文件。不带 `sha256` 的 `skip` 返回同名最新可见文件。`duplicate` 总是再创建一份。 -- `uploadStream` 只把 `sha256` 写入 metadata。它不校验字节，也不接受 `ifExists`。 +- 通过 `metadata` 对象传入的 `metadata.sha256` 是调用方数据，不会被验证。`upload()` 如需校验字节，必须使用顶层 `sha256` 选项。 +- `uploadStream` 的顶层 `sha256` 只写入 metadata，不校验字节，也不接受 `ifExists`。 这些助手写入的 `metadata.sha256` 会覆盖调用方自己放在 `metadata.sha256` 里的值。提供的 `contentType` 写入 `metadata.contentType`，并覆盖调用方在该字段上的值。`FileInfo.contentType` 会返回它。旧文件上残留的顶层 `contentType` 同样会返回。 diff --git a/scripts/maintenance/gridfs-recover.cjs b/scripts/maintenance/gridfs-recover.cjs index d55af79..d80f669 100644 --- a/scripts/maintenance/gridfs-recover.cjs +++ b/scripts/maintenance/gridfs-recover.cjs @@ -27,10 +27,10 @@ function toBuffer(data) { fail('Encountered a chunk with unsupported BSON binary data'); } -async function verifyVisibleFile(db, bucket, id, session) { +async function verifyVisibleFile(db, bucket, id, session, owner) { const files = db.collection(bucket + '.files'); const doc = await files.findOne({ _id: id }); - if (!doc) return false; + if (!doc) return null; if (!Number.isSafeInteger(doc.length) || doc.length < 0 || !Number.isSafeInteger(doc.chunkSize) || doc.chunkSize < 1) { fail('Visible file has invalid length/chunkSize; preserve ownership'); @@ -52,22 +52,50 @@ async function verifyVisibleFile(db, bucket, id, session) { if (length !== doc.length || n !== totalChunks) { fail('Visible file is incomplete; preserve ownership and chunks'); } - const actualSha = digest.digest('hex'); + const actualSha256 = digest.digest('hex'); + let sha256Evidence; if (session) { + // Resumable session SHA is part of the upload protocol and is trusted + // only after the actual persisted chunks have been re-hashed. if (session.length !== doc.length || session.chunkSizeBytes !== doc.chunkSize || - doc.metadata?.sha256 !== session.sha256 || actualSha !== session.sha256) { + doc.metadata?.sha256 !== session.sha256 || actualSha256 !== session.sha256) { fail('Finalizing session and visible bytes mismatch; preserve ownership and chunks'); } - } else if (typeof doc.metadata?.sha256 === 'string') { - // UploadStream SHA is caller-supplied metadata, not proof of integrity. - // A mismatched claimed hash is nevertheless evidence of inconsistency: - // do not unlock an anomalous direct upload as if it had been verified. - if (!/^[0-9a-f]{64}$/i.test(doc.metadata.sha256) || - actualSha !== doc.metadata.sha256.toLowerCase()) { - fail('Visible direct upload metadata SHA mismatch; preserve ownership and chunks'); + sha256Evidence = { + sha256Policy: 'resumable-session-expected', + sha256Verified: true, expectedSha256: session.sha256, actualSha256, warnings: [], + }; + } else { + const claimedSha256 = typeof owner?.claimedSha256 === 'string' + ? owner.claimedSha256 + : typeof doc.metadata?.sha256 === 'string' ? doc.metadata.sha256 : undefined; + if (owner?.sha256Verified === true) { + // Only upload() records a digest that was checked against the full + // payload before any GridFS bytes were written. + if (owner.uploadMethod !== 'upload' || !/^[0-9a-f]{64}$/.test(owner.expectedSha256 || '') || + owner.expectedSha256 !== doc.metadata?.sha256 || actualSha256 !== owner.expectedSha256) { + fail('Verified direct upload SHA provenance does not match visible bytes; preserve ownership and chunks'); + } + sha256Evidence = { + sha256Policy: 'direct-upload-preverified', + sha256Verified: true, expectedSha256: owner.expectedSha256, actualSha256, warnings: [], + }; + } else { + // uploadStream SHA and arbitrary metadata.sha256 are caller claims, + // not integrity guarantees. Keep the physical chunk checks above, + // but record a warning instead of permanently locking a complete file. + const warnings = []; + if (claimedSha256 !== undefined) { + if (!/^[0-9a-f]{64}$/i.test(claimedSha256)) warnings.push('metadata_sha256_invalid_unverified'); + else if (actualSha256 !== claimedSha256.toLowerCase()) warnings.push('metadata_sha256_mismatch_unverified'); + } + sha256Evidence = { + sha256Policy: owner?.uploadMethod === 'uploadStream' ? 'uploadStream-metadata-only' : 'unverified-metadata', + sha256Verified: false, claimedSha256: claimedSha256 ?? null, actualSha256, warnings, + }; } } - return true; + return { fileLength: doc.length, chunkCount: n, ...sha256Evidence }; } function snapshotToken(doc) { @@ -211,6 +239,7 @@ async function recoverGridFSOwnership(input) { } } const held = { _id: id, status: 'recovering', 'recovery.token': recoveryToken }; + let verification = null; try { const chunks = db.collection(bucket + '.chunks'); const files = db.collection(bucket + '.files'); @@ -233,9 +262,9 @@ async function recoverGridFSOwnership(input) { // Verify before ANY deletion. Never delete chunks of a visible file // unless this is an explicitly quiesced, retained "deleting" owner. if (sourceStatus === 'finalizing') { - await verifyVisibleFile(db, bucket, id, doc); + verification = await verifyVisibleFile(db, bucket, id, doc, doc); } else if ((sourceStatus === 'uploading' || sourceStatus === 'renaming') && visible) { - await verifyVisibleFile(db, bucket, id, null); + verification = await verifyVisibleFile(db, bucket, id, null, doc); } else if (sourceStatus === 'renaming' && !visible) { fail('Renaming owner has no visible file; preserve ownership for manual inspection'); } @@ -249,8 +278,8 @@ async function recoverGridFSOwnership(input) { const removed = await uploads.deleteOne(held); if (removed.deletedCount !== 1) fail('Lost recovery claim while releasing owner'); } - await audit.updateOne({ _id: attemptId }, { $set: { outcome: 'completed', finishedAt: new Date() } }); - return { outcome: 'completed', modified: true, plan, attemptId }; + await audit.updateOne({ _id: attemptId }, { $set: { outcome: 'completed', finishedAt: new Date(), ...(verification ? { verification } : {}) } }); + return { outcome: 'completed', modified: true, plan, attemptId, ...(verification ? { verification } : {}) }; } catch (error) { // Deliberately retain recovering state; retry using its exact token and // updatedAt only after re-verifying process quiescence. diff --git a/scripts/validation/doc-example-matrix.json b/scripts/validation/doc-example-matrix.json index 92ed8cb..347c1f9 100644 --- a/scripts/validation/doc-example-matrix.json +++ b/scripts/validation/doc-example-matrix.json @@ -401,6 +401,13 @@ "runner": "check:docs-examples", "status": "covered" }, + { + "doc": "gridfs-recovery", + "kind": "doc-check", + "target": "docs/en/gridfs-recovery.md", + "runner": "check:docs-examples", + "status": "covered" + }, { "doc": "hooks", "kind": "example", diff --git a/src/capabilities/gridfs/operations.ts b/src/capabilities/gridfs/operations.ts index 0847055..f425cd8 100644 --- a/src/capabilities/gridfs/operations.ts +++ b/src/capabilities/gridfs/operations.ts @@ -86,7 +86,12 @@ export async function uploadFile( // The visible-file preflight above is not sufficient across collections or processes. let owner: Awaited>; try { - owner = await claimOwner(ctx, id, 'direct', 'upload'); + owner = await claimOwner(ctx, id, 'direct', 'upload', { + uploadMethod: 'upload', + sha256Verified: options.sha256 !== undefined, + ...(options.sha256 !== undefined ? { expectedSha256: options.sha256 } : {}), + ...(typeof metadata?.sha256 === 'string' ? { claimedSha256: metadata.sha256 } : {}), + }); } catch (error) { if (source) releaseReadable(source); throw error; @@ -138,7 +143,12 @@ export function openUploadStream(ctx: GridFSContext, filename: string, opts?: Gr let target: GridFSDriverStream | undefined; let owner: Awaited> | undefined; try { - owner = await claimOwner(ctx, id, 'direct', 'uploadStream'); + owner = await claimOwner(ctx, id, 'direct', 'uploadStream', { + uploadMethod: 'uploadStream', + // uploadStream's SHA is explicitly metadata-only; never mark it verified. + sha256Verified: false, + ...(typeof metadata?.sha256 === 'string' ? { claimedSha256: metadata.sha256 } : {}), + }); // Check again after the ID is reserved, in case an older unreserved file exists. if (await getFile(ctx, id)) { throw fileExists('uploadStream', ctx, { id, filename, sha256: options.sha256 }); diff --git a/src/capabilities/gridfs/ownership.ts b/src/capabilities/gridfs/ownership.ts index caac89b..d26d531 100644 --- a/src/capabilities/gridfs/ownership.ts +++ b/src/capabilities/gridfs/ownership.ts @@ -12,14 +12,45 @@ import { chunksCollectionName, filesCollectionName, uploadsCollectionName } from export type GridFSOwnerKind = 'direct' | 'delete' | 'rename' | 'cleanup'; export interface GridFSOwner { id: ObjectId; token: string; kind: GridFSOwnerKind } +/** + * Provenance for a direct upload's optional SHA metadata. + * upload() validates its supplied sha256 against the complete payload before + * claiming ownership; uploadStream() stores caller SHA as metadata only. + */ +export interface GridFSDirectOwnerProvenance { + uploadMethod: 'upload' | 'uploadStream'; + sha256Verified: boolean; + expectedSha256?: string; + claimedSha256?: string; +} + export async function claimOwner( ctx: GridFSContext, id: ObjectId, kind: GridFSOwnerKind, operation: string, + provenance?: GridFSDirectOwnerProvenance, ): Promise { await ctx.indexes.ensure(); const token = randomUUID(); + if (kind === 'direct' && !provenance) { + throw new Error('Direct GridFS ownership requires upload SHA provenance'); + } + if (kind !== 'direct' && provenance) { + throw new Error('Only direct GridFS ownership accepts upload SHA provenance'); + } + if (provenance?.sha256Verified && ( + provenance.uploadMethod !== 'upload' || + !provenance.expectedSha256 || + !/^[0-9a-f]{64}$/.test(provenance.expectedSha256) || + provenance.claimedSha256 !== provenance.expectedSha256 + )) { + throw new Error('Verified direct GridFS ownership requires a consistent pre-verified upload SHA-256'); + } + if (provenance && !provenance.sha256Verified && provenance.expectedSha256 !== undefined) { + throw new Error('Unverified direct GridFS ownership cannot carry a trusted expected SHA-256'); + } try { await ctx.db.collection(uploadsCollectionName(ctx.bucketName)).insertOne({ _id: id, kind, status: kind === 'direct' ? 'uploading' : kind === 'delete' ? 'deleting' : kind === 'rename' ? 'renaming' : 'cleaning', + ...(provenance ?? {}), ownerToken: token, createdAt: new Date(), updatedAt: new Date(), }); } catch (error) { diff --git a/src/capabilities/gridfs/types.ts b/src/capabilities/gridfs/types.ts index cbc6321..baaf27f 100644 --- a/src/capabilities/gridfs/types.ts +++ b/src/capabilities/gridfs/types.ts @@ -16,6 +16,7 @@ export interface GridFSFileInfo { chunkSize: number; uploadDate: Date; contentType?: string; + /** Arbitrary metadata; metadata.sha256 is not verified. Use the top-level sha256 option for byte verification. */ metadata?: Record; } @@ -30,6 +31,7 @@ export interface GridFSUploadOptions { chunkSizeBytes?: number; } +/** uploadStream's top-level sha256 is metadata-only and does not validate the stream bytes. */ export type GridFSUploadStreamOptions = Omit; export type GridFSIdOrName = ObjectId | { filename: string; revision?: number }; diff --git a/test/integration/gridfs/gridfs-ownership-regressions.test.ts b/test/integration/gridfs/gridfs-ownership-regressions.test.ts index 1cf3a9c..f2c45ff 100644 --- a/test/integration/gridfs/gridfs-ownership-regressions.test.ts +++ b/test/integration/gridfs/gridfs-ownership-regressions.test.ts @@ -11,6 +11,18 @@ function sha(data: Buffer): string { return createHash('sha256').update(data).digest('hex'); } +/** + * GridFS Driver 6 deletes chunks with { files_id: id }, while Driver 7.8+ + * uses { files_id: { $eq: id } } for an exact ID match. Match both without + * interpreting arbitrary filter objects as valid IDs. + */ +function matchesDeletedFileId(value: unknown, expected: ObjectId): boolean { + const candidate = value && typeof value === 'object' && '$eq' in value + ? (value as { $eq: unknown }).$eq + : value; + return candidate instanceof ObjectId && candidate.equals(expected); +} + /** * Deterministic regression tests for MZ-15 / MZ-16 / MZ-18. @@ -181,6 +193,17 @@ describe('GridFS MZ-15 / MZ-16 / MZ-18 ownership regressions', () => { }); }); + it('recognizes Driver 6 bare and Driver 7.8 exact-$eq chunk deletion filters', () => { + const id = new ObjectId(); + const other = new ObjectId(); + assert.equal(matchesDeletedFileId(id, id), true); + assert.equal(matchesDeletedFileId({ $eq: id }, id), true); + assert.equal(matchesDeletedFileId(other, id), false); + assert.equal(matchesDeletedFileId({ $eq: other }, id), false); + assert.equal(matchesDeletedFileId({ $in: [id] }, id), false); + assert.equal(matchesDeletedFileId(undefined, id), false); + }); + it('MZ-18 prevents delete removing chunks from a replacement upload with the same id', { timeout: 30_000 }, async () => { await withRuntime('gridfs_mz18_delete_upload_race', async (runtime, db) => { const bucket = runtime.gridfs('b', { chunkSizeBytes: 4 }); @@ -201,8 +224,8 @@ describe('GridFS MZ-15 / MZ-16 / MZ-18 ownership regressions', () => { (Collection.prototype as any).deleteMany = async function (this: Collection, ...args: unknown[]) { const filter = args[0] as Record; - const filesId = filter.files_id as ObjectId | undefined; - if (armed && this.collectionName === 'b.chunks' && filesId?.equals(id)) { + if (armed && this.collectionName === 'b.chunks' && + matchesDeletedFileId(filter.files_id, id)) { armed = false; reached(); await gate; @@ -213,8 +236,17 @@ describe('GridFS MZ-15 / MZ-16 / MZ-18 ownership regressions', () => { let deleting: Promise | undefined; try { // GridFSBucket.delete removes files first, then deletes chunks. - deleting = bucket.delete(id); - await hit; + const pendingDelete: Promise = bucket.delete(id); + deleting = pendingDelete; + // Fail immediately if the driver completes without reaching the + // chunk-delete barrier. Never await a gate that cannot fire. + await Promise.race([ + hit, + pendingDelete.then(() => { + throw new Error('GridFS delete completed without reaching the exact-ID chunks barrier'); + }), + ]); + assert.equal(armed, false, 'the delete must be paused at the chunk-delete barrier'); assert.equal(await bucket.getFile(id), null); // The delete owns the ID until both files and chunks have gone. diff --git a/test/integration/gridfs/gridfs-stale-recovery.test.ts b/test/integration/gridfs/gridfs-stale-recovery.test.ts index 4a19ad8..0f95c7f 100644 --- a/test/integration/gridfs/gridfs-stale-recovery.test.ts +++ b/test/integration/gridfs/gridfs-stale-recovery.test.ts @@ -81,17 +81,35 @@ describe('GridFS operator recovery: deterministic post-crash state and chunk saf }); }); - it('preserves committed direct upload bytes while releasing a stranded owner', async () => { + it('preserves a pre-verified upload() SHA when the process crashes before owner release', { timeout: 30_000 }, async () => { await withRuntime(async (bucket, db) => { const id = new ObjectId(); const data = Buffer.from('GOODGOOD'); - await bucket.upload('committed.bin', data, { id, sha256: digest(data) }); - const now = new Date(); - await db.collection('b.uploads').insertOne({ - _id: id, kind: 'direct', status: 'uploading', ownerToken: 'post-commit', - createdAt: now, updatedAt: now, - }); - await recover(db, id); + const hash = digest(data); + const original = Collection.prototype.deleteOne; + let intercepted = false; + (Collection.prototype as any).deleteOne = async function (this: Collection, ...args: unknown[]) { + const filter = args[0] as Record; + if (!intercepted && this.collectionName === 'b.uploads' && + filter?._id?.equals(id) && typeof filter?.ownerToken === 'string') { + intercepted = true; + throw new Error('simulated process interruption before owner release'); + } + return (original as any).apply(this, args); + }; + try { + await bucket.upload('committed.bin', data, { id, sha256: hash }); + } finally { + Collection.prototype.deleteOne = original; + } + const owner = await db.collection('b.uploads').findOne({ _id: id }); + assert.equal(owner?.uploadMethod, 'upload'); + assert.equal(owner?.sha256Verified, true); + assert.equal(owner?.expectedSha256, hash); + assert.equal(owner?.claimedSha256, hash); + const receipt = await recover(db, id); + assert.equal(receipt.verification.sha256Policy, 'direct-upload-preverified'); + assert.equal(receipt.verification.sha256Verified, true); assert.equal(await db.collection('b.uploads').countDocuments({ _id: id }), 0); assert.equal(await db.collection('b.chunks').countDocuments({ files_id: id }), 2); assert.deepEqual(await bucket.download(id), data); @@ -365,25 +383,86 @@ describe('GridFS operator recovery: deterministic post-crash state and chunk saf }); - it('does not release a visible direct upload with inconsistent claimed SHA', async () => { + it('keeps a trusted upload SHA strict when recovering a visible direct file', async () => { await withRuntime(async (bucket, db) => { const id = new ObjectId(); const actual = Buffer.from('EVIL'); + const expectedSha256 = digest(Buffer.from('GOOD')); await bucket.upload('direct.bin', actual, { id }); await db.collection('b.files').updateOne({ _id: id }, { - $set: { 'metadata.sha256': digest(Buffer.from('GOOD')) }, + $set: { 'metadata.sha256': expectedSha256 }, }); const now = new Date(); await db.collection('b.uploads').insertOne({ _id: id, kind: 'direct', status: 'uploading', ownerToken: 'crash', + uploadMethod: 'upload', sha256Verified: true, + expectedSha256, claimedSha256: expectedSha256, createdAt: now, updatedAt: now, }); - await assert.rejects(recover(db, id), /metadata SHA mismatch/); + await assert.rejects(recover(db, id), /Verified direct upload SHA provenance/); assert.equal((await db.collection('b.uploads').findOne({ _id: id }))?.status, 'recovering'); + assert.equal(await db.collection('b.chunks').countDocuments({ files_id: id }), 1); assert.deepEqual(await bucket.download(id), actual); }); }); + it('recovers a complete uploadStream with mismatching metadata-only SHA and audits a warning', { timeout: 30_000 }, async () => { + await withRuntime(async (bucket, db) => { + const id = new ObjectId(); + const actual = Buffer.from('EVIL'); + const claimedSha256 = digest(Buffer.from('GOOD')); + const original = Collection.prototype.deleteOne; + let intercepted = false; + (Collection.prototype as any).deleteOne = async function (this: Collection, ...args: unknown[]) { + const filter = args[0] as Record; + if (!intercepted && this.collectionName === 'b.uploads' && + filter?._id?.equals(id) && typeof filter?.ownerToken === 'string') { + intercepted = true; + throw new Error('simulated process interruption before direct owner release'); + } + return (original as any).apply(this, args); + }; + try { + const writable = bucket.uploadStream('stream.bin', { id, sha256: claimedSha256 }); + writable.end(actual); + const uploaded = await writable.done; + assert.equal(String(uploaded.id), String(id)); + } finally { + Collection.prototype.deleteOne = original; + } + const stranded = await db.collection('b.uploads').findOne({ _id: id }); + assert.equal(stranded?.uploadMethod, 'uploadStream'); + assert.equal(stranded?.sha256Verified, false); + assert.equal(stranded?.claimedSha256, claimedSha256); + + const receipt = await recover(db, id); + assert.equal(receipt.outcome, 'completed'); + assert.equal(receipt.verification.sha256Policy, 'uploadStream-metadata-only'); + assert.equal(receipt.verification.sha256Verified, false); + assert.equal(receipt.verification.actualSha256, digest(actual)); + assert.equal(receipt.verification.claimedSha256, claimedSha256); + assert.deepEqual(receipt.verification.warnings, ['metadata_sha256_mismatch_unverified']); + assert.deepEqual(await bucket.download(id), actual); + assert.equal(await db.collection('b.chunks').countDocuments({ files_id: id }), 1); + const audit = await db.collection('b.ownership_recovery_audit').findOne({ fileId: id, outcome: 'completed' }); + assert.equal(audit?.verification?.sha256Policy, 'uploadStream-metadata-only'); + assert.deepEqual(audit?.verification?.warnings, ['metadata_sha256_mismatch_unverified']); + }); + }); + + it('rejects upload() SHA mismatch before creating files, chunks or an ownership record', async () => { + await withRuntime(async (bucket, db) => { + const id = new ObjectId(); + const actual = Buffer.from('EVIL'); + const expectedSha256 = digest(Buffer.from('GOOD')); + await assert.rejects(bucket.upload('bad.bin', actual, { id, sha256: expectedSha256 }), + (error: unknown) => (error as { code?: string }).code === 'GRIDFS_INTEGRITY_MISMATCH'); + assert.equal(await db.collection('b.files').countDocuments({ _id: id }), 0); + assert.equal(await db.collection('b.chunks').countDocuments({ files_id: id }), 0); + assert.equal(await db.collection('b.uploads').countDocuments({ _id: id }), 0); + }); + }); + it('releases a quiesced rename only after checking a visible complete file', async () => { await withRuntime(async (bucket, db) => { const id = new ObjectId(); @@ -434,4 +513,38 @@ describe('GridFS operator recovery: deterministic post-crash state and chunk saf }); }); + it('treats upload() metadata.sha256 without options.sha256 as an unverified claim', { timeout: 30_000 }, async () => { + await withRuntime(async (bucket, db) => { + const id = new ObjectId(); + const actual = Buffer.from('EVIL'); + const claimedSha256 = digest(Buffer.from('GOOD')); + const original = Collection.prototype.deleteOne; + let intercepted = false; + (Collection.prototype as any).deleteOne = async function (this: Collection, ...args: unknown[]) { + const filter = args[0] as Record; + if (!intercepted && this.collectionName === 'b.uploads' && + filter?._id?.equals(id) && typeof filter?.ownerToken === 'string') { + intercepted = true; + throw new Error('simulated process interruption before owner release'); + } + return (original as any).apply(this, args); + }; + try { + await bucket.upload('metadata.bin', actual, { id, metadata: { sha256: claimedSha256 } }); + } finally { + Collection.prototype.deleteOne = original; + } + const owner = await db.collection('b.uploads').findOne({ _id: id }); + assert.equal(owner?.uploadMethod, 'upload'); + assert.equal(owner?.sha256Verified, false); + assert.equal(owner?.claimedSha256, claimedSha256); + const receipt = await recover(db, id); + assert.equal(receipt.verification.sha256Policy, 'unverified-metadata'); + assert.equal(receipt.verification.sha256Verified, false); + assert.deepEqual(receipt.verification.warnings, ['metadata_sha256_mismatch_unverified']); + assert.deepEqual(await bucket.download(id), actual); + assert.equal(await db.collection('b.uploads').countDocuments({ _id: id }), 0); + }); + }); + }); diff --git a/types/gridfs.d.ts b/types/gridfs.d.ts index e412588..e01bf53 100644 --- a/types/gridfs.d.ts +++ b/types/gridfs.d.ts @@ -18,6 +18,7 @@ export interface GridFSFileInfo { uploadDate: Date; /** Present when the file was stored with a content type. */ contentType?: string; + /** Arbitrary metadata. metadata.sha256 is not verified; use the top-level sha256 option to request byte verification. */ metadata?: Record; } @@ -56,7 +57,7 @@ export interface GridFSUploadOptions { chunkSizeBytes?: number; } -/** `uploadStream` writes `sha256` into metadata without validating bytes, and does not support `ifExists`. */ +/** `uploadStream` writes the top-level `sha256` into metadata without validating bytes, and does not support `ifExists`. */ export interface GridFSUploadStreamOptions extends Omit {} /**