Skip to content

Commit f0cc16e

Browse files
fix(service-storage): presigned S3 PUT URLs no longer carry the CRC32 of an empty body (#21164)
Fixes #21147 Clause-②: no ## What changed The S3 adapter's `S3Client` now sets `requestChecksumCalculation` and `responseChecksumValidation` to `WHEN_REQUIRED` (`packages/services/service-storage/src/s3-storage-adapter.ts`, `getClient()`). Before, the SDK default (`WHEN_SUPPORTED`) put `x-amz-checksum-crc32=AAAAAA==` (the CRC32 of an empty body) and `x-amz-sdk-checksum-algorithm=CRC32` into the signed query of every presigned PUT, because a presign prepares the command before any byte exists. A store that enforces a query-signed checksum against the uploaded body would refuse every browser upload while server-side `upload()` kept working. Landing is exactly the card's file surface: the producer is this adapter, no other package builds an `S3Client`. | File | Change | |---|---| | `packages/services/service-storage/src/s3-storage-adapter.ts` | the two client options plus the reason | | `packages/services/service-storage/src/s3-storage-adapter.presign-checksum.test.ts` | new offline pin, real SDK | | `.changeset/21147-s3-presign-no-checksum.md` | `patch` for `@objectstack/service-storage` (not private; `dist` ships the option, grep hit in `dist/index.js` and `dist/index.cjs`) | 3 files, +164 / -1. ## Evidence **Red before, green after.** The new test drives the real `S3Client` and the real presigner, fully offline: fake credentials, fixed endpoint `http://127.0.0.1:9000`, no socket opened. A test that mocks `@aws-sdk/client-s3` (as `s3-storage-adapter.key-prefix.test.ts` does) cannot see this defect, because the checksum is stamped by the SDK's own middleware. - On `origin/main`'s adapter (base `70dae533`), the new file: 2 failed. - upload case: `expected [ 'x-amz-checksum-crc32', …(1) ] to deeply equal []`, received `x-amz-checksum-crc32`, `x-amz-sdk-checksum-algorithm`. - download case: `expected [ 'x-amz-checksum-mode' ] to deeply equal []`. - With the fix: `Tests 2 passed (2)`; whole package `pnpm --filter @objectstack/service-storage test`: `Test Files 41 passed (41)`, `Tests 629 passed (629)`; `typecheck` (including the test-layer check) exit 0. **Ablation of the pin** (committed fix at HEAD, each leg through `scripts/ablation-replace.mjs`: anchor hit once, blob hash moved, restore proven by blob equal to HEAD and an empty `git diff HEAD`). The subject resolves to `src/`, not a dependency's `dist/`, so no rebuild leg applies. | Removed | upload case | download case | |---|---|---| | both options | red (`crc32`, `sdk-checksum-algorithm`) | red (`checksum-mode`) | | `requestChecksumCalculation` only | red | red (`crc32` on the GET URL) | | `responseChecksumValidation` only | green | red (`checksum-mode`) | **The two options are a pair.** Presigning with `@aws-sdk/client-s3` 3.1090.0 under each combination: | request / response | PUT URL | GET URL | |---|---|---| | default / default | crc32, sdk-checksum-algorithm | checksum-mode | | WHEN_REQUIRED / default | none | checksum-mode | | default / WHEN_REQUIRED | crc32, sdk-checksum-algorithm | crc32 (worse than default) | | WHEN_REQUIRED / WHEN_REQUIRED | none | none | Setting only the response option puts a CRC32 on a GET URL that had none, so the pair is set together and the test header records the table. **What changes for `upload()` and the other doors** (measured against a loopback server over both plain HTTP and a self-signed HTTPS listener, same result): a server-side `PutObject` with a Buffer body and an `UploadPart` no longer send `x-amz-sdk-checksum-algorithm` / `x-amz-checksum-crc32`; `x-amz-content-sha256` (the SigV4 payload hash) is identical before and after, and the request still succeeds. A `GetObject` no longer sends `x-amz-checksum-mode: ENABLED`, so `download()` stops asking for and validating a response checksum. `getPresignedDownload` loses the `x-amz-checksum-mode=ENABLED` query parameter. This is the pre-flexible-checksum behaviour; the SDK's default added it. None of the nine commands the adapter issues is checksum-required: in `@aws-sdk/client-s3` 3.1090.0 `requestChecksumRequired: true` is bound to bucket-config, ACL, tagging, retention, `DeleteObjects` and similar commands, and `PutObject`, `GetObject`, `DeleteObject`, `HeadObject`, `ListObjectsV2` and the four multipart commands are not among them. Peer range is `^3.0.0`: SDKs older than the flexible-checksum release ignore the two options and never stamped a checksum. ## Local verification (HEAD `33e30baf`) - Dependency closure built first: `pnpm --workspace-concurrency=2 --filter '@objectstack/service-storage^...' build` exit 0 (direction: upstream dependencies of this package, the closure the test imports through `dist`). - Gates: `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands` derived 63 commands, each run with its exit code recorded; `--ran` reconciliation: `63 derived famil(ies) accounted for — 61 run, 2 NOT-MEASURED (2 DERIVED from a recorded exit 3)`. 61 exited 0. - NOT MEASURED: `pnpm check:dual-build-cjs-loads` and `pnpm check:i18n`, both exit 3, `PREREQUISITE NOT MET` (they read built output of dozens of packages outside this closure; CI builds them). Substitute for the first, narrower than the gate: `pnpm --filter @objectstack/service-storage build` exit 0 and `node -e "require('./dist/index.cjs')"` loads, `S3StorageAdapter` is a function. The diff touches no translation, extract config or metadata form, so `check:i18n`'s population is not moved by it, but that is a reading of the diff, not a measurement. - One turbo-driven gate appended its managed block to `AGENTS.md`; restored with `git checkout HEAD -- AGENTS.md`, never staged. ## Optional live leg Not done. The repo has no container pattern for an enforcing store: no testcontainers or localstack dependency and no MinIO service in any workflow (`git grep` hits for MinIO are config prose and settings labels only). Whether AWS S3 or MinIO enforce a query-signed checksum stays NOT MEASURED, as on the card; this change removes the dependence on the answer. ## Acceptance notes - Observation, not filed: the option-pair coupling in the table above is an SDK behaviour; it is recorded only in the test header and the adapter comment, so a future SDK bump that changes it surfaces as the upload or download case going red. - The new test file is deliberately a single self-contained file with no mock of the SDK; it does not replace the mocked command-level coverage in `s3-storage-adapter.key-prefix.test.ts`. --- _Generated by [Claude Code](https://claude.ai/code/session_01DiCSbmJrkzNhuEAier4VoJ)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent bafb8c9 commit f0cc16e

3 files changed

Lines changed: 164 additions & 1 deletion

File tree

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
---
2+
'@objectstack/service-storage': patch
3+
---
4+
5+
fix(service-storage): presigned S3 uploads no longer bake the CRC32 of an empty body into the signed URL (#21147)
6+
7+
Clause-②: no
8+
9+
`S3StorageAdapter.getPresignedUpload()` handed the browser a URL carrying
10+
`x-amz-sdk-checksum-algorithm=CRC32&x-amz-checksum-crc32=AAAAAA==`. `AAAAAA==`
11+
is the CRC32 of an empty body: the AWS SDK's default (`WHEN_SUPPORTED`) stamps a
12+
checksum into every `PutObjectCommand` it prepares, and a presign prepares the
13+
command before any byte exists. A store that enforces a query-signed checksum
14+
against the uploaded body refuses every presigned browser upload with a checksum
15+
mismatch, while server-side `upload()` keeps working. Hosted R2 does not enforce
16+
it, which is why this stayed quiet; self-hosted deployments on other
17+
S3-compatible stores may.
18+
19+
The adapter's `S3Client` now sets `requestChecksumCalculation` and
20+
`responseChecksumValidation` to `WHEN_REQUIRED`. None of the commands the adapter
21+
issues is checksum-required, so the presigned PUT URL carries no `x-amz-checksum-*`
22+
or `x-amz-sdk-checksum-algorithm` parameter, and a presigned GET URL no longer
23+
carries `x-amz-checksum-mode=ENABLED`.
24+
25+
Two server-side effects of the same setting, measured against a loopback server:
26+
`upload()` and multipart `uploadChunk()` no longer send the client-computed
27+
`x-amz-checksum-crc32` header (the SigV4 `x-amz-content-sha256` payload hash is
28+
unchanged), and `download()` no longer asks the store for, or validates, a
29+
response checksum. Both were added only by the SDK's default change; the older
30+
behaviour is restored. SDKs older than the flexible-checksum release ignore the
31+
two options.
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* Presigned URLs carry no flexible-checksum parameters (#21147).
5+
*
6+
* ## What is being asserted, and why it needs the REAL SDK
7+
*
8+
* Since the AWS SDK v3 flexible-checksum default (`WHEN_SUPPORTED`), a client
9+
* built without `requestChecksumCalculation` stamps a CRC32 into every
10+
* `PutObjectCommand` it prepares — and a presign prepares the command with NO
11+
* body, so the value it bakes into the signed query is the CRC32 of the EMPTY
12+
* body (`x-amz-checksum-crc32=AAAAAA==`, plus `x-amz-sdk-checksum-algorithm=
13+
* CRC32`). The browser then PUTs real bytes to that URL. A store that enforces
14+
* a query-signed checksum against the uploaded body refuses every one of them
15+
* with a checksum mismatch, while server-side `upload()` — which does compute
16+
* the checksum over real bytes — keeps working: "presign 200, upload never
17+
* lands".
18+
*
19+
* The defect lives in the SDK's own request pipeline, not in anything the
20+
* adapter writes itself, so a test that mocks `@aws-sdk/client-s3` (as
21+
* `s3-storage-adapter.key-prefix.test.ts` does, deliberately, to watch the
22+
* commands) can never see it: the fake `S3Client` has no checksum middleware.
23+
* This file therefore drives the REAL `S3Client` and the REAL presigner — still
24+
* fully offline, because presigning is pure local computation over the fake
25+
* credentials and the fixed endpoint below; nothing here opens a socket.
26+
*
27+
* ## The two options are a pair, and the cases below hold both halves
28+
*
29+
* `requestChecksumCalculation` governs the upload URL (the card). The adapter
30+
* also sets `responseChecksumValidation`, and its observable footprint is on
31+
* the download URL: the default bakes `x-amz-checksum-mode=ENABLED` into a
32+
* presigned GET, which asks the store to answer with checksum headers.
33+
*
34+
* They cannot be set one at a time. Measured on `@aws-sdk/client-s3` 3.1090.0,
35+
* presigning with each combination (query names that start `x-amz-checksum` or
36+
* `x-amz-sdk-checksum`):
37+
*
38+
* request / response PUT URL GET URL
39+
* default / default crc32, sdk-...-algorithm mode
40+
* REQUIRED / default (none) mode
41+
* default / REQUIRED crc32, sdk-...-algorithm crc32 <- worse than default
42+
* REQUIRED / REQUIRED (none) (none)
43+
*
44+
* so setting only the response option would put a CRC32 on a GET URL that had
45+
* none. The upload case goes red when the request option is dropped, the
46+
* download case when either is.
47+
*/
48+
49+
import { describe, it, expect } from 'vitest';
50+
import { S3StorageAdapter } from './s3-storage-adapter.js';
51+
52+
const ENDPOINT = 'http://127.0.0.1:9000';
53+
const BUCKET = 'test-bucket';
54+
55+
/**
56+
* Every query-string name the SDK's flexible-checksum feature can put on a
57+
* presigned URL: the `x-amz-checksum-*` family (`-crc32`, `-sha256`, `-mode`,
58+
* ...) and the `x-amz-sdk-checksum-algorithm` selector that travels with it.
59+
* SigV4 query parameter names are case-insensitive on the wire, so the match is.
60+
*/
61+
const CHECKSUM_PARAM = /^x-amz-(checksum-|sdk-checksum-algorithm$)/i;
62+
63+
function checksumParams(url: string): string[] {
64+
return [...new URL(url).searchParams.keys()].filter((name) => CHECKSUM_PARAM.test(name));
65+
}
66+
67+
function makeAdapter(): S3StorageAdapter {
68+
return new S3StorageAdapter({
69+
bucket: BUCKET,
70+
region: 'us-east-1',
71+
keyPrefix: null,
72+
endpoint: ENDPOINT,
73+
forcePathStyle: true,
74+
accessKeyId: 'AKIAFAKEFAKEFAKEFAKE',
75+
secretAccessKey: 'fake-secret-access-key',
76+
});
77+
}
78+
79+
describe('S3StorageAdapter presigned URLs carry no flexible-checksum parameters', () => {
80+
it('getPresignedUpload: the signed PUT URL has no x-amz-checksum-* / x-amz-sdk-checksum-algorithm parameter', async () => {
81+
const desc = await makeAdapter().getPresignedUpload('uploads/a.txt', 300, {
82+
contentType: 'text/plain',
83+
metadata: { owner: 'u1' },
84+
});
85+
86+
// The URL is a real presigned PUT for THIS bucket/key at THIS endpoint —
87+
// without this the absence below could be the absence of any URL at all.
88+
const url = new URL(desc.uploadUrl);
89+
expect(url.origin).toBe(ENDPOINT);
90+
expect(url.pathname).toBe(`/${BUCKET}/uploads/a.txt`);
91+
expect(url.searchParams.get('X-Amz-Algorithm')).toBe('AWS4-HMAC-SHA256');
92+
expect(url.searchParams.get('X-Amz-Expires')).toBe('300');
93+
expect(url.searchParams.get('X-Amz-Signature')).toMatch(/^[0-9a-f]{64}$/);
94+
expect(desc.method).toBe('PUT');
95+
96+
expect(checksumParams(desc.uploadUrl)).toEqual([]);
97+
});
98+
99+
it('getPresignedDownload / getSignedUrl: the signed GET URL has no x-amz-checksum-mode parameter', async () => {
100+
const adapter = makeAdapter();
101+
const desc = await adapter.getPresignedDownload('uploads/a.txt', 300, {
102+
contentType: 'text/plain',
103+
filename: 'a.txt',
104+
});
105+
106+
const url = new URL(desc.downloadUrl);
107+
expect(url.origin).toBe(ENDPOINT);
108+
expect(url.pathname).toBe(`/${BUCKET}/uploads/a.txt`);
109+
expect(url.searchParams.get('X-Amz-Signature')).toMatch(/^[0-9a-f]{64}$/);
110+
111+
expect(checksumParams(desc.downloadUrl)).toEqual([]);
112+
expect(checksumParams(await adapter.getSignedUrl('uploads/a.txt', 300))).toEqual([]);
113+
});
114+
});

‎packages/services/service-storage/src/s3-storage-adapter.ts‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -230,7 +230,25 @@ export class S3StorageAdapter implements IStorageService {
230230
);
231231
}
232232
const { S3Client } = s3Mod;
233-
const clientOpts: any = { region: this.region };
233+
const clientOpts: any = {
234+
region: this.region,
235+
// Flexible checksums only where the operation REQUIRES one. The SDK
236+
// default (`WHEN_SUPPORTED`) stamps a CRC32 into every PutObject it
237+
// prepares, and a presign prepares the command with NO body — so the
238+
// signed upload URL would bake in `x-amz-checksum-crc32=AAAAAA==`
239+
// (the CRC32 of an empty body) and `x-amz-sdk-checksum-algorithm`. A
240+
// store that enforces a query-signed checksum then refuses every
241+
// browser PUT with a checksum mismatch, while server-side `upload()`
242+
// keeps working: presign answers 200 and the object never lands.
243+
// None of the commands this adapter issues is checksum-required.
244+
// `responseChecksumValidation` is the other half: it is what bakes
245+
// `x-amz-checksum-mode=ENABLED` into a presigned GET, and what makes
246+
// `download()` validate a response checksum. The two are set TOGETHER
247+
// on purpose: the response option alone puts a CRC32 on the GET URL.
248+
// Pinned in `s3-storage-adapter.presign-checksum.test.ts`.
249+
requestChecksumCalculation: 'WHEN_REQUIRED',
250+
responseChecksumValidation: 'WHEN_REQUIRED',
251+
};
234252
if (this.endpoint) clientOpts.endpoint = this.endpoint;
235253
if (this.forcePathStyle) clientOpts.forcePathStyle = true;
236254
if (this.options.accessKeyId && this.options.secretAccessKey) {

0 commit comments

Comments
 (0)