Repository navigation
Conversation
Replace minio-go with aws-sdk-go-v2 for S3 uploads and fix the issue that originally motivated the move to minio-go: since service/s3 v1.73 the SDK sends flexible checksums (CRC32, trailing checksums) by default, which many S3-compatible providers reject. With a custom endpoint, checksums are now only sent when the operation requires them, including in the transfer manager which does not inherit the client setting. - restore the full AWS credential chain (profiles, assume role, SSO, credential_process), as documented in the README - upload with feature/s3/transfermanager (multipart for large files) - USE_PATH_STYLE now defaults to path-style with a custom endpoint, matching what minio-go did for non-AWS hosts, and "false" selects virtual-hosted-style, which was not possible before - ignore blank and space-padded age recipient keys - bump dependencies (talos v1.14.5, cosi runtime v1.16.3, age v1.3.2, compress v1.20.3) and drop minio-go - integration test: use versitygw instead of the minio/minio image, which is no longer available on Docker Hub, drop the hostDNS patch rejected by Talos 1.14 and read the container IP from the bridge network (empty NetworkSettings.IPAddress on recent Docker) - add unit tests for the S3 client and GetServiceConfig Signed-off-by: Cedric Grard <cedric.grard@eurofiber.com>
cgrard
force-pushed
the
feat/aws-sdk-s3
branch
from
October 27, 2026 08:32
3f269c0 to
8051cab
Compare
minio-go silently fell back to us-east-1 when no region was set, the AWS SDK does not: without AWS_REGION, uploads to an S3-compatible endpoint failed with "A region must be set when sending requests to S3", after the snapshot was taken. Default to us-east-1 with a custom endpoint, as most S3-compatible providers ignore the region, and return an explicit error for AWS S3. Signed-off-by: Cedric Grard <cedric.grard@eurofiber.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Moves the S3 client back to
aws-sdk-go-v2and fixes the root cause that led to minio-go in 0b3984d: sinceservice/s3v1.73 the SDK sends flexible checksums by default, and many S3-compatible providers reject them.USE_PATH_STYLEwith a custom endpoint:truefalseIt has no effect on AWS S3, where there is no custom endpoint.
Also included:
GetServiceConfigreturned["", ""]when no key was set, and"age1a, age1b"failed to parse because of the space.minio/miniois no longer pullable from Docker Hub, so the test is switched toversity/versitygw(Apache-2.0). It is also fixed for Talos 1.14 (dropped thehostDNSpatch) and for recent Docker (container IP read from the bridge network).Evidence
TestPushSnapshotCustomEndpointfails withunexpected checksum header X-Amz-Checksum-Crc32(SDK defaults applied to the upload).After: passes, request goes to
/bucket/prefix/snap.dbwith nox-amz-checksum-*header and no trailing checksum.TestGetServiceConfig/defaultsfails withAgeRecipientPublicKey: []string{"", ""}.After: all 5 cases pass.
pull access denied for minio/minio.After:
TZ=UTC go test -tags integration ./internal/integration/passes (snapshot taken from a Talos 1.14 node, compressed, encrypted, uploaded, listed back).make lint-golangci-lint lint-gofumpt lint-govulncheck lint-markdown: clean, no vulnerabilities found.Merge Danger
Door: two-way
The only behaviour change for existing users is
USE_PATH_STYLE=false, which now really selects virtual-hosted-style. Unset andtruebehave as before. One edge case: a custom endpoint pointing at a Google or Aliyun host used virtual-hosted-style under minio-go's auto mode; it now needsUSE_PATH_STYLE=false.Blast Radius: uploads
Every backup goes through this client. Exercised here against versitygw and an
httptestserver, not against real AWS, Backblaze, R2 or Ceph.Notes for maintainers:
go.modnow requiresgo 1.26.8(pulled in by dependencies) while the kresMakefilepinsGO_VERSION 1.26.1. Amake rekresis probably needed.feature/s3/transfermanageris the replacement for the deprecatedfeature/s3/manager, but it is still v0.x.TZ=UTC: the key regex expects aZsuffix. This predates this PR and is left unchanged.