[jonathan343]

Note

1. This is the first of three PRs. Follow-ups add CRC64NVME and then XXHash64, XXHash3, and XXHash128 to this package.
2. This PR description is AI generated but reviewed by me.

Summary

S3's flexible checksums support CRC32, CRC32C, CRC64NVME, SHA-1, SHA-256, SHA-512, MD5, and three XXHash variants. The standard library only covers CRC32, SHA, and MD5. botocore gets the rest from awscrt, which is an optional dependency, so those algorithms are unavailable without it. smithy-python has no checksum implementations yet, so generated clients can't support flexible checksums.

This PR adds a smithy-checksums package, starting with CRC32 and CRC32C. Each algorithm has a Rust implementation (PyO3 and the crc-fast crate, which the AWS SDK for Rust also uses) and a pure-Python fallback that gives identical results. SHA-1, SHA-256, SHA-512, and MD5 come from hashlib through the same new() entry point. The fallback means the package installs everywhere, so clients can take a hard dependency on it. Platforms with native wheels get fast checksums.

The PR also sets up the repository for packages with Rust extensions: build, lint, dependency, and license checks, plus CI jobs that build and test native wheels on Linux, macOS, and Windows. Wiring checksums into generated clients is out of scope.

Example

import base64

import smithy_checksums

checksum = smithy_checksums.new("CRC32C")
checksum.update(b"Hello ")
checksum.update(b"world")
print(checksum.name, checksum.digest().hex(), base64.b64encode(checksum.digest()))
print("native:", smithy_checksums.HAS_NATIVE_EXTENSION)

print(type(smithy_checksums.new("SHA256")))

try:
    smithy_checksums.new("crc16")
except ValueError as error:
    print(error)

Native wheel

crc32c 72b51f78 b'crUfeA=='
native: True
<class '_hashlib.HASH'>
Unsupported checksum algorithm: 'crc16'

Pure-Python wheel

crc32c 72b51f78 b'crUfeA=='
native: False
<class '_hashlib.HASH'>
Unsupported checksum algorithm: 'crc16'

The public API is Checksum (a protocol that hashlib objects also satisfy), CRC32, CRC32C, new(), and HAS_NATIVE_EXTENSION. Digests are big-endian, the byte order x-amz-checksum-* headers base64-encode.

Decisions

1. One package, two build backends. An in-tree build backend uses maturin when cargo is available and hatchling (pure Python) otherwise. A separate native package couldn't be a safe hard dependency, since there's no way to require it only where a wheel exists.
2. Fall back only when Rust is missing. Without Rust, source builds quietly produce the pure-Python wheel. If Rust is installed but too old or the build fails, the build stops and suggests SMITHY_PYTHON_NATIVE_BUILD=never.
3. The sdist is always built with hatchling. maturin needs cargo even to build an sdist, and the release build doesn't have Rust.
4. Native wheels ship third-party licenses. They contain compiled MIT and Apache-2.0 crates, so a generated LICENSE-THIRD-PARTY is included and CI checks it's current. That file and Cargo.lock are 1,224 of the 2,759 added lines.

Additional Testing

Correctness is checked against independent references. Those are the standard CRC check values, the CRC32C vectors from RFC 3720 appendix B.4, and the CRT (awscrt), which botocore uses. Native and pure are also compared against each other across 300+ input lengths:

uv run pytest packages/smithy-checksums -k "known_answers or matches_crt or native_matches_pure" --no-cov -q

12 passed, 73 deselected in 0.06s

Built wheels locally on macOS arm64, installed each into a fresh virtual environment, and ran the package tests against the installed wheel:

Python	Wheel	Implementation	Tests
3.12, 3.13, 3.14, 3.15	cp312-abi3	native	81 passed, 4 skipped
3.14t	cp314-cp314t	native, GIL stays disabled	81 passed, 4 skipped
3.15t	cp315-abi3.abi3t	native, GIL stays disabled	81 passed, 4 skipped

The four skips are the CRT comparisons, since those environments didn't have awscrt.

Installing from the sdist without a Rust toolchain falls back to pure Python:

uv build packages/smithy-checksums --sdist --out-dir dist
uv venv --quiet .venv-sdist
# Hide cargo by running with a minimal PATH.
PATH="/usr/bin:/bin:$(dirname "$(command -v uv)")" \
  uv pip install --quiet --python .venv-sdist --no-cache dist/smithy_checksums-0.0.0.tar.gz
.venv-sdist/bin/python -c "import smithy_checksums; print(smithy_checksums.HAS_NATIVE_EXTENSION)"

False

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.