feat: bundle fetcher (jl2py.bundles.ensure) with injected transport callback#18
Merged
Conversation
…allback Add jl2py.bundles — a zero-dep (stdlib-only) fetcher that owns the bundle end-state contract while the transport/auth is an injected `fetch` callback (resolves design O2). jl2py never learns about R2/awscli/boto3/presigned URLs. `ensure(name, *, version=None, platform=None, julia_version=None, fetch=None, manifest=None, cache_dir=None) -> Path`: - resolves version (default: manifest latest_version), platform (default: the host's Julia triple), and julia_version (default: highest available for the resolved version+platform) against an R2-shaped manifest; - verifies the downloaded tarball's sha256 against the manifest entry (fails hard, leaving nothing in the cache); - unpacks tar.zst (zstd CLI, with an optional `zstandard` fallback and a clear error if neither is present) into a temp dir, then atomically renames into `<cache_dir>/<name>-<version>-<platform>-julia<julia_version>/`; - validates the unpacked bundle.json (exists, parseable, julia_version match); - is idempotent (a cached bundle is returned without any fetch — not even the manifest, when fully pinned); - honors `$JL2PY_BUNDLE_PATH` (air-gapped) and `$JL2PY_BUNDLE_CACHE`. The module is import-independent of the native lib, so it fetches without a built libjl2py / a live Julia. 14 stdlib pytest cases (no Julia needed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V9jVcSYDNXtbBGfcAEPwvh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
New module
src/jl2py/bundles.pyimplementing the bundle fetcher from the jl2py convergence + bundle-pipeline design (§4), resolving O2: jl2py owns the end-state contract while the transport/auth is an injected callback.ensure(name, *, version=None, platform=None, julia_version=None, fetch=None, manifest=None, cache_dir=None) -> pathlib.Pathfetch(relative_key, dest); example transports (awscli / boto3 / presigned) are documented in the docstring without importing any of them.versiondefaults to the manifestlatest_version;platformdefaults to the host's Julia triple (platform.machine()/sys.platform→x86_64-linux-gnu,aarch64-apple-darwin, …);julia_versiondefaults to the highest available for the resolved(version, platform).tar.zstis unpacked (zstd CLI, with an optionalzstandardfallback and a clear error if neither is present) into a temp dir, then atomically renamed into<cache_dir>/<name>-<version>-<platform>-julia<julia_version>/.bundle.jsonmust exist, parse, and itsjulia_versionmust match the requested one — otherwise a clean error (converting the cryptic deep-loader build_id failure into an actionable message).$JL2PY_BUNDLE_PATH(air-gapped, returned after bundle.json validation) and$JL2PY_BUNDLE_CACHE(cache root; default~/.cache/jl2py/bundles).The module is import-independent of the native lib (added to
jl2py/__init__.pyasjl2py.bundles), so a bundle can be pre-fetched without a builtlibjl2pyor a live Julia.Tests
tests/test_bundles.py— 14 stdlib pytest cases, no Julia needed: env-override + air-gapped julia mismatch, corrupt/missingbundle.jsonrejection, manifest resolution (latest_version + highest julia, from dict/file/fetch), no-matching-build error, sha256 mismatch → error + no cache pollution, idempotent second call invokes no fetch, unpacked julia mismatch rejection,$JL2PY_BUNDLE_CACHE, platform-triple shape. Tarball-dependent cases build a tiny.tar.zstwith thezstdCLI and skip with a clear reason if zstd is absent (Ubuntu CI has it). They run in the existingtestworkflow alongside the Julia-backed suite.🤖 Generated with Claude Code
https://claude.ai/code/session_01V9jVcSYDNXtbBGfcAEPwvh