Skip to content

feat: bundle fetcher (jl2py.bundles.ensure) with injected transport callback#18

Merged
jack-champagne merged 1 commit into
masterfrom
feat/bundle-fetcher
Jul 5, 2026
Merged

feat: bundle fetcher (jl2py.bundles.ensure) with injected transport callback#18
jack-champagne merged 1 commit into
masterfrom
feat/bundle-fetcher

Conversation

@jack-champagne

Copy link
Copy Markdown
Member

What

New module src/jl2py/bundles.py implementing the bundle fetcher from the jl2py convergence + bundle-pipeline design (§4), resolving O2: jl2py owns the end-state contract while the transport/auth is an injected callback.

from jl2py import bundles
path = bundles.ensure("piccolissimo-pkgimage", fetch=my_fetch)  # -> cached bundle dir

ensure(name, *, version=None, platform=None, julia_version=None, fetch=None, manifest=None, cache_dir=None) -> pathlib.Path

  • Zero-dep (stdlib only). jl2py never imports R2/awscli/boto3/urllib — the caller supplies fetch(relative_key, dest); example transports (awscli / boto3 / presigned) are documented in the docstring without importing any of them.
  • Resolution: version defaults to the manifest latest_version; platform defaults to the host's Julia triple (platform.machine() / sys.platformx86_64-linux-gnu, aarch64-apple-darwin, …); julia_version defaults to the highest available for the resolved (version, platform).
  • Contract enforcement: sha256 of the downloaded tarball must match the manifest entry (fails hard, leaves nothing in the cache); tar.zst is unpacked (zstd CLI, with an optional zstandard fallback and a clear error if neither is present) into a temp dir, then atomically renamed into <cache_dir>/<name>-<version>-<platform>-julia<julia_version>/.
  • Validation: the unpacked bundle.json must exist, parse, and its julia_version must match the requested one — otherwise a clean error (converting the cryptic deep-loader build_id failure into an actionable message).
  • Idempotent: an already-cached bundle is returned without any fetch (not even the manifest, when fully pinned).
  • Escape hatches: $JL2PY_BUNDLE_PATH (air-gapped, returned after bundle.json validation) and $JL2PY_BUNDLE_CACHE (cache root; default ~/.cache/jl2py/bundles).

The module is import-independent of the native lib (added to jl2py/__init__.py as jl2py.bundles), so a bundle can be pre-fetched without a built libjl2py or a live Julia.

Tests

tests/test_bundles.py — 14 stdlib pytest cases, no Julia needed: env-override + air-gapped julia mismatch, corrupt/missing bundle.json rejection, manifest resolution (latest_version + highest julia, from dict/file/fetch), no-matching-build error, sha256 mismatch → error + no cache pollution, idempotent second call invokes no fetch, unpacked julia mismatch rejection, $JL2PY_BUNDLE_CACHE, platform-triple shape. Tarball-dependent cases build a tiny .tar.zst with the zstd CLI and skip with a clear reason if zstd is absent (Ubuntu CI has it). They run in the existing test workflow alongside the Julia-backed suite.

🤖 Generated with Claude Code

https://claude.ai/code/session_01V9jVcSYDNXtbBGfcAEPwvh

…allback

Add jl2py.bundles — a zero-dep (stdlib-only) fetcher that owns the bundle
end-state contract while the transport/auth is an injected `fetch` callback
(resolves design O2). jl2py never learns about R2/awscli/boto3/presigned URLs.

`ensure(name, *, version=None, platform=None, julia_version=None, fetch=None,
manifest=None, cache_dir=None) -> Path`:

- resolves version (default: manifest latest_version), platform (default: the
  host's Julia triple), and julia_version (default: highest available for the
  resolved version+platform) against an R2-shaped manifest;
- verifies the downloaded tarball's sha256 against the manifest entry (fails
  hard, leaving nothing in the cache);
- unpacks tar.zst (zstd CLI, with an optional `zstandard` fallback and a clear
  error if neither is present) into a temp dir, then atomically renames into
  `<cache_dir>/<name>-<version>-<platform>-julia<julia_version>/`;
- validates the unpacked bundle.json (exists, parseable, julia_version match);
- is idempotent (a cached bundle is returned without any fetch — not even the
  manifest, when fully pinned);
- honors `$JL2PY_BUNDLE_PATH` (air-gapped) and `$JL2PY_BUNDLE_CACHE`.

The module is import-independent of the native lib, so it fetches without a
built libjl2py / a live Julia. 14 stdlib pytest cases (no Julia needed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V9jVcSYDNXtbBGfcAEPwvh
@jack-champagne jack-champagne marked this pull request as ready for review July 5, 2026 01:51
@jack-champagne jack-champagne merged commit 0de4bba into master Jul 5, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant