Make pip install pamtra work: static OpenBLAS/FFTW, cibuildwheel pipeline - #65
Merged
Conversation
A dynamic OpenBLAS bundled into a PyPI wheel would collide at runtime with numpy/scipy own bundled copy in the same process (duplicate global symbols, shared thread-pool state) -- version matching alone does not prevent this, since nothing renames or namespaces the symbols. tools/build_openblas_static.sh builds OpenBLAS USE_THREAD=0 (PAMTRA only uses it for small per-particle T-matrix solves, not large GEMMs) and static-only. meson.build's pyPamtraLib target now also restricts its exported dynamic symbol table to just PyInit_pyPamtraLib at link time, since -fvisibility=hidden on OpenBLAS own build does not reach its hand-written assembly kernels (they set .globl directly). Verified with nm/otool (only PyInit_pyPamtraLib exported, no libopenblas linked) and the full pytest suite (66 passed, including the T-matrix-exercising regression tests). This is a building block for an eventual cibuildwheel pipeline, not a full one -- FFTW and netCDF-Fortran still need their own bundling story. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
OpenBLAS own build output warns about exactly this: any flags passed to make during build must also be passed to make install, or install can fail. On Linux this is not just a warning -- without NO_SHARED=1 at install time, make install tries to `install` a shared library that was never built and exits non-zero (install: cannot stat libopenblas...so: No such file or directory), so the whole script aborts under set -euo pipefail before ever generating openblas.pc. macOS masked this because the equivalent step there uses cp in a make recipe line that is allowed to fail, so it looked like a warning instead of a hard error. Verified on a real Debian x86_64 machine: install now completes and openblas.pc is generated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pyproject.toml: [project] name pyPamtra -> pamtra, so `pip install pamtra` works once published. The import name stays `import pyPamtra` unchanged (driven by meson.build's install_sources subdir, not this field) -- same pattern as `pip install beautifulsoup4` -> `import bs4`. This also aligns with conda-recipe/recipe.yaml, which was already named pamtra, not pyPamtra. tools/build_fftw_static.sh mirrors build_openblas_static.sh: builds FFTW static-only (--disable-shared, --with-pic) so it needs no auditwheel/ delocate bundling step later. Unlike OpenBLAS, FFTW has no known symbol- collision risk with other commonly-bundled packages, so this is purely to keep the wheel's runtime dependency surface minimal, not a correctness requirement. Verified on both macOS arm64 (here) and Debian x86_64 (over SSH): built against both static libs together, confirmed via otool/ldd that neither libopenblas nor libfftw3 is a runtime dependency anymore, confirmed the exported symbol table still contains only PyInit_pyPamtraLib, and ran the full pytest suite on both (66/66 on macOS, 65/66 on Debian -- the one Debian failure is test_data_autofetch.py's network-failure test, unrelated to this change). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… build meson.build: pyPamtraLib listed netcdfdep as a dependency, but nothing it compiles ever calls netCDF -- confirmed via grep that the only Fortran source using netCDF at all is write_nc_results.f90, which belongs exclusively to the standalone pamtra executable's own source list, never pyPamtraLib's. Verified empirically too: removed the dependency, rebuilt with no netcdf present in the environment at all, still links cleanly, import pyPamtra works, and the full pytest suite still passes (66/66). This matters for wheel-building: import pyPamtra already gets NetCDF I/O purely through the netCDF4 Python package (a runtime dependency that ships its own self-contained wheels), so the compiled extension needed no direct C-level link to libnetcdf at all -- it was dead weight, and arguably a latent risk of its own (a second, separately-linked copy of netCDF-C sharing a process with netCDF4's bundled copy). Only the standalone pamtra CLI binary target still needs netCDF-C/-Fortran. tools/build_netcdf_stack.sh builds that stack (HDF5 -> netCDF-C --disable-dap --disable-nczarr -> netCDF-Fortran) as ordinary dynamic libraries into one prefix, for the pamtra CLI target specifically. Unlike OpenBLAS/FFTW this is intentionally left dynamic rather than static: HDF5/netCDF have no known symbol-collision risk with other commonly- bundled PyPI packages (unlike OpenBLAS with numpy/scipy), so the standard auditwheel/delocate dynamic-bundling path -- what netCDF4's and h5py's own PyPI wheels already use -- is the right, lower-risk tool here rather than reinventing static linking for a much larger, more complex dependency chain. Verified locally: all three stages built cleanly on the first try. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
meson.build's netcdf-fortran includedir workaround used include_directories(), which meson rejects for absolute paths that resolve inside the source tree. That's exactly what happens with tools/build_netcdf_stack.sh's default prefix (<repo>/build-deps/netcdf), failing with "Tried to form an absolute path to a dir in the source tree." Switched to a raw '-I' compile_args string instead, which the compiler accepts with no such restriction. This isn't just a local- testing artifact -- a real cibuildwheel before-all hook building dependencies in-tree would hit the same thing. Verified end-to-end on both macOS arm64 and Debian x86_64: built pyPamtra + the standalone pamtra CLI against all three custom dependency prefixes (static OpenBLAS, static FFTW, dynamic HDF5/netCDF-C/netCDF-Fortran) together, confirmed via otool/ldd (with LD_LIBRARY_PATH set, since these are dynamic libraries at a non-standard prefix -- see the AI.md note added here) that everything resolves to our own builds rather than system copies, and ran the full pytest suite on both machines. Also documents tools/build_fftw_static.sh and tools/build_netcdf_stack.sh in AI.md alongside the existing OpenBLAS entry, including the LD_LIBRARY_PATH gotcha for local (pre-wheel) testing of the dynamic netCDF stack. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pyproject.toml: [tool.cibuildwheel] targets cp311-cp314 on manylinux_2_28 (Linux x86_64) and macOS (both archs), matching the existing pip-build CI matrix. before-all runs the new tools/cibw_before_all.sh, which installs gfortran/zlib-devel (Linux) or gcc (macOS) then builds all three bundled dependencies -- static OpenBLAS, static FFTW, dynamic HDF5/netCDF-C/ netCDF-Fortran -- into /tmp/pamtra-deps, deliberately outside the checked- out source tree (a prefix under the repo can trip meson.build's netcdf-fortran includedir handling, per the commit that fixed that for local testing -- building outside the tree sidesteps the question rather than relying on that fix alone). test-command reuses the existing pytest suite plus a `pamtra -h` smoke test, both already used in ci.yml, with no LD_LIBRARY_PATH set -- unlike local dev testing, this doubles as a check that auditwheel/delocate's repair step actually rewrote the wheel to load its own bundled libraries rather than relying on an ambient env var. tools/build_openblas_static.sh: switched from the default `make` target to `make libs` explicitly. Discovered by running cibuildwheel locally (it can build macOS wheels without Docker): OpenBLAS's default target is `all :: tests`, which also builds and links its own BLAS/LAPACK self-test programs -- unneeded here, and failing to link specifically inside cibuildwheel's sandboxed build environment even though the exact same script worked in a plain interactive shell. `libs` builds only the library itself; `make install` was already a separate, unaffected target. Locally verified end-to-end (via `cibuildwheel --only cp313-macosx_arm64`) that before-all now succeeds and produces the dependency prefixes correctly. Couldn't verify the full wheel build to completion on this machine: cibuildwheel's macOS path needs the official python.org "Framework" CPython installs that GitHub-hosted macOS runners ship with for exactly this purpose, which this dev machine (conda-only Python) doesn't have -- not something to install system-wide without asking. Real CI is the next step. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
.github/workflows/wheels.yml runs cibuildwheel across [ubuntu-latest, macos-13, macos-14] (Linux x86_64 + macOS Intel + macOS Apple Silicon), triggered manually or on pushes touching packaging files. Deliberately not on every PR -- a full wheel build (compiling OpenBLAS, FFTW, and the whole HDF5/netCDF stack from source per commit) is expensive enough to not want on the normal PR loop. After each build, unpacks one wheel and re-runs this session's manual verification (nm/ldd on Linux, nm/otool on macOS) inline in CI: exactly one exported symbol (PyInit_pyPamtraLib) and no libopenblas/libfftw3 linked. This is deliberately not just "did cibuildwheel exit 0" -- it confirms the actual property the static+hidden-symbol OpenBLAS work exists for survives the real auditwheel/delocate repair step, not just the hand-built local case already verified on two real machines. No publish step yet -- that's gated on manually setting up PyPI trusted publishing first (needs a PyPI account login, not something automatable). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@v4 exists as a git tag but GitHub Actions' action resolver could not find it (Unable to resolve action pypa/cibuildwheel@v4, unable to find version v4) -- likely a marketplace-indexing quirk rather than a real missing tag, but pinning the exact release avoids relying on that resolving correctly either way. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both found by actually running the wheels.yml workflow on GitHub Actions (the local cibuildwheel test never got far enough to catch either, since it hit the missing-Framework-Python wall first). tools/build_openblas_static.sh: `make libs` (added in the previous commit to skip OpenBLAS's failing self-test suite) turned out to skip more than just the tests -- OpenBLAS's own target graph is `shared : libs netlib $(RELA)`, so LAPACK (netlib) is a separate prerequisite of `shared`, not of `libs`. A `libs`-only build produces a static archive with BLAS but no LAPACK at all. This went unnoticed locally because pyPamtraLib never calls into LAPACK and linked fine either way; only the standalone pamtra executable's radmat.f90 -> minvert_lapack_ -> dgetri_ chain exposed it, failing on macos-14 in CI with "Undefined symbols ... _dgetri_". Switched to `make shared`, which builds libs+netlib but -- since NO_SHARED=1 is already set -- still skips both the actual .dylib/.so assembly (wrapped in `ifneq ($(NO_SHARED), 1)` in OpenBLAS's Makefile) and the `tests` target that depends on `shared` succeeding. Confirmed locally: `nm` now shows _dgetri_ present in the archive, and the full local build (both pyPamtraLib and the pamtra executable, tested via the existing pytest suite) is back to 66/66 passing. tools/cibw_before_all.sh: added libxml2-devel to the manylinux yum/dnf install list. netCDF-C's ./configure wants libxml2-config even with --disable-dap set (some other feature depends on it, not just DAP) -- failed on ubuntu-latest in CI with "Cannot find xml2-config utility". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The standalone pamtra executable kept segfaulting specifically inside
cibuildwheel's manylinux container, on real computation (not just -h),
after building and auditwheel-repairing successfully with no visible
errors. Extensive investigation (rebuilding the same wheel + repair step
by hand on a real Debian machine, reproducing the actual auditwheel
repair mechanism outside the container) could not pin down the cause --
a properly repaired wheel worked fine there once a red herring (a stale
system-linked pamtra binary shadowing the venv's own on PATH) was ruled
out, so the container-specific segfault remains unexplained. Given
pyPamtraLib itself needs no netCDF at all (see the previous commit) and
the CLI was the only reason wheel builds needed netCDF-C/-Fortran/HDF5,
the pragmatic fix is to just not ship the CLI in the wheel.
meson_options.txt: new build_cli boolean option, default true.
meson.build: netCDF-C and netCDF-Fortran resolution (previously
unconditional -- netcdfdep was a hard `dependency('netcdf')` that would
fail the whole build if netCDF-C were absent) is now wrapped in
`if build_cli`, with both left as disabler() otherwise. Since the pamtra
executable target lists both as dependencies, disabler() propagation
means the executable target itself is silently skipped when build_cli is
off, with no separate `if` needed around the executable() call.
pyproject.toml: [tool.cibuildwheel] passes -Dbuild_cli=false via
config-settings; tools/cibw_before_all.sh no longer calls
build_netcdf_stack.sh (still exists for local/dev CLI-included builds).
Verified locally: `pip install . -Csetup-args=-Dbuild_cli=false` with
only static OpenBLAS/FFTW on PKG_CONFIG_PATH (no netcdf at all) builds
cleanly, installs no pamtra binary, and the full pytest suite (which
skips test_cli_binary.py cleanly via its own pamtra_binary fixture when
none is found) still passes.
Also fixes the macOS build failure found in the same CI run: delocate
refused to repair a wheel tagged for macOS 11.0 (cibuildwheel's arm64
default) when the bundled libgfortran/libquadmath (from GitHub-hosted
runners' Homebrew gcc) themselves require macOS 14.0 --
"Library dependencies do not satisfy target MacOS version 11.0". Set via
MACOSX_DEPLOYMENT_TARGET=14.0 in [tool.cibuildwheel.macos]'s environment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GitHub's Intel-macOS runner queue is badly backlogged -- a build sat queued for 3+ hours here with no sign of clearing, while the same commit built cleanly on ubuntu-latest and macos-14 (Apple Silicon) within minutes. Consistent with GitHub winding that pool down as Apple's own Intel Mac lineup ages out. Dropping it intentionally rather than continuing to wait on it; revisit if the queue situation changes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Uses trusted publishing (OIDC via the testpypi GitHub Actions environment, no stored token) per pypa/gh-action-pypi-publish's documented pattern. Downloads the per-OS wheel artifacts from the build job (wheels-ubuntu-latest, wheels-macos-14) into one dist/ directory before publishing, since this workflow's matrix build uploads one artifact per platform rather than a single combined one. Manual dispatch only, not on the workflow's other (packaging-file-push) trigger: TestPyPI rejects re-uploading the same version, so publishing on every such push would just fail after the first success. This is meant as a one-off dry run to validate the pipeline before promoting to a real, tag-gated PyPI publish job. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pyproject.toml/ci.yml: add cp310/"3.10" to the cibuildwheel build target and the pip-build test matrix respectively. Both previously started at 3.11 even though pyproject.toml already declares requires-python >=3.10 -- a real gap now that this is heading toward an actual PyPI release, not just an oversight to leave in place. wheels.yml: new publish-pypi job, gated on an actual vX.Y.Z release tag (startsWith(github.ref, 'refs/tags/v')) -- never on this workflow's other triggers (workflow_dispatch, packaging-file pushes), so nothing here can accidentally publish a real release. Uses trusted publishing (OIDC) via the pypi GitHub Actions environment, same pattern as the existing publish-testpypi job. Adding `tags: - 'v*'` to the workflow's push trigger is safe alongside the existing `paths:` filter even though a tag push touches none of those files: GitHub doesn't apply path filtering to tag pushes at all, only to branch pushes. No tag pushed as part of this commit -- that (and the one remaining manual prerequisite, registering a pypi.org trusted publisher for this project) is a deliberate separate step, not automated here. RELEASING.md: replaces the now-outdated "Why not PyPI" section (this session's earlier work solved everything it described as blocking) with a "PyPI" section covering how the bundling actually works and the trusted-publisher setup it depends on, plus a note in step 2 that pushing the release tag alone already triggers the real publish -- no separate manual step beyond that. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed empirically: the previous commit's tags: addition (with no branches: key) caused this exact push to not trigger the workflow at all, not just narrow which branches matched. GitHub's documented (if not obviously so) behavior: adding tags: with no branches: suppresses branch-push triggering entirely. branches: ['**'] restores it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pyproject.toml classifiers claimed Windows support, which was never actually true (no Windows CI, no Windows wheel, conda-recipe explicitly skips win) -- removed. No classifier granularity exists for "macOS arm64 only, not Intel", so that caveat goes in the actual install docs instead, added in three places since pip install pamtra never had any documentation at all before this branch: - readme.md (also PyPI's project-page description via pyproject.toml's readme field) - doc/source/installation.rst, a new section ahead of the from-source instructions Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
maahn
added a commit
that referenced
this pull request
Aug 20, 2026
First real PyPI-availability release -- no functional/physics changes to PAMTRA itself, this is the packaging work from #64/#65 (meteo_si split, static-OpenBLAS/FFTW wheel pipeline, cibuildwheel, Python 3.10 support). Bumped rather than kept at 1.0.3 (the existing conda-forge version) to mark that milestone distinctly. All three places RELEASING.md documents (pyproject.toml, meson.build, conda-recipe/recipe.yaml) updated together. conda-recipe/recipe.yaml's source.sha256 is intentionally left at the old 1.0.3 tarball's checksum for now -- RELEASING.md's own process computes that from the real v1.1.0 tag's generated tarball, which doesn't exist until after this is tagged. That recipe isn't used by this release itself (only PyPI); the conda-forge feedstock update is a separate, later step. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes
pip install pamtrawork as a real, self-contained wheel for the first time -- the whole point of this branch. Builds on top of the earliermeteo_simerge (#64), already reflected here via a clean merge frommaster.Platform scope: Linux x86_64 and macOS Apple Silicon (arm64) only. No wheels for Windows or Intel macOS (
osx-64) -- seewheels.yml's matrix comment for why Intel macOS specifically was dropped.pyproject.toml's classifiers,readme.md, anddoc/source/installation.rstall now say this explicitly.PyInit_pyPamtraLibat link time -- avoids colliding with numpy/scipy's own bundled OpenBLAS in the same process (tools/build_openblas_static.sh).tools/build_fftw_static.sh).pamtraCLI, notpyPamtraLib(import pyPamtragets NetCDF I/O through the pure-PythonnetCDF4package) -- so wheel builds pass-Dbuild_cli=falseand skip this dependency chain entirely (tools/build_netcdf_stack.shstill used bypip install ./conda builds, which build the CLI by default)..github/workflows/wheels.yml:cibuildwheelacross Linux x86_64 + macOS arm64 (Intel macOS dropped -- GitHub's runner queue for it was badly backlogged), Python 3.10-3.14, with an in-CI check that the built wheel is genuinely self-contained (nolibopenblas/libfftw3linked, exactly one exported symbol). Also wires up both a manual TestPyPI dry-run job and the real, tag-gated PyPI publish job.RELEASING.mdupdated to describe the PyPI process (replacing the now-outdated "why not PyPI" section) andAI.mddocuments the build scripts.Verified end-to-end multiple times: local builds on macOS arm64 and a real Debian x86_64 machine (including catching several bugs
cibuildwheel's manylinux container surfaced that a local build didn't -- BFLOAT16/LAPACK target selection in OpenBLAS's own Makefile, missinglibxml2-devel, macOS deployment-target mismatches), a full TestPyPI publish + clean-venv install proving the wheel actually works with zero system libraries preinstalled, and the CI matrix (including the newly-added Python 3.10 and themeteo_sidependency together).Test plan
pytest tests/full suite green on everypip install .variation exercised (static deps,build_cli=false, withmeteo_si)wheels.ymlbuild-only run green on Linux + macOS arm64, Python 3.10-3.14RELEASING.md, after this merges)🤖 Generated with Claude Code