build: migrate to scikit-build-core + uv, add manylinux wheels, re-arm PyPI release - #1117
Merged
Conversation
Replace poetry-core + build.py with scikit-build-core + CMake, and move the dev workflow to uv. Prerequisite for publishing to PyPI and bioconda: the previous backend could not build in CI at all, since build.py hard-failed without CONDA_PREFIX and no workflow installed conda or htslib. Build system: - Add CMakeLists.txt building scanmst._cppext with exact source parity to build.py (binding.cpp stays uncompiled -- it declares a second PYBIND11_MODULE and would give a duplicate PyInit__cppext symbol). - Resolve htslib via HTSLIB_ROOT then CONDA_PREFIX, treating empty strings as unset. HTSLIB_ROOT is the non-conda entry point for CI, Docker, cibuildwheel and bioconda. - Set INSTALL_RPATH behind SCANMST_EMBED_RPATH. BUILD_RPATH alone does not survive cmake --install, and the previous build only had an RPATH by accident, via conda's LDFLAGS carrying -Wl,-rpath for the base env. - Install the extension to scanmst/, not scanmst/cppext/: cppext/__init__.py does `from scanmst._cppext.cppext import *`. Packaging: - Convert [tool.poetry] to PEP 621. requires-python is >=3.9,<3.11, capped by pysam==0.19.0, which ships no wheels above cp310. - Declare pyyaml and requests, previously imported by scanmst.base.helper and scanmst.blat but only ever resolved transitively. - Move pybind11 from a runtime to a build dependency. - Use a PEP 639 license expression; the file form embedded 675 lines of GPL text into METADATA (52KB -> 11KB). - Exclude the runtime-downloaded UCSC BLAT binaries from wheels, which the poetry build shipped whenever they were present on disk. - Read __version__ from package metadata; pyproject said 0.1.9 while __init__.py said 0.1.8. Dev workflow: - Drop nox-poetry; replace export_requirements() with `uv export`. Commit uv.lock and remove the *.lock catch-all that would have swallowed it. - Add PEP 735 dependency groups. - Fix mypy being handed a nonexistent docs/conf.py, and refurb passing its args to install() instead of run() so it never ran. CI: - tests.yml installs htslib and builds via uv; matrix drops 3.8, which was already below the declared floor. - release.yml is disarmed to workflow_dispatch until Trusted Publishing is configured; it built with poetry and would otherwise fire on push to main.
Step 2 of the PyPI/bioconda migration. The wheel Step 1 produces is tagged linux_x86_64 and links the developer's conda libhts, so PyPI rejects it and it is not portable. Build proper manylinux wheels with htslib bundled instead. - scripts/build-htslib.sh compiles htslib 1.23.1 inside the manylinux container, pinned to the version the verified local build linked against. libcurl/gcs/s3/libdeflate are disabled: the extension calls only core BAM/SAM APIs, and enabling them would drag libcurl and openssl into the bundled wheel. - [tool.cibuildwheel] targets cp39/cp310 manylinux_2_28 x86_64 only. macOS is excluded because pysam==0.19.0 ships no arm64 wheel, so Apple Silicon can never resolve the dependency chain; tests.yml drops its macos-latest entry for the same reason, as those runners are arm64. - SCANMST_EMBED_RPATH=OFF hands RPATH ownership to auditwheel. An RPATH pointing at the build container's /usr/local/lib is meaningless on a user's machine. - scripts/check_wheel.py gates on the two failure modes that yield a wheel which installs cleanly and then breaks at import: the extension landing somewhere other than scanmst/_cppext*.so, and libhts not being bundled. It also rejects a non-manylinux tag. - wheels.yml runs on workflow_dispatch and v* tags only, uploading artifacts. Nothing is published; that is Step 3. cibuildwheel rebuilds htslib per interpreter, which is too slow for per-commit CI.
workflow_dispatch requires the workflow file to be on the default branch, so this is the only way to prove the manylinux build before it lands. The branches: block is removed before merge.
The manylinux build is proven (run 31572447407): auditwheel bundled libhts-ead13fb8.so.1.23.1 and the wheels install and run on a non-conda Python. wheels.yml is back to workflow_dispatch + v* tags only.
release.yml now builds via wheels.yml (converted to a reusable workflow to avoid double-building on tag push) and publishes with `uv publish` under OIDC trusted publishing -- no stored token. A check-version job gates the whole run on the tag matching pyproject.toml before anything builds or publishes, since PyPI uploads are permanent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four-step PyPI + bioconda publishing migration. Steps 1-3 are in this PR; bioconda (Step 4) is a follow-up.
9930985) — replace poetry-core +build.pywith scikit-build-core + CMake; move dev workflow to uv. Fixes packaging bugs found along the way: undeclaredpyyaml/requestsruntime deps,pybind11misclassified as a runtime dep, oversized LICENSE embedding, stale commitizen tag format.9d5c0d3) — cibuildwheel manylinux wheels with htslib 1.23.1 bundled via auditwheel.scripts/check_wheel.pygates the two failure modes that produce a wheel which installs cleanly and breaks at import (extension in the wrong place, htslib not bundled). Verified by a real green CI run with artifacts downloaded and re-checked locally (seeCLAUDE.mdfor the run link and details).efce9c8) — re-armrelease.ymlonv*tags with PyPI Trusted Publishing (OIDC,uv publish, no stored token).wheels.ymlis now a reusable workflow (workflow_call) thatrelease.ymlinvokes, avoiding a duplicate cibuildwheel build on tag push. Acheck-versionjob blocks the whole run if the tag doesn't matchpyproject.toml's version, since PyPI uploads are permanent.Full context and load-bearing constraints are documented in
CLAUDE.mdat the repo root.Test plan
wheels.ymlbuild, both jobs green; wheel artifacts downloaded and manually verified (.soplacement, RPATH, clean-venv smoke test withLD_LIBRARY_PATHunset)tests.ymlpasses on this branch (htslib via apt,HTSLIB_ROOT=/usr)release.ymlagainst thev0.1.9tag ref (the tag predates this branch and can't be re-pushed to fire the tag trigger) to perform the actual first publish to PyPI, now that the pending Trusted Publisher is configured (ylab-hi/ScanMST, workflowrelease.yml, environmentpypi)