Nightly tests of the published packages against the newest dependencies

.github/workflows/package-tests.yaml

@@ -0,0 +1,115 @@
+# Nightly: the tests bundled in the published packages, run again against the newest
+# conda-forge dependencies (tools/package_tests.py), to catch breakages that come from
+# conda-forge rather than from RoboStack, e.g. ABI changes (RoboStack/ros-humble#274).
+# Every package with a test file (shared/tests/, distros/<d>/tests/) is tested in its
+# newest build, on each platform. The results are kept in one issue per distribution
+# ("<distro>: package tests", label package-tests), closed when every test passes.
+name: package-tests
+
+on:
+  schedule:
+    - cron: "31 2 * * *"
+  workflow_dispatch:
+    inputs:
+      distros:
+        description: Space-separated distributions to test (default all)
+        required: false
+
+concurrency:
+  group: ${{ github.workflow }}
+  cancel-in-progress: false
+
+permissions:
+  contents: read
+
+env:
+  PYTHONIOENCODING: utf-8
+
+jobs:
+  plan:
+    runs-on: ubuntu-latest
+    outputs:
+      distros: ${{ steps.plan.outputs.distros }}
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          persist-credentials: false
+      - name: Distributions
+        id: plan
+        env:
+          SELECTED: ${{ inputs.distros }}
+        run: |
+          distros=$(python3 -c 'import json, os, pathlib
+          selected = os.environ.get("SELECTED", "").split()
+          print(json.dumps(selected or sorted(p.parent.name for p in pathlib.Path("distros").glob("*/vinca.yaml"))))')
+          echo "distros=$distros" | tee -a "$GITHUB_OUTPUT"
+
+  test:
+    needs: plan
+    strategy:
+      fail-fast: false
+      matrix:
+        distro: ${{ fromJSON(needs.plan.outputs.distros) }}
+        target:
+          - { os: ubuntu-latest, platform: linux-64 }
+          - { os: ubuntu-24.04-arm, platform: linux-aarch64 }
+          - { os: macos-15-intel, platform: osx-64 }
+          - { os: macos-15, platform: osx-arm64 }
+          - { os: windows-2022, platform: win-64 }
+    name: ${{ matrix.distro }} ${{ matrix.target.platform }}
+    runs-on: ${{ matrix.target.os }}
+    timeout-minutes: 180
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          persist-credentials: false
+      # git auto-maintenance can leave a lock file in the git clones pixi caches
+      # (e.g. vinca's), which breaks every later run restoring that cache
+      - name: Disable git auto-maintenance
+        run: git config --global maintenance.auto false && git config --global gc.auto 0 && git config --global core.longpaths true
+      - uses: prefix-dev/setup-pixi@v0.11.0
+        with:
+          locked: true  # the lock file must match pixi.toml
+          pixi-version: v0.81.0
+          cache-write: false  # saved from main only (pixi-cache.yaml)
+      - name: Test the published packages
+        # a failed test is reported in the issue (report job), not as a failed job
+        continue-on-error: true
+        shell: bash
+        env:
+          DISTRO: ${{ matrix.distro }}
+          PLATFORM: ${{ matrix.target.platform }}
+        run: pixi run python tools/package_tests.py run --distro "$DISTRO" --platform "$PLATFORM" --results "results/$DISTRO-$PLATFORM.json"
+      - uses: actions/upload-artifact@v6
+        if: always()
+        with:
+          name: package-tests-${{ matrix.distro }}-${{ matrix.target.platform }}
+          path: results/
+          if-no-files-found: ignore
+          retention-days: 14
+
+  report:
+    needs: [plan, test]
+    if: always()
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      issues: write  # the "<distro>: package tests" issues
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          persist-credentials: false
+      - uses: prefix-dev/setup-pixi@v0.11.0
+        with:
+          locked: true  # the lock file must match pixi.toml
+          pixi-version: v0.81.0
+          cache-write: false  # saved from main only (pixi-cache.yaml)
+      - uses: actions/download-artifact@v7
+        with:
+          pattern: package-tests-*
+          path: results/
+      - name: Update the issues
+        env:
+          GH_TOKEN: ${{ github.token }}
+          DISTROS: ${{ join(fromJSON(needs.plan.outputs.distros), ' ') }}
+        run: pixi run python tools/package_tests.py report results/ --distros "$DISTROS"

README.md

@@ -151,6 +151,20 @@ There are three ways to trigger a command:
 
 "Maintainers" means the repository's owners, members and collaborators.
 
+### Nightly package tests
+
+`.github/workflows/package-tests.yaml` runs `tools/package_tests.py` every night (or via *Actions › package-tests › Run workflow*, for some distributions):
+- **What:** every package with a test file (`shared/tests/<package>.yaml`, `distros/<distro>/tests/<package>.yaml`), in its newest build on the distribution's channel, with every variant of that build (e.g. each Python), on all five platforms.
+- **How:** `rattler-build test` runs the tests bundled in the published package, with the newest dependencies from the distribution's channel and conda-forge. This catches breakages that come from conda-forge rather than from RoboStack, e.g. an ABI change in a dependency a package was built against.
+- **Report:** one issue per distribution, *`<distro>`: package tests* (label `package-tests`), with the failed tests and their output; it is closed when every test passes. Builds from before their test file was added carry no tests and are listed as such.
+
+Locally:
+
+```bash
+pixi run python tools/package_tests.py run --distro jazzy --platform osx-arm64 --results results/jazzy.json [--package ros2-rclpy]
+pixi run python tools/package_tests.py report results/
+```
+
 ### New distribution
 
 ```bash

tools/maintenance.py

@@ -1077,32 +1077,38 @@ def report_issue_body(distro: str, data: dict, migrations: str = "", run_url: st
     return body
 
 
-def publish_report(repo: str, distro: str, body: str, conflicts: bool) -> str | None:
-    """Create or update the distribution's dependency-report issue: open while there
-    are conflicts, closed otherwise. Returns its URL."""
-    title = f"{distro}: dependency report"
+def publish_issue(repo: str, title: str, label: str, body: str, open_: bool) -> str | None:
+    """Create or update the issue with this title and label: open while open_ (e.g.
+    there are conflicts), closed otherwise. Returns its URL (None: nothing to report
+    and no issue to close)."""
     query = f'.[] | select(.title == {json.dumps(title)}) | {{number, state, html_url}} | @json'
-    out = _gh(["api", "--paginate", f"repos/{repo}/issues?labels={REPORT_LABEL}&state=all&per_page=100", "--jq", query])
+    out = _gh(["api", "--paginate", f"repos/{repo}/issues?labels={label}&state=all&per_page=100", "--jq", query])
     # the open one, else the newest
     issues = sorted((json.loads(l) for l in out.splitlines() if l.strip()),
                     key=lambda i: (i["state"] == "open", i["number"]))
-    if not issues and not conflicts:
+    if not issues and not open_:
         return None
     with tempfile.NamedTemporaryFile("w", suffix=".md", delete=False) as fh:
         fh.write(body)
     try:
         if issues:
             issue = issues[-1]
             _gh(["api", "-X", "PATCH", f"repos/{repo}/issues/{issue['number']}", "-F", f"body=@{fh.name}",
-                 "-f", f"state={'open' if conflicts else 'closed'}"])
+                 "-f", f"state={'open' if open_ else 'closed'}"])
             return issue["html_url"]
         created = _gh(["api", f"repos/{repo}/issues", "-f", f"title={title}", "-F", f"body=@{fh.name}",
-                       "-f", f"labels[]={REPORT_LABEL}"])
+                       "-f", f"labels[]={label}"])
         return json.loads(created)["html_url"]
     finally:
         os.unlink(fh.name)
 
 
+def publish_report(repo: str, distro: str, body: str, conflicts: bool) -> str | None:
+    """Create or update the distribution's dependency-report issue: open while there
+    are conflicts, closed otherwise. Returns its URL."""
+    return publish_issue(repo, f"{distro}: dependency report", REPORT_LABEL, body, conflicts)
+
+
 def dependency_report(distro: str) -> Result:
     """Check the distribution's dependencies against its pins (linux-64) and keep its
     dependency-report issue up to date (in GitHub Actions; locally only the report)."""