Skip to content

Commit 697aa31

Browse files
Merge pull request #3 from Tobias-Fischer/share-more-files
Share vinca.yaml, pkg_additional_info.yaml and dependencies.yaml settings
2 parents 3533c2b + 393f968 commit 697aa31

30 files changed

Lines changed: 1666 additions & 2126 deletions
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
name: robostack-bot command
2+
description: Ask robostack-bot to run a maintenance task (maintainers only)
3+
title: "robostack-bot: "
4+
labels: ["robostack-bot"]
5+
body:
6+
- type: markdown
7+
attributes:
8+
value: |
9+
robostack-bot runs the command, replies here and closes the issue. Commands that
10+
change files open (or update) a pull request. You can also comment
11+
`@robostack-bot <command> [<distro>]` on any issue or pull request.
12+
13+
| command | what it does |
14+
|---|---|
15+
| `update-rosdistro-snapshot` | Refresh the distribution's `rosdistro_snapshot.yaml` to the latest rosdistro release; the PR lists the version bumps. Runs weekly for every distribution. |
16+
| `find-stale-packages` | List the distribution's published packages that were built against pins that no longer match, with the build-number snippet to rebuild them. Replies only. |
17+
| `update-conda-forge-pinning` | Move the shared conda-forge pinning (`shared/pinning/conda_forge.yaml`) to the latest version, with migrations selected for all distributions; re-renders `conda_build_config.yaml` of those that follow it. Runs weekly. |
18+
- type: dropdown
19+
id: command
20+
attributes:
21+
label: Command
22+
options: [update-rosdistro-snapshot, find-stale-packages, update-conda-forge-pinning]
23+
validations:
24+
required: true
25+
- type: dropdown
26+
id: distro
27+
attributes:
28+
label: Distribution
29+
description: For update-rosdistro-snapshot and find-stale-packages
30+
options: [rolling, lyrical, kilted, jazzy, humble]

‎.github/workflows/bot.yml‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
# robostack-bot: routine maintenance (see tools/maintenance.py).
2+
#
3+
# - weekly: refresh every distribution's rosdistro snapshot (one PR per distribution)
4+
# and the shared conda-forge pinning (one PR)
5+
# - on demand (owners, members and collaborators):
6+
# Actions > robostack-bot > Run workflow, an issue from the "robostack-bot command"
7+
# template, or a comment: @robostack-bot <command> [<distro>]
8+
# commands: update-rosdistro-snapshot <distro> | find-stale-packages <distro> |
9+
# update-conda-forge-pinning
10+
#
11+
# PRs are opened with the robostack-bot GitHub App (vars.ROBOSTACK_BOT_APP_ID +
12+
# secrets.ROBOSTACK_BOT_PRIVATE_KEY, falling back to secrets.GHA_PAT) so CI runs on them.
13+
name: robostack-bot
14+
15+
on:
16+
schedule:
17+
- cron: "17 3 * * 1"
18+
workflow_dispatch:
19+
inputs:
20+
command:
21+
type: choice
22+
options: [update-rosdistro-snapshot, find-stale-packages, update-conda-forge-pinning]
23+
distro:
24+
description: Distribution (for update-rosdistro-snapshot / find-stale-packages)
25+
required: false
26+
issue_comment:
27+
types: [created]
28+
issues:
29+
types: [opened]
30+
31+
permissions:
32+
contents: read
33+
34+
jobs:
35+
plan:
36+
if: >-
37+
github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' ||
38+
(github.event_name == 'issue_comment' && contains(github.event.comment.body, '@robostack-bot')) ||
39+
(github.event_name == 'issues' && contains(github.event.issue.labels.*.name, 'robostack-bot'))
40+
runs-on: ubuntu-latest
41+
outputs:
42+
jobs: ${{ steps.plan.outputs.jobs }}
43+
steps:
44+
- uses: actions/checkout@v6
45+
with:
46+
persist-credentials: false
47+
- uses: prefix-dev/setup-pixi@v0.10.2
48+
with:
49+
frozen: true
50+
pixi-version: v0.75.0
51+
- id: plan
52+
env:
53+
EVENT: ${{ github.event_name }}
54+
COMMAND: ${{ inputs.command }}
55+
DISTRO: ${{ inputs.distro }}
56+
BODY: ${{ github.event.comment.body || github.event.issue.body }}
57+
ASSOCIATION: ${{ github.event.comment.author_association || github.event.issue.author_association }}
58+
run: |
59+
case "$EVENT" in
60+
schedule)
61+
jobs=$(pixi run -q python tools/robostack.py changed-distros --base HEAD --all \
62+
| python3 -c 'import json,sys; print(json.dumps([{"command":"update-rosdistro-snapshot","distro":d} for d in json.load(sys.stdin)] + [{"command":"update-conda-forge-pinning","distro":""}]))') ;;
63+
workflow_dispatch)
64+
jobs=$(python3 -c 'import json,os; print(json.dumps([{"command":os.environ["COMMAND"],"distro":os.environ["DISTRO"]}]))') ;;
65+
*)
66+
parsed=$(pixi run -q rs parse-command --body "$BODY" --association "$ASSOCIATION")
67+
jobs=$(echo "$parsed" | python3 -c 'import json,sys; c=json.load(sys.stdin); print(json.dumps([c] if c["command"] else []))') ;;
68+
esac
69+
echo "jobs=$jobs" | tee -a "$GITHUB_OUTPUT"
70+
71+
run:
72+
needs: plan
73+
if: needs.plan.outputs.jobs != '[]'
74+
runs-on: ubuntu-latest
75+
strategy:
76+
fail-fast: false
77+
matrix:
78+
include: ${{ fromJSON(needs.plan.outputs.jobs) }}
79+
permissions:
80+
contents: write
81+
pull-requests: write
82+
issues: write
83+
steps:
84+
- name: Bot token
85+
id: app
86+
if: vars.ROBOSTACK_BOT_APP_ID != ''
87+
uses: actions/create-github-app-token@v2
88+
with:
89+
app-id: ${{ vars.ROBOSTACK_BOT_APP_ID }}
90+
private-key: ${{ secrets.ROBOSTACK_BOT_PRIVATE_KEY }}
91+
- uses: actions/checkout@v6
92+
with:
93+
token: ${{ steps.app.outputs.token || secrets.GHA_PAT || github.token }}
94+
- uses: prefix-dev/setup-pixi@v0.10.2
95+
with:
96+
frozen: true
97+
pixi-version: v0.75.0
98+
- name: robostack-bot ${{ matrix.command }} ${{ matrix.distro }}
99+
id: bot
100+
env:
101+
COMMAND: ${{ matrix.command }}
102+
DISTRO: ${{ matrix.distro }}
103+
GITHUB_TOKEN: ${{ github.token }}
104+
run: |
105+
set +e
106+
case "$COMMAND" in
107+
update-rosdistro-snapshot) pixi run rs "$DISTRO" update-snapshot --summary "$RUNNER_TEMP/summary.md" ;;
108+
find-stale-packages) pixi run rs "$DISTRO" find-stale --summary "$RUNNER_TEMP/summary.md" ;;
109+
update-conda-forge-pinning) pixi run rs update-pinning --summary "$RUNNER_TEMP/summary.md" ;;
110+
esac
111+
status=$?
112+
cat "$RUNNER_TEMP/summary.md" >> "$GITHUB_STEP_SUMMARY" 2>/dev/null
113+
exit $status
114+
- name: Open or update the pull request
115+
if: steps.bot.outputs.changed == 'true'
116+
uses: peter-evans/create-pull-request@v7
117+
with:
118+
token: ${{ steps.app.outputs.token || secrets.GHA_PAT || github.token }}
119+
branch: bot/${{ matrix.command }}${{ matrix.distro && format('-{0}', matrix.distro) || '' }}
120+
delete-branch: true
121+
commit-message: ${{ steps.bot.outputs.title }}
122+
title: ${{ steps.bot.outputs.title }}
123+
body-path: ${{ runner.temp }}/summary.md
124+
labels: robostack-bot
125+
- name: Reply on the issue / pull request
126+
if: always() && (github.event_name == 'issue_comment' || github.event_name == 'issues')
127+
env:
128+
GH_TOKEN: ${{ github.token }}
129+
NUMBER: ${{ github.event.issue.number }}
130+
TITLE: ${{ steps.bot.outputs.title || 'failed' }}
131+
run: |
132+
{
133+
echo "**robostack-bot ${{ matrix.command }} ${{ matrix.distro }}**: $TITLE"
134+
echo
135+
[ -f "$RUNNER_TEMP/summary.md" ] && cat "$RUNNER_TEMP/summary.md"
136+
echo
137+
echo "[Workflow run]($GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID)"
138+
} > "$RUNNER_TEMP/comment.md"
139+
gh issue comment "$NUMBER" --repo "$GITHUB_REPOSITORY" --body-file "$RUNNER_TEMP/comment.md"
140+
if [ "$GITHUB_EVENT_NAME" = issues ] && [ "${{ job.status }}" = success ]; then
141+
gh issue close "$NUMBER" --repo "$GITHUB_REPOSITORY"
142+
fi

‎.github/workflows/main.yml‎

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,11 @@
22
# merge touched, one build branch per distribution and platform
33
# (buildbranch_<distro>_<platform>, workflows .github/workflows/build_<distro>_*.yml).
44
#
5-
# Pushing build branches starts the real builds and uploads, so in this prototype
6-
# it only does that in the RoboStack organisation; elsewhere (or with dry_run) the
7-
# generated workflows are uploaded as an artifact instead.
5+
# Pushing build branches starts the real builds, so this only pushes in the RoboStack
6+
# organisation or where the repository variable PUSH_BUILD_BRANCHES is 'true'; otherwise
7+
# (or with dry_run) the generated workflows are uploaded as an artifact. Outside the
8+
# RoboStack org the builds don't upload unless ROBOSTACK_UPLOAD_CHANNEL names a (test)
9+
# prefix.dev channel.
810
name: Generate build branches
911

1012
on:
@@ -56,8 +58,16 @@ jobs:
5658
distro: ${{ fromJSON(needs.plan.outputs.distros) }}
5759
env:
5860
DISTRO: ${{ matrix.distro }}
59-
PUSH: ${{ github.repository_owner == 'RoboStack' && !inputs.dry_run }}
61+
# Push in the RoboStack org, or where the repository variable PUSH_BUILD_BRANCHES is 'true'.
62+
PUSH: ${{ (github.repository_owner == 'RoboStack' || vars.PUSH_BUILD_BRANCHES == 'true') && !inputs.dry_run }}
6063
steps:
64+
- name: Bot token (pushing workflow files needs more than GITHUB_TOKEN)
65+
id: app
66+
if: vars.ROBOSTACK_BOT_APP_ID != ''
67+
uses: actions/create-github-app-token@v2
68+
with:
69+
app-id: ${{ vars.ROBOSTACK_BOT_APP_ID }}
70+
private-key: ${{ secrets.ROBOSTACK_BOT_PRIVATE_KEY }}
6171
- uses: actions/checkout@v6
6272
with:
6373
persist-credentials: false
@@ -69,7 +79,7 @@ jobs:
6979
- name: Generate and push the build branches
7080
env:
7181
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
72-
PUSH_TOKEN: ${{ secrets.GHA_PAT }}
82+
PUSH_TOKEN: ${{ steps.app.outputs.token || secrets.GHA_PAT }}
7383
run: |
7484
set -e
7585
git config --local user.email "action@github.com"
@@ -89,7 +99,7 @@ jobs:
8999
branch="buildbranch_${DISTRO}_${platform//-/_}"
90100
git add -f . ':!.pixi'
91101
git commit -qm "Build files for $DISTRO $platform $(date '+%Y-%m-%d-%H%M')"
92-
git push "https://${GITHUB_ACTOR}:${PUSH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" "HEAD:$branch" --force
102+
git push "https://x-access-token:${PUSH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" "HEAD:$branch" --force
93103
fi
94104
done
95105
- uses: actions/upload-artifact@v6

‎.github/workflows/testpr.yml‎

Lines changed: 6 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ jobs:
4646
fi
4747
echo "distros=$distros" | tee -a "$GITHUB_OUTPUT"
4848
49-
sort-check:
49+
check:
5050
runs-on: ubuntu-latest
5151
steps:
5252
- uses: actions/checkout@v6
@@ -56,12 +56,8 @@ jobs:
5656
with:
5757
frozen: true
5858
pixi-version: v0.75.0
59-
- name: Check sorting
60-
run: pixi run sort --check
61-
- name: Check conda_build_config.yaml is up to date
62-
run: |
63-
pixi run render-pinning
64-
git diff --exit-code -- 'distros/*/conda_build_config.yaml'
59+
- name: Sanity checks (every distribution assembles, generated files up to date, sorted)
60+
run: pixi run rs check
6561

6662
build:
6763
needs: plan
@@ -81,8 +77,8 @@ jobs:
8177
env:
8278
DISTRO: ${{ matrix.distro }}
8379
PLATFORM: ${{ matrix.target.platform }}
84-
# rattler-build writes to ./output of the distribution, or to CONDA_BLD_PATH (win-64)
85-
CACHE_DIR: ${{ matrix.target.platform == 'win-64' && 'C:/bld/win-64' || format('distros/{0}/output/{1}', matrix.distro, matrix.target.platform) }}
80+
# rattler-build writes to distros/<d>/work/output, or to CONDA_BLD_PATH (win-64)
81+
CACHE_DIR: ${{ matrix.target.platform == 'win-64' && 'C:/bld/win-64' || format('distros/{0}/work/output/{1}', matrix.distro, matrix.target.platform) }}
8682
steps:
8783
- uses: actions/checkout@v6
8884
with:
@@ -145,7 +141,7 @@ jobs:
145141
- name: Apply ci.yaml cache controls
146142
shell: bash -l {0}
147143
run: |
148-
pixi run python tools/ci_config.py --cache-dir "$CACHE_DIR" --config "distros/$DISTRO/ci.yaml" --vinca "distros/$DISTRO/vinca.yaml"
144+
pixi run python tools/ci_config.py --cache-dir "$CACHE_DIR" --config "distros/$DISTRO/ci.yaml" --vinca "distros/$DISTRO/work/vinca.yaml"
149145
mkdir -p "$CACHE_DIR"
150146
pixi run rattler-index fs "$CACHE_DIR/.." --force
151147

‎.gitignore‎

Lines changed: 2 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,5 @@
22
__pycache__
33
*.egg-info
44
.DS_Store
5-
# generated per distribution by tools/robostack.py
6-
distros/*/recipes/
7-
distros/*/recipes_only_patch/
8-
distros/*/tests/
9-
distros/*/vinca_pinning.yaml
10-
distros/*/output/
11-
distros/*/*.json
12-
distros/*/buildorder.txt
5+
# assembled per distribution by tools/robostack.py (prepare)
6+
distros/*/work/

‎.scripts/build_unix.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ export PYTHONUNBUFFERED=1
1818

1919
# ROBOSTACK_DISTRO is set by the generated build workflow (tools/robostack.py gha).
2020
: "${ROBOSTACK_DISTRO:?ROBOSTACK_DISTRO is not set}"
21-
cd "distros/${ROBOSTACK_DISTRO}"
21+
cd "distros/${ROBOSTACK_DISTRO}/work"
2222
export FEEDSTOCK_ROOT=`pwd`
2323
export "CONDA_BLD_PATH=$HOME/conda-bld/"
2424

‎.scripts/build_win.bat‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ rmdir /Q/S C:\Strawberry\
88
rmdir /Q/S "C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\"
99

1010
:: ROBOSTACK_DISTRO is set by the generated build workflow (tools/robostack.py gha).
11-
cd distros\%ROBOSTACK_DISTRO%
11+
cd distros\%ROBOSTACK_DISTRO%\work
1212
set "FEEDSTOCK_ROOT=%cd%"
1313

1414
mkdir %CONDA_BLD_PATH%

‎AGENTS.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,11 @@
33
Working notes for coding agents in the RoboStack all-distributions repository.
44

55
- Every distribution lives in `distros/<distro>/` (only distro-specific data); shared
6-
data is in `shared/`, tools in `tools/`. See README.md for the layout.
6+
data is in `shared/` (vinca.yaml, pkg_additional_info.yaml, dependencies.yaml,
7+
robostack.yaml, pinning, tests), tools in `tools/`. See README.md for the layout and
8+
how they combine. Never edit `distros/<d>/work/` (generated).
9+
- A fix that applies to every distribution goes into `shared/`; a version-specific one
10+
into `distros/<d>/`.
711
- Run everything per distribution with `pixi run rs <distro> <task>`
812
(`generate-recipes`, `build`, `build-one <pkg>`, `check-patches`, `check-deps`,
913
`create-snapshot`, `render-pinning`, `sort`).

‎README.md‎

Lines changed: 38 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,24 +10,57 @@ The data is copied from those repositories (rolling, humble, lyrical and kilted
1010
## Layout
1111

1212
```
13-
distros/<distro>/ everything specific to one distribution
13+
distros/<distro>/ only what is specific to one distribution
1414
distro.yaml channel, upload target, conda-forge pinning version/migrations, own pins
15-
vinca.yaml package selection, build number, mutex
16-
pkg_additional_info.yaml, rosdistro_additional_recipes.yaml, rosdistro_snapshot.yaml
17-
patch/
15+
vinca.yaml ros_distro, build number, mutex, its own package selection
16+
pkg_additional_info.yaml its own per-package settings (build numbers, version-specific cmake args)
17+
patch/ its patches; patch/dependencies.yaml its own dependency fixes
18+
rosdistro_snapshot.yaml, rosdistro_additional_recipes.yaml
1819
ci.yaml temporary PR-build controls (full rebuild, cache evictions)
1920
conda_build_config.yaml generated: `pixi run rs <distro> render-pinning`
21+
work/ generated, git-ignored: what vinca and rattler-build actually run on
2022
shared/ the same for every distribution
23+
vinca.yaml settings and packages selected in every distribution
24+
pkg_additional_info.yaml per-package settings that are the same everywhere
25+
dependencies.yaml dependency fixes that are the same everywhere
2126
robostack.yaml, packages-ignore.yaml rosdep key -> conda package mapping
2227
pinning/ conda-forge pinning version, migrations and overrides
2328
tests/ package tests (ros2-<pkg>.yaml; *.jinja for distro-specific bits)
24-
tools/robostack.py `pixi run rs <distro> <task>`: runs everything for one distribution
29+
tools/robostack.py `pixi run rs <distro> <task>`: assembles work/ and runs everything there
2530
tools/*.py check_patches_clean_apply, check_dependency_compat, ...
2631
.scripts/ staged build scripts (build_unix.sh, build_win.bat)
2732
.github/workflows/ testpr.yml (PR builds), main.yml (staged build branches)
2833
pixi.toml one environment for all distributions (one vinca, one rattler-build)
2934
```
3035

36+
### How shared and distribution files combine
37+
38+
`pixi run rs <distro> <task>` first assembles `distros/<distro>/work/`:
39+
40+
- **`vinca.yaml`:** lists (`packages_select_by_deps`, `packages_skip_by_deps`,
41+
`packages_remove_from_deps`) are `shared/` + the distribution's. Other keys come
42+
from the distribution if it sets them, else from `shared/`. A distribution that
43+
differs from a shared setting just sets it, e.g. kilted's `package_name_mode: legacy`.
44+
`skip_existing`, `conda_index`, `patch_dir` and the snapshot paths are filled in.
45+
- **`pkg_additional_info.yaml`, `patch/dependencies.yaml`:** per package, the
46+
distribution's keys win over the shared ones. For example, a shared
47+
`additional_cmake_args` plus the distribution's own `build_number`.
48+
- **`tests/`:** shared tests, with `*.jinja` rendered for the distribution.
49+
50+
**What is shared:** an entry moves to `shared/` when every distribution that has it
51+
agrees, and no other distribution contains that package. Adding the entry there
52+
can't change anything.
53+
54+
**Checked by regenerating the full linux-64 recipe sets:** the split leaves the recipes
55+
unchanged. rolling (1,927), humble (2,214) and lyrical (1,885) are byte-identical before
56+
and after.
57+
58+
**Sharing more is a decision:**
59+
- a package selected in only some distributions;
60+
- a cmake flag that differs between versions.
61+
62+
For each, either keep it per distribution or decide to unify, as with `robostack.yaml`.
63+
3164
## Working with it
3265

3366
```bash

0 commit comments

Comments
 (0)