Repository navigation
build: unify Python setup with bootstrap - #2204
Conversation
6461cbb to
d3862a6
Compare
GitHub Actions' hosted Ubuntu runners do not support IPv6 connectivity. When the LLVM signing-key download resolves to IPv6, it fails with "Network is unreachable" and prevents the development image from building. Force IPv4 for APT and the Dockerfile's wget and curl downloads so they use the network available on the CI runners.
Fresh firmware builds can regenerate protobuf bindings with whichever protoc and mypy-protobuf happen to be on PATH. This can change generated code or require a newer runtime, while separate local and CI setup makes the tool versions easy to diverge. Have make bootstrap create or reuse the repository .venv and install Python dependencies, including protoc 21.2 and mypy-protobuf 3.6.0, from py/requirements.txt. Install the checkout's bitbox02 package in editable mode so scripts use its current APIs and pick up local changes without reinstallation. Top-level Make commands prefer that environment so builds use the installed tools without shell activation. Run bootstrap in the shared CI build, test and lint jobs, letting it initialize submodules and configure Cargo. Use the same requirements in the development image, track them as an image input, and bump the container version to 60. Document the common setup for host and container development.
CI only checked Rust protobuf bindings, leaving committed Python modules and type stubs unchecked. Existing output files could also hide bindings that are no longer generated. Delete both generated output directories before regenerating Rust and Python bindings. Fail on modified, deleted or untracked files and print the diff and affected paths so missing regeneration is visible in CI.
d3862a6 to
ff3f747
Compare
Direct Cargo builds invoke protobuf generation without the Makefile's local virtualenv PATH setup, so they can select a different protoc version. Set PROTOC to the repository's .venv/bin/protoc when it is executable. Otherwise, retain prost's existing PROTOC override and PATH lookup so container and global installations continue to work without bootstrapping.
| python3 scripts/build_config.py $(CONFIG_ARGS) | ||
|
|
||
| bootstrap: | ||
| test -d .venv || python3 -m venv .venv |
There was a problem hiding this comment.
Reusing .venv solely because the directory exists breaks switching between host and container. dockerdev bind-mounts the checkout, including .venv, but the environment's interpreter links and installed packages are specific to the Python version and platform that created it. Python's documentation describes virtual environments as inherently non-portable.
I reproduced this with a .venv created on a Python 3.13 host and mounted at the same path in the existing Python 3.10 development image. .venv/bin/python3 points to /usr/bin/python3, which resolves to Python 3.10 in the container, while the installed packages are under lib/python3.13/site-packages. Inside the container:
$ .venv/bin/python3 -m pip --version
/tmp/bitbox-pr2204-cvaef263/.venv/bin/python3: No module named pip
$ .venv/bin/protoc --version
Traceback (most recent call last):
File "/tmp/bitbox-pr2204-cvaef263/.venv/bin/protoc", line 5, in <module>
from protoc import exec_protoc
ModuleNotFoundError: No module named 'protoc'
make bootstrap skips creation and then fails at the pip install. The new PATH/PROTOC preference also makes builds select this broken environment even when the container has working tools installed.
Please keep host and container virtualenvs separate, or use the image's installed dependencies in the container.
There was a problem hiding this comment.
I know, and this was a known limitation. Is this a real problem for you? (I assumed you did "make clean" when going between host/container. Which is reasonable because cmake sets up paths to all the other tools that are environment specific.)
There was a problem hiding this comment.
fixed by adding a make distclean target and documented the limitation
Switching between host and container builds requires clearing Python packages and build artifacts from the previous environment. The existing clean target leaves the repository virtualenv in place. Add a phony distclean target that depends on clean and removes .venv. Document running it before switching environments and bootstrapping again in the new environment.
Fresh firmware builds can regenerate protobuf bindings with whichever protoc and mypy-protobuf happen to be on PATH. This can change generated code or require a newer runtime, while separate local and CI setup makes the tool versions easy to diverge.
Have make bootstrap create or reuse the repository .venv and install Python dependencies, including protoc 21.2 and mypy-protobuf 3.6.0, from py/requirements.txt. Install the checkout's bitbox02 package in editable mode so scripts use its current APIs and pick up local changes without reinstallation. Top-level Make commands prefer that environment so builds use the installed tools without shell activation.
Run bootstrap in the shared CI build, test and lint jobs, letting it initialize submodules and configure Cargo. Use the same requirements in the development image, track them as an image input, and bump the container version to 60. Document the common setup for host and container development.