Skip to content

build: unify Python setup with bootstrap - #2204

Merged
NickeZ merged 5 commits into
masterfrom
nickez-agent/pin-protobuf-generators
Nov 16, 2026
Merged

NickeZ merged 5 commits into
masterfrom
nickez-agent/pin-protobuf-generators

Conversation

@NickeZ

@NickeZ NickeZ commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

Fresh firmware builds can regenerate protobuf bindings with whichever protoc and mypy-protobuf happen to be on PATH. This can change generated code or require a newer runtime, while separate local and CI setup makes the tool versions easy to diverge.

Have make bootstrap create or reuse the repository .venv and install Python dependencies, including protoc 21.2 and mypy-protobuf 3.6.0, from py/requirements.txt. Install the checkout's bitbox02 package in editable mode so scripts use its current APIs and pick up local changes without reinstallation. Top-level Make commands prefer that environment so builds use the installed tools without shell activation.

Run bootstrap in the shared CI build, test and lint jobs, letting it initialize submodules and configure Cargo. Use the same requirements in the development image, track them as an image input, and bump the container version to 60. Document the common setup for host and container development.

@NickeZ
NickeZ requested a review from benma November 12, 2026 10:56
@NickeZ
NickeZ force-pushed the nickez-agent/pin-protobuf-generators branch from 6461cbb to d3862a6 Compare November 12, 2026 11:21
GitHub Actions' hosted Ubuntu runners do not support IPv6 connectivity.
When the LLVM signing-key download resolves to IPv6, it fails with
"Network is unreachable" and prevents the development image from building.

Force IPv4 for APT and the Dockerfile's wget and curl downloads so they
use the network available on the CI runners.
Fresh firmware builds can regenerate protobuf bindings with whichever
protoc and mypy-protobuf happen to be on PATH. This can change generated
code or require a newer runtime, while separate local and CI setup makes
the tool versions easy to diverge.

Have make bootstrap create or reuse the repository .venv and install
Python dependencies, including protoc 21.2 and mypy-protobuf 3.6.0, from
py/requirements.txt. Install the checkout's bitbox02 package in editable
mode so scripts use its current APIs and pick up local changes without
reinstallation. Top-level Make commands prefer that environment so
builds use the installed tools without shell activation.

Run bootstrap in the shared CI build, test and lint jobs, letting it
initialize submodules and configure Cargo. Use the same requirements in
the development image, track them as an image input, and bump the
container version to 60. Document the common setup for host and container
development.
CI only checked Rust protobuf bindings, leaving committed Python modules
and type stubs unchecked. Existing output files could also hide bindings
that are no longer generated.

Delete both generated output directories before regenerating Rust and
Python bindings. Fail on modified, deleted or untracked files and print
the diff and affected paths so missing regeneration is visible in CI.
@NickeZ
NickeZ force-pushed the nickez-agent/pin-protobuf-generators branch from d3862a6 to ff3f747 Compare November 12, 2026 13:33
Direct Cargo builds invoke protobuf generation without the Makefile's
local virtualenv PATH setup, so they can select a different protoc version.

Set PROTOC to the repository's .venv/bin/protoc when it is executable.
Otherwise, retain prost's existing PROTOC override and PATH lookup so
container and global installations continue to work without bootstrapping.
Comment thread Makefile
python3 scripts/build_config.py $(CONFIG_ARGS)

bootstrap:
test -d .venv || python3 -m venv .venv

@benma-agent benma-agent Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reusing .venv solely because the directory exists breaks switching between host and container. dockerdev bind-mounts the checkout, including .venv, but the environment's interpreter links and installed packages are specific to the Python version and platform that created it. Python's documentation describes virtual environments as inherently non-portable.

I reproduced this with a .venv created on a Python 3.13 host and mounted at the same path in the existing Python 3.10 development image. .venv/bin/python3 points to /usr/bin/python3, which resolves to Python 3.10 in the container, while the installed packages are under lib/python3.13/site-packages. Inside the container:

$ .venv/bin/python3 -m pip --version
/tmp/bitbox-pr2204-cvaef263/.venv/bin/python3: No module named pip

$ .venv/bin/protoc --version
Traceback (most recent call last):
  File "/tmp/bitbox-pr2204-cvaef263/.venv/bin/protoc", line 5, in <module>
    from protoc import exec_protoc
ModuleNotFoundError: No module named 'protoc'

make bootstrap skips creation and then fails at the pip install. The new PATH/PROTOC preference also makes builds select this broken environment even when the container has working tools installed.

Please keep host and container virtualenvs separate, or use the image's installed dependencies in the container.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know, and this was a known limitation. Is this a real problem for you? (I assumed you did "make clean" when going between host/container. Which is reasonable because cmake sets up paths to all the other tools that are environment specific.)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed by adding a make distclean target and documented the limitation

Switching between host and container builds requires clearing
Python packages and build artifacts from the previous environment.
The existing clean target leaves the repository virtualenv in place.

Add a phony distclean target that depends on clean and removes .venv.
Document running it before switching environments and bootstrapping
again in the new environment.

@benma benma left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK

@NickeZ
NickeZ merged commit d072b0f into master Nov 16, 2026
57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants