Merge branch 'nickez-agent/pin-protobuf-generators'
What changed, and why it matters
This commit is a build-system and developer-environment change. It pins the protobuf compiler and related Python tools to specific versions via a project-managed Python virtual environment, instead of relying on whatever version a developer or CI worker happens to have installed. There is no change to firmware code, device behavior, or security logic, and no security vulnerability is being fixed.
No security action required. Treat as a normal build-hygiene improvement. Reviewers may optionally verify that pinned versions match the previously used protoc 21.x and that CI reproducibly generates identical protobuf bindings.
Security signals we found
Pinning build tools to exact versions reduces non-determinism in generated code
Moves protobuf compiler from system/Docker-installed binary to a version-locked Python wheel
No changes to firmware, cryptography, authentication, or device protocol logic
Evidence from the diff
The merge commit ‘nickez-agent/pin-protobuf-generators’ refactors how the protobuf compiler (protoc) and Python protobuf/mypy generators are provisioned. Key changes: (1) py/requirements.txt now pins protoc-wheel-0==21.2 and mypy-protobuf==3.6.0; (2) make bootstrap creates a .venv and installs those requirements plus the local bitbox02 package in editable mode; (3) scripts/generate-protobuf-rust.sh prefers .venv/bin/protoc if present; (4) .ci/check-protobuf now also regenerates and checks Python protobuf bindings; (5) Dockerfile removes the system-wide protoc install and adds python3-venv; (6) CI workflows replace ad-hoc protoc/submodule steps with make bootstrap. The change improves reproducibility and supply-chain hygiene but does not alter runtime firmware or fix a reported security issue.
Changed components
Build system (Makefile, Dockerfile, CI workflows)Python development environment (py/requirements.txt, .venv bootstrap)Protobuf code-generation scripts (scripts/generate-protobuf-rust.sh, .ci/check-protobuf)Inspect captured patch +107 / −64
### .ci/check-container-sources-modified
@@ -7,7 +7,7 @@
set -e
-if git diff --name-only HEAD^1 HEAD | grep -E '^(\.containerversion|Dockerfile)' >/dev/null; then
+if git diff --name-only HEAD^1 HEAD | grep -E '^(\.containerversion|Dockerfile|py/requirements\.txt)' >/dev/null; then
echo "true"
exit
fi
### .ci/check-protobuf
@@ -2,5 +2,16 @@
set -euo pipefail
+rust_dir=src/rust/bitbox-proto
+python_dir=py/bitbox02/bitbox02/communication/generated
+
+rm -rf -- "$rust_dir/src/generated" "$python_dir"
./scripts/generate-protobuf-rust.sh
-git diff --exit-code -- src/rust/bitbox-proto
+make -C py/bitbox02
+
+changes=$(git status --porcelain --untracked-files=all -- "$rust_dir" "$python_dir")
+if [[ -n "$changes" ]]; then
+ git diff -- "$rust_dir" "$python_dir"
+ printf '%s\n' "$changes"
+ exit 1
+fi
### .containerversion
@@ -1 +1 @@
-59
+60
### .github/workflows/ci-common.yml
@@ -54,6 +54,11 @@ jobs:
- name: Mark directory as safe
run: git config --global --add safe.directory $GITHUB_WORKSPACE
+ - name: Bootstrap
+ run: |
+ make bootstrap
+ echo "$PWD/.venv/bin" >> "$GITHUB_PATH"
+
- name: Lint python (black, mypy, pylint)
run: |
make -C py clean
@@ -79,11 +84,13 @@ jobs:
# Unfortunately linting the sources requires running cmake, so we need to do a full checkout
fetch-depth: 0
fetch-tags: true
- submodules: recursive
- name: Mark directory as safe
run: git config --global --add safe.directory $GITHUB_WORKSPACE
+ - name: Bootstrap
+ run: make bootstrap
+
- name: Lint src code style (clang-format, rustfmt)
run: ./.ci/check-style
@@ -104,7 +111,12 @@ jobs:
- name: Mark directory as safe
run: git config --global --add safe.directory $GITHUB_WORKSPACE
- - name: Check protobuf Rust bindings
+ - name: Bootstrap
+ run: |
+ make bootstrap
+ echo "$PWD/.venv/bin" >> "$GITHUB_PATH"
+
+ - name: Check protobuf bindings
run: ./.ci/check-protobuf
unit-tests:
@@ -117,11 +129,13 @@ jobs:
with:
fetch-depth: 0
fetch-tags: true
- submodules: recursive
- name: Mark directory as safe
run: git config --global --add safe.directory $GITHUB_WORKSPACE
+ - name: Bootstrap
+ run: make bootstrap
+
- name: Build unit-tests
run: make -j$(($(nproc) + 1)) unit-test
@@ -146,16 +160,12 @@ jobs:
with:
fetch-depth: 0
fetch-tags: true
- submodules: recursive
- name: Install system dependencies
run: brew install cbindgen bindgen hidapi
- - name: Install protoc
- uses: arduino/setup-protoc@v3
- with:
- version: "21.x"
- repo-token: ${{ secrets.GITHUB_TOKEN }}
+ - name: Bootstrap
+ run: make bootstrap
- name: install rust
uses: actions-rust-lang/setup-rust-toolchain@v1
@@ -177,11 +187,13 @@ jobs:
with:
fetch-depth: 0
fetch-tags: true
- submodules: recursive
- name: Mark directory as safe
run: git config --global --add safe.directory $GITHUB_WORKSPACE
+ - name: Bootstrap
+ run: make bootstrap
+
- name: Run unit tests under valgrind
run: make run-valgrind-on-unit-tests SANITIZE=OFF
@@ -239,13 +251,12 @@ jobs:
ref: ${{ github.ref }}
fetch-depth: 0
fetch-tags: true
- submodules: recursive
- name: Mark directory as safe
run: git config --global --add safe.directory $GITHUB_WORKSPACE
- - name: Bootstrap Cargo config
- run: ./scripts/bootstrap-cargo-config
+ - name: Bootstrap
+ run: make bootstrap
- name: Configure product
run: make config CONFIG_ARGS="--product ${{ matrix.product }} --edition ${{ matrix.edition }} --debug ${{ matrix.debug || 'no' }} --locked ${{ matrix.locked || 'no' }} --development ${{ matrix.development || 'yes' }}"
@@ -313,16 +324,12 @@ jobs:
ref: ${{ github.ref }}
fetch-depth: 0
fetch-tags: true
- submodules: recursive
- name: Install system dependencies
run: brew install cbindgen bindgen hidapi
- - name: Install protoc
- uses: arduino/setup-protoc@v3
- with:
- version: "21.x"
- repo-token: ${{ secrets.GITHUB_TOKEN }}
+ - name: Bootstrap
+ run: make bootstrap
- name: install rust
uses: actions-rust-lang/setup-rust-toolchain@v1
@@ -370,14 +377,14 @@ jobs:
with:
fetch-depth: 0
fetch-tags: true
- submodules: recursive
- name: Mark directory as safe
run: git config --global --add safe.directory $GITHUB_WORKSPACE
- name: Build documentation
run: |
make bootstrap
+ . .venv/bin/activate
cmake -S . -B build -DDOC_GRAPHS=NO
make -C build -j$(($(nproc)+1)) doc
### BUILD.md
@@ -20,6 +20,11 @@ For automation and editor integrations, use `./scripts/dev_exec.sh <command>` as
entrypoint. It runs commands natively on the host by default and can be switched to container
execution with `BITBOX_FW_EXEC_MODE=docker`.
+Before switching between host and container development environments, you must run
+`make distclean` to remove `.venv` and build artifacts from the previous environment. On the host,
+use `./scripts/dev_exec.sh make distclean`. After switching, run `make bootstrap` in the new
+environment before building.
+
Accessing USB devices, like the flashing tool and the bitbox, is easier outside
of the container. So it is recommended to install the J-Link Software on your
development machine to follow the instructions below.
@@ -99,7 +104,7 @@ Make sure you have [Homebrew](https://brew.sh) installed. Install the
dependencies with:
```sh
-brew install hidapi cmake protobuf@21
+brew install hidapi cmake
brew install automake libtool
brew tap osx-cross/arm
brew install arm-gcc-bin
@@ -152,18 +157,36 @@ cd bitbox02-firmware
### Configure and build
-Inside the development container, run `make bootstrap` once to initialize submodules. From the
-host, use `./scripts/dev_exec.sh make <target>` for builds (or set
-`BITBOX_FW_EXEC_MODE=docker` to use the container toolchain).
+Run `make bootstrap` once to initialize submodules, configure Cargo, and install the Python
+dependencies, including pinned protobuf generators, from [py/requirements.txt](py/requirements.txt).
+It also installs the local `bitbox02` Python package in editable mode.
+
+Bootstrap creates `.venv/` in the repository root if it is missing and reuses it otherwise,
+including inside the development container. It installs the dependencies there regardless of which
+virtualenv is active in your shell. All top-level Make targets prefer `.venv/bin` on `PATH`, so
+builds use the installed tools automatically.
+
+Run the following from the repository root:
```sh
+make bootstrap
make config
make firmware
make factorysetup
make bootloader-stage0
make bootloader-stage1
```
+For Python commands outside Make, activate the environment from the repository root in each new
+shell:
+
+```sh
+. .venv/bin/activate
+```
+
+Use `./scripts/dev_exec.sh make <target>` for host builds (or set `BITBOX_FW_EXEC_MODE=docker` to use
+the container toolchain).
+
`make config` remembers selections in the git-ignored `config.mk`. It prompts for product, board,
edition (`multi` or `btc-only`), debug builds, locked hardware, development mode, probe software, and
SWD speed in kHz, automatically selecting fields with one choice. `DEBUG` defaults to `no`; `SWD_SPEED` defaults to
### Dockerfile
@@ -12,12 +12,14 @@ ENV DEBIAN_FRONTEND=noninteractive
ARG TARGETPLATFORM
ARG TARGETARCH
+RUN echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4
+
RUN apt-get update && apt-get upgrade -y && apt-get install -y wget nano rsync curl gnupg2 jq unzip bzip2 xz-utils
# for clang-*-22, see https://apt.llvm.org/
RUN echo "deb http://apt.llvm.org/jammy/ llvm-toolchain-jammy-22 main" >> /etc/apt/sources.list && \
echo "deb-src http://apt.llvm.org/jammy/ llvm-toolchain-jammy-22 main" >> /etc/apt/sources.list && \
- wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add -
+ wget -4 -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add -
# Install gcc8-arm-none-eabi
RUN if [ "${TARGETPLATFORM}" = "linux/arm64" ]; then \
@@ -27,7 +29,7 @@ RUN if [ "${TARGETPLATFORM}" = "linux/arm64" ]; then \
GNU_TOOLCHAIN=https://developer.arm.com/-/media/Files/downloads/gnu/15.2.rel1/binrel/arm-gnu-toolchain-15.2.rel1-x86_64-arm-none-eabi.tar.xz \
GNU_TOOLCHAIN_HASH=597893282ac8c6ab1a4073977f2362990184599643b4c5ee34870a8215783a16; \
fi; \
- curl -sSL -o gcc.tar.xz ${GNU_TOOLCHAIN} && \
+ curl -4 -sSL -o gcc.tar.xz ${GNU_TOOLCHAIN} && \
echo "$GNU_TOOLCHAIN_HASH gcc.tar.xz" | sha256sum -c && \
tar -xvf gcc.tar.xz -C /usr/local --strip-components=1 && \
rm -f gcc.tar.xz
@@ -68,6 +70,7 @@ RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 100
RUN apt-get update && apt-get install -y \
python3 \
python3-pip \
+ python3-venv \
clang-format-22 \
clang-tidy-22
@@ -86,27 +89,14 @@ RUN python3 -m pip install --upgrade \
pylint==3.3.7 \
pylint-protobuf==0.22.0 \
black==25.1.0 \
- mypy==1.15.0 \
- mypy-protobuf==3.6.0
+ mypy==1.15.0
# Python modules for packaging
RUN python3 -m pip install --upgrade \
setuptools==41.2.0 \
wheel==0.33.6 \
twine==1.15.0
-#Install protoc from release, because the version available on the repo is too old
-RUN if [ "${TARGETPLATFORM}" = "linux/arm64" ]; then \
- PROTOC_URL=https://github.com/protocolbuffers/protobuf/releases/download/v21.2/protoc-21.2-linux-aarch_64.zip; \
- else \
- PROTOC_URL=https://github.com/protocolbuffers/protobuf/releases/download/v21.2/protoc-21.2-linux-x86_64.zip; \
- fi; \
- mkdir -p /opt/protoc && \
- curl -L0 ${PROTOC_URL} -o /tmp/protoc-21.2.zip && \
- unzip /tmp/protoc-21.2.zip -d /opt/protoc && \
- rm /tmp/protoc-21.2.zip
-ENV PATH=/opt/protoc/bin:$PATH
-
# Developer tools
RUN apt-get update && apt-get install -y \
bash-completion
@@ -116,13 +106,13 @@ ENV GOPATH=/opt/go
ENV GOROOT=/opt/go_dist/go
ENV PATH=$GOROOT/bin:$GOPATH/bin:$PATH
RUN mkdir -p /opt/go_dist && \
- curl https://dl.google.com/go/go1.19.3.linux-${TARGETARCH}.tar.gz | tar -xz -C /opt/go_dist
+ curl -4 https://dl.google.com/go/go1.19.3.linux-${TARGETARCH}.tar.gz | tar -xz -C /opt/go_dist
# Install rust compiler
ENV PATH=/opt/cargo/bin:$PATH
ENV RUSTUP_HOME=/opt/rustup
COPY rust-toolchain.toml rust-toolchain.toml
-RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | CARGO_HOME=/opt/cargo sh -s -- --default-toolchain $(grep -oP '(?<=channel = ")[^"]+' rust-toolchain.toml) -y
+RUN curl -4 --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | CARGO_HOME=/opt/cargo sh -s -- --default-toolchain $(grep -oP '(?<=channel = ")[^"]+' rust-toolchain.toml) -y
RUN CARGO_HOME=/opt/cargo cargo install cbindgen --version 0.29.2 --locked
RUN CARGO_HOME=/opt/cargo cargo install bindgen-cli --version 0.72.1 --locked
### Makefile
@@ -4,6 +4,7 @@
.DEFAULT_GOAL := firmware
.NOTPARALLEL:
SANITIZE ?= ON
+export PATH := $(CURDIR)/.venv/bin:$(PATH)
ifneq ($(filter config,$(MAKECMDGOALS)),)
ifneq ($(words $(MAKECMDGOALS)),1)
@@ -26,8 +27,11 @@ config:
python3 scripts/build_config.py $(CONFIG_ARGS)
bootstrap:
+ test -d .venv || python3 -m venv .venv
git submodule update --init --recursive
./scripts/bootstrap-cargo-config
+ .venv/bin/python3 -m pip install --requirement py/requirements.txt --editable ./py/bitbox02
+ @echo 'For Python commands in your shell, run: . .venv/bin/activate'
IMAGES := firmware factorysetup bootloader-stage0 bootloader-stage1
FLASH_IMAGES := $(IMAGES)
@@ -125,6 +129,8 @@ prepare-tidy:
$(MAKE) -C build-build rust-cbindgen
clean:
rm -rf build build-debug build-build build-build-noasan build-bitbox02-cmake-* build-bitbox02nova-cmake-* build-bitbox03-cargo src/rust/target
+distclean: clean
+ rm -rf .venv
# When you vendor rust libs avoid duplicates
vendor-rust-deps:
@@ -137,4 +143,4 @@ vendor-rust-deps:
run-simulator unit-test run-unit-tests run-rust-unit-tests run-rust-clippy \
run-valgrind-on-unit-tests flash-dev-firmware \
rtt-client dockerinit dockerpull dockerdev dockerrel generate-protobufs \
- generate-atecc608-config ci prepare-tidy clean vendor-rust-deps
+ generate-atecc608-config ci prepare-tidy clean distclean vendor-rust-deps
### py/README.md
@@ -8,7 +8,7 @@ This directory contains scripts to talk to the BitBox device directly via the co
These instructions require Python 3.10 or newer, pip, and venv (a lightweight "virtual
environment"). Inside the virtual environment, `python` and `pip` refer to Python 3.
-All commands below assume you are in the `py/bitbox02` directory.
+All commands below assume you are in the `py` directory, unless stated otherwise.
### Requirements
@@ -20,21 +20,24 @@ Older pip versions will fail due to changes in how editable installs are handled
### Installing dependencies
-Install the required Python dependencies listed in `requirements.txt`:
+Use the shared `.venv` in the repository root. `make bootstrap` creates or reuses it automatically;
+see [Configure and build](../BUILD.md#configure-and-build).
+
+From the repository root, install the dependencies with bootstrap and activate the environment:
```bash
-python3 -m venv .venv
+make bootstrap
source .venv/bin/activate
-pip install -r requirements.txt
+cd py
```
## Activate the virtual environment
-If you open a new shell, remember to re-activate the virtual environment,
-to use the scripts and communicate with the BitBox.
+In a new shell, activate `.venv` from the repository root before returning to `py/`:
```
source .venv/bin/activate
+cd py
```
You can deactivate a virtual environment by typing `deactivate` in your shell.
@@ -114,13 +117,11 @@ For building the BitBox firmware please refer to the
## Development
-To work on the library or scripts, install them in editable mode.
-Editable installs are only needed if you want to modify the scripts or library code.
-
-```bash
-pip install -e ./bitbox02
-```
+`make bootstrap` installs the local `bitbox02` package in editable mode, so changes to the library
+are used without reinstalling it.
For developing the Python sources, almost all of it can be done on the host easily.
-To regenerate protobufs it is recommended to use the Docker container.
-Read more about dockerized setup in [BUILD.md](../BUILD.md).
+Before regenerating protobufs, run `make bootstrap` from the repository root to install the pinned
+generators. Activate the repository's `.venv` before running Python scripts or regenerating
+bindings directly with `make -C py/bitbox02`.
+Read more about the development environment in [BUILD.md](../BUILD.md#configure-and-build).
### py/bitbox02/README.md
@@ -68,9 +68,8 @@ Python strings and protobuf objects are not guaranteed to be erased from host me
It is highly recommended that you use the dockerized setup while developing, guide for setting it up
can be found in [BUILD.md](../../BUILD.md).
-If you intended to develop instead on your local environment be vary that some development procedures
-such as regenerating files have their libraries NOT in requirements.txt as they are not necessary for the bitbox02
-Python package.
+For local development, run `make bootstrap` from the repository root to install the Python
+dependencies and protobuf generators from [py/requirements.txt](../requirements.txt).
It is highly advisable to use same version that is used in the [Dockerfile](../../Dockerfile) that is used in dockerized
setup.
### py/requirements.txt
@@ -1,4 +1,4 @@
-# Modules needed to run scripts in this directory
+# Dependencies for Python scripts and protobuf generation
tzlocal>=1.5,<2.0
types-tzlocal
bitbox02
@@ -7,3 +7,5 @@ requests
types-requests
rlp
pyhpke>=0.6.4,<0.7
+protoc-wheel-0==21.2
+mypy-protobuf==3.6.0
### scripts/generate-protobuf-rust.sh
@@ -15,6 +15,10 @@ trap cleanup EXIT
mkdir -p "${OUT_DIR}"
+if [[ -x "${ROOT_DIR}/.venv/bin/protoc" ]]; then
+ export PROTOC="${ROOT_DIR}/.venv/bin/protoc"
+fi
+
prost-build-proto \
"${ROOT_DIR}/messages" \
"${TMP_DIR}"Why this scored 15/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.