Skip to content

docs(release): make "disclosure never yields to a stale test" an invariant #478

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BitHighlander merged 1 commit into from
Aug 17, 2026
Merged

docs(release): make "disclosure never yields to a stale test" an invariant #478

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 53 additions & 21 deletions docs/release/7.14.2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -320,27 +320,59 @@ reproducible path that silently removes setup protections is not defensible.
| 5 — triage all residual findings | **met** | 30/30 triaged under index partitioning. 1 must-fix (#445, fixed), 3 already fixed, 0 false positives, 18 deferrals filed (#446–#463). |
| 6 — final adversarial round | **not started** | |

### Known-red at this head

- `python-integration-tests` has never completed on this branch. It is not slow —
`test_msg_eos_signtx.py` and `test_msg_thorchain_signtx.py` stall at both
`440619703` and head, consuming the 30-minute budget, so **Phase 2 — the documented
gate — never runs** and every file after the stall is unmeasured (#466).
- `deps/python-keepkey` is pinned to `81e581f`, a commit that exists only on an
unmerged topic branch. That pinned suite asserts the **pre-EIP-155** behaviour
that #445 now correctly refuses: 5 failures in `test_msg_ethereum_signtx.py` at
head. python-keepkey PR #215 fixes all of them (verified: 13 passed, 2 skipped
against this firmware on a clean emulator). **The pin must be updated before tag.**
- `FW_VERSION` detection silently resolves to `7.14.0` on this branch — BusyBox grep
has no `-P` (#467).
- 44 of the 47 commits in `1af2ffe7de..7e07b2d92` were pushed directly to the release
branch with no pull request. For a security release, most of the diff has had no
second reader.

Measurement note: do not assess the integration suite by running it sequentially
against one emulator. A killed hang leaves the device wedged and every subsequent
file reports spurious errors. Five "collection errors" and four "failures" recorded
that way evaporated when each file was re-run against a fresh emulator.
### RELEASE INVARIANT: disclosure never yields to a stale test

**Tests adapt to newly required disclosure. Firmware disclosure is never removed to
satisfy a stale test.** The dependency runs in one direction only.

This is written down because the pressure to invert it is real and arrives disguised
as housekeeping. The integration suite is red, the quickest way to green is to stop
showing a screen, and the commit that does it looks like a test fix. The extra
THORChain screens disclose affiliate-fee bytes that were **previously signed
invisibly**; deleting them to make CI pass would reintroduce the defect this release
exists to close, and it would look like progress while doing it.

If a test fails because the device now shows more, the test is wrong.

### Known-red at this head, and why that is now useful

Every remaining failure is a stale pinned expectation. **None indicates a firmware
defect.** With the per-test timeout (#477) the suite completes in 134s instead of
being killed at 30 minutes, so this list exists at all:

| count | message | owner |
|---|---|---|
| 9 | `Transaction signing disabled by policy` (TON) | documented BREAKING gate; tests must opt in |
| 5 | `Chain Id out of bounds` | #445 working as designed; fixed by python-keepkey #215 |
| 3 | `Enable AdvancedMode to blind-sign` (TRON) | documented BREAKING gate |
| 2 | `Timeout >60.0s` | extra disclosure screens the pinned suite never acks (#466) |
| 1 | `Arbitrary contract data signing disabled by policy` | 0x `transformERC20` via #468; master's copy already opts in |
| 1 | `Enable AdvancedMode to blind-sign typed hashes` | documented BREAKING gate |
| 1 | `Structured EIP-712 disabled pending canonical display hardening` | deliberately disabled here |

`22 failed, 362 passed, 47 skipped`. The single action that resolves most of it is the
`deps/python-keepkey` repin: it is pinned to `81e581f`, a commit on an unmerged topic
branch, and python-keepkey #215 plus master already fix six of the 22. **The pin must
be updated before tag.**

Also red, and separately owned: `FW_VERSION` silently resolves to `7.14.0` because
BusyBox grep has no `-P` (#467). And 44 of the 47 commits in `1af2ffe7de..7e07b2d92`
were pushed straight to the release branch with no pull request — for a security
release, most of the diff has had no second reader.

The branch reports **red**, and after #474 that is a fact rather than an appearance:
`ci-gate` fails when any required job is `failure`, `cancelled` **or `skipped`**, so a
gate-stage failure can no longer make the build graph vanish behind green ticks.

Two measurement traps, recorded so they are not rediscovered:

- Do not assess the integration suite sequentially against one emulator. A killed hang
wedges the device and every later file reports spurious errors. Five "collection
errors" and four "failures" recorded that way evaporated on a per-file re-run.
- **The pinned submodule commit is the test source of truth, never the working tree.**
`test__sign_transformERC20` enables AdvancedMode on python-keepkey master and does
not at the pinned `81e581f`; reading the wrong one produced a confident and wrong
"this breaks nothing".

## Definition of done

Expand Down