From 36ccbfde0a454c400836dd4a3d4c7155329eeb03 Mon Sep 17 00:00:00 2001 From: highlander Date: Mon, 17 Aug 2026 17:17:49 -0600 Subject: [PATCH] docs(release): make "disclosure never yields to a stale test" an invariant The integration suite is red, the quickest route to green is to stop showing a screen, and the commit that does it looks like a test fix. Writing the rule down is the cheapest defence against that, because the pressure arrives disguised as housekeeping and the result would look like progress. Tests adapt to newly required disclosure. Firmware disclosure is never removed to satisfy a stale test. If a test fails because the device now shows more, the test is wrong. The extra THORChain screens disclose affiliate-fee bytes that were previously signed invisibly. Also replaces the known-red section, which was wrong in two ways. It said the integration stall was pre-existing at 440619703 -- a controlled run (same test files, only firmware varying) showed 1af2ffe7de passes and head hangs, so this release introduced it. And it could not enumerate the failures at all, because the job was killed before producing JUnit XML. With #477 the suite completes in 134s and the list is finite: 22 failed, 362 passed, 47 skipped, every failure a stale pinned expectation and none a firmware defect. Recorded as a table with an owner per row, since that is now the concrete work list for the repin. Adds the two measurement traps that cost real time here: do not run the suite sequentially against one emulator, and treat the PINNED submodule commit as the test source of truth rather than the working tree -- reading master's copy of test__sign_transformERC20 produced a confident and wrong "this breaks nothing". Refs #466 #474 #477 --- docs/release/7.14.2.md | 74 ++++++++++++++++++++++++++++++------------ 1 file changed, 53 insertions(+), 21 deletions(-) diff --git a/docs/release/7.14.2.md b/docs/release/7.14.2.md index f806169ae..2f8d5a301 100644 --- a/docs/release/7.14.2.md +++ b/docs/release/7.14.2.md @@ -320,27 +320,59 @@ reproducible path that silently removes setup protections is not defensible. | 5 — triage all residual findings | **met** | 30/30 triaged under index partitioning. 1 must-fix (#445, fixed), 3 already fixed, 0 false positives, 18 deferrals filed (#446–#463). | | 6 — final adversarial round | **not started** | | -### Known-red at this head - -- `python-integration-tests` has never completed on this branch. It is not slow — - `test_msg_eos_signtx.py` and `test_msg_thorchain_signtx.py` stall at both - `440619703` and head, consuming the 30-minute budget, so **Phase 2 — the documented - gate — never runs** and every file after the stall is unmeasured (#466). -- `deps/python-keepkey` is pinned to `81e581f`, a commit that exists only on an - unmerged topic branch. That pinned suite asserts the **pre-EIP-155** behaviour - that #445 now correctly refuses: 5 failures in `test_msg_ethereum_signtx.py` at - head. python-keepkey PR #215 fixes all of them (verified: 13 passed, 2 skipped - against this firmware on a clean emulator). **The pin must be updated before tag.** -- `FW_VERSION` detection silently resolves to `7.14.0` on this branch — BusyBox grep - has no `-P` (#467). -- 44 of the 47 commits in `1af2ffe7de..7e07b2d92` were pushed directly to the release - branch with no pull request. For a security release, most of the diff has had no - second reader. - -Measurement note: do not assess the integration suite by running it sequentially -against one emulator. A killed hang leaves the device wedged and every subsequent -file reports spurious errors. Five "collection errors" and four "failures" recorded -that way evaporated when each file was re-run against a fresh emulator. +### RELEASE INVARIANT: disclosure never yields to a stale test + +**Tests adapt to newly required disclosure. Firmware disclosure is never removed to +satisfy a stale test.** The dependency runs in one direction only. + +This is written down because the pressure to invert it is real and arrives disguised +as housekeeping. The integration suite is red, the quickest way to green is to stop +showing a screen, and the commit that does it looks like a test fix. The extra +THORChain screens disclose affiliate-fee bytes that were **previously signed +invisibly**; deleting them to make CI pass would reintroduce the defect this release +exists to close, and it would look like progress while doing it. + +If a test fails because the device now shows more, the test is wrong. + +### Known-red at this head, and why that is now useful + +Every remaining failure is a stale pinned expectation. **None indicates a firmware +defect.** With the per-test timeout (#477) the suite completes in 134s instead of +being killed at 30 minutes, so this list exists at all: + +| count | message | owner | +|---|---|---| +| 9 | `Transaction signing disabled by policy` (TON) | documented BREAKING gate; tests must opt in | +| 5 | `Chain Id out of bounds` | #445 working as designed; fixed by python-keepkey #215 | +| 3 | `Enable AdvancedMode to blind-sign` (TRON) | documented BREAKING gate | +| 2 | `Timeout >60.0s` | extra disclosure screens the pinned suite never acks (#466) | +| 1 | `Arbitrary contract data signing disabled by policy` | 0x `transformERC20` via #468; master's copy already opts in | +| 1 | `Enable AdvancedMode to blind-sign typed hashes` | documented BREAKING gate | +| 1 | `Structured EIP-712 disabled pending canonical display hardening` | deliberately disabled here | + +`22 failed, 362 passed, 47 skipped`. The single action that resolves most of it is the +`deps/python-keepkey` repin: it is pinned to `81e581f`, a commit on an unmerged topic +branch, and python-keepkey #215 plus master already fix six of the 22. **The pin must +be updated before tag.** + +Also red, and separately owned: `FW_VERSION` silently resolves to `7.14.0` because +BusyBox grep has no `-P` (#467). And 44 of the 47 commits in `1af2ffe7de..7e07b2d92` +were pushed straight to the release branch with no pull request — for a security +release, most of the diff has had no second reader. + +The branch reports **red**, and after #474 that is a fact rather than an appearance: +`ci-gate` fails when any required job is `failure`, `cancelled` **or `skipped`**, so a +gate-stage failure can no longer make the build graph vanish behind green ticks. + +Two measurement traps, recorded so they are not rediscovered: + +- Do not assess the integration suite sequentially against one emulator. A killed hang + wedges the device and every later file reports spurious errors. Five "collection + errors" and four "failures" recorded that way evaporated on a per-file re-run. +- **The pinned submodule commit is the test source of truth, never the working tree.** + `test__sign_transformERC20` enables AdvancedMode on python-keepkey master and does + not at the pinned `81e581f`; reading the wrong one produced a confident and wrong + "this breaks nothing". ## Definition of done