Screenshot audit is presence-only and is satisfied by setUp's wipe-confirmation frames, so a test that draws none of its declared screens still passes

[BitHighlander]

Finding

Severity: medium · Dimension: test-integrity · Location: deps/python-keepkey/scripts/generate-test-report.py:2562

Every KeepKeyTest.setUp calls wipe_device() (common.py:77), fsm_msgWipeDevice unconditionally raises ButtonRequest_WipeDevice, and DebugLinkMixin.callback_ButtonRequest writes a PNG into the test's own screenshot directory. reset_screenshots() — the only thing that deletes those frames — is invoked solely from the setup_mnemonic_* helpers (client.py:468), so any test class that does not call one keeps them. TestDisplayDisclosesSignedContent is exactly such a class: D1-D4 declare screenshots in SECTIONS (generate-test-report.py:2259, 2265, 2271, 2277) and, per finding 1, draw nothing of their own — yet screenshot_audit finds a non-empty PNG list at line 2562 and reports 'every declared screen was captured'. The audit added in response to the 7.14.2 '345 PNGs while every changed suite captured zero' incident therefore cannot detect that same failure for any test whose class skips setup_mnemonic_*, and the release PDF embeds a wipe-confirmation frame as evidence for 'Bytes after a NUL are shown'.

Evidence

deps/python-keepkey/scripts/generate-test-report.py:2561-2563:
            d = _os.path.join(screenshot_root, mod.replace('test_', '', 1), meth)
            if not _os.path.isdir(d) or not [f for f in _os.listdir(d) if f.endswith('.png')]:
                missing.append((mod, meth))

deps/python-keepkey/tests/common.py:57-64 (the per-test dir is set on the client BEFORE the wipe at line 77):
            sdir = os.path.join(os.environ.get('SCREENSHOT_DIR', 'screenshots'), mod, test_name)
            os.makedirs(sdir, exist_ok=True)
            self.client.screenshot_dir = sdir

deps/python-keepkey/keepkeylib/client.py:526-537 (every ButtonRequest is captured):
    def callback_ButtonRequest(self, msg):
        ...
        # Capture OLED screenshot BEFORE pressing button (confirmation screen)
        self._capture_oled()

lib/firmware/fsm_msg_common.h:502-506 (the wipe always draws a confirm screen):
void fsm_msgWipeDevice(WipeDevice* msg) {
  (void)msg;

  if (!confirm(ButtonRequestType_ButtonRequest_WipeDevice, "Wipe Device",
               "Do you want to erase your private keys and settings?")) {

deps/python-keepkey/keepkeylib/client.py:465-472 — the only cleanup, and it is not called by every test:
    def reset_screenshots(self):
        """Drop screenshots captured so far this test and restart numbering.
        Called at the end of the setup_mnemonic_* helpers ..."""

Suggested fix

Make the audit content-aware rather than presence-aware: have _capture_oled tag frames by originating ButtonRequest phase (or have common.KeepKeyTest.setUp always call drop_setup_screenshots() after the wipe, not only inside setup_mnemonic*), and have screenshot_audit require at least as many post-setup PNGs as the SECTIONS entry declares screens, instead of not [f for f in listdir(d) if f.endswith('.png')].

Verification

(not independently verified)

---

Found by an adversarial audit of release/7.15 (628b09257). Each finding was independently re-checked by a separate reviewer instructed to refute it by default; this one survived at confidence ?.

[BitHighlander]

Resolved on alpha by #587 (merge 893c93d). The final immutable head e4ec531 passed both complete CI workflow runs.