Skip to content

Commit 178fefd

Browse files
committed
fix(report): the console summary counted the wrong skips
render() computes the catalog's own skip count, then the scope paragraph rebound the same name to the run-wide census: skipped = sum(... for t in s[5] if _lookup(...) == 'skip') # catalog ... skipped = JUNIT_CENSUS['skipped'] # whole run so the line CI operators actually read -- N sections, 374 tests (323 passed, 0 failed, 3 skipped, 48 pending) printed the whole run's skips inside a breakdown of the catalog. The four numbers did not add up to the total, and the error ran in the alarming direction: it inflates skips, which reads as "lots of this was not exercised" against a catalog where only three entries were actually gated. Use the census value inline where the paragraph needs it and leave `skipped` meaning one thing. Added the assertion that would have caught it, since the breakdown is only ever right when it reconciles: assert passed + failed + skipped + missing == total Now: 374 tests (323 passed, 0 failed, 3 skipped, 48 pending) -> 374.
1 parent 1dacea4 commit 178fefd

1 file changed

Lines changed: 4 additions & 2 deletions

File tree

scripts/generate-test-report.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2845,13 +2845,12 @@ def _section_state(s):
28452845
ran = JUNIT_CENSUS['ran']
28462846
if ran:
28472847
pb.gap(3)
2848-
skipped = JUNIT_CENSUS['skipped']
28492848
for line in _w('Scope: this report is a curated catalog of %d tests. The CI run collected %d '
28502849
'(%d of them native firmware unit tests); %d SKIPPED and did not execute, '
28512850
'usually because the emulator predates the firmware the test targets -- a skip '
28522851
'is not evidence the feature works. Absence from this report is NOT '
28532852
'evidence that a feature is untested -- check the JUnit artifacts.'
2854-
% (total, ran, JUNIT_CENSUS['native'], skipped), 100):
2853+
% (total, ran, JUNIT_CENSUS['native'], JUNIT_CENSUS['skipped']), 100):
28552854
pb.text(8, line, color=GRAY)
28562855
pb.gap(6)
28572856
pb.text(12, 'Sections', bold=True)
@@ -2994,6 +2993,9 @@ def _section_state(s):
29942993

29952994
pb.finish()
29962995
pdf.write(output_path)
2996+
assert passed + failed + skipped + missing == total, (
2997+
'catalog counts do not reconcile: %d+%d+%d+%d != %d'
2998+
% (passed, failed, skipped, missing, total))
29972999
print(f'{output_path}: fw={fw_version}, {len(active)} sections, {total} tests '
29983000
f'({passed} passed, {failed} failed, {skipped} skipped, {missing} pending)')
29993001

0 commit comments

Comments
 (0)