Summary
playwright-coverage-threshold is documented as an enforced minimum. It has never been able to fail a run. Separately, the number it compares against the threshold does not measure what its name says, and it returns a perfect score for a repo with no spec scenarios at all.
Three defects, in .github/workflows/quality.yml@main, "Generate spec-to-test coverage report" step.
1. The threshold cannot fail anything
Lines 2659-2661:
if (report.coverage < ${{ inputs.playwright-coverage-threshold }}) {
console.log('::warning::Spec-to-test coverage ' + report.coverage + '% is below threshold ' + ... + '%');
}::warning:: and no non-zero exit. The branch is taken, a warning is annotated, and the job continues green.
Measured: pipelinq sets playwright-coverage-threshold: 75 and its Code Quality run 31082256226 printed
Spec-to-test coverage: 28%
##[warning]Spec-to-test coverage 28% is below threshold 75%
28 against a declared floor of 75, and the setting contributed nothing to the verdict. A repo can set any threshold it likes; the value is decorative.
The input description (line 146) reads "Minimum line coverage percentage for Playwright tests (0-100). Only enforced when enable-playwright-coverage is true." — which tells a reader that enabling the flag turns enforcement on. It does not.
2. It is not line coverage
Line 2647:
coverage: scenarios.length > 0 ? Math.round((tests.length / scenarios.length) * 100) : 100,
That is count(test(...) calls) / count(spec scenarios) — a ratio of two independent totals. It is not line coverage, and it is not coverage in any sense: it never checks that any test corresponds to any scenario. Adding ten unrelated test() calls raises the number exactly as much as covering ten scenarios does. The metric can exceed 100% while covering nothing.
The two sides are also collected by regex over source text — scenarios by /^###?\s+(S\d+|Scenario[:\s]|REQ-)[^\n]*/gm and tests by /test\([']([^']+)/g — so a commented-out test( counts, and a scenario heading that does not match the pattern does not.
3. Zero scenarios scores 100%
The ternary's else branch is : 100. A repo whose openspec/specs/ and openspec/changes/ yield no matching headings — because the directory is absent, empty, or its headings use a different form — reports 100% coverage. The worst-covered possible repo and the best-covered one produce the same number, and the worse one produces it without running anything.
This is the standard dead-gate shape: the failure mode and the success mode are indistinguishable in the output.
Suggested fix
- Decide whether this gate enforces. If it does,
::error:: and a non-zero exit; if it does not, remove playwright-coverage-threshold rather than shipping a knob that does nothing, and reword the description.
- Rename the metric to what it is (
tests-per-scenario), or make it real by matching tests to scenarios — gate-19 (@e2e traceability) already does per-scenario matching and could supply the mapping.
scenarios.length === 0 must not score 100. Report it as "no scenarios found" and, if the gate enforces, treat it as a failure to measure rather than a pass.
Whatever is chosen, it needs a test that shows the gate CAN fail — a fixture with coverage below the threshold that turns the job red. Right now no such run can exist.
Found while burning down E2E failures across pipelinq / decidesk / openbuild / nldesign. Related: #188 (Playwright trace and globalTimeout instrument defects).
Summary
playwright-coverage-thresholdis documented as an enforced minimum. It has never been able to fail a run. Separately, the number it compares against the threshold does not measure what its name says, and it returns a perfect score for a repo with no spec scenarios at all.Three defects, in
.github/workflows/quality.yml@main, "Generate spec-to-test coverage report" step.1. The threshold cannot fail anything
Lines 2659-2661:
::warning::and no non-zero exit. The branch is taken, a warning is annotated, and the job continues green.Measured: pipelinq sets
playwright-coverage-threshold: 75and its Code Quality run31082256226printed28 against a declared floor of 75, and the setting contributed nothing to the verdict. A repo can set any threshold it likes; the value is decorative.
The input description (line 146) reads "Minimum line coverage percentage for Playwright tests (0-100). Only enforced when
enable-playwright-coverageis true." — which tells a reader that enabling the flag turns enforcement on. It does not.2. It is not line coverage
Line 2647:
That is
count(test(...) calls) / count(spec scenarios)— a ratio of two independent totals. It is not line coverage, and it is not coverage in any sense: it never checks that any test corresponds to any scenario. Adding ten unrelatedtest()calls raises the number exactly as much as covering ten scenarios does. The metric can exceed 100% while covering nothing.The two sides are also collected by regex over source text — scenarios by
/^###?\s+(S\d+|Scenario[:\s]|REQ-)[^\n]*/gmand tests by/test\([']([^']+)/g— so a commented-outtest(counts, and a scenario heading that does not match the pattern does not.3. Zero scenarios scores 100%
The ternary's else branch is
: 100. A repo whoseopenspec/specs/andopenspec/changes/yield no matching headings — because the directory is absent, empty, or its headings use a different form — reports 100% coverage. The worst-covered possible repo and the best-covered one produce the same number, and the worse one produces it without running anything.This is the standard dead-gate shape: the failure mode and the success mode are indistinguishable in the output.
Suggested fix
::error::and a non-zero exit; if it does not, removeplaywright-coverage-thresholdrather than shipping a knob that does nothing, and reword the description.tests-per-scenario), or make it real by matching tests to scenarios — gate-19 (@e2etraceability) already does per-scenario matching and could supply the mapping.scenarios.length === 0must not score 100. Report it as "no scenarios found" and, if the gate enforces, treat it as a failure to measure rather than a pass.Whatever is chosen, it needs a test that shows the gate CAN fail — a fixture with coverage below the threshold that turns the job red. Right now no such run can exist.
Found while burning down E2E failures across pipelinq / decidesk / openbuild / nldesign. Related: #188 (Playwright trace and globalTimeout instrument defects).