A failing playwright-seed-command silently skips its && tail, exits 0, and prints 'Seed command completed.' — Playwright then tests an unseeded instance

[rubenvdlinde]

Summary

playwright-seed-command is eval'd with an unquoted interpolation, so any && in a repo's seed command is parsed by the step's own shell rather than by eval. Combined with bash's set -e exemption for && lists, a seed command whose first stage fails will:

1. silently skip every later stage,
2. not fail the step, and
3. print Seed command completed. — a statement that is false.

Playwright then runs against a half-seeded instance and produces a full, credible-looking tally.

The code

.github/workflows/quality.yml@main, "Seed test data" step (around line 2422):

run: |
  cd server
  echo "Running seed command: ${{ inputs.playwright-seed-command }}"
  eval ${{ inputs.playwright-seed-command }}
  echo "Seed command completed."

eval has no quotes around the interpolation, so for an input like a.sh && b && c the shell parses eval a.sh && b && c. eval only ever receives the first stage.

The set -e exemption

GitHub's default shell for run: is bash -e {0}, which usually means a failing command aborts the step. It does not here. From the bash manual, set -e is suppressed for a command that is "part of any command executed in a && or || list except the command following the final && or ||". The failing first stage is exempt, and the final stage never runs at all, so nothing is ever eligible to trigger the exit.

Demonstrated:

$ bash -e -c 'echo "Running seed command: X"; false && echo "TAIL RAN"; echo "Seed command completed."'
Running seed command: X
Seed command completed.
$ echo $?
0

TAIL RAN is absent, and the step exits 0.

Measured in the wild

ConductionNL/pipelinq, run 30800304506 (branch ci/bundle-truncation-control, 2026-08-03), E2E job 91645954167. Its seed command was:

bash apps/pipelinq/tests/e2e/ci-seed.sh && echo "[control] bundle before: $(stat -c%s apps/pipelinq/js/pipelinq-main.js) bytes" && truncate -s 65 apps/pipelinq/js/pipelinq-main.js && echo "[control] bundle after: $(stat -c%s apps/pipelinq/js/pipelinq-main.js) bytes"

ci-seed.sh failed:

##[error]These setup steps are still unmet: ['organisation'].
Seed command completed.

so the truncate never ran. The branch existed for one purpose — to run the suite against a zeroed JS bundle as a positive control — and it ran against a full bundle on a half-seeded instance instead. It then reported 107 passed / 61 failed / 41 skipped / 1 did not run out of 211, a tally with the shape of a real experimental result and no relationship to the experiment.

The control's own readback could not have caught it either. $(stat …) sits inside the double-quoted argument of the echo "Running seed command: …" line, where nested quotes end the outer quoting, so both substitutions are expanded by that echo — before any truncation. The log shows:

Running seed command: … && echo [control] bundle before: 3299693 bytes && truncate -s 65 … && echo [control] bundle after: 3299693 bytes

Identical before and after, by construction. A successful truncation would have printed exactly the same line.

Why it matters generally

This is not one repo's mistake. && is the natural way to write a multi-stage seed, the step's own log line actively encourages treating the input as a shell fragment, and the failure is invisible three ways over: no non-zero exit, a success message, and a plausible test tally. Any repo whose playwright-seed-command chains stages is one failing stage away from silently testing an unseeded instance — and the resulting failures will be attributed to application code.

Suggested fix

1. Quote the interpolation: eval "${{ inputs.playwright-seed-command }}" so the whole command reaches eval as one unit and its combined exit status is the step's.
2. Check the status explicitly rather than relying on set -e:

   eval "${{ inputs.playwright-seed-command }}" && RC=0 || RC=$?
   if [ "$RC" -ne 0 ]; then
     echo "::error::Seed command failed with exit ${RC}. Not running Playwright against an unseeded instance."
     exit "$RC"
   fi
   echo "Seed command completed."

   The && RC=0 || RC=$? idiom is already used elsewhere in this file (the axe and hydra-gates runners) for the same reason.
3. Only print Seed command completed. when it did.
4. Consider passing the input through an intermediate variable rather than interpolating it into the script body, so a seed command containing quotes cannot reshape the surrounding shell.

A test that proves the gate can fail: a seed command of false && true must turn the job red. Today it turns it green.

Related

• Playwright instrument: 2 repos write zero traces (retries:0 + on-first-retry), and no repo in the fleet sets globalTimeout #188 — Playwright trace and globalTimeout instrument defects.
• playwright-coverage-threshold has never gated: warning-only, not line coverage, and zero scenarios scores 100% #189 — playwright-coverage-threshold has never gated.
• E2E is not a required check anywhere: 0 of 20 fleet repos gate on Playwright, and 17 have no branch protection at all #190 — E2E is not a required check in any fleet repo.

[rubenvdlinde]

Current exposure, for scoping: I swept the playwright-seed-command on development for all 15 fleet repos that set one. None currently uses an && chain — 14 are a bare bash apps/<app>/tests/e2e/ci-seed.sh and nldesign is a single php occ config:app:set. So no repo is silently mis-seeding today.

That makes this a latent trap rather than a live outage, but not a theoretical one. The one time a chain was used it was for a bundle-truncation positive control — the exact experiment where a silently-skipped tail is most damaging, because the whole point is to prove the suite can go red, and the broken control instead produced a plausible tally that argued the opposite.

Chained seeds are also the natural shape for any temporary experiment (truncate an asset, disable a feature flag, seed an extra fixture), so this will be reached for again. Worth fixing before it is.