Repository navigation
Make non-vlt e2e legs fail when every test soft-skips #1054
Description
Follow-up from #1046 (audit B69/test hygiene).
The generalized passed > 0 guard in the ci.yml e2e step (ci.yml ~1645-1651) can still go green without testing anything: libtest counts an early-return soft-skip as passed, so a leg whose tests all skip reports passed > 0. Only per-suite *_REQUIRED env gates catch this, and not every row sets one (e.g. e2e_redirect_rush_sim).
Options:
- have the non-vlt e2e step grep a standard skip marker, as
check-vlt-legs.pydoes for the vlt step; or - give every e2e row a
*_REQUIREDgate.
🤖 Generated with Claude Code
Activity
mikolalysenko commentedon Oct 7, 2026
[agent] Triaged as
priority:p3(CI only). No open PR covers it yet. #1046 links it as a follow-up but doesn't implement the skip-marker check. I found no duplicate.Generated by Claude Code
mikolalysenko commentedon Oct 11, 2026
[agent] Re-verified on origin/main 6f0ea2c: still applies. The lean
e2e_redirect_pnpm_buildrow (.github/workflows/ci.yml:1262) sets no*_REQUIREDgate, so its 8 tests would reportokif corepack pnpm went missing (thepassed > 0guard at ci.yml:1759 can't tell). Same fore2e_safety_pnpmande2e_redirect_rush_simin the full tier. Today they do run (merge-queue run 38170303367, 8 passed).[agent] Claiming this issue (shared root cause: pnpm-family e2e rows have no fail-instead-of-skip gate). Branch: agent/fix-ci-pnpm-e2e-required. Claim-ID: 2026-10-11T20:51:03Z-19075f
Generated by Claude Code
mikolalysenko commentedon Oct 11, 2026
[agent] Draft PR: #1400 (Linux rows; Windows/macOS rows stay as follow-up here).
Generated by Claude Code
CI: gate pnpm rows with a matrix key
mikolalysenko commentedon Oct 11, 2026
[agent] #1400 merged: the Linux rows of
e2e_redirect_pnpm_build,e2e_safety_pnpmande2e_redirect_rush_simnow setpnpm_required: '1', so a missing pnpm fails those legs instead of passing. Left open for the remaining slice: the Windows and macOS rows of the same suites (full tier,e2e-windows/e2e-macos) are not gated yet. Gating them needs one full-scope run to confirm corepack/pnpm is present on those runners. Releasing the claim.Generated by Claude Code
mikolalysenko commentedon Oct 12, 2026
[agent] Claiming this issue (Windows/macOS slice; root cause: on windows-latest the pnpm-family suites spawn bare
pnpm/corepack, which can't resolve the.cmdshims, so every test skips). Evidence from nightly run 38115955529: Windowse2e_safety_pnpm4 passed in 0.02s,e2e_redirect_pnpm_build8 passed in 0.02s,e2e_redirect_rush_sim3 passed in 1.01s, against 2.9s / 14.4s / 73.6s for the same tests on macOS. Branch: agent/fix-ci-windows-pnpm-shims. Claim-ID: 2026-10-12T00:50:38Z-ab3670Generated by Claude Code
mikolalysenko commentedon Oct 12, 2026
[agent] Draft PR: #1407
Generated by Claude Code
Run Windows pnpm e2e tests instead of skipping
Fail Windows and macOS pnpm legs on a skip
Tighten pnpm probe and shim regression guard
mikolalysenko commentedon Oct 12, 2026
[agent] Done. #1054 is fixed by #1400 plus #1407 (merged). On origin/main every row of
e2e_safety_pnpm,e2e_redirect_pnpm_buildande2e_redirect_rush_simsetspnpm_required: '1': the Linux rows from #1400, and the Windows and macOS rows from #1407.scripts/tests/test_ci_e2e_tiers.py::PnpmRequiredGatepins the gate on all three OSes. #1407 also fixed the Windows legs, which had been skipping every test because barepnpm/corepackspawns couldn't run the.cmdshims. Full-scope run 38191238691 shows them really running: 5.7 s, 93 s and 81 s, up from 0.02–1 s.Not covered here: e2e rows of other suites that still lack a
*_REQUIREDgate, and thecommon::has_commandshim follow-up noted in #1407's description. Neither is part of the pnpm slice tracked here.Generated by Claude Code