Skip to content

bench(ci): measure what TEST_SELECTION=enforce would have saved over recent CI runs - #258

Merged
PHPCraftdream merged 1 commit into
mainfrom
sel-impact-20261116
Nov 16, 2026
Merged

PHPCraftdream merged 1 commit into
mainfrom
sel-impact-20261116

Conversation

@PHPCraftdream

Copy link
Copy Markdown
Owner

Summary

Measurement only (no selector, planner or workflow-behaviour change): what the repository variable TEST_SELECTION=enforce would have changed over 28 recent CI runs.

  • scripts/test-selection-impact.mjs — cargo-free generator (--report, --check), reuses the timestamp helpers of scripts/ci-job-durations.mjs, asserts every printed ratio.
  • docs/perf/TEST_SELECTION_ENFORCE_IMPACT.md — generated report; docs/perf/test_selection_inputs/ — compact inputs (91 KiB).
  • tests/test_selection_impact.test.mjs — offline fixtures (non-zero savings, unknown reasons, damaged inputs, determinism, tampered report), registered in the always-on docs-consistency action.

Result (this window only)

  • Subset runs: 0 / 28; estimated saved test-core seconds 0.0 / 6669.0 s.
  • Reasons for all: 8 full-run requests (CI: full), 15 a changed Node test file (tests/*.test.mjs / tests/*.mjs) "executed by no mapped test", 5 a bench/example change not attributable to a test.
  • Limits stated in the report: success-preferred sampling, 28 instead of ~40 artifacts, gh-call budget exceeded during collection. No GO/NO-GO decision.

Input for plan step P1.5 (selection by the crate graph): the dominant cause in this window is Node test files reaching the Rust selector's unmapped-file rule.

🤖 Generated with Claude Code

…recent CI runs

scripts/test-selection-impact.mjs derives, from saved `test-selection`
artifacts and the `test (core tiers)` job timings of 28 recent CI runs
(2026-11-14 20:43 to 2026-11-16 06:21 UTC), how many runs enforce mode would
have reduced and the proportional test-core seconds it would have saved;
docs/perf/TEST_SELECTION_ENFORCE_IMPACT.md is generated by it (`--check`
verifies the committed report byte for byte) and the compact inputs live in
docs/perf/test_selection_inputs/.

Result for this window: 0 / 28 runs would have run a subset; all 28 selected
`all` (8 full-run requests, 15 a changed file executed by no mapped test - in
every case a Node `tests/*.test.mjs` or `tests/*.mjs` file, 5 a bench/example
change not attributable to a test). Estimated saving 0.0 / 6669.0 s.

Sampling limits are stated in the report: success-preferred sampling (failed
and cancelled runs were not sampled), 28 artifacts instead of about 40, and
the collection exceeded its gh-call budget (69 vs 45). The report makes no
GO/NO-GO decision. The Node test suite is registered in the always-on
docs-consistency action; the decomposition digest of that step is updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PHPCraftdream
PHPCraftdream merged commit a319401 into main Nov 16, 2026
62 checks passed
@PHPCraftdream
PHPCraftdream deleted the sel-impact-20261116 branch November 16, 2026 18:26
PHPCraftdream added a commit that referenced this pull request Nov 16, 2026
main gained tests/test_selection_impact.test.mjs (PR #258); the derived classification CSV and report are regenerated with scripts/classify-tests.mjs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant