[automated] Track conditional PR CI routing defects found by live audit

[radical]

[automated]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

A live exact-head audit of conditional PR CI found one confirmed safety defect,
three systematic over-selection defects, and one unconditional job that should
participate in selection. The audit replayed the selector and
eng/github-ci/test-trigger-map.yml from each PR head, then independently
derived the expected projects/jobs from ProjectGraph, runtime dependencies,
workflow consumers, and loose inputs.

Across 228 enforcing-mode decisions audited so far, 2 runs had the same
false-negative defect, 25 had confirmed false-positive jobs, and 4 rename runs
safely but unnecessarily selected ALL. The cases below are distinct defects
but share the same selector/map ownership and should be tracked together.

1. Runtime Hosting-package consumers can be skipped

PR #19720 changed publish/deploy behavior and several Hosting integrations. Its
selector chose 49 of 99 PR test projects but omitted
Aspire.Cli.EndToEnd.Tests, so Tests / Build CLI E2E Docker image was skipped
in runs
33595798642
and
33458260779.

Aspire.Cli.EndToEnd.Tests exercises the affected aspire publish and
aspire deploy paths using built NuGet packages, but those runtime package
dependencies are not visible through its ordinary ProjectReference graph and
are not represented in Layer 2. This is a confirmed false negative.

2. Broad Hosting rules run extension E2E for unrelated integrations

The current Hosting path/project rules select the full VS Code extension E2E
workflow for leaf integrations that no extension fixture restores, loads, or
references. Confirmed examples include PRs #17742, #19024, #19243, #19810, and
#19826.

Across 24 audited runs, this instantiated 772 unnecessary extension E2E jobs
and consumed about 3,388 aggregate runner-minutes, excluding extension
bootstrap prerequisites. For example, PR #19810's Azure Sandbox-only diff
added about 180 runner-minutes in
run 33596440076.

3. CLI test-only changes run extension E2E

PR #19727 changed one file in Aspire.Cli.Tests and one in
Aspire.Cli.EndToEnd.Tests. The selector correctly selected those .NET test
projects, but the two broad test-directory rules also instantiated 35
extension E2E jobs consuming about 163 aggregate runner-minutes in
run 33432144442.

The extension workflow consumes production CLI/package artifacts; it does not
build or run either changed test project.

4. Rename tombstones force ALL

PR #19486 renamed:

eng/scripts/aspire-skills-bundle.common.ps1
-> eng/scripts/aspire-skills-bundles.common.ps1

The exact-head map routes the new path to Infrastructure.Tests, but the
selector also evaluates the deleted old path under --no-renames. That path
cannot match the map at HEAD, so
run 33414997820
escalated to ALL, selecting 99 unrelated test projects and four unrelated
gated jobs. The same behavior occurred in three subsequent runs of the PR.

This fallback is safe but unnecessarily expensive when the rename destination
is already attributed.

5. Stabilization Check is unconditional

stabilization_check lives outside the reusable tests.yml workflow and does
not consume selector output. PR #19455 changed only analyze-ci-failure
workflow files and an Infrastructure.Tests file; the selector correctly chose
only Infrastructure.Tests and no gated jobs, but Stabilization Check still
ran in
run 33678547895.

Six measured runs averaged 6m39s on 8-core-ubuntu-latest. The job's useful
coverage is stable-package metadata/dependency validation, eleven
version-sensitive CLI test classes, and the stable-feed init/restore smoke
test—not arbitrary source, workflow, or infrastructure-test changes.

Unresolved routing questions

These are not classified as defects yet because independent reviews disagreed
materially:

• PR Keep Aspire CLI notification claim timestamps consistent #19854 /
  run 33582827098:
  whether a TypeScript production module with precise unit coverage but no E2E
  assertion should select extension E2E.
• PR Avoid live CLI rendering when stdout is redirected #19673 /
  run 33689651134:
  whether generic CLI artifact/runtime reachability makes installer and API
  jobs relevant.
• PR Remove dead Playwright scanner applicators #19878 /
  run 33687040476:
  whether eager scanner-constructor resolution makes every CLI-consuming job
  relevant, or only jobs that reach scanner behavior.

Expected Behavior

Conditional PR CI should have zero high-confidence false negatives while
selecting only test projects and gated jobs that can consume the changed
production code, test source, workflow input, or runtime package.

Specifically:

• Runtime NuGet consumers such as CLI E2E must be represented when ProjectGraph
  cannot see them.
• Extension E2E must be routed by actual production/fixture dependencies rather
  than every Hosting integration or CLI test file.
• A mapped rename should not force ALL solely because the deleted source path
  is absent from the head map.
• Stabilization Check should participate in the same auditable selection
  decision, while preserving fail-safe ALL behavior.
• The three unresolved cases should receive explicit routing policy decisions
  and focused acceptance coverage.

Steps To Reproduce

1. Check out PR Add publish and deploy finalization boundaries #19720 head 954fb57812f08b31cc868ead7ba7f7b87fd750bc
   with base 8e47a3dbf220a74a79fd734ea7b4b941c842595e available.
2. Run the exact enforcing selector:

SELECT_TESTS_COMMENT_FILE=/tmp/select-tests-comment.md \
SELECT_TESTS_JSON_FILE=/tmp/select-tests-selection.json \
./dotnet.sh run --project tools/SelectTests/SelectTests.csproj -- \
  --repo-root . \
  --before-build-props /tmp/BeforeBuildProps.props \
  --map eng/github-ci/test-trigger-map.yml \
  --enforce \
  --from 8e47a3dbf220a74a79fd734ea7b4b941c842595e \
  --to 954fb57812f08b31cc868ead7ba7f7b87fd750bc

3. Inspect /tmp/select-tests-comment.md. Aspire.Cli.EndToEnd.Tests is
   absent despite exercising the affected publish/deploy packages at runtime.
4. Use the linked PRs and workflow runs above for the over-selection, rename,
   and unconditional Stabilization Check examples.

Exceptions (if any)

N/A

Aspire doctor output

N/A - this affects GitHub Actions routing.

Anything else?

Relevant implementation and configuration:

• tools/SelectTests/
• eng/github-ci/test-trigger-map.yml
• .github/workflows/tests.yml
• .github/workflows/ci.yml

The extension false positives and unconditional Stabilization Check are
inherited routing behavior rather than selector-computation regressions. False
positives reduce efficiency; the skipped CLI E2E coverage is the
safety-critical finding. No existing issue or pull request matching these
routing defects was found in targeted searches.

[radical]

[automated] Status after the merged #19882 routing work:

• The runtime CLI E2E consumer gap and the broad Hosting/CLI-test extension E2E over-selection in sections 1-3 are addressed by ci: Run extension and CLI E2E only for actual consumers #19882.
• Rename tombstone handling remains tracked separately in [automated] Investigate renamed-file handling in selective PR CI #19786.
• Stabilization Check remains unconditional with respect to selector output and still needs a routing decision.
• The unresolved routing-policy questions in the issue body have not been closed out.

Keeping this umbrella issue open for the remaining Stabilization Check and policy work. Once those items are resolved or split into dedicated trackers, this issue can close.