Run daily full-fleet public validation cadence

[Brandon Miller (brandom-msft)]

Summary

• run full-fleet validation daily at 07:00 UTC with manual dispatch support
• discover the deterministic sample manifest and matrix from samples/**/sample.yaml
• validate all supported samples at L3, map JavaScript through TypeScript/node, and emit explicit skipped records for Rust
• preserve normalized schema v1, completeness enforcement, same-run reporting, and the declared-L4 OIDC warm-project seam without provisioning

Validation

• python .github/scripts/test/test_validation_pilot.py (3 tests)
• python .github/scripts/test/test-render-validation-report.py (4 tests)
• Python compilation for changed validation scripts
• dynamic discovery verified against 72 samples and 11 explicit Rust skips

ADO 5449701: https://msdata.visualstudio.com/Vienna/_workitems/edit/5449701

[Copilot]

Pull request overview

This PR changes the validation “pilot” workflow into a daily full-fleet validation cadence by dynamically discovering all samples/**/sample.yaml, generating a deterministic manifest/matrix for GitHub Actions, running normalized validation across the fleet (with explicit skipped records for unsupported languages), and producing a same-run report using the generated manifest.

Changes:

• Add a discovery job to generate a deterministic manifest and Actions matrix from samples/**/sample.yaml, and wire the rest of the workflow to use it.
• Update reporting/completeness to consume the generated manifest from the run artifact (instead of a checked-in curated JSON).
• Add a new discovery script and update tests/docs to reflect full-fleet, scheduled execution.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
.github/workflows/validation-report.yml	Adds manifest-path input and points completeness/reporting scripts at the run-produced manifest.
.github/workflows/validation-pilot.yml	Adds daily schedule + discovery job, switches validation to a generated matrix, and uploads the generated manifest with the normalized run artifact.
.github/workflows/scripts-selftest.yml	Updates paths triggering selftest to include the new discovery script instead of the removed curated matrix.
.github/validation-pilot.README.md	Updates workflow documentation to describe daily full-fleet discovery + execution and same-run reporting behavior.
.github/validation-pilot-matrix.json	Removes the curated static manifest/matrix.
.github/scripts/test/test_validation_pilot.py	Replaces curated-manifest assertions with discovery-based inventory assertions.
.github/scripts/run-validation-pilot.py	Adds validator-language mapping support, explicit skip records, and optional L4 execution after L3 pass.
.github/scripts/discover-validation-samples.py	New script to discover all sample manifests and emit a deterministic manifest + Actions matrix.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/scripts/discover-validation-samples.py:56

• The discovered manifest’s samples array is emitted in filesystem path order, but render-validation-report.py requires the manifest samples to be sorted and unique by id (it fails if ids != sorted(set(ids))). Sorting by path doesn’t strictly guarantee id-sorted order when path segments contain -, and there’s also no explicit duplicate-id detection here.

Sort samples by id before building identities, and fail closed if duplicate ids are discovered so the workflow fails deterministically instead of producing a malformed manifest/report.

    identities = [
        {key: sample[key] for key in ("id", "path", "language", "shape")}
        for sample in samples
    ]