Skip to content

Measure the condition gate on Windows, macOS and Linux on demand - #1211

Merged
jmrplens merged 1 commit into
mainfrom
ci/1210-conditions-on-demand
Oct 6, 2026
Merged

jmrplens merged 1 commit into
mainfrom
ci/1210-conditions-on-demand

Conversation

@jmrplens

@jmrplens jmrplens commented Oct 6, 2026

Copy link
Copy Markdown
Owner

The condition gate (gobco, through scripts/coverage-conditions.sh) runs on Linux before I commit, and since #1206 the Windows and macOS legs of CI gate only the files those systems alone build. A figure for a whole package on Windows or macOS still had to be taken by hand: the internal/toolutil figures I quoted in #1206 came from a Windows virtual machine, before 20 more conditions landed in errors.go, and I had no way to take them on macOS at all.

.github/workflows/conditions.yml measures it on demand. It is triggered by workflow_dispatch and by nothing else, takes the packages, the gate (all or beyond:linux/amd64) and the systems (windows-latest, macos-latest, ubuntu-latest), and fans out over package by system. Each run is scripts/coverage-conditions.sh through bash, exactly as the CI legs and make coverage-conditions run it, so every figure is the gate's own on that system. An empty package list means every package whose built files differ between two of the platforms .goreleaser.yml builds, found by listing the module under each of them, which today is cmd/server, internal/toolutil and internal/tools/packages.

The summary job writes one table into the job summary: package, system, gobco's figure as covered of total, how many conditions were not evaluated both ways and how many of those the gate held, then every such condition with its file and line (a condition written across lines is joined onto one, so it stays one row), then why any run that could not measure could not. Each run's raw gobco output and its record are uploaded as an artifact. No figure fails the workflow; a run that cannot measure at all (a package go cannot load, a failing test, a 0/0 report, a runtime crash) fails its own job and says why in one annotation. The measuring step removes setup-go's problem matcher before the script prints anything: it would file every condition gobco names as an error annotation, so a run that measured would look failed, and past GitHub's ten per step the step's own annotation would be dropped.

To tell a held figure from no figure, scripts/coverage-conditions.sh now exits 3 when its gate holds a condition, a status no other failure uses: every other one exits 1 or 2 (go run reports any failure of gobco as 1). The CI step, the Makefile target and scripts/check-spec-conditions.sh read any non-zero status as a failure, as before, and the script's Python tests now pin 3 for the gate.

The plan and the table are a new command, cmd/measure_conditions, rather than shell in the workflow. plan validates the inputs (only the three hosted runners, a package path in the characters this module's directories use, a gate the script reads) and prints the matrix; summarize reads the records wherever download-artifact puts them, joins them to the plan, and refuses a record that disagrees with the plan or with its own status rather than tabulating a guess. Its tests hold the workflow file to the record format they read. It is at 100% of statements, 174/174 conditions under gobco, and 108 mutants killed with none lived or not covered under gremlins.

The workflow holds contents: read and nothing more, every uses: is pinned to the SHA ci.yml already pins, and no step resolves code at run time. make check-supply-chain passes, and so does actionlint 1.7.7 with shellcheck on it.

docs/development/testing/README.md says how to dispatch it and that its figures are measured, not inferred. docs/development/cmd-utilities.md, CLAUDE.md, the Copilot instructions and the testing reference name the new command, and the testing reference records its measured coverage rather than n/a.

GitHub offers a workflow for dispatch only once its file is on the default branch, so this pull request cannot run it. The first dispatch happens after the merge, for cmd/server and internal/toolutil on all three systems, and it is what the issue's last acceptance line asks for: gh workflow run conditions.yml -f packages="./cmd/server ./internal/toolutil" -f systems="windows-latest macos-latest ubuntu-latest" -f gate=all. I will link that run on the issue, and reopen it if the run does not produce the table.

Closes #1210

A figure for a whole package on Windows or macOS could only be taken by
hand on a machine of that system: the internal/toolutil figures quoted in
#1206 came from a Windows virtual machine, and none could be taken on
macOS. .github/workflows/conditions.yml takes them when dispatched, and
on nothing else. It runs scripts/coverage-conditions.sh through bash on
each runner named, exactly as the CI legs and make coverage-conditions
run it, so every figure is the one the gate computes on that system.

The packages default to every package whose built files differ between
two of the platforms .goreleaser.yml builds (cmd/server,
internal/toolutil and internal/tools/packages today), the gate is all or
beyond:linux/amd64, and the systems are any of windows-latest,
macos-latest and ubuntu-latest. The plan and the job summary are a new
command, cmd/measure_conditions: one row per package and runner with
gobco's figure, every condition not evaluated both ways with its file
and line, and why a run that could not measure could not. Each run's raw
output and record are uploaded as an artifact. No figure fails the
workflow; a run that cannot measure fails its own job and says why.

To tell the two apart, the script now exits 3 when its gate holds a
condition, a status no other failure uses. Every caller that only asks
whether a run passed reads any non-zero status as before.

Closes #1210
@jmrplens jmrplens added this to the 3.2.0 milestone Oct 6, 2026
@sourcery-ai

sourcery-ai Bot commented Oct 6, 2026

Copy link
Copy Markdown

Sorry @jmrplens, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 6 days and 11 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@github-actions github-actions Bot added ci Build pipeline, workflows, or lint configuration tooling The audit and generator commands under cmd/, and the Makefile targets that run them platform Windows and macOS specific source, and the cross-platform CI leg v3.2.0 Targeted at the 3.2.0 release labels Oct 6, 2026
@coderabbitai

coderabbitai Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: jmrplens/gitlab-mcp-server/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: bc3e8929-f055-42d2-970b-12ab722bb266
📥 Commits

Reviewing files that changed from the base of the PR and between 66a032f and 0e961f9.

📒 Files selected for processing (34)
  • .github/copilot-instructions.md
  • .github/workflows/ci.yml
  • .github/workflows/conditions.yml
  • CLAUDE.md
  • Makefile
  • cmd/internal/golist/doc.go
  • cmd/measure_conditions/discover.go
  • cmd/measure_conditions/discover_test.go
  • cmd/measure_conditions/doc.go
  • cmd/measure_conditions/main.go
  • cmd/measure_conditions/main_test.go
  • cmd/measure_conditions/measurement.go
  • cmd/measure_conditions/measurement_test.go
  • cmd/measure_conditions/plan.go
  • cmd/measure_conditions/plan_test.go
  • cmd/measure_conditions/record.go
  • cmd/measure_conditions/record_test.go
  • cmd/measure_conditions/summarize.go
  • cmd/measure_conditions/summarize_test.go
  • cmd/measure_conditions/testdata/platforms/arch/arch.go
  • cmd/measure_conditions/testdata/platforms/arch/arch_arm64.go
  • cmd/measure_conditions/testdata/platforms/go.mod
  • cmd/measure_conditions/testdata/platforms/plain/plain.go
  • cmd/measure_conditions/testdata/platforms/split/split.go
  • cmd/measure_conditions/testdata/platforms/split/split_windows.go
  • cmd/measure_conditions/testdata/platforms/tagged/tagged.go
  • cmd/measure_conditions/testdata/platforms/tagged/tagged_race.go
  • cmd/measure_conditions/testdata/platforms/testonly/probe.go
  • cmd/measure_conditions/testdata/platforms/testonly/probe_unix_test.go
  • docs/development/cmd-utilities.md
  • docs/development/testing/README.md
  • docs/development/testing/testing.md
  • scripts/coverage-conditions.sh
  • scripts/coverage_conditions_sh_test.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Summary

Summary by CodeRabbit

  • New Features
    • Added an on-demand condition-coverage workflow for Windows, macOS, and Ubuntu. It can target selected packages or discover packages that vary across release platforms, then report coverage results and conditions not evaluated both ways.
    • Added clearer reporting for runs without measurements and for issues with their results.
  • Documentation
    • Documented how to plan and summarize condition-coverage runs, choose packages and platforms, and interpret workflow outcomes.
  • Bug Fixes
    • Coverage gate failures now use a distinct exit status from other script failures.

Walkthrough

This change adds a manually dispatched condition-coverage workflow and the measure_conditions command. The command plans package and runner combinations, processes measurement records, and writes coverage results to the workflow summary. Gate failures now use exit status 3.

Changes

Condition coverage measurement

Layer / File(s) Summary
Gate failure status
scripts/coverage-conditions.sh, scripts/coverage_conditions_sh_test.py, Makefile, docs/development/testing/README.md
The coverage script returns status 3 when the gate finds conditions not evaluated both ways. Tests and documentation reflect the status.
Package discovery and matrix planning
cmd/measure_conditions/discover.go, cmd/measure_conditions/plan.go, cmd/measure_conditions/main.go, cmd/measure_conditions/*_test.go, cmd/measure_conditions/testdata/platforms/*, cmd/internal/golist/doc.go, CLAUDE.md
The command discovers packages with differing file lists across release targets and emits a validated package-by-runner matrix. Tests cover discovery, input handling, and command dispatch.
Measurement and artifact parsing
cmd/measure_conditions/measurement.go, cmd/measure_conditions/record.go, cmd/measure_conditions/measurement_test.go, cmd/measure_conditions/record_test.go
The command parses coverage-script output and validates run records and raw output files. Tests cover condition details, status consistency, and record errors.
Run joining and Markdown summary
cmd/measure_conditions/summarize.go, cmd/measure_conditions/summarize_test.go, cmd/measure_conditions/doc.go, docs/development/cmd-utilities.md, docs/development/testing/testing.md, CLAUDE.md
The summarize command matches records to planned runs and reports coverage, held conditions, missing measurements, and record problems. Documentation and test statistics include the command.
Dispatch workflow and integration
.github/workflows/conditions.yml, .github/workflows/ci.yml, .github/copilot-instructions.md, cmd/measure_conditions/record_test.go, docs/development/testing/README.md
The workflow plans and runs package-by-runner measurements, uploads records and raw output, then writes a job summary. Tests check workflow settings and steps. Documentation describes how to dispatch it.

Priority: ⬇️ Low

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  actor Operator
  participant ConditionsWorkflow
  participant MeasureConditionsPlan
  participant CoverageScript
  participant ArtifactStorage
  participant MeasureConditionsSummarize
  Operator->>ConditionsWorkflow: Dispatch with package, runner, and gate inputs
  ConditionsWorkflow->>MeasureConditionsPlan: Plan package-runner matrix
  MeasureConditionsPlan-->>ConditionsWorkflow: Return matrix
  ConditionsWorkflow->>CoverageScript: Run coverage script for each matrix entry
  ConditionsWorkflow->>ArtifactStorage: Upload run record and raw output
  ConditionsWorkflow->>ArtifactStorage: Download available measurement artifacts
  ConditionsWorkflow->>MeasureConditionsSummarize: Summarize plan and downloaded records
  MeasureConditionsSummarize-->>ConditionsWorkflow: Write Markdown coverage summary
Loading

Merge Risk: ⚪ Minimal · up to 0e961

The workflow is mergeable after normal checks. Its first manual cross-platform dispatch remains to be completed after it reaches the default branch.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 0e961

The workflow adds optional measurements without granting repository-write permissions or weakening existing gates. Input validation limits runner selection and shell-facing values. Remaining uncertainty concerns report freshness after interrupted reruns, rather than demonstrated access to production systems.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The supported exposure is repository-package test execution on hosted CI runners, including ordinary runner filesystem and network access. The compared workflow does not add production deployment, repository-write, or secret authority. Existing cross-platform CI already executes the same coverage script.

Trust Boundaries and Controls

  • observed — The dispatch-to-execution boundary uses environment variables rather than expression interpolation into shell code, quotes execution arguments, rejects traversal and shell-syntax characters in explicit package paths, and allowlists hosted runner labels. Artifact-name collisions are refused before execution.

Resilience and Maintainability Implications

  • inferred — Record identity is sufficient for ordinary package/system joins but does not establish execution-attempt freshness. Records omit attempt provenance, while upload ignores absent files and requests overwrite. Whether an older record survives an interrupted rerun and is consumed by summary remains a platform-behavior gap, not a verified attack path or mandatory-gate bypass.

Hardening Proposals

  • proposed — Consider retaining per-cell execution provenance and showing when a summary reuses an earlier attempt’s measurement. Preserve intentional reuse of successful cells during failed-job reruns rather than requiring every record to have the latest global attempt number.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: an on-demand condition-gate measurement across Windows, macOS, and Linux.
Description check ✅ Passed The description explains the purpose, workflow behavior, command, status codes, validation, documentation, and reported checks. It links issue #1210 and explains that the required post-merge dispatch …
Linked Issues check ✅ Passed For #1210, .github/workflows/conditions.yml is dispatch-only and runs scripts/coverage-conditions.sh across the selected package/system matrix. It records and uploads raw output, then uses `cmd/me…
Out of Scope Changes check ✅ Passed The command, tests, test fixtures, script exit-status change, workflow, and documentation support #1210. The related package-count and testing-reference updates document those additions. No unrelated …
Docstring Coverage ✅ Passed Docstring coverage is 93.06% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 72 functions across 25 files. (9 skipped: 9…
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sonarqubecloud

sonarqubecloud Bot commented Oct 6, 2026

Copy link
Copy Markdown

@jmrplens
jmrplens merged commit 754bc28 into main Oct 6, 2026
5 checks passed
@jmrplens
jmrplens deleted the ci/1210-conditions-on-demand branch October 6, 2026 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Build pipeline, workflows, or lint configuration platform Windows and macOS specific source, and the cross-platform CI leg tooling The audit and generator commands under cmd/, and the Makefile targets that run them v3.2.0 Targeted at the 3.2.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Measure the condition gate on Windows and macOS on demand

1 participant