feat(standards): add a scheduled org-wide conformance sweep - #43
feat(standards): add a scheduled org-wide conformance sweep#43WomB0ComB0 wants to merge 3 commits into
Conversation
repo-standards catches drift on the PR that introduces it. It cannot catch a
repo that already drifted and simply is not being touched — which is exactly
how four repos ended up with a security scan that had never run once. Nobody
had opened a PR against them since the day it broke.
Runs the same checks against every non-archived repo weekly and reports a job
summary. Read-only.
Two things learned by actually running it:
* A repo the token cannot read must be reported as UNREADABLE, never as
clean. Silence is not conformance — that is the same failure mode the
checks exist to catch, and it would be self-inflicted.
* `head -1` and `grep -m1` in a pipe close it early, the upstream writer
takes SIGPIPE, and under `set -o pipefail` the whole sweep aborts with
exit 141 having assessed nothing. First run did precisely that. Sliced
with parameter expansion instead, and no-match greps now tolerate their
exit 1.
Also fixes a false positive in the merged repo-standards check: it matched
`zima` anywhere in a runs-on line, including the dynamic USE_SELF_HOSTED
toggle. That form embeds the label in an expression actionlint cannot
evaluate, so it never errors there — landing was being flagged for a problem
it does not have. Both now match only the literal array form.
Verified against the live org: 21 repos scanned, and it found vcpkg — a fifth
repo whose security workflow has been startup_failure since July, which the
manual pass missed entirely because it only sampled known repos.
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Warning Review limit reached
Next review available in: 18 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe pull request adds a scheduled and manually triggered organization-wide conformance sweep. It checks repository workflows and configuration, reports findings, optionally fails the job, and narrows ChangesConformance validation
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Scheduler
participant ConformanceSweep
participant GitHubAPI
participant Repository
Scheduler->>ConformanceSweep: Trigger weekly or manually
ConformanceSweep->>GitHubAPI: List non-archived repositories
GitHubAPI->>ConformanceSweep: Return repository list
ConformanceSweep->>GitHubAPI: Fetch workflow and configuration files
GitHubAPI->>Repository: Read repository contents
Repository->>ConformanceSweep: Return files or readability errors
ConformanceSweep->>ConformanceSweep: Validate repository conventions
ConformanceSweep->>Scheduler: Publish summary, warnings, and optional failure
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/org-conformance-sweep.yml:
- Around line 101-104: Update the security.yml validation in the sweep script to
detect org reusable-workflow calls and evaluate the caller’s effective
permissions using the same caller-scoped check as repo-standards.yml. Report no
finding only when either the calling job’s permissions or the workflow-level
permissions grant actions: read; otherwise retain the existing security.yml
finding.
- Line 50: Update the workflow’s GH_TOKEN configuration before repository
discovery so the sweep requires ORG_READ_TOKEN and does not fall back to
github.token. Fail the workflow configuration or otherwise stop the sweep when
ORG_READ_TOKEN is absent, ensuring gh repo list cannot perform a partial
discovery that produces a clean summary.
- Around line 72-73: Update the repository enumeration command in the workflow’s
repos assignment to use a limit of 1000 instead of 200, ensuring all
non-archived organization repositories are included while preserving the
existing JSON extraction and sorting.
- Around line 67-70: The ls_workflows() readability check currently treats a
GitHub 404 for a missing .github/workflows directory as an unreadable
repository. Update ls_workflows() and its caller to distinguish a missing
directory from genuine API or permission failures, returning an empty workflow
list for 404 while preserving unreadable classification for other failures; use
a read-only contents endpoint.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5bd89f81-a97c-4f88-a999-90d0dc330261
📒 Files selected for processing (2)
.github/workflows/org-conformance-sweep.yml.github/workflows/repo-standards.yml
`scripts` is intentionally out of reach, so the sweep reported it UNREADABLE on every run. A weekly report that cries wolf is one people stop reading — which is the same failure mode these checks exist to prevent, just self inflicted. Adds a `skip-repos` input defaulting to `scripts`. Scheduled runs pass no inputs, so the default is applied in `env:` too. Forwarded through env rather than inlined into the shell, matching the pattern the other workflows use. Verified live: 20 repos scanned, scripts skipped, 0 unreadable, 2 findings — resQ (pending resQ#722) and vcpkg (pending vcpkg#34).
Every one of these could report a repo as fine when it is not — the failure mode this sweep exists to prevent. Readability was tested by listing .github/workflows, but a repo without that directory returns 404, which is indistinguishable from an access failure by exit status alone. `scripts` was being reported UNREADABLE while being perfectly readable — it simply has no workflows. Readability is now established against the repo object, and a missing directory means "no workflows". That also removes the need for the skip-repos default added a commit ago: scripts now scans clean on its own. The input stays as an escape hatch but defaults to empty — a repo suppressed there is a repo nobody is checking. The actions: read check was the loose file-wide grep, i.e. the exact bug caught in repo-standards during #42 review and fixed there. I had copied the pre-fix version into the sweep. It now uses the same caller-scoped awk, so a workflow whose caller job overrides permissions without actions: read is caught. A failed or partial `gh repo list` silently produced a short sweep that read as all-clean. It now errors out, including when enumeration returns fewer than two repos, which is the signature of GITHUB_TOKEN being used without ORG_READ_TOKEN. Limit raised 200 -> 1000. Verified live with and without skip-repos: 21 repos scanned, 0 unreadable, 2 findings (resQ pending resQ#722, vcpkg pending vcpkg#34).
All four correct — and one of them proved a claim in this PR wrongThe 404 finding was the important one, and it invalidated something I asserted. I had reported $ gh api /repos/resq-software/scripts --jq .name # readable
scripts
$ gh api /repos/resq-software/scripts/contents/.github/workflows
404
That also removes the need for the The Enumeration failures read as all-clean. Fixed: it now errors out, and additionally errors when enumeration returns fewer than two repos, which is the signature of VerificationRun live against the org both with and without
|
1 participant
The hole this closes
repo-standards.yml(#42) catches drift on the PR that introduces it. It cannot catch a repo that already drifted and simply is not being touched — which is exactly how four repos ended up with a security scan that had never run once. Nobody had opened a PR against them since the day it broke.This runs the same checks against every non-archived repo weekly, read-only, reporting a job summary.
It found a repo the manual pass missed
Verified against the live org — 21 repos:
resQzimawithout actionlint declaring it (fix pending in resQ#722)scriptsvcpkgsecurity.ymlmissingactions: read(startup_failure)vcpkgis a fifth repo whose security workflow has never run —startup_failureon every run back to July. I missed it in the manual sweep because I only sampled repos I already knew about. That is the entire argument for automating this.Three things learned by running it, not reasoning about it
1. Unreadable ≠ clean.
GITHUB_TOKENcannot read sibling repos. A naive sweep would report every repo as passing. Unreadable repos are now an explicit finding and count toward the failure gate — silence must never be mistaken for conformance, which is the same failure mode these checks exist to catch.2. The first run aborted having assessed nothing, and exited 0-ish.
head -1/grep -m1in a pipe close it early, the upstream writer takes SIGPIPE, and underset -o pipefailthe whole script dies — exit 141, empty summary. Now sliced with parameter expansion, and no-match greps tolerate their exit 1.3. A false positive in the already-merged check.
#42matchedzimaanywhere in aruns-on:line, including the dynamic toggle:actionlint cannot evaluate that expression, so it never errors on it — which is why
landingpasses today despite not declaring the label. Only the literal array form (runs-on: [self-hosted, linux, x64, zima]) trips it.landingwas being flagged for a problem it does not have. Both the sweep andrepo-standards.ymlnow match the literal form only.Setup required
Needs an
ORG_READ_TOKENsecret — fine-grained, org-wide,Contents: Read. Without it the sweep honestly reports every repo as UNREADABLE rather than falsely green.Verification
actionlintclean on both files (it shellchecks therun:blocks)actionlint.yaml-present-but-missing-label still caughtlandingcorrectly no longer flaggedSummary by CodeRabbit
New Features
Bug Fixes