Skip to content

ci: make the required check real, and add secret and code scanning - #3284

Merged
forgetso merged 9 commits into
mainfrom
ci/required-check-and-security-scanning
Sep 16, 2026
Merged

forgetso merged 9 commits into
mainfrom
ci/required-check-and-security-scanning

Conversation

@forgetso

@forgetso forgetso commented Sep 16, 2026 •

Copy link
Copy Markdown
Member

What was broken

main requires exactly one status check to pass before a PR can merge, and it is called check.

Seven separate workflows each published a job called check — changesets, cypress, lint, pinned-versions, provider_image, tests and typecheck. When several check runs share a name, GitHub merges them into one context and takes the answer from whichever finished first. That was always the cheapest one. So a PR could go green and merge while the tests, the typecheck and the lint were all still running, or had already failed.

The comments in those workflows actually described this as a feature: a docs-only PR skips the path-filtered jobs, so changesets' unfiltered check was there to stop the PR hanging forever. It did do that. It was also the hole.

What changed

Each of the seven is now named after its own workflow, so no two report the same context. A new required-check workflow takes over the check name and earns it: it has no path filter, runs on every non-draft PR, and waits for every other check run on the commit, failing if any of them failed.

The dev and staging rulesets used to require the same context. Both branches are dead — dev last committed 2024-07, staging 2025-07 — so those rulesets were deleted separately and this workflow is scoped to main and release/*.

The aggregator is the one that can be required precisely because it always reports. The others can't be — they are path-filtered, and a required context that never reports blocks the PR forever.

Two other workflows also shared a name (build, in android-webview and ios-webview), so they are renamed too. That matters because the aggregator de-duplicates by name, so a shared name would hide one of them.

Also in here: this repo had no security scanning

Not a scan of any kind. Dependabot opens PRs for known-bad versions, which is useful, but it is not a scan and it gates nothing.

Added, on every PR plus a weekly full-history run:

  • gitleaks — committed credentials
  • semgrep — static analysis
  • npm audit and cargo audit — known-vulnerable dependencies

All of it runs on GitHub-hosted runners rather than our own machines, because this repo is public and a fork PR would otherwise execute a contributor's code on hardware we own. Hosted runners are free and unlimited on public repos.

.gitleaks.toml tunes the scanner to what is actually secret here. The entropy heuristic flags every site key, and a site key is a public identifier — it is in the HTML of every page that loads the widget. Those are allowlisted by the shape of the value rather than by directory, so a real credential dropped into an allowlisted folder still gets reported.

Test coverage

Run locally against the real tree before opening this:

Check Result
gitleaks, full history (11,868 commits) no leaks found, exit 0
gitleaks, PR-range mode no leaks found
npm audit --audit-level=critical (the gating level) exit 0 — 19 advisories, none critical
cargo audit, native-ja4 clean, 35 dependencies
cargo audit, native-merkle clean, 33 dependencies
every workflow file parses as YAML pass
aggregator shell script, bash -n pass
no two PR jobs share a display name verified across all 20 workflows

The npm audit and semgrep gates are deliberately set so they cannot fail on debt that already exists — npm audit gates at critical and reports high, semgrep only reports findings a PR introduces. Otherwise every unrelated PR gets blocked by a backlog its author did not create, and the job gets deleted instead of the debt getting paid.

🤖 Generated with Claude Code

Seven workflows each published a check run named `check`, which is the one
status context the branch rulesets require. GitHub resolves a same-named
context from whichever reports first, so the gate went green on the cheapest
job while the rest were still running. Each is now named after its workflow and
a new `required-check` job takes the name, waiting for every other check on the
commit.

Adds gitleaks, semgrep, npm audit and cargo audit on every PR plus a weekly
full-history sweep. All on GitHub-hosted runners, so a fork PR cannot run
contributor code on the self-hosted fleet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@forgetso
forgetso force-pushed the ci/required-check-and-security-scanning branch from bdeb01a to 0e38fc1 Compare September 16, 2026 12:32
@forgetso

Copy link
Copy Markdown
Member Author

Heads up on the red check: the failing jobs are the Android ones, and they are not caused by this PR.

This PR's entire change to those workflow files is a single name: line. The failure is inside android-actions/setup-android@v4, at the "Set up Android SDK" step, before any of our code runs:

Found preinstalled sdkmanager in /usr/local/lib/android/sdk/cmdline-tools/latest
Wrong version in preinstalled sdkmanager
[command] .../cmdline-tools/20.0/bin/sdkmanager --licenses
6 of 7 SDK package licenses not accepted.
...
All SDK package licenses accepted
Error: The process '.../sdkmanager' failed with exit code 1

Three jobs across two repos fail identically at that same step. The action's v4 tag has not moved since 2026-04-04, so the change is on GitHub's runner image side — the preinstalled Android cmdline-tools version moved and the action's licence-acceptance path no longer exits 0. The last green run of this job was 2026-09-10.

So this is pre-existing breakage that was previously invisible: the old gate took its answer from whichever same-named check reported first, so a broken Android job did not stop anything. The new aggregator noticing it is the change working, not a regression.

Fixing it looks like a one-line cmdline-tools-version input, but it is an unrelated toolchain change and I would rather not land it inside the PR that an auditor will read as the evidence for the branch-protection and scanning controls. Happy to raise it separately.

forgetso and others added 7 commits September 16, 2026 13:38
Ignores key material by shape rather than by path, since a path-specific ignore
only ever covers the directory someone thought of. Public keys and certificates
stay trackable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
setup-android's `packages` input defaults to `tools platform-tools`. `tools` is
obsolete and no longer in the SDK repository. The runner image ships
cmdline-tools 12.0, the action treats that as the wrong version and downloads
20.0, and 20.0 cannot resolve `tools`:

  sdkmanager tools
  Warning: Failed to find package 'tools'
  Error: ... failed with exit code 1

Licence acceptance succeeded; it was the package install that failed. Now asks
for platform-tools only and lets Gradle resolve the rest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A .gitleaksignore lists each known finding as <commit>:<path>:<rule>:<line>,
which in a public repository is a precise index to a credential in history --
readable with one `git show`. Out of hundreds of entropy hits that are nearly
all false positives, it publishes exactly the ones that are real, which is the
triage step done on the reader's behalf.

The file is removed and purged from this branch's history. Known history
findings are tracked privately instead, so the weekly full-history sweep is
report-only and must not gate. The PR-range scan is the gating one and needs no
exception list, because it only sees commits the PR adds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@forgetso
forgetso force-pushed the ci/required-check-and-security-scanning branch from cead02f to 08e3ae2 Compare September 16, 2026 14:33
@forgetso
forgetso merged commit 8daae99 into main Sep 16, 2026
20 checks passed
@forgetso
forgetso deleted the ci/required-check-and-security-scanning branch September 16, 2026 16:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant