Skip to content

feat(workflows): add the peer-advisory-drift reusable check - #17

Merged
msalvatti merged 3 commits into
mainfrom
feat/peer-advisory-drift
Aug 1, 2026
Merged

feat(workflows): add the peer-advisory-drift reusable check#17
msalvatti merged 3 commits into
mainfrom
feat/peer-advisory-drift

Conversation

@msalvatti

Copy link
Copy Markdown
Member

Dependabot audits what is installed. Nothing audits what a package declares it supports — and that is where the rot happens.

A library pinned to a current @nestjs/common in its own devDependencies is invisible to Dependabot while its peerDependencies range still admits a version with a published advisory. The consumer resolving that range gets to pick the low end.

The failure mode is that the declared range never changes and becomes wrong anyway. The advisory database is what moves. A range audited clean today starts admitting a vulnerable version the day a new advisory is published — no commit, no PR, no alert. That is why this is scheduled and not a release-checklist item: the degradation happens between releases, when nobody is reading the file.

This is not hypothetical. It is exactly how nest-notification went from "the only repo with peer ranges verified against the advisory database" to admitting a vulnerable @nestjs/common, with nobody touching the file.

What it reports

One row per declared range, because that is the unit of repair — one edit fixes every advisory it admits. Run against nest-auth today:

Field Package Declared Raise floor to Advisories Worst
peerDependencies @nestjs/common ^11.0.0 ^11.0.16 1 moderate
peerDependencies @nestjs/core ^11.0.0 ^11.1.18 1 moderate
peerDependencies next ^16.0.0 ^16.2.11 37 high

Grouping matters here: ungrouped, next alone produced 37 rows that all resolve to the same single edit, burying the one number the reader needs. Full advisory lists stay in collapsed <details>.

"Raise floor to" is the highest first-patched version among the matches, since that is what clears everything below it — not the first one in the list.

Behaviour

  • Opens a tracking issue, rewrites it on every run, and closes it automatically once every range is clear. Nobody has to remember to close it after raising a floor.
  • Ranges that are not semver (workspace:*, file:../x) are reported as unaudited rather than skipped silently — a silent skip is how a dependency quietly stops being covered.
  • Withdrawn advisories are excluded.
  • fail-on-drift is off by default. The issue is the signal; a permanently red scheduled run on the default branch just trains people to ignore it.

Verification

Not desk-checked — actually executed against real manifests:

Case Expected Got
nest-queue (floors already raised) 0 0
nest-auth (^11.0.0 on both) finds them 3 ranges / 39 matches
The suggested floors ^11.0.16 / ^11.1.18 / ^16.2.11 0 0
workspace:* and file:../x surfaced, not crashed 2 reported as unaudited

Two defects were found and fixed by running it rather than reading it:

  1. NODE_PATH does not apply to ESM resolution. The script imported semver and died with ERR_MODULE_NOT_FOUND on the first run. Fixed by writing the script beside the installed node_modules so import resolves by walking up from the file.
  2. The advisory database emits >= 1.2.0, < 1.4.1 — comma-separated. semver wants a space-separated conjunction; parsed as-is the comma reads as a union and matches far more than it should.

Supply chain

semver is installed into a throwaway $RUNNER_TEMP directory with --ignore-scripts, never into the caller's workspace — this job must not perturb the tree it audits, and a pnpm workspace would reject a stray npm install at its root anyway. Only GitHub-owned actions are used, so the org's third-party allow-list is unaffected.

semver rather than a hand-rolled comparison because range intersection is the kind of logic that looks trivial and is not: prerelease tails, || unions, and the >=x, <y form all have to be handled the way the package managers handle them.

⚠️ Needs the v1 tag moved

Callers pin bymaxone/.github/....yml@v1. After merge, v1 has to be moved to include this commit or every caller fails with "workflow not found". Say the word and I will move it.


Ready for your review — not merging it.

Dependabot audits what is installed. Nothing audits what a package declares
it supports, so a peerDependencies range keeps admitting a vulnerable version
long after the advisory lands — the range never changes, the advisory database
does. The drift arrives with no commit, no pull request and no alert.

Scheduled rather than event-driven for that reason: the degradation happens
between releases, when nobody is reading the file.

Reports one row per declared range with the floor to move to, opens a tracking
issue, rewrites it as findings change, and closes it once every range is clear.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a reusable GitHub Actions workflow (peer-advisory-drift) that audits declared dependency ranges (e.g. peerDependencies) against GitHub’s advisory database, and maintains a self-healing tracking issue when drift is detected.

Changes:

  • Introduces a reusable workflow that queries GitHub’s advisory GraphQL API and reports vulnerable version-range intersections in the job summary.
  • Creates/updates/closes a labeled tracking issue based on drift presence, with optional fail-on-drift behavior.
  • Installs semver in a temporary runner directory to avoid mutating the audited workspace.
Suppressed comments (1)

.github/workflows/peer-advisory-drift.yml:262

  • drift-count is documented as the number of declared ranges that admit a vulnerable version, but the script currently writes findings.length (number of advisory matches). This makes the reusable workflow output (and the optional fail message) report the wrong unit.
          appendFileSync(process.env.GITHUB_OUTPUT, `drift-count=${findings.length}\n`)

Comment thread .github/workflows/peer-advisory-drift.yml Outdated
Comment thread .github/workflows/peer-advisory-drift.yml Outdated
…byte

The query stopped at first: 100 with no pagination. That is not a theoretical
cap: `next` returns 159 advisories today, so the audit silently dropped a
third of them. Measured on a manifest declaring next ^16.0.0 -- 37 matches and
a worst severity of "high" before, 39 and "critical" after. Under-reporting
is the one failure mode a scheduled detector cannot have, because it reads as
clean. The page loop is bounded and THROWS on the bound rather than returning
a partial result.

The group key also carried a literal NUL byte where a space belonged. It did
not change behaviour -- the key stayed unique -- but it made the file binary to
grep and truncated it under awk, which is a trap for whoever edits this next.
Copilot AI review requested due to automatic review settings August 1, 2026 13:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

.github/workflows/peer-advisory-drift.yml:284

  • drift-count is documented as the number of declared ranges that admit a vulnerable version, but the script writes drift-count=${findings.length} (advisory matches). This makes the output misleading (e.g., “39” instead of “3”) and can confuse callers consuming the reusable workflow output.
          appendFileSync(process.env.GITHUB_STEP_SUMMARY, `## Peer advisory drift\n\n${report}\n`)
          appendFileSync(process.env.GITHUB_OUTPUT, `drift-count=${findings.length}\n`)
          appendFileSync(process.env.GITHUB_OUTPUT, `skipped-count=${skipped.length}\n`)

.github/workflows/peer-advisory-drift.yml:190

  • The script can fetch the same package’s vulnerability list multiple times if a package appears in more than one audited field (the comment above explicitly calls this out). This increases runtime and risks hitting GitHub GraphQL rate limits unnecessarily; cache vulnerabilitiesFor(name) per package name and reuse it across declarations.
            for (const vuln of await vulnerabilitiesFor(name)) {

Comment thread .github/workflows/peer-advisory-drift.yml
…package

drift-count is documented as the number of declared ranges that admit a
vulnerable version, but it was emitting the advisory total -- 39 where the
maintainer has one line to edit. It now emits the range count, and the total
moves to a new advisory-count output.

The advisory list is also cached per package. A package declared in more than
one audited field was fetched once per declaration; measured with next in two
fields, 4 round trips become 2. Cached only after the last page, so a partial
result is never served as the whole answer.
Copilot AI review requested due to automatic review settings August 1, 2026 13:22
@msalvatti

Copy link
Copy Markdown
Member Author

Picking up the two suppressed comments from the last review — both were right, both fixed in 7a7bd95.

drift-count was reporting the wrong quantity

The output is documented as "number of declared ranges that admit a vulnerable version" and was emitting findings.length, the advisory total. On a manifest declaring next ^16.0.0 it said 39 where the maintainer has one line to edit. A caller gating on it would have been reading a number that does not mean what its own description says.

drift-count=1   advisory-count=39   skipped-count=0

drift-count now counts ranges — the unit of repair — and the total moved to a new advisory-count output rather than being dropped, since the report still cites it. The description was tightened to say which is which, because "drift count" reads naturally as either one.

The same package was fetched once per declaration

Real, and my own comment had flagged the multi-field case without acting on it. Cached per package name and measured with next declared in both peerDependencies and dependencies, counting actual fetch calls:

GraphQL round trips
next in one field 2
next in two fields, before 4
next in two fields, after 2

Two rather than one because next spans two pages — the pagination fix from the sibling thread.

The cache is populated only after the final page, never mid-loop. Storing a partial page set would hand every later caller a short list as though it were complete, which is the same silent-under-report failure the pagination fix exists to remove.

Verified after both changes

nest-queue still reports drift-count=0 advisory-count=0, and the next case reports 1 / 39 with ^16.2.11 as the floor.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (4)

.github/workflows/peer-advisory-drift.yml:333

  • The tracking issue is closed/skipped when DRIFT_COUNT=0, even if skipped-count>0 (unaudited ranges). If the issue is meant to be the primary signal, include SKIPPED_COUNT in the “no drift” condition so unaudited declarations still open/update the issue.
          if [ "$DRIFT_COUNT" -eq 0 ]; then

.github/workflows/peer-advisory-drift.yml:126

  • SEVERITY_THRESHOLD.toUpperCase() will throw if the caller passes an empty string (or omits the env var), and an empty DEPENDENCY_TYPES will silently result in fields=[] (no declarations audited). Add defensive defaults/validation so a misconfigured caller fails loudly instead of appearing “clean”.
          const RANK = { LOW: 0, MODERATE: 1, HIGH: 2, CRITICAL: 3 }
          const threshold = RANK[process.env.SEVERITY_THRESHOLD.toUpperCase()] ?? 0
          const manifestPath = process.env.MANIFEST_PATH
          const fields = process.env.DEPENDENCY_TYPES.split(',').map((f) => f.trim()).filter(Boolean)

.github/workflows/peer-advisory-drift.yml:329

  • gh label create … 2>/dev/null || true hides real failures (e.g. missing permissions / API errors), which makes scheduled-run breakages harder to diagnose. Prefer a create-or-edit flow that keeps unexpected errors visible.
          gh label create "$LABEL" --color B60205 \
            --description "A declared range admits a version with a published advisory" \
            2>/dev/null || true

.github/workflows/peer-advisory-drift.yml:320

  • The audit script writes skipped-count, but the issue logic only considers DRIFT_COUNT. If a repo only has unaudited (non-semver) ranges, this workflow will close/avoid the tracking issue even though the run explicitly found uncovered declarations. Export SKIPPED_COUNT so the issue logic can keep the signal visible.

This issue also appears on line 333 of the same file.

        env:
          GH_TOKEN: ${{ github.token }}
          LABEL: ${{ inputs.issue-label }}
          DRIFT_COUNT: ${{ steps.audit.outputs.drift-count }}
          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

@msalvatti
msalvatti merged commit bc2aa4c into main Aug 1, 2026
5 checks passed
@msalvatti
msalvatti deleted the feat/peer-advisory-drift branch August 1, 2026 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants