Skip to content

build: migrate from npm to pnpm 11 #106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
HughParry wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

build: migrate from npm to pnpm 11 #106

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 51 additions & 27 deletions .github/scripts/check-pinned-versions.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#!/usr/bin/env node
// Supply-chain guard: fail CI if any package.json declares a non-exact (floating)
// version for a runtime dependency, or if package-lock.json is missing integrity
// version for a runtime dependency, or if pnpm-lock.yaml is missing integrity
// hashes for resolved registry packages.
//
// Floating ranges (^, ~, *, >=, "latest", ...) let `npm install` silently pull a
// Floating ranges (^, ~, *, >=, "latest", ...) let `pnpm install` silently pull a
// newer release than the one that was reviewed. When that newer release is
// malicious, every fresh install / CI run is compromised before anyone notices.
// Recent npm supply-chain attacks that worked exactly this way:
Expand All @@ -19,8 +19,9 @@
// password stealer on install.
// - Nov 2018 event-stream / flatmap-stream — transitive dep backdoored to steal
// bitcoin-wallet credentials.
// Pinning exact versions + committing the lockfile + `npm ci` means a new malicious
// release is NOT pulled until the version is explicitly bumped and reviewed.
// Pinning exact versions + committing the lockfile + `pnpm install --frozen-lockfile`
// means a new malicious release is NOT pulled until the version is explicitly
// bumped and reviewed.
//
// peerDependencies are intentionally exempt: they express a compatibility *range*
// against whatever the consumer installs, so pinning them would wrongly constrain
Expand Down Expand Up @@ -145,37 +146,60 @@ function checkPackageJson(file) {
/**
* Every resolved registry package in the lockfile must carry an integrity hash,
* so a tampered tarball cannot be substituted for the reviewed one.
*
* pnpm-lock.yaml is parsed line by line so this check runs without installing
* a YAML parser. pnpm writes each `packages:` entry as a two-space-indented key
* followed by a four-space-indented flow-style `resolution: {...}` line.
*/
function checkLockfile(file) {
let lock;
try {
lock = JSON.parse(readFileSync(file, "utf8"));
} catch (e) {
errors.push(`${relative(ROOT, file)}: invalid JSON (${e.message})`);
return;
}
const rel = relative(ROOT, file);
if ((lock.lockfileVersion ?? 0) < 2) {
const lines = readFileSync(file, "utf8").split("\n");

const rawVersion =
lines
.find((l) => l.startsWith("lockfileVersion:"))
?.replace(/^lockfileVersion:\s*/, "")
.replace(/['"]/g, "") ?? "(missing)";
if (!(Number.parseFloat(rawVersion) >= 9)) {
errors.push(
`${rel}: lockfileVersion ${lock.lockfileVersion} is too old; needs >= 2 for integrity hashes`,
`${rel}: lockfileVersion ${rawVersion} is not supported; needs pnpm lockfile v9+`,
);
return;
}
const packages = lock.packages || {};
for (const [key, entry] of Object.entries(packages)) {
// Root project and workspace members ("" and workspace dirs) and local
// links have no registry tarball / integrity — skip them.
if (key === "" || !key.includes("node_modules/")) continue;
if (entry.link === true) continue;
// Only registry-resolved deps must have integrity. git/file/url deps are
// pinned by their resolved field instead.
const resolved = entry.resolved || "";

let inPackages = false;
let current = null;
const unresolved = new Set();
for (const line of lines) {
if (/^\S/.test(line)) {
inPackages = line === "packages:";
current = null;
continue;
}
if (!inPackages) continue;
const key = /^ {2}(\S.*):$/.exec(line);
if (key) {
current = key[1].replace(/^'(.*)'$/, "$1");
unresolved.add(current);
continue;
}
const res = /^ {4}resolution:\s*\{(.*)\}\s*$/.exec(line);
if (!res || current === null) continue;
unresolved.delete(current);
const fields = res[1];
// git and local directory deps are pinned by commit / path instead.
if (/\btype:\s*(git|directory)\b/.test(fields)) continue;
// A bare tarball URL outside the registry is pinned by the URL itself.
const tarball = /\btarball:\s*([^,\s}]+)/.exec(fields)?.[1] ?? "";
const isRegistry =
resolved === "" || /^https?:\/\/[^/]*registry\./.test(resolved);
if (isRegistry && !entry.integrity) {
errors.push(`${rel} ${key}: missing integrity hash`);
tarball === "" || /^https?:\/\/[^/]*registry\./.test(tarball);
if (isRegistry && !/\bintegrity:\s*\S/.test(fields)) {
errors.push(`${rel} ${current}: missing integrity hash`);
}
}
for (const name of unresolved) {
errors.push(`${rel} ${name}: no resolution entry`);
}
}

const pkgFiles = findPackageJsons(ROOT);
Expand All @@ -184,7 +208,7 @@ for (const f of pkgFiles) checkPackageJson(f);
// Lockfiles live next to each package.json that owns one.
const seenLocks = new Set();
for (const f of pkgFiles) {
const lock = join(dirname(f), "package-lock.json");
const lock = join(dirname(f), "pnpm-lock.yaml");
if (seenLocks.has(lock)) continue;
try {
statSync(lock);
Expand All @@ -203,7 +227,7 @@ if (errors.length > 0) {
console.error(
"\nDependencies in dependencies/devDependencies/optionalDependencies must use an" +
'\nexact version (e.g. "1.2.3", not "^1.2.3"). peerDependencies may use ranges.' +
"\nThis prevents `npm install` from silently pulling a malicious newer release." +
"\nThis prevents `pnpm install` from silently pulling a malicious newer release." +
"\nRun `node .github/scripts/check-pinned-versions.mjs` locally to reproduce.",
);
process.exit(1);
Expand Down
141 changes: 38 additions & 103 deletions .github/scripts/pin-versions.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,19 @@
#!/usr/bin/env node
// One-shot helper used to introduce exact pinning: rewrites every floating
// (^, ~, range, *, latest) specifier in dependencies/devDependencies/
// optionalDependencies to the exact version that package-lock.json already
// resolved it to. Because the lockfile is the source of truth for `npm ci`,
// this does NOT change what gets installed — it only makes package.json declare
// the version explicitly. peerDependencies are left untouched.
// optionalDependencies to the exact version pnpm-lock.yaml already resolved it
// to. Because the lockfile is the source of truth for
// `pnpm install --frozen-lockfile`, this does NOT change what gets installed —
// it only makes package.json declare the version explicitly. peerDependencies
// are left untouched.
//
// Resolution: for a workspace at dir D depending on N, the installed version is
// the lockfile `packages` entry nearest to D walking up node_modules dirs, then
// the hoisted root node_modules/N. This mirrors Node's module resolution.
// Resolution: installed versions come from `pnpm list --recursive --json`, which
// reports each workspace project's direct dependencies as pnpm resolved them.
// Run `pnpm install` before this script, and again afterwards so the lockfile's
// recorded specifiers match the rewritten package.json files.

import { readFileSync, readdirSync, statSync, writeFileSync } from "node:fs";
import { execFileSync } from "node:child_process";
import { readFileSync, readdirSync, writeFileSync } from "node:fs";
import { dirname, join, relative, resolve } from "node:path";

const ROOT = process.cwd();
Expand Down Expand Up @@ -70,73 +73,37 @@ function findPackageJsons(dir, out = []) {
return out;
}

// Find the package-lock.json that governs a given package.json (nearest ancestor).
function findLockFor(pkgFile) {
let dir = dirname(pkgFile);
for (;;) {
const lock = join(dir, "package-lock.json");
try {
statSync(lock);
return lock;
} catch {}
const parent = dirname(dir);
if (parent === dir || !dir.startsWith(ROOT)) return null;
dir = parent;
}
}

// Detect a file's indentation (tab, or N spaces) so we re-serialise it the same
// way npm did and keep the diff to just the changed lines.
function detectIndent(text) {
const m = /\n(\t+|[ ]+)"/.exec(text);
if (!m) return "\t";
return m[1][0] === "\t" ? "\t" : " ".repeat(m[1].length);
}
const lockIndent = new Map();

const lockCache = new Map();
function loadLock(lockFile) {
if (!lockCache.has(lockFile)) {
const text = readFileSync(lockFile, "utf8");
lockIndent.set(lockFile, detectIndent(text));
lockCache.set(lockFile, JSON.parse(text));
}
return lockCache.get(lockFile);
}

// Resolve installed version of dep `name` for the workspace at `pkgDir`.
function resolveVersion(lock, lockFile, pkgDir, name) {
const packages = lock.packages || {};
// workspace path relative to the lockfile root, using forward slashes
const relDir = relative(dirname(lockFile), pkgDir).split("\\").join("/");
const segments = relDir === "" ? [] : relDir.split("/");
// Walk up: <ws>/node_modules/name, then parents, then root node_modules/name
for (let i = segments.length; i >= 0; i--) {
const prefix = segments.slice(0, i).join("/");
const key = `${prefix ? `${prefix}/` : ""}node_modules/${name}`;
if (packages[key]?.version) return packages[key].version;
// Map each workspace project's directory to section -> dep name -> version.
function loadInstalledVersions() {
const out = execFileSync(
"pnpm",
["list", "--recursive", "--depth", "0", "--json"],
{ cwd: ROOT, encoding: "utf8", maxBuffer: 64 * 1024 * 1024 },
);
const projects = new Map();
for (const project of JSON.parse(out)) {
const sections = {};
for (const section of ENFORCED_SECTIONS) {
sections[section] = {};
for (const [name, info] of Object.entries(project[section] || {})) {
if (info?.version) sections[section][name] = info.version;
}
}
projects.set(resolve(project.path), sections);
}
return null;
return projects;
}
const installed = loadInstalledVersions();

const pkgFiles = findPackageJsons(ROOT);
let totalChanged = 0;
const unresolved = [];

for (const file of pkgFiles) {
const pkg = JSON.parse(readFileSync(file, "utf8"));
const lockFile = findLockFor(file);
if (!lockFile) continue;
const lock = loadLock(lockFile);
const pkgDir = dirname(file);

// Only resolve a version from the lockfile when this package.json is an actual
// workspace tracked by that lockfile. For standalone dirs (not installed as
// workspaces), walking up node_modules would borrow an unrelated, hoisted
// version from another package — possibly a different major — so we must NOT
// resolve them from the lockfile; we floor-strip their range instead.
const relDir = relative(dirname(lockFile), pkgDir).split("\\").join("/");
const isWorkspace = relDir === "" || Boolean(lock.packages?.[relDir]);
// Standalone dirs that are not workspace projects have no resolved versions
// here, so their ranges are floor-stripped below instead.
const resolved = installed.get(resolve(dirname(file)));

// Collect replacements: section -> name -> newVersion
const targets = {};
Expand All @@ -145,7 +112,7 @@ for (const file of pkgFiles) {
if (!deps) continue;
for (const [name, spec] of Object.entries(deps)) {
if (isPinned(String(spec))) continue;
let v = isWorkspace ? resolveVersion(lock, lockFile, pkgDir, name) : null;
let v = resolved?.[section]?.[name] ?? null;
// Fallback for packages not resolvable from the lockfile (standalone
// dirs, or unlisted deps): pin a simple ^/~ range to its floor version,
// which is an exact pin that stays within the declared major.
Expand Down Expand Up @@ -208,40 +175,8 @@ if (unresolved.length) {
for (const u of unresolved) console.log(` ${u}`);
}

// Sync the recorded ranges inside each lockfile's workspace entries so they
// mirror the now-pinned package.json. We do NOT re-resolve the tree (which would
// drop entries for any uninitialised submodule workspace) — we only rewrite the
// declared ranges for workspace packages, keeping resolved versions/integrity
// untouched. npm ci requires these recorded ranges to match package.json.
for (const lockFile of lockCache.keys()) {
const lock = loadLock(lockFile);
const lockRoot = dirname(lockFile);
const packages = lock.packages || {};
let synced = 0;
for (const file of pkgFiles) {
if (findLockFor(file) !== lockFile) continue;
const key = relative(lockRoot, dirname(file)).split("\\").join("/");
const entry = packages[key];
if (!entry) continue; // standalone dir not tracked as a workspace
const pkg = JSON.parse(readFileSync(file, "utf8"));
for (const section of ENFORCED_SECTIONS) {
if (!pkg[section] || !entry[section]) continue;
for (const name of Object.keys(entry[section])) {
if (
pkg[section][name] !== undefined &&
entry[section][name] !== pkg[section][name]
) {
entry[section][name] = pkg[section][name];
synced++;
}
}
}
}
if (synced > 0) {
const indent = lockIndent.get(lockFile) || "\t";
writeFileSync(lockFile, `${JSON.stringify(lock, null, indent)}\n`);
console.log(
`Synced ${synced} recorded range(s) in ${relative(ROOT, lockFile)}`,
);
}
if (totalChanged > 0) {
console.log(
"\nRun `pnpm install` to record the pinned specifiers in pnpm-lock.yaml.",
);
}
29 changes: 6 additions & 23 deletions .github/workflows/bump_version.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,28 +59,11 @@ jobs:
submodules: "recursive"
fetch-depth: 0 # Fetch all history for all branches

- uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
- uses: actions/setup-node@v4
with:
node-version-file: 'package.json'
- run: npm i -g "npm@$(jq -r .engines.npm < package.json)"

- run: mkdir -p ~/.npm
- run: mkdir -p ~/.cache/Cypress

- name: Restore npm cache
if: ${{ runner.environment != 'self-hosted' }} # don't restore cache on self-hosted runners, network speed not good enough
uses: actions/cache/restore@v4
with:
# must restore all cache dirs, and they must exist ahead of this!
path: |
~/.npm
~/.cache/Cypress
# note that restoring a cache in github is a pain. The trailing '-' matches any string after the '-', therefore 'abc-' would match a cache named 'abc-1234' or 'abc-5678', etc.
# the problem is 'abc-' will not match a cache named 'abc'! So if you're using wildcard cache name selectors like this, you need a field that changes as the suffix to become the wildcard
# here we're setting the key to an unused cache key so it falls back to the wildcard selector in `restore-keys`
key: some-unused-cache-key
restore-keys: |
npm-${{ runner.os }}-${{ runner.arch }}-
cache: pnpm

- name: Bump version
run: |
Expand Down Expand Up @@ -136,10 +119,10 @@ jobs:
done

# the json may be poorly formatted now, so we need to fix it
npm i
npm run -w @prosopo/scripts build
npm run -w @prosopo/lint build
npm run lint-fix
pnpm install
pnpm --filter @prosopo/scripts run build
pnpm --filter @prosopo/lint run build
pnpm run lint-fix

# commit the version changes
git add .
Expand Down
40 changes: 4 additions & 36 deletions .github/workflows/cache.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,43 +41,11 @@ jobs:
with:
submodules: "recursive"

# make cache dirs, these must exist before doing cache stuff
- run: mkdir -p ~/.npm
- run: mkdir -p ~/.cache/Cypress

# don't restore the cache, otherwise we get artifacts left over from the previous run sit in the cache for eternity. Do a clean build every time.
- run: npm cache clean --force

- uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
- uses: actions/setup-node@v4
with:
node-version-file: 'package.json'
- run: npm i -g "npm@$(jq -r .engines.npm < package.json)"

- run: npm ci --no-audit --no-fund --prefer-offline


# install cypress so we don't have to do that every time
- run: npx cypress install

- name: Save npm cache
uses: actions/cache/save@v4
if: always()
with:
path: |
~/.npm
~/.cache/Cypress
key: npm-${{ runner.os }}-${{ runner.arch }}-${{ github.run_id }}-${{ github.run_attempt }}
cache: pnpm

- name: Cleanup npm caches
if: always()
run: |
# remove all but the latest cache, leaving only the cache we just saved
echo "Fetching list of cache key"
cacheKeys=$(gh cache list --sort created_at --order desc --limit 100 -R "${{ github.repository }}" --key "npm-${{ runner.os }}-${{ runner.arch }}-" | cut -f 1 | tail -n +3)
echo caches to be removed:
echo "${cacheKeys}"
set +e
for cacheKey in $cacheKeys
do
gh cache delete "$cacheKey" -R "${{ github.repository }}"
done
# setup-node saves the pnpm store on its post step, keyed on the pnpm-lock.yaml hash
- run: pnpm install --frozen-lockfile
Loading