[dgesteves]

What was wrong

The corpus installs nothing but MUI for the two MUI apps (install in corpus/repos.json), so it never sees what types decide: which props and variants a component takes, and whether a native element's attributes fit a design-system component. Users and the hook run onsystem with node_modules installed. On the five repositories bench/v2 installs, installing each one from its own lockfile changes these counts (node scripts/corpus.mjs --installed, src as on main at 7ca0f69):

Run	Rule	Without dependencies	With them
shadcn-website	prefer-design-system-component	9 (7 fixable)	9 (9 fixable)
openstatus-dashboard	prefer-design-system-component	15 (8 fixable)	15 (15 fixable)
openstatus-root	prefer-design-system-component	44 (37 fixable)	44 (44 fixable)
dub-web	no-unknown-prop	0	4 (0 fixable)
dub-web	prefer-design-system-component	1000 (589 fixable)	1000 (968 fixable)
dub-web-configured	no-unknown-prop	0	4 (0 fixable)
dub-web-configured	prefer-design-system-component	973 (589 fixable)	1000 (995 fixable)
dub-root	no-unknown-prop	0	4 (0 fixable)
dub-root	prefer-design-system-component	1015 (604 fixable)	1015 (983 fixable)

Colour, spacing, radius and icon-button counts are the same either way. None of these rows was in corpus/snapshot.json, and its blind sample was drawn without types.

What changed

• pnpm corpus --installed (also with fetch, sample, --only and --update). It takes the repositories in bench/v2/config/repos.json at the commits and folders corpus/repos.json pins, checks them out again under $CORPUS_DIR/.installed (the plain runs never see that node_modules), and installs each with bench/v2's own command, pnpm install --frozen-lockfile --ignore-scripts, plus --ignore-pnpmfile. The install gets no environment variable that looks like a secret. It runs every corpus run of those repositories except dub-web-package, whose init copies the checkout and would copy the install with it: 11 runs, named <run>-installed so their fingerprints and labels stay apart from the plain runs. It compares the counts with the new corpus/snapshot-installed.json and fails on a change without --update, as the plain corpus does, and prints the table above against corpus/snapshot.json.

• Weekly CI. A new installed job in .github/workflows/corpus.yml runs on Mondays at 04:41 UTC and on workflow_dispatch, and writes its report to the job summary. The daily job does not run on the weekly schedule. The concurrency group now includes the schedule, so the two scheduled runs cannot cancel each other. It does not run on pull requests.

• A frozen holdout, corpus/installed-holdout-20261016/, drawn with pnpm corpus draw --installed --seed 20261016 --per-rule 30: 109 of the 12,945 distinct installed findings, up to 30 per rule, so each rule gets a rate of its own:

  Rule	Population	Sampled
  no-hardcoded-color	11,752	30
  prefer-design-system-component	1,089	30
  icon-button-accessible-name	85	30
  no-hardcoded-radius	8	8
  no-hardcoded-spacing	7	7
  no-unknown-prop	4	4

  The folder has the labeller's input (sample-unlabelled.json: no fix, severity or label), the key, the allocation with the onsystem commit and a population hash, a README with the method and the commands that rebuild it, and BRIEF.md, the labelling brief: what the labeller may read, must not open (the key, the allocation, corpus/labels.json, both snapshots, the earlier sample) and writes. It is not labelled yet. Once sample-labelled.json is there, pnpm corpus --installed prints each rule's false-positive rate with a 95% Wilson interval, and whether every drawn finding is still reported. Its labels stay in that folder: test/readme.test.ts and site/scripts/generate.mjs expect one blind draw in corpus/labels.json.

• scripts/corpus-lib.ts: installedRepos (refuses an install without --frozen-lockfile and --ignore-scripts, or anything but pnpm), typeDriven and typeDrivenTable, distinctFindings (the README's definition: no re-check with a config, only what a root run adds), drawHoldout, wilson and ruleRates. outside is now generic over the finding type.

The finding proposed pinning @types/react and the primitives per repository. Reusing bench/v2's lockfile installs instead covers every type the apps use, at the versions they use, with nothing new to pin. The finding's figures (66 vs 56 prefer-design-system-component fixes in shadcn) came from an older main; the table above is today's.

CI time

Measured on this Mac, which other jobs share (1-minute load between 4 and 29 during the runs):

• fetch of the five repositories: 13.6 s;
• installs with an empty pnpm store: 98 s (saas-starter 4, chatbot 11, shadcn-ui 31, openstatus 22, dub 30), or 54 s with a warm store;
• the check of the 11 runs: 66 s;
• the setup steps the job shares with the corpus job (job setup, checkout, pnpm, Node, install, build): 25 s in corpus run 38866423410.

That is about 3.4 minutes once a week. Nothing is cached, since the installs take gigabytes. The first Monday run, or a manual one, will show the runner's figure.

How it was tested

• test/corpus-installed.test.ts, 15 tests: the installed repositories and runs from the real configs; the install command checks; that the installed snapshot covers every bench repository and shows a no-unknown-prop count and a prefer-design-system-component fix count that only types give; typeDriven; the workflow's weekly job; distinctFindings, drawHoldout (same seed, same draw, whatever the order) and the Wilson interval, which matches the earlier sample's stats.py (9 of 80: 6.0-20.0%); and the holdout's own files: allocation, fingerprints, no fix, severity or label for the labeller, and the brief's list of files not to open. Run against main's scripts and corpus files, all 11 of its tests that exist there fail (the four per-holdout tests need the folder): installedRepos does not exist and neither does corpus/snapshot-installed.json.
• node scripts/corpus.mjs --installed: "Counts match corpus/snapshot-installed.json.", the root runs report what their project runs do, file for file, and "Holdout installed-holdout-20261016: 109 of its 109 findings still reported."
• node scripts/corpus.mjs --only saas-starter --only chatbot --only shadcn-ui --only openstatus --only dub (plain): "Counts match corpus/snapshot.json." The corpus job on this pull request runs every plain run.
• A second draw into another folder wrote byte-identical sample-unlabelled.json, sample-key.json and allocation.json.
• pnpm install --frozen-lockfile --prefer-offline, pnpm format:check, pnpm lint, pnpm typecheck, pnpm test (74 files and 972 tests passed, 3 skipped) and pnpm build.

No changeset: scripts, CI and corpus data only, nothing in the published package.

Follow-ups

• Label the holdout per its BRIEF.md, by a reviewer who did not draw it, then quote the per-rule rates and say they come from the installed corpus.
• CONTRIBUTING.md's "Real-world corpus" section should describe pnpm corpus --installed, the weekly job and the holdout. It is outside this change's files.
• feat: read installed design-system packages that publish their sources #176 (installed design-system packages that publish their sources) and fix(cli): check only JS and TS files, gate on nothing checked, and summarize long runs #180 (check only JS and TS files) change what the installed runs report. After they merge, the weekly job fails until someone runs pnpm corpus --installed --update. If a change takes out a holdout finding, draw a fresh holdout with a new seed.
• no-unknown-variant reports nothing in these five repositories even with types, so the Radix type="multiple" false positive is not in this corpus. Its unit test with real Radix types stays with OS2-02.

Roadmap item: OS2-39.