feat(corpus): check the bench repositories with their dependencies installed, weekly, and draw a holdout to label blind

.github/workflows/corpus.yml

@@ -3,6 +3,9 @@ name: Corpus
 # Checks the built CLI on public repositories pinned by commit (corpus/repos.json) and compares
 # the findings with corpus/snapshot.json and the labels in corpus/labels.json. Not a required
 # check yet: it reports, and a change of counts needs `pnpm corpus --update` in the pull request.
+# Once a week, and on demand, the installed-types corpus checks the repositories bench/v2
+# installs with their dependencies installed, as users run onsystem, against
+# corpus/snapshot-installed.json (`pnpm corpus --installed --update` records a change).
 on:
   pull_request:
     paths:
@@ -14,18 +17,22 @@ on:
       - '.github/workflows/corpus.yml'
   schedule:
     - cron: '23 3 * * *'
+    # The installed-types corpus (the `installed` job), on Mondays.
+    - cron: '41 4 * * 1'
   workflow_dispatch:
 
 permissions:
   contents: read
 
 concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
+  # The daily and the weekly schedule are separate groups, so neither cancels the other.
+  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.schedule }}
   cancel-in-progress: true
 
 jobs:
   corpus:
     name: Real-world corpus
+    if: github.event.schedule != '41 4 * * 1'
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v7
@@ -68,3 +75,37 @@ jobs:
       # The same runs through the ESLint plugin, which must report what check reports.
       - name: Check the corpus through ESLint
         run: pnpm --silent corpus eslint
+
+  installed:
+    name: Installed-types corpus
+    if: github.event_name == 'workflow_dispatch' || github.event.schedule == '41 4 * * 1'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: pnpm/action-setup@v6
+      - uses: actions/setup-node@v7
+        with:
+          node-version: 24
+          cache: pnpm
+      - run: pnpm install --frozen-lockfile
+      - run: pnpm build
+      # Outside the checkout: the CLI looks upwards for node_modules and workspace roots.
+      - run: echo "CORPUS_DIR=${{ runner.temp }}/corpus" >> "$GITHUB_ENV"
+      # Fetches the pinned commits, then installs each repository from its own lockfile with
+      # bench/v2's command (bench/v2/config/repos.json): pnpm with --frozen-lockfile and
+      # --ignore-scripts, plus --ignore-pnpmfile. Not cached: the installs take gigabytes.
+      - run: pnpm --silent corpus fetch --installed
+      - name: Check the installed-types corpus
+        shell: bash
+        run: |
+          set +e
+          pnpm --silent corpus --installed > corpus-report.txt 2>&1
+          status=$?
+          cat corpus-report.txt
+          {
+            echo '## Installed-types corpus'
+            echo '```text'
+            cat corpus-report.txt
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"
+          exit "$status"

corpus/installed-holdout-20261016/BRIEF.md

@@ -0,0 +1,62 @@
+# Labelling brief: installed-holdout-20261016
+
+You label 109 findings that onsystem reported in five public repositories with their dependencies installed. Each label says whether the finding is right. Your labels give each rule's false-positive rate, so they must be blind: judge every finding from the code alone, without any other label, rate or opinion about onsystem.
+
+Who labels: a reviewer who did not draw this sample and has not worked on onsystem's rules, its corpus or its samples. When the reviewer is an AI, its labels are published as "blind labels by an AI reviewer", never as labels by a person, and `LABELLER.md` names the model by its id.
+
+## Your input
+
+`sample-unlabelled.json` in this folder: the findings H001 to H109, in a shuffled order. For each one it gives:
+
+- where it is: the repository, its pinned commit, a permalink, the file, line and column;
+- the rule id;
+- the message, exactly as onsystem printed it;
+- the matched source text;
+- 15 lines of code from the pinned commit, with the reported line marked `>`.
+
+These findings come from `onsystem check` run with each repository's dependencies installed from its own lockfile, so onsystem saw the TypeScript types of the components. Judge them the same way. When a finding says a component takes no such prop or variant, read the props the component's types declare, including what it spreads from a native element or a library primitive.
+
+## What you may read
+
+- This brief and `sample-unlabelled.json`. Nothing else in this folder.
+- onsystem's `docs/rules.md` on `main`: only the introduction, the rules table and "How fixes are chosen".
+- The five repositories, read-only, at the commits the sample pins: source tarballs from codeload.github.com or the GitHub API. Read the finding's whole file, the project's theme and CSS variables (Tailwind config, `globals.css` and the like) and its components.
+- The type declarations of a package a finding depends on, at the version the repository's lockfile pins, from that version's npm tarball (`.d.ts` files only).
+
+Nothing from the repositories or the packages is installed, built or run.
+
+## What you must not open
+
+- In this folder: `sample-key.json` (severity, fixes and weights), `allocation.json` (how many findings each rule has), `README.md`, and any `sample-labelled.json` or `LABELLER.md` from an earlier attempt.
+- In onsystem: `corpus/labels.json`, `corpus/snapshot.json`, `corpus/snapshot-installed.json`, anything in `corpus/fp-sample-20261016`, any other sample, the README's paragraph on false positives, the website's numbers, the source in `src/`, and issues or pull requests about false positives.
+- The web beyond the sources above: no search.
+- Credential files and tokens of any kind. Write nothing anywhere except the two files below.
+
+## How to label
+
+Use the definitions of the earlier blind sample, so the rates compare:
+
+- **TP**: a maintainer who cares about that project's design system would want the finding flagged, and the message's claim is correct.
+- **FP**: the code is fine as written, or the message's claim is false.
+- **debatable**: reasonable maintainers would disagree, or the answer depends on intent you cannot see.
+
+Record for each finding:
+
+- `label`: `TP`, `FP` or `debatable`.
+- `own_ds_value`: `true` when the flagged value appears deliberately in the project's design-system package, theme or CSS variables; `false` when it does not; `null` for rules that are not about a value (component, prop, variant, accessible name).
+- `replacement_wrong`: `null` when the message names no replacement; `true` only when the replacement it names is factually wrong (a component that does not fit, a token with another value); `false` otherwise.
+- `reason`: one or two sentences with the evidence, citing files and lines.
+- `looked_at`: the permalinks you read for it.
+
+Settle on working rules for cases that repeat, apply them to every finding, and write them down.
+
+## What to write
+
+1. `sample-labelled.json` in this folder: `sample-unlabelled.json` with the same findings in the same order, each with the five fields above added and nothing else changed, and a `$comment` that says the labels were added and by which reviewer.
+2. `LABELLER.md` in this folder: who labelled (for an AI, its model id), the date, what you read and what you did not open, your working rules, how long it took, and the findings you found hard.
+
+Compute no rates or statistics, and do not add the labels to `corpus/labels.json`: that file keeps the plain corpus's blind sample, which the README quotes.
+
+## After the labels
+
+Whoever opens the pull request with the two files runs `pnpm build`, then `pnpm corpus --installed`. It prints, for each rule, the share of false positives with its 95% Wilson interval, and how many of the holdout's findings are still reported; the table goes in the pull request. The pull request changes nothing else in this folder, says that the labels are blind labels by an AI reviewer when they are, and names the model.

corpus/installed-holdout-20261016/README.md

@@ -0,0 +1,63 @@
+# Installed-types holdout, seed 20261016
+
+A seeded random draw of 109 of the 12,945 distinct findings that the installed-types corpus reported on onsystem `7ca0f69` (5 repositories pinned by commit, 11 runs, each repository installed from its own lockfile with scripts off). It takes up to 30 findings of each rule, and every finding of a rule that has fewer, so each rule gets a false-positive rate of its own. It is not labelled yet: [BRIEF.md](BRIEF.md) is the labelling brief.
+
+## Why a second sample
+
+The corpus job checks its repositories without their dependencies, apart from MUI for the two MUI apps. Its blind sample ([corpus/fp-sample-20261016](../fp-sample-20261016)) therefore measures onsystem without types. Users and the Claude Code hook run onsystem with `node_modules` installed, and types decide which props and variants a component takes and whether a native element's attributes fit a design-system component. `pnpm corpus --installed` checks the five repositories that bench/v2 installs (`bench/v2/config/repos.json`) with their dependencies, and prints what installing changes against the same runs without them. This holdout gives the false-positive rate of each rule in that setting.
+
+## Frozen
+
+- The draw is never redone or edited. Once its findings change (a fix takes one out, or the population it was drawn from moves on), its rates no longer hold: draw a fresh holdout with a new seed into a new folder.
+- It is a holdout: its labels measure the rules, and are not used to find false positives to fix. Work on a rule uses `corpus/labels.json` and `pnpm corpus sample`.
+- Its labels stay in this folder, in `sample-labelled.json`. `corpus/labels.json` keeps the plain corpus's labels.
+
+## Allocation (n = 109, seed 20261016)
+
+From [allocation.json](allocation.json):
+
+| Rule                           | Population | Sampled | Weight |
+| ------------------------------ | ---------: | ------: | -----: |
+| no-hardcoded-color             |     11,752 |      30 | 391.73 |
+| prefer-design-system-component |      1,089 |      30 |  36.30 |
+| icon-button-accessible-name    |         85 |      30 |   2.83 |
+| no-hardcoded-radius            |          8 |       8 |   1.00 |
+| no-hardcoded-spacing           |          7 |       7 |   1.00 |
+| no-unknown-prop                |          4 |       4 |   1.00 |
+| **Total**                      | **12,945** | **109** |        |
+
+The population counts distinct findings as the README counts the plain corpus's: `dub-web-configured-installed`, which re-checks dub-web's files with a config, is left out, and a run from a monorepo root (`dub-root-installed`, `openstatus-root-installed`) counts only what it adds outside the folders its project runs check. Neither root run adds a finding. `no-unknown-variant` and `no-unknown-component` reported nothing.
+
+Method (`drawHoldout` in [scripts/corpus-lib.ts](../../scripts/corpus-lib.ts)):
+
+1. Each rule gets min(30, its population) slots.
+2. One mulberry32 generator seeded with 20261016, the one `random()` in scripts/corpus-lib.ts is, makes every draw. Rules are taken in id order, and each rule's findings in the order of their ids (`repository:file:line:column:rule`). Each draw takes index floor(next() × remaining) out of that list, the corpus `sample` command's step.
+3. The 109 picks are shuffled with the same generator (Fisher-Yates) and numbered H001 to H109, so the labeller does not see them grouped by rule.
+4. A rule's weight, population / sampled, is how many findings each of its sampled findings stands for.
+
+Sampled findings by repository: dub 81 (dub-web 78, dub-ui 3), openstatus 13 (dashboard 11, web 2), shadcn-ui 9, chatbot 5, saas-starter 1.
+
+## Files
+
+- `sample-unlabelled.json`: the labeller's only input. For each finding, where it is (repository, pinned commit, permalink, file, line and column), its rule, its message exactly as onsystem printed it, the matched source text, and 15 lines of code from the pinned commit. No suggested fix, no severity, no label. `run`, `occurrence` and `fingerprint` are the corpus keys.
+- `sample-key.json`: severity, whether onsystem offers a fix and its suggestion, and the rule's population and weight, for each finding. Not for the labeller.
+- `allocation.json`: the allocation above, the onsystem commit and version, and `populationSha256`, the sha256 of every distinct finding's id, sorted, one per line.
+- [BRIEF.md](BRIEF.md): what the labeller reads, must not open, and writes.
+- To come from the labeller: `sample-labelled.json` and `LABELLER.md`. `pnpm corpus --installed` then prints each rule's false-positive rate with its 95% Wilson interval.
+
+## Commands that rebuild it
+
+`$W` is a working folder outside any project. The draw needs `scripts/corpus.mjs` from the change that added this folder, and `src/` as it was at `7ca0f69`. Node v24.18.0, pnpm 11.6.0.
+
+```sh
+cd onsystem
+pnpm install --frozen-lockfile
+pnpm build
+# Fetches the five repositories at their pinned commits into $W/corpus/.installed and installs
+# each from its own lockfile (pnpm --frozen-lockfile --ignore-scripts --ignore-pnpmfile).
+node scripts/corpus.mjs fetch --installed --dir "$W/corpus"
+# Checks every run, requires the counts to match corpus/snapshot-installed.json, then draws.
+node scripts/corpus.mjs draw --installed --seed 20261016 --per-rule 30 --out "$W/redraw" --dir "$W/corpus"
+```
+
+A second draw into another folder on the same day wrote byte-identical `sample-unlabelled.json`, `sample-key.json` and `allocation.json` (its `drawn` date and `onsystem.commit` change with the day and the checkout).

corpus/installed-holdout-20261016/allocation.json

@@ -0,0 +1,121 @@
+{
+  "$comment": "How the holdout was drawn. populationSha256 is the sha256 of every distinct finding id (repository:file:line:column:rule), sorted, one per line with a final newline.",
+  "corpus": "installed",
+  "seed": 20261016,
+  "perRule": 30,
+  "n": 109,
+  "populationTotal": 12945,
+  "generator": "mulberry32 (scripts/corpus-lib.ts random)",
+  "drawn": "2026-10-16",
+  "onsystem": {
+    "commit": "7ca0f69b077b53fcfccf15fed534f303b06f6f1f",
+    "version": "0.7.0"
+  },
+  "populationSha256": "451e90f8bd6c252e3db4e28f9de2d204737f58e72d5b736a6b07fec6ef0ea9d3",
+  "runs": [
+    {
+      "run": "saas-starter-installed",
+      "repo": "saas-starter",
+      "sha": "6e33e58b1e553a41fe22e6b941a7229a002de361",
+      "distinct": 111
+    },
+    {
+      "run": "chatbot-installed",
+      "repo": "chatbot",
+      "sha": "c2f8235e1f3ea903ad8b7f61447c4f74164b5c58",
+      "distinct": 73
+    },
+    {
+      "run": "shadcn-website-installed",
+      "repo": "shadcn-ui",
+      "sha": "0132174664c07d41262fb51012d0cc782e458e6c",
+      "distinct": 124
+    },
+    {
+      "run": "openstatus-dashboard-installed",
+      "repo": "openstatus",
+      "sha": "e36446dc7d6980c16959f07d683f6261bc9c37d0",
+      "distinct": 37
+    },
+    {
+      "run": "openstatus-web-installed",
+      "repo": "openstatus",
+      "sha": "e36446dc7d6980c16959f07d683f6261bc9c37d0",
+      "distinct": 45
+    },
+    {
+      "run": "openstatus-ui-installed",
+      "repo": "openstatus",
+      "sha": "e36446dc7d6980c16959f07d683f6261bc9c37d0",
+      "distinct": 19
+    },
+    {
+      "run": "openstatus-root-installed",
+      "repo": "openstatus",
+      "sha": "e36446dc7d6980c16959f07d683f6261bc9c37d0",
+      "distinct": 0
+    },
+    {
+      "run": "dub-web-installed",
+      "repo": "dub",
+      "sha": "70c96f90701451358bc9438f8b7fdf4b3b46a6f4",
+      "distinct": 12211
+    },
+    {
+      "run": "dub-ui-installed",
+      "repo": "dub",
+      "sha": "70c96f90701451358bc9438f8b7fdf4b3b46a6f4",
+      "distinct": 325
+    },
+    {
+      "run": "dub-web-configured-installed",
+      "repo": "dub",
+      "sha": "70c96f90701451358bc9438f8b7fdf4b3b46a6f4",
+      "distinct": 0
+    },
+    {
+      "run": "dub-root-installed",
+      "repo": "dub",
+      "sha": "70c96f90701451358bc9438f8b7fdf4b3b46a6f4",
+      "distinct": 0
+    }
+  ],
+  "allocation": [
+    {
+      "rule": "icon-button-accessible-name",
+      "population": 85,
+      "sampled": 30,
+      "weight": 2.8333333333333335
+    },
+    {
+      "rule": "no-hardcoded-color",
+      "population": 11752,
+      "sampled": 30,
+      "weight": 391.73333333333335
+    },
+    {
+      "rule": "no-hardcoded-radius",
+      "population": 8,
+      "sampled": 8,
+      "weight": 1
+    },
+    {
+      "rule": "no-hardcoded-spacing",
+      "population": 7,
+      "sampled": 7,
+      "weight": 1
+    },
+    {
+      "rule": "no-unknown-prop",
+      "population": 4,
+      "sampled": 4,
+      "weight": 1
+    },
+    {
+      "rule": "prefer-design-system-component",
+      "population": 1089,
+      "sampled": 30,
+      "weight": 36.3
+    }
+  ]
+}