bench: publish a reproducible fixture benchmark

[DerMayer1]

docs/benchmark-methodology.md described a fixture benchmark but deliberately withheld numbers until the corpus, policy snapshot and per-fixture results were in the repository. This adds all three.

pnpm install
pnpm --filter @fencier/core build
node benchmark/run.mjs

Corpus

82 fixtures covering all seven policy rules:

• 58 carrying a violation, across blocked_path, outside_allowed_paths, sensitive_path, missing_tests, max_files_changed, max_lines_changed, secret_pattern
• 25 seeded secrets, at least two per pattern across all nine detectors. Synthetic values, structurally valid for their pattern, none real
• 14 near misses that resemble violations and must not be flagged: exactly 8 files against a limit of 8, 499 lines against a limit of 500, a variable named tokenizer, a comment mentioning an API key, a protected path changed together with its test
• 13 clean patches, including files under ignored_paths that must be dropped before evaluation

The near misses and clean patches are the point. A detection rate without them is unfalsifiable, because a checker that failed every input would report 100%.

Results

Measure	Value
Expected signals detected	74/74
Seeded secrets detected	25/25
False positives	1 of 24 clean fixtures
Unexpected findings on violation fixtures	0
Fixtures matching expectations exactly	81/82

The one mismatch is real, and stays

process.env.API_KEY = undefined;   ->  flagged as generic_api_key

The pattern matches API_KEY followed by = and eight or more non-quote characters; undefined; qualifies. Clearing a key is reported as leaking one. Left in the corpus and documented rather than deleted, so the headline number stays honest.

Also recorded: the default policy blocks .env.*, which matches a committed .env.example. The engine behaves as configured. Whether the shipped policy should carve that out is a separate question.

What the numbers do not establish

docs/benchmark.md carries a limitations section. 100% detection measures coverage of the rules the engine implements, on inputs written to exercise them. It is not evidence that the policy model covers every way an agent can damage a repository, and it is not an independent audit, since the corpus and the detectors share an author.

Notes

• Policy is pinned in benchmark/policy.snapshot.yaml, so editing the project's own fencier.yaml does not move published results
• Runs are deterministic: no network, no model calls, no clock-dependent behaviour in the evaluated path. Verified by diffing two runs
• pnpm benchmark added
• Existing suite unchanged at 54 tests, typecheck clean