Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 0 additions & 9 deletions artifacts/rebaseline-2025/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,15 +84,6 @@ uses MIT for its own code and documentation, but that license does not relicense
third-party web pages. Keep web extracts hash-only unless the exact row has a
redistribution review that permits publishing the text.

To run the private KatFish KO calibration after downloading `essay.jsonl`, `abstract.jsonl`, and `poetry.jsonl` into `artifacts/rebaseline-2025/private/katfish/`:

```bash
npm run benchmark:katfish-ko -- --write --basename katfish-ko-latest
```

The command writes only aggregate metrics to `docs/benchmarks/`; it must not
commit KatFish raw rows unless a license review explicitly allows it.

To refresh their deterministic score/outcome fields from a private raw-text
intake file:

Expand Down
3 changes: 1 addition & 2 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -256,8 +256,7 @@ The checker enforces only three reachability rules:
2. `playground/**` cannot reach server-secret modules, API handlers, or Node
built-ins.
3. `src/**`, `api/**`, `bin/**`, and every declared published package bin
cannot reach research modules (`scripts/research/`, `tests/quality/`, and
`scripts/iterative-rewrite-baseline.mjs`).
cannot reach research modules (`scripts/research/` and `tests/quality/`).

Deterministic shared modules are classified rather than blanket-banned:
`src/edit-controls.js`, `src/errors.js`, `src/logger.js`,
Expand Down
6 changes: 1 addition & 5 deletions docs/COOKBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,11 +75,7 @@ The Action finds changed Markdown files with `dorny/paths-filter@v4`, runs patin

### README score badge

`npm run badge` writes a [Shields.io endpoint](https://shields.io/endpoint) JSON file from the same deterministic prose score. It reports the highest scored file (`maxScore`) as an editing-hotspot percentage, not an authorship verdict.

```bash
npm run badge -- README.md docs/FAQ.md > patina-badge.json
```
The Action's `patina-badge.json` is a [Shields.io endpoint](https://shields.io/endpoint) file built from the same deterministic prose score. It reports the highest scored file (`maxScore`) as an editing-hotspot percentage, not an authorship verdict.

```json
{ "schemaVersion": 1, "label": "patina", "message": "25% · human-ish", "color": "brightgreen" }
Expand Down
8 changes: 3 additions & 5 deletions docs/HARNESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@ INPUT ENGINE (deterministic, LLM-free) SURFACES
│ HARNESS │
│ (1) regression (det.) benchmark · benchmark:ranges · benchmark:report │
│ (2) calibration (det.) signal-impact · rebaseline:score/report · │
│ low-fpr · katfish-ko · lexicon:freshness │
│ low-fpr · lexicon:freshness │
│ (3) robustness/perf robustness · perf │
│ (4) LLM quality (opt-in) quality:live · quality:rewrite-ab · adversarial-mps │
│ (4) LLM quality (opt-in) quality:live · adversarial-mps │
│ (5) comparison (det.) detector-comparison │
└──────────────────────────────────────────────────────────────────────────┘
│ enforced by
Expand Down Expand Up @@ -72,7 +72,6 @@ analysis below ablates each one to report its marginal contribution.
| Rebaseline summary | `npm run benchmark:rebaseline` / `:report` | Validates the public manifest + computes the headline catch/FP claim | [tests/quality/README.md](../tests/quality/README.md) |
| Rebaseline score | `npm run benchmark:rebaseline:score` | Re-runs the analyzer over private rows → public scored manifest (predicted_hot + trigger_counts) | [tests/quality/README.md](../tests/quality/README.md) |
| Low-FPR metrics | `npm run benchmark:rebaseline:low-fpr` | TPR@1%/5%FPR operating points | `scripts/rebaseline-low-fpr-report.mjs` |
| KatFish calibration | `npm run benchmark:katfish-ko` | KO diagnostic catch/FP deltas (aggregate-only) | [tests/quality/README.md](../tests/quality/README.md) |
| Lexicon freshness | `npm run lexicon:freshness` | AI-lexicon provenance sidecar check | `scripts/lexicon-freshness.mjs` |
| FP fixture export | `node scripts/fp-fixture-export.mjs` | Turns FP reports into suspect-zone fixtures | `scripts/fp-fixture-export.mjs` |

Expand All @@ -82,7 +81,6 @@ analysis below ablates each one to report its marginal contribution.
|---|---|---|---|
| Live rewrite quality | `npm run quality:live` (`PATINA_LIVE=1` to call a model) | before/after AI score, MPS, fidelity on rewrites | [tests/quality/README.md](../tests/quality/README.md) |
| Adversarial MPS | `npm run quality:adversarial-mps` | Guards against MPS hiding unchanged AI style | [tests/quality/README.md](../tests/quality/README.md) |
| Rewrite A/B | `npm run quality:rewrite-ab` (`--live`) | Compares two rewrite configs (default `single` vs `iterative-baseline`) on the same fixtures: after-AI/MPS/fidelity/edit-churn + per-fixture winner. Answers whether the baseline comparison flow produces a better rewrite. | [tests/quality/README.md](../tests/quality/README.md) |

## Gates (deterministic, run in CI / pre-publish)

Expand All @@ -92,7 +90,7 @@ analysis below ablates each one to report its marginal contribution.
| Tests | `npm test` | unit + e2e (`node --test`) |
| Release metadata | `npm run release:check` | version sync across all version-bearing surfaces |
| Private-asset leak | `npm run check:no-private-assets` | no private/vendor text in the npm tarball or tree |
| Prose score gate | `patina-score` / `npm run badge` | hot-paragraph ratio CI gate (default 30) |
| Prose score gate | `patina-score` | hot-paragraph ratio CI gate (default 30) |

## Signal impact harness (`scripts/signal-impact.mjs`)

Expand Down
3 changes: 1 addition & 2 deletions docs/QA.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,10 @@ result `inconclusive` until an approved command exists.
| `performance-package-size` | `npm run benchmark:perf -- --costmetrics` | Explicit report-only tarball size, cold-process CLI timing, and warm analyzer measurements; no rebaseline or latency gate |
| `dogfood` | `npm run dogfood` | Configured public-document checks |
| `release-safety` | `npm run release:check` and `npm run check:no-private-assets` | Release metadata and private-asset boundary |
| `mock-quality` | `npm run qa:mdx` | Existing MDX QA script, when its scope applies |
| `live-model` | `npm run quality:live` | Opt-in model-backed quality; approval and budget required |

Other package scripts may fit the changed boundary (for example,
`benchmark:robustness`, `quality:adversarial-mps`, or `quality:rewrite-ab`).
`benchmark:robustness` or `quality:adversarial-mps`).
Record the exact command and arguments rather than assuming a similarly named
script exists. `benchmark:rebaseline*` and other research commands are not
routine acceptance profiles.
Expand Down
4 changes: 0 additions & 4 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@
"quality:adversarial-mps": "node scripts/adversarial-mps-report.mjs --check",
"quality:persona-ablation": "node scripts/persona-ablation.mjs",
"dogfood": "node tests/quality/dogfood.mjs",
"badge": "node scripts/badge-json.mjs",
"card": "node scripts/share-card.mjs",
"lint": "npm run lint:syntax && npm run lint:eslint && npm run typecheck && npm run spellcheck && npm run check:architecture",
"release:check": "node scripts/check-release-metadata.mjs && node scripts/check-retired-concepts.mjs",
Expand All @@ -49,16 +48,13 @@
"lint:eslint": "eslint .",
"typecheck": "tsc -p .",
"spellcheck": "cspell \"patterns/**/*.md\" \"lexicon/**/*.md\" --gitignore-root .",
"qa:mdx": "node scripts/qa/mdx-score.mjs",
"quality:live": "node tests/quality/live-quality.mjs",
"benchmark:scorer:live": "node tests/quality/live-scorer-benchmark.mjs",
"quality:model-rewrite": "node scripts/research/model-rewrite-benchmark.mjs",
"quality:rewrite-ab": "node scripts/rewrite-ab.mjs",
"benchmark:rebaseline:intake": "node scripts/rebaseline-intake.mjs",
"benchmark:rebaseline:score": "node scripts/rebaseline-score.mjs",
"benchmark:signal-impact": "node scripts/signal-impact.mjs",
"benchmark:rebaseline:web": "node scripts/rebaseline-web-collect.mjs",
"benchmark:katfish-ko": "node scripts/katfish-calibration.mjs",
"lexicon:freshness": "node scripts/lexicon-freshness.mjs --check",
"benchmark:rebaseline:generate-modern": "node scripts/rebaseline-generate-modern.mjs",
"benchmark:rebaseline:claim-manifest": "node scripts/rebaseline-build-claim-manifest.mjs",
Expand Down
79 changes: 0 additions & 79 deletions scripts/badge-json.mjs

This file was deleted.

1 change: 0 additions & 1 deletion scripts/check-architecture.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,6 @@ const SERVER_SECRET_MODULES = Object.freeze([
]);

const RESEARCH_MODULES = Object.freeze([
'scripts/iterative-rewrite-baseline.mjs',
'scripts/research/',
'tests/quality/',
]);
Expand Down
Loading
Loading