Empirical evidence about the evaluator proposed by the Judgment Pack Specification's RFC 0006 (Draft), in two tracks that answer two different questions and are judged by different standards:
- Agreement — do independent implementations produce the same result? Clean-room evaluators derived from the specification text alone. Their agreement tests whether the prose actually pins the semantics; their divergences locate the places it does not. Deterministic, offline, CI-verified.
- Efficacy — does representing a policy this way actually help? Preregistered experiments on third-party benchmarks. API-dependent, costly, non-deterministic, and run manually — never in CI.
The two are kept visibly separate on purpose. Conformance evidence is something a specification project is the right author of; efficacy evidence is something it is the wrong author of, and the only remedy is method: preregistration, public harness and data, and reported negative results.
Nothing in this repository claims any JPS conformance. JPS 0.1.0-draft defines no evaluator
conformance class and forbids evaluator-conformance claims outright (Core §3.4). Every
implementation here is experimental, may change or be removed without compatibility promise, and
evaluates nothing normatively: a disposition is data, not an authorization, a decision, or an
executed action.
| Path | What it is |
|---|---|
CLEAN-ROOM-PROTOCOL.md |
The reusable recipe every implementation must follow to count as evidence |
reference/ |
Pinned snapshots of the specification texts the rooms are built from |
harness/ |
The post-hoc referee: runs identical inputs through implementations and diffs dispositions |
python/ |
Agreement track. Implementation #2 (the first lives in judgment-pack-runtime): stdlib-only Python, written clean-room by a coding agent from the reference texts, with its interpretation log in python/DECISIONS.md |
studies/ |
Efficacy, expressiveness, and interoperability track. Preregistered experiments on third-party benchmarks and independently developed systems, from policy expressiveness (001–003) through trustworthy-input lineage (005–008) to receipt-protocol interoperability (013, 014). See the studies index for the full matrix and the open-source projects each one builds on. |
docs/adr/ |
Decision records for this repository — why a given study is being run, and what comes next |
Future implementations (TypeScript, Rust, …) get sibling directories — but the clean room itself never happens inside this repository: an implementer working here could read the existing implementations. See the protocol.
Navigation and status only — every finding, and every qualification that finding carries, lives in the study's own directory. Statuses below are the ones the studies index records; it remains the canonical matrix, with the external source each study builds on.
| Study | Track | Status | Evidence |
|---|---|---|---|
| 001 | Expressiveness / efficacy | Prepared expressiveness result; comparison arms not yet run | Results, study |
| 002 | Expressiveness | Completed expressiveness result | Results, study |
| 003 | Expressiveness | Completed escape census | Results, study |
| 004 | Composition | Completed composition-closure study; no efficacy claim | RESULTS.md, run-log.md |
| 005 | Trustworthy input | Completed semantic-source-discovery study | README.md, RESULTS.md, ANALYSIS.md |
| 006 | Trustworthy input | Deterministic phase completed; model-authoring phase terminated before inference because of infrastructure, so model usability is not estimable | README.md, RESULTS.md |
| 007 | Trustworthy input | Completed evidence-lineage model replication | README.md, RESULTS.md, ANALYSIS.md |
| 008 | Trustworthy input | Completed offline replay and probe with no model runs; read with the study's stated qualifications | README.md, RESULTS.md, ANALYSIS.md |
| 009 | Blinded authorship | Completed corrected retained attempt as a constructed existence witness — not discovery evidence, not general pipeline validation | README.md, RESULTS.json, ANALYSIS.md |
| 010 | Blinded authorship | Run (frozen) | ANALYSIS.md, RESULTS.json |
| 011 | Blinded authorship | Run (frozen) | ANALYSIS.md, RESULTS.json |
| 012 | Blinded authorship | Frozen + run — R1 unsupported; retracts a published claim | ANALYSIS.md, CORRECTION.md |
| 013 | Interoperability | Frozen + run — R1 holds (both strata) | ANALYSIS.md, results/ |
| 014 | Interoperability | Frozen + run — R1 holds (both strata) | ANALYSIS.md, results/ |
| 015 | Interoperability | Draft — five review rounds, not frozen | PREREG-REVIEW.md |
| 016 | Interoperability | Frozen + run — R1 holds (both strata) | ANALYSIS.md, results/ |
| 017 | Currency governance | Frozen + run — R1 holds (both strata) | ANALYSIS.md, results/ |
| 018 | Currency governance | Frozen + run — R1 holds; reviewer holdout diverged on three preregistered cells | ANALYSIS.md, results/ |
This repository claims no JPS conformance for anything in it, and the table above adds no aggregate headline: each study answers a different preregistered question and must be read with its own qualifications.
- 13/13 semantic agreement between the Go reference runtime's experimental evaluator
(v0.2.0) and
python/on RFC 0006's nine appendix instances plus three probes — identical kind, outcome, reason set, and handoff state (harness/README.md). - Two specification gaps found and recorded as RFC 0006 unresolved questions: number
representability, and the disposition's concrete JSON serialization (the two implementations
agreed on all semantics while serializing
handoffincompatibly — seepython/DECISIONS.mdentry 3).
Later studies have retained results; each answers a different preregistered question and must be read with its study-level qualifications. On the expressiveness arms below, no comparison arm has been run in either study. Two expressiveness results, on deliberately opposite policy types and by different model families:
| Study 001 — CBA, arithmetic-dense | Study 002 — airline, qualitative | Study 003 — census, 12 decisions | |
|---|---|---|---|
| Prepared facts | 124 | 5 | 58 |
| Prepared determinations | 13 | 1 | 40 |
| Decisions with ≥1 determination | 1/1 | 1/1 | 12/12 |
The escape is universal in the census frame because real request inputs are collections — and one device dominates: quantification over collections forced 25 of 40 determinations (arithmetic: 6). Encoders were hypothesis-blind; classification was two-way independent with zero fact-class disagreements. See Study 003, Study 002 and RFC 0007.
Apache-2.0. This repository is maintained alongside, but is not part of, the normative specification; the specification repository owns the standard and its conformance corpus.