Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 48 additions & 6 deletions EVALUATION-STANDARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ LGD-Eval is an **independent third-party evaluation group**. Evaluations we publ
Out of scope: certification, accreditation, conformity assessment with legal effect,
regulatory opinion, or endorsement of any product or vendor.

## 2. Evidence Model (evidence-grader v1.2)
## 2. Evidence Model (evidence-grader v1.3)

Each evaluated claim is graded on two axes:

Expand All @@ -38,12 +38,24 @@ Each evaluated claim is graded on two axes:

### 2.2 Authorization verdict (per decision)

The verification verdict has three states. This is the vocabulary used in `outcome.verification`.

| Verdict | Meaning | Operator meaning |
|---|---|---|
| `verified` | Evidence measured and consistent with authorization | ALLOW |
| `unresolved` | Evidence absent or unmeasurable — not yet decidable | DEFER (do not default to allow) |
| `rejected` | Evidence measured and found inconsistent | DENY |
| `quarantined` | Normative semantics of the field still unsettled | EXCLUDE from formal result until spec settles |
| `verified` | Authorization established and current | ALLOW |
| `unresolved` | Authorization cannot be established — not yet decidable | DEFER (do not default to allow) |
| `rejected` | Authorization affirmatively not valid — established by evidence | DENY |

**Operator action is a separate axis.** What a relying party does is policy-dependent, so it is recorded in its own field (`outcome.policy_action`) and never merged into the verdict. `reject` is not available as an action name: the word is reserved for the verdict, so the two levels stay distinguishable at a glance.

| Operator action | Meaning | Typical trigger |
|---|---|---|
| `allow` | Proceed under normal policy | `verified` |
| `defer` | Do not default to allow; hold for review or escalation | `unresolved` |
| `deny` | Do not accept | `rejected` |
| `exclude` | Keep out of the formal result until the normative semantics settle | Field semantics still unsettled — this is **not** a verification verdict |

`quarantined` was a fourth value of the verdict up to v1.2 and was removed in v1.3: it described a handling decision over an unsettled field, not something that had been verified. It is now `policy_action: exclude`. Evidence-level `failure_class` still reports `quarantined` for malformed input; that is the evidence axis (§2.1) and is unchanged.

### 2.3 Payload digest convention

Expand All @@ -52,6 +64,33 @@ Digests are truncated to the first 16 hex chars as vector ids in worked examples
quoted in full in evidence records. Re-running the same canonical input must reproduce
the same digest (determinism check, see §4).

The digest covers the substantive objects of a conclusion (`domains`, `authorization`,
`outcome`) and excludes `timestamp`, so reproduction is date-independent. `verdict` and
`overall_failure_class` are pure functions of `domains` and are covered transitively.

> **Correction (v1.3, 2026-10-23).** The convention above has been published in this
> section since v0.1, but the v1.2 implementation computed the digest with default JSON
> separators (`", "` / `": "`) and insertion order rather than `sort_keys=True` with
> compact separators. A third party following this section could therefore not reproduce
> any published digest. The implementation now matches the text; all four v1.2 digests
> were recomputed, with the reasons listed in `worked-examples/cases/README.md`.

### 2.4 Case outcome vs coverage verdict

A conclusion reports two summary fields that answer different questions, and reading the
wrong one reports a passing case as a failure:

| Field | Answers | Scope |
|---|---|---|
| `outcome.verification` | What is the case-level conclusion for the dimension under test? | The case |
| `verdict` | How much of the six-domain evidence grid is covered at grade V/S? | The grid |

A case set that exercises one or two of the six domains is `Insufficient evidence` under
`verdict` by construction, even when its authorization outcome is `verified`. Readers and
programs should take the case conclusion from `outcome.verification`. Each conclusion also
carries a `scopes` object stating this in machine-readable form, so the distinction
travels with the data instead of living only in documentation.

## 3. Evaluation Report Format

Every published evaluation follows the REVIEW format (see `reviews/` for samples):
Expand Down Expand Up @@ -83,7 +122,10 @@ displayed with the disclaimer text defined there.

## 6. Versioning & Change

- Protocol version: v1.2 (evidence-grader). This document: v0.1.
- Protocol version: v1.3 (evidence-grader). This document: v0.2.
- v1.3 (2026-10-23): case outcome separated from the coverage verdict; operator action split
out of the verdict vocabulary; digest computation corrected to match §2.3. All worked-example
digests recomputed.
- Changes are tracked in git; worked examples updated atomically with the grader.
- Zenodo record for protocol releases: https://zenodo.org/records/23428476 (CC-BY-4.0).

Expand Down
12 changes: 6 additions & 6 deletions WORKED_EXAMPLES_L0-L13.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,13 +165,13 @@

## Appendix A — Reproduction from live runs / 附录 A:实测复现对照

The following three cases were executed with the reference grader (`evaluation-protocol/tools/evidence-grader.py` v1.2, SHA-256 `98C9C1F6FABA8E1E688D0CE01FF35C3715EF51EC25072685B23A8782A1C9C91B`); machine-readable outputs with payload digests are in `worked-examples/cases/` and `tools/demo/`. They illustrate L0–L2 judgment in practice and are offered as optional evaluation aid. / 以下用例已用参考分级器 v1.2 实跑(输出含 payload digest,位于 `worked-examples/cases/` 与 `tools/demo/`),用于演示 L0–L2 的判定实践,仅作可选评估辅助。
The following four cases were executed with the reference grader (`evaluation-protocol/tools/evidence-grader.py` v1.3, SHA-256 `06C89E1BA30E4C331058AB90511B6DEA62D8641A8AB15D284BCAFD4005925098` (computed over the server-side LF bytes, 34795 B)); machine-readable outputs with payload digests are in `worked-examples/cases/` and `tools/demo/`. They illustrate L0–L2 judgment in practice and are offered as optional evaluation aid. / 以下用例已用参考分级器 v1.3 实跑(输出含 payload digest,位于 `worked-examples/cases/` 与 `tools/demo/`),用于演示 L0–L2 的判定实践,仅作可选评估辅助。

| Case / 用例 | Scenario / 场景 | Judgment / 判定 | Outcome / 结果 |
|---|---|---|---|
| A — Valid registered credential / 有效已注册凭证 | Signed credential from an authorized issuer, status source available | L0/L1/L2 contract, static and functional checks pass; signature grade V; authorization **verified** | **positive** (reason: `none`) |
| B1 — Unverifiable issuer scope / 签发范围不可判定 | Signature verifies, but authoritative issuer-scope information is unavailable | Signature grade V; authorization **unresolved** | **indeterminate** (reason: `authorization unresolved`; missing evidence ≠ invalid) |
| B2 — Confirmed out-of-scope issuer / 已证实越权签发者 | Signature verifies, authoritative scope registry confirms the issuer lacks the required scope | Signature grade V; authorization **rejected** | **rejected** (reason: `authorization rejected`) |
| C — Status source unavailable / 状态源不可用 | Credential cannot be confirmed or denied because the status source is down | Authorization **unresolved**; `unavailable ≠ revoked` | **indeterminate** (reason: `status unavailable ≠ revoked`) |
| A — Valid registered credential / 有效已注册凭证 | Signed credential from an authorized issuer, status source available | L0/L1/L2 contract, static and functional checks pass; signature grade V; authorization **verified** | **verified** (reason_code `none`; policy `allow`) |
| B1 — Unverifiable issuer scope / 签发范围不可判定 | Signature verifies, but authoritative issuer-scope information is unavailable | Signature grade V; authorization **unresolved** | **unresolved** (reason_code `authorization_unresolved`; policy `defer`; missing evidence ≠ invalid) |
| B2 — Confirmed out-of-scope issuer / 已证实越权签发者 | Signature verifies, authoritative scope registry confirms the issuer lacks the required scope | Signature grade V; authorization **rejected** | **rejected** (reason_code `authorization_rejected`; policy `deny`) |
| C — Status source unavailable / 状态源不可用 | Credential cannot be confirmed or denied because the status source is down | Authorization **unresolved**; `unavailable ≠ revoked` | **unresolved** (reason_code `status_unavailable`; policy `defer`; unavailable ≠ revoked) |

*Case B was split into B1/B2 per the themes#42 review (missing evidence of authorization ≠ demonstrated invalid authorization). Worked examples drafted 2026-10-16, updated 2026-10-17 by LGD-Eval (evaluation account) — bilingual, English first, Chinese second. / 实例由 LGD-Eval 评测号起草并更新(双语,英前中后);原 Case B 按 themes#42 评审意见拆分为 B1/B2。*
*Case B was split into B1/B2 per the themes#42 review (missing evidence of authorization ≠ demonstrated invalid authorization). Worked examples drafted 2026-10-16, updated 2026-10-23 by LGD-Eval (v1.3: outcome vocabulary aligned, policy actions separated) (evaluation account) — bilingual, English first, Chinese second. / 实例由 LGD-Eval 评测号起草并更新(双语,英前中后);原 Case B 按 themes#42 评审意见拆分为 B1/B2。*
Loading
Loading