diff --git a/EVALUATION-STANDARD.md b/EVALUATION-STANDARD.md index 5df3a80..ec3fb14 100644 --- a/EVALUATION-STANDARD.md +++ b/EVALUATION-STANDARD.md @@ -22,7 +22,7 @@ LGD-Eval is an **independent third-party evaluation group**. Evaluations we publ Out of scope: certification, accreditation, conformity assessment with legal effect, regulatory opinion, or endorsement of any product or vendor. -## 2. Evidence Model (evidence-grader v1.2) +## 2. Evidence Model (evidence-grader v1.3) Each evaluated claim is graded on two axes: @@ -38,12 +38,24 @@ Each evaluated claim is graded on two axes: ### 2.2 Authorization verdict (per decision) +The verification verdict has three states. This is the vocabulary used in `outcome.verification`. + | Verdict | Meaning | Operator meaning | |---|---|---| -| `verified` | Evidence measured and consistent with authorization | ALLOW | -| `unresolved` | Evidence absent or unmeasurable — not yet decidable | DEFER (do not default to allow) | -| `rejected` | Evidence measured and found inconsistent | DENY | -| `quarantined` | Normative semantics of the field still unsettled | EXCLUDE from formal result until spec settles | +| `verified` | Authorization established and current | ALLOW | +| `unresolved` | Authorization cannot be established — not yet decidable | DEFER (do not default to allow) | +| `rejected` | Authorization affirmatively not valid — established by evidence | DENY | + +**Operator action is a separate axis.** What a relying party does is policy-dependent, so it is recorded in its own field (`outcome.policy_action`) and never merged into the verdict. `reject` is not available as an action name: the word is reserved for the verdict, so the two levels stay distinguishable at a glance. + +| Operator action | Meaning | Typical trigger | +|---|---|---| +| `allow` | Proceed under normal policy | `verified` | +| `defer` | Do not default to allow; hold for review or escalation | `unresolved` | +| `deny` | Do not accept | `rejected` | +| `exclude` | Keep out of the formal result until the normative semantics settle | Field semantics still unsettled — this is **not** a verification verdict | + +`quarantined` was a fourth value of the verdict up to v1.2 and was removed in v1.3: it described a handling decision over an unsettled field, not something that had been verified. It is now `policy_action: exclude`. Evidence-level `failure_class` still reports `quarantined` for malformed input; that is the evidence axis (§2.1) and is unchanged. ### 2.3 Payload digest convention @@ -52,6 +64,33 @@ Digests are truncated to the first 16 hex chars as vector ids in worked examples quoted in full in evidence records. Re-running the same canonical input must reproduce the same digest (determinism check, see §4). +The digest covers the substantive objects of a conclusion (`domains`, `authorization`, +`outcome`) and excludes `timestamp`, so reproduction is date-independent. `verdict` and +`overall_failure_class` are pure functions of `domains` and are covered transitively. + +> **Correction (v1.3, 2026-10-23).** The convention above has been published in this +> section since v0.1, but the v1.2 implementation computed the digest with default JSON +> separators (`", "` / `": "`) and insertion order rather than `sort_keys=True` with +> compact separators. A third party following this section could therefore not reproduce +> any published digest. The implementation now matches the text; all four v1.2 digests +> were recomputed, with the reasons listed in `worked-examples/cases/README.md`. + +### 2.4 Case outcome vs coverage verdict + +A conclusion reports two summary fields that answer different questions, and reading the +wrong one reports a passing case as a failure: + +| Field | Answers | Scope | +|---|---|---| +| `outcome.verification` | What is the case-level conclusion for the dimension under test? | The case | +| `verdict` | How much of the six-domain evidence grid is covered at grade V/S? | The grid | + +A case set that exercises one or two of the six domains is `Insufficient evidence` under +`verdict` by construction, even when its authorization outcome is `verified`. Readers and +programs should take the case conclusion from `outcome.verification`. Each conclusion also +carries a `scopes` object stating this in machine-readable form, so the distinction +travels with the data instead of living only in documentation. + ## 3. Evaluation Report Format Every published evaluation follows the REVIEW format (see `reviews/` for samples): @@ -83,7 +122,10 @@ displayed with the disclaimer text defined there. ## 6. Versioning & Change -- Protocol version: v1.2 (evidence-grader). This document: v0.1. +- Protocol version: v1.3 (evidence-grader). This document: v0.2. +- v1.3 (2026-10-23): case outcome separated from the coverage verdict; operator action split + out of the verdict vocabulary; digest computation corrected to match §2.3. All worked-example + digests recomputed. - Changes are tracked in git; worked examples updated atomically with the grader. - Zenodo record for protocol releases: https://zenodo.org/records/23428476 (CC-BY-4.0). diff --git a/WORKED_EXAMPLES_L0-L13.md b/WORKED_EXAMPLES_L0-L13.md index e47e0b0..4299f38 100644 --- a/WORKED_EXAMPLES_L0-L13.md +++ b/WORKED_EXAMPLES_L0-L13.md @@ -165,13 +165,13 @@ ## Appendix A — Reproduction from live runs / 附录 A:实测复现对照 -The following three cases were executed with the reference grader (`evaluation-protocol/tools/evidence-grader.py` v1.2, SHA-256 `98C9C1F6FABA8E1E688D0CE01FF35C3715EF51EC25072685B23A8782A1C9C91B`); machine-readable outputs with payload digests are in `worked-examples/cases/` and `tools/demo/`. They illustrate L0–L2 judgment in practice and are offered as optional evaluation aid. / 以下用例已用参考分级器 v1.2 实跑(输出含 payload digest,位于 `worked-examples/cases/` 与 `tools/demo/`),用于演示 L0–L2 的判定实践,仅作可选评估辅助。 +The following four cases were executed with the reference grader (`evaluation-protocol/tools/evidence-grader.py` v1.3, SHA-256 `06C89E1BA30E4C331058AB90511B6DEA62D8641A8AB15D284BCAFD4005925098` (computed over the server-side LF bytes, 34795 B)); machine-readable outputs with payload digests are in `worked-examples/cases/` and `tools/demo/`. They illustrate L0–L2 judgment in practice and are offered as optional evaluation aid. / 以下用例已用参考分级器 v1.3 实跑(输出含 payload digest,位于 `worked-examples/cases/` 与 `tools/demo/`),用于演示 L0–L2 的判定实践,仅作可选评估辅助。 | Case / 用例 | Scenario / 场景 | Judgment / 判定 | Outcome / 结果 | |---|---|---|---| -| A — Valid registered credential / 有效已注册凭证 | Signed credential from an authorized issuer, status source available | L0/L1/L2 contract, static and functional checks pass; signature grade V; authorization **verified** | **positive** (reason: `none`) | -| B1 — Unverifiable issuer scope / 签发范围不可判定 | Signature verifies, but authoritative issuer-scope information is unavailable | Signature grade V; authorization **unresolved** | **indeterminate** (reason: `authorization unresolved`; missing evidence ≠ invalid) | -| B2 — Confirmed out-of-scope issuer / 已证实越权签发者 | Signature verifies, authoritative scope registry confirms the issuer lacks the required scope | Signature grade V; authorization **rejected** | **rejected** (reason: `authorization rejected`) | -| C — Status source unavailable / 状态源不可用 | Credential cannot be confirmed or denied because the status source is down | Authorization **unresolved**; `unavailable ≠ revoked` | **indeterminate** (reason: `status unavailable ≠ revoked`) | +| A — Valid registered credential / 有效已注册凭证 | Signed credential from an authorized issuer, status source available | L0/L1/L2 contract, static and functional checks pass; signature grade V; authorization **verified** | **verified** (reason_code `none`; policy `allow`) | +| B1 — Unverifiable issuer scope / 签发范围不可判定 | Signature verifies, but authoritative issuer-scope information is unavailable | Signature grade V; authorization **unresolved** | **unresolved** (reason_code `authorization_unresolved`; policy `defer`; missing evidence ≠ invalid) | +| B2 — Confirmed out-of-scope issuer / 已证实越权签发者 | Signature verifies, authoritative scope registry confirms the issuer lacks the required scope | Signature grade V; authorization **rejected** | **rejected** (reason_code `authorization_rejected`; policy `deny`) | +| C — Status source unavailable / 状态源不可用 | Credential cannot be confirmed or denied because the status source is down | Authorization **unresolved**; `unavailable ≠ revoked` | **unresolved** (reason_code `status_unavailable`; policy `defer`; unavailable ≠ revoked) | -*Case B was split into B1/B2 per the themes#42 review (missing evidence of authorization ≠ demonstrated invalid authorization). Worked examples drafted 2026-10-16, updated 2026-10-17 by LGD-Eval (evaluation account) — bilingual, English first, Chinese second. / 实例由 LGD-Eval 评测号起草并更新(双语,英前中后);原 Case B 按 themes#42 评审意见拆分为 B1/B2。* +*Case B was split into B1/B2 per the themes#42 review (missing evidence of authorization ≠ demonstrated invalid authorization). Worked examples drafted 2026-10-16, updated 2026-10-23 by LGD-Eval (v1.3: outcome vocabulary aligned, policy actions separated) (evaluation account) — bilingual, English first, Chinese second. / 实例由 LGD-Eval 评测号起草并更新(双语,英前中后);原 Case B 按 themes#42 评审意见拆分为 B1/B2。* diff --git a/demo/evidence-grading-demo.html b/demo/evidence-grading-demo.html index 1dc6c50..b2f2066 100644 --- a/demo/evidence-grading-demo.html +++ b/demo/evidence-grading-demo.html @@ -4,7 +4,7 @@