From 5ced9aa994112e1e24e6547733959c083142ed6d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?LGD-Eval=20=E2=80=94=20Independent=20AI=20Evaluation?= Date: Fri, 23 Oct 2026 17:41:57 +0800 Subject: [PATCH 1/2] =?UTF-8?q?v1.3:=20separate=20case=20outcome=20from=20?= =?UTF-8?q?coverage=20verdict;=20split=20policy=20actions=20out=20of=20the?= =?UTF-8?q?=20verdict=20vocabulary;=20fix=20payload=20digest=20to=20=C2=A7?= =?UTF-8?q?2.3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three defects found while reviewing the case set for FG-TIDA/use-cases#27 against the 2026-10-23 review comment. None of them is a change of position; all three are the same position applied consistently. 1) `verdict` is six-domain evidence coverage, not a case outcome. Case A — a valid, current, in-registration credential — reports "Insufficient evidence（证据不足）" because only 2 of the 6 domains are exercised. A program that reads `verdict` as the conclusion gets a false negative on the positive case. v1.3 adds `outcome.verification` (verified / unresolved / rejected) as the field to read, keeps `verdict` with its original meaning, and states both scopes inside every conclusion in a machine-readable `scopes` object. The field was not wrong; it was being asked the wrong question. 2) The verdict vocabulary mixed policy actions with verification states. `quarantined` was a fourth verdict, but it describes what an operator does with an unsettled field, not what was verified. It is now `policy_action: exclude`, and `policy_action` reuses the four operator meanings already published in EVALUATION-STANDARD.md §2.2 (allow / defer / deny / exclude) instead of introducing a fifth word set. `reason_code` becomes a required machine-readable field (none / authorization_unresolved / authorization_rejected / status_unavailable). 3) The payload digest did not match the convention in EVALUATION-STANDARD.md §2.3. The v1.2 implementation computed the digest with default JSON separators and insertion order; §2.3 specifies sort_keys=True with compact separators. Any third party following §2.3 could not reproduce a published digest. This was verified against the v1.2 files before changing anything: recomputed under §2.3 they hash to ab56339d…, be1677e8…, ff4ff95a…, 6c37e632…, none of which matches the published values. The implementation now matches the text. All four digests change (schema lgd-eval/conclusion/v1.3): Case A fa8b1420cca29307 Case B1 a353ad64f911030a Case B2 d709c4ec85ef5d6e Case C 982c390107058485 Selftest: 35 invariants (was 15). New invariants cover B1/B2 outcome separation, reason_code consistency with the verdict, rejection of `quarantined` as a verdict, digest equality with an independently computed §2.3 value, and key-order independence. Reproduction: from a clean copy of this tree with the conclusions deleted, re-running the grader on the four inputs reproduces all four published digests (4/4 MATCH), and the selftest passes in that copy. Also in scope: demo/evidence-grading-demo.html was stale (three cases, "12 invariants", and the pre-split digests) and is regenerated from the four conclusions; the "external verification" wording is replaced with what actually happened (a reproduction run from a fresh download, made by the maintainers), since claiming third-party verification for our own run would be the exact overstatement this project exists to avoid. No files removed. The `three_cases_outcome_flow.svg` asset keeps its file name so that links from existing threads do not break; the caption and the page state that it shows four cases. --- EVALUATION-STANDARD.md | 54 ++- WORKED_EXAMPLES_L0-L13.md | 12 +- demo/evidence-grading-demo.html | 284 ++++++++++++-- tools/demo/demo-conclusion.json | 327 ++++++++-------- tools/demo/demo-conclusion.md | 74 ++-- tools/evidence-grader.py | 358 ++++++++++++++---- worked-examples/cases/README.md | 91 +++-- ...alid_registered_credential_conclusion.json | 247 ++++++------ ..._valid_registered_credential_conclusion.md | 41 ++ ...seA_valid_registered_credential_input.json | 78 ++-- ..._unverifiable_issuer_scope_conclusion.json | 227 ++++++----- ...B1_unverifiable_issuer_scope_conclusion.md | 41 ++ ...aseB1_unverifiable_issuer_scope_input.json | 4 +- ...firmed_out_of_scope_issuer_conclusion.json | 227 ++++++----- ...onfirmed_out_of_scope_issuer_conclusion.md | 41 ++ ...2_confirmed_out_of_scope_issuer_input.json | 4 +- ..._unavailable_status_source_conclusion.json | 209 +++++----- ...eC_unavailable_status_source_conclusion.md | 41 ++ ...caseC_unavailable_status_source_input.json | 42 +- 19 files changed, 1592 insertions(+), 810 deletions(-) create mode 100644 worked-examples/cases/caseA_valid_registered_credential_conclusion.md create mode 100644 worked-examples/cases/caseB1_unverifiable_issuer_scope_conclusion.md create mode 100644 worked-examples/cases/caseB2_confirmed_out_of_scope_issuer_conclusion.md create mode 100644 worked-examples/cases/caseC_unavailable_status_source_conclusion.md diff --git a/EVALUATION-STANDARD.md b/EVALUATION-STANDARD.md index 5df3a80..ec3fb14 100644 --- a/EVALUATION-STANDARD.md +++ b/EVALUATION-STANDARD.md @@ -22,7 +22,7 @@ LGD-Eval is an **independent third-party evaluation group**. Evaluations we publ Out of scope: certification, accreditation, conformity assessment with legal effect, regulatory opinion, or endorsement of any product or vendor. -## 2. Evidence Model (evidence-grader v1.2) +## 2. Evidence Model (evidence-grader v1.3) Each evaluated claim is graded on two axes: @@ -38,12 +38,24 @@ Each evaluated claim is graded on two axes: ### 2.2 Authorization verdict (per decision) +The verification verdict has three states. This is the vocabulary used in `outcome.verification`. + | Verdict | Meaning | Operator meaning | |---|---|---| -| `verified` | Evidence measured and consistent with authorization | ALLOW | -| `unresolved` | Evidence absent or unmeasurable — not yet decidable | DEFER (do not default to allow) | -| `rejected` | Evidence measured and found inconsistent | DENY | -| `quarantined` | Normative semantics of the field still unsettled | EXCLUDE from formal result until spec settles | +| `verified` | Authorization established and current | ALLOW | +| `unresolved` | Authorization cannot be established — not yet decidable | DEFER (do not default to allow) | +| `rejected` | Authorization affirmatively not valid — established by evidence | DENY | + +**Operator action is a separate axis.** What a relying party does is policy-dependent, so it is recorded in its own field (`outcome.policy_action`) and never merged into the verdict. `reject` is not available as an action name: the word is reserved for the verdict, so the two levels stay distinguishable at a glance. + +| Operator action | Meaning | Typical trigger | +|---|---|---| +| `allow` | Proceed under normal policy | `verified` | +| `defer` | Do not default to allow; hold for review or escalation | `unresolved` | +| `deny` | Do not accept | `rejected` | +| `exclude` | Keep out of the formal result until the normative semantics settle | Field semantics still unsettled — this is **not** a verification verdict | + +`quarantined` was a fourth value of the verdict up to v1.2 and was removed in v1.3: it described a handling decision over an unsettled field, not something that had been verified. It is now `policy_action: exclude`. Evidence-level `failure_class` still reports `quarantined` for malformed input; that is the evidence axis (§2.1) and is unchanged. ### 2.3 Payload digest convention @@ -52,6 +64,33 @@ Digests are truncated to the first 16 hex chars as vector ids in worked examples quoted in full in evidence records. Re-running the same canonical input must reproduce the same digest (determinism check, see §4). +The digest covers the substantive objects of a conclusion (`domains`, `authorization`, +`outcome`) and excludes `timestamp`, so reproduction is date-independent. `verdict` and +`overall_failure_class` are pure functions of `domains` and are covered transitively. + +> **Correction (v1.3, 2026-10-23).** The convention above has been published in this +> section since v0.1, but the v1.2 implementation computed the digest with default JSON +> separators (`", "` / `": "`) and insertion order rather than `sort_keys=True` with +> compact separators. A third party following this section could therefore not reproduce +> any published digest. The implementation now matches the text; all four v1.2 digests +> were recomputed, with the reasons listed in `worked-examples/cases/README.md`. + +### 2.4 Case outcome vs coverage verdict + +A conclusion reports two summary fields that answer different questions, and reading the +wrong one reports a passing case as a failure: + +| Field | Answers | Scope | +|---|---|---| +| `outcome.verification` | What is the case-level conclusion for the dimension under test? | The case | +| `verdict` | How much of the six-domain evidence grid is covered at grade V/S? | The grid | + +A case set that exercises one or two of the six domains is `Insufficient evidence` under +`verdict` by construction, even when its authorization outcome is `verified`. Readers and +programs should take the case conclusion from `outcome.verification`. Each conclusion also +carries a `scopes` object stating this in machine-readable form, so the distinction +travels with the data instead of living only in documentation. + ## 3. Evaluation Report Format Every published evaluation follows the REVIEW format (see `reviews/` for samples): @@ -83,7 +122,10 @@ displayed with the disclaimer text defined there. ## 6. Versioning & Change -- Protocol version: v1.2 (evidence-grader). This document: v0.1. +- Protocol version: v1.3 (evidence-grader). This document: v0.2. +- v1.3 (2026-10-23): case outcome separated from the coverage verdict; operator action split + out of the verdict vocabulary; digest computation corrected to match §2.3. All worked-example + digests recomputed. - Changes are tracked in git; worked examples updated atomically with the grader. - Zenodo record for protocol releases: https://zenodo.org/records/23428476 (CC-BY-4.0). diff --git a/WORKED_EXAMPLES_L0-L13.md b/WORKED_EXAMPLES_L0-L13.md index e47e0b0..25843a3 100644 --- a/WORKED_EXAMPLES_L0-L13.md +++ b/WORKED_EXAMPLES_L0-L13.md @@ -165,13 +165,13 @@ ## Appendix A — Reproduction from live runs / 附录 A：实测复现对照 -The following three cases were executed with the reference grader (`evaluation-protocol/tools/evidence-grader.py` v1.2, SHA-256 `98C9C1F6FABA8E1E688D0CE01FF35C3715EF51EC25072685B23A8782A1C9C91B`); machine-readable outputs with payload digests are in `worked-examples/cases/` and `tools/demo/`. They illustrate L0–L2 judgment in practice and are offered as optional evaluation aid. / 以下用例已用参考分级器 v1.2 实跑（输出含 payload digest，位于 `worked-examples/cases/` 与 `tools/demo/`），用于演示 L0–L2 的判定实践，仅作可选评估辅助。 +The following four cases were executed with the reference grader (`evaluation-protocol/tools/evidence-grader.py` v1.3, SHA-256 `74CEB2B8266478014BB5953DC63534CF68B12BC08721162A889E1E0175A7B74B`); machine-readable outputs with payload digests are in `worked-examples/cases/` and `tools/demo/`. They illustrate L0–L2 judgment in practice and are offered as optional evaluation aid. / 以下用例已用参考分级器 v1.3 实跑（输出含 payload digest，位于 `worked-examples/cases/` 与 `tools/demo/`），用于演示 L0–L2 的判定实践，仅作可选评估辅助。 | Case / 用例 | Scenario / 场景 | Judgment / 判定 | Outcome / 结果 | |---|---|---|---| -| A — Valid registered credential / 有效已注册凭证 | Signed credential from an authorized issuer, status source available | L0/L1/L2 contract, static and functional checks pass; signature grade V; authorization **verified** | **positive** (reason: `none`) | -| B1 — Unverifiable issuer scope / 签发范围不可判定 | Signature verifies, but authoritative issuer-scope information is unavailable | Signature grade V; authorization **unresolved** | **indeterminate** (reason: `authorization unresolved`; missing evidence ≠ invalid) | -| B2 — Confirmed out-of-scope issuer / 已证实越权签发者 | Signature verifies, authoritative scope registry confirms the issuer lacks the required scope | Signature grade V; authorization **rejected** | **rejected** (reason: `authorization rejected`) | -| C — Status source unavailable / 状态源不可用 | Credential cannot be confirmed or denied because the status source is down | Authorization **unresolved**; `unavailable ≠ revoked` | **indeterminate** (reason: `status unavailable ≠ revoked`) | +| A — Valid registered credential / 有效已注册凭证 | Signed credential from an authorized issuer, status source available | L0/L1/L2 contract, static and functional checks pass; signature grade V; authorization **verified** | **verified** (reason_code `none`; policy `allow`) | +| B1 — Unverifiable issuer scope / 签发范围不可判定 | Signature verifies, but authoritative issuer-scope information is unavailable | Signature grade V; authorization **unresolved** | **unresolved** (reason_code `authorization_unresolved`; policy `defer`; missing evidence ≠ invalid) | +| B2 — Confirmed out-of-scope issuer / 已证实越权签发者 | Signature verifies, authoritative scope registry confirms the issuer lacks the required scope | Signature grade V; authorization **rejected** | **rejected** (reason_code `authorization_rejected`; policy `deny`) | +| C — Status source unavailable / 状态源不可用 | Credential cannot be confirmed or denied because the status source is down | Authorization **unresolved**; `unavailable ≠ revoked` | **unresolved** (reason_code `status_unavailable`; policy `defer`; unavailable ≠ revoked) | -*Case B was split into B1/B2 per the themes#42 review (missing evidence of authorization ≠ demonstrated invalid authorization). Worked examples drafted 2026-10-16, updated 2026-10-17 by LGD-Eval (evaluation account) — bilingual, English first, Chinese second. / 实例由 LGD-Eval 评测号起草并更新（双语，英前中后）；原 Case B 按 themes#42 评审意见拆分为 B1/B2。* +*Case B was split into B1/B2 per the themes#42 review (missing evidence of authorization ≠ demonstrated invalid authorization). Worked examples drafted 2026-10-16, updated 2026-10-23 by LGD-Eval (v1.3: outcome vocabulary aligned, policy actions separated) (evaluation account) — bilingual, English first, Chinese second. / 实例由 LGD-Eval 评测号起草并更新（双语，英前中后）；原 Case B 按 themes#42 评审意见拆分为 B1/B2。* diff --git a/demo/evidence-grading-demo.html b/demo/evidence-grading-demo.html index 1dc6c50..b2f2066 100644 --- a/demo/evidence-grading-demo.html +++ b/demo/evidence-grading-demo.html @@ -4,7 +4,7 @@ Evidence Grading for Verifier Decisions — Interactive Demo - +