From 1b96b09c51ece227f847bbb73683f3fd7d58b131 Mon Sep 17 00:00:00 2001 From: Hermes Date: Fri, 23 Oct 2026 19:33:33 -0500 Subject: [PATCH 1/2] feat(evidence): ship the trust harness action in the kit One workflow file runs the eight authority cases on the reader's own account, verifies the sealed records offline, and writes a trustharness.report/v1 report. The action reads the kit at a pinned commit, so the command on the evidence page and the command in CI are the same command. The report carries only what the run measured: cost rows are null with a reason, and comparison figures stay on the evidence page. --- .../authority-at-dispatch/action/README.md | 68 +++++ .../authority-at-dispatch/action/action.yml | 156 ++++++++++++ .../authority-at-dispatch/action/report.mjs | 235 ++++++++++++++++++ .../action/report.test.mjs | 132 ++++++++++ .../authority-at-dispatch/action/schema.json | 127 ++++++++++ 5 files changed, 718 insertions(+) create mode 100644 public/evidence/aaif/authority-at-dispatch/action/README.md create mode 100644 public/evidence/aaif/authority-at-dispatch/action/action.yml create mode 100644 public/evidence/aaif/authority-at-dispatch/action/report.mjs create mode 100644 public/evidence/aaif/authority-at-dispatch/action/report.test.mjs create mode 100644 public/evidence/aaif/authority-at-dispatch/action/schema.json diff --git a/public/evidence/aaif/authority-at-dispatch/action/README.md b/public/evidence/aaif/authority-at-dispatch/action/README.md new file mode 100644 index 0000000..11c1339 --- /dev/null +++ b/public/evidence/aaif/authority-at-dispatch/action/README.md @@ -0,0 +1,68 @@ +# Proofable trust harness — GitHub Action + +Runs the eight authority-at-dispatch cases on your own Proofable account from your CI, verifies +every sealed record offline, and writes a `trustharness.report/v1` report your reviewer can check. + +This action does not reimplement the measurement. It reads the kit at a pinned commit of +`proofable/docs` and runs it, so the command in the [reproduce guide](../reproduce/README.md) and +the command in your CI are the same command. + +## Use it + +```yaml +# .github/workflows/proofable.yml +name: Proofable trust harness +on: [workflow_dispatch, pull_request] + +jobs: + authority: + runs-on: ubuntu-latest + steps: + - uses: proofable/docs/public/evidence/aaif/authority-at-dispatch/action@ + with: + access-key: ${{ secrets.PROOFABLE_ACCESS_KEY }} + operator: ${{ github.repository_owner }} + kit-ref: +``` + +Add `PROOFABLE_ACCESS_KEY` as a repository secret. Create the key under +[Profile → Account → Access keys](https://proofable.me/profile?tab=account). The cases run hosted +jobs, so they spend the account's credits (one full run is eight jobs, roughly six to eight +minutes). + +Pin `kit-ref` to a commit sha for a reproducible run. The resolved sha is recorded in the report +either way, so an unpinned run is still attributable. + +## Inputs + +| Input | Required | Default | Purpose | +| --- | --- | --- | --- | +| `access-key` | yes | — | `npk_…` access key from the account that pays for the run | +| `operator` | yes | — | Who ran it; recorded as the custodian in every record | +| `agent-id` | no | `aaif-authority-review` | Review agent id; give each reviewer on one account its own | +| `cases` | no | `all` | Space-separated case names, or `all` | +| `kit-ref` | no | `main` | docs ref the kit is read from | +| `fail-on-failure` | no | `true` | Fail the job when the offline check does not pass | +| `report-path` | no | `proofable-report.json` | Where to write the report | + +## Outputs + +| Output | Meaning | +| --- | --- | +| `report` | Path to the `trustharness.report/v1` JSON | +| `passed` / `total` | Cases passed, cases attempted | + +The sealed run directory and the report are uploaded as the `proofable-trust-harness` artifact, and +the report summary is written to the job summary. + +## What the report says and does not say + +The report carries only what the run measured: per-case outcome, whether each record verified +offline, and the deployed revision observed. Cost and performance rows are `null` with a reason, +because this kit does not measure them. Comparison figures are a link to +[docs.proofable.me/evidence](https://docs.proofable.me/evidence), which owns every number. + +`arm` is always `self-run` here: you drove the run on your own account. An independent run by a +third party is the next tier, not this one. + +Report shape: [`schema.json`](schema.json). diff --git a/public/evidence/aaif/authority-at-dispatch/action/action.yml b/public/evidence/aaif/authority-at-dispatch/action/action.yml new file mode 100644 index 0000000..f62894a --- /dev/null +++ b/public/evidence/aaif/authority-at-dispatch/action/action.yml @@ -0,0 +1,156 @@ +name: Proofable trust harness +description: Run the eight authority-at-dispatch cases on your own Proofable account and publish a scored report whose records verify offline. +author: Proofable +branding: + icon: shield + color: black + +inputs: + access-key: + description: A Proofable access key (npk_...) from the account that pays for the run. Pass it from a repository secret; never commit it. + required: true + operator: + description: Who is running this (organization or person). Recorded as the custodian in every sealed record. + required: true + agent-id: + description: Review agent id registered on that account. Give each reviewer sharing an account its own id. + required: false + default: aaif-authority-review + cases: + description: Space-separated case names, or "all". List them with `node aad.mjs cases`. + required: false + default: all + kit-ref: + description: docs commit or ref the kit is read from. Pin a commit sha for a reproducible run; the resolved sha is recorded in the report either way. + required: false + default: main + fail-on-failure: + description: Fail the job when any case does not pass. + required: false + default: "true" + report-path: + description: Where to write the report JSON. + required: false + default: proofable-report.json + +outputs: + report: + description: Path to the trustharness.report/v1 JSON. + value: ${{ steps.report.outputs.path }} + passed: + description: Cases that passed. + value: ${{ steps.report.outputs.passed }} + total: + description: Cases attempted. + value: ${{ steps.report.outputs.total }} + +runs: + using: composite + steps: + - name: Read the kit at a pinned ref + shell: bash + env: + KIT_REF: ${{ inputs.kit-ref }} + run: | + set -euo pipefail + kit="$RUNNER_TEMP/proofable-kit" + rm -rf "$kit" + git init -q "$kit" + cd "$kit" + git remote add origin https://github.com/proofable/docs.git + git fetch -q --depth 1 origin "$KIT_REF" + git checkout -q FETCH_HEAD + echo "PROOFABLE_KIT_COMMIT=$(git rev-parse HEAD)" >> "$GITHUB_ENV" + echo "PROOFABLE_KIT_DIR=$kit/public/evidence/aaif/authority-at-dispatch/reproduce" >> "$GITHUB_ENV" + + - name: Install the kit + shell: bash + run: | + set -euo pipefail + cd "$PROOFABLE_KIT_DIR" + npm ci --no-audit --no-fund + + - name: Register the review agent + shell: bash + env: + PROOFABLE_ACCESS_KEY: ${{ inputs.access-key }} + AAD_OPERATOR: ${{ inputs.operator }} + AAD_AGENT_ID: ${{ inputs.agent-id }} + run: | + set -euo pipefail + cd "$PROOFABLE_KIT_DIR" + node aad.mjs setup + + - name: Run the cases + shell: bash + env: + PROOFABLE_ACCESS_KEY: ${{ inputs.access-key }} + AAD_OPERATOR: ${{ inputs.operator }} + AAD_AGENT_ID: ${{ inputs.agent-id }} + AAD_CASES: ${{ inputs.cases }} + run: | + set -euo pipefail + cd "$PROOFABLE_KIT_DIR" + # The newest sealed run is the one this job reports on. + if [ "$AAD_CASES" = "all" ]; then + node aad.mjs run + else + # shellcheck disable=SC2086 + node aad.mjs run $AAD_CASES + fi + run_dir="$(ls -1dt runs/*/ | head -n 1)" + echo "PROOFABLE_RUN_DIR=$PROOFABLE_KIT_DIR/${run_dir%/}" >> "$GITHUB_ENV" + + - name: Verify the records offline + shell: bash + run: | + set -euo pipefail + cd "$PROOFABLE_KIT_DIR" + set +e + node aad.mjs verify "$PROOFABLE_RUN_DIR" > "$RUNNER_TEMP/proofable-verify.log" 2>&1 + verify_status=$? + set -e + cat "$RUNNER_TEMP/proofable-verify.log" + echo "PROOFABLE_VERIFY_STATUS=$verify_status" >> "$GITHUB_ENV" + + - name: Build the report + id: report + shell: bash + env: + AAD_OPERATOR: ${{ inputs.operator }} + REPORT_PATH: ${{ inputs.report-path }} + SOURCE_URL: ${{ github.server_url }}/${{ github.repository }} + run: | + set -euo pipefail + node "$PROOFABLE_KIT_DIR/../action/report.mjs" "$PROOFABLE_RUN_DIR" \ + --verify-log "$RUNNER_TEMP/proofable-verify.log" \ + --source "$SOURCE_URL" \ + --out "$REPORT_PATH" \ + --summary "$RUNNER_TEMP/proofable-report.md" + node -e ' + const r = require(process.env.REPORT_PATH.replace(/^/, "./")); + const fs = require("fs"); + fs.appendFileSync(process.env.GITHUB_OUTPUT, `path=${process.env.REPORT_PATH}\n`); + fs.appendFileSync(process.env.GITHUB_OUTPUT, `passed=${r.authority.passed}\n`); + fs.appendFileSync(process.env.GITHUB_OUTPUT, `total=${r.authority.total}\n`); + ' + cat "$RUNNER_TEMP/proofable-report.md" >> "$GITHUB_STEP_SUMMARY" + + - name: Keep the sealed run and the report + uses: actions/upload-artifact@v4 + with: + name: proofable-trust-harness + path: | + ${{ inputs.report-path }} + ${{ env.PROOFABLE_RUN_DIR }} + if-no-files-found: error + + - name: Fail when a case did not pass + if: ${{ inputs.fail-on-failure == 'true' }} + shell: bash + run: | + set -euo pipefail + if [ "${PROOFABLE_VERIFY_STATUS:-1}" != "0" ]; then + echo "The offline check did not pass; see the report and the verify log." + exit 1 + fi diff --git a/public/evidence/aaif/authority-at-dispatch/action/report.mjs b/public/evidence/aaif/authority-at-dispatch/action/report.mjs new file mode 100644 index 0000000..2f64d52 --- /dev/null +++ b/public/evidence/aaif/authority-at-dispatch/action/report.mjs @@ -0,0 +1,235 @@ +#!/usr/bin/env node +/** + * Build a `trustharness.report/v1` report from one sealed authority-at-dispatch run. + * + * This reads only what the kit wrote. It measures nothing itself and copies no + * comparison figure: cost and performance rows are null with a reason until + * something measured them, and the evidence page stays the owner of every + * comparison number. + * + * node report.mjs [--verify-log verify.log] [--source ] + * [--out proofable-report.json] [--summary report.md] + * + * Exit codes: 0 report written, 1 the run directory is unusable. + */ +import { createHash } from 'node:crypto'; +import { existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from 'node:fs'; +import { basename, dirname, join, resolve } from 'node:path'; + +const NOT_MEASURED = [ + 'revocation latency (how long a revocation takes to reach the check)', + 'target-side committed effect (effects are recorded as the service observed them)', + 'model cost per correct outcome (no paired arm ran in this report)', + 'false-refusal rate (no benign twin ran in this report)', +]; + +const COST_ROWS = [ + { + metric: 'authority decision overhead', + value: null, + unit: 'ms/decision', + reason: 'Not measured in this run. The kit records decisions and effects, not per-decision latency.', + }, + { + metric: 'cost per correct outcome (receipt check vs redo)', + value: null, + unit: 'USD', + reason: 'No paired arm ran here. The money comparison is a separate suite with its own run record.', + }, + { + metric: 'offline verification latency', + value: null, + unit: 'ms', + reason: 'The verify step reports pass/fail, not a timed median.', + }, +]; + +function parseArgs(argv) { + const out = { runDir: '', verifyLog: '', source: '', out: '', summary: '' }; + for (let i = 0; i < argv.length; i += 1) { + const a = argv[i]; + const next = () => { + const v = argv[i + 1]; + i += 1; + if (v === undefined) throw new Error(`${a} needs a value`); + return v; + }; + if (a === '--verify-log') out.verifyLog = next(); + else if (a === '--source') out.source = next(); + else if (a === '--out') out.out = next(); + else if (a === '--summary') out.summary = next(); + else if (!a.startsWith('--') && !out.runDir) out.runDir = a; + else throw new Error(`unknown argument ${a}`); + } + return out; +} + +/** Parse `aad.mjs verify` stdout: ` / envelopes`. */ +function parseVerifyLog(path) { + if (!path || !existsSync(path)) return { checksums: null, perCase: new Map() }; + const perCase = new Map(); + let failures = 0; + for (const line of readFileSync(path, 'utf8').split(/\r?\n/)) { + const m = line.match(/^(\S+)\s+(PASS|FAIL|MISSING)\s+(\d+)\/(\d+) envelopes\s*$/); + if (m) { + perCase.set(m[1], { outcome: m[2], envelopes: Number(m[3]), total: Number(m[4]) }); + if (m[2] !== 'PASS') failures += 1; + continue; + } + if (/^\d+ failure\(s\)$/.test(line.trim())) failures = Math.max(failures, 1); + } + return { checksums: failures === 0 && perCase.size > 0 ? 'pass' : 'fail', perCase }; +} + +function sha256(buf) { + return createHash('sha256').update(buf).digest('hex'); +} + +function build(args) { + const runDir = resolve(args.runDir); + const manifestPath = join(runDir, 'MANIFEST.json'); + if (!existsSync(manifestPath)) { + throw new Error(`No MANIFEST.json in ${runDir}. Pass the runs/ directory the kit sealed.`); + } + const manifest = JSON.parse(readFileSync(manifestPath, 'utf8')); + if (manifest.schema !== 'proofable-aad-run-manifest/v1') { + throw new Error(`Unexpected manifest schema ${JSON.stringify(manifest.schema)}; expected proofable-aad-run-manifest/v1.`); + } + + const expected = Array.isArray(manifest.expected_cases) ? manifest.expected_cases : []; + const manifestRows = new Map( + (Array.isArray(manifest.cases) ? manifest.cases : []).map((row) => [String(row.case), row]), + ); + const errorFiles = new Set( + (Array.isArray(manifest.errors) ? manifest.errors : []).map((name) => String(name).replace(/\.error\.json$/, '')), + ); + const verify = parseVerifyLog(args.verifyLog); + + const cases = expected.map((name) => { + const row = manifestRows.get(name); + const verified = verify.perCase.get(name); + const evidenceFile = join(runDir, `${name}.json`); + const record = existsSync(evidenceFile) ? JSON.parse(readFileSync(evidenceFile, 'utf8')) : null; + let outcome = 'not-run'; + if (errorFiles.has(name)) outcome = 'error'; + else if (row) outcome = row.passed ? 'pass' : 'fail'; + if (verified && verified.outcome !== 'PASS' && outcome === 'pass') { + // The seal disagreed with the verdict: the stricter reading wins. + outcome = 'fail'; + } + return { + case: name, + outcome, + job_id: row?.jobId ? String(row.jobId) : null, + evidence: record?.receipt?.qHash ? String(record.receipt.qHash) : (existsSync(evidenceFile) ? basename(evidenceFile) : null), + note: verified ? `${verified.envelopes}/${verified.total} envelopes verified offline` : null, + }; + }); + + const passed = cases.filter((c) => c.outcome === 'pass').length; + const failed = cases.filter((c) => c.outcome === 'fail' || c.outcome === 'error').length; + const total = cases.length; + + const envelopesVerified = [...verify.perCase.values()].reduce((sum, row) => sum + row.envelopes, 0); + const envelopesTotal = [...verify.perCase.values()].reduce((sum, row) => sum + row.total, 0); + + const assets = new Map(); + for (const name of readdirSync(runDir).sort()) { + if (!/\.(json|md|SHA256SUMS)$/.test(name) && !name.endsWith('SHA256SUMS')) continue; + assets.set(name, sha256(readFileSync(join(runDir, name)))); + } + + const seen = new Set(); + const notMeasured = []; + for (const row of [ + ...(Array.isArray(manifest.not_measured) ? manifest.not_measured : []), + ...NOT_MEASURED, + ]) { + const key = String(row).toLowerCase().split('(')[0].trim(); + if (seen.has(key)) continue; + seen.add(key); + notMeasured.push(String(row)); + } + + return { + schema: 'trustharness.report/v1', + arm: 'self-run', + generated_at: new Date().toISOString(), + instrument: { + kit: manifest.schema, + kit_url: 'https://docs.proofable.me/evidence', + kit_commit: String(process.env.PROOFABLE_KIT_COMMIT || '').trim() || null, + run_id: manifest.run_id ? String(manifest.run_id) : null, + }, + subject: { + operator: String(process.env.AAD_OPERATOR || manifest.custody || '').trim() || 'unknown', + agent_id: manifest.agentId ? String(manifest.agentId) : null, + agent_wallet: manifest.agentWallet ? String(manifest.agentWallet) : null, + controller: String(manifest.delegationRoot || '').split(':').pop() || null, + product_revision: manifest.product_revision ? String(manifest.product_revision) : null, + source: args.source || null, + }, + authority: { + cases, + passed, + total, + unauthorized_actions_through: failed, + }, + integrity: { + checksums: verify.checksums || 'fail', + envelopes_verified: Number.isFinite(envelopesVerified) ? envelopesVerified : null, + envelopes_total: Number.isFinite(envelopesTotal) ? envelopesTotal : null, + }, + cost: COST_ROWS, + gaps: [], + not_measured: notMeasured, + limits: [ + 'One run on the operator\'s own account: a sample, not a rate.', + 'Measures the decision the service returned for each case and whether a record verified offline; not the target system\'s committed effect.', + 'Self-run: the operator drove the run. An independent rerun is the next tier, not this one.', + 'Scope and comparison figures live on the evidence page; this report does not restate them.', + ], + evidence_url: 'https://docs.proofable.me/evidence', + assets: Object.fromEntries(assets), + }; +} + +function summaryMarkdown(report) { + const lines = [ + `## Proofable trust harness — ${report.authority.passed} of ${report.authority.total} authority cases passed`, + '', + `Arm: **self-run** · operator: **${report.subject.operator}** · revision: \`${report.subject.product_revision || 'unknown'}\` · run \`${report.instrument.run_id || 'unknown'}\``, + '', + '| Case | Outcome | Offline check |', + '| --- | --- | --- |', + ...report.authority.cases.map( + (c) => `| \`${c.case}\` | ${c.outcome} | ${c.note || '—'} |`, + ), + '', + `Sealed records: **${report.integrity.checksums}** · envelopes verified offline: **${report.integrity.envelopes_verified ?? '—'}/${report.integrity.envelopes_total ?? '—'}**`, + '', + 'Not measured here:', + ...report.not_measured.map((row) => `- ${row}`), + '', + `Comparisons and their scope: ${report.evidence_url}`, + ]; + return `${lines.join('\n')}\n`; +} + +try { + const args = parseArgs(process.argv.slice(2)); + if (!args.runDir) throw new Error('usage: node report.mjs [--verify-log ] [--source ] [--out ] [--summary ]'); + const report = build(args); + const outPath = args.out || 'proofable-report.json'; + mkdirSync(dirname(resolve(outPath)), { recursive: true }); + writeFileSync(outPath, `${JSON.stringify(report, null, 2)}\n`, 'utf8'); + if (args.summary) { + mkdirSync(dirname(resolve(args.summary)), { recursive: true }); + writeFileSync(args.summary, summaryMarkdown(report), 'utf8'); + } + console.log(`report ${outPath}: ${report.authority.passed}/${report.authority.total} passed, seals ${report.integrity.checksums}`); + process.exitCode = report.authority.passed === report.authority.total ? 0 : 2; +} catch (error) { + console.error(`report failed: ${error?.message || error}`); + process.exitCode = 1; +} diff --git a/public/evidence/aaif/authority-at-dispatch/action/report.test.mjs b/public/evidence/aaif/authority-at-dispatch/action/report.test.mjs new file mode 100644 index 0000000..dddac0c --- /dev/null +++ b/public/evidence/aaif/authority-at-dispatch/action/report.test.mjs @@ -0,0 +1,132 @@ +/** + * Smoke test for the report builder: a fixture run directory shaped exactly like + * the kit's output, so the parser can be exercised without paying for a run. + * + * The fixture is a test input, never a result: it is written to a temp directory, + * labelled `Fixture Org`, and its `product_revision` is `fixture-rev`. + * + * node --test action/report.test.mjs + */ +import assert from 'node:assert/strict'; +import { execFileSync } from 'node:child_process'; +import { createHash } from 'node:crypto'; +import { mkdtempSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import test from 'node:test'; + +const here = dirname(fileURLToPath(import.meta.url)); +const reportScript = join(here, 'report.mjs'); +const CASES = [ + 'hosted_allow', + 'binding_veto', + 'approval_reachable', + 'approval_unavailable', + 'revoke_before_dispatch', + 'expiry_before_dispatch', + 'stale_authority', + 'post_dispatch_revoke', +]; + +function buildFixture({ failing = [] } = {}) { + const root = mkdtempSync(join(tmpdir(), 'proofable-report-')); + const dir = join(root, 'runs', 'fixture-run'); + mkdirSync(dir, { recursive: true }); + const identity = { + run_id: 'fixture-run', + operator: 'Fixture Org', + custody: 'OPERATOR: Fixture Org', + product_revision: 'fixture-rev', + mcp_url: 'https://mcp.proofable.me/mcp', + delegationRoot: 'did:pkh:eip155:8453:0x1111111111111111111111111111111111111111', + agentId: 'aaif-authority-review', + agentWallet: '0x2222222222222222222222222222222222222222', + }; + const rows = []; + for (const [i, name] of CASES.entries()) { + const passed = !failing.includes(name); + const record = { + schema: 'proofable-aaif-authority-suite/v1', + case: name, + ...identity, + passed, + jobId: `job_${i}`, + receipt: { qHash: `0x${String(i).repeat(64)}` }, + }; + writeFileSync(join(dir, `${name}.json`), `${JSON.stringify(record, null, 2)}\n`); + rows.push({ case: name, passed, jobId: `job_${i}`, grant: null }); + } + const manifest = { + schema: 'proofable-aad-run-manifest/v1', + kit: 'fixture', + ...identity, + expected_cases: CASES, + cases: rows, + errors: [], + passed_count: rows.filter((r) => r.passed).length, + not_measured: ['revocation latency', 'target-side committed effect'], + }; + writeFileSync(join(dir, 'MANIFEST.json'), `${JSON.stringify(manifest, null, 2)}\n`); + const files = [...CASES.map((c) => `${c}.json`), 'MANIFEST.json'].sort(); + writeFileSync( + join(dir, 'SHA256SUMS'), + `${files.map((f) => `${createHash('sha256').update(readFileSync(join(dir, f))).digest('hex')} ${f}`).join('\n')}\n`, + ); + const verifyLog = join(root, 'verify.log'); + writeFileSync( + verifyLog, + `${CASES.map((c) => `${c.padEnd(24)} ${failing.includes(c) ? 'FAIL' : 'PASS'} 2/2 envelopes`).join('\n')}\n\nall checks pass\n`, + ); + return { root, dir, verifyLog }; +} + +function runReport(args) { + try { + const stdout = execFileSync(process.execPath, [reportScript, ...args], { encoding: 'utf8' }); + return { status: 0, stdout }; + } catch (error) { + return { status: error.status, stdout: `${error.stdout || ''}${error.stderr || ''}` }; + } +} + +test('a clean run reports eight passes with seals verified', () => { + const { root, dir, verifyLog } = buildFixture(); + const out = join(root, 'report.json'); + const result = runReport([dir, '--verify-log', verifyLog, '--out', out, '--summary', join(root, 'report.md')]); + assert.equal(result.status, 0); + const report = JSON.parse(readFileSync(out, 'utf8')); + assert.equal(report.schema, 'trustharness.report/v1'); + assert.equal(report.arm, 'self-run'); + assert.equal(report.authority.passed, 8); + assert.equal(report.authority.total, 8); + assert.equal(report.authority.unauthorized_actions_through, 0); + assert.equal(report.integrity.checksums, 'pass'); + assert.equal(report.integrity.envelopes_verified, 16); + // Nothing measured cost, so every cost row is null with a reason. + for (const row of report.cost) { + assert.equal(row.value, null); + assert.ok(row.reason, `${row.metric} needs a reason`); + } + assert.ok(report.not_measured.length >= 2); + assert.ok(!/48 of 48|48\/48/.test(JSON.stringify(report.cost)), 'the report must not carry a comparison figure'); +}); + +test('a failing case is counted as one that got through and fails the build', () => { + const { root, dir, verifyLog } = buildFixture({ failing: ['binding_veto'] }); + const out = join(root, 'report.json'); + const result = runReport([dir, '--verify-log', verifyLog, '--out', out]); + assert.equal(result.status, 2); + const report = JSON.parse(readFileSync(out, 'utf8')); + assert.equal(report.authority.passed, 7); + assert.equal(report.authority.unauthorized_actions_through, 1); + assert.equal(report.authority.cases.find((c) => c.case === 'binding_veto').outcome, 'fail'); + assert.equal(report.integrity.checksums, 'fail'); +}); + +test('a directory without a sealed manifest is refused, not guessed', () => { + const { root } = buildFixture(); + const result = runReport([join(root, 'runs'), '--out', join(root, 'nope.json')]); + assert.equal(result.status, 1); + assert.match(result.stdout + '', /MANIFEST\.json/); +}); diff --git a/public/evidence/aaif/authority-at-dispatch/action/schema.json b/public/evidence/aaif/authority-at-dispatch/action/schema.json new file mode 100644 index 0000000..4969ddd --- /dev/null +++ b/public/evidence/aaif/authority-at-dispatch/action/schema.json @@ -0,0 +1,127 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://docs.proofable.me/schemas/trustharness.report.v1.json", + "title": "Proofable trust harness report", + "description": "One report shape for both arms of the trust harness: `review` (a public-surface reading, no run) and `self-run` (the operator's own production run of the authority-at-dispatch cases, verified offline). The arm field is never optional. Every value is either what this report measured or null with a reason; a comparison figure is a link to its owner, never a copy.", + "type": "object", + "additionalProperties": false, + "required": ["schema", "arm", "generated_at", "subject", "authority", "not_measured", "limits"], + "properties": { + "schema": { "const": "trustharness.report/v1" }, + "arm": { + "enum": ["review", "self-run"], + "description": "review = read of public material only, nothing executed against the subject. self-run = the subject's own account ran the kit and the records verify offline." + }, + "generated_at": { "type": "string", "format": "date-time" }, + "instrument": { + "type": ["object", "null"], + "additionalProperties": false, + "properties": { + "kit": { "type": "string", "description": "Record schema the kit sealed, e.g. proofable-aad-run-manifest/v1." }, + "kit_url": { "type": "string" }, + "kit_commit": { "type": ["string", "null"], "description": "Resolved docs commit the kit was read from. null when unknown." }, + "run_id": { "type": ["string", "null"] } + } + }, + "subject": { + "type": "object", + "additionalProperties": false, + "required": ["operator"], + "properties": { + "operator": { "type": "string", "description": "Who ran it (self-run) or whose material was read (review)." }, + "agent_id": { "type": ["string", "null"] }, + "agent_wallet": { "type": ["string", "null"] }, + "controller": { "type": ["string", "null"] }, + "product_revision": { "type": ["string", "null"], "description": "Deployed revision the run observed." }, + "source": { "type": ["string", "null"], "description": "review only: repository or endpoint read." } + } + }, + "authority": { + "type": "object", + "additionalProperties": false, + "required": ["cases", "passed", "total"], + "properties": { + "cases": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["case", "outcome"], + "properties": { + "case": { "type": "string" }, + "outcome": { "enum": ["pass", "fail", "not-run", "error"] }, + "job_id": { "type": ["string", "null"] }, + "evidence": { "type": ["string", "null"], "description": "Where to look: proof qHash, file:line for a review, or null." }, + "note": { "type": ["string", "null"] } + } + } + }, + "passed": { "type": "integer", "minimum": 0 }, + "total": { "type": "integer", "minimum": 0 }, + "unauthorized_actions_through": { + "type": ["integer", "null"], + "description": "Sum of failed cases. A case with no recorded outcome counts as one that got through only when the arm is self-run; otherwise null." + } + } + }, + "integrity": { + "type": ["object", "null"], + "additionalProperties": false, + "description": "self-run only: the offline check of the sealed records. null in the review arm.", + "properties": { + "checksums": { "enum": ["pass", "fail"] }, + "envelopes_verified": { "type": ["integer", "null"] }, + "envelopes_total": { "type": ["integer", "null"] } + } + }, + "assets": { + "type": "object", + "description": "sha256 of every sealed file in the run directory, so a reader can prove the report describes bytes they can check.", + "additionalProperties": { "type": "string" } + }, + "cost": { + "type": "array", + "description": "Cost and performance are never inferred. A row with no measurement carries value null and a reason.", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["metric", "value", "reason"], + "properties": { + "metric": { "type": "string" }, + "value": { "type": ["number", "null"] }, + "unit": { "type": ["string", "null"] }, + "reason": { "type": ["string", "null"], "description": "Why the value is null. Required whenever value is null." } + } + } + }, + "gaps": { + "type": "array", + "description": "review only: findings, each with what it means and the smallest fix.", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["finding", "means"], + "properties": { + "finding": { "type": "string" }, + "means": { "type": "string" }, + "fix": { "type": ["string", "null"] }, + "evidence": { "type": ["string", "null"] } + } + } + }, + "not_measured": { + "type": "array", + "items": { "type": "string" }, + "description": "Named limits. Never empty: every arm has something it does not measure." + }, + "limits": { + "type": "array", + "items": { "type": "string" }, + "description": "Scope statements that must travel with the report wherever it is quoted." + }, + "evidence_url": { + "type": "string", + "description": "The one page that owns every comparison figure. This report never restates them." + } + } +} From d374a47a145a1a2cc381b4b11b4d7b2a889a9421 Mon Sep 17 00:00:00 2001 From: Hermes Date: Fri, 23 Oct 2026 19:33:52 -0500 Subject: [PATCH 2/2] docs(evidence): point the reproduce section at the CI action The same eight cases, run per pull request, with the report shape and the pin-it rule stated on the page that owns the numbers. --- evidence.mdx | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/evidence.mdx b/evidence.mdx index afa53e4..a4b0cb9 100644 --- a/evidence.mdx +++ b/evidence.mdx @@ -139,6 +139,20 @@ node aad.mjs verify runs/ The same records are part of the [AAIF Identity & Trust working-group comparison](https://github.com/aaif/wg-identity-and-trust/issues/13) at the Linux Foundation. +### Run it on every pull request + +Add one workflow file and the same eight cases run in your CI, on your account, with every record verified offline and kept as a build artifact: + +```yaml +- uses: proofable/docs/public/evidence/aaif/authority-at-dispatch/action@ + with: + access-key: ${{ secrets.PROOFABLE_ACCESS_KEY }} + operator: ${{ github.repository_owner }} + kit-ref: +``` + +Pin both refs to the same commit so a rerun measures the same kit. The action writes a `trustharness.report/v1` report listing the outcome for each case, whether each record verified offline, and the revision observed. It fails the job when the offline check does not pass, and it reports what it measured and nothing else: cost and performance rows stay `null` with a reason, and every comparison figure stays on this page. [Action and report schema](https://github.com/proofable/docs/tree/main/public/evidence/aaif/authority-at-dispatch/action). + ## How the AgentGovBench run works [AgentGovBench](https://github.com/agentic-control-plane/agentgovbench) is an open library of 48 agent-governance scenarios in eight areas: audit completeness, cross-tenant isolation, delegation provenance, fail-mode discipline, identity propagation, per-user policy, rate limits and scope inheritance. Each scenario is tagged with the NIST AI RMF controls it tests. Its maintainers revised the library in October 2026 so that nothing passes on a technicality.