Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions evidence.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,20 @@ node aad.mjs verify runs/<run-id>

The same records are part of the [AAIF Identity & Trust working-group comparison](https://github.com/aaif/wg-identity-and-trust/issues/13) at the Linux Foundation.

### Run it on every pull request

Add one workflow file and the same eight cases run in your CI, on your account, with every record verified offline and kept as a build artifact:

```yaml
- uses: proofable/docs/public/evidence/aaif/authority-at-dispatch/action@<commit-sha>
with:
access-key: ${{ secrets.PROOFABLE_ACCESS_KEY }}
operator: ${{ github.repository_owner }}
kit-ref: <commit-sha>
```

Pin both refs to the same commit so a rerun measures the same kit. The action writes a `trustharness.report/v1` report listing the outcome for each case, whether each record verified offline, and the revision observed. It fails the job when the offline check does not pass, and it reports what it measured and nothing else: cost and performance rows stay `null` with a reason, and every comparison figure stays on this page. [Action and report schema](https://github.com/proofable/docs/tree/main/public/evidence/aaif/authority-at-dispatch/action).

## How the AgentGovBench run works

[AgentGovBench](https://github.com/agentic-control-plane/agentgovbench) is an open library of 48 agent-governance scenarios in eight areas: audit completeness, cross-tenant isolation, delegation provenance, fail-mode discipline, identity propagation, per-user policy, rate limits and scope inheritance. Each scenario is tagged with the NIST AI RMF controls it tests. Its maintainers revised the library in October 2026 so that nothing passes on a technicality.
Expand Down
68 changes: 68 additions & 0 deletions public/evidence/aaif/authority-at-dispatch/action/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Proofable trust harness — GitHub Action

Runs the eight authority-at-dispatch cases on your own Proofable account from your CI, verifies
every sealed record offline, and writes a `trustharness.report/v1` report your reviewer can check.

This action does not reimplement the measurement. It reads the kit at a pinned commit of
`proofable/docs` and runs it, so the command in the [reproduce guide](../reproduce/README.md) and
the command in your CI are the same command.

## Use it

```yaml
# .github/workflows/proofable.yml
name: Proofable trust harness
on: [workflow_dispatch, pull_request]

jobs:
authority:
runs-on: ubuntu-latest
steps:
- uses: proofable/docs/public/evidence/aaif/authority-at-dispatch/action@<commit-sha>
with:
access-key: ${{ secrets.PROOFABLE_ACCESS_KEY }}
operator: ${{ github.repository_owner }}
kit-ref: <commit-sha>
```

Add `PROOFABLE_ACCESS_KEY` as a repository secret. Create the key under
[Profile → Account → Access keys](https://proofable.me/profile?tab=account). The cases run hosted
jobs, so they spend the account's credits (one full run is eight jobs, roughly six to eight
minutes).

Pin `kit-ref` to a commit sha for a reproducible run. The resolved sha is recorded in the report
either way, so an unpinned run is still attributable.

## Inputs

| Input | Required | Default | Purpose |
| --- | --- | --- | --- |
| `access-key` | yes | — | `npk_…` access key from the account that pays for the run |
| `operator` | yes | — | Who ran it; recorded as the custodian in every record |
| `agent-id` | no | `aaif-authority-review` | Review agent id; give each reviewer on one account its own |
| `cases` | no | `all` | Space-separated case names, or `all` |
| `kit-ref` | no | `main` | docs ref the kit is read from |
| `fail-on-failure` | no | `true` | Fail the job when the offline check does not pass |
| `report-path` | no | `proofable-report.json` | Where to write the report |

## Outputs

| Output | Meaning |
| --- | --- |
| `report` | Path to the `trustharness.report/v1` JSON |
| `passed` / `total` | Cases passed, cases attempted |

The sealed run directory and the report are uploaded as the `proofable-trust-harness` artifact, and
the report summary is written to the job summary.

## What the report says and does not say

The report carries only what the run measured: per-case outcome, whether each record verified
offline, and the deployed revision observed. Cost and performance rows are `null` with a reason,
because this kit does not measure them. Comparison figures are a link to
[docs.proofable.me/evidence](https://docs.proofable.me/evidence), which owns every number.

`arm` is always `self-run` here: you drove the run on your own account. An independent run by a
third party is the next tier, not this one.

Report shape: [`schema.json`](schema.json).
156 changes: 156 additions & 0 deletions public/evidence/aaif/authority-at-dispatch/action/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
name: Proofable trust harness
description: Run the eight authority-at-dispatch cases on your own Proofable account and publish a scored report whose records verify offline.
author: Proofable
branding:
icon: shield
color: black

inputs:
access-key:
description: A Proofable access key (npk_...) from the account that pays for the run. Pass it from a repository secret; never commit it.
required: true
operator:
description: Who is running this (organization or person). Recorded as the custodian in every sealed record.
required: true
agent-id:
description: Review agent id registered on that account. Give each reviewer sharing an account its own id.
required: false
default: aaif-authority-review
cases:
description: Space-separated case names, or "all". List them with `node aad.mjs cases`.
required: false
default: all
kit-ref:
description: docs commit or ref the kit is read from. Pin a commit sha for a reproducible run; the resolved sha is recorded in the report either way.
required: false
default: main
fail-on-failure:
description: Fail the job when any case does not pass.
required: false
default: "true"
report-path:
description: Where to write the report JSON.
required: false
default: proofable-report.json

outputs:
report:
description: Path to the trustharness.report/v1 JSON.
value: ${{ steps.report.outputs.path }}
passed:
description: Cases that passed.
value: ${{ steps.report.outputs.passed }}
total:
description: Cases attempted.
value: ${{ steps.report.outputs.total }}

runs:
using: composite
steps:
- name: Read the kit at a pinned ref
shell: bash
env:
KIT_REF: ${{ inputs.kit-ref }}
run: |
set -euo pipefail
kit="$RUNNER_TEMP/proofable-kit"
rm -rf "$kit"
git init -q "$kit"
cd "$kit"
git remote add origin https://github.com/proofable/docs.git
git fetch -q --depth 1 origin "$KIT_REF"
git checkout -q FETCH_HEAD
echo "PROOFABLE_KIT_COMMIT=$(git rev-parse HEAD)" >> "$GITHUB_ENV"
echo "PROOFABLE_KIT_DIR=$kit/public/evidence/aaif/authority-at-dispatch/reproduce" >> "$GITHUB_ENV"

- name: Install the kit
shell: bash
run: |
set -euo pipefail
cd "$PROOFABLE_KIT_DIR"
npm ci --no-audit --no-fund

- name: Register the review agent
shell: bash
env:
PROOFABLE_ACCESS_KEY: ${{ inputs.access-key }}
AAD_OPERATOR: ${{ inputs.operator }}
AAD_AGENT_ID: ${{ inputs.agent-id }}
run: |
set -euo pipefail
cd "$PROOFABLE_KIT_DIR"
node aad.mjs setup

- name: Run the cases
shell: bash
env:
PROOFABLE_ACCESS_KEY: ${{ inputs.access-key }}
AAD_OPERATOR: ${{ inputs.operator }}
AAD_AGENT_ID: ${{ inputs.agent-id }}
AAD_CASES: ${{ inputs.cases }}
run: |
set -euo pipefail
cd "$PROOFABLE_KIT_DIR"
# The newest sealed run is the one this job reports on.
if [ "$AAD_CASES" = "all" ]; then
node aad.mjs run
else
# shellcheck disable=SC2086
node aad.mjs run $AAD_CASES
fi
run_dir="$(ls -1dt runs/*/ | head -n 1)"
echo "PROOFABLE_RUN_DIR=$PROOFABLE_KIT_DIR/${run_dir%/}" >> "$GITHUB_ENV"

- name: Verify the records offline
shell: bash
run: |
set -euo pipefail
cd "$PROOFABLE_KIT_DIR"
set +e
node aad.mjs verify "$PROOFABLE_RUN_DIR" > "$RUNNER_TEMP/proofable-verify.log" 2>&1
verify_status=$?
set -e
cat "$RUNNER_TEMP/proofable-verify.log"
echo "PROOFABLE_VERIFY_STATUS=$verify_status" >> "$GITHUB_ENV"

- name: Build the report
id: report
shell: bash
env:
AAD_OPERATOR: ${{ inputs.operator }}
REPORT_PATH: ${{ inputs.report-path }}
SOURCE_URL: ${{ github.server_url }}/${{ github.repository }}
run: |
set -euo pipefail
node "$PROOFABLE_KIT_DIR/../action/report.mjs" "$PROOFABLE_RUN_DIR" \
--verify-log "$RUNNER_TEMP/proofable-verify.log" \
--source "$SOURCE_URL" \
--out "$REPORT_PATH" \
--summary "$RUNNER_TEMP/proofable-report.md"
node -e '
const r = require(process.env.REPORT_PATH.replace(/^/, "./"));
const fs = require("fs");
fs.appendFileSync(process.env.GITHUB_OUTPUT, `path=${process.env.REPORT_PATH}\n`);
fs.appendFileSync(process.env.GITHUB_OUTPUT, `passed=${r.authority.passed}\n`);
fs.appendFileSync(process.env.GITHUB_OUTPUT, `total=${r.authority.total}\n`);
'
cat "$RUNNER_TEMP/proofable-report.md" >> "$GITHUB_STEP_SUMMARY"

- name: Keep the sealed run and the report
uses: actions/upload-artifact@v4
with:
name: proofable-trust-harness
path: |
${{ inputs.report-path }}
${{ env.PROOFABLE_RUN_DIR }}
if-no-files-found: error

- name: Fail when a case did not pass
if: ${{ inputs.fail-on-failure == 'true' }}
shell: bash
run: |
set -euo pipefail
if [ "${PROOFABLE_VERIFY_STATUS:-1}" != "0" ]; then
echo "The offline check did not pass; see the report and the verify log."
exit 1
fi
Loading
Loading