From e9910752d6d0ac277900cff8e99aa0d5cda28653 Mon Sep 17 00:00:00 2001 From: stacknil Date: Mon, 27 Apr 2026 18:45:29 +0800 Subject: [PATCH] add reviewer evidence pack --- README.md | 131 +++++++------- tools/sbom-diff-and-risk/README.md | 2 +- .../sbom-diff-and-risk/docs/reviewer-brief.md | 96 +++++----- .../docs/reviewer-evidence-pack.md | 171 ++++++++++++++++++ 4 files changed, 287 insertions(+), 113 deletions(-) create mode 100644 tools/sbom-diff-and-risk/docs/reviewer-evidence-pack.md diff --git a/README.md b/README.md index c564778..b536ffb 100644 --- a/README.md +++ b/README.md @@ -1,69 +1,70 @@ -# scientific-computing-toolkit - -This repository is a portfolio space for scientific-computing infrastructure, systems tooling, and supply-chain-security experiments that favor deterministic behavior, auditable outputs, and clear release evidence. - -## Current Flagship Project - -[`tools/sbom-diff-and-risk`](tools/sbom-diff-and-risk/README.md) is the current flagship tool. It compares SBOMs and dependency manifests, produces JSON, Markdown, and SARIF review artifacts, supports local policy checks, and can optionally record PyPI provenance and OpenSSF Scorecard evidence. - -For a fast reviewer overview, start with the [`sbom-diff-and-risk` reviewer brief](tools/sbom-diff-and-risk/docs/reviewer-brief.md). - -## Why This Repository Exists - -Scientific and security-oriented engineering often needs small, inspectable tools that make evidence easier to review. This repository collects projects that emphasize: - -- deterministic local analysis -- machine-readable security and review output -- conservative policy checks -- explicit provenance and release verification boundaries -- documentation that separates tool behavior from distribution evidence - -## Project Map - -| Project | Status | What to review | -| --- | --- | --- | -| [`sbom-diff-and-risk`](tools/sbom-diff-and-risk/README.md) | Released at `v0.5.0` | Deterministic SBOM/dependency diffing, JSON/Markdown/SARIF output, local policy checks, optional provenance and Scorecard evidence. | - -Useful entry points: - +# scientific-computing-toolkit + +This repository is a portfolio space for scientific-computing infrastructure, systems tooling, and supply-chain-security experiments that favor deterministic behavior, auditable outputs, and clear release evidence. + +## Current Flagship Project + +[`tools/sbom-diff-and-risk`](tools/sbom-diff-and-risk/README.md) is the current flagship tool. It compares SBOMs and dependency manifests, produces JSON, Markdown, and SARIF review artifacts, supports local policy checks, and can optionally record PyPI provenance and OpenSSF Scorecard evidence. + +For a fast reviewer overview, start with the [`sbom-diff-and-risk` reviewer brief](tools/sbom-diff-and-risk/docs/reviewer-brief.md). + +## Why This Repository Exists + +Scientific and security-oriented engineering often needs small, inspectable tools that make evidence easier to review. This repository collects projects that emphasize: + +- deterministic local analysis +- machine-readable security and review output +- conservative policy checks +- explicit provenance and release verification boundaries +- documentation that separates tool behavior from distribution evidence + +## Project Map + +| Project | Status | What to review | +| --- | --- | --- | +| [`sbom-diff-and-risk`](tools/sbom-diff-and-risk/README.md) | Released at `v0.5.0` | Deterministic SBOM/dependency diffing, JSON/Markdown/SARIF output, local policy checks, optional provenance and Scorecard evidence. | + +Useful entry points: + - [`sbom-diff-and-risk` README](tools/sbom-diff-and-risk/README.md) - [Reviewer brief](tools/sbom-diff-and-risk/docs/reviewer-brief.md) +- [Reviewer evidence pack](tools/sbom-diff-and-risk/docs/reviewer-evidence-pack.md) - [v0.5.0 release notes](tools/sbom-diff-and-risk/RELEASE_NOTES_v0.5.0.md) - [Examples](tools/sbom-diff-and-risk/examples/) - -## Verification And Release Evidence - -`sbom-diff-and-risk` has separate verification surfaces. They are related, but they do not prove the same thing. - -| Evidence | Where to start | -| --- | --- | -| Tool verification guide | [`docs/verification.md`](tools/sbom-diff-and-risk/docs/verification.md) | -| GitHub Release asset verification | [`docs/release-provenance.md`](tools/sbom-diff-and-risk/docs/release-provenance.md) | -| TestPyPI Trusted Publishing dry-run | [`docs/pypi-trusted-publishing-readiness.md`](tools/sbom-diff-and-risk/docs/pypi-trusted-publishing-readiness.md) | -| Production PyPI decision gate | [`docs/pypi-production-publishing-decision.md`](tools/sbom-diff-and-risk/docs/pypi-production-publishing-decision.md) | - -The TestPyPI Trusted Publishing dry-run has been validated. Production PyPI publishing is intentionally deferred. - -## What This Repository Intentionally Does Not Claim - -- It does not claim that `sbom-diff-and-risk` is a vulnerability scanner. -- It does not claim to resolve CVEs, advisories, exploitability, or package safety verdicts. -- It does not treat optional provenance or Scorecard evidence as proof that a dependency is safe. -- It does not imply that production PyPI publishing is enabled. -- It does not treat GitHub release verification, GitHub workflow artifact attestations, and PyPI Trusted Publishing provenance as interchangeable evidence. - -## Reviewer Quick Path - -1. Read the [`sbom-diff-and-risk` reviewer brief](tools/sbom-diff-and-risk/docs/reviewer-brief.md). -2. Skim the [`sbom-diff-and-risk` README](tools/sbom-diff-and-risk/README.md) for CLI scope and examples. -3. Check the [v0.5.0 release notes](tools/sbom-diff-and-risk/RELEASE_NOTES_v0.5.0.md). -4. Use the [verification guide](tools/sbom-diff-and-risk/docs/verification.md) to choose the right provenance check. -5. Inspect the [examples](tools/sbom-diff-and-risk/examples/) for sample reports and policy files. - -## Status - -- Current flagship release: `sbom-diff-and-risk` `v0.5.0` -- GitHub Release assets: available for `v0.5.0` -- TestPyPI Trusted Publishing dry-run: completed -- Production PyPI publishing: intentionally deferred - + +## Verification And Release Evidence + +`sbom-diff-and-risk` has separate verification surfaces. They are related, but they do not prove the same thing. + +| Evidence | Where to start | +| --- | --- | +| Tool verification guide | [`docs/verification.md`](tools/sbom-diff-and-risk/docs/verification.md) | +| GitHub Release asset verification | [`docs/release-provenance.md`](tools/sbom-diff-and-risk/docs/release-provenance.md) | +| TestPyPI Trusted Publishing dry-run | [`docs/pypi-trusted-publishing-readiness.md`](tools/sbom-diff-and-risk/docs/pypi-trusted-publishing-readiness.md) | +| Production PyPI decision gate | [`docs/pypi-production-publishing-decision.md`](tools/sbom-diff-and-risk/docs/pypi-production-publishing-decision.md) | + +The TestPyPI Trusted Publishing dry-run has been validated. Production PyPI publishing is intentionally deferred. + +## What This Repository Intentionally Does Not Claim + +- It does not claim that `sbom-diff-and-risk` is a vulnerability scanner. +- It does not claim to resolve CVEs, advisories, exploitability, or package safety verdicts. +- It does not treat optional provenance or Scorecard evidence as proof that a dependency is safe. +- It does not imply that production PyPI publishing is enabled. +- It does not treat GitHub release verification, GitHub workflow artifact attestations, and PyPI Trusted Publishing provenance as interchangeable evidence. + +## Reviewer Quick Path + +1. Read the [`sbom-diff-and-risk` reviewer brief](tools/sbom-diff-and-risk/docs/reviewer-brief.md). +2. Skim the [`sbom-diff-and-risk` README](tools/sbom-diff-and-risk/README.md) for CLI scope and examples. +3. Check the [v0.5.0 release notes](tools/sbom-diff-and-risk/RELEASE_NOTES_v0.5.0.md). +4. Use the [verification guide](tools/sbom-diff-and-risk/docs/verification.md) to choose the right provenance check. +5. Inspect the [examples](tools/sbom-diff-and-risk/examples/) for sample reports and policy files. + +## Status + +- Current flagship release: `sbom-diff-and-risk` `v0.5.0` +- GitHub Release assets: available for `v0.5.0` +- TestPyPI Trusted Publishing dry-run: completed +- Production PyPI publishing: intentionally deferred + diff --git a/tools/sbom-diff-and-risk/README.md b/tools/sbom-diff-and-risk/README.md index a8cdea9..7dfef4b 100644 --- a/tools/sbom-diff-and-risk/README.md +++ b/tools/sbom-diff-and-risk/README.md @@ -10,7 +10,7 @@ It uses conservative heuristics for change intelligence. By default it does not This project has two different provenance stories: -For a concise reviewer-facing overview, start with [docs/reviewer-brief.md](docs/reviewer-brief.md). +For a concise reviewer-facing overview, start with [docs/reviewer-brief.md](docs/reviewer-brief.md). For reproducible review evidence and verification commands, use [docs/reviewer-evidence-pack.md](docs/reviewer-evidence-pack.md). 1. If you want to verify `sbom-diff-and-risk` itself, start with [docs/verification.md](D:/OneDrive/Code/scientific-computing-toolkit/tools/sbom-diff-and-risk/docs/verification.md). 2. If you want to use `sbom-diff-and-risk` to analyze third-party dependency provenance, start with [Dependency provenance analysis](#dependency-provenance-analysis-opt-in) and [Dependency provenance reporting](#dependency-provenance-reporting). diff --git a/tools/sbom-diff-and-risk/docs/reviewer-brief.md b/tools/sbom-diff-and-risk/docs/reviewer-brief.md index 1639259..67fa952 100644 --- a/tools/sbom-diff-and-risk/docs/reviewer-brief.md +++ b/tools/sbom-diff-and-risk/docs/reviewer-brief.md @@ -1,56 +1,58 @@ -# Reviewer brief - -## Summary - -`sbom-diff-and-risk` is a local CLI for comparing two SBOMs or dependency manifests and producing deterministic review artifacts: JSON, Markdown, and SARIF. It is built for conservative supply-chain review, not for vulnerability scanning or package reputation scoring. - -## Why this project matters - -Dependency review often needs evidence that is stable enough for code review, CI, and audit trails. This project turns dependency changes into repeatable findings, optional policy outcomes, and machine-readable security output while keeping default analysis offline and file-based. - -## Capability map - -| Area | What exists | -| --- | --- | -| Deterministic local analysis | Compares CycloneDX, SPDX, `requirements.txt`, and conservative `pyproject.toml` inputs without hidden network access by default. | -| Reviewer output | Produces JSON and Markdown reports for dependency diffs, heuristic risk buckets, and policy outcomes. | -| Security tooling output | Emits a conservative SARIF subset for selected high-signal findings and explicit policy violations. | -| Provenance-aware reporting | Optionally records PyPI provenance and integrity evidence when `--enrich-pypi` is enabled. | -| Scorecard signals | Optionally records OpenSSF Scorecard evidence when `--enrich-scorecard` is enabled and a repository mapping is explicit enough. | -| Policy support | Supports local YAML policies for thresholds, source allowlists, provenance requirements, and Scorecard thresholds. | - +# Reviewer brief + +## Summary + +`sbom-diff-and-risk` is a local CLI for comparing two SBOMs or dependency manifests and producing deterministic review artifacts: JSON, Markdown, and SARIF. It is built for conservative supply-chain review, not for vulnerability scanning or package reputation scoring. + +## Why this project matters + +Dependency review often needs evidence that is stable enough for code review, CI, and audit trails. This project turns dependency changes into repeatable findings, optional policy outcomes, and machine-readable security output while keeping default analysis offline and file-based. + +## Capability map + +| Area | What exists | +| --- | --- | +| Deterministic local analysis | Compares CycloneDX, SPDX, `requirements.txt`, and conservative `pyproject.toml` inputs without hidden network access by default. | +| Reviewer output | Produces JSON and Markdown reports for dependency diffs, heuristic risk buckets, and policy outcomes. | +| Security tooling output | Emits a conservative SARIF subset for selected high-signal findings and explicit policy violations. | +| Provenance-aware reporting | Optionally records PyPI provenance and integrity evidence when `--enrich-pypi` is enabled. | +| Scorecard signals | Optionally records OpenSSF Scorecard evidence when `--enrich-scorecard` is enabled and a repository mapping is explicit enough. | +| Policy support | Supports local YAML policies for thresholds, source allowlists, provenance requirements, and Scorecard thresholds. | + ## Evidence map | Question | Evidence path | | --- | --- | | What does the tool do? | `README.md`, examples, tests, and generated sample reports. | +| How can a reviewer reproduce the core evidence? | [reviewer-evidence-pack.md](reviewer-evidence-pack.md) for demo, release, TestPyPI, and SARIF verification paths. | | Are default runs offline? | CLI docs, tests for no-enrichment behavior, and explicit enrichment flags. | | Can code scanning consume the output? | `docs/github-code-scanning.md` and `examples/sample-sarif.sarif`. | | Can the tool's own artifacts be verified? | `docs/self-provenance.md` for workflow artifact attestations. | -| Can GitHub release assets be verified? | `docs/release-provenance.md` for release asset verification. | -| Did Trusted Publishing get exercised safely? | `docs/pypi-trusted-publishing-readiness.md` documents the completed TestPyPI dry-run. | -| Is production PyPI enabled? | `docs/pypi-production-publishing-decision.md` documents that production PyPI is intentionally deferred. | - -## Quick verification path - +| Can GitHub release assets be verified? | `docs/release-provenance.md` for release asset verification. | +| Did Trusted Publishing get exercised safely? | `docs/pypi-trusted-publishing-readiness.md` documents the completed TestPyPI dry-run. | +| Is production PyPI enabled? | `docs/pypi-production-publishing-decision.md` documents that production PyPI is intentionally deferred. | + +## Quick verification path + 1. Read this brief for the 30-second project shape. -2. Read `README.md` for CLI scope, supported inputs, and examples. -3. Read `docs/verification.md` to choose the right verification path. -4. Use `docs/self-provenance.md` when verifying workflow-built wheel or source distribution artifacts. -5. Use `docs/release-provenance.md` when verifying GitHub Release assets. -6. Use `docs/pypi-production-publishing-decision.md` before making any production PyPI publishing decision. - -## What this project intentionally does not claim - -- It does not claim to be a vulnerability scanner. -- It does not resolve CVEs, advisories, or exploitability. -- It does not score package reputation or declare packages safe. -- It does not perform hidden network enrichment. -- It does not treat TestPyPI success as production PyPI readiness. -- It does not currently publish to production PyPI. -- It does not treat PyPI Trusted Publishing provenance, GitHub workflow artifact attestations, and GitHub Release asset verification as interchangeable evidence. - -## Resume / application wording - -Built `sbom-diff-and-risk`, a deterministic SBOM and dependency diff CLI that produces JSON, Markdown, and SARIF review artifacts; supports local policy checks and optional provenance/Scorecard evidence; and documents a release verification story covering GitHub artifact attestations, GitHub Release assets, TestPyPI Trusted Publishing validation, and intentionally deferred production PyPI publishing. - +2. Read [reviewer-evidence-pack.md](reviewer-evidence-pack.md) for reproducible commands and evidence paths. +3. Read `README.md` for CLI scope, supported inputs, and examples. +4. Read `docs/verification.md` to choose the right verification path. +5. Use `docs/self-provenance.md` when verifying workflow-built wheel or source distribution artifacts. +6. Use `docs/release-provenance.md` when verifying GitHub Release assets. +7. Use `docs/pypi-production-publishing-decision.md` before making any production PyPI publishing decision. + +## What this project intentionally does not claim + +- It does not claim to be a vulnerability scanner. +- It does not resolve CVEs, advisories, or exploitability. +- It does not score package reputation or declare packages safe. +- It does not perform hidden network enrichment. +- It does not treat TestPyPI success as production PyPI readiness. +- It does not currently publish to production PyPI. +- It does not treat PyPI Trusted Publishing provenance, GitHub workflow artifact attestations, and GitHub Release asset verification as interchangeable evidence. + +## Resume / application wording + +Built `sbom-diff-and-risk`, a deterministic SBOM and dependency diff CLI that produces JSON, Markdown, and SARIF review artifacts; supports local policy checks and optional provenance/Scorecard evidence; and documents a release verification story covering GitHub artifact attestations, GitHub Release assets, TestPyPI Trusted Publishing validation, and intentionally deferred production PyPI publishing. + diff --git a/tools/sbom-diff-and-risk/docs/reviewer-evidence-pack.md b/tools/sbom-diff-and-risk/docs/reviewer-evidence-pack.md new file mode 100644 index 0000000..4925302 --- /dev/null +++ b/tools/sbom-diff-and-risk/docs/reviewer-evidence-pack.md @@ -0,0 +1,171 @@ +# Reviewer evidence pack + +This page is a reproducible evidence checklist for reviewing `sbom-diff-and-risk`. It focuses on what can be verified from the repository, examples, GitHub release assets, and TestPyPI dry-run documentation. It does not introduce new CLI behavior. + +## Project Identity + +`sbom-diff-and-risk` is a local-first deterministic CLI for comparing SBOMs and dependency manifests. It is designed to produce stable review evidence for dependency changes. + +Core identity: + +- local deterministic SBOM/dependency diffing +- JSON, Markdown, and SARIF output +- local policy checks over diff and risk findings +- optional provenance-aware reporting through explicit PyPI enrichment +- optional OpenSSF Scorecard evidence when repository mapping is explicit enough +- release and distribution documentation that separates tool behavior from artifact provenance + +## Reproducible Demo Path + +From `tools/sbom-diff-and-risk`, install the package in editable development mode: + +```powershell +python -m pip install -e .[dev] +``` + +Generate the default CycloneDX example reports: + +```powershell +sbom-diff-risk compare ` + --before examples/cdx_before.json ` + --after examples/cdx_after.json ` + --format auto ` + --out-json outputs/report.json ` + --out-md outputs/report.md +``` + +Expected output files: + +- `outputs/report.json` +- `outputs/report.md` + +Compare the outputs against the checked-in sample reports: + +```powershell +Compare-Object (Get-Content examples/sample-report.json) (Get-Content outputs/report.json) +Compare-Object (Get-Content examples/sample-report.md) (Get-Content outputs/report.md) +``` + +No differences means the sample path reproduced the committed example output. + +Generate the strict-policy SARIF sample: + +```powershell +sbom-diff-risk compare ` + --before examples/sarif_before.json ` + --after examples/sarif_after.json ` + --policy examples/policy-strict.yml ` + --out-sarif outputs/report.sarif +``` + +Compare the SARIF output against the sample: + +```powershell +Compare-Object (Get-Content examples/sample-sarif.sarif) (Get-Content outputs/report.sarif) +``` + +The SARIF sample is intentionally conservative. It covers selected high-signal findings and explicit policy violations, not every enrichment fact. + +## Release Verification Path + +Start with the GitHub Release for the version under review. For `v0.5.0`, inspect the release and assets: + +```powershell +gh release view v0.5.0 --repo stacknil/scientific-computing-toolkit --json tagName,name,isDraft,isPrerelease,assets,url +``` + +Expected release assets: + +- `sbom_diff_and_risk-0.5.0-py3-none-any.whl` +- `sbom_diff_and_risk-0.5.0.tar.gz` + +For workflow-built artifacts downloaded from a trusted workflow run, verify artifact attestations with the signer workflow: + +```powershell +gh attestation verify path/to/sbom_diff_and_risk-0.5.0-py3-none-any.whl ` + --repo stacknil/scientific-computing-toolkit ` + --signer-workflow stacknil/scientific-computing-toolkit/.github/workflows/sbom-diff-and-risk-ci.yml +``` + +```powershell +gh attestation verify path/to/sbom_diff_and_risk-0.5.0.tar.gz ` + --repo stacknil/scientific-computing-toolkit ` + --signer-workflow stacknil/scientific-computing-toolkit/.github/workflows/sbom-diff-and-risk-ci.yml +``` + +`gh release verify` and `gh release verify-asset` are conditional on immutable releases. Use them only when the repository release is immutable and GitHub has generated release attestations: + +```powershell +gh release view v0.5.0 --repo stacknil/scientific-computing-toolkit --json isImmutable,assets,url +``` + +If `isImmutable` is true, release verification can check the release record and downloaded release assets: + +```powershell +gh release verify v0.5.0 --repo stacknil/scientific-computing-toolkit +gh release verify-asset v0.5.0 path/to/sbom_diff_and_risk-0.5.0-py3-none-any.whl --repo stacknil/scientific-computing-toolkit +``` + +If `isImmutable` is false, use the workflow artifact attestation path as the primary artifact verification story. + +## TestPyPI Evidence Path + +The TestPyPI Trusted Publishing dry-run completed for `sbom-diff-and-risk`. See `pypi-trusted-publishing-readiness.md` for the exact workflow identity and setup notes. + +What this proves: + +- the package metadata can render on TestPyPI +- the TestPyPI upload path can use Trusted Publishing / OIDC +- the workflow separates build/check from upload +- TestPyPI upload was manually gated + +What this does not prove: + +- production PyPI publishing is ready +- production PyPI has a project, pending publisher, or trusted publisher +- future production distributions will be byte-identical to GitHub Release assets +- dependency analysis results are safety verdicts + +Production PyPI is intentionally deferred. See `pypi-production-publishing-decision.md` before making any production publishing decision. + +## Code Scanning / SARIF Evidence Path + +The SARIF output is designed for GitHub code scanning consumption. Start with: + +- `docs/github-code-scanning.md` +- `examples/sample-sarif.sarif` +- `examples/sample-provenance-report.sarif` +- `examples/sample-scorecard-report.sarif` + +The SARIF renderer intentionally emits a conservative subset: + +- selected heuristic findings such as suspicious source, unknown license, and major upgrade +- explicit blocking policy decisions +- selected provenance or Scorecard policy violations when policy turns them into findings + +Avoid overclaiming: + +- SARIF output is not a CVE scanner +- SARIF output is not a malware or reputation verdict +- missing provenance is an evidence gap, not proof of compromise +- Scorecard evidence is advisory unless policy explicitly gates it + +## Non-Claims + +- No hidden network access occurs by default. +- No production PyPI package exists yet. +- No dependency safety verdicts are produced. +- No CVE resolution is performed. +- No advisory database or exploitability analysis is performed. +- No production PyPI publishing workflow is enabled. +- TestPyPI validation is not production PyPI readiness. + +## 30-Second Reviewer Checklist + +- Can I identify what the tool does? Read `README.md` and `reviewer-brief.md`. +- Can I reproduce a deterministic demo? Run the CycloneDX example and compare `outputs/report.*` to `examples/sample-report.*`. +- Can I see machine-readable security output? Inspect or regenerate `examples/sample-sarif.sarif`. +- Can I verify release/distribution evidence? Read `verification.md`, `self-provenance.md`, and `release-provenance.md`. +- Can I distinguish TestPyPI from production PyPI? Read `pypi-trusted-publishing-readiness.md` and `pypi-production-publishing-decision.md`. +- Can I state the non-claims? No CVE scanner, no reputation oracle, no dependency safety verdicts, no production PyPI package yet. +