Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,9 @@ TestForge is advisory verification machinery. It does not prove defect freedom,
## Repository map

- [`docs/`](docs/) - the tailored GitHub Pages site, generated hero artwork, and site-source boundary.
- [`testforge/`](testforge/) - the complete portable TestForge Augment v1.1.4.
- [`testforge/`](testforge/) - the complete portable TestForge Augment v1.1.5.
- [`testforge/docs/QUICK-START.md`](testforge/docs/QUICK-START.md) - install and first-use guide.
- [`RELEASE-NOTES-v1.1.4.md`](RELEASE-NOTES-v1.1.4.md) - plugin-publication changes and exact untested boundary.
- [`RELEASE-NOTES-v1.1.5.md`](RELEASE-NOTES-v1.1.5.md) - plugin-publication changes and exact untested boundary.
- [`ARCHIVE-CUSTODY.md`](ARCHIVE-CUSTODY.md) - canonical Augment, plugin, standalone-skill, Claude, GitHub, and backup custody.
- [`PLUGIN-DIRECTORY-SUBMISSION-v1.1.4.md`](PLUGIN-DIRECTORY-SUBMISSION-v1.1.4.md) - exact OpenAI draft listing, portal-specific upload custody, reviewer cases, and owner-only submission gate.
- [`testforge/docs/SALES-DEMO.md`](testforge/docs/SALES-DEMO.md) - a compact proof-of-value scenario.
Expand All @@ -59,7 +59,7 @@ codex plugin marketplace add Stunspot/TestForge
codex plugin add testforge@cd-testforge
```

Start a new Codex task, then invoke `$software-verification` or `$verification-reviewer`. The plugin bundles the two self-contained TestForge v1.1.4 skills so their doctrine, tools, examples, and status vocabulary stay aligned. The separate Augment behavioral-evaluation harness remains in this repository rather than the skills-only plugin. Its marketplace namespace is product-specific, so TestForge can coexist with other Collaborative Dynamics plugin repositories.
Start a new Codex task, then invoke `$software-verification` or `$verification-reviewer`. The plugin bundles the two self-contained TestForge v1.1.5 skills so their doctrine, tools, examples, and status vocabulary stay aligned. The separate Augment behavioral-evaluation harness remains in this repository rather than the skills-only plugin. Its marketplace namespace is product-specific, so TestForge can coexist with other Collaborative Dynamics plugin repositories.

## Quick start: use the standalone Agent SKILLs

Expand Down
11 changes: 11 additions & 0 deletions RELEASE-NOTES-v1.1.5.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# TestForge v1.1.5

## What changed

This patch release restores the verification custody boundary after an unexpected result. TestForge must first classify the cause. A product defect or missing product invariant ends the current verification cycle and returns the candidate upstream as not ready or insufficiently evidenced. TestForge does not patch the product, continue through product failures, or rerun the repaired product inside the same cycle.

A repaired product is a new frozen candidate with a new evidence cutoff and a new verification cycle. Only defects proven to belong to the test, tool, fixture, or execution environment may be corrected and rerun within the existing cycle.

## Package boundary

The release contains synchronized Codex plugin and Claude skill distributions. Static package verification does not prove live host activation, customer outcomes, or defect freedom.
Binary file added claude-ai/software-verification-v1.1.5.zip
Binary file not shown.
Binary file added claude-ai/verification-reviewer-v1.1.5.zip
Binary file not shown.
4 changes: 2 additions & 2 deletions documentation-manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"testforge/docs/VALIDATION.md",
"testforge/docs/LIMITATIONS.md",
"testforge/docs/SUPPORT-AND-VERSIONING.md",
"RELEASE-NOTES-v1.1.4.md",
"RELEASE-NOTES-v1.1.5.md",
"ARCHIVE-CUSTODY.md",
"PLUGIN-DIRECTORY-SUBMISSION-v1.1.4.md",
"testforge/PROVENANCE.md"
Expand All @@ -30,7 +30,7 @@
"recovery": ["README.md", "testforge/docs/TROUBLESHOOTING.md", "testforge/docs/SUPPORT-AND-VERSIONING.md"],
"privacy_security": ["testforge/docs/DATA-AND-PRIVACY.md", "testforge/docs/TERMS-OF-USE.md", "testforge/SECURITY.md"],
"evidence_limits": ["testforge/docs/VALIDATION.md", "testforge/docs/LIMITATIONS.md"],
"support_maintenance": ["testforge/docs/SUPPORT-AND-VERSIONING.md", "RELEASE-NOTES-v1.1.4.md", "ARCHIVE-CUSTODY.md", "PLUGIN-DIRECTORY-SUBMISSION-v1.1.4.md"],
"support_maintenance": ["testforge/docs/SUPPORT-AND-VERSIONING.md", "RELEASE-NOTES-v1.1.5.md", "ARCHIVE-CUSTODY.md", "PLUGIN-DIRECTORY-SUBMISSION-v1.1.4.md"],
"provenance": ["testforge/PROVENANCE.md"]
}
}
2 changes: 1 addition & 1 deletion plugins/testforge/.codex-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "testforge",
"version": "1.1.4",
"version": "1.1.5",
"description": "Risk-driven software verification and independent evidence review for coding Agents.",
"author": {
"name": "Collaborative Dynamics",
Expand Down
6 changes: 5 additions & 1 deletion plugins/testforge/skills/software-verification/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,11 @@ Keep diagnostic and reproduction commands capability-matched, read-only where po
- `scripts/normalize_test_results.py` for JUnit XML, Jest JSON, or generic command records.
- `scripts/assemble_report.py` only after the manifest and referenced evidence validate.

Classify every failure before anything is patched: `PRODUCT_DEFECT`, `TEST_DEFECT`, `ENVIRONMENT_FAILURE`, `FLAKY_OR_NONDETERMINISTIC`, `EXPECTED_CONTRACT_CHANGE`, `TOOLING_FAILURE`, or `INSUFFICIENT_EVIDENCE`. A product defect withdraws the readiness claim and returns to builder custody. A newly exposed requirement, invariant, or design decision means the candidate entered TestForge before the upstream work was complete; preserve that discovery as `INSUFFICIENT_EVIDENCE` and return it upstream. TestForge owns the evidence and verdict, not product completion. Change the verification apparatus here only when evidence identifies a test or tooling defect. Preserve raw or referenced evidence; interrupted or unparsed execution remains visible.
Classify every unexpected result before anything is changed: `PRODUCT_DEFECT`, `TEST_DEFECT`, `ENVIRONMENT_FAILURE`, `FLAKY_OR_NONDETERMINISTIC`, `EXPECTED_CONTRACT_CHANGE`, `TOOLING_FAILURE`, or `INSUFFICIENT_EVIDENCE`. Preserve the exact failure, locate the earliest observed divergence, keep plausible causes live until evidence separates them, and use the smallest discriminating check needed to support a cause or bound the remaining uncertainty. A workaround that makes the symptom disappear is not a diagnosis.

The classification controls custody. A `PRODUCT_DEFECT` immediately withdraws the submitted candidate's readiness claim, produces a `NOT_READY` finding, and ends that TestForge cycle. A newly exposed requirement, invariant, or design decision produces `INSUFFICIENT_EVIDENCE` and also ends the cycle. TestForge does not patch the product, continue down a queue of subsequent product failures, or rerun the repaired product inside the same verification cycle. Return the finding and evidence to builder custody. If a completed repair is later submitted, treat it as a new frozen candidate with a new verification cycle and evidence cutoff.

TestForge may change and rerun only its own verification apparatus when evidence identifies a `TEST_DEFECT` or `TOOLING_FAILURE`, or make a bounded environment correction when the environment, not the product, is proven to be the cause and the correction does not alter the submitted candidate. If that intervention exposes a different result, reopen the causal model before acting. Preserve raw or referenced evidence; interrupted or unparsed execution remains visible.

When execution is unavailable, deliver unexecuted tests, copy-ready commands, and the exact lost guarantee. Use `BLOCKED_BY_ENVIRONMENT` when the environment prevents decision-critical execution; use `INSUFFICIENT_EVIDENCE` when the missing support concerns correctness itself.

Expand Down
2 changes: 1 addition & 1 deletion plugins/testforge/skills/verification-reviewer/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Ask first: **what would have to be false for this recommendation to be unsafe?**

Use `review-rubric.md` and `adversarial-checks.md`. Re-run `scripts/validate_manifest.py` and `scripts/validate_traceability.py` when tool access exists. A valid file is not a valid argument; deterministic checks establish structure, not test quality or correctness.

Challenge in this order:
Challenge in this order. Before scoring any other lens, enforce custody after failure: a product defect or newly exposed requirement must end that candidate's verification cycle. Treat product patching or retesting inside the same cycle as a review failure.

1. **Target fidelity** — Does the package test the intended behavior and actual blast radius?
2. **Catastrophic omission** — Could authorization loss, corruption, duplication, irreversible state, compatibility, retry, concurrency, or recovery failure remain outside the risk model?
Expand Down
4 changes: 2 additions & 2 deletions release-docs/INSTALL-CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ Python 3.10+ is recommended for the portable verifier but is not required by the

## Available archives

- [software-verification ZIP](../releases/v1.1.4/claude/software-verification-v1.1.4.zip)
- [verification-reviewer ZIP](../releases/v1.1.4/claude/verification-reviewer-v1.1.4.zip)
- [software-verification ZIP](../releases/v1.1.5/claude/software-verification-v1.1.5.zip)
- [verification-reviewer ZIP](../releases/v1.1.5/claude/verification-reviewer-v1.1.5.zip)

## Procedure

Expand Down
4 changes: 2 additions & 2 deletions release-docs/INSTALL-CODEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@

Python 3.10+ is recommended for the portable verifier but is not required by the skills at runtime. Without Python, follow the checksum and reduced-assurance path in the [quick start](QUICK-START.md).

- An extracted `TestForge-v1.1.4.zip` release.
- An extracted `TestForge-v1.1.5.zip` release.
- A Codex build that supports local plugin import or a configured local plugin source directory.
- Permission to add a local plugin on the host.

## Procedure

1. From the extracted release root, run `python tools/verify_release.py .` and require `"ok": true`.
2. Confirm the payload contains [plugin.json](../releases/v1.1.4/codex/testforge/.codex-plugin/plugin.json) and a `codex/testforge/skills/` directory.
2. Confirm the payload contains [plugin.json](../releases/v1.1.5/codex/testforge/.codex-plugin/plugin.json) and a `codex/testforge/skills/` directory.
3. In Codex's supported local-plugin import flow, select the complete `codex/testforge/` directory. If the host instead uses a configured plugin source directory, copy that whole directory there unchanged; do not copy individual skill files out of it.
4. Let Codex reload plugins, then open a fresh task so discovery is tested without stale task state.
5. Confirm `TestForge` and its expected handles are listed by the host.
Expand Down
16 changes: 8 additions & 8 deletions release-docs/MAINTAINER-GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,22 @@ Build each release from the maintained repository on a clean release branch. A p

## Rebuild procedure

1. Confirm `plugins/testforge/skills/` and `testforge/skills/` are byte-identical and the plugin, package, eval suite, and release target all declare version `1.1.4`.
1. Confirm `plugins/testforge/skills/` and `testforge/skills/` are byte-identical and the plugin, package, eval suite, and release target all declare version `1.1.5`.
2. Run `python -B tools/build_public_release_v114.py` from the repository root.
3. Run it a second time and require the same SHA-256 digest.
4. Run `python -B releases/v1.1.4/tools/verify_release.py releases/v1.1.4` and require `ok: true` with no findings.
4. Run `python -B releases/v1.1.5/tools/verify_release.py releases/v1.1.5` and require `ok: true` with no findings.
5. Run the repository unit suites, package validator, eval-suite validator, release-manifest validator, and line-ending verifier.
6. Review all thirteen customer documents as a reader journey, including installation, first value, expected success, troubleshooting, removal, and rollback.
7. Require an independent skeptical review before publication.
8. After publication, download the GitHub asset and compare its SHA-256 with the canonical repository artifact and release shelf copy.

## Evidence pointers

- [manifest.json](../releases/v1.1.4/manifest.json): exact Codex source-file hashes and Claude archive receipts.
- [verification-report.json](../releases/v1.1.4/verification-report.json): portable post-build verification.
- [description-custody.json](../releases/v1.1.4/description-custody.json): customer-facing product description custody.
- [package-receipt.json](../releases/v1.1.4/package-receipt.json): package identity and static claim boundary.
- [receipt.json](../releases/v1.1.4/receipt.json): release identity and evidence boundary.
- `TestForge-v1.1.4.zip.sha256`: detached canonical archive digest.
- [manifest.json](../releases/v1.1.5/manifest.json): exact Codex source-file hashes and Claude archive receipts.
- [verification-report.json](../releases/v1.1.5/verification-report.json): portable post-build verification.
- [description-custody.json](../releases/v1.1.5/description-custody.json): customer-facing product description custody.
- [package-receipt.json](../releases/v1.1.5/package-receipt.json): package identity and static claim boundary.
- [receipt.json](../releases/v1.1.5/receipt.json): release identity and evidence boundary.
- `TestForge-v1.1.5.zip.sha256`: detached canonical archive digest.

Never infer installation, discovery, invocation, or healthy behavior from a passing static package check.
12 changes: 6 additions & 6 deletions release-docs/PACKAGE-REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@ package-receipt.json
verification-report.json
```

The canonical archive is `TestForge-v1.1.4.zip`. The release tree contains `receipt.json`. The `.sha256` file lives beside the archive because an archive cannot contain its own final digest.
The canonical archive is `TestForge-v1.1.5.zip`. The release tree contains `receipt.json`. The `.sha256` file lives beside the archive because an archive cannot contain its own final digest.

## Key records

- [Plugin manifest](../releases/v1.1.4/codex/testforge/.codex-plugin/plugin.json)
- [Release manifest](../releases/v1.1.4/manifest.json)
- [Description custody](../releases/v1.1.4/description-custody.json)
- [Portable verification report](../releases/v1.1.4/verification-report.json)
- [Package receipt](../releases/v1.1.4/package-receipt.json)
- [Plugin manifest](../releases/v1.1.5/codex/testforge/.codex-plugin/plugin.json)
- [Release manifest](../releases/v1.1.5/manifest.json)
- [Description custody](../releases/v1.1.5/description-custody.json)
- [Portable verification report](../releases/v1.1.5/verification-report.json)
- [Package receipt](../releases/v1.1.5/package-receipt.json)
- [Validation procedure](VALIDATION.md)
2 changes: 1 addition & 1 deletion release-docs/PROVENANCE.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# TestForge: provenance

Each [manifest source record](../releases/v1.1.4/manifest.json) identifies a handle and exact included-file hash inventory without embedding an absolute selected-source path. [Description custody](../releases/v1.1.4/description-custody.json) binds the exact model-visible and UI-short prompt surfaces. [Package verification](../releases/v1.1.4/verification-report.json) binds the assembled Codex and Claude bytes.
Each [manifest source record](../releases/v1.1.5/manifest.json) identifies a handle and exact included-file hash inventory without embedding an absolute selected-source path. [Description custody](../releases/v1.1.5/description-custody.json) binds the exact model-visible and UI-short prompt surfaces. [Package verification](../releases/v1.1.5/verification-report.json) binds the assembled Codex and Claude bytes.

## Promotion procedure

Expand Down
4 changes: 2 additions & 2 deletions release-docs/QUICK-START.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Use this path to reach a first verification result without confusing a valid pac

1. Extract the canonical release ZIP into a new directory.
2. If Python 3.10 or newer is available, open a terminal in the extracted directory and run `python tools/verify_release.py .`. Continue when it returns `"ok": true` with no findings.
3. If Python is unavailable, compare the ZIP's SHA-256 with `TestForge-v1.1.4.zip.sha256` using an operating-system checksum tool. Record the portable verifier as unexecuted. If you cannot perform either check, use only an archive obtained from the canonical GitHub release, retain it unchanged, and treat local package integrity as reduced assurance rather than a pass.
3. If Python is unavailable, compare the ZIP's SHA-256 with `TestForge-v1.1.5.zip.sha256` using an operating-system checksum tool. Record the portable verifier as unexecuted. If you cannot perform either check, use only an archive obtained from the canonical GitHub release, retain it unchanged, and treat local package integrity as reduced assurance rather than a pass.
4. Complete the [Codex installation](INSTALL-CODEX.md) or [Claude installation](INSTALL-CLAUDE.md), then start a fresh task or chat.

## First value: verify a completed candidate
Expand All @@ -27,7 +27,7 @@ A useful review returns an independent review verdict, actionable findings or an

## If first value does not appear

1. Confirm the intended TestForge handle is listed by the host and that version `1.1.4` is selected.
1. Confirm the intended TestForge handle is listed by the host and that version `1.1.5` is selected.
2. Name the handle explicitly once to distinguish routing from installation.
3. Confirm the input is a completed candidate for the operator or an existing verification package for the reviewer.
4. Follow [support and recovery](SUPPORT.md), recording package verification, installation, discovery, invocation, and behavior as separate observations.
8 changes: 4 additions & 4 deletions release-docs/SUPPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,15 @@ Do not include credentials, private corpus content, customer data, or unrelated
## Evidence bundle

- Family: `testforge`
- Version: `1.1.4`
- Version: `1.1.5`
- Intended handle
- Host name and host version
- Installation method and exact step that failed
- Expected result and observed result
- Output from `python tools/verify_release.py .`
- [manifest.json](../releases/v1.1.4/manifest.json)
- [verification-report.json](../releases/v1.1.4/verification-report.json)
- [description-custody.json](../releases/v1.1.4/description-custody.json)
- [manifest.json](../releases/v1.1.5/manifest.json)
- [verification-report.json](../releases/v1.1.5/verification-report.json)
- [description-custody.json](../releases/v1.1.5/description-custody.json)
- Whether failure occurs during packaging, installation, discovery, invocation, tool use, or output review

## Issue body
Expand Down
6 changes: 3 additions & 3 deletions release-docs/VALIDATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
```

3. Require exit code `0`, `"ok": true`, and an empty findings list.
4. Compare the result with [verification-report.json](../releases/v1.1.4/verification-report.json).
4. Compare the result with [verification-report.json](../releases/v1.1.5/verification-report.json).

The verifier checks manifest-to-Codex byte parity, Claude ZIP hashes and members, ZIP path safety, plugin metadata, the documentation set, and private-topology leakage.

Expand All @@ -19,8 +19,8 @@ The verifier checks manifest-to-Codex byte parity, Claude ZIP hashes and members
From the unextracted staging or download directory in PowerShell:

```powershell
Get-FileHash -Algorithm SHA256 '.\TestForge-v1.1.4.zip'
Get-Content '.\TestForge-v1.1.4.zip.sha256'
Get-FileHash -Algorithm SHA256 '.\TestForge-v1.1.5.zip'
Get-Content '.\TestForge-v1.1.5.zip.sha256'
```

The computed digest must match the detached checksum supplied beside the archive.
Expand Down
Loading
Loading