Skip to content

fix: preserve workflow evidence and independently validate repairs - #57

Merged
timbrinded merged 5 commits into
masterfrom
fix/workflow-evidence-and-validation
Sep 17, 2026
Merged

timbrinded merged 5 commits into
masterfrom
fix/workflow-evidence-and-validation

Conversation

@timbrinded

@timbrinded timbrinded commented Sep 16, 2026

Copy link
Copy Markdown
Owner

Advisory reviews could lose distinct candidates or failed branches and then appear clean. Patch previews could also appear successful based only on the implementer's report. This PR preserves the evidence behind each result and independently validates repair candidates.

  • Transfer semaphore slots directly to queued waiters, including cancellation after selection. This prevents newcomers from exceeding the cap or bypassing FIFO admission.
  • Assign candidate IDs at discovery. Replace location-bucket deduplication with conservative claim matching, retaining contributing IDs and evidence. Keep review-comment anchors separate from cross-file evidence locations.
  • Share discovery, verification, and synthesis across all four advisory workflows. Require candidate identities after discovery, preserve explicit model-output boundaries, and represent completed challenges with both evidence records.
  • Use settled advisory stages with branch counts and failure details. Reconstruct final findings from verified IDs; retain verified records if synthesis fails. Show incomplete coverage as a warning in the terminal.
  • Add opt-in --challenge[=N] stages, with a default limit of three and hard limit of ten. Malformed challenge limits fail before agents start. Challengers try to falsify selected claims; independent adjudication preserves unresolved disagreement.
  • Reconstruct captured repairs on the exact reviewed baseline in fresh evaluator and check workspaces. Record baseline identity, patch hash, evaluator decision, commands, process output, and verified / rejected / blocked / no-patch outcomes. Preserve failed candidates and specific validation failure reasons, including recorded checks when worktree cleanup fails. Cleanup failure blocks a passed candidate and preserves an existing rejection. Apply candidate and regression patches through the worktree registry. Support baseline-fail / repaired-pass regression checks.

Validation:

  • bun run typecheck
  • Full Bun suite: 520 tests passed, including deterministic semaphore handoff/cancellation tests, advisory failure/provenance tests, and real Git-worktree repair validation tests.
  • npm pack --dry-run
  • Interactive pi v0.85.1 under tmux with a scripted local provider: ten-agent challenge workflow, total verifier failure with an incomplete result, extension reload, and inspector navigation. This checks the real bundled runtime and UI; it does not assess live-model review quality.
  • Updated existing test assertions for the new result contract. Made a temp-path assertion respect tmpdir() and removed an incidental tied-timestamp ordering assumption from a persistence test.

No new dependencies. Challenges remain opt-in. Required validation that cannot run blocks candidate verification.

Closes #52
Closes #53
Closes #54
Closes #55
Closes #56

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
🔒 Security Review Completed 2026-09-16T16:17:28.340271Z 10b4f21 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@timbrinded
timbrinded merged commit 5517bfe into master Sep 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment