Skip to content

fix(evals): wait for active native execution before grading - #209

Merged
vriesd merged 1 commit into
mainfrom
fix/eval-active-session-wait
Oct 28, 2026
Merged

vriesd merged 1 commit into
mainfrom
fix/eval-active-session-wait

Conversation

@vriesd

@vriesd vriesd commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

OpenCode can still be executing while its next model response has not appeared in exported messages. The evaluator could mistake that gap for quiet and grade unfinished work. It now includes the native active-session state for the root and owned descendants in its existing busy predicate, while retaining the same deadline and cancellation paths.

The bug reproduces with a scripted loopback provider. The retained paid run did not capture active state at its stop, so its exact causal connection remains unproven. Its original failed grade and receipts remain unchanged.

Proof: canonical push preflight passed 5,761 tests, with 27 skips and zero failures. Pinned Bun 1.4.0 evaluator checks pass 686 tests with four skips. The same native OpenCode 2.1.4 smoke assertions fail on base while the session is active and pass on head (128 assertions). The packaged plugin remains byte-identical at SHA-256 7883cfbf… .

Reviewed with sol-loop: 1 round with gpt-6.1-sol on high. Sol verdict: approved.

Created with gpt-6.1-sol in T3 Code through the Codex harness.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T04:09:55.938794Z a12eedd PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@vriesd
vriesd merged commit 2c83c38 into main Oct 28, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants