Skip to content

ci(trader-joes): gate MCP evals in Modal - #57

Merged
walkerhughes merged 2 commits into
codex/trader-joes-marketplacefrom
codex/trader-joes-evals
Sep 7, 2026
Merged

walkerhughes merged 2 commits into
codex/trader-joes-marketplacefrom
codex/trader-joes-evals

Conversation

@walkerhughes

Copy link
Copy Markdown
Owner

Depends on #56.

Summary

  • add four deterministic Harbor tasks for store lookup, price search, SKU lookup, and shopping-list pricing
  • score both outcome correctness and required MCP use
  • detect direct endpoint bypasses in main and delegated agent transcripts
  • run the Claude Code agent in Modal as a serialized CI merge gate
  • pin eval images to the PR head SHA and upload trial artifacts for diagnosis

Validation

  • Harbor resolves all four tasks with the Modal environment
  • 28 synthetic process-verifier cases pass
  • reward gate self-test passes
  • full plugin suite passes with 92% coverage

@walkerhughes
walkerhughes force-pushed the codex/trader-joes-evals branch from 7f0ef9a to 51aa574 Compare September 7, 2026 04:39
@walkerhughes
walkerhughes merged commit a318d3d into main Sep 7, 2026
3 checks passed
@walkerhughes
walkerhughes deleted the codex/trader-joes-evals branch September 7, 2026 05:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant