Skip to content

ci(costco): gate MCP evals in Modal - #61

Merged
walkerhughes merged 1 commit into
codex/costco-marketplacefrom
codex/costco-evals
Sep 7, 2026
Merged

walkerhughes merged 1 commit into
codex/costco-marketplacefrom
codex/costco-evals

Conversation

@walkerhughes

Copy link
Copy Markdown
Owner

Summary

  • add four deterministic Harbor tasks covering warehouse discovery, catalog search, item lookup, and shopping-list pricing
  • score outcomes separately from MCP-use and endpoint-bypass process checks
  • run the full agent loop in Modal as a required Costco CI job
  • pin Harbor 0.18.0 and install the exact PR SHA inside each eval image

Validation

  • make check
  • make coverage (87.60%)
  • harbor run ... --print-config with the Modal environment
  • official plugin and skill validators
  • workflow YAML parse

Stack

Depends on #60, which depends on #59.

@walkerhughes
walkerhughes merged commit bdb77bb into main Sep 7, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant