Skip to content

test(costco): add real-world shopping evals - #62

Merged
walkerhughes merged 1 commit into
codex/costco-evalsfrom
codex/costco-real-world-evals
Sep 7, 2026
Merged

walkerhughes merged 1 commit into
codex/costco-evalsfrom
codex/costco-real-world-evals

Conversation

@walkerhughes

Copy link
Copy Markdown
Owner

Summary

  • add a location-to-list eval for a two-dinner Costco run with ingredient reuse
  • verify bulk package math and extended price calculations
  • require hidden prices to remain unpriced instead of being treated as zero
  • compare qualifying alternatives by public price
  • require every MCP tool in multi-step tasks while continuing to reject endpoint bypasses

Why these evals

The original suite proves individual tools. These scenarios exercise decisions people need before a warehouse trip: where to go, how many bulk packages to buy, whether the estimate is complete, which alternative is economical, and how products can cover multiple meals.

Validation

  • make check
  • make coverage (87.60%)
  • 57 process-check scenarios and 32 outcome-check scenarios across 8 evals
  • deterministic eval generation
  • git diff --check

Stack

Depends on #61, which depends on #60 and #59.

@walkerhughes
walkerhughes merged commit cb5152f into main Sep 7, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant