Skip to content

ci: non-blocking paid evals + upload eval traces - #4

Open
nyashkn wants to merge 1 commit into
mainfrom
fix/paid-eval-tools
Open

nyashkn wants to merge 1 commit into
mainfrom
fix/paid-eval-tools

Conversation

@nyashkn

@nyashkn nyashkn commented Sep 16, 2026

Copy link
Copy Markdown
Owner

First post-merge paid eval run on Haiku (run 35137457868) failed the 0.8 threshold:

  • archetype-pick: 1, 0, 0 (judge: did not pick incident-timeline)
  • baseline-dashboard: 1, 1, 0

The traces were written to /tmp/claude-eval-*/out/ and not uploaded, so the failing runs cannot be inspected.

This PR:

  • marks the paid job continue-on-error, since it only runs after merge and cannot block anything
  • uploads each run trace with the eval report

🤖 Generated with Claude Code

Haiku scored archetype-pick 1/3 and baseline-dashboard 2/3 on the first
main run. Traces lived in /tmp and were not uploaded, so the cause can't
be diagnosed yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant