feat(benchmarks): cost & energy per solve — the axis nobody else can win - #2473
Merged
Conversation
Joel: 'let's do something unexpected.' Everyone reports raw pass-rate; nobody reports score-per-dollar or score-per-watt — because for a metered cloud harness that number is humiliating and for a local mesh it's the whole point. A $0-per-attempt local system playing the SAME official exams isn't competing in their category, it defines its own: test-time compute (best-of-k, deep web research, retries) is nearly free for us and prohibitive for them, which is exactly why the lean cloud harnesses STAY lean. The section frames the three economies (local $0 / metered API / subsidized datacenter) and notes the SHAPE that matters: as the retake+transfer curves climb, our cost-per-solve stays flat at the hardware while a metered rival's climbs with every attempt. Rows can carry attempt_cost_usd/attempt_wh for a computed column; local is $0 by construction. Average cards, creative strategy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The unexpected reframe: score-per-dollar / score-per-watt. A $0-per-attempt local system on the same official exams defines its own category — and it's why test-time compute is free for us, prohibitive for metered rivals. Three-economy framing + the cost-curve SHAPE argument.
🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo