Skip to content

feat: cache the stable system prompt across runs, always-on no-overth… - #39

Merged
AlleyBo55 merged 1 commit into
masterfrom
feat/eval-compare-cache-split
Nov 2, 2026
Merged

AlleyBo55 merged 1 commit into
masterfrom
feat/eval-compare-cache-split

Conversation

@AlleyBo55

Copy link
Copy Markdown
Owner

…ink rules, agent comparison evals

System prompt

  • The run-specific environment (working directory, date, git branch and changed-file count) moves to the end of the system prompt, under "# Environment". On Anthropic the cache breakpoint now sits before that section, so runs in other directories, on other days or with other git state reuse one cached prefix instead of writing a new one. Other providers get the same text in one piece.
  • The no-overthink rules (github.com/AlleyBo55/no-overthink, 6591c77) are embedded and always on. Two lines that contradicted them changed: "be proactive, anticipate what the user did not ask" now says to carry the asked task through, and "briefly summarize what you did" now says to lead with the result.
  • System prompt: 2,331 -> 2,806 tokens (Anthropic count_tokens).

Evals

  • New -agent mode compares agent CLIs (gocode, opencode, crush, claurst) on one Claude model: fresh HOME and git repo per run, one macOS sandbox profile for all, and cost metered by a local proxy from the usage the API returns. Agents only ever get throwaway keys. -runs, -budget, -agent-path, -models-catalog.
  • Fix: durations were always recorded as 0s (deferred write to an unnamed result).

Measured on claude-sonnet-5-5, 15 tasks x 3 runs, all agents 45/45: $ per solved task: gocode 0.013, claurst 0.018, crush 0.039, opencode 0.041. gocode before this change: 0.025 (1 run); cache writes per sweep 92k -> 19-28k.

…ink rules, agent comparison evals

System prompt
- The run-specific environment (working directory, date, git branch and
  changed-file count) moves to the end of the system prompt, under
  "# Environment". On Anthropic the cache breakpoint now sits before that
  section, so runs in other directories, on other days or with other git
  state reuse one cached prefix instead of writing a new one. Other
  providers get the same text in one piece.
- The no-overthink rules (github.com/AlleyBo55/no-overthink, 6591c77) are
  embedded and always on. Two lines that contradicted them changed: "be
  proactive, anticipate what the user did not ask" now says to carry the
  asked task through, and "briefly summarize what you did" now says to
  lead with the result.
- System prompt: 2,331 -> 2,806 tokens (Anthropic count_tokens).

Evals
- New -agent mode compares agent CLIs (gocode, opencode, crush, claurst)
  on one Claude model: fresh HOME and git repo per run, one macOS sandbox
  profile for all, and cost metered by a local proxy from the usage the
  API returns. Agents only ever get throwaway keys. -runs, -budget,
  -agent-path, -models-catalog.
- Fix: durations were always recorded as 0s (deferred write to an
  unnamed result).

Measured on claude-sonnet-5-5, 15 tasks x 3 runs, all agents 45/45:
$ per solved task: gocode 0.013, claurst 0.018, crush 0.039, opencode 0.041.
gocode before this change: 0.025 (1 run); cache writes per sweep 92k -> 19-28k.
@gitguardian

gitguardian Bot commented Nov 2, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
39430256 Triggered Generic High Entropy Secret 127f73a evals/runner/meter.go View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@AlleyBo55
AlleyBo55 merged commit bb2cb8b into master Nov 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant