harness(Part B): jinja-pin + route runner scoring through score_branch - #5
Merged
Conversation
… score_branch Two fixes the Part-B smoke surfaced (validated $0; smoke spent ~$1.65, no bad data committed): 1. jinja2 EDITABLE-INSTALL CONTAMINATION: litellm transitively imports jinja2, which in the WSL workspace is pip-editable-installed from the commit0 repos/jinja benchmark — the stub jinja deliberately fails to import (`_failing_new`), so `import litellm` crashed and aider/smolagents produced nothing. Added pin_jinja_for_litellm() (checkout repos/jinja to a branch whose jinja2 imports: smolagents=3.1.4 / kaizen_stage2=3.1.6) BEFORE import litellm in both runners + a preflight step. Verified litellm imports from the broken state. 2. NON-CANONICAL LOCAL-FALLBACK SCORING: the runners' run_pytest_via_commit0 had the stale-log bug fixed in score_branch but never ported here -> it fell back to LOCAL pytest, which collects differently than the container (portalocker_smolagents: 52 local vs 40 canonical commit0). Routed _persist_and_score through the robust score_branch (commit0-only, mtime watermark + container self-heal + retry); dropped the local fallback. 0-collected cells are now honestly tagged "no-summary" (-> pending) instead of a bogus local count. Validated via verify_patches ($0): portalocker_smolagents canonical = 40/0/0 (not 52); marshmallow/jinja aider = 0/0/0 (genuine floor-lib ceilings). Pipeline now produces canonical, faithful artifacts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adjusts the Part-B harness to (1) prevent litellm from crashing due to an editable-install jinja2 resolving to the commit0 repos/jinja stub, and (2) ensure scoring always uses the canonical, robust commit0 test --branch path via the shared score_branch scorer (removing the previously non-canonical local pytest fallback).
Changes:
- Add a
pin_jinja_for_litellm()pre-import hook (and a preflight step) to forcerepos/jinjaonto an importable branch beforeimport litellm. - Route
_persist_and_scorethrough the sharedscore_branchrobust scorer and remove local pytest fallback scoring. - Add
sampling/tosys.pathin runners to accessscore_branch.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| commit0/baselines/smolagents/_smolagents_runner.py | Adds sampling import path, pins jinja before litellm, and routes scoring through score_branch (no local fallback). |
| commit0/baselines/single_shot_sonnet.py | Introduces pin_jinja_for_litellm() helper used by runners to avoid litellm import failures. |
| commit0/baselines/preflight_clean.sh | Adds a preflight step to pin repos/jinja to an importable branch for litellm’s transitive jinja2 import. |
| commit0/baselines/aider/_aider_runner.py | Adds sampling import path, pins jinja before litellm, and routes scoring through score_branch (no local fallback). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
56
to
60
| git, | ||
| load_dotenv, | ||
| pin_jinja_for_litellm, | ||
| run_pytest_via_commit0, | ||
| ) |
Contributor
Author
There was a problem hiding this comment.
Fixed in the latest commit.
Comment on lines
54
to
61
| from single_shot_sonnet import ( # noqa: E402 | ||
| discover_stub_files, | ||
| extract_pdf_text, | ||
| git, | ||
| load_dotenv, | ||
| pin_jinja_for_litellm, | ||
| run_pytest_via_commit0, | ||
| ) |
Contributor
Author
There was a problem hiding this comment.
Fixed in the latest commit.
Comment on lines
+257
to
+263
| repo = Path.home() / "kaizen-commit0" / "repos" / "jinja" | ||
| if not repo.is_dir(): | ||
| return | ||
| for b in ("smolagents", "kaizen_stage2"): | ||
| if subprocess.run(["git", "-C", str(repo), "checkout", "-f", b], | ||
| capture_output=True, text=True).returncode == 0: | ||
| return |
Contributor
Author
There was a problem hiding this comment.
Fixed in the latest commit.
- aider/smolagents: remove now-unused run_pytest_via_commit0 from imports (scoring routed through score_branch). - single_shot pin_jinja_for_litellm: use the module WORKSPACE constant instead of a hardcoded Path.home()/"kaizen-commit0". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment on lines
57
to
61
| git, | ||
| load_dotenv, | ||
| pin_jinja_for_litellm, | ||
| run_pytest_via_commit0, | ||
| ) |
Comment on lines
56
to
60
| git, | ||
| load_dotenv, | ||
| pin_jinja_for_litellm, | ||
| run_pytest_via_commit0, | ||
| ) |
Comment on lines
+202
to
+208
| # 2. Authoritative full-suite score via the SHARED robust scorer (score_branch): | ||
| # commit0-only with mtime watermark + orphaned-container self-heal + retry. NO local | ||
| # pytest fallback — local collects differently than the container, producing | ||
| # non-canonical counts. A 0-collected result is honestly tagged "no-summary" (not a | ||
| # valid full-suite score). See ../../REPRODUCIBILITY.md. | ||
| sc = _score_branch(lib_name, branch, repo_dir=repo_dir, commit=False) | ||
| summary, counts, scoring = sc["summary"], sc["counts"], sc["scoring"] |
Comment on lines
+208
to
+214
| # 2. Authoritative full-suite score via the SHARED robust scorer (score_branch): | ||
| # commit0-only with mtime watermark + orphaned-container self-heal + retry. NO local | ||
| # pytest fallback — local collects differently than the container (e.g. portalocker | ||
| # 52 local vs 40 commit0), producing non-canonical counts. A 0-collected result is | ||
| # honestly tagged "no-summary" (not a valid full-suite score). See ../../REPRODUCIBILITY.md. | ||
| sc = _score_branch(lib_name, branch, repo_dir=repo_dir, commit=False) | ||
| summary, counts, scoring = sc["summary"], sc["counts"], sc["scoring"] |
| Pin repos/jinja to a branch whose jinja2 actually imports (smolagents=3.1.4, | ||
| kaizen_stage2=3.1.6) first. WSL-only env workaround; see ../RERUN_CHECKLIST.md (A5). | ||
| """ | ||
| repo = Path.home() / "kaizen-commit0" / "repos" / "jinja" |
Comment on lines
+33
to
+35
| for b in smolagents kaizen_stage2; do | ||
| if git -C "$jr" checkout -qf "$b" 2>/dev/null; then echo " repos/jinja -> $b"; break; fi | ||
| done |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two harness fixes surfaced by the Part-B smoke (validated $0; smoke spent ~$1.65):
import jinja2resolves to the editable commit0repos/jinja(stub fails to import) →import litellmcrashed → aider/smolagents produced nothing. Addedpin_jinja_for_litellm()beforeimport litellm(+ preflight step).run_pytest_via_commit0had the stale-log bug (fixed inscore_branchbut not ported) → fell back to local pytest, which collects differently than the container (portalocker_smolagents: 52 local vs 40 canonical). Routed_persist_and_scorethroughscore_branch(commit0-only, mtime watermark + container self-heal); dropped the local fallback. 0-collected → honestno-summary.Validation (verify_patches, $0)
portalocker_smolagents_openai: canonical 40/0/0 (was mis-scored 52 local)marshmallow/jinjaaider: 0/0/0 (genuine floor-lib ceilings →no-summary)No regenerated data committed yet — this PR is the harness fix; the data regeneration is the spend-gated Part-B run, now on a proven pipeline.
🤖 Generated with Claude Code