Repository navigation
fix(scripts): setup_benchmark.py is not executable, but 10 error messages tell you to run it - #105
Open
aashish254 wants to merge 1 commit into
Open
aashish254 wants to merge 1 commit into
aashish254 wants to merge 1 commit into
Conversation
…messages say Ten shipped error messages tell the user to run `scripts/setup_benchmark.py <benchmark>`, but the file is committed 100644, so following one of them literally ends in "Permission denied" (exit 126) instead of setup. The two messages in third_party/jb_runner/run_benchmark_codex_cli_adapt.sh cannot be reworded without a dependencies/ASSETS.json digest change, so the mode bit is the only way to make those readable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ten shipped error messages tell the user to run
scripts/setup_benchmark.py <benchmark>, but thefile is committed mode
100644, so typing that line ends in a kernel rejection instead of setup:This PR sets the executable bit on that one file -- the diff is
old mode 100644 / new mode 100755, no content line changes (871d1cb9…before and after,sha256sum80291a59cf6e…identical in both trees).
The sites, counted over all tracked
.pyand.shfiles:run scripts/setup_benchmark.py …(bare path)benchmarks/jobbench.py:970,1081,benchmarks/lab.py:428,473,benchmarks/workbuddy.py:945,core/proposer.py:402,substrate/codex.py:219,substrate/opencode.py:313,third_party/jb_runner/run_benchmark_codex_cli_adapt.sh:493,494python scripts/setup_benchmark.py …jobbench.py:981,lab.py:936,958,workbuddy.py:959One of the ten is the line a first-time deployer hits first (
harness2/substrate/opencode.py:311-314):Why the mode bit rather than rewording the messages
third_party/jb_runner/run_benchmark_codex_cli_adapt.sh, which iscovered by a
dependencies/ASSETS.jsonchecksum -- editing it means editing ASSETS.json, and8 open PRs are already fighting over that file. The mode bit makes those two readable without
touching them.
neighbouring lines (
jobbench.pyalone is in fix(jobbench): parse run identity from its path suffix #18/fix(jobbench): keep judge worker exceptions ungraded #28/fix: stop passing API keys and host tokens to agent subprocesses #37/Add Claude Code (cc) and Pi (pi) substrates for JobBench #66/core: include task inputs in the reuse identity of a rollout #76/feat: support saved ChatGPT login for native Codex runs #81/Let Claude Code authenticate without Vertex AI; selectable claude_cli improver #84/Add a claude_cli JobBench judge backend #85/fix(jobbench): a judge that produced no total is ungraded, not a zero #89/fix(jobbench): clean up render directory after rollout #90). Awording sweep there is a merge-conflict generator; a one-file mode change is not.
docs/EXPERIMENTS.md:59-61and thescripts/run_*.shheaders always invoke those wrappers asbash scripts/run_lab.sh, so their100644is correct and left alone. Onlysetup_benchmark.pyis told to users as a path toexecute, and only it gets the bit.
harness2-setup jb, which README anddocs/SETUP.mddocument) is ofcourse already fine; this fixes the script-path form the product itself prints.
Verification
Python 3.12.13, measured in a pristine
1cfe254worktree (/tmp/h2base) and a worktree carryingonly this mode change (
/tmp/h2exec). The venv PATH stands for README's documented steps 1-3(
uv venv --python 3.12 .venv && source .venv/bin/activate && uv pip install -e '.[render]').100644, venv PATHscripts/setup_benchmark.py --helpenv: scripts/setup_benchmark.py: Permission denied100755, venv PATHscripts/setup_benchmark.py --helpInstall a benchmark's pinned tools, runner overlays, dependencies, and data../scripts/setup_benchmark.py --helppython scripts/setup_benchmark.py --helppython-prefixed messages already work, which is why only the bare sites are brokenRows A vs B differ in nothing but the mode bit, so the fix is proven at the exact seam the message
describes. Shebang resolution was checked rather than assumed: inside the activated venv
/usr/bin/env python3is Python 3.12.13, which is insiderequires-python = ">=3.12,<3.14".Everything else is unchanged, as it must be for a content-free diff:
python -m unittest discover -s testsRan 5 tests/OKRan 5 tests/OKruff check .git diff --stat 1cfe254..HEAD1 file changed, 0 insertions(+), 0 deletions(-)What this does not fix, stated plainly: with the bit set but no venv activated, the script
still fails --
env PATH=/usr/bin:/bin ./scripts/setup_benchmark.py --helpgives rc=1 andModuleNotFoundError: No module named 'harness2', because the system interpreter here is 3.9.6and the package is not installed.
python scripts/setup_benchmark.pyfails identically at base,so this PR neither adds nor removes that constraint; the venv is the only supported environment
either way.
No test is added on purpose: a
os.access(..., X_OK)assertion would pass or fail depending onthe checkout's platform and
core.fileMode, not on the code under review -- a Windows checkout ora tarball without mode bits would report red on a correct tree. The A/B measurement above is the
witness.
Not a duplicate
permission denied,setup_benchmark,executable. The hits are incidental uses of the word"execute" in unrelated reports.
gh pr list --state open --limit 200 --json filesover all 75 open PRs: 0 touchscripts/setup_benchmark.py. The only PRs underscripts/are fix(workflow): propagate benchmark report failures #9 and fix(workflow): preserve seed across refinement and grading #17, which editrun_jobbench.sh/run_lab.sh/run_workbuddy.sh(untouched here), and my own style(scripts): make render_readme_typing.py satisfy ruff format #103, whichreformats
scripts/render_readme_typing.py-- a different file, so no overlap.git merge-tree --write-treeontomain, in both orders: clean. Re-run against the twoneighbouring
scripts/PRs at their current heads (fix(workflow): propagate benchmark report failures #9e72bb12, fix(workflow): preserve seed across refinement and grading #17d1bf6f0): clean bothways, and
git diff-tree --summaryshows neither changes a file mode. (My first pass reportedthese as conflicts only because the refs had not been fetched yet -- corrected by fetching and
re-measuring, not by reasoning about it.)
ruff format --check .reports1 file would be reformatted, 49 files already formattedin both trees, forscripts/render_readme_typing.py. That is pre-existing onmainand is what PR style(scripts): make render_readme_typing.py satisfy ruff format #103 fixes; it is not introduced or affected here.