Skip to content

fix(scripts): setup_benchmark.py is not executable, but 10 error messages tell you to run it - #105

Open
aashish254 wants to merge 1 commit into
google-research:mainfrom
aashish254:fix/setup-benchmark-is-executable
Open

aashish254 wants to merge 1 commit into
google-research:mainfrom
aashish254:fix/setup-benchmark-is-executable

Conversation

@aashish254

Copy link
Copy Markdown

What

Ten shipped error messages tell the user to run scripts/setup_benchmark.py <benchmark>, but the
file is committed mode 100644, so typing that line ends in a kernel rejection instead of setup:

$ scripts/setup_benchmark.py jb
env: scripts/setup_benchmark.py: Permission denied
$ echo $?
126

This PR sets the executable bit on that one file -- the diff is old mode 100644 / new mode 100755, no content line changes (871d1cb9… before and after, sha256sum 80291a59cf6e…
identical in both trees).

The sites, counted over all tracked .py and .sh files:

Form Count Where
run scripts/setup_benchmark.py … (bare path) 10 benchmarks/jobbench.py:970,1081, benchmarks/lab.py:428,473, benchmarks/workbuddy.py:945, core/proposer.py:402, substrate/codex.py:219, substrate/opencode.py:313, third_party/jb_runner/run_benchmark_codex_cli_adapt.sh:493,494
python scripts/setup_benchmark.py … 4 jobbench.py:981, lab.py:936,958, workbuddy.py:959

One of the ten is the line a first-time deployer hits first (harness2/substrate/opencode.py:311-314):

OpenCode checkout missing at <root>/dependencies/opencode (no packages/opencode/src/index.ts):
run scripts/setup_benchmark.py <benchmark> to fetch the pinned checkout at 23fb5e0516c9 (v1.14.18).

Why the mode bit rather than rewording the messages

Verification

Python 3.12.13, measured in a pristine 1cfe254 worktree (/tmp/h2base) and a worktree carrying
only this mode change (/tmp/h2exec). The venv PATH stands for README's documented steps 1-3
(uv venv --python 3.12 .venv && source .venv/bin/activate && uv pip install -e '.[render]').

# Environment Invocation rc Output
A base 100644, venv PATH scripts/setup_benchmark.py --help 126 env: scripts/setup_benchmark.py: Permission denied
B this change 100755, venv PATH scripts/setup_benchmark.py --help 0 argparse usage, Install a benchmark's pinned tools, runner overlays, dependencies, and data.
C this change, venv PATH ./scripts/setup_benchmark.py --help 0 same usage
D base, venv PATH python scripts/setup_benchmark.py --help 0 same usage -- the 4 python-prefixed messages already work, which is why only the bare sites are broken

Rows A vs B differ in nothing but the mode bit, so the fix is proven at the exact seam the message
describes. Shebang resolution was checked rather than assumed: inside the activated venv
/usr/bin/env python3 is Python 3.12.13, which is inside requires-python = ">=3.12,<3.14".

Everything else is unchanged, as it must be for a content-free diff:

Check base this change
python -m unittest discover -s tests Ran 5 tests / OK Ran 5 tests / OK
ruff check . rc=0 rc=0
git diff --stat 1cfe254..HEAD -- 1 file changed, 0 insertions(+), 0 deletions(-)

What this does not fix, stated plainly: with the bit set but no venv activated, the script
still fails -- env PATH=/usr/bin:/bin ./scripts/setup_benchmark.py --help gives rc=1 and
ModuleNotFoundError: No module named 'harness2', because the system interpreter here is 3.9.6
and the package is not installed. python scripts/setup_benchmark.py fails identically at base,
so this PR neither adds nor removes that constraint; the venv is the only supported environment
either way.

No test is added on purpose: a os.access(..., X_OK) assertion would pass or fail depending on
the checkout's platform and core.fileMode, not on the code under review -- a Windows checkout or
a tarball without mode bits would report red on a correct tree. The A/B measurement above is the
witness.

Not a duplicate

…messages say

Ten shipped error messages tell the user to run `scripts/setup_benchmark.py
<benchmark>`, but the file is committed 100644, so following one of them
literally ends in "Permission denied" (exit 126) instead of setup. The two
messages in third_party/jb_runner/run_benchmark_codex_cli_adapt.sh cannot be
reworded without a dependencies/ASSETS.json digest change, so the mode bit is
the only way to make those readable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant