Skip to content

On a branch cut from origin/main, commit 5c182c0 (local branch goal/20-scripts-musebench-import-score) is cherry-picked: - #17

Merged
unohee merged 1 commit into
mainfrom
goal/22-on-a-branch-cut-from-origin-ma
Nov 17, 2026
Merged

unohee merged 1 commit into
mainfrom
goal/22-on-a-branch-cut-from-origin-ma

Conversation

@unohee

@unohee unohee commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Milestone of an auto-claude run.

Check: export XDG_RUNTIME_DIR="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}" && systemd-run --user --scope -q -p MemoryMax=6G ~/RTX_ENV/bin/python -m pytest -m "not slow" -q -p no:cacheprovider && ~/RTX_ENV/bin/python scripts/check_docs.py && test -f scripts/musebench/cohort.py && ~/RTX_ENV/bin/python scripts/musebench/report.py --help | grep -q -- "--cohort" && ~/RTX_ENV/bin/python scripts/musebench/import_scores.py --help | grep -q -- "--cohort" && ls tests/test_musebench_cohort*.py && ! grep -rnE "/home/unohee|/media/unohee" artifactbench scripts --include='*.py'

Plan: .claude/auto-plan.md (local to the run).

…reports)

scripts/musebench/cohort.py: a cohort is a JSON list of track ids or {"track_ids": [...]}; load() rejects empty,
duplicated or malformed lists, restrict() keeps eval-manifest order and errors on ids not in the eval manifest,
and the fingerprint is sha256 of the sorted ids (same set -> same fingerprint, whatever file or order).

import_scores.py --cohort: imports only the cohort rows, so coverage is relative to the cohort (scored + failed ==
cohort size, a cohort row without a score is a 'missing' failure); score lines for eval rows outside the cohort
are ignored, not counted as unmatched (the scorer may have run the full set); run_manifest.json records
cohort {path, size, sha256}.

report.py --cohort: reports only the cohort rows and records run.cohort. A run imported for a cohort must be
reported with the same cohort (exit 2 on a missing or different one), so a pilot run cannot pass as a full one.
Without --cohort nothing changes: no cohort key, and the report equals build_full_report on all rows.

Both scripts load cohort.py by file path because they drop their own directory from sys.path (select.py
shadows the stdlib module). Tests: tests/test_musebench_cohort.py (list/object forms, malformed files,
unknown id, cohort import+report coverage, mismatch refusal, unchanged full-cohort report).
@unohee
unohee merged commit 5854b13 into main Nov 17, 2026
1 check passed
@unohee
unohee deleted the goal/22-on-a-branch-cut-from-origin-ma branch November 17, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant