diff --git a/docs/MUSEBENCH.md b/docs/MUSEBENCH.md index 9c8dc7a..7885884 100644 --- a/docs/MUSEBENCH.md +++ b/docs/MUSEBENCH.md @@ -161,10 +161,13 @@ ArtifactBench evaluates detectors that were not trained on MuseBench. The rules: calibration) is selected on MuseBench data. 2. **Native input.** Each detector receives the audio through its own adapter (for ArtifactNet: 44.1 kHz, seven 4 s chunks, centre crop), not the paper's 10 s / 48 kHz excerpt. -3. **Eval cohort** (deterministic, sealed): all Jamendo human tracks, all OOD-FMA tracks, - up to 2,000 clean tracks per generator (including Suno and Udio), up to 2,000 co-created - tracks, and up to 200 tracks per transform × generator. Rows in the cohort are marked - `musebench_role = eval` and must never be used for training. +3. **Eval cohort** (deterministic, sealed), chosen by `scripts/musebench/select.py`: + 3,000 Jamendo human tracks, 3,000 OOD-FMA tracks, 1,000 clean tracks per generator (all + ten, including Suno and Udio), 1,000 co-created tracks, and 200 tracks per transform × + generator (8 × 10 = 80 groups). Clean AI tracks are drawn from tracks that have transformed + variants (the paper's test split), and the transform samples are the transforms of those + same tracks, so clean/transform results are paired. Rows in the cohort are marked + `musebench_role = eval` (also `protocol_split = eval`) and must never be used for training. 4. **Leak annotation.** Suno and Udio are SONICS tracks; FMA and Jamendo audio may also appear in a detector's training history. Every row carries `seen_in_training` for the detector under test, and every metric is reported for seen and unseen rows separately. @@ -183,8 +186,57 @@ acquired selectively: python scripts/musebench/index_shards.py --root ~/data/musebench --workers 75 # Assert the index is complete and internally consistent. python scripts/musebench/verify.py --root ~/data/musebench index +# Choose the eval cohort and the budgeted train pool (deterministic). +python scripts/musebench/select.py plan --root ~/data/musebench --budget-gb 170 --free-floor-gb 80 +# Download the chosen members by byte range; resumable, stops at the free-space floor. +python scripts/musebench/fetch.py --root ~/data/musebench --workers 16 --free-floor-gb 80 +# Write the runtime manifest, then check files, hashes, cohort shape and disk policy. +python scripts/musebench/select.py manifest --root ~/data/musebench +python scripts/musebench/verify.py --root ~/data/musebench acquire ``` +### Families and the train pool + +Every member is assigned a *family*: the source track it derives from (`caption_set/base +name`; Suno and Udio tracks are their own family). A Jamendo song, its co-created remix, every +generation conditioned on its caption and every transform of those generations share one +family. Selection is ordered by `sha256(seed:family)`, so the same families are preferred +across generators, and the **train pool excludes every family that has an eval member**. + +The train pool fills the remaining disk budget in priority tiers, each source up to a cap, +round-robin within a tier: + +1. generators that are new to most detectors' training data: AudioLDM2, MAGNeT, JASCO, + Mustango, Lyria RealTime (5,000 each); +2. co-created (5,000); +3. human Jamendo (3,000) and OOD-FMA (2,000); +4. MusicGen, Riffusion, Stable Audio Open (2,000 each); +5. Suno and Udio: none (they are SONICS tracks, usually available locally already). + +Transforms are never put in the train pool; all of them except PGD can be regenerated from +clean audio with the Appendix A.3 parameters. + +### Acquired corpus (2026-11-04) + +With the defaults above (`--budget-gb 170 --free-floor-gb 80`): 64,756 audio files, 169.9 GB — +33,000 eval rows in 93 sources (91.3 GB) and 31,756 train-pool rows (78.6 GB: 5,000 each of +AudioLDM2, MAGNeT, JASCO, Mustango, Lyria RealTime and co-created, 877 Jamendo, 879 +OOD-FMA; the budget ran out inside tier 3). Every file's SHA-256 was recomputed and matches the +fetch ledger. The index allows fetching more of the train pool later with a larger budget. + +### Observations from the shipped files + +- Suno and Udio transforms sit under a literal `root/` folder + (`aac_generated_music/suno_generated_music/root/…`); PGD files have no caption-set folder and + are named `adv_`; the PGD folders for MAGNeT and MusicGen are lower case. +- Transform file names end with the transform parameters (`_aac128`, `_mp3128`, `_ogg3`, + `_eq-cut_low`, `_pt-ps+2st-ts0.9`, `_lnLUFS-14`, `_gn0.0025`). The Gaussian files say + `gn0.0025`, while Appendix A.3 states σ = 0.025. +- The shipped CSVs (`Caption/captions-*.csv`, `Genius-Lyrics/metadata.csv`, + `Genius-Lyrics/segments.csv`, `Jamendo/segments.csv`) carry captions, segment descriptions and + YouTube ids only. The per-track Jamendo `license_ccurl` that the card mentions is **not** + shipped; manifest rows carry `license_ccurl: null` until it is joined from JamendoMaxCaps. + Rules for the data root (`$MUSEBENCH_ROOT`, default `~/data/musebench`): - Shard bytes and extracted audio are untrusted: they live only under the data root, scripts live @@ -200,7 +252,8 @@ Rules for the data root (`$MUSEBENCH_ROOT`, default `~/data/musebench`): - Collection: CC BY-SA 3.0 (inherited from JamendoMaxCaps). - Jamendo audio: per-track Creative Commons licenses (CC BY, BY-SA, BY-NC, BY-NC-ND), recorded - in the `license_ccurl` metadata; NC and ND tracks restrict commercial use and derivatives. + in JamendoMaxCaps' `license_ccurl` metadata (not shipped with MuseBench); NC and ND tracks + restrict commercial use and derivatives. - FMA audio: per-track Creative Commons licenses; FMA metadata CC BY 4.0. - AI audio: bound by each generator's terms; most open weights are non-commercial (MusicGen / MAGNeT / JASCO weights CC BY-NC 4.0, AudioLDM and Mustango CC BY-NC-SA 4.0, diff --git a/scripts/musebench/fetch.py b/scripts/musebench/fetch.py new file mode 100644 index 0000000..49971b3 --- /dev/null +++ b/scripts/musebench/fetch.py @@ -0,0 +1,213 @@ +#!/usr/bin/env python3 +# Created: 2026-11-04 +# Purpose: Download the members chosen by select.py from the MuseBench shards with HTTP range +# requests, byte-identical, SHA-256 recorded, atomic, resumable, with a free-space floor. +# Dependencies: standard library only. +# Test Status: tests/test_musebench_fetch.py +"""Fetch selected MuseBench members. + +Reads ``/selection/selection.json`` and writes + +* audio to ``/audio/`` and metadata CSVs to ``/meta/``, +* one ledger line per finished file to ``/selection/fetch_ledger.jsonl`` + (``path, kind, size, sha256, fetched_at``). + +Each member is read with one range request ``[data_offset, data_offset + size)``, written to a +``.part`` file, checked for length, hashed, and renamed into place. A file already in the +ledger with the right size on disk is skipped, so the command can be re-run after any +interruption. Before each download the free space of the data root's filesystem must stay at +or above ``--free-floor-gb`` after the write, otherwise the run stops. + +Member names are untrusted: each is validated by ``select.safe_relpath`` before use. +""" +from __future__ import annotations + +import os as _os +import sys as _sys + +# This directory holds select.py, which shadows the stdlib ``select`` module when the +# directory is on sys.path (``python scripts/musebench/x.py`` without -I). Drop it. +if _sys.path and _os.path.abspath(_sys.path[0] or ".") == _os.path.dirname(_os.path.abspath(__file__)): + _sys.path.pop(0) + +import argparse # noqa: E402 +import hashlib # noqa: E402 +import json # noqa: E402 +import os # noqa: E402 +import shutil # noqa: E402 +import sys # noqa: E402 +import threading # noqa: E402 +import time # noqa: E402 +from concurrent.futures import ThreadPoolExecutor, as_completed # noqa: E402 +from pathlib import Path # noqa: E402 +from typing import Callable # noqa: E402 + +REPO_ROOT = Path(__file__).resolve().parents[2] +if str(REPO_ROOT) not in sys.path: + sys.path.insert(0, str(REPO_ROOT)) + +from scripts.musebench.index_shards import RangeFetcher, resolve_url # noqa: E402 +from scripts.musebench.select import load_ledger, safe_relpath # noqa: E402 + +DEFAULT_ROOT = Path(os.environ.get("MUSEBENCH_ROOT", "~/data/musebench")).expanduser() +GB = 10 ** 9 +ROLE_ORDER = {"meta": 0, "eval": 1, "train_pool": 2} + + +class DiskFloorReached(RuntimeError): + pass + + +def work_items(selection: dict) -> list[dict]: + items = [{**m, "kind": "meta", "role": "meta"} for m in selection["meta"]] + items += [{"path": r["path"], "shard": r["shard"], "data_offset": r["data_offset"], + "size": r["size"], "kind": "audio", "role": r["musebench_role"]} + for r in selection["rows"]] + items.sort(key=lambda it: (ROLE_ORDER[it["role"]], it["shard"], it["data_offset"])) + return items + + +def dest_for(root: Path, item: dict) -> Path: + rel = safe_relpath(item["path"]) + base = root / ("meta" if item["kind"] == "meta" else "audio") + dest = (base / rel).resolve() + if base.resolve() not in dest.parents: + raise ValueError(f"destination escapes data root: {item['path']!r}") + return dest + + +class Fetcher: + """Thread-safe member downloader; one persistent RangeFetcher per (thread, shard).""" + + def __init__(self, root: Path, shards: dict[str, dict], free_floor_bytes: int, + reader_factory: Callable[[dict], Callable[[int, int], bytes]] | None = None, + disk_free: Callable[[Path], int] | None = None): + self.root = root + self.shards = shards + self.floor = free_floor_bytes + self.reader_factory = reader_factory or ( + lambda s: RangeFetcher(resolve_url(s["repo"], s["revision"], s["file"]))) + self.disk_free = disk_free or (lambda p: shutil.disk_usage(p).free) + self.local = threading.local() + self.ledger_lock = threading.Lock() + self.space_lock = threading.Lock() + self.reserved = 0 + self.ledger_path = root / "selection" / "fetch_ledger.jsonl" + self.ledger_path.parent.mkdir(parents=True, exist_ok=True) + self.stop = threading.Event() + + def _reader(self, shard_file: str): + # Work items are sorted by shard, so one open connection per thread is enough. + current = getattr(self.local, "current", None) + if current is None or current[0] != shard_file: + if current is not None and hasattr(current[1], "_close"): + current[1]._close() + current = self.local.current = (shard_file, self.reader_factory(self.shards[shard_file])) + return current[1] + + def _reserve(self, size: int) -> None: + with self.space_lock: + free = self.disk_free(self.root) - self.reserved + if free - size < self.floor: + self.stop.set() + raise DiskFloorReached( + f"free space {free / GB:.1f} GB - {size / GB:.3f} GB < floor {self.floor / GB:.0f} GB") + self.reserved += size + + def _release(self, size: int) -> None: + with self.space_lock: + self.reserved -= size + + def fetch(self, item: dict, attempts: int = 3) -> dict: + if self.stop.is_set(): + raise DiskFloorReached("stopped") + dest = dest_for(self.root, item) + self._reserve(item["size"]) + try: + for attempt in range(attempts): + data = self._reader(item["shard"])(item["data_offset"], item["size"]) \ + if item["size"] else b"" + if len(data) == item["size"]: + break + if attempt == attempts - 1: + raise OSError(f"short read for {item['path']!r}: {len(data)} of {item['size']}") + digest = hashlib.sha256(data).hexdigest() + dest.parent.mkdir(parents=True, exist_ok=True) + part = dest.with_name(dest.name + ".part") + with open(part, "wb") as f: + f.write(data) + f.flush() + os.fsync(f.fileno()) + os.replace(part, dest) + finally: + self._release(item["size"]) + entry = {"path": item["path"], "kind": item["kind"], "size": item["size"], + "sha256": digest, "fetched_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())} + with self.ledger_lock: + with open(self.ledger_path, "a", encoding="utf-8") as f: + f.write(json.dumps(entry, ensure_ascii=True) + "\n") + return entry + + +def pending(root: Path, items: list[dict]) -> list[dict]: + """Items not yet in the ledger with a correctly sized file on disk.""" + ledger = load_ledger(root) + out = [] + for it in items: + entry = ledger.get(it["path"]) + dest = dest_for(root, it) + if entry and entry["size"] == it["size"] and dest.exists() \ + and dest.stat().st_size == it["size"]: + continue + out.append(it) + return out + + +def run(root: Path, selection: dict, shards: dict[str, dict], workers: int, + free_floor_bytes: int, **fetcher_kw) -> tuple[int, int, list[str]]: + items = pending(root, work_items(selection)) + fetcher = Fetcher(root, shards, free_floor_bytes, **fetcher_kw) + done, failed = 0, [] + t0, last = time.time(), 0.0 + nbytes = 0 + with ThreadPoolExecutor(max_workers=max(1, workers)) as pool: + futures = {pool.submit(fetcher.fetch, it): it for it in items} + for fut in as_completed(futures): + it = futures[fut] + try: + fut.result() + done += 1 + nbytes += it["size"] + except DiskFloorReached as exc: + failed.append(f"{it['path']}: {exc}") + except Exception as exc: # noqa: BLE001 - keep going, report at the end + failed.append(f"{it['path']}: {exc!r}") + now = time.time() + if now - last >= 60: + last = now + rate = nbytes / max(now - t0, 1e-9) / 1e6 + print(f"[{time.strftime('%H:%M:%S')}] {done}/{len(items)} files, " + f"{nbytes / GB:.1f} GB, {rate:.0f} MB/s, {len(failed)} failed", + file=sys.stderr, flush=True) + return len(items), done, failed + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description="Fetch selected MuseBench members by range.") + ap.add_argument("--root", type=Path, default=DEFAULT_ROOT) + ap.add_argument("--workers", type=int, default=16) + ap.add_argument("--free-floor-gb", type=float, default=80.0) + args = ap.parse_args(argv) + selection = json.loads((args.root / "selection" / "selection.json").read_text(encoding="utf-8")) + shards = {s["file"]: s for s in + json.loads((args.root / "index" / "shards.json").read_text(encoding="utf-8"))} + total, done, failed = run(args.root, selection, shards, args.workers, + int(args.free_floor_gb * GB)) + print(f"fetched {done}/{total} pending files; {len(failed)} failed", file=sys.stderr) + for line in failed[:20]: + print(f" {line}", file=sys.stderr) + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/musebench/index_shards.py b/scripts/musebench/index_shards.py index 3960ca8..0f372a4 100644 --- a/scripts/musebench/index_shards.py +++ b/scripts/musebench/index_shards.py @@ -26,6 +26,15 @@ """ from __future__ import annotations +import os as _os +import sys as _sys + +# This directory holds select.py, which shadows the stdlib ``select`` module when the +# directory is on sys.path (``python scripts/musebench/x.py`` without -I). Drop it. +if _sys.path and _os.path.abspath(_sys.path[0] or ".") == _os.path.dirname(_os.path.abspath(__file__)): + _sys.path.pop(0) + + import argparse import http.client import json diff --git a/scripts/musebench/inventory.py b/scripts/musebench/inventory.py index 91d7092..6ca164b 100644 --- a/scripts/musebench/inventory.py +++ b/scripts/musebench/inventory.py @@ -10,6 +10,15 @@ """ from __future__ import annotations +import os as _os +import sys as _sys + +# This directory holds select.py, which shadows the stdlib ``select`` module when the +# directory is on sys.path (``python scripts/musebench/x.py`` without -I). Drop it. +if _sys.path and _os.path.abspath(_sys.path[0] or ".") == _os.path.dirname(_os.path.abspath(__file__)): + _sys.path.pop(0) + + import argparse import json import os diff --git a/scripts/musebench/select.py b/scripts/musebench/select.py new file mode 100644 index 0000000..2a576e2 --- /dev/null +++ b/scripts/musebench/select.py @@ -0,0 +1,471 @@ +#!/usr/bin/env python3 +# Created: 2026-11-04 +# Purpose: Classify every MuseBench shard member, choose a deterministic sealed eval cohort and a +# disk-budgeted train pool from the shard index, and (after fetch.py) write the runtime +# manifest that artifactbench.data.manifest.load_manifest accepts. +# Dependencies: standard library only. +# Test Status: tests/test_musebench_select.py +"""Select MuseBench members and build the runtime manifest. + +Subcommands:: + + python -I scripts/musebench/select.py plan # index -> selection/selection.json + python -I scripts/musebench/select.py manifest # selection + fetch ledger -> manifest + +Determinism: every choice is ordered by ``sha256(seed:family)``, where *family* is the source +track a member derives from (``caption_set/base name``). A Jamendo song, its co-created remix, +every AI generation conditioned on its caption and every transform of those generations share +one family, so + +* the eval cohort prefers the same families across generators and transforms (paired + clean/transform and human/co-created comparisons), and +* the train pool excludes every family that has any eval member (no family straddles roles). + +Clean AI eval tracks are drawn from the tracks that have transformed variants: MuseBench +applied transforms to its test split only, so this approximates the paper's test split. +""" +from __future__ import annotations + +import os as _os +import sys as _sys + +# This directory holds select.py, which shadows the stdlib ``select`` module when the +# directory is on sys.path (``python scripts/musebench/x.py`` without -I). Drop it. +if _sys.path and _os.path.abspath(_sys.path[0] or ".") == _os.path.dirname(_os.path.abspath(__file__)): + _sys.path.pop(0) + + +import argparse +import hashlib +import json +import os +import re +import shutil +import sys +import time +from collections import Counter, defaultdict +from pathlib import Path, PurePosixPath + +DEFAULT_ROOT = Path(os.environ.get("MUSEBENCH_ROOT", "~/data/musebench")).expanduser() +SEED = "musebench-20261104" +GB = 10 ** 9 + +GENERATORS = { # folder name (lower case) -> canonical generator id + "magnet_generated_music": "magnet", + "musicgen_generated_music": "musicgen", + "audioldm_generated_music": "audioldm2", + "jasco_generated_music": "jasco", + # The card calls it "Magenta"; its 19,903 files match the paper's Lyria RealTime row. + "magenta_generated_music": "lyria_rt", + "mustango_generated_music": "mustango", + "riffusion_generated_music": "riffusion", + "stable_audio_generated_music": "stable_audio_open", + "suno_generated_music": "suno", + "udio_generated_music": "udio", +} +TRANSFORMS = { + "aac_generated_music": "aac", + "mp3_noised_music": "mp3", + "ogg_noised_music": "ogg", + "eq_generated_music": "eq", + "pitch_time_generated_music": "pitch_time", + "loudness_generation_music": "loudness", + "gaussion_noised_music": "gaussian", + "pgd_generated_music": "pgd", +} +TRANSFORM_SUFFIX = { + "aac": re.compile(r"_(aac\d+)$"), + "mp3": re.compile(r"_(mp3\d+)$"), + "ogg": re.compile(r"_(ogg\d+)$"), + "eq": re.compile(r"_(eq-(?:cut|boost)_(?:low|mid|high))$"), + "pitch_time": re.compile(r"_(pt-ps[+-]\d+st-ts[\d.]+)$"), + "loudness": re.compile(r"_(lnLUFS-?\d+)$"), + "gaussian": re.compile(r"_(gn[\d.]+)$"), +} +CAPTION_SETS = {"genius-lyrics": "genius", "jamedo": "jamendo", "jamendo": "jamendo"} +META_PREFIXES = ("Caption/", "Genius-Lyrics/") +AUDIO_EXT = (".wav", ".mp3", ".flac", ".ogg", ".m4a") + +# Eval cohort caps (per source group). See docs/MUSEBENCH.md for the rationale. +EVAL_CAPS = {"human_jamendo": 3000, "human_fma_ood": 3000, "clean_per_generator": 1000, + "co_created": 1000, "transform_per_generator": 200} +# Train pool: tiers filled in order, each source up to its cap, until the byte budget binds. +TRAIN_TIERS = [ + {"audioldm2": 5000, "magnet": 5000, "jasco": 5000, "mustango": 5000, "lyria_rt": 5000}, + {"co_created": 5000}, + {"human_jamendo": 3000, "human_fma_ood": 2000}, + {"musicgen": 2000, "riffusion": 2000, "stable_audio_open": 2000}, + {"suno": 0, "udio": 0}, # already on local disk as SONICS +] + + +class UnsafePath(ValueError): + pass + + +# --------------------------------------------------------------------------- classification + +def strip_audio_ext(name: str) -> str: + for ext in (".mp3.wav", ".wav", ".mp3", ".m4a", ".ogg", ".flac"): + if name.lower().endswith(ext): + return name[:-len(ext)] + return name + + +def classify(path: str) -> dict | None: + """Return classification fields for a member path, or None if it is not recognised.""" + parts = path.split("/") + name = parts[-1] + top = parts[0] + if path.endswith(".csv") and (path.startswith(META_PREFIXES) or top == "Jamendo"): + return {"kind": "meta"} + if not name.lower().endswith(AUDIO_EXT): + return None + if top == "Jamendo" and len(parts) == 2: + base = strip_audio_ext(name) + return _row("human", "human_jamendo", "real", "human", "clean", "jamendo", base) + if top == "OOD FMA" and len(parts) == 2: + base = strip_audio_ext(name) + return _row("human", "human_fma_ood", "real", "human", "clean", "fma", base) + if top == "ai_human_co-created_music" and len(parts) == 3: + base = strip_audio_ext(name) + base = base[:-len("_edited")] if base.endswith("_edited") else base + cset = CAPTION_SETS.get(parts[1].lower()) + if cset is None: + return None + return _row("ai", "co_created", "ai", "musicgen_style", "co_created", cset, base) + gen = GENERATORS.get(top.lower()) + if gen is not None: + if gen in ("suno", "udio") and len(parts) == 2: + return _row("ai", gen, "ai", gen, "clean", "none", strip_audio_ext(name)) + if len(parts) == 3 and parts[1].lower() in CAPTION_SETS: + return _row("ai", gen, "ai", gen, "clean", CAPTION_SETS[parts[1].lower()], + strip_audio_ext(name)) + return None + variant = TRANSFORMS.get(top) + if variant is None or len(parts) < 3: + return None + gen = GENERATORS.get(parts[1].lower()) + if gen is None: + return None + if len(parts) == 4 and parts[2].lower() in CAPTION_SETS: + cset = CAPTION_SETS[parts[2].lower()] + elif len(parts) == 4 and parts[2] == "root" and gen in ("suno", "udio"): + cset = "none" # standalone generators: transforms sit under a literal "root/" folder + elif len(parts) == 3: + cset = "none" if gen in ("suno", "udio") else "*" # pgd has no caption-set level + else: + return None + stem = strip_audio_ext(name) + if variant == "pgd": + param = "pgd" + base = stem[len("adv_"):] if stem.startswith("adv_") else stem + else: + m = TRANSFORM_SUFFIX[variant].search(stem) + if m is None: + return None + param, base = m.group(1), stem[:m.start()] + row = _row("ai", f"{gen}__{variant}", "ai", gen, variant, cset, base) + row["transform_param"] = param + return row + + +def _row(kind, group, label, generator, variant, caption_set, base): + return {"kind": kind, "group": group, "label": label, "generator": generator, + "variant": variant, "caption_set": caption_set, "base_key": base} + + +def source_name(group: str) -> str: + return f"musebench_{group}" + + +def rank(seed: str, key: str) -> str: + return hashlib.sha256(f"{seed}:{key}".encode("utf-8", "surrogateescape")).hexdigest() + + +def family_of(row: dict) -> str: + """Source-track identity shared by a song, its remix, generations and their transforms.""" + cset = row["caption_set"] + if row["generator"] in ("suno", "udio"): + return f"{row['generator']}/{row['base_key']}" # standalone tracks: own family + if cset == "fma": + return f"fma/{row['base_key']}" + return f"{cset}/{row['base_key']}" + + +def safe_relpath(path: str) -> str: + """Reject member names that could escape the data root.""" + if not path or "\0" in path or path.startswith("/") or "\\" in path: + raise UnsafePath(path) + parts = PurePosixPath(path).parts + if any(p in ("", ".", "..") for p in parts) or path.split("/") != list(parts): + raise UnsafePath(path) + return path + + +# --------------------------------------------------------------------------- planning + +def load_index(root: Path) -> list[dict]: + recs = [] + for jsonl in sorted((root / "index").glob("*.jsonl")): + with open(jsonl, encoding="utf-8") as f: + recs.extend(json.loads(line) for line in f if line.strip()) + return recs + + +def classify_index(records: list[dict]) -> tuple[list[dict], list[dict], list[str]]: + """Split index records into (audio rows, meta records, unclassified paths).""" + audio, meta, unknown = [], [], [] + for rec in records: + if rec.get("type") != "file": + continue + cls = classify(rec["path"]) + if cls is None: + unknown.append(rec["path"]) + continue + if cls["kind"] == "meta": + meta.append(rec) + continue + row = {**cls, "path": rec["path"], "shard": rec["shard"], + "data_offset": rec["data_offset"], "size": rec["size"]} + row["family"] = family_of(row) + audio.append(row) + return audio, meta, unknown + + +def _resolve_pgd_caption_sets(rows: list[dict]) -> None: + """PGD files lack the caption-set folder; borrow it from the clean track with that base.""" + clean_sets = defaultdict(set) + for r in rows: + if r["variant"] == "clean" and r["kind"] == "ai": + clean_sets[(r["generator"], r["base_key"])].add(r["caption_set"]) + for r in rows: + if r["caption_set"] == "*": + sets = clean_sets.get((r["generator"], r["base_key"]), set()) + r["caption_set"] = sets.pop() if len(sets) == 1 else "unknown" + r["family"] = family_of(r) + + +def plan(records: list[dict], seed: str = SEED, eval_caps: dict = EVAL_CAPS, + train_tiers: list[dict] = TRAIN_TIERS, train_budget_bytes: int = 85 * GB) -> dict: + audio, meta, unknown = classify_index(records) + _resolve_pgd_caption_sets(audio) + fam_rank = {r["family"]: rank(seed, r["family"]) for r in audio} + + def by_rank(rows): + return sorted(rows, key=lambda r: (fam_rank[r["family"]], r["path"])) + + groups = defaultdict(list) + for r in audio: + groups[r["group"]].append(r) + + eval_rows: list[dict] = [] + expected: dict[str, int] = {} + + def take(group: str, rows: list[dict], cap: int) -> list[dict]: + chosen = by_rank(rows)[:cap] + expected[group] = len(chosen) + eval_rows.extend(chosen) + return chosen + + take("human_jamendo", groups["human_jamendo"], eval_caps["human_jamendo"]) + take("human_fma_ood", groups["human_fma_ood"], eval_caps["human_fma_ood"]) + take("co_created", groups["co_created"], eval_caps["co_created"]) + + # Families that have transformed variants per generator ~ the paper's test split. + transformed = defaultdict(set) + for r in audio: + if r["variant"] not in ("clean", "co_created"): + transformed[r["generator"]].add(r["family"]) + clean_eval_families: dict[str, set] = {} + for gen in sorted(set(GENERATORS.values())): + rows = groups.get(gen, []) + test_like = [r for r in rows if r["family"] in transformed[gen]] + rest = [r for r in rows if r["family"] not in transformed[gen]] + cap = eval_caps["clean_per_generator"] + chosen = by_rank(test_like)[:cap] + if len(chosen) < cap: + chosen += by_rank(rest)[:cap - len(chosen)] + expected[gen] = len(chosen) + eval_rows.extend(chosen) + clean_eval_families[gen] = {r["family"] for r in chosen} + + for group in sorted(g for g in groups if "__" in g): + gen = group.split("__")[0] + rows = groups[group] + paired = [r for r in rows if r["family"] in clean_eval_families.get(gen, set())] + unpaired = [r for r in rows if r["family"] not in clean_eval_families.get(gen, set())] + cap = eval_caps["transform_per_generator"] + chosen = by_rank(paired)[:cap] + if len(chosen) < cap: + chosen += by_rank(unpaired)[:cap - len(chosen)] + expected[group] = len(chosen) + eval_rows.extend(chosen) + + eval_families = {r["family"] for r in eval_rows} + eval_paths = {r["path"] for r in eval_rows} + + train_rows: list[dict] = [] + budget_left = train_budget_bytes + for tier in train_tiers: + queues = {} + for group, cap in tier.items(): + pool = [r for r in groups.get(group, []) + if r["family"] not in eval_families and r["path"] not in eval_paths] + queues[group] = by_rank(pool)[:cap] + # Round-robin across the tier so a binding budget is shared evenly. + pos = {g: 0 for g in queues} + active = [g for g in sorted(queues) if queues[g]] + while active: + nxt = [] + for g in active: + r = queues[g][pos[g]] + pos[g] += 1 + if r["size"] <= budget_left: + train_rows.append(r) + budget_left -= r["size"] + if pos[g] < len(queues[g]): + nxt.append(g) + active = nxt + + for r in eval_rows: + r["musebench_role"] = "eval" + for r in train_rows: + r["musebench_role"] = "train_pool" + return { + "seed": seed, + "eval_caps": eval_caps, + "train_tiers": train_tiers, + "train_budget_bytes": train_budget_bytes, + "expected_eval_counts": dict(sorted(expected.items())), + "unclassified": unknown, + "meta": [{"path": m["path"], "shard": m["shard"], "data_offset": m["data_offset"], + "size": m["size"]} for m in meta], + "rows": eval_rows + train_rows, + "bytes": {"eval": sum(r["size"] for r in eval_rows), + "train_pool": sum(r["size"] for r in train_rows), + "meta": sum(m["size"] for m in meta)}, + } + + +def selection_fingerprint(selection: dict) -> str: + h = hashlib.sha256() + for r in sorted(selection["rows"], key=lambda r: r["path"]): + h.update(f"{r['path']}\t{r['musebench_role']}\n".encode("utf-8", "surrogateescape")) + return h.hexdigest() + + +# --------------------------------------------------------------------------- manifest + +def track_id(path: str) -> str: + return "mb_" + hashlib.sha1(path.encode("utf-8", "surrogateescape")).hexdigest()[:16] + + +def load_ledger(root: Path) -> dict[str, dict]: + ledger = {} + path = root / "selection" / "fetch_ledger.jsonl" + if path.exists(): + with open(path, encoding="utf-8") as f: + for line in f: + if not line.strip(): + continue + try: + entry = json.loads(line) + except json.JSONDecodeError: + continue # torn line from an interrupted run: that file is fetched again + ledger[entry["path"]] = entry # last entry wins + return ledger + + +def build_manifest(selection: dict, ledger: dict[str, dict], root: Path) -> dict: + tracks, missing = [], [] + for r in selection["rows"]: + entry = ledger.get(r["path"]) + if entry is None or entry.get("size") != r["size"]: + missing.append(r["path"]) + continue + tracks.append({ + "track_id": track_id(r["path"]), + "path": str(root / "audio" / r["path"]), + "label": r["label"], + "source": source_name(r["group"]), + "generator": r["generator"], + "variant": r["variant"], + "caption_set": r["caption_set"], + "musebench_role": r["musebench_role"], + "protocol_split": r["musebench_role"], + "musebench_path": r["path"], + "family": r["family"], + "base_key": r["base_key"], + "transform_param": r.get("transform_param"), + "size": r["size"], + "sha256": entry["sha256"], + "shard": r["shard"], + "data_offset": r["data_offset"], + # The shipped CSVs carry captions only; per-track Jamendo licenses are not included. + "license_ccurl": None, + "license_source": "not shipped in MuseBench; see JamendoMaxCaps / FMA metadata", + }) + return { + "tracks": tracks, + "metadata": { + "dataset": "MuseBench (arXiv:2610.20310)", + "created_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + "seed": selection["seed"], + "selection_fingerprint": selection_fingerprint(selection), + "n_tracks": len(tracks), + "n_missing": len(missing), + "roles": dict(Counter(t["musebench_role"] for t in tracks)), + }, + }, missing + + +def atomic_write_json(path: Path, obj: object) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(path.suffix + ".tmp") + tmp.write_text(json.dumps(obj, ensure_ascii=True) + "\n", encoding="utf-8") + os.replace(tmp, path) + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description="Select MuseBench members / build the manifest.") + ap.add_argument("command", choices=["plan", "manifest"]) + ap.add_argument("--root", type=Path, default=DEFAULT_ROOT) + ap.add_argument("--budget-gb", type=float, default=170.0, + help="cap on all kept audio (eval + train pool), in GB") + ap.add_argument("--free-floor-gb", type=float, default=80.0, + help="free space that must remain on the data root's filesystem") + args = ap.parse_args(argv) + sel_path = args.root / "selection" / "selection.json" + + if args.command == "plan": + records = load_index(args.root) + probe = plan(records, train_budget_bytes=0) + eval_bytes = probe["bytes"]["eval"] + probe["bytes"]["meta"] + have = sum(e["size"] for e in load_ledger(args.root).values()) + free_now = shutil.disk_usage(args.root).free + cap_total = int(args.budget_gb * GB) + cap_free = free_now + have - int(args.free_floor_gb * GB) + train_budget = max(0, min(cap_total, cap_free) - eval_bytes) + selection = plan(records, train_budget_bytes=train_budget) + selection["created_at"] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + selection["budget"] = {"budget_gb": args.budget_gb, "free_floor_gb": args.free_floor_gb, + "free_at_plan": free_now} + atomic_write_json(sel_path, selection) + counts = Counter((r["musebench_role"], r["group"]) for r in selection["rows"]) + print(f"eval {selection['bytes']['eval'] / GB:.1f} GB, train_pool " + f"{selection['bytes']['train_pool'] / GB:.1f} GB, unclassified " + f"{len(selection['unclassified'])}") + for (role, group), n in sorted(counts.items()): + print(f" {role:10s} {group:28s} {n}") + return 1 if selection["unclassified"] else 0 + + selection = json.loads(sel_path.read_text(encoding="utf-8")) + manifest, missing = build_manifest(selection, load_ledger(args.root), args.root) + atomic_write_json(args.root / "manifests" / "musebench_runtime_manifest.json", manifest) + print(f"manifest: {manifest['metadata']['n_tracks']} tracks, {len(missing)} missing") + return 1 if missing else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/musebench/verify.py b/scripts/musebench/verify.py index 767cf52..f2e7f00 100644 --- a/scripts/musebench/verify.py +++ b/scripts/musebench/verify.py @@ -14,9 +14,20 @@ """ from __future__ import annotations +import os as _os +import sys as _sys + +# This directory holds select.py, which shadows the stdlib ``select`` module when the +# directory is on sys.path (``python scripts/musebench/x.py`` without -I). Drop it. +if _sys.path and _os.path.abspath(_sys.path[0] or ".") == _os.path.dirname(_os.path.abspath(__file__)): + _sys.path.pop(0) + + import argparse +import hashlib import json import os +import shutil import sys from collections import Counter from pathlib import Path @@ -114,6 +125,124 @@ def verify_index(root: Path, expected_parts=EXPECTED_PARTS, return problems, stats +REQUIRED_ROW_FIELDS = ("track_id", "path", "label", "source", "generator", "variant", + "caption_set", "musebench_role") +MANIFEST_REL = Path("manifests") / "musebench_runtime_manifest.json" + + +def sha256_file(path: Path) -> str: + h = hashlib.sha256() + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +def verify_acquire(root: Path, sha_sample: int = 300, full_sha: bool = False, + free_floor_gb: float | None = None) -> tuple[list[str], dict]: + """Check selection determinism, fetched files, and the runtime manifest.""" + from artifactbench.data.manifest import load_manifest + from scripts.musebench import select as sel_mod + + problems: list[str] = [] + sel_path = root / "selection" / "selection.json" + man_path = root / MANIFEST_REL + for p in (sel_path, man_path): + if not p.exists(): + problems.append(f"missing {p}") + if problems: + return problems, {} + selection = json.loads(sel_path.read_text(encoding="utf-8")) + if selection.get("unclassified"): + problems.append(f"{len(selection['unclassified'])} unclassified index members") + + # 1. The selection is reproducible from the index and its recorded parameters. + replanned = sel_mod.plan(sel_mod.load_index(root), seed=selection["seed"], + eval_caps=selection["eval_caps"], + train_tiers=selection["train_tiers"], + train_budget_bytes=selection["train_budget_bytes"]) + if sel_mod.selection_fingerprint(replanned) != sel_mod.selection_fingerprint(selection): + problems.append("selection.json is not reproducible from the index (fingerprint differs)") + + # 2. The manifest loads through the runner's loader and covers the selection exactly. + entries, by_source = load_manifest(str(man_path), split="all") + manifest = json.loads(man_path.read_text(encoding="utf-8")) + if manifest.get("metadata", {}).get("selection_fingerprint") != sel_mod.selection_fingerprint(selection): + problems.append("manifest was built from a different selection") + rows = {r["path"]: r for r in selection["rows"]} + seen_ids: set[str] = set() + man_paths = set() + for e in entries: + missing = [k for k in REQUIRED_ROW_FIELDS if e.get(k) in (None, "")] + if missing: + problems.append(f"{e.get('track_id')}: missing fields {missing}") + continue + if e["track_id"] in seen_ids: + problems.append(f"duplicate track_id {e['track_id']}") + seen_ids.add(e["track_id"]) + if e["label"] not in ("ai", "real") or e["musebench_role"] not in ("eval", "train_pool"): + problems.append(f"{e['track_id']}: bad label/role {e['label']}/{e['musebench_role']}") + man_paths.add(e["musebench_path"]) + r = rows.get(e["musebench_path"]) + if r is None: + problems.append(f"manifest row not in selection: {e['musebench_path']!r}") + continue + if r["musebench_role"] != e["musebench_role"]: + problems.append(f"{e['track_id']}: role differs from selection") + p = Path(e["path"]) + if not p.is_file(): + problems.append(f"missing audio {p}") + elif p.stat().st_size != r["size"]: + problems.append(f"size mismatch {p}: {p.stat().st_size} != index {r['size']}") + absent = set(rows) - man_paths + if absent: + problems.append(f"{len(absent)} selected members are not in the manifest") + + # 3. Content: SHA-256 of a deterministic sample (or all) equals the ledger/manifest value. + ledger = sel_mod.load_ledger(root) + ordered = sorted(entries, key=lambda e: e["track_id"]) + sample = ordered if full_sha else ordered[::max(1, len(ordered) // max(1, sha_sample))] + for e in sample: + p = Path(e["path"]) + if not p.is_file(): + continue + digest = sha256_file(p) + led = ledger.get(e["musebench_path"], {}).get("sha256") + if digest != e.get("sha256") or digest != led: + problems.append(f"sha256 mismatch {p}") + for m in selection["meta"]: + mp = root / "meta" / m["path"] + if not mp.is_file() or mp.stat().st_size != m["size"]: + problems.append(f"missing or wrong-size metadata file {mp}") + + # 4. Cohort shape: eval counts as planned, roles disjoint by family. + eval_counts = Counter(r["group"] for r in selection["rows"] if r["musebench_role"] == "eval") + expected = {k: v for k, v in selection["expected_eval_counts"].items() if v} + if dict(eval_counts) != expected: + problems.append("eval counts differ from expected_eval_counts") + eval_fams = {r["family"] for r in selection["rows"] if r["musebench_role"] == "eval"} + straddle = [r["path"] for r in selection["rows"] + if r["musebench_role"] == "train_pool" and r["family"] in eval_fams] + if straddle: + problems.append(f"{len(straddle)} train_pool rows share a family with eval rows") + + # 5. Disk policy. + budget = selection.get("budget", {}) + floor = budget.get("free_floor_gb", 80.0) if free_floor_gb is None else free_floor_gb + free = shutil.disk_usage(root).free + if free < floor * 1e9: + problems.append(f"free space {free / 1e9:.1f} GB below floor {floor} GB") + kept = sum(r["size"] for r in selection["rows"]) + if "budget_gb" in budget and kept > budget["budget_gb"] * 1e9: + problems.append(f"kept audio {kept / 1e9:.1f} GB exceeds budget {budget['budget_gb']} GB") + + stats = {"tracks": len(entries), "sources": len(by_source), + "roles": dict(Counter(e.get("musebench_role") for e in entries)), + "kept_gb": round(kept / 1e9, 1), "free_gb": round(free / 1e9, 1), + "sha_checked": len(sample)} + return problems, stats + + def not_implemented(stage: str): def check(root: Path) -> tuple[list[str], dict]: return [f"stage '{stage}' has no verifier yet (later milestone)"], {} @@ -122,7 +251,7 @@ def check(root: Path) -> tuple[list[str], dict]: STAGES = { "index": verify_index, - "acquire": not_implemented("acquire"), + "acquire": verify_acquire, "leak": not_implemented("leak"), "run": not_implemented("run"), } diff --git a/tests/test_musebench_fetch.py b/tests/test_musebench_fetch.py new file mode 100644 index 0000000..84eef05 --- /dev/null +++ b/tests/test_musebench_fetch.py @@ -0,0 +1,171 @@ +# Created: 2026-11-04 +# Purpose: Cover scripts/musebench/fetch.py (byte-identical range fetch, ledger, resume, +# free-space floor, unsafe names) and verify.py's acquire stage end to end on a +# synthetic shard. +# Dependencies: pytest +# Test Status: passing +import hashlib +import io +import json +import tarfile + +import pytest + +from scripts.musebench import fetch as fx +from scripts.musebench import index_shards as ix +from scripts.musebench import select as sl +from scripts.musebench import verify as vf + +CAPS = {"human_jamendo": 2, "human_fma_ood": 2, "clean_per_generator": 2, "co_created": 2, + "transform_per_generator": 1} +TIERS = [{"magnet": 3}, {"human_jamendo": 1}] + + +def build_root(tmp_path): + names = ["Caption/captions-jamendo.csv"] + for i in range(6): + names += [f"Jamendo/s{i} - a.mp3", f"OOD FMA/{i:05d}.mp3", + f"ai_human_co-created_music/Jamendo/s{i} - a_edited.wav", + f"MagNet_Generated_Music/Jamedo/s{i} - a.mp3.wav"] + if i < 3: + names.append(f"aac_generated_music/MagNet_Generated_Music/Jamedo/s{i} - a_aac128.m4a") + buf = io.BytesIO() + with tarfile.open(fileobj=buf, mode="w", format=tarfile.GNU_FORMAT) as tf: + for n in names: + data = hashlib.sha256(n.encode()).digest() * (5 + len(n) % 7) + info = tarfile.TarInfo(n) + info.size = len(data) + tf.addfile(info, io.BytesIO(data)) + blob = buf.getvalue() + shard = {"part": 1, "repo": "r", "revision": "x", "file": "part1_001.tar", + "size": len(blob), "lfs_sha256": None} + (tmp_path / "index").mkdir(parents=True) + ix.index_shard(shard, tmp_path / "index", lambda o, n: blob[o:o + n]) + (tmp_path / "index" / "shards.json").write_text(json.dumps([shard])) + sel = sl.plan(sl.load_index(tmp_path), eval_caps=CAPS, train_tiers=TIERS, + train_budget_bytes=10 ** 6) + sel["budget"] = {"budget_gb": 1.0, "free_floor_gb": 0.0} + sl.atomic_write_json(tmp_path / "selection" / "selection.json", sel) + return blob, {"part1_001.tar": shard}, sel + + +def reader_for(blob, calls=None): + def factory(shard): + def read(offset, length): + if calls is not None: + calls.append(offset) + return blob[offset:offset + length] + return read + return factory + + +def test_fetch_writes_identical_bytes_and_ledger(tmp_path): + blob, shards, sel = build_root(tmp_path) + total, done, failed = fx.run(tmp_path, sel, shards, 3, 0, reader_factory=reader_for(blob)) + assert failed == [] and done == total == len(sel["rows"]) + len(sel["meta"]) + ledger = sl.load_ledger(tmp_path) + for r in sel["rows"]: + got = (tmp_path / "audio" / r["path"]).read_bytes() + assert got == blob[r["data_offset"]:r["data_offset"] + r["size"]] + assert ledger[r["path"]]["sha256"] == hashlib.sha256(got).hexdigest() + assert (tmp_path / "meta" / "Caption" / "captions-jamendo.csv").is_file() + assert not list(tmp_path.rglob("*.part")) + + +def test_fetch_resume_skips_finished_files(tmp_path): + blob, shards, sel = build_root(tmp_path) + fx.run(tmp_path, sel, shards, 2, 0, reader_factory=reader_for(blob)) + victim = sel["rows"][0] + (tmp_path / "audio" / victim["path"]).write_bytes(b"truncated") + calls = [] + total, done, failed = fx.run(tmp_path, sel, shards, 2, 0, reader_factory=reader_for(blob, calls)) + assert (total, done, failed) == (1, 1, []) + assert calls == [victim["data_offset"]] + + +def test_free_space_floor_stops_the_run(tmp_path): + blob, shards, sel = build_root(tmp_path) + def disk_free(_): + written = sum(f.stat().st_size for f in tmp_path.rglob("*") if f.is_file() + and ("audio" in f.parts or "meta" in f.parts)) + return 10_000 - written + + total, done, failed = fx.run(tmp_path, sel, shards, 1, 9_000, reader_factory=reader_for(blob), + disk_free=disk_free) + assert failed and all("floor" in f or "stopped" in f for f in failed) + assert done < total + + +def test_short_read_is_an_error_and_leaves_no_file(tmp_path): + blob, shards, sel = build_root(tmp_path) + item = fx.work_items(sel)[1] + f = fx.Fetcher(tmp_path, shards, 0, reader_factory=lambda s: (lambda o, n: blob[o:o + n - 1])) + with pytest.raises(OSError, match="short read"): + f.fetch(item) + assert not fx.dest_for(tmp_path, item).exists() + + +def test_unsafe_member_name_is_refused(tmp_path): + with pytest.raises(sl.UnsafePath): + fx.dest_for(tmp_path, {"path": "../../etc/x.wav", "kind": "audio"}) + + +# ----------------------------------------------------------------------- verify acquire + +def fetched_root(tmp_path): + blob, shards, sel = build_root(tmp_path) + fx.run(tmp_path, sel, shards, 2, 0, reader_factory=reader_for(blob)) + man, missing = sl.build_manifest(sel, sl.load_ledger(tmp_path), tmp_path) + assert missing == [] + sl.atomic_write_json(tmp_path / vf.MANIFEST_REL, man) + return sel, man + + +def test_verify_acquire_passes_on_complete_fetch(tmp_path): + sel, man = fetched_root(tmp_path) + problems, stats = vf.verify_acquire(tmp_path, full_sha=True, free_floor_gb=0) + assert problems == [] + assert stats["tracks"] == len(sel["rows"]) and stats["sha_checked"] == len(sel["rows"]) + + +def test_verify_acquire_detects_corruption_and_gaps(tmp_path): + sel, man = fetched_root(tmp_path) + first = man["tracks"][0] + p = tmp_path / "audio" / first["musebench_path"] + p.write_bytes(bytes(b ^ 1 for b in p.read_bytes())) # same size, different content + man["tracks"] = man["tracks"][1:] + [dict(man["tracks"][1], track_id="mb_dup")] + sl.atomic_write_json(tmp_path / vf.MANIFEST_REL, man) + problems, _ = vf.verify_acquire(tmp_path, full_sha=True, free_floor_gb=0) + text = "\n".join(problems) + assert "not in the manifest" in text # the dropped first row + # Corrupt content is caught on rows still listed; restore row 0 to check sha path. + man["tracks"] = [first] + man["tracks"][:-1] + sl.atomic_write_json(tmp_path / vf.MANIFEST_REL, man) + problems, _ = vf.verify_acquire(tmp_path, full_sha=True, free_floor_gb=0) + assert any("sha256 mismatch" in x for x in problems) + + +def test_verify_acquire_detects_non_reproducible_selection(tmp_path): + sel, _ = fetched_root(tmp_path) + sel["seed"] = "tampered" + sl.atomic_write_json(tmp_path / "selection" / "selection.json", sel) + problems, _ = vf.verify_acquire(tmp_path, free_floor_gb=0) + assert any("not reproducible" in x for x in problems) + + +def test_verify_acquire_enforces_free_space_floor(tmp_path): + fetched_root(tmp_path) + problems, _ = vf.verify_acquire(tmp_path, free_floor_gb=10 ** 9) + assert any("below floor" in x for x in problems) + + +def test_torn_ledger_line_is_ignored_and_refetched(tmp_path): + blob, shards, sel = build_root(tmp_path) + fx.run(tmp_path, sel, shards, 2, 0, reader_factory=reader_for(blob)) + ledger_path = tmp_path / "selection" / "fetch_ledger.jsonl" + lines = ledger_path.read_text().splitlines() + victim = json.loads(lines[-1])["path"] + ledger_path.write_text("\n".join(lines[:-1]) + "\n" + lines[-1][:20]) + assert victim not in sl.load_ledger(tmp_path) + total, done, failed = fx.run(tmp_path, sel, shards, 2, 0, reader_factory=reader_for(blob)) + assert (total, done, failed) == (1, 1, []) diff --git a/tests/test_musebench_select.py b/tests/test_musebench_select.py new file mode 100644 index 0000000..7cbcca3 --- /dev/null +++ b/tests/test_musebench_select.py @@ -0,0 +1,183 @@ +# Created: 2026-11-04 +# Purpose: Cover scripts/musebench/select.py: path classification, deterministic family-level +# eval/train selection with a byte budget, and manifest construction. +# Dependencies: pytest +# Test Status: passing +import json + +import pytest + +from scripts.musebench import select as sl + + +@pytest.mark.parametrize("path, expect", [ + ("Jamendo/#charles - Romain ROMERO.mp3", + dict(group="human_jamendo", label="real", variant="clean", caption_set="jamendo", + base_key="#charles - Romain ROMERO")), + ("OOD FMA/10000_Vena_Cava_50_Foot_Wave.mp3", + dict(group="human_fma_ood", label="real", caption_set="fma")), + ("ai_human_co-created_music/Jamendo/#charles - Romain ROMERO_edited.wav", + dict(group="co_created", label="ai", variant="co_created", base_key="#charles - Romain ROMERO")), + ("MagNet_Generated_Music/Genius-Lyrics/-_KILL.ZERO_4wUVupuI2Ko.mp3.wav", + dict(group="magnet", generator="magnet", caption_set="genius", base_key="-_KILL.ZERO_4wUVupuI2Ko")), + ("jasco_generated_music/Genius-lyrics/x.mp3.wav", dict(group="jasco", caption_set="genius")), + ("magenta_generated_music/Jamedo/y - z.mp3.wav", + dict(group="lyria_rt", generator="lyria_rt", caption_set="jamendo")), + ("suno_generated_music/fake_00001_suno_0.mp3", + dict(group="suno", caption_set="none", base_key="fake_00001_suno_0")), + ("aac_generated_music/MagNet_Generated_Music/Genius-Lyrics/a_b_c_aac128.m4a", + dict(group="magnet__aac", variant="aac", base_key="a_b_c", transform_param="aac128")), + ("eq_generated_music/audioldm_generated_music/Jamedo/t - a_eq-cut_low.wav", + dict(group="audioldm2__eq", base_key="t - a", transform_param="eq-cut_low")), + ("pitch_time_generated_music/riffusion_generated_music/Jamedo/t_pt-ps+2st-ts0.9.wav", + dict(group="riffusion__pitch_time", base_key="t", transform_param="pt-ps+2st-ts0.9")), + ("loudness_generation_music/jasco_generated_music/Jamedo/t_lnLUFS-14.wav", + dict(group="jasco__loudness", transform_param="lnLUFS-14")), + ("gaussion_noised_music/suno_generated_music/root/fake_00025_suno_0_gn0.0025.mp3", + dict(group="suno__gaussian", caption_set="none", base_key="fake_00025_suno_0")), + ("pgd_generated_music/musicgen_generated_music/adv_t - a.mp3.wav", + dict(group="musicgen__pgd", variant="pgd", caption_set="*", base_key="t - a")), +]) +def test_classify(path, expect): + row = sl.classify(path) + for k, v in expect.items(): + assert row[k] == v, (k, row) + + +@pytest.mark.parametrize("path", ["Caption/captions-genius.csv", "Genius-Lyrics/metadata.csv", + "Jamendo/segments.csv"]) +def test_metadata_csvs_are_meta(path): + assert sl.classify(path) == {"kind": "meta"} + + +@pytest.mark.parametrize("path", ["README.txt", "mystery_generated_music/a/b.wav", + "aac_generated_music/MagNet_Generated_Music/Jamedo/no_suffix.m4a"]) +def test_unknown_paths_are_not_classified(path): + assert sl.classify(path) is None + + +@pytest.mark.parametrize("bad", ["/etc/passwd", "../x.wav", "a/../../x.wav", "a//b.wav", + "a/./b.wav", "a\\b.wav", "", "a\0b"]) +def test_safe_relpath_rejects_escapes(bad): + with pytest.raises(sl.UnsafePath): + sl.safe_relpath(bad) + + +def test_safe_relpath_accepts_odd_but_safe_names(): + assert sl.safe_relpath("OOD FMA/#x & y (1).mp3") == "OOD FMA/#x & y (1).mp3" + + +# ----------------------------------------------------------------------- planning + +def synthetic_index(n_jam=40, n_gen=30): + recs, off = [], 0 + + def add(path, size=1000): + nonlocal off + recs.append({"path": path, "type": "file", "shard": "part1_001.tar", + "data_offset": off + 512, "size": size, "offset": off}) + off += 512 + (size + 511) // 512 * 512 + + add("Caption/captions-jamendo.csv", 10) + for i in range(n_jam): + add(f"Jamendo/song{i:03d} - artist.mp3", 5000) + add(f"ai_human_co-created_music/Jamendo/song{i:03d} - artist_edited.wav", 3000) + add(f"OOD FMA/{i:05d}_x.mp3", 8000) + for folder in ("MagNet_Generated_Music", "jasco_generated_music"): + for i in range(n_gen): + add(f"{folder}/Jamedo/song{i:03d} - artist.mp3.wav", 2000) + if i % 2 == 0: # "test split": only even songs have transforms + add(f"aac_generated_music/{folder}/Jamedo/song{i:03d} - artist_aac128.m4a", 500) + add(f"pgd_generated_music/{folder.lower()}/adv_song{i:03d} - artist.mp3.wav", 2000) + for i in range(n_gen): + add(f"suno_generated_music/fake_{i:05d}_suno_0.mp3", 900) + return recs + + +CAPS = {"human_jamendo": 5, "human_fma_ood": 4, "clean_per_generator": 6, "co_created": 5, + "transform_per_generator": 3} +TIERS = [{"magnet": 100, "jasco": 100}, {"co_created": 3}, {"human_jamendo": 100}] + + +def run_plan(budget=10 ** 9, seed=sl.SEED, recs=None): + return sl.plan(recs or synthetic_index(), seed=seed, eval_caps=CAPS, train_tiers=TIERS, + train_budget_bytes=budget) + + +def test_plan_eval_caps_and_test_split_preference(): + sel = run_plan() + assert sel["unclassified"] == [] + ev = [r for r in sel["rows"] if r["musebench_role"] == "eval"] + counts = {g: sum(r["group"] == g for r in ev) for g in sel["expected_eval_counts"]} + assert counts == sel["expected_eval_counts"] + assert counts["human_jamendo"] == 5 and counts["magnet"] == 6 and counts["magnet__aac"] == 3 + # Clean eval tracks come from the families that have transforms (even song numbers). + for r in ev: + if r["generator"] in ("magnet", "jasco") and r["variant"] == "clean": + assert int(r["base_key"][4:7]) % 2 == 0 + # PGD rows borrowed the caption set of their clean track. + assert all(r["caption_set"] == "jamendo" for r in ev if r["variant"] == "pgd") + + +def test_plan_transforms_pair_with_clean_eval_and_roles_are_family_disjoint(): + sel = run_plan() + ev = [r for r in sel["rows"] if r["musebench_role"] == "eval"] + clean = {(r["generator"], r["family"]) for r in ev if r["variant"] == "clean"} + transforms = [r for r in ev if r["variant"] in ("aac", "pgd")] + assert transforms and all((r["generator"], r["family"]) in clean for r in transforms) + eval_fams = {r["family"] for r in ev} + train = [r for r in sel["rows"] if r["musebench_role"] == "train_pool"] + assert train and not any(r["family"] in eval_fams for r in train) + assert not any(r["variant"] in ("aac", "pgd") for r in train) # transforms are eval-only + assert sum(r["group"] == "co_created" for r in train) == 3 + + +def test_plan_is_deterministic_and_seed_dependent(): + a, b = run_plan(), run_plan() + assert sl.selection_fingerprint(a) == sl.selection_fingerprint(b) + c = run_plan(seed="other-seed") + assert sl.selection_fingerprint(a) != sl.selection_fingerprint(c) + + +def test_plan_respects_train_budget_round_robin(): + sel = run_plan(budget=9000) + train = [r for r in sel["rows"] if r["musebench_role"] == "train_pool"] + assert sum(r["size"] for r in train) <= 9000 + groups = [r["group"] for r in train] + assert groups.count("magnet") == 2 and groups.count("jasco") == 2 # shared evenly + assert sel["bytes"]["train_pool"] == sum(r["size"] for r in train) + assert run_plan(budget=0)["bytes"]["train_pool"] == 0 + + +# ----------------------------------------------------------------------- manifest + +def test_build_manifest_uses_ledger_and_reports_missing(tmp_path): + sel = run_plan() + ledger = {r["path"]: {"path": r["path"], "size": r["size"], "sha256": "ab" * 32} + for r in sel["rows"][1:]} + man, missing = sl.build_manifest(sel, ledger, tmp_path) + assert missing == [sel["rows"][0]["path"]] + assert man["metadata"]["n_tracks"] == len(sel["rows"]) - 1 + t = man["tracks"][0] + for k in ("track_id", "path", "label", "source", "generator", "variant", "caption_set", + "musebench_role", "protocol_split", "sha256"): + assert t[k] not in (None, "") + assert t["path"].startswith(str(tmp_path / "audio")) + assert t["source"].startswith("musebench_") + assert len({x["track_id"] for x in man["tracks"]}) == len(man["tracks"]) + + +def test_manifest_loads_with_runner_loader(tmp_path): + from artifactbench.data.manifest import load_manifest + + sel = run_plan() + ledger = {r["path"]: {"path": r["path"], "size": r["size"], "sha256": "0" * 64} + for r in sel["rows"]} + man, _ = sl.build_manifest(sel, ledger, tmp_path) + path = tmp_path / "m.json" + path.write_text(json.dumps(man)) + entries, by_source = load_manifest(str(path), split="all") + assert len(entries) == len(sel["rows"]) + assert "musebench_magnet__aac" in by_source + evals, _ = load_manifest(str(path), split="all", protocol_split="eval") + assert len(evals) == sum(r["musebench_role"] == "eval" for r in sel["rows"]) diff --git a/tests/test_musebench_verify.py b/tests/test_musebench_verify.py index d6102e5..fd0fb67 100644 --- a/tests/test_musebench_verify.py +++ b/tests/test_musebench_verify.py @@ -90,6 +90,6 @@ def test_cli_runs_isolated_and_reports_failure(tmp_path): out = subprocess.run([sys.executable, "-I", str(script), "--root", str(tmp_path), "index"], capture_output=True, text=True) assert out.returncode == 1 and "[index] FAIL" in out.stdout - out = subprocess.run([sys.executable, "-I", str(script), "--root", str(tmp_path), "acquire"], + out = subprocess.run([sys.executable, "-I", str(script), "--root", str(tmp_path), "leak"], capture_output=True, text=True) assert out.returncode == 1 and "no verifier yet" in out.stdout