Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 58 additions & 5 deletions docs/MUSEBENCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,10 +161,13 @@ ArtifactBench evaluates detectors that were not trained on MuseBench. The rules:
calibration) is selected on MuseBench data.
2. **Native input.** Each detector receives the audio through its own adapter (for ArtifactNet:
44.1 kHz, seven 4 s chunks, centre crop), not the paper's 10 s / 48 kHz excerpt.
3. **Eval cohort** (deterministic, sealed): all Jamendo human tracks, all OOD-FMA tracks,
up to 2,000 clean tracks per generator (including Suno and Udio), up to 2,000 co-created
tracks, and up to 200 tracks per transform × generator. Rows in the cohort are marked
`musebench_role = eval` and must never be used for training.
3. **Eval cohort** (deterministic, sealed), chosen by `scripts/musebench/select.py`:
3,000 Jamendo human tracks, 3,000 OOD-FMA tracks, 1,000 clean tracks per generator (all
ten, including Suno and Udio), 1,000 co-created tracks, and 200 tracks per transform ×
generator (8 × 10 = 80 groups). Clean AI tracks are drawn from tracks that have transformed
variants (the paper's test split), and the transform samples are the transforms of those
same tracks, so clean/transform results are paired. Rows in the cohort are marked
`musebench_role = eval` (also `protocol_split = eval`) and must never be used for training.
4. **Leak annotation.** Suno and Udio are SONICS tracks; FMA and Jamendo audio may also appear in
a detector's training history. Every row carries `seen_in_training` for the detector under
test, and every metric is reported for seen and unseen rows separately.
Expand All @@ -183,8 +186,57 @@ acquired selectively:
python scripts/musebench/index_shards.py --root ~/data/musebench --workers 75
# Assert the index is complete and internally consistent.
python scripts/musebench/verify.py --root ~/data/musebench index
# Choose the eval cohort and the budgeted train pool (deterministic).
python scripts/musebench/select.py plan --root ~/data/musebench --budget-gb 170 --free-floor-gb 80
# Download the chosen members by byte range; resumable, stops at the free-space floor.
python scripts/musebench/fetch.py --root ~/data/musebench --workers 16 --free-floor-gb 80
# Write the runtime manifest, then check files, hashes, cohort shape and disk policy.
python scripts/musebench/select.py manifest --root ~/data/musebench
python scripts/musebench/verify.py --root ~/data/musebench acquire
```

### Families and the train pool

Every member is assigned a *family*: the source track it derives from (`caption_set/base
name`; Suno and Udio tracks are their own family). A Jamendo song, its co-created remix, every
generation conditioned on its caption and every transform of those generations share one
family. Selection is ordered by `sha256(seed:family)`, so the same families are preferred
across generators, and the **train pool excludes every family that has an eval member**.

The train pool fills the remaining disk budget in priority tiers, each source up to a cap,
round-robin within a tier:

1. generators that are new to most detectors' training data: AudioLDM2, MAGNeT, JASCO,
Mustango, Lyria RealTime (5,000 each);
2. co-created (5,000);
3. human Jamendo (3,000) and OOD-FMA (2,000);
4. MusicGen, Riffusion, Stable Audio Open (2,000 each);
5. Suno and Udio: none (they are SONICS tracks, usually available locally already).

Transforms are never put in the train pool; all of them except PGD can be regenerated from
clean audio with the Appendix A.3 parameters.

### Acquired corpus (2026-11-04)

With the defaults above (`--budget-gb 170 --free-floor-gb 80`): 64,756 audio files, 169.9 GB —
33,000 eval rows in 93 sources (91.3 GB) and 31,756 train-pool rows (78.6 GB: 5,000 each of
AudioLDM2, MAGNeT, JASCO, Mustango, Lyria RealTime and co-created, 877 Jamendo, 879
OOD-FMA; the budget ran out inside tier 3). Every file's SHA-256 was recomputed and matches the
fetch ledger. The index allows fetching more of the train pool later with a larger budget.

### Observations from the shipped files

- Suno and Udio transforms sit under a literal `root/` folder
(`aac_generated_music/suno_generated_music/root/…`); PGD files have no caption-set folder and
are named `adv_<clean name>`; the PGD folders for MAGNeT and MusicGen are lower case.
- Transform file names end with the transform parameters (`_aac128`, `_mp3128`, `_ogg3`,
`_eq-cut_low`, `_pt-ps+2st-ts0.9`, `_lnLUFS-14`, `_gn0.0025`). The Gaussian files say
`gn0.0025`, while Appendix A.3 states σ = 0.025.
- The shipped CSVs (`Caption/captions-*.csv`, `Genius-Lyrics/metadata.csv`,
`Genius-Lyrics/segments.csv`, `Jamendo/segments.csv`) carry captions, segment descriptions and
YouTube ids only. The per-track Jamendo `license_ccurl` that the card mentions is **not**
shipped; manifest rows carry `license_ccurl: null` until it is joined from JamendoMaxCaps.

Rules for the data root (`$MUSEBENCH_ROOT`, default `~/data/musebench`):

- Shard bytes and extracted audio are untrusted: they live only under the data root, scripts live
Expand All @@ -200,7 +252,8 @@ Rules for the data root (`$MUSEBENCH_ROOT`, default `~/data/musebench`):

- Collection: CC BY-SA 3.0 (inherited from JamendoMaxCaps).
- Jamendo audio: per-track Creative Commons licenses (CC BY, BY-SA, BY-NC, BY-NC-ND), recorded
in the `license_ccurl` metadata; NC and ND tracks restrict commercial use and derivatives.
in JamendoMaxCaps' `license_ccurl` metadata (not shipped with MuseBench); NC and ND tracks
restrict commercial use and derivatives.
- FMA audio: per-track Creative Commons licenses; FMA metadata CC BY 4.0.
- AI audio: bound by each generator's terms; most open weights are non-commercial
(MusicGen / MAGNeT / JASCO weights CC BY-NC 4.0, AudioLDM and Mustango CC BY-NC-SA 4.0,
Expand Down
213 changes: 213 additions & 0 deletions scripts/musebench/fetch.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,213 @@
#!/usr/bin/env python3
# Created: 2026-11-04
# Purpose: Download the members chosen by select.py from the MuseBench shards with HTTP range
# requests, byte-identical, SHA-256 recorded, atomic, resumable, with a free-space floor.
# Dependencies: standard library only.
# Test Status: tests/test_musebench_fetch.py
"""Fetch selected MuseBench members.

Reads ``<root>/selection/selection.json`` and writes

* audio to ``<root>/audio/<member path>`` and metadata CSVs to ``<root>/meta/<member path>``,
* one ledger line per finished file to ``<root>/selection/fetch_ledger.jsonl``
(``path, kind, size, sha256, fetched_at``).

Each member is read with one range request ``[data_offset, data_offset + size)``, written to a
``.part`` file, checked for length, hashed, and renamed into place. A file already in the
ledger with the right size on disk is skipped, so the command can be re-run after any
interruption. Before each download the free space of the data root's filesystem must stay at
or above ``--free-floor-gb`` after the write, otherwise the run stops.

Member names are untrusted: each is validated by ``select.safe_relpath`` before use.
"""
from __future__ import annotations

import os as _os
import sys as _sys

# This directory holds select.py, which shadows the stdlib ``select`` module when the
# directory is on sys.path (``python scripts/musebench/x.py`` without -I). Drop it.
if _sys.path and _os.path.abspath(_sys.path[0] or ".") == _os.path.dirname(_os.path.abspath(__file__)):
_sys.path.pop(0)

import argparse # noqa: E402
import hashlib # noqa: E402
import json # noqa: E402
import os # noqa: E402
import shutil # noqa: E402
import sys # noqa: E402
import threading # noqa: E402
import time # noqa: E402
from concurrent.futures import ThreadPoolExecutor, as_completed # noqa: E402
from pathlib import Path # noqa: E402
from typing import Callable # noqa: E402

REPO_ROOT = Path(__file__).resolve().parents[2]
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))

from scripts.musebench.index_shards import RangeFetcher, resolve_url # noqa: E402
from scripts.musebench.select import load_ledger, safe_relpath # noqa: E402

DEFAULT_ROOT = Path(os.environ.get("MUSEBENCH_ROOT", "~/data/musebench")).expanduser()
GB = 10 ** 9
ROLE_ORDER = {"meta": 0, "eval": 1, "train_pool": 2}


class DiskFloorReached(RuntimeError):
pass


def work_items(selection: dict) -> list[dict]:
items = [{**m, "kind": "meta", "role": "meta"} for m in selection["meta"]]
items += [{"path": r["path"], "shard": r["shard"], "data_offset": r["data_offset"],
"size": r["size"], "kind": "audio", "role": r["musebench_role"]}
for r in selection["rows"]]
items.sort(key=lambda it: (ROLE_ORDER[it["role"]], it["shard"], it["data_offset"]))
return items


def dest_for(root: Path, item: dict) -> Path:
rel = safe_relpath(item["path"])
base = root / ("meta" if item["kind"] == "meta" else "audio")
dest = (base / rel).resolve()
if base.resolve() not in dest.parents:
raise ValueError(f"destination escapes data root: {item['path']!r}")
return dest


class Fetcher:
"""Thread-safe member downloader; one persistent RangeFetcher per (thread, shard)."""

def __init__(self, root: Path, shards: dict[str, dict], free_floor_bytes: int,
reader_factory: Callable[[dict], Callable[[int, int], bytes]] | None = None,
disk_free: Callable[[Path], int] | None = None):
self.root = root
self.shards = shards
self.floor = free_floor_bytes
self.reader_factory = reader_factory or (
lambda s: RangeFetcher(resolve_url(s["repo"], s["revision"], s["file"])))
self.disk_free = disk_free or (lambda p: shutil.disk_usage(p).free)
self.local = threading.local()
self.ledger_lock = threading.Lock()
self.space_lock = threading.Lock()
self.reserved = 0
self.ledger_path = root / "selection" / "fetch_ledger.jsonl"
self.ledger_path.parent.mkdir(parents=True, exist_ok=True)
self.stop = threading.Event()

def _reader(self, shard_file: str):
# Work items are sorted by shard, so one open connection per thread is enough.
current = getattr(self.local, "current", None)
if current is None or current[0] != shard_file:
if current is not None and hasattr(current[1], "_close"):
current[1]._close()
current = self.local.current = (shard_file, self.reader_factory(self.shards[shard_file]))
return current[1]

def _reserve(self, size: int) -> None:
with self.space_lock:
free = self.disk_free(self.root) - self.reserved
if free - size < self.floor:
self.stop.set()
raise DiskFloorReached(
f"free space {free / GB:.1f} GB - {size / GB:.3f} GB < floor {self.floor / GB:.0f} GB")
self.reserved += size

def _release(self, size: int) -> None:
with self.space_lock:
self.reserved -= size

def fetch(self, item: dict, attempts: int = 3) -> dict:
if self.stop.is_set():
raise DiskFloorReached("stopped")
dest = dest_for(self.root, item)
self._reserve(item["size"])
try:
for attempt in range(attempts):
data = self._reader(item["shard"])(item["data_offset"], item["size"]) \
if item["size"] else b""
if len(data) == item["size"]:
break
if attempt == attempts - 1:
raise OSError(f"short read for {item['path']!r}: {len(data)} of {item['size']}")
digest = hashlib.sha256(data).hexdigest()
dest.parent.mkdir(parents=True, exist_ok=True)
part = dest.with_name(dest.name + ".part")
with open(part, "wb") as f:
f.write(data)
f.flush()
os.fsync(f.fileno())
os.replace(part, dest)
finally:
self._release(item["size"])
entry = {"path": item["path"], "kind": item["kind"], "size": item["size"],
"sha256": digest, "fetched_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())}
with self.ledger_lock:
with open(self.ledger_path, "a", encoding="utf-8") as f:
f.write(json.dumps(entry, ensure_ascii=True) + "\n")
return entry


def pending(root: Path, items: list[dict]) -> list[dict]:
"""Items not yet in the ledger with a correctly sized file on disk."""
ledger = load_ledger(root)
out = []
for it in items:
entry = ledger.get(it["path"])
dest = dest_for(root, it)
if entry and entry["size"] == it["size"] and dest.exists() \
and dest.stat().st_size == it["size"]:
continue
out.append(it)
return out


def run(root: Path, selection: dict, shards: dict[str, dict], workers: int,
free_floor_bytes: int, **fetcher_kw) -> tuple[int, int, list[str]]:
items = pending(root, work_items(selection))
fetcher = Fetcher(root, shards, free_floor_bytes, **fetcher_kw)
done, failed = 0, []
t0, last = time.time(), 0.0
nbytes = 0
with ThreadPoolExecutor(max_workers=max(1, workers)) as pool:
futures = {pool.submit(fetcher.fetch, it): it for it in items}
for fut in as_completed(futures):
it = futures[fut]
try:
fut.result()
done += 1
nbytes += it["size"]
except DiskFloorReached as exc:
failed.append(f"{it['path']}: {exc}")
except Exception as exc: # noqa: BLE001 - keep going, report at the end
failed.append(f"{it['path']}: {exc!r}")
now = time.time()
if now - last >= 60:
last = now
rate = nbytes / max(now - t0, 1e-9) / 1e6
print(f"[{time.strftime('%H:%M:%S')}] {done}/{len(items)} files, "
f"{nbytes / GB:.1f} GB, {rate:.0f} MB/s, {len(failed)} failed",
file=sys.stderr, flush=True)
return len(items), done, failed


def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(description="Fetch selected MuseBench members by range.")
ap.add_argument("--root", type=Path, default=DEFAULT_ROOT)
ap.add_argument("--workers", type=int, default=16)
ap.add_argument("--free-floor-gb", type=float, default=80.0)
args = ap.parse_args(argv)
selection = json.loads((args.root / "selection" / "selection.json").read_text(encoding="utf-8"))
shards = {s["file"]: s for s in
json.loads((args.root / "index" / "shards.json").read_text(encoding="utf-8"))}
total, done, failed = run(args.root, selection, shards, args.workers,
int(args.free_floor_gb * GB))
print(f"fetched {done}/{total} pending files; {len(failed)} failed", file=sys.stderr)
for line in failed[:20]:
print(f" {line}", file=sys.stderr)
return 1 if failed else 0


if __name__ == "__main__":
sys.exit(main())
9 changes: 9 additions & 0 deletions scripts/musebench/index_shards.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,15 @@
"""
from __future__ import annotations

import os as _os
import sys as _sys

# This directory holds select.py, which shadows the stdlib ``select`` module when the
# directory is on sys.path (``python scripts/musebench/x.py`` without -I). Drop it.
if _sys.path and _os.path.abspath(_sys.path[0] or ".") == _os.path.dirname(_os.path.abspath(__file__)):
_sys.path.pop(0)


import argparse
import http.client
import json
Expand Down
9 changes: 9 additions & 0 deletions scripts/musebench/inventory.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,15 @@
"""
from __future__ import annotations

import os as _os
import sys as _sys

# This directory holds select.py, which shadows the stdlib ``select`` module when the
# directory is on sys.path (``python scripts/musebench/x.py`` without -I). Drop it.
if _sys.path and _os.path.abspath(_sys.path[0] or ".") == _os.path.dirname(_os.path.abspath(__file__)):
_sys.path.pop(0)


import argparse
import json
import os
Expand Down
Loading
Loading