Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions docs/MUSEBENCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,39 @@ Rules for the data root (`$MUSEBENCH_ROOT`, default `~/data/musebench`):
- Nothing derived from MuseBench audio (manifests with local paths, scores, reports) is committed
to this repository.

## Leak map

`scripts/musebench/leak_map.py` marks every manifest row `seen_in_training` (boolean) for the
detector under test and writes `$MUSEBENCH_ROOT/leak/leak_report.json`. The detector and its
training pool are described in `$MUSEBENCH_ROOT/leak/references.json` (machine paths stay out of
this repository): the model, the training manifest(s) or dataset builder it came from, a SONICS
`fake_songs` directory, FMA track-id lists, MTG-Jamendo `raw.meta.tsv` with the audio folders
that were in the pool, and generator-level exposure.

```bash
python scripts/musebench/leak_map.py --root ~/data/musebench --references ~/data/musebench/leak/references.json
python scripts/musebench/verify.py --root ~/data/musebench leak
```

Identity rules:

| MuseBench rows | Key | Confirmation |
|---|---|---|
| Suno, Udio | SONICS file name `fake_<id>_<generator>_<seed>` | file size |
| OOD FMA | FMA track id (leading digits of the file name) | file size vs a local FMA copy, where available |
| Jamendo | normalised (title, artist) vs MTG-Jamendo `raw.meta.tsv`, any ` - ` split | duration within 2 s |

A transform inherits the flag of its clean track and a co-created remix that of its human
source. Generations conditioned on a seen track's caption are different audio and are not
"seen"; exposure to a *generator* is recorded separately in `generator_in_training`. The match
is against the training **candidate pool**, an upper bound on what a checkpoint saw.

Dataset facts found on the way: all MuseBench Suno and Udio files checked so far (2,000 of
2,000 in the acquired corpus) have the name **and byte size** of a SONICS `fake_songs` file, so
any detector trained on SONICS has seen them. JamendoMaxCaps names match MTG-Jamendo metadata
for only a minority of tracks (266 of 3,877 acquired Jamendo rows), so most MuseBench Jamendo
audio is outside MTG-Jamendo.

## Licenses

- Collection: CC BY-SA 3.0 (inherited from JamendoMaxCaps).
Expand Down
330 changes: 330 additions & 0 deletions scripts/musebench/leak_map.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,330 @@
#!/usr/bin/env python3
# Created: 2026-11-04
# Purpose: Annotate every MuseBench runtime-manifest row with whether its audio (or its direct
# audio ancestor) is in the training candidate pool of the detector under test, and
# write a leak report with per-source seen/unseen counts and the evidence used.
# Dependencies: standard library; ffprobe (optional, duration confirmation of Jamendo matches).
# Test Status: tests/test_musebench_leak_map.py
"""Leak map for MuseBench against a detector's training history.

MuseBench overlaps three corpora that detectors are commonly trained on:

* **Suno / Udio** tracks are SONICS tracks with SONICS file names
(``fake_<id>_<generator>_<seed>.mp3``) -> matched by name, confirmed by file size.
* **OOD FMA** files are named ``<fma track id>_<title>.mp3`` -> matched by FMA track id
against the FMA track lists used for training.
* **Jamendo** files are named ``<title> - <artist>.mp3`` (JamendoMaxCaps) -> matched by
normalised (title, artist) against MTG-Jamendo ``raw.meta.tsv``, restricted to the local
MTG-Jamendo folders that were part of the training pool, confirmed by duration.

A row is ``seen_in_training`` when its own audio, or its direct audio ancestor, is in that pool:
a transform inherits from its clean track, a co-created remix from its human source track.
Generations conditioned on a seen track's caption are different audio and are not "seen";
``generator_in_training`` records generator-level exposure separately.

The match is against the training *candidate pool* (what the dataset builder could draw from),
which is an upper bound on what the checkpoint actually saw.
"""
from __future__ import annotations

import os as _os
import sys as _sys

# This directory holds select.py, which shadows the stdlib ``select`` module when the
# directory is on sys.path (``python scripts/musebench/x.py`` without -I). Drop it.
if _sys.path and _os.path.abspath(_sys.path[0] or ".") == _os.path.dirname(_os.path.abspath(__file__)):
_sys.path.pop(0)

import argparse # noqa: E402
import csv # noqa: E402
import hashlib # noqa: E402
import json # noqa: E402
import os # noqa: E402
import re # noqa: E402
import subprocess # noqa: E402
import sys # noqa: E402
import time # noqa: E402
import unicodedata # noqa: E402
from collections import Counter, defaultdict # noqa: E402
from pathlib import Path # noqa: E402
from typing import Callable # noqa: E402

DEFAULT_ROOT = Path(os.environ.get("MUSEBENCH_ROOT", "~/data/musebench")).expanduser()
MANIFEST_REL = Path("manifests") / "musebench_runtime_manifest.json"
REPORT_REL = Path("leak") / "leak_report.json"
DURATION_TOLERANCE_S = 2.0
LEAK_FIELDS = ("seen_in_training", "seen_evidence", "generator_in_training",
"in_local_mtg_jamendo", "mtg_jamendo_ids", "leak_note")


# --------------------------------------------------------------------------- identity keys

def norm(text: str) -> str:
text = unicodedata.normalize("NFKC", text).replace("&amp;", "&").casefold()
return re.sub(r"\s+", " ", text).strip()


def load_mtg_meta(tsv: Path) -> dict[tuple[str, str], set[int]]:
"""(normalised title, normalised artist) -> MTG-Jamendo track ids."""
index: dict[tuple[str, str], set[int]] = defaultdict(set)
with open(tsv, encoding="utf-8", newline="") as f:
reader = csv.reader(f, delimiter="\t")
header = next(reader)
col = {name: i for i, name in enumerate(header)}
for row in reader:
if len(row) <= max(col["TRACK_ID"], col["TRACK_NAME"], col["ARTIST_NAME"]):
continue
tid = int(row[col["TRACK_ID"]].split("_")[-1])
index[(norm(row[col["TRACK_NAME"]]), norm(row[col["ARTIST_NAME"]]))].add(tid)
return index


def jamendo_candidates(base: str, meta: dict[tuple[str, str], set[int]]) -> set[int]:
"""MTG ids whose (title, artist) matches ``"<title> - <artist>"`` at any ' - ' split."""
parts = base.split(" - ")
out: set[int] = set()
for i in range(1, len(parts)):
out |= meta.get((norm(" - ".join(parts[:i])), norm(" - ".join(parts[i:]))), set())
return out


def scan_mtg_audio(audio_dir: Path, folders: set[str] | None) -> dict[int, Path]:
"""MTG track id -> local file, for the given two-digit folders (all if None)."""
out: dict[int, Path] = {}
for sub in sorted(audio_dir.iterdir()):
if not sub.is_dir() or (folders is not None and sub.name not in folders):
continue
for f in sub.iterdir():
stem = f.name.split(".")[0]
if stem.isdigit():
out[int(stem)] = f
return out


FMA_ID = re.compile(r"^(\d+)")


def fma_id(name: str) -> int | None:
m = FMA_ID.match(Path(name).name)
return int(m.group(1)) if m else None


def load_fma_ids(path: Path) -> set[int]:
"""FMA track ids from a text list of paths or a JSON list of {filename|path} records."""
text = path.read_text(encoding="utf-8")
names: list[str] = []
if path.suffix == ".json":
data = json.loads(text)
for item in data:
names.append(item if isinstance(item, str) else
item.get("filename") or item.get("filepath") or item.get("path", ""))
else:
names = [line.strip() for line in text.splitlines() if line.strip()]
return {i for i in (fma_id(n) for n in names) if i is not None}


def scan_fma_audio(dirs: list[Path]) -> dict[int, Path]:
"""FMA track id -> local file (flat or fma_full/<000>/ layout)."""
out: dict[int, Path] = {}
for d in dirs:
for f in d.rglob("*.mp3"):
fid = fma_id(f.name)
if fid is not None:
out.setdefault(fid, f)
return out


def scan_sonics(fake_dir: Path) -> dict[str, int]:
"""SONICS fake file stem -> size in bytes."""
return {f.name.rsplit(".", 1)[0]: f.stat().st_size for f in fake_dir.iterdir() if f.is_file()}


def ffprobe_duration(path: Path) -> float | None:
try:
out = subprocess.run(["ffprobe", "-v", "error", "-show_entries", "format=duration",
"-of", "default=nw=1:nk=1", str(path)],
capture_output=True, text=True, timeout=60)
return float(out.stdout.strip())
except (OSError, ValueError, subprocess.SubprocessError):
return None


# --------------------------------------------------------------------------- annotation

def annotate(tracks: list[dict], *, sonics: dict[str, int], fma_ids: set[int],
mtg_meta: dict[tuple[str, str], set[int]], mtg_pool: dict[int, Path],
mtg_all: dict[int, Path], generators_in_training: dict[str, bool],
fma_files: dict[int, Path] | None = None,
duration: Callable[[Path], float | None] | None = None,
tolerance_s: float = DURATION_TOLERANCE_S) -> dict:
"""Add ``seen_in_training``/``seen_evidence``/... to each row; return match statistics."""
stats = Counter()
for t in tracks: # idempotent: drop fields from an earlier annotation
for key in LEAK_FIELDS:
t.pop(key, None)
# 1. Ancestor identities: clean Suno/Udio, FMA and Jamendo human rows.
seen_family: dict[str, str] = {}
for t in tracks:
gen, var, base = t["generator"], t["variant"], t["base_key"]
if gen in ("suno", "udio") and var == "clean":
size = sonics.get(base)
if size is not None:
stats["sonics_name_match"] += 1
same = size == t["size"]
stats["sonics_size_equal" if same else "sonics_size_differs"] += 1
seen_family[t["family"]] = f"SONICS file {base} ({'same size' if same else 'size differs'})"
elif t["source"] == "musebench_human_fma_ood":
fid = fma_id(base)
if fid is not None and fid in fma_ids:
stats["fma_id_match"] += 1
note = f"FMA track {fid} in training FMA list"
ref = (fma_files or {}).get(fid)
if ref is not None:
same = ref.stat().st_size == t["size"]
stats["fma_size_equal" if same else "fma_size_differs"] += 1
note += f" ({'same size as' if same else 'size differs from'} local copy)"
seen_family[t["family"]] = note
elif t["source"] == "musebench_human_jamendo":
cands = jamendo_candidates(base, mtg_meta)
if not cands:
continue
stats["jamendo_meta_match"] += 1
local = [c for c in sorted(cands) if c in mtg_all]
if local:
t["mtg_jamendo_ids"] = local
pooled = [c for c in local if c in mtg_pool]
if not pooled:
continue
note = f"MTG-Jamendo track {pooled[0]} (title/artist match)"
if duration is not None:
d_ref = duration(mtg_pool[pooled[0]])
d_mb = duration(Path(t["path"]))
if d_ref is not None and d_mb is not None:
if abs(d_ref - d_mb) > tolerance_s:
stats["jamendo_duration_mismatch"] += 1
t["leak_note"] = f"{note}; duration {d_mb:.1f}s vs {d_ref:.1f}s: not counted"
continue
note += f", duration {d_mb:.1f}s vs {d_ref:.1f}s"
stats["jamendo_duration_confirmed"] += 1
stats["jamendo_pool_match"] += 1
seen_family[t["family"]] = note

# 2. Every row: own identity or direct ancestor (same family, and the row derives from it).
for t in tracks:
evidence = None
fam = t["family"]
if fam in seen_family:
if t["variant"] == "clean" and t["generator"] in ("human", "suno", "udio"):
evidence = seen_family[fam]
elif t["generator"] in ("suno", "udio"):
evidence = "transform of " + seen_family[fam] # transforms of a seen SONICS track
elif t["variant"] == "co_created":
evidence = "co-created remix of " + seen_family[fam]
t["seen_in_training"] = evidence is not None
t["seen_evidence"] = evidence
g = t["generator"]
t["generator_in_training"] = None if g == "human" else bool(generators_in_training.get(g, False))
if t["source"] == "musebench_human_jamendo":
t["in_local_mtg_jamendo"] = bool(t.get("mtg_jamendo_ids"))
return dict(stats)


def per_source_counts(tracks: list[dict]) -> dict[str, dict]:
out: dict[str, Counter] = defaultdict(Counter)
for t in tracks:
c = out[t["source"]]
c["total"] += 1
c["seen" if t["seen_in_training"] else "unseen"] += 1
c[f"{t['musebench_role']}_{'seen' if t['seen_in_training'] else 'unseen'}"] += 1
return {k: {"total": v["total"], "seen": v["seen"], "unseen": v["unseen"],
"eval_seen": v["eval_seen"], "eval_unseen": v["eval_unseen"],
"train_pool_seen": v["train_pool_seen"], "train_pool_unseen": v["train_pool_unseen"]}
for k, v in sorted(out.items())}


def manifest_fingerprint(tracks: list[dict]) -> str:
h = hashlib.sha256()
for t in sorted(tracks, key=lambda t: t["track_id"]):
h.update(f"{t['track_id']}\t{t['musebench_path']}\n".encode("utf-8", "surrogateescape"))
return h.hexdigest()


def atomic_write_json(path: Path, obj: object) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(path.suffix + ".tmp")
tmp.write_text(json.dumps(obj, ensure_ascii=True, indent=1) + "\n", encoding="utf-8")
os.replace(tmp, path)


def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(description="Annotate the MuseBench manifest with training overlap.")
ap.add_argument("--root", type=Path, default=DEFAULT_ROOT)
ap.add_argument("--references", type=Path, default=None,
help="JSON describing the detector and its training pool "
"(default: <root>/leak/references.json)")
ap.add_argument("--no-duration-check", action="store_true")
args = ap.parse_args(argv)
ref_path = args.references or args.root / "leak" / "references.json"
refs = json.loads(ref_path.read_text(encoding="utf-8"))

sonics = scan_sonics(Path(refs["sonics_fake_dir"]))
fma_ids: set[int] = set()
for p in refs["fma_id_lists"]:
fma_ids |= load_fma_ids(Path(p))
fma_files = scan_fma_audio([Path(p) for p in refs.get("fma_audio_dirs", [])])
mtg_meta = load_mtg_meta(Path(refs["mtg_meta_tsv"]))
mtg_dir = Path(refs["mtg_audio_dir"])
mtg_pool = scan_mtg_audio(mtg_dir, set(refs["mtg_pool_folders"]))
mtg_all = scan_mtg_audio(mtg_dir, None)

man_path = args.root / MANIFEST_REL
manifest = json.loads(man_path.read_text(encoding="utf-8"))
tracks = manifest["tracks"]
stats = annotate(tracks, sonics=sonics, fma_ids=fma_ids, mtg_meta=mtg_meta,
mtg_pool=mtg_pool, mtg_all=mtg_all,
generators_in_training=refs["generators_in_training"],
fma_files=fma_files, duration=None if args.no_duration_check else ffprobe_duration)
manifest["metadata"]["leak_annotation"] = {
"model": refs["model"]["name"], "report": str(args.root / REPORT_REL),
"annotated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())}
atomic_write_json(man_path, manifest)

report = {
"model": refs["model"],
"training_manifests": refs["training_manifests"],
"reference_corpora": {
"sonics_fake_dir": {"path": refs["sonics_fake_dir"], "files": len(sonics)},
"fma_id_lists": {"paths": refs["fma_id_lists"], "track_ids": len(fma_ids),
"audio_dirs": refs.get("fma_audio_dirs", []),
"local_files": len(fma_files)},
"mtg_jamendo": {"meta_tsv": refs["mtg_meta_tsv"], "audio_dir": refs["mtg_audio_dir"],
"pool_folders": refs["mtg_pool_folders"],
"pool_tracks": len(mtg_pool), "local_tracks": len(mtg_all)},
},
"method": {
"suno_udio": "SONICS file-name match (fake_<id>_<gen>_<seed>), size compared",
"fma": "FMA track id (leading digits of the file name) in the training FMA id lists; "
"size compared with a local FMA copy where one exists",
"jamendo": "normalised (title, artist) vs MTG-Jamendo raw.meta.tsv, restricted to the "
f"pool folders; duration within {DURATION_TOLERANCE_S}s"
+ (" (not checked)" if args.no_duration_check else ""),
"inheritance": "transforms inherit from their clean track; co-created remixes from "
"their human source; caption-conditioned generations do not",
"scope": "candidate pool of the training dataset builder: an upper bound on exposure",
},
"generators_in_training": refs["generators_in_training"],
"caveats": refs.get("caveats", []),
"match_stats": stats,
"per_source": per_source_counts(tracks),
"totals": dict(Counter("seen" if t["seen_in_training"] else "unseen" for t in tracks)),
"manifest": str(man_path),
"manifest_fingerprint": manifest_fingerprint(tracks),
"created_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
}
atomic_write_json(args.root / REPORT_REL, report)
print(json.dumps({"match_stats": stats, "totals": report["totals"]}, indent=1))
return 0


if __name__ == "__main__":
sys.exit(main())
Loading
Loading