Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions docs/MUSEBENCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -281,6 +281,26 @@ any detector trained on SONICS has seen them. JamendoMaxCaps names match MTG-Jam
for only a minority of tracks (266 of 3,877 acquired Jamendo rows), so most MuseBench Jamendo
audio is outside MTG-Jamendo.

### A second detector: lineage manifests and a model key

A detector's real history is the chain of training manifests of every generation in its weight
lineage, not a candidate-pool directory. A references JSON may list them as `lineage_manifests`
(plus `mtg_meta_tsv` and `mtg_audio_dir`); their rows are read by `path`, `track_id` and the
`origin_*` fields of derived rows (codec cells such as `cell:<cell>:<source>:<id>`), and yield
SONICS stems, FMA ids and MTG-Jamendo ids by the same identity rules. The union over train, val
and test rows is an upper bound on exposure for the manifests listed; a lineage step whose
manifest is missing makes it a lower bound, which the references' `caveats` should say.

`--model-key KEY` writes `seen_in_training_KEY` (and the other annotation fields with the same
suffix) and `leak/leak_report_KEY.json`, so the first model's unkeyed annotation and report stay
as they are. A SONICS name whose training file is not on this host counts as
`sonics_size_unknown` rather than a size mismatch.

```bash
python scripts/musebench/leak_map.py --root ~/data/musebench \
--references ~/data/musebench/leak/references_<key>.json --model-key <key>
```

## Running a detector

The runner's `--protocol-split` has no `eval` choice, so the eval rows are written to their own
Expand Down Expand Up @@ -331,6 +351,32 @@ that environment on `LD_LIBRARY_PATH`; otherwise onnxruntime silently falls back
(about 7× slower for ArtifactNet). CUDA and CPU scores differed by at most 0.016 on a 30-track
probe, with no decision changes at 0.5.

### Externally scored runs

A pipeline that is not wrapped by `artifactbench.bench` (for example a production stack with its
own batch runner and decision rule) writes one JSON line per file with `path`, `probability`, a
decision field and `error`. `import_scores.py` turns that into the runner's result layout, keeping
the pipeline's own decision per track (`decision_ai`), which `report.py` honours over
`prob >= threshold`. Tracks with no line or a failing decision value become recorded failures.
`report.py --seen-field` splits seen/unseen by a model-keyed annotation; the field is joined from
the runtime manifest by track id, so the eval manifest a run consumed is never rewritten.

```bash
python scripts/musebench/import_scores.py --root ~/data/musebench \
--scores ~/data/musebench/results/<run>/scores.jsonl --run-dir ~/data/musebench/results/<run> \
--model-name <name> --threshold <tau> --decision-field verdict
python scripts/musebench/report.py --root ~/data/musebench --run-dir ~/data/musebench/results/<run> \
--seen-field seen_in_training_<key>
python scripts/musebench/verify.py --root ~/data/musebench production
```

`verify.py production` reads which run and model key to check from
`$MUSEBENCH_ROOT/results/production_run.json` (`{"run_dir": "results/<run>", "model_key": "<key>"}`),
so private model names stay in the data root. It checks that the score lines cover the eval cohort
once each and are unchanged since the import, that scored plus failed tracks equal the cohort with
an explicit decision on every score, that the keyed leak annotation is present, coherent and current,
and that the saved report equals a rebuild split by the keyed field. It is part of `verify.py all`.

## Licenses

- Collection: CC BY-SA 3.0 (inherited from JamendoMaxCaps).
Expand Down
228 changes: 186 additions & 42 deletions scripts/musebench/leak_map.py
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,81 @@ def scan_sonics(fake_dir: Path) -> dict[str, int]:
return {f.name.rsplit(".", 1)[0]: f.stat().st_size for f in fake_dir.iterdir() if f.is_file()}


# --------------------------------------------------------------------------- lineage manifests
# A detector's real training history is a chain of training manifests (one per generation), not a
# candidate-pool directory. Their rows name files by path and sometimes by a derived track id
# (codec cells: "cell:<cell>:<source>:<id>"), and derived rows point back with origin_* fields.
LINEAGE_SPLITS = ("train", "val", "test", "hardneg", "codec_cells", "rows", "tracks", "items")
LINEAGE_ROW_KEYS = ("path", "track_id", "origin_path", "origin_track_id", "source_path")
SONICS_STEM = re.compile(r"fake_\d+_(suno|udio)")
UNKNOWN_SIZE = -1 # a SONICS name taken from a manifest whose file is not on this host


def lineage_rows(data: object) -> list:
"""Rows of one training manifest: a list, or a dict of split lists (a dict of lists is flattened)."""
if isinstance(data, list):
return data
if not isinstance(data, dict):
return []
out: list = []
for split in LINEAGE_SPLITS:
v = data.get(split)
if isinstance(v, dict):
v = [r for sub in v.values() if isinstance(sub, list) for r in sub]
if isinstance(v, list):
out.extend(v)
return out


def lineage_identity(value: object) -> tuple[str, object] | None:
"""('sonics', stem) | ('fma', id) | ('mtg', id) | None for one path or track id.

The identity is the last path component (after the last ':' for cell ids), up to the first '.'.
FMA and MTG-Jamendo are told apart by the corpus name appearing anywhere in the string; both
name their files by a bare numeric track id."""
if not value:
return None
s = str(value)
stem = s.rsplit("/", 1)[-1].rsplit(":", 1)[-1].split(".")[0]
low = s.lower()
if SONICS_STEM.match(stem):
return "sonics", stem
if "fma" in low and stem.isdigit():
return "fma", int(stem)
if "jamendo" in low and stem.isdigit():
return "mtg", int(stem)
return None


def load_lineage(paths: list[Path]) -> tuple[dict[str, int], set[int], set[int], dict[str, int]]:
"""(SONICS stem -> size, FMA ids, MTG-Jamendo ids, rows per manifest) over a lineage of manifests."""
sonics: dict[str, int] = {}
fma: set[int] = set()
mtg: set[int] = set()
per_manifest: dict[str, int] = {}
for mp in paths:
rows = lineage_rows(json.loads(Path(mp).read_text(encoding="utf-8")))
if not rows:
raise ValueError(f"{mp}: no rows (unsupported manifest shape)")
for r in rows:
values = [r] if isinstance(r, str) else [r.get(k) for k in LINEAGE_ROW_KEYS] if isinstance(r, dict) else []
for v in values:
ident = lineage_identity(v)
if ident is None:
continue
kind, key = ident
if kind == "sonics":
if key not in sonics:
p = Path(str(v))
sonics[key] = p.stat().st_size if p.is_absolute() and p.is_file() else UNKNOWN_SIZE
elif kind == "fma":
fma.add(key)
else:
mtg.add(key)
per_manifest[str(mp)] = len(rows)
return sonics, fma, mtg, per_manifest


def ffprobe_duration(path: Path) -> float | None:
try:
out = subprocess.run(["ffprobe", "-v", "error", "-show_entries", "format=duration",
Expand Down Expand Up @@ -170,9 +245,14 @@ def annotate(tracks: list[dict], *, sonics: dict[str, int], fma_ids: set[int],
size = sonics.get(base)
if size is not None:
stats["sonics_name_match"] += 1
same = size == t["size"]
stats["sonics_size_equal" if same else "sonics_size_differs"] += 1
seen_family[t["family"]] = f"SONICS file {base} ({'same size' if same else 'size differs'})"
if size == UNKNOWN_SIZE: # named by a training manifest, file not on this host
stats["sonics_size_unknown"] += 1
note = "size not checked: training file not on this host"
else:
same = size == t["size"]
stats["sonics_size_equal" if same else "sonics_size_differs"] += 1
note = "same size" if same else "size differs"
seen_family[t["family"]] = f"SONICS file {base} ({note})"
elif t["source"] == "musebench_human_fma_ood":
fid = fma_id(base)
if fid is not None and fid in fma_ids:
Expand Down Expand Up @@ -229,13 +309,33 @@ def annotate(tracks: list[dict], *, sonics: dict[str, int], fma_ids: set[int],
return dict(stats)


def per_source_counts(tracks: list[dict]) -> dict[str, dict]:
def keyed(field: str, model_key: str | None) -> str:
"""Field name for a model-specific annotation: seen_in_training -> seen_in_training_<key>."""
return f"{field}_{model_key}" if model_key else field


def annotate_keyed(tracks: list[dict], model_key: str, **kwargs) -> dict:
"""Run annotate() on copies and store its fields under model-specific names, so the rows keep any
earlier (unkeyed) annotation untouched. Idempotent: stale keyed fields are dropped first."""
if not model_key or not re.fullmatch(r"[a-z0-9_]+", model_key):
raise ValueError(f"model key must be [a-z0-9_]+, got {model_key!r}")
copies = [dict(t) for t in tracks]
stats = annotate(copies, **kwargs)
for t, c in zip(tracks, copies):
for f in LEAK_FIELDS:
t.pop(keyed(f, model_key), None)
if f in c:
t[keyed(f, model_key)] = c[f]
return stats


def per_source_counts(tracks: list[dict], seen_field: str = "seen_in_training") -> dict[str, dict]:
out: dict[str, Counter] = defaultdict(Counter)
for t in tracks:
c = out[t["source"]]
c["total"] += 1
c["seen" if t["seen_in_training"] else "unseen"] += 1
c[f"{t['musebench_role']}_{'seen' if t['seen_in_training'] else 'unseen'}"] += 1
c["seen" if t[seen_field] else "unseen"] += 1
c[f"{t['musebench_role']}_{'seen' if t[seen_field] else 'unseen'}"] += 1
return {k: {"total": v["total"], "seen": v["seen"], "unseen": v["unseen"],
"eval_seen": v["eval_seen"], "eval_unseen": v["eval_unseen"],
"train_pool_seen": v["train_pool_seen"], "train_pool_unseen": v["train_pool_unseen"]}
Expand All @@ -256,73 +356,117 @@ def atomic_write_json(path: Path, obj: object) -> None:
os.replace(tmp, path)


def report_rel(model_key: str | None) -> Path:
"""leak/leak_report.json for the unkeyed (first) annotation, leak/leak_report_<key>.json otherwise."""
return REPORT_REL if not model_key else REPORT_REL.with_name(f"leak_report_{model_key}.json")


def load_references(refs: dict) -> dict:
"""Reference identities from a references JSON.

Two shapes: a *candidate pool* (``sonics_fake_dir`` + ``fma_id_lists`` + ``mtg_pool_folders``)
or a *lineage* (``lineage_manifests``: the training manifests of every generation in the model's
weight lineage, read row by row). Both need ``mtg_meta_tsv`` and ``mtg_audio_dir``."""
mtg_meta = load_mtg_meta(Path(refs["mtg_meta_tsv"]))
mtg_all = scan_mtg_audio(Path(refs["mtg_audio_dir"]), None)
if refs.get("lineage_manifests"):
sonics, fma_ids, mtg_ids, per_manifest = load_lineage([Path(p) for p in refs["lineage_manifests"]])
return dict(kind="lineage", sonics=sonics, fma_ids=fma_ids, mtg_meta=mtg_meta, mtg_all=mtg_all,
mtg_pool={i: p for i, p in mtg_all.items() if i in mtg_ids}, fma_files={},
corpora={"lineage_manifests": per_manifest, "sonics_stems": len(sonics),
"fma_ids": len(fma_ids), "mtg_ids": len(mtg_ids),
"mtg_ids_local": sum(1 for i in mtg_ids if i in mtg_all),
"mtg_meta_tsv": refs["mtg_meta_tsv"], "mtg_audio_dir": refs["mtg_audio_dir"]})
sonics = scan_sonics(Path(refs["sonics_fake_dir"]))
fma_ids: set[int] = set()
for p in refs["fma_id_lists"]:
fma_ids |= load_fma_ids(Path(p))
fma_files = scan_fma_audio([Path(p) for p in refs.get("fma_audio_dirs", [])])
mtg_pool = scan_mtg_audio(Path(refs["mtg_audio_dir"]), set(refs["mtg_pool_folders"]))
return dict(kind="pool", sonics=sonics, fma_ids=fma_ids, mtg_meta=mtg_meta, mtg_all=mtg_all,
mtg_pool=mtg_pool, fma_files=fma_files,
corpora={
"sonics_fake_dir": {"path": refs["sonics_fake_dir"], "files": len(sonics)},
"fma_id_lists": {"paths": refs["fma_id_lists"], "track_ids": len(fma_ids),
"audio_dirs": refs.get("fma_audio_dirs", []),
"local_files": len(fma_files)},
"mtg_jamendo": {"meta_tsv": refs["mtg_meta_tsv"], "audio_dir": refs["mtg_audio_dir"],
"pool_folders": refs["mtg_pool_folders"],
"pool_tracks": len(mtg_pool), "local_tracks": len(mtg_all)}})


def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(description="Annotate the MuseBench manifest with training overlap.")
ap.add_argument("--root", type=Path, default=DEFAULT_ROOT)
ap.add_argument("--references", type=Path, default=None,
help="JSON describing the detector and its training pool "
help="JSON describing the detector and its training pool or lineage "
"(default: <root>/leak/references.json)")
ap.add_argument("--model-key", default=None,
help="write seen_in_training_<key> etc. and leak/leak_report_<key>.json, keeping the "
"unkeyed annotation of another model intact")
ap.add_argument("--no-duration-check", action="store_true")
args = ap.parse_args(argv)
ref_path = args.references or args.root / "leak" / "references.json"
refs = json.loads(ref_path.read_text(encoding="utf-8"))

sonics = scan_sonics(Path(refs["sonics_fake_dir"]))
fma_ids: set[int] = set()
for p in refs["fma_id_lists"]:
fma_ids |= load_fma_ids(Path(p))
fma_files = scan_fma_audio([Path(p) for p in refs.get("fma_audio_dirs", [])])
mtg_meta = load_mtg_meta(Path(refs["mtg_meta_tsv"]))
mtg_dir = Path(refs["mtg_audio_dir"])
mtg_pool = scan_mtg_audio(mtg_dir, set(refs["mtg_pool_folders"]))
mtg_all = scan_mtg_audio(mtg_dir, None)
ref = load_references(refs)
key = args.model_key
seen_field = keyed("seen_in_training", key)

man_path = args.root / MANIFEST_REL
manifest = json.loads(man_path.read_text(encoding="utf-8"))
tracks = manifest["tracks"]
stats = annotate(tracks, sonics=sonics, fma_ids=fma_ids, mtg_meta=mtg_meta,
mtg_pool=mtg_pool, mtg_all=mtg_all,
generators_in_training=refs["generators_in_training"],
fma_files=fma_files, duration=None if args.no_duration_check else ffprobe_duration)
manifest["metadata"]["leak_annotation"] = {
"model": refs["model"]["name"], "report": str(args.root / REPORT_REL),
"annotated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())}
kwargs = dict(sonics=ref["sonics"], fma_ids=ref["fma_ids"], mtg_meta=ref["mtg_meta"],
mtg_pool=ref["mtg_pool"], mtg_all=ref["mtg_all"],
generators_in_training=refs["generators_in_training"], fma_files=ref["fma_files"],
duration=None if args.no_duration_check else ffprobe_duration)
stats = annotate_keyed(tracks, key, **kwargs) if key else annotate(tracks, **kwargs)
rep_rel = report_rel(key)
note = {"model": refs["model"]["name"], "report": str(args.root / rep_rel), "seen_field": seen_field,
"annotated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())}
if key:
manifest["metadata"].setdefault("leak_annotations", {})[key] = note
else:
manifest["metadata"]["leak_annotation"] = {k: v for k, v in note.items() if k != "seen_field"}
atomic_write_json(man_path, manifest)

lineage = ref["kind"] == "lineage"
report = {
"model": refs["model"],
"model_key": key,
"seen_field": seen_field,
"training_manifests": refs["training_manifests"],
"reference_corpora": {
"sonics_fake_dir": {"path": refs["sonics_fake_dir"], "files": len(sonics)},
"fma_id_lists": {"paths": refs["fma_id_lists"], "track_ids": len(fma_ids),
"audio_dirs": refs.get("fma_audio_dirs", []),
"local_files": len(fma_files)},
"mtg_jamendo": {"meta_tsv": refs["mtg_meta_tsv"], "audio_dir": refs["mtg_audio_dir"],
"pool_folders": refs["mtg_pool_folders"],
"pool_tracks": len(mtg_pool), "local_tracks": len(mtg_all)},
},
"reference_corpora": ref["corpora"],
"method": {
"suno_udio": "SONICS file-name match (fake_<id>_<gen>_<seed>), size compared",
"fma": "FMA track id (leading digits of the file name) in the training FMA id lists; "
"size compared with a local FMA copy where one exists",
"suno_udio": "SONICS file-name match (fake_<id>_<gen>_<seed>), size compared"
+ (" when the training file is on this host" if lineage else ""),
"fma": ("FMA track id (numeric file stem of a training row whose path or id names FMA)"
if lineage else
"FMA track id (leading digits of the file name) in the training FMA id lists; "
"size compared with a local FMA copy where one exists"),
"jamendo": "normalised (title, artist) vs MTG-Jamendo raw.meta.tsv, restricted to the "
f"pool folders; duration within {DURATION_TOLERANCE_S}s"
+ ("MTG ids named by training rows" if lineage else "pool folders")
+ f"; duration within {DURATION_TOLERANCE_S}s"
+ (" (not checked)" if args.no_duration_check else ""),
"inheritance": "transforms inherit from their clean track; co-created remixes from "
"their human source; caption-conditioned generations do not",
"scope": "candidate pool of the training dataset builder: an upper bound on exposure",
"scope": ("union of the train/val/test rows of every manifest in the weight lineage: an upper "
"bound on exposure for the manifests listed, a lower bound if one is missing"
if lineage else
"candidate pool of the training dataset builder: an upper bound on exposure"),
},
"generators_in_training": refs["generators_in_training"],
"caveats": refs.get("caveats", []),
"match_stats": stats,
"per_source": per_source_counts(tracks),
"totals": dict(Counter("seen" if t["seen_in_training"] else "unseen" for t in tracks)),
"per_source": per_source_counts(tracks, seen_field),
"totals": dict(Counter("seen" if t[seen_field] else "unseen" for t in tracks)),
"eval_seen": sum(1 for t in tracks if t[seen_field] and t.get("musebench_role") == "eval"),
"manifest": str(man_path),
"manifest_fingerprint": manifest_fingerprint(tracks),
"created_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
}
atomic_write_json(args.root / REPORT_REL, report)
print(json.dumps({"match_stats": stats, "totals": report["totals"]}, indent=1))
atomic_write_json(args.root / rep_rel, report)
print(json.dumps({"match_stats": stats, "totals": report["totals"], "eval_seen": report["eval_seen"],
"report": str(args.root / rep_rel)}, indent=1))
return 0


Expand Down
Loading
Loading