feat(music-data): read decoded master data with its manifest (--decoded-master)

README.en.md

@@ -49,7 +49,7 @@ nnnotes config check             # each setting's origin and whether it is valid
 | `player` | APK | render-related global settings (color space, quality levels, renderers) as JSON |
 | `live` | music ID + difficulty | a full chart directory: chart and runtime notes, 3D scene, note and effect assets, BGM and sounds, sound routing |
 | `web` | `--pair music:difficulty` (repeatable) or `--all`; `--live2d model` (repeatable) or `--all-live2d`; `--story episode` (repeatable) or `--all-stories`; `--region region` (repeatable) or `--all-regions` | an ournotes-player static site: shared player + per-chart / per-model / per-episode manifests + content-addressed assets; the Live2D models of the stories are built first as models, listed in `models.json`, and the story manifests reference them; one site can serve several regions, with listing texts in five languages; a story's interface texts are grouped by language, with TextMeshPro font assets generated from open fonts (the game's fonts with `--fonts game`); compressible assets (JSON, shaders, moc3, ...) are stored gzip encoded by default (`--compress br` / `none`) |
-| `music-data` | master data files (`--master-files` directory or `--apk-master`) | one JSON file with every song and chart: titles and credits in five languages, bands, vocal characters, category, tags, release time, score ranks, BGM length; per difficulty the level, note counts, BPM, chart times, skill events and fever ranges; and the chart statistics the deck model ournotes-deck (built into nnnotes) measures on its whole-live simulation (the no-skill score, the weight of every score-up skill kind at every position). `--full` adds the deck model's input: every chart's runtime notes and the master data tables about cards, skills, bonuses, scores and events ([format](https://github.com/MetaSekaiLab/nnnotes/blob/main/docs/music-data.md)) |
+| `music-data` | master data files (`--master-files` directory or `--apk-master`), or decoded master data with its manifest (`--decoded-master`) | one JSON file with every song and chart: titles and credits in five languages, bands, vocal characters, category, tags, release time, score ranks, BGM length; per difficulty the level, note counts, BPM, chart times, skill events and fever ranges; and the chart statistics the deck model ournotes-deck (built into nnnotes) measures on its whole-live simulation (the no-skill score, the weight of every score-up skill kind at every position). `--full` adds the deck model's input: every chart's runtime notes and the master data tables about cards, skills, bonuses, scores and events ([format](https://github.com/MetaSekaiLab/nnnotes/blob/main/docs/music-data.md)) |
 
 Export conventions:
 

README.md

@@ -40,7 +40,7 @@ nnnotes config check             # 每项设置的来源和格式是否有效，
 | `player` | APK | 渲染相关的全局设置（色彩空间、画质等级、渲染器）JSON |
 | `live` | 曲目 ID + 难度 | 完整谱面目录：谱面与运行时音符、3D 场景、音符与特效资源、BGM 与音效、声音路由 |
 | `web` | `--pair 曲目:难度`（可重复）或 `--all`；`--live2d 模型`（可重复）或 `--all-live2d`；`--story 剧情 ID`（可重复）或 `--all-stories`；`--region 区服`（可重复）或 `--all-regions` | ournotes-player 静态站点：共享播放器 + 每谱 / 每模型 / 每集剧情清单 + 内容寻址资源；剧情用到的 Live2D 模型先按模型构建并列入 `models.json`，剧情清单引用它们；一个站点可服务多个区服，列表文本含五种语言；剧情的界面文字按语言分组，字形由开源字体生成 TextMeshPro 字体资源（`--fonts game` 时用游戏字体）；可压缩的资源（JSON、着色器、moc3 等）默认以 gzip 存储，`--compress br` / `none` 可改 |
-| `music-data` | masterdata 文件（`--master-files` 目录或 `--apk-master`） | 全部歌曲与谱面的单个 JSON：五语标题与作词作曲编曲、乐队、演唱角色、分类、标签、上线时间、评级线、BGM 时长；每个难度的等级、音符数、BPM、谱面时间、技能事件与 fever 区间；以及组卡模型 ournotes-deck（内置于 nnnotes）在整场模拟上实测的谱面统计（无技能得分、每种加分技能在每个演出位的权重）。`--full` 另附组卡模型的输入：每张谱面的运行时音符与卡牌、技能、加成、分数、活动相关的 masterdata 表（[格式](docs/music-data.md)） |
+| `music-data` | masterdata 文件（`--master-files` 目录或 `--apk-master`），或带清单的解码 masterdata（`--decoded-master`） | 全部歌曲与谱面的单个 JSON：五语标题与作词作曲编曲、乐队、演唱角色、分类、标签、上线时间、评级线、BGM 时长；每个难度的等级、音符数、BPM、谱面时间、技能事件与 fever 区间；以及组卡模型 ournotes-deck（内置于 nnnotes）在整场模拟上实测的谱面统计（无技能得分、每种加分技能在每个演出位的权重）。`--full` 另附组卡模型的输入：每张谱面的运行时音符与卡牌、技能、加成、分数、活动相关的 masterdata 表（[格式](docs/music-data.md)） |
 
 导出约定：
 

docs/commands.md

@@ -670,8 +670,8 @@ in the printed summary and in `SITE.failures.json`, models that fail in the summ
 ## music-data
 
 ```
-nnnotes music-data (--master-files DIR | --apk-master) [--full] [--no-deck] [--seeds N] [--workers N]
-                   [--no-bgm] [--jackets DIR] -o FILE
+nnnotes music-data (--master-files DIR | --apk-master | --decoded-master) [--full] [--no-deck] [--seeds N]
+                   [--workers N] [--no-bgm] [--jackets DIR] -o FILE
 ```
 
 Writes one JSON file with every `MasterLiveMusic` song and its charts for one master data version: titles, readings
@@ -689,10 +689,13 @@ master data tables about cards, skills, bonuses, scores and events. `--no-deck`
 luck range and its threads. The master data is decoded from the files as served: `--master-files DIR` reads
 `DIR/MasterManifest.json` and the `.bin` files it lists (`master download`; the file's region is `[catalog] region`),
 `--apk-master` the same files inside `[paths] apk` (region `embedded`); each file is checked against the manifest's
-SHA-256. `--no-bgm` skips the cue sheets (every `bgm.length` is null). `--jackets DIR` also writes every song's
-jacket as `DIR/<jacket>.webp` (at most 320 px on the longer side). `FILE` ending in `.gz` is written
-gzip-compressed; the file is canonical: the same inputs and nnnotes version give the same bytes. Prints `{out,
-format, region, masterSource, masterVersion, songs, charts, deck, unplayable, full, bgm, jackets, bytes, fileBytes,
-sha256}`. A missing or unreadable input (a master data file, a column, a text id, a chart asset, a cue sheet or cue,
-a jacket), a chart the deck model cannot measure, or deck statistics that disagree with the chart facts stop the
-command with exit status 1 before the file is written. The format is described in [music-data.md](music-data.md).
+SHA-256. `--decoded-master` reads master data decoded elsewhere instead, without the master key: the `<Table>.json`
+files of the master data directory (`[paths] master`, `--master`) and the `MasterManifest.json` of the files they were
+decoded from, whose version and SHA-256 the file records (region `[catalog] region`). `--no-bgm` skips the cue sheets
+(every `bgm.length` is null). `--jackets DIR` also writes every song's jacket as `DIR/<jacket>.webp` (at most 320 px
+on the longer side). `FILE` ending in `.gz` is written gzip-compressed; the file is canonical: the same inputs and
+nnnotes version give the same bytes. Prints `{out, format, region, masterSource, masterVersion, songs, charts, deck,
+unplayable, full, bgm, jackets, bytes, fileBytes, sha256}`. A missing or unreadable input (a master data file, a
+column, a text id, a chart asset, a cue sheet or cue, a jacket), a chart the deck model cannot measure, or deck
+statistics that disagree with the chart facts stop the command with exit status 1 before the file is written. The
+format is described in [music-data.md](music-data.md).

docs/configuration.md

@@ -155,7 +155,7 @@ story text); it is not a setting.
 | `web --pair` / `--all` | as `live`, plus `[paths] player`, `node`; with `--region` / `--all-regions` each region's `[servers.<region>]` table (its `cdn` for what must be downloaded) and master data (`[servers.<region>] master`; `[paths] master` for at most one region) |
 | `web --live2d` / `--all-live2d` | catalog (bundles from the CDN of the site's first region), `[paths] apk`, `[paths] player`; not `node`; master data only for the model names (optional: without it `models.json` has no names) |
 | `web --player-only` / `--reingest-json` | `[paths] player` |
-| `music-data` | catalog, `[master] key` + `iv`; `--master-files` also `[catalog] region`; `--apk-master` also `[paths] apk`; `[paths] apk` (optional otherwise) for the client version |
+| `music-data` | catalog, `[master] key` + `iv` (not with `--decoded-master`); `--master-files` also `[catalog] region`; `--apk-master` also `[paths] apk`; `--decoded-master` also `[catalog] region` and master data with its `MasterManifest.json`; `[paths] apk` (optional otherwise) for the client version |
 | `export`, `plan` | the store (`[paths] store` or `[paths] cache`); catalog (bundles are fetched into the cache); `[paths] apk` for the bundles inside the APK (without it they are reported as `source.absent`); master data for `--views`; with `--catalog-version` an imported catalog version instead of the current catalog |
 | `run-stage` | the store; `[paths] cache` for inputs located in the cache; `--fetch` also what fetching needs (region, `cdn`, bundle key, `[paths] apk`) |
 | `catalogs list` / `import` / `diff`, `store verify` | the store; `import` reads the APK's catalog from `[paths] apk` when it is set |
@@ -216,6 +216,8 @@ nnnotes: game API call Version to [servers.tw] api failed: UNAVAILABLE (server u
   `nnnotes master download --latest` (or `--version <version>`) or on the game client's own files.
   `[servers.<region>] master` the same for one region: `nnnotes --region <region> master download --latest -o <dir>`,
   then `nnnotes master decode <dir> -o <region dir>` (the regions serve different master data versions).
+  `music-data --decoded-master` also reads the `MasterManifest.json` of the decoded files there (copy it from the
+  download directory; a published master data snapshot may carry it).
 - **CRI HCA keycode**: not a setting. `audio`, `story`, `live` and `web` read it from the APK's boot data;
   `nnnotes crikey` shows whether one was found and can write it as a `.hcakey` file for vgmstream.
 - **Tools**: [vgmstream](https://vgmstream.org/) (`vgmstream-cli`), [FFmpeg](https://ffmpeg.org/) and, for

docs/music-data.md

@@ -9,14 +9,18 @@ its extension module `nnnotes._deck`. The format is `nnnotes.music-data/1`; its
 [schema/music-data.schema.json](schema/music-data.schema.json).
 
 ```
-nnnotes music-data (--master-files DIR | --apk-master) [--full] [--no-deck] [--seeds N] [--workers N]
-                   [--no-bgm] [--jackets DIR] -o FILE
+nnnotes music-data (--master-files DIR | --apk-master | --decoded-master) [--full] [--no-deck] [--seeds N]
+                   [--workers N] [--no-bgm] [--jackets DIR] -o FILE
 ```
 
 - `--master-files DIR`: master data files as served, `DIR/MasterManifest.json` and the `.bin` files it lists
   (`nnnotes master download`). The file's `region` is `[catalog] region` (`--region`).
 - `--apk-master`: the master data files the APK ships (`assets/Master/` of `[paths] apk`, the same layout). The
   file's `region` is `embedded`.
+- `--decoded-master`: master data decoded elsewhere, the directory the other commands read (`[paths] master`,
+  `--master`, `[servers.<region>] master`): one `<Table>.json` per table (`nnnotes master decode`) and the
+  `MasterManifest.json` of the files they were decoded from, as a master data snapshot published with its manifest
+  carries it. The file's `region` is `[catalog] region`; no master key is needed.
 - `--full`: also write the deck model's input, every chart's runtime notes and the master data tables about cards,
   skills, bonuses, scores and events ([the deck input](#the-deck-input---full)), for tools that run a deck model of
   their own.
@@ -30,6 +34,9 @@ nnnotes music-data (--master-files DIR | --apk-master) [--full] [--no-deck] [--s
   page next to the file finds a song's jacket at `<DIR>/<song.jacket>.webp`.
 
 Each master data file is checked against the SHA-256 the manifest lists and decoded with `[master] key` and `iv`.
+With `--decoded-master` the tables are read as decoded, and the master data version and each file's SHA-256 are the
+manifest's: the decoded tables cannot be checked against the files as served, so the file records what the manifest
+lists (the same values as `--master-files` on those files).
 Charts are read from the catalog of `[catalog] language` (bundles fetched into the cache as for every command); the
 BGM length from the cue sheet's ACB (its `CueTable` and `WaveformTable`, no audio is decoded). `FILE` ending in `.gz`
 is written gzip-compressed. The command prints a summary (`songs`, `charts`, `deck`: the deck model's commit,
@@ -79,7 +86,7 @@ A **text** is an object with one string per language of `languages` (`{"ja": ...
 | `client.versionName`, `client.versionCode` | the APK's version name and code (null without `[paths] apk`) |
 | `catalog.resourceVersion` | the resource version recorded for the catalog in the catalog store (`nnnotes catalogs fetch` / `import`), null when none is recorded |
 | `catalog.sha256` | SHA-256 of the remote catalog file the charts were read with |
-| `master.source` | `api` (`--master-files`) or `embedded` (`--apk-master`) |
+| `master.source` | `api` (`--master-files`, `--decoded-master`: the region's files) or `embedded` (`--apk-master`) |
 | `master.version` | the `version` of the master data manifest |
 | `master.tables.<Table>.sha256` | SHA-256 of each table's file as served, before decoding: the song tables (`MasterLiveMusic`, `MasterLiveMusicScore`, `MasterText`, `MasterBand`, `MasterCharacter`, `MasterTag`, `MasterLiveMusicCategory`, `MasterSound`, `MasterSoundCueSheet`, `MasterLiveScoreRank`) and, when the deck model runs or with `--full`, the tables of [the deck input](#the-deck-input---full) |
 | `exporter.name`, `exporter.version` | `nnnotes` and its version |

docs/schema/music-data.schema.json

@@ -90,7 +90,7 @@
                 "api",
                 "embedded"
               ],
-              "description": "api: files as downloaded from the region; embedded: the files the APK ships"
+              "description": "api: the region's files (downloaded, or with --decoded-master decoded elsewhere with their manifest); embedded: the files the APK ships"
             },
             "version": {
               "type": [
@@ -101,7 +101,7 @@
             },
             "tables": {
               "type": "object",
-              "description": "per table read, the SHA-256 of its file as served (before decoding)",
+              "description": "per table read, the SHA-256 of its file as served (before decoding; with --decoded-master as the manifest lists it)",
               "additionalProperties": {
                 "type": "object",
                 "required": [

src/nnnotes/cli.py

@@ -36,8 +36,8 @@
                          [--story 10462 [--story ...] | --all-stories] [--story-languages en,ja] [--font en=<file>]
                          [--font emoji=<file>]
                          [--region <region> [--region ...] | --all-regions]
-    nnnotes music-data --master-files <master download dir> | --apk-master [--full] [--no-deck] [--no-bgm]
-                       [--jackets DIR] -o out/music-data.json[.gz]
+    nnnotes music-data --master-files <master download dir> | --apk-master | --decoded-master [--full] [--no-deck]
+                       [--no-bgm] [--jackets DIR] -o out/music-data.json[.gz]
     nnnotes export -o out/assets [--select group:<group> | key:<prefix> | bundle:<glob> ...] [--layout original,cas]
     nnnotes plan [--select ...] [--json] [--check] [--emit-tasks <dir>]
     nnnotes run-stage <task.json> [...]
@@ -646,9 +646,11 @@ def cmd_music_data(args, cfg):
         deck = None if args.no_deck else musicdata.Deck(seeds=args.seeds, workers=args.workers)
         if args.apk_master:
             src, region = deckdata.apk_master(apk), deckdata.EMBEDDED
+        elif args.decoded_master:                    # decoded elsewhere: no master key
+            src, region = deckdata.decoded_master(master_dir(cfg)), cfg.region()
         else:
             src, region = deckdata.master_files(Path(args.master_files)), cfg.region()
-        key = master_key(cfg)
+        key = None if src.decoded else master_key(cfg)
         cat = open_catalog(cfg)
         r = musicdata.export(Path(args.out), src, key, deckdata.catalog_fetch(cat),
                              None if args.no_bgm else musicdata.catalog_bgm(cat), region=region,
@@ -933,6 +935,9 @@ def target(m, what):
                    help="master data files as served: MasterManifest.json and the .bin files it lists "
                         "(`master download`)")
     g.add_argument("--apk-master", action="store_true", help="the master data files of base.apk ([paths] apk)")
+    g.add_argument("--decoded-master", action="store_true",
+                   help="decoded master data ([paths] master or --master) with the MasterManifest.json of the files "
+                        "it was decoded from; no master key")
     c.add_argument("--full", action="store_true",
                    help="also write the deck model's input: every chart's runtime notes and the master data tables "
                         "about cards, skills, bonuses, scores and events")