feat(jp): support Japanese release assets and split APKs

README.en.md

@@ -1,5 +1,9 @@
 # nnnotes？！
 
+Japanese-release support includes anonymous version discovery, authenticated CDN downloads, gzip catalogs,
+snapshot-isolated caches and split APKs. See the [JP guide](https://github.com/MetaSekaiLab/nnnotes/blob/main/docs/jp.md)
+for configuration and validation limits.
+
 [简体中文](https://github.com/MetaSekaiLab/nnnotes/blob/main/README.md) | [English](https://github.com/MetaSekaiLab/nnnotes/blob/main/README.en.md)
 
 nnnotes is an offline data toolkit for the game files of BanG Dream! Our Notes: it reads Addressables catalogs,

README.md

@@ -52,6 +52,9 @@ nnnotes config check             # 每项设置的来源和格式是否有效，
 
 ## 完成度
 
+日服现已支持匿名版本查询、CDN 认证下载、gzip catalog、资源路径与缓存隔离，以及 Android 分包读取。
+配置、命令和日服验证范围见 [日服支持](docs/jp.md)。
+
 以下结果基于台服 1.0.1（zh-Hant）的全部数据：
 
 | 部分 | 状态 |

docs/configuration.md

@@ -1,5 +1,7 @@
 # Configuration
 
+For Japanese-release endpoints, catalog snapshots and split APKs, see [Japanese release](jp.md).
+
 nnnotes has no built-in keys, server addresses or data paths. Every setting comes from one of three sources, and a
 command that needs a setting nobody gave stops before it does any work.
 
@@ -87,6 +89,10 @@ exist when they are given.
 | `[catalog] region` | `NNNOTES_CATALOG_REGION` | `--region` | name of one `[servers.<region>]` table |
 | `[catalog] language` | `NNNOTES_CATALOG_LANGUAGE` | `--language` | catalog language: the `<language>` of `catalog_main_<language>.bin`: `ja`, `en`, `zh-Hant`, `zh-Hans` or `ko`; also the client language of `live`, `story` and `web` (the text field, fonts and line spacing of their UI) and of the model labels of `web --live2d` |
 | `[servers.<region>] name` | `NNNOTES_SERVERS_<REGION>_NAME` | — | label of the region in `browse` (default: the region name) |
+| `[servers.<region>] provider` | `NNNOTES_SERVERS_<REGION>_PROVIDER` | — | `international` or `jp`; empty selects `jp` for the region named jp, international otherwise |
+| `[servers.<region>] client_version` | `NNNOTES_SERVERS_<REGION>_CLIENT_VERSION` | — | per-region API client version; overrides `[client] version`, then falls back to the region's APK versionName |
+| `[servers.<region>] apk` | `NNNOTES_SERVERS_<REGION>_APK` | `--apk` | per-region APK, APKS/XAPK or directory with base.apk; adjacent splits of base.apk are read automatically; the flag overrides it |
+| `[servers.<region>] catalog` | `NNNOTES_SERVERS_<REGION>_CATALOG` | `--catalog` | per-region catalog file; JP needs the matching `<file>.source.json`; the flag overrides it |
 | `[servers.<region>] cdn` | `NNNOTES_SERVERS_<REGION>_CDN` | — | CDN base URL of the region (a trailing `/` is ignored) |
 | `[servers.<region>] languages` | `NNNOTES_SERVERS_<REGION>_LANGUAGES` | — | catalog languages `browse` lists: a TOML array of strings; comma-separated in the environment |
 | `[servers.<region>] api` | `NNNOTES_SERVERS_<REGION>_API` | — | API root of the region: `https://host[:port]` (TLS, port 443 by default), `host[:port]`, or `http://host[:port]` for a plain-text local server; no path |

docs/jp.md

@@ -0,0 +1,114 @@
+# Japanese release
+
+JP uses the same catalog, bundle and master parsers, with a separate version discovery and CDN download path.
+The anonymous `MasterdataService/Version` response provides the master version, Android asset version/hash and
+CDN authentication. No player account or login is needed.
+
+## Configuration
+
+Use your own client addresses and encryption settings. As elsewhere in nnnotes, neither addresses nor keys are
+built into the package. Add a region table with these settings to your private configuration:
+
+```toml
+[catalog]
+region = "jp"
+language = "ja"
+
+[servers.jp]
+provider = "jp"
+name = "JP"
+api = "https://YOUR_API_HOST"
+cdn = "https://YOUR_CDN_HOST"
+client_version = "YOUR_CURRENT_CLIENT_VERSION"
+languages = ["ja"]
+apk = "/path/to/jp/base.apk"
+master = "/path/to/jp/master"
+
+[paths]
+cache = "/path/to/cache"
+```
+
+Keep `[bundle] key/nonce_seed` and `[master] key/iv` in your configuration. Master keys are unnecessary when
+reading already decoded tables with `music-data --decoded-master`.
+
+`provider` defaults to `jp` for the region named `jp`, and to `international` otherwise. An alias such as
+`servers.japan` needs `provider = "jp"`. `client_version` overrides `[client] version`; when neither is set,
+nnnotes reads the APK versionName. Use the current accepted JP client version: an old extracted APK can still
+contain readable local data even when its version is no longer accepted by the API.
+
+The `cdn` setting is also the allowed CDN origin for authentication. A different origin in Version fails before
+any authenticated download. The JP transport uses HTTPS, follows no redirects, keeps authentication in memory,
+and refreshes it once on 401/403. A 429 does not cause an authentication-refresh loop. Credentials are not written
+to catalog metadata or task files.
+
+## APKs and split data
+
+`apk` accepts:
+
+- a normal APK;
+- `base.apk` with adjacent `split_*.apk` / `config.*.apk` files;
+- a directory containing `base.apk` and its splits;
+- an `.apks` or `.xapk` archive containing `base.apk` and its splits.
+
+JP stores boot data in base.apk and the embedded catalog/master/bundles in the Unity data split. Both are read
+through the same APK-set interface. `datapack.unity3d` is loaded alongside `data.unity3d` for external resources
+such as materials and shaders. Native libraries are also read from the splits. No repacked APK is required.
+
+`[servers.<region>] apk` and `catalog` override their `[paths]` equivalents; explicit `--apk` / `--catalog` flags
+still win. For mixed-release chart builds, configure separate master directories and APK sets. JP stories and
+models must use a separate site directory from international releases, since their model IDs may overlap.
+
+## Commands
+
+```sh
+nnnotes --region jp master version
+nnnotes --region jp master download --latest -o work/jp-master-bin
+nnnotes master decode work/jp-master-bin -o work/jp-master
+nnnotes --region jp catalog --prefix Live/MusicScore/ --limit 10
+nnnotes --region jp pull Live/MusicScore/0001/0001_00
+nnnotes --region jp catalogs fetch
+nnnotes --region jp export --select key:Image/Jacket/jacket_temporary --views none -o out/jp-sample
+nnnotes --region jp music-data --decoded-master --no-bgm -o out/jp-music-data.json
+```
+
+For `--decoded-master`, keep `MasterManifest.json` with the decoded tables (copy it from the download directory,
+or use a snapshot from your masterdata service). `--no-bgm` omits song audio length inspection; omit that flag to
+include it. `master version` reports `resourceHash` for JP as well as the two version fields. `music-data` records
+the JP hash in `provenance.catalog.resourceHash`.
+
+JP catalogs are `catalog_main.bin` in a version/hash directory, without a language suffix. The HTTP payload can
+be gzip; nnnotes keeps its original bytes and SHA-256, and bounds decompression before parsing. Remote bundle and
+CRI locations use `{Fwk.Resource.RemoteAssetDir}`. These are indexed, resolved and downloaded through the selected
+snapshot, including in `browse`, `catalogs`, `export`, `plan` and `run-stage`.
+
+The cache lives under `<cache>/jp/<CDN-origin-digest>/<asset-version>/<asset-hash>/`. It cannot reuse the
+international catalog cache. Embedded files are further isolated under `apk/<APK-catalog-SHA256>/`, so an APK
+update cannot reuse old local bundles when the CDN snapshot stays unchanged. Each cached `catalog_main.bin` has a `catalog_main.bin.source.json` with its SHA-256
+and public source metadata. For an offline `--catalog` or `catalogs import`, copy the pair together. Credentials
+are reacquired only when a missing file must be downloaded.
+
+Catalog database records include this source metadata, and JP catalog identity includes version/hash even when
+the catalog bytes remain the same. Old international records retain their previous IDs. Cached historical files
+can be read offline; downloading a missing historical file fails if Version now selects a different snapshot.
+Refresh the catalog to use the new snapshot. Historical local bundles require the APK set matching the imported
+APK catalog. A JP master download likewise rejects a different current master snapshot.
+
+## Validation and current limits
+
+The synthetic tests use local gRPC and HTTP servers, including gzip catalogs, split packages, authentication
+rotation, redirects, 429, truncated responses, source isolation, APK-only and hash-only updates, offline replay
+and invalid paths.
+
+Live validation on 2026-09-30 used JP client 1.0.4 and local JP Android 1.0.3 data:
+
+- 238 master tables downloaded with manifest hashes and decoded successfully;
+- a 73-note chart and a 512×512 jacket read successfully;
+- one Live2D runtime model and one complete story exported; the story's 22 FLAC files passed full FFmpeg decoding;
+- selected `export` produced six files without failed tasks;
+- `music-data --decoded-master --no-bgm` built 85 songs / 340 charts, including the pinned deck model's
+  statistics, with no unplayable chart; output passed JSON Schema validation.
+
+This does not establish full JP story/Live2D/movie export coverage, browser playback, every BGM, or native scoring
+parity. JP's text tables retain five language columns, but most translated values are empty; story web builds
+default to Japanese. Existing Bili chat skin view rules still refer to international `_iconAssetPath` fields;
+JP rows with a different schema report no-value and are not represented as verified skin mappings.

docs/schema/music-data.schema.json

@@ -71,6 +71,11 @@
               ],
               "description": "the resource version recorded for the catalog in the catalog store, null when none is recorded"
             },
+            "resourceHash": {
+              "type": "string",
+              "pattern": "^[0-9a-f]{32}$",
+              "description": "JP Android asset directory hash, when the catalog source is JP"
+            },
             "sha256": {
               "$ref": "#/$defs/sha256",
               "description": "SHA-256 of the remote catalog file the charts were read with"

docs/stages.md

@@ -13,7 +13,8 @@ The asset export is built from three layers:
 ## Stages
 
 Every stage is at version 1, except `unity.census` (version 2: it lists the scripts animation clip bindings name)
-and `cri.movie` (version 2: every stream kind, codec and channel of a USM).
+and `cri.movie` (version 2: every stream kind, codec and channel of a USM), and `catalog.index` (version 2:
+gzip input and Japanese remote asset placeholders).
 
 | Stage | Subject | Inputs | Outputs |
 |---|---|---|---|

src/nnnotes/addressables.py

@@ -15,6 +15,8 @@
 from __future__ import annotations
 
 import hashlib
+import gzip
+import io
 import struct
 from dataclasses import dataclass, field
 from html import escape
@@ -32,6 +34,23 @@
 CATALOG_MAGIC = 0x0DE38942
 CATALOG_VERSION = 2
 NONE = 0xFFFFFFFF
+REMOTE_PREFIX = "{Fwk.Resource.RemoteAssetDir}/"
+MAX_CATALOG = 128 * 1024 * 1024
+
+
+def catalog_bytes(data: bytes) -> bytes:
+    """Normalize HTTP gzip or raw binary catalog bytes with a bounded expanded size."""
+    if len(data) > MAX_CATALOG:
+        raise ValueError("catalog exceeds the size limit")
+    if data.startswith(b"\x1f\x8b"):
+        try:
+            with gzip.GzipFile(fileobj=io.BytesIO(data)) as stream:
+                data = stream.read(MAX_CATALOG + 1)
+        except (OSError, EOFError):
+            raise ValueError("invalid gzip catalog") from None
+        if len(data) > MAX_CATALOG:
+            raise ValueError("expanded catalog exceeds the size limit")
+    return data
 
 
 @dataclass(frozen=True)
@@ -56,6 +75,9 @@ def decrypt(data: bytes, filename: str, key: BundleKey) -> bytes:
 
 def remote_path(internal_id: str) -> str | None:
     """The path below the CDN base of a remote location (an absolute URL: everything after its host), else None."""
+    if internal_id.startswith(REMOTE_PREFIX):
+        from .jp import relative_path
+        return "/" + relative_path(internal_id[len(REMOTE_PREFIX):])
     scheme, sep, rest = internal_id.partition("://")
     if not sep or not scheme.isalpha():
         return None
@@ -65,6 +87,7 @@ def remote_path(internal_id: str) -> str | None:
 
 def parse(data: bytes) -> list[dict]:
     """Every location of a binary catalog: {offset, primary_key, internal_id, dependencies (location offsets)}."""
+    data = catalog_bytes(data)
     def u32(offset):
         return struct.unpack_from("<I", data, offset)[0]
 
@@ -229,6 +252,7 @@ def request_options(self, offset: int) -> dict:
 def parse_header(data: bytes) -> dict:
     """The header of a binary catalog: magic, version, keysOffset, locatorId, instanceProvider, sceneProvider
     ({id, assembly, type, data}), initObjects (the same, the providers the catalog initializes), buildResultHash."""
+    data = catalog_bytes(data)
     if len(data) < HEADER.size:
         raise ValueError("unsupported catalog format")
     magic, version, keys, locator, instance, scene, init, build = HEADER.unpack_from(data, 0)
@@ -244,6 +268,7 @@ def parse_header(data: bytes) -> dict:
 def parse_keys(data: bytes) -> list[dict]:
     """The key table of a binary catalog in stored order (ContentCatalogData.ResourceLocator.KeyData {u32 key object,
     u32 location set}): {"key": value, "type": the key's type name, "locations": location offsets}."""
+    data = catalog_bytes(data)
     header = parse_header(data)
     buf = _Buffer(data)
     out = []
@@ -267,6 +292,7 @@ def parse_locations(data: bytes) -> list[dict]:
     offsets), dependency hash (i32), extra data (an ObjectTypeData) and resource type (a TypeSerializer.Data).
     extra_data of bundle and raw file locations is an AssetBundleRequestOptions: {type, hash, bundleName, crc,
     bundleSize, timeout, redirectLimit, retryCount, flags and the flag bits by name}."""
+    data = catalog_bytes(data)
     header = parse_header(data)
     buf = _Buffer(data)
     offsets: set[int] = set()
@@ -309,6 +335,7 @@ class Region:
     label: str
     cdn: str = field(repr=False)
     languages: list[str]
+    config: object = field(default=None, repr=False)
 
 
 class Handler(BaseHTTPRequestHandler):
@@ -329,6 +356,12 @@ def listing(self, label: str, rows) -> None:
     def catalog(self, region: Region, language: str):
         key = (region.name, language)
         if key not in self.server.catalogs:
+            if region.config is not None and region.config.provider(region.name) == "jp":
+                from .jp import open_catalog
+                cat = open_catalog(region.config, region.name, bundle_key=self.server.bundle_key)
+                self.server.catalogs[key] = browse(cat.entries)
+                self.server.jp_catalogs[key] = cat
+                return self.server.catalogs[key]
             catalog = self.server.cache / "catalogs" / region.name / f"catalog_main_{language}.bin"
             if not catalog.exists():
                 with urlopen(region.cdn + f"/asset/Android/{catalog.name}", timeout=60) as response:
@@ -358,7 +391,13 @@ def do_GET(self):
         if language not in region.languages:
             self.send_error(404)
             return
-        bundles, files = self.catalog(region, language)
+        from .gameapi import GameApiError
+        from .config import ConfigError
+        try:
+            bundles, files = self.catalog(region, language)
+        except (GameApiError, ConfigError, ValueError):
+            self.send_error(502, "Catalog could not be loaded")
+            return
         base = f"/{quote(region.name)}/{quote(language)}/"
         if len(parts) > 2 and parts[2] == "download":
             ident = parts[3] if len(parts) == 4 else ""
@@ -367,8 +406,17 @@ def do_GET(self):
                 return
             entry = bundles[int(ident)]
             name = entry["internal_id"].rsplit("/", 1)[1]
-            with urlopen(region.cdn + remote_path(entry["internal_id"]), timeout=60) as response:
-                data = decrypt(response.read(), name, self.server.bundle_key)
+            try:
+                cat = getattr(self.server, "jp_catalogs", {}).get((region.name, language))
+                if cat is not None:
+                    from .catalog import Bundle
+                    data = cat.fetch(Bundle(int(ident), entry["internal_id"], name, True)).read_bytes()
+                else:
+                    with urlopen(region.cdn + remote_path(entry["internal_id"]), timeout=60) as response:
+                        data = decrypt(response.read(), name, self.server.bundle_key)
+            except (GameApiError, ConfigError, ValueError):
+                self.send_error(502, "Resource could not be downloaded")
+                return
             self.send_response(200)
             self.send_header("Content-Type", "application/octet-stream")
             self.send_header("Content-Disposition", "attachment; filename*=UTF-8''" + quote(name))
@@ -404,6 +452,7 @@ def serve(regions: list[Region], bundle_key: BundleKey, cache: Path, port: int,
     server.bundle_key = bundle_key
     server.cache = Path(cache)
     server.catalogs = {}
+    server.jp_catalogs = {}
     print(f"nnnotes browse: serving on {host}:{port}", flush=True)
     try:
         server.serve_forever()

src/nnnotes/apkset.py

@@ -0,0 +1,77 @@
+"""Read a base APK and its splits, an APK directory, or an APKS/XAPK archive.
+
+The base manifest wins over split manifests. Resource members are resolved across the set;
+no merged APK is produced. Nested APKs are spooled so large asset packs need not stay in RAM.
+"""
+from __future__ import annotations
+
+import shutil
+import tempfile
+import zipfile
+from contextlib import ExitStack
+from pathlib import Path
+
+
+class ApkSet:
+    def __init__(self, source):
+        self.source = source
+        self._stack = ExitStack()
+        self._members = {}
+
+    def __enter__(self):
+        try:
+            self._open()
+            return self
+        except BaseException:
+            self._stack.close()
+            raise
+
+    def _add(self, source):
+        archive = self._stack.enter_context(zipfile.ZipFile(source))
+        for info in archive.infolist():
+            self._members.setdefault(info.filename, (archive, info))
+
+    def _open(self):
+        if hasattr(self.source, "read"):
+            self._add(self.source)
+            return
+        path = Path(self.source)
+        if path.is_dir():
+            base = path / "base.apk"
+            if not base.is_file():
+                raise FileNotFoundError("APK directory has no base.apk")
+            paths = [base] + sorted(p for p in path.glob("*.apk") if p != base)
+        elif path.suffix.lower() in (".apks", ".xapk"):
+            outer = self._stack.enter_context(zipfile.ZipFile(path))
+            names = [n for n in outer.namelist() if n.lower().endswith(".apk")]
+            bases = [n for n in names if Path(n).name == "base.apk"]
+            if len(bases) != 1:
+                raise ValueError("APK archive must contain exactly one base.apk")
+            for name in bases + sorted(n for n in names if n not in bases):
+                tmp = self._stack.enter_context(tempfile.SpooledTemporaryFile(max_size=16 * 1024 * 1024))
+                with outer.open(name) as stream:
+                    shutil.copyfileobj(stream, tmp)
+                tmp.seek(0)
+                self._add(tmp)
+            return
+        else:
+            paths = [path]
+            if path.name == "base.apk":
+                paths += sorted(p for p in path.parent.glob("*.apk")
+                                if p.name.startswith(("split_", "config.")))
+        for apk in paths:
+            self._add(apk)
+
+    def namelist(self):
+        return list(self._members)
+
+    def infolist(self):
+        return [info for _, info in self._members.values()]
+
+    def read(self, name):
+        name = name.filename if isinstance(name, zipfile.ZipInfo) else name
+        archive, info = self._members[name]
+        return archive.read(info)
+
+    def __exit__(self, *args):
+        return self._stack.__exit__(*args)