mbforge

mbforge is a Go CLI that downloads the latest MusicBrainz JSON dump and builds a flattened libSQL metadata database for Oxygen/Chromakopia. mbforge sync keeps that database current by applying the hourly incremental JSON dumps from the MusicBrainz Live Data Feed, and mbforge prune applies the deletions and merges those dumps cannot convey.

It is intentionally narrow:

• Full builds plus hourly incremental sync and prune
• No raw JSON blobs
• No annotation import

Commands

mbforge build

• Resolves the latest dump directory from LATEST

• Downloads the selected *.tar.xz archives into --dump-dir

• Streams xz -> tar -> JSONL without extracting the full dumps to disk

• --mirror accepts any base URL serving the upstream path layout (LATEST + <dir>/<entity>.tar.xz), including an Azure blob container URL carrying a SAS query string, which is appended to every request

• If the mirror also serves <dir>/<entity>.chunks.json, the entity is downloaded as pre-split zstd chunks (<dir>/<entity>.chunks/<name>.jsonl.zst) and decompressed on all cores instead of one serial xz stream; pass --no-chunks to force the original archives. The manifest format is:

  {
    "format": "mbforge-chunks/1",
    "entity": "release",
    "metadata": {
      "json_dumps_schema_number": "...",  // JSON_DUMPS_SCHEMA_NUMBER
      "replication_sequence": "...",      // REPLICATION_SEQUENCE
      "schema_sequence": "...",           // SCHEMA_SEQUENCE
      "timestamp": "..."                  // TIMESTAMP
    },
    "chunks": [{"name": "000000.jsonl.zst", "size": 123}]
  }

  Each chunk is a standalone zstd stream of complete JSONL lines from the tar's mbdump/<entity> member (lines never span chunks; ordering across chunks is irrelevant since import is unordered). metadata carries the four dump metadata files from the tar root, so the chunked path never touches the tar.

• Imports artists (incl. artist–artist relationships), labels, works (incl. recording→work links), release groups, releases, embedded recordings/tracks, and standalone recordings

• Defers secondary index creation until after the bulk load

• Optionally builds a full-text search index with --search-index; track rows are excluded by default (see mbforge search-index below), pass --search-index-tracks to include them

• Writes _meta with dump and replication metadata

• Runs PRAGMA optimize and VACUUM

mbforge sync

• Applies MusicBrainz Live Data Feed incremental JSON dumps to a database built by mbforge build
• Requires a MetaBrainz Live Data Feed access token (--token or $MBFORGE_LDF_TOKEN)
• Resumes from _meta.replication_sequence and fetches every packet up to the current one reported by the replication-info endpoint
• Applies one packet per transaction and advances _meta.replication_sequence in the same transaction, so an interrupted sync can simply be rerun
• A 404 for an entity archive means the packet has no changes for that entity and is skipped
• Runs in WAL mode with synchronous=NORMAL, so the database stays safe for concurrent readers (the aggressive build-time pragmas are never used); note the database file remains in WAL mode afterwards
• Updates the full-text search index rows for changed entities when the search index exists; track rows are only refreshed when _meta.search_index_tracks is not "false", so an index built without tracks stays that way
• Creates any tables added by newer mbforge versions (labels, works, relationship link tables) on databases built before them, so older databases keep syncing; their historical label/work rows still require a full rebuild to backfill
• Stops with an error instructing a full rebuild if a packet's SCHEMA_SEQUENCE differs from _meta.schema_sequence
• Pass --with-prune to run mbforge prune afterwards in the same invocation

The incremental JSON dumps do not carry deletions or merges; mbforge prune covers those. With sync and prune both running hourly, full rebuilds are only needed when the MusicBrainz schema changes (a SCHEMA_SEQUENCE bump, roughly twice a year) or for disaster recovery.

mbforge prune

• Applies entity deletions and merges from the hourly RAW replication packets (replication-<SEQ>-v2.tar.bz2, dbmirror2 format) of the MusicBrainz Live Data Feed, using the same access token as sync
• Deletes removed artists, labels, works, release groups, releases, recordings, and tracks, including their child rows and search-index rows
• Records merges in a gid_redirects table (entity_type, old_mbid, new_mbid) so chromakopia's catalog can forward lookups of merged MBIDs, and deletes the merged-away entity's rows
• Resumes from _meta.prune_replication_sequence, seeded from _meta.replication_sequence on the first run; applies one packet per transaction, so an interrupted prune can simply be rerun
• Packets reference merge targets by MusicBrainz row id, which the mbforge schema does not store. Most targets resolve from row data in the same packet; the rest are kept in gid_redirects_pending, retried against ids harvested from later packets (mb_row_ids), and finally resolved via the MusicBrainz web service at one request per second (--resolve-remote=false to disable)
• Must not run concurrently with sync against the same database: both are single-writer jobs. Run them back to back from one scheduler slot, or simply use mbforge sync --with-prune

mbforge info

• Prints database path, size, core row counts, and _meta values

mbforge search

• Searches artists, labels, works, release groups, releases, recordings, and tracks from a single free-form query
• Accepts artist names, track titles, album titles, MBIDs, ISRCs, ISWCs, and release barcodes
• Labels and works are only covered by the fast indexed path, not the slow SQL fallback
• Uses the full-text search index when present
• Falls back to the older slower SQL path when the search index is absent
• With an index built without track rows (the default), the fast path returns no track results; recording rows cover title search, and the slow SQL fallback still searches tracks when no index exists

mbforge search-index

• Builds or rebuilds the full-text search index on an existing database
• Useful when you already finished a long build run without --search-index
• Excludes track rows by default: they are near-duplicates of recording titles and roughly 57M of the ~91M index rows, so leaving them out keeps the index small enough for a small node's page cache; pass --search-index-tracks to include them
• Records the choice in _meta.search_index_tracks so mbforge sync does not reintroduce track rows on an index built without them

mbforge version

• Prints build-time version metadata

Requirements

• Go 1.24+
• CGO-enabled build environment
• Network access to the MusicBrainz dump mirror

Build

go build ./cmd/mbforge

Example:

mbforge build \
  --output /mnt/nvme/metadata.db \
  --dump-dir /mnt/nvme/mbdump \
  --workers 16 \
  --batch-size 5000 \
  --search-index \
  --verbose

Import a subset of entities:

mbforge build \
  --output ./musicbrainz.db \
  --entities artist,release-group

Apply the latest Live Data Feed packets (including deletions and merges) to an existing database:

export MBFORGE_LDF_TOKEN=your-metabrainz-token
mbforge sync \
  --db /mnt/nvme/metadata.db \
  --dump-dir /mnt/nvme/mbdump \
  --with-prune

Inspect a finished database:

mbforge info --db ./musicbrainz.db

Search across the main entities:

mbforge search --db ./musicbrainz.db "nirvana"

Build the fast search index on an existing database:

mbforge search-index --db ./musicbrainz.db

Pipeline

Import order:

1. artist
2. label
3. work
4. release-group
5. release
6. recording

Important detail: release.tar.xz contains the overwhelming majority of recordings at media[].tracks[].recording. recording.tar.xz only covers the standalone subset. mbforge imports both, and uses INSERT OR IGNORE for recordings so the two sources can coexist safely.

The same pattern applies to works: work.tar.xz is imported first so its full rows win, and works embedded in recording work-relations (in both recording.tar.xz and release.tar.xz) only fill gaps via INSERT OR IGNORE.

The importer uses multiple JSON parse workers and a single batched SQLite writer to avoid write contention.

Schema Notes

• Areas are flattened inline instead of being normalized into separate tables.
• Labels are first-class rows in labels (with label_aliases, label_tags, label_genres); release_labels.label_mbid joins releases to them while keeping the flattened label_name/catalog_number columns.
• Works live in works (with work_aliases, work_iswcs, work_tags); recording_works links recordings to works from the performance relationship, so covers/live versions of the same song connect through the shared work. attributes holds a sorted JSON array like ["cover","live"] (empty string when the relation has no attributes).
• recording_works rows are extracted from both sides of the relationship (recording lines and work lines) and deduplicate via INSERT OR IGNORE; rows from a work line may reference recordings imported later in the pipeline, which is why recording_works.recording_mbid carries no foreign-key clause.
• artist_relationships stores artist–artist relationships (band membership, collaborations, teachers, …) with one row per direction as rendered in each artist's dump line: query it by artist_mbid to get everything that artist participates in. begin_date/end_date/attributes are part of the primary key because the same pair can hold the same relationship type over several stints (empty strings stand in for NULLs there, as in release_labels).
• Secondary indexes are created after bulk import for speed.
• SQLite does not allow expressions inside a table primary key, so nullable key parts from the draft schema are normalized to empty strings for artist_aliases.locale and release_labels key columns.

Azure Build VM

deploy/cloud-init.yaml is a starting point for the one-shot build VM path:

1. Install Go
2. Clone the repo
3. Build mbforge
4. Build metadata.db on local NVMe
5. Copy the finished database to an attached disk
6. Stop the VM

You still need to customize the final handoff step to your long-running sqld VM.