fix(translate): load BabelDOC's fonts once per process

CLAUDE.md

@@ -117,7 +117,7 @@ singleton with no job id, so it is polled via `GET /setup/status`.
 | `engine_assets.py` | Single source of truth for "the offline engine is installed" |
 | `config/` | `ConfigManager` + Pydantic `AppSettings` (`providers` map, `ModelRef`s) |
 | `providers/` | `registry.py` — one `ProviderSpec` per provider, the source of every per-provider rule (stdlib-only); `listing.py` — one model lister per protocol; `catalog.py` — the model catalog cache and `list_models` |
-| `processors/` | `PDFProcessor` (BabelDOC), `page_selection.py`, `pdf_pages.py`, `pdf_cache.py`, `doc_layout_cache.py` |
+| `processors/` | `PDFProcessor` (BabelDOC), `page_selection.py`, `pdf_pages.py`, `pdf_cache.py`, `doc_layout_cache.py`, `font_verify_cache.py`, `font_cache.py` |
 | `translators/` | `BaseTranslator` + OpenAI/Gemini/Anthropic/Argos, `factory.py`, `capabilities.py`, `rate_limiter.py`, `translation_cache.py`, `param_compat.py`, `usage_estimate.py` |
 | `rag/` | `rag_chain.py` (`EnhancedRAGChain`), `conversation.py` (`ChatTurn`, stdlib-only), `ingest.py` (PDF → chunks, page kinds), `ocr.py`, `retrieval.py` (RRF, page evidence), `citations.py`, `page_images.py`, `vector_store.py`, `index_spec.py` (embedding model + `EmbeddingSpec`s), `onnx_embeddings.py`, `keyword_search.py` |
 | `benchmarks/rag/` | Corpus, questions with gold pages, harness (baseline from a `main` worktree), results — `README.md` there has the numbers |
@@ -262,6 +262,13 @@ rule has, unless noted, a test. They are the things most easily undone by
   (and a hit is linked back), via `pdf_cache.link_or_copy`, so **never rewrite
   a translated PDF in place**. Write a new file and `os.replace` it, or the
   write reaches the cache entry too.
+- **BabelDOC's fonts are loaded once per process** (`processors/font_cache.py`).
+  BabelDOC builds a `FontMapper` per stage, 6–7 a page, and each loaded all 34
+  embedding fonts (266 MB) into reference cycles only a full GC frees — on a
+  book that ran the sidecar out of memory. The shared constructor is a copy of
+  BabelDOC's, installed only on a babeldoc in `VERIFIED_BABELDOC`: re-check it
+  against `fontmap.py` when bumping babeldoc. Sharing a font across pages relies
+  on PyMuPDF never releasing the GIL; re-check that when bumping PyMuPDF.
 
 **Data and chat** → `data-and-chat.md`
 - Three stores under one root: `pdfusion.db` (system of record), `vectors/`
@@ -537,9 +544,9 @@ isolation (real ChromaDB under `tmp_path`), chat ingestion (real PDFs built with
 PyMuPDF), OCR respacing, rank fusion and page evidence, citation checking,
 page images, multimodal and streamed `generate()` per SDK, the answer chain's
 image, citation, fence and conversation rules, answer ratings, the benchmark's
-interval statistics, chunking and rolling-PDF assembly,
-the data root per platform, packaging config agreement, and the viewer's pure
-half.
+interval statistics, chunking and rolling-PDF assembly, BabelDOC's shared
+fonts, the data root per platform, packaging config agreement, and the viewer's
+pure half.
 
 Not covered: `processors/processor.py` proper, chat quality with the real
 embedding/OCR models and a real LLM (that is what `benchmarks/rag/` measures —

docs/architecture/translation-pipeline.md

@@ -97,33 +97,31 @@ A failed load leaves the module global `None` rather than caching the failure,
 so the next chunk retries; `run_one_chunk`'s `except BaseException` still routes
 it into `babeldoc_chunk_error`.
 
-**What this does not fix, and why chunk size is a live question.** The layout model was never the largest per-chunk fixed cost.
+**What this does not fix, and why chunk size is a live question.** The layout
+model was never the largest per-chunk fixed cost: the fonts were.
 `FontMapper.__init__` sha3_256-verifies and loads **all 34 embedding fonts
 (266 MB)** for `lang_out`, and BabelDOC memoizes neither — not `fontmap.py`, not
 `assets.py`. It constructs one per stage: `ILCreater`, `ParagraphFinder`,
-`StylesAndFormulas`, `ILTranslator` (a second inside `ILTranslatorLLMOnly`),
-`Typesetting`, `PDFCreater` (a second for scanned-page detection) — 7 per page
-on the Argos path, 8 on the LLM path. The hash is now remembered (next section);
-what is left is ~0.26 s per `FontMapper` of `pymupdf.Font` parsing, ≈2 s per
-chunk, which this cache does not touch either. So "the reload is fixed" is not
-on its own a reason to shrink a chunk — shrinking one multiplies that fixed cost
-by the extra chunks it creates, and on the Argos path
-(`_MAX_PARALLEL_CHUNKS_ARGOS = 2`) that can outweigh what the cache saves.
-`_PAGES_PER_CHUNK_ARGOS` is 1 today, like the LLM path; measure an end-to-end
-Argos run before trusting that, or before changing either constant. What a
-larger chunk buys is amortized per-chunk fixed overhead in general, not the ONNX
-reload specifically.
+`StylesAndFormulas`, `ILTranslator`, `Typesetting`, `PDFCreater` (a second for
+scanned-page detection) — 6 per page on the Argos path, 7 on the LLM path,
+counted on real runs. The hash is now remembered and the fonts loaded once per
+process (the next two sections), but every chunk is still a whole BabelDOC run,
+with its own IR, rendering and PDF save. `_PAGES_PER_CHUNK_ARGOS` is 1 today,
+like the LLM path; measure an end-to-end Argos run before changing either
+constant. What a larger chunk buys is amortized per-chunk fixed overhead in
+general, not the ONNX reload specifically.
 
 ## A font that passed its hash check is not hashed again
 
 A `FontMapper` took ~1.3 s, of which ~1.0 s was `assets.verify_file` hashing
 the 34 fonts (measured alone: 0.99 s; parsing them alone: 0.13 s; the ~0.26 s a
-mapper still costs with the hash remembered is that parse plus the rest of the
-constructor, under the contention of a real run). Traced over real one-page
-LLM-path runs on two different documents: **272 checks per page — 34 fonts × 8
-mappers (238 on Argos, with 7) — against the same 34 (file, expected hash)
-pairs**, none derived from the document; every pair passed,
-and every file was byte-identical before and after both runs. Only a process's
+mapper still cost with the hash remembered was that parse plus the rest of the
+constructor, under the contention of a real run — a parse only a process's first
+mapper pays now, as the next section explains). That is **238 checks per page
+— 34 fonts × 7 mappers (204 on Argos, with 6) — against the same 34 (file,
+expected hash) pairs**, none derived from the document; traced over real
+one-page runs on two different documents, every pair passed, and every file was
+byte-identical before and after both runs. Only a process's
 first check can find anything (a missing or corrupt font, which BabelDOC then
 re-downloads); the rest re-hash unchanged bytes.
 
@@ -163,6 +161,67 @@ usually loses that race and just logs, which is what makes the symptom
 intermittent rather than absent — do not read "it works here" as "the ordering
 is safe".)
 
+## Each font is loaded once per process
+
+A `FontMapper` also *loads* all 34 fonts, each into a `pymupdf.Font` that holds
+the whole file: 266 MB for any target language, because BabelDOC's
+`__add_fallback_to_font_family` appends every other family's fonts to each one.
+At 6 mappers a page (7 on the LLM path) that is 1.6 GB of fonts a page, and
+BabelDOC never gave it back on time. The constructor memoizes by wrapping bound
+methods in `lru_cache` — its own `has_char`/`map_in_type`, and every font's
+`has_glyph`/`char_lengths` — so each mapper and each font sits in a reference
+cycle that only CPython's full collection frees. Full collections are scheduled
+by counts of objects, which font buffers barely move, and nothing in BabelDOC,
+Argos or PDFusion calls `gc.collect()`.
+
+On a long document that was an out-of-memory kill, whatever the translator. A
+346-page book on Argos (2 pages in flight) took the sidecar to 12 GB, and the
+kernel killed it 238 pages in. Replayed through the real pipeline, four pages
+took the process from 0.26 GB to 6.5 GB with no full collection in between; the
+LLM path's 6 pages in flight (`cpu // 2` on 12 cores; a stub translator that
+returns its input, so no request was made) passed 6.5 GB before its first page
+finished. Two things about the numbers in BabelDOC's log: "Peak memory usage" is
+the whole sidecar and its children, not one sub-job, and on Linux it
+over-counts. Between its two-second PSS samples it adds up the RSS of the
+sidecar and of each forked PDF-save child, and a forked child's RSS includes
+every page it shares with the sidecar.
+
+`processors/font_cache.py` replaces `FontMapper.__init__` with a copy of
+BabelDOC's whose font loop takes each font from a process-wide cache, which gives
+a font BabelDOC's per-font attributes and memos once, at its first load. On the
+same book afterwards, its 301 pages with text went through in 23 minutes with
+the sidecar never above 2.4 GB — 1,806 mappers sharing one set of fonts — and a
+page translated before and after the change came out the same, every word in
+the same place. Memory still drifts up by about 1.7 MB a page: most likely the
+rolling PDF, rebuilt whole after every page (43.5 MB by the end), and per-page
+garbage that still waits for a full collection, such as the source PDF's own
+fonts, which `ILCreater` memoizes the same way.
+
+- **Sharing is safe because nothing writes to a font after construction.** The
+  stages only read a mapper (`has_char`, `map`, `fontid2font`, `base_font`,
+  `add_font`), and PyMuPDF never releases the GIL — neither `_mupdf.so` nor
+  `_extra.so` imports `PyEval_SaveThread` — so MuPDF calls from different pages
+  run one at a time. Python does run inside a call, in PyMuPDF's warning and
+  error callbacks, where threads can switch: the same exposure as the one MuPDF
+  context every page already shares. The shared memos are keyed by a character
+  (and, for `char_lengths`, a font size) and capped at 10,240 entries each;
+  `char_lengths` hands every caller the same list, which they only read.
+- **The fonts stay.** ~266 MB is resident from the first translation until the
+  sidecar exits. The cache is never pruned: every mapper wants all 34.
+- **It is a copy of BabelDOC 0.5.24's constructor**, so `install()` patches only
+  a BabelDOC whose version is in `VERIFIED_BABELDOC`; any other keeps its own
+  constructor and logs a warning. babeldoc is pinned to a range, and a copy that
+  no longer matches could break every stage, where an unpatched one only costs
+  the memory. `test_font_cache.py` compares the copy with the installed
+  constructor attribute by attribute, and fails by name on an unverified
+  version: re-check the copy against `fontmap.py` and extend the set when
+  bumping babeldoc. Re-check the GIL claim above when bumping PyMuPDF.
+- **Installed by `routes/translation.py:_load_engine`**, which every job runs
+  before BabelDOC builds anything. The class is patched, not the module's name
+  for it: each stage imported `FontMapper` by name.
+- **A failed load is not remembered**, so a font BabelDOC re-downloads is loaded
+  by the next mapper.
+
 ## Page ranges
 
 A translation covers the pages the toolbar's Pages box names (#33) — `1-20, 35`,

src/desktop_pdf_translator/api/routes/translation.py

@@ -81,13 +81,16 @@ def _load_engine() -> tuple[type, type]:
     beats that thread — or a warm-up that failed, which it does silently —
     lands here instead. Callers must run it off the event loop.
     """
-    from ...processors import font_verify_cache
+    from ...processors import font_cache, font_verify_cache
     from ...processors.events import EventType
     from ...processors.processor import PDFProcessor
 
     # Normally done by the warm thread too; a click that beats it would
     # otherwise re-hash every font in every stage of the first job.
     font_verify_cache.install()
+    # Before any stage builds a `FontMapper`: BabelDOC's own loads all 34 fonts
+    # for each, and on a book that runs the sidecar out of memory.
+    font_cache.install()
     return EventType, PDFProcessor
 
 

src/desktop_pdf_translator/processors/font_cache.py

@@ -0,0 +1,182 @@
+"""BabelDOC's embedding fonts, loaded once per process instead of once per
+`FontMapper`.
+
+BabelDOC builds a `FontMapper` in every pipeline stage — six per page on the
+Argos path, seven on the LLM path, which also runs scanned-page detection —
+and each one reads every embedding font of the target language into a
+`pymupdf.Font`: all 34 of them, 266 MB, for any language, because BabelDOC
+appends every other family's fonts to each family as fallbacks. None of it was
+freed when a stage ended. The constructor memoizes by wrapping bound methods
+in `lru_cache` — its own `has_char`/`map_in_type`, and each font's
+`has_glyph`/`char_lengths` — so every mapper and every font sits in a
+reference cycle, which only a full collection frees, and full collections
+follow counts of objects, which 266 MB of font buffers barely move. On a book
+that ran the sidecar out of memory: replayed through the real pipeline, four
+pages took it from 0.26 GB to 6.5 GB without one full collection.
+
+`install()` replaces `FontMapper.__init__` with a copy of BabelDOC's whose font
+loop asks `_shared_font` instead. Sharing is safe because nothing in BabelDOC
+writes to a mapper's fonts after construction — the stages only read them —
+and PyMuPDF never releases the GIL, so MuPDF calls from different pages run
+one at a time (Python runs inside a call only in PyMuPDF's warning and error
+callbacks; see "Each font is loaded once per process" in
+docs/architecture/translation-pipeline.md). The cost is that the fonts stay:
+~266 MB from the first translation until the sidecar exits.
+
+The copy is babeldoc 0.5.24's constructor, so it is only installed on a version
+in `VERIFIED_BABELDOC`: babeldoc is pinned to a range, and a newer constructor
+the copy doesn't match could break every stage, where leaving it alone only
+costs the memory. `tests/test_font_cache.py` compares the copy with the
+installed BabelDOC's own constructor; re-check it, then extend the set, when
+bumping babeldoc.
+
+Kept free of babeldoc imports at module scope, like `doc_layout_cache.py`.
+"""
+
+from __future__ import annotations
+
+import functools
+import logging
+import os
+import threading
+from pathlib import Path
+from typing import Any, Mapping
+
+logger = logging.getLogger(__name__)
+
+# The babeldoc versions whose `FontMapper.__init__` `_sharing_init` was checked
+# against.
+VERIFIED_BABELDOC = frozenset({"0.5.24"})
+
+# Marks the constructor this module installed, so `install()` can tell it from
+# BabelDOC's own — or from a test's, which `monkeypatch` restores.
+_MARK = "_pdfusion_shared_fonts"
+
+_install_lock = threading.Lock()
+_fonts_lock = threading.Lock()
+# Font file path → its one `pymupdf.Font`. Never pruned: every mapper of every
+# page wants all of them.
+_fonts: dict[str, Any] = {}
+
+
+def _shared_font(font_file_name: str, font_path: Path, font_metadata: Mapping[str, Any]):
+    """The process's `pymupdf.Font` for `font_path`, loaded on first use and
+    given what BabelDOC's constructor gives each font it loads — once, where
+    BabelDOC did it again for every mapper. A load that raises is not
+    remembered, so the next mapper tries again."""
+    key = os.fspath(font_path)
+    with _fonts_lock:
+        font = _fonts.get(key)
+        if font is None:
+            import pymupdf
+
+            font = pymupdf.Font(fontfile=key)
+            font.has_glyph = functools.lru_cache(maxsize=10240, typed=True)(font.has_glyph)
+            font.char_lengths = functools.lru_cache(maxsize=10240, typed=True)(
+                font.char_lengths
+            )
+            font.font_id = font_file_name
+            font.font_path = font_path
+            font.ascent_fontmap = font_metadata["ascent"]
+            font.descent_fontmap = font_metadata["descent"]
+            font.encoding_length = font_metadata["encoding_length"]
+            _fonts[key] = font
+    return font
+
+
+def _sharing_init(fontmap, original):
+    """BabelDOC 0.5.24's `FontMapper.__init__`, with its font loop asking
+    `_shared_font`. Everything else is BabelDOC's, line for line, and reaches
+    BabelDOC's `assets` through the fontmap module, as the original does."""
+
+    def __init__(self, translation_config):
+        self.translation_config = translation_config
+        assert translation_config.primary_font_family in [
+            None,
+            "serif",
+            "sans-serif",
+            "script",
+        ]
+        self.primary_font_family = fontmap.PrimaryFontFamily.from_str(
+            translation_config.primary_font_family,
+        )
+
+        font_family = fontmap.assets.get_font_family(translation_config.lang_out)
+        self.font_file_names = []
+        for k in (
+            "normal",
+            "script",
+            "fallback",
+            "base",
+        ):
+            self.font_file_names.extend(font_family[k])
+
+        self.fonts = {}
+        self.fontid2fontpath = {}
+        for font_file_name in self.font_file_names:
+            if font_file_name in self.fontid2fontpath:
+                continue
+            font_path, font_metadata = fontmap.assets.get_font_and_metadata(font_file_name)
+            self.fonts[font_file_name] = _shared_font(font_file_name, font_path, font_metadata)
+            self.fontid2fontpath[font_file_name] = font_path
+
+        self.normal_font_ids = font_family["normal"]
+        self.script_font_ids = font_family["script"]
+        self.fallback_font_ids = font_family["fallback"]
+        self.base_font_ids = font_family["base"]
+        self.fontid2fontpath["base"] = self.fontid2fontpath[font_family["base"][0]]
+
+        self.fontid2font = {f.font_id: f for f in self.fonts.values()}
+
+        self.fontid2font["base"] = self.fontid2font[self.base_font_ids[0]]
+
+        self.normal_fonts = [self.fontid2font[font_id] for font_id in self.normal_font_ids]
+        self.script_fonts = [self.fontid2font[font_id] for font_id in self.script_font_ids]
+        self.fallback_fonts = [
+            self.fontid2font[font_id] for font_id in self.fallback_font_ids
+        ]
+
+        self.base_font = self.fontid2font["base"]
+
+        self.type2font = {
+            "normal": self.normal_fonts,
+            "script": self.script_fonts,
+            "fallback": self.fallback_fonts,
+            "base": [self.base_font],
+        }
+
+        self.has_char = functools.lru_cache(maxsize=10240, typed=True)(self.has_char)
+        self.map_in_type = functools.lru_cache(maxsize=10240, typed=True)(
+            self.map_in_type
+        )
+
+    __init__.__wrapped__ = original  # type: ignore[attr-defined]
+    setattr(__init__, _MARK, True)
+    return __init__
+
+
+def install() -> None:
+    """Have every BabelDOC `FontMapper` share one loaded copy of each font.
+
+    Patches the class, not the module's name for it: every stage imported
+    `FontMapper` by name, so only the class object reaches them all.
+    Idempotent. Blocking (imports BabelDOC); call off the event loop.
+    """
+    import babeldoc
+    from babeldoc.format.pdf.document_il.utils import fontmap
+
+    with _install_lock:
+        current = fontmap.FontMapper.__init__
+        if getattr(current, _MARK, False):
+            return
+        # A babeldoc that reports no version is as unchecked as a new one, and
+        # leaving its constructor alone only costs memory.
+        version = getattr(babeldoc, "__version__", None)
+        if version not in VERIFIED_BABELDOC:
+            logger.warning(
+                "babeldoc %s is not one font_cache was checked against; its "
+                "FontMapper keeps loading every font for every stage",
+                version,
+            )
+            return
+        fontmap.FontMapper.__init__ = _sharing_init(fontmap, current)