fix(translate): stop the sidecar's RAM climbing with every translated page

CLAUDE.md

@@ -122,7 +122,7 @@ singleton with no job id, so it is polled via `GET /setup/status`.
 | `rag/` | `rag_chain.py` (`EnhancedRAGChain`), `conversation.py` (`ChatTurn`, stdlib-only), `ingest.py` (PDF → chunks, page kinds), `ocr.py`, `retrieval.py` (RRF, page evidence), `citations.py`, `page_images.py`, `vector_store.py`, `index_spec.py` (embedding model + `EmbeddingSpec`s), `onnx_embeddings.py`, `keyword_search.py` |
 | `benchmarks/rag/` | Corpus, questions with gold pages, harness (baseline from a `main` worktree), results — `README.md` there has the numbers |
 | `storage/` | `sqlite.py` (pragmas, per-thread connections), `migrations.py`, `records.py` (`pdfusion.db`) |
-| `utils/` | `encryption.py`, `file_export.py`, `logging_setup.py`, `paths.py` |
+| `utils/` | `encryption.py`, `file_export.py`, `logging_setup.py`, `memory.py` (glibc `malloc_trim`), `paths.py` |
 
 ### HTTP API
 
@@ -269,6 +269,14 @@ rule has, unless noted, a test. They are the things most easily undone by
   BabelDOC's, installed only on a babeldoc in `VERIFIED_BABELDOC`: re-check it
   against `fontmap.py` when bumping babeldoc. Sharing a font across pages relies
   on PyMuPDF never releasing the GIL; re-check that when bumping PyMuPDF.
+- **PyMuPDF stays ≥ 1.27.2.** Older ones leak two lists per link in `insert_pdf`,
+  and `rebuild_rolling_pdf` re-inserts the whole document after every page:
+  O(pages²) on a book, invisible to gc. `test_pdf_pages.py` fails on a version
+  that leaks. Also keep it `<1.28`, which deprecates `fitz`.
+- **Each finished page ends with `release_freed_memory`** (`malloc_trim`,
+  glibc only), after its rebuild and `chunk_ready`. glibc kept ~420 MB of freed
+  heap otherwise. Never add `gc.collect()` there: ~277 ms a page, freed nothing.
+  (No unit test for the call site: it is in `processor.py`.)
 
 **Data and chat** → `data-and-chat.md`
 - Three stores under one root: `pdfusion.db` (system of record), `vectors/`
@@ -308,6 +316,10 @@ rule has, unless noted, a test. They are the things most easily undone by
   embeds means editing `rag/index_spec.py`. How a model is run — ONNX file,
   pooling, its `query: `/`passage: ` prefixes — is its `EmbeddingSpec` there;
   a wrong one gives worse vectors and no error.
+- **Line-break hyphens are joined by `ingest._join_broken_words`, which
+  reproduces `TEXT_DEHYPHENATE` exactly** — PyMuPDF 1.27 ignores the flag. Keep
+  it exact: the parity test's expected text is what 1.25.5 indexed, and a
+  "smarter" rule sparing compounds lost two benchmark questions.
 - **A follow-up is searched for as its standalone rewrite, never fused with
   the question as typed** (the raw words outvote it). The rewrite prompt has no
   answer-language instruction; the history is the same document's last
@@ -544,8 +556,9 @@ isolation (real ChromaDB under `tmp_path`), chat ingestion (real PDFs built with
 PyMuPDF), OCR respacing, rank fusion and page evidence, citation checking,
 page images, multimodal and streamed `generate()` per SDK, the answer chain's
 image, citation, fence and conversation rules, answer ratings, the benchmark's
-interval statistics, chunking and rolling-PDF assembly, BabelDOC's shared
-fonts, the data root per platform, packaging config agreement, and the viewer's
+interval statistics, chunking and rolling-PDF assembly (and that rebuilding it
+leaks nothing), BabelDOC's shared
+fonts, handing freed heap back to glibc, the data root per platform, packaging config agreement, and the viewer's
 pure half.
 
 Not covered: `processors/processor.py` proper, chat quality with the real

docs/architecture/data-and-chat.md

@@ -124,8 +124,32 @@ index (once per document, rag/ingest.py)            ask (rag/rag_chain.py)
   to be refused outright ("No text could be extracted"). OCR is slow — ~7 s a
   dense page on an i5 — and runs once, with per-page progress on the index job.
 - Text is NFKC-normalised (ligatures: "firmware" never matched "firmware")
-  and dehyphenated. `CHUNKER_VERSION` is "2", so every existing index is
-  rebuilt the next time chat opens its document.
+  and dehyphenated. Changing how a PDF becomes chunks means bumping
+  `CHUNKER_VERSION` (now "3"), which rebuilds every existing index the next
+  time chat opens its document.
+- **Line-break hyphens are joined by `ingest._join_broken_words`, not by
+  PyMuPDF.** `TEXT_DEHYPHENATE` stopped working in PyMuPDF 1.27, which the
+  translation memory fix requires (translation-pipeline.md, "PyMuPDF is at
+  least 1.27.2"). Without a replacement, 1,129 words on 278 of a 346-page
+  book's pages were indexed as halves ("trans- action"), which neither search
+  matches. The helper is the flag's rule reproduced exactly:
+  - A line ending in a hyphen loses it and runs into the next, unless the
+    line is only the hyphen.
+  - A hyphen is "-", U+00AD, U+2010 or U+2011, each checked against 1.25.5.
+    Word processors export the Unicode ones, and the benchmark has none.
+  - Checked chunk for chunk, the ingestor's output on 1.25.5 is identical
+    before and after the change on all seven text documents in the benchmark.
+  - `CHUNKER_VERSION` stays, even though 1.27's `find_tables` changes some
+    chunks (ResNet, below). An index built on 1.25.5 keeps working, and it was
+    the better of the two on that document. Rebuilding every user's index,
+    OCR included, for a one-question difference would cost more than it buys.
+  - The rule glues compounds ("Bose-" / "Einstein" → "BoseEinstein"). A
+    variant that joined only lowercase continuations was measured and lost two
+    questions on BERT, whose tables use "-" for an empty cell, so the exact
+    rule stayed.
+  - Retrieval-only hit@1, PyMuPDF 1.25.5 → 1.27: 0.857 → 0.838 with no join,
+    0.848 with the helper. The one question still lost (`p10`, ResNet) comes
+    from 1.27's table finder, not from the join.
 
 **Retrieval** (`EnhancedRAGChain.retrieve`) is local and costs one read of the
 index's chunks:

docs/architecture/testing-and-ci.md

@@ -14,7 +14,12 @@ CI gates.
   `generate()` per SDK and the answer chain's image and citation rules
   (`test_rag_*.py`, `test_multimodal_generate.py`).
   Splitting a document into chunks and assembling the rolling PDF are covered
-  against real PDFs (`test_pdf_pages.py`).
+  against real PDFs (`test_pdf_pages.py`), including that rebuilding a linked
+  document's rolling PDF again and again leaves no objects alive. The
+  PyMuPDF < 1.27.2 leak failed exactly that. `test_memory_release.py` covers
+  the glibc trim helper: it trims where glibc is, and is a no-op off Linux,
+  on musl and without a loadable libc. The call after each page is in
+  `processor.py` and is checked by hand.
   Cross-platform (#69), `test_storage_paths.py` covers the data root per
   platform — the marked cases run only where they apply, the shell-override
   ones run everywhere — `test_config_security.py` covers the keystore scheme

docs/architecture/translation-pipeline.md

@@ -192,10 +192,13 @@ a font BabelDOC's per-font attributes and memos once, at its first load. On the
 same book afterwards, its 301 pages with text went through in 23 minutes with
 the sidecar never above 2.4 GB — 1,806 mappers sharing one set of fonts — and a
 page translated before and after the change came out the same, every word in
-the same place. Memory still drifts up by about 1.7 MB a page: most likely the
-rolling PDF, rebuilt whole after every page (43.5 MB by the end), and per-page
-garbage that still waits for a full collection, such as the source PDF's own
-fonts, which `ILCreater` memoizes the same way.
+the same place. Memory still drifted up afterwards, by +1.25 MB a page past a
+~40-page warm-up. It was not garbage waiting for a full collection, which was
+the first guess: 29 full collections ran on their own over 129 pages, and a
+forced one at the end freed next to nothing. It had two causes: PyMuPDF
+leaking ~1,690 objects a page in the rolling-PDF rebuild (the next section),
+and glibc keeping ~410 MB of freed heap by the end of that run ("glibc's heap
+is trimmed after every page").
 
 - **Sharing is safe because nothing writes to a font after construction.** The
   stages only read a mapper (`has_char`, `map`, `fontid2font`, `base_font`,
@@ -215,12 +218,88 @@ fonts, which `ILCreater` memoizes the same way.
   the memory. `test_font_cache.py` compares the copy with the installed
   constructor attribute by attribute, and fails by name on an unverified
   version: re-check the copy against `fontmap.py` and extend the set when
-  bumping babeldoc. Re-check the GIL claim above when bumping PyMuPDF.
+  bumping babeldoc. Re-check the GIL claim above when bumping PyMuPDF (last
+  checked on 1.25.5 and 1.27.2.3).
 - **Installed by `routes/translation.py:_load_engine`**, which every job runs
   before BabelDOC builds anything. The class is patched, not the module's name
   for it: each stage imported `FontMapper` by name.
 - **A failed load is not remembered**, so a font BabelDOC re-downloads is loaded
   by the next mapper.
+- **The shared memos fill slowly and stop.** Over 129 pages the 34 fonts held
+  ~8,300 `has_glyph` / `char_lengths` entries, out of a 696,320 cap. That is
+  the few thousand lists that still appear on PyMuPDF 1.27. They are not a
+  leak, so don't chase them.
+
+## PyMuPDF is at least 1.27.2
+
+Before 1.27.2, PyMuPDF's C++ helper behind `annot_xrefs`
+(`_extra.JM_get_annot_xref_list`) leaks a list on every call. `Link.next`
+calls it twice per link, so `get_links()` on a page with 23 links leaked 46
+lists. `insert_pdf` copies every page's links through it.
+
+`rebuild_rolling_pdf` re-inserts the *whole* document after every page, so
+each rebuild leaked two lists for every link it copied. On the 346-page book
+that was ~2,120 lists (~150 KB) per rebuild while the book was still
+untranslated. Translated pages carry no links, so the count falls as they
+replace linked ones: ~1,690 live objects a page, averaged over a 129-page
+run. That is O(pages) per page and O(pages²) for a whole document. The lists
+carry one extra reference and no referrer, so gc never sees them as garbage.
+- 1.25.5 and 1.26.7 leak. 1.27.2.3 and 1.28.2 don't.
+- Measured after warm-up: +1.25 MB/page on 1.25.5, +0.18 on 1.27.2.3, and
+  live objects flat.
+- The slope is far bigger than the lists' own ~150 KB. The link between them
+  is the before/after measurement of the bump, not their size. Each leaked
+  list most likely keeps the memory around it from being returned — that
+  part is not measured.
+- `test_rebuilding_the_rolling_pdf_over_and_over_leaves_no_objects_behind`
+  guards it. 20 rebuilds of a 10-page, 40-link document left 1,278 objects on
+  1.25.5 and 0 on 1.27.2.3. Pages without links never leaked, which is why
+  the older rebuild tests could not catch it.
+- `<1.28` because 1.28 deprecates the `fitz` name every module imports.
+
+What else the bump changed:
+- `TEXT_DEHYPHENATE` is ignored, so chat joins line-break hyphens itself.
+  See data-and-chat.md.
+- `find_tables` reads more as tables. It turns ResNet's architecture figure
+  into half-empty two-column grids that the ingestor's mostly-empty guard lets
+  through, which costs one benchmark question.
+- `find_tables` prints a one-line `pymupdf_layout` notice to stdout once per
+  process. The shell drains stdout into shell.log.
+- The translation output was checked over 129 pages: it differs from 1.25.5
+  no more than two runs of 1.25.5 differ from each other. Argos's batching
+  depends on timing.
+- Speed is unchanged: 4.33/4.57 s a page before, 4.47/4.38 after. That was
+  A-B-B-A, 46 pages each, and measured together with the trim below.
+
+## glibc's heap is trimmed after every page
+
+glibc keeps what Python frees inside its arenas, and gives memory back on its
+own only from the very top of a heap. Untrimmed, on PyMuPDF 1.27, a 129-page
+Argos run ended ~420 MB above what `malloc_trim` returned it to. A forced
+`gc.collect()` freed next to nothing: the memory was already free, just never
+returned.
+
+So the consumer loop in `_process_with_babeldoc` ends each finished page with
+`utils/memory.release_freed_memory`, off the event loop. That is
+`malloc_trim(0)` on glibc, and a no-op on Windows, macOS and musl.
+- **Where it sits.** After the rolling rebuild, so the rebuild's document is
+  freed too. After `chunk_ready`, so the viewer's swap never waits on it.
+- **What it buys.** The trim lowers the plateau by ~420 MB. It hardly changes
+  the slope: that was the PyMuPDF leak.
+- **What it costs.** 8–22 ms a page; speed is in the section above.
+  `gc.collect()` is deliberately not called: it cost ~277 ms a page and freed
+  nothing.
+- **What it does not lower.** The peak of two pages in flight. That is
+  `processing.max_parallel_chunks`.
+- **Watch the trimmed baseline.** If it ever creeps across a long run,
+  something really leaks, and trimming only hides it.
+
+Recycling worker processes every N pages was weighed: it reclaims memory on
+any OS. It was not taken, because concurrent chunks share paragraph batches,
+the provider's rate limiter and the loaded layout model. Splitting them costs
+API calls, risks 429s and reloads the model per worker. Windows' heap returns
+large blocks as they are freed, so whether Windows needs anything at all is
+being measured (#134), not assumed.
 
 ## Page ranges
 

pyproject.toml

@@ -38,7 +38,12 @@ dependencies = [
     "cryptography>=41.0.0,<43.0.0",
 
     # PDF processing
-    "PyMuPDF>=1.25.1,<1.26.0",
+    # Before 1.27.2 the helper behind `get_links` leaks two lists per link,
+    # and `insert_pdf` copies every link through it; the rolling PDF re-inserts
+    # the whole document after each page, so a 346-page book leaked ~1,700
+    # objects a page, invisible to gc (tests/test_pdf_pages.py guards it).
+    # <1.28: 1.28 deprecates the `fitz` name every module imports.
+    "PyMuPDF>=1.27.2,<1.28",
     "Pillow>=9.5.0,<11.0.0",
     "babeldoc>=0.5.20,<0.6.0",
     # bitstring 5 removed `BitStream`, which babeldoc 0.5's PDF writer still imports

requirements.txt

@@ -6,7 +6,12 @@ tomlkit>=0.12.0,<1.0.0
 python-dotenv>=1.0.0,<2.0.0
 cryptography>=41.0.0,<43.0.0
 
-PyMuPDF>=1.25.1,<1.26.0
+# Before 1.27.2 the helper behind `get_links` leaks two lists per link,
+# and `insert_pdf` copies every link through it; the rolling PDF re-inserts
+# the whole document after each page, so a 346-page book leaked ~1,700
+# objects a page, invisible to gc (tests/test_pdf_pages.py guards it).
+# <1.28: 1.28 deprecates the `fitz` name every module imports.
+PyMuPDF>=1.27.2,<1.28
 Pillow>=9.5.0,<11.0.0
 
 babeldoc>=0.5.20,<0.6.0

src/desktop_pdf_translator/processors/processor.py

@@ -23,6 +23,7 @@
 from ..translators.argos_translator import ArgosTranslator
 from ..translators.base import describe_fatal_error
 from ..translators.capabilities import resolve_effective_service, resolve_languages
+from ..utils.memory import release_freed_memory
 from .events import (
     ProcessingEvent,
     ProgressEvent,
@@ -1080,6 +1081,13 @@ async def paragraph_ticker() -> None:
                         progress_percent=global_progress,
                         message=f"{pages_label} translated",
                     )
+                    # glibc keeps freed heap in its arenas: untrimmed, a
+                    # 129-page run ended ~420 MB above where malloc_trim
+                    # returns it. Here, after the rebuild, so its document is
+                    # freed too, and after chunk_ready, so the viewer's swap
+                    # never waits on it. Never gc.collect(): ~277 ms a page,
+                    # and it freed nothing (utils/memory.py).
+                    await asyncio.to_thread(release_freed_memory)
             finally:
                 # Stop the ticker; completion_task is already done by sentinel.
                 ticker_task.cancel()

src/desktop_pdf_translator/rag/ingest.py

@@ -144,6 +144,40 @@ def _union(rects: Iterable[Rect]) -> Rect:
     )
 
 
+# What `TEXT_DEHYPHENATE` took for a line-break hyphen, checked one by one on
+# 1.25.5: hyphen-minus, soft hyphen, hyphen, non-breaking hyphen. Never a dash.
+_LINE_BREAK_HYPHENS = ("-", "­", "‐", "‑")
+
+
+def _join_broken_words(lines: List[_Line]) -> List[_Line]:
+    """A word hyphenated across a line break, put back together.
+
+    PyMuPDF's `TEXT_DEHYPHENATE` did this up to 1.26 and is ignored from 1.27,
+    which split ~1,100 words of a 346-page book into halves that neither
+    search matches. This is that flag's rule, reproduced exactly: a line
+    ending in a hyphen loses it and runs into the next, unless the line is
+    nothing but the hyphen, which stays. That glues compounds too
+    ("BoseEinstein"), and is kept anyway: a rule that spared them moved two
+    benchmark questions off their page, and matching the flag keeps the
+    dehyphenated text of an index built before the upgrade the same as after.
+    """
+    merged: List[_Line] = []
+    for line in lines:
+        if merged and merged[-1].text.endswith(_LINE_BREAK_HYPHENS):
+            prev = merged[-1]
+            head = prev.text if prev.text.strip() in _LINE_BREAK_HYPHENS else prev.text[:-1]
+            merged[-1] = _Line(
+                text=head + line.text,
+                size=max(prev.size, line.size),
+                bbox=_union([prev.bbox, line.bbox]),
+                bold=prev.bold and line.bold,
+                horizontal=prev.horizontal and line.horizontal,
+            )
+            continue
+        merged.append(line)
+    return merged
+
+
 def _clip(rect: Rect, page: fitz.Rect) -> Rect:
     return (
         float(max(rect[0], page.x0)),
@@ -159,9 +193,10 @@ def _inside(rect: Rect, area: Rect) -> bool:
 
 
 def _read_page(page: fitz.Page) -> _Page:
-    # Ligatures expanded and line-break hyphens joined by PyMuPDF itself:
-    # without `TEXT_PRESERVE_LIGATURES`, and with `TEXT_DEHYPHENATE`.
-    flags = fitz.TEXT_PRESERVE_WHITESPACE | fitz.TEXT_MEDIABOX_CLIP | fitz.TEXT_DEHYPHENATE
+    # Ligatures expanded by PyMuPDF itself: without `TEXT_PRESERVE_LIGATURES`.
+    # Not `TEXT_DEHYPHENATE`: 1.27 ignores it, and `_join_broken_words` is the
+    # one place its rule lives now.
+    flags = fitz.TEXT_PRESERVE_WHITESPACE | fitz.TEXT_MEDIABOX_CLIP
     data = page.get_text("dict", flags=flags)
     result = _Page(number=page.number, rect=page.rect)
     for block in data.get("blocks", []):
@@ -183,7 +218,7 @@ def _read_page(page: fitz.Page) -> _Page:
                 horizontal=abs(direction[0] - 1.0) < 0.01 and abs(direction[1]) < 0.01,
             ))
         if lines:
-            result.blocks.append(_Block(lines=lines, bbox=tuple(block["bbox"])))
+            result.blocks.append(_Block(lines=_join_broken_words(lines), bbox=tuple(block["bbox"])))
     result.text_chars = sum(len(b.text.replace(" ", "")) for b in result.blocks)
 
     area = abs(page.rect) or 1.0

src/desktop_pdf_translator/utils/memory.py

@@ -0,0 +1,45 @@
+"""Hand freed heap back to the OS between translated pages.
+
+A translation runs every page through BabelDOC inside the one long-lived
+sidecar, and glibc keeps what Python frees inside its arenas: it gives memory
+back on its own only from the very top of a heap. Measured on a 129-page Argos
+run, the sidecar ended ~420 MB above the level `malloc_trim` brought it back to,
+and a full `gc.collect()` freed next to nothing — the memory was already free,
+just not returned. `malloc_trim` also releases free pages from the middle of a
+heap, which is the part that matters here.
+
+Elsewhere this is a no-op. Windows' heap returns large blocks as they are
+freed; whether it holds anything else over a long run is measured separately
+(#134), not assumed. musl has no `malloc_trim`. Stdlib-only, so free to import.
+"""
+
+from __future__ import annotations
+
+import ctypes
+import functools
+import sys
+from typing import Callable, Optional
+
+
+@functools.cache
+def _malloc_trim() -> Optional[Callable[[int], int]]:
+    if not sys.platform.startswith("linux"):
+        return None
+    try:
+        libc = ctypes.CDLL("libc.so.6")
+    except OSError:
+        return None
+    return getattr(libc, "malloc_trim", None)
+
+
+def release_freed_memory() -> bool:
+    """Return freed heap pages to the OS; True if glibc was asked to.
+
+    Blocking (it walks every arena), so call it off the event loop.
+    """
+    trim = _malloc_trim()
+    if trim is None:
+        return False
+    # 0 keeps no slack at the top of the heap; the next page grows it again.
+    trim(0)
+    return True