[TranBaDat2607]

Why

During a translation the sidecar's RAM climbed with every page, about +1.25 MB a page after warm-up on a 346-page book. That held even after #135 fixed the fonts. Measuring the real pipeline after each page found two causes:

1. A PyMuPDF refcount leak (< 1.27.2). _extra.JM_get_annot_xref_list leaks a list per call, and insert_pdf copies every link through it (two lists per link).
   • rebuild_rolling_pdf re-inserts the whole document after every page. On the book that leaked ~2,120 lists per rebuild while it was untranslated, and ~1,690 live objects a page averaged over a run.
   • That is O(pages²) for a whole document, and gc can't see it.
2. glibc keeping freed heap. A final malloc_trim handed back ~410 MB. A forced gc.collect() freed nothing: the memory was already free, just never returned.

What changes

commit	what
fix(chat): join line-break hyphens without PyMuPDF's flag	PyMuPDF 1.27 silently ignores TEXT_DEHYPHENATE. Without a replacement, 1,129 words of the book were indexed as halves (trans- action). ingest._join_broken_words reproduces the flag's rule exactly.
fix(translate): require PyMuPDF>=1.27.2,<1.28	Adds a regression test, and re-checks the GIL property the shared fonts rely on, on 1.27.2.3.
fix(translate): malloc_trim(0) after every finished page	utils/memory.py, run off the event loop, after the rebuild and after chunk_ready. A no-op on Windows, macOS and musl. Ported from the unmerged fix/release-freed-memory, whose root-cause prose was wrong (that 7 GB OOM was the fonts).
fix(chat): the Unicode hyphens too	The flag also joined U+00AD, U+2010 and U+2011. Found in review; the benchmark corpus has none of them.
docs(translate): reconcile the figures	Each number in the notes now names what it was measured on.

No chat index is rebuilt. On PyMuPDF 1.25.5 the ingestor's chunks are byte-identical to main on all 7 text documents of the RAG benchmark, so CHUNKER_VERSION is unchanged. A "smarter" join that kept compounds apart (Bose-/Einstein) was measured and dropped: it lost 2 questions.

Results

129 text pages of the book, Argos, 2 pages in flight, measured after warm-up:

	RSS slope p47–129	RSS at end	freed by a final trim	live objects p47→p129
main (PyMuPDF 1.25.5)	+1.25 MB/page	2,118 MB	408 MB	+123,761
PyMuPDF 1.27.2.3 only	+0.18 MB/page	2,066 MB	419 MB	−8,080
this branch	−0.19 MB/page	1,656 MB	2 MB	−4,786

Speed was compared A-B-B-A, sequentially, 46 pages each:

	run 1	run 2
main	4.33 s/page	4.57 s/page
this branch	4.47 s/page	4.38 s/page

That's no difference beyond main's own run-to-run spread. The trim costs 8–22 ms a page, and the 129-page run on this branch did 4.50 s/page.

Behaviour changes worth knowing

• Chat retrieval (retrieval-only, 131 questions), hit@1:

  main, 1.25.5	1.27, no join	1.27 + this branch
  0.857	0.838	0.848 (McNemar p = 1.0)

  The one question still lost (p10, ResNet) comes from 1.27's find_tables. It now reads ResNet's architecture figure as half-empty two-column tables (121 → 133 chunks on that paper). Tightening the ingestor's mostly-empty-grid guard could be a follow-up; the rag-quality gate is 0.80.

• find_tables prints a one-line pymupdf_layout notice to stdout once per process; it goes to shell.log.

• Translation output: it differs from main on a few pages' wording, but no more than two runs of main differ from each other. Argos's batching depends on timing.

Test plan

• pytest tests: 1440 passed (1425 before, plus 5 trim, 1 leak and 9 dehyphenation cases); ruff clean.
• Every new test was seen red for the reason it targets:
  • the leak test failed on 1.25.5 with growth 1,279 (0 on 1.27.2.3);
  • the parity cases failed on the abandoned rule and on the ASCII-only rule;
  • the trim tests failed against two stubs.
• Expected values come from the reference implementation (main's ingestor on 1.25.5), never from the new code.
• RAG retrieval benchmark: pm1255, pm1272, pm1272-dehyph, pm1272-mupdfrule, compared with compare.py.
• 129-page memory run through the patched processor.py. The call site has no unit test, so this is the check.
• Frozen sidecar, built locally with ./build-sidecar.sh (Linux): pytest -m smoke 21 passed, frozen half included.
• Through the frozen exe over HTTP:
  • Argos translate of pages 35–40: done in 30 s, 0 failed paragraphs, Vietnamese output.
  • Cancel on a 55-page run: cancelled 0.1 s after the request.
  • Chat index of resnet.pdf: done, 133 chunks.
• frozen-sidecar workflow dispatched (Windows + Ubuntu): https://github.com/TranBaDat2607/PDFusion/actions/runs/39939620038
• Not measured: Windows memory (Measure sidecar memory across a long translation on Windows #134). The trim is a no-op there; the PyMuPDF fix applies everywhere.

Found along the way, not fixed here

• An intermittent startup import race. The first chat index, requested 0.7 s after READY, failed once inside the frozen exe with cannot import name 'NDArray' from partially initialized module 'numpy._typing'. ChromaDB's import ran while the engine-warm thread was still importing numpy through BabelDOC. Two reruns, one immediately and one after 60 s, passed. Probably on main as well, since neither import path changes here, but not verified there.
• Blank pages still fail a translation on main ("contains no paragraphs"). The fix sits on the unmerged fix/skip-textless-pages.
• BabelDOC's concurrent PDF-save forks still trip filelock's at-fork check, and BabelDOC retries them.

🤖 Generated with Claude Code