From c94d03192900cbb312e484c849ee5f16ac1c56ca Mon Sep 17 00:00:00 2001 From: Alex Lee <13192723+leeaky@users.noreply.github.com> Date: Mon, 2 Nov 2026 00:12:46 +1100 Subject: [PATCH] Map: stage 12 stand-in deployed, one-off rerun counts Co-Authored-By: Claude Opus 5.5 --- docs/build-map.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/build-map.md b/docs/build-map.md index 7c1e2f0..d8cb22a 100644 --- a/docs/build-map.md +++ b/docs/build-map.md @@ -343,6 +343,7 @@ Before any feature work, dev and live are separated: the live worker, bot, datab - **Status:** done - **Changed:** 2026-11-01 — Stage added at the owner's request after stage 11. For papers already saved, the owner chose to recheck only items shown as Short ("Only items shown as Short"). 2026-11-01 — After the live one-off fix (2 papers named a PDF that couldn't be read: Nature, MIT DSpace), owner: "for any url that has a citation URL, assume long and put '>45 min' as a placeholder length"; chose a stand-in count over a true ">45 min" label (no database change). A page whose PDF tag points to a PDF that can't be downloaded or read is now stored as 45 minutes' worth of words (10,350), so it shows Long · 45 min; the one-off fix does the same and prints a fourth count. arXiv is unchanged (a withdrawn paper really is short). Tests: `test_paper_pdf.py`. Local run, `https://www.nature.com/articles/s41562-017-0064`: page 1,646 words, PDF unreadable (PdfStreamError) → stored 10,350, long, 45 min; one-off fix after a reset: back to 10,350. `uv run pytest -q`: 154 passed. Deploy: merge → `scripts/live-worker-update.ps1` → rerun `readurlist-paper-lengths` in the live folder. + 2026-11-02 — Deployed: owner merged #24 (live on `e7f1830`, one live `readurlist.exe`, PID 34672, started 2026-11-01 23:54) and reran the one-off fix: "Short items counted from their PDF: 0; counted as 45 min (PDF unreadable): 4; no PDF tag: 101; kept the page count: 28". The 4 are the three saved copies of Nature s41562-017-0064 and MIT DSpace; the 28 are sites that refuse the page (HTTP 403), among them Science, ScienceDirect and SSRN. - **Tasks:** - [x] (a) A new link whose page names its PDF in the standard citation tag is counted from that PDF; title and summary still come from the page. Detail: the tag is `citation_pdf_url`. Same rules as stage 11: PDF up to 40 MB, its text is counted and not kept, never shown to the model; if the PDF can't be fetched or read (some sites block downloads), the page's count is kept and one line says why. Detail: built: `extract.pdf_link` reads the tag (relative links resolved, http(s) only); `extract_article` now also returns it; `arxiv.pdf_word_count` counts any PDF with stage 11's rules; an arXiv link never falls through to its abstract page's tag. Tests: `test_paper_pdf.py`, `test_arxiv.py`.