Stage 11: arXiv papers get their real length

README.md

@@ -164,6 +164,14 @@ readurlist-effort-backfill
 
 It only fills items still missing a count, so running it again is harmless.
 
+**arXiv papers:** for an arXiv link (abs, pdf or html), the title and summary come from the abstract page, but the word count comes from the full paper's PDF, so papers show their real reading time. If the PDF can't be fetched, the abstract's count is kept. Papers saved before this change get their count once with:
+
+```bash
+readurlist-arxiv-lengths
+```
+
+It pauses 3 seconds between papers, as arXiv asks.
+
 ### Migrate an old local SQLite file
 
 ```bash

docs/build-map.md

@@ -313,6 +313,25 @@ Before any feature work, dev and live are separated: the live worker, bot, datab
 - **Verified:** 2026-11-01 — `uv run pytest -q`: 120 passed; `web/`: `npm test` 84 passed, `npx tsc --noEmit` clean, `npm run build` clean. Local database: "Last 7 days: read 1 of 5 picks (20%, below half: tune the ranking). Skipped 0. Unread pile 9 → 14."; today (Sunday) list has no weekly line. Dev site at 375 px: panel "… Unread pile 10 → 14." at the end of Today; no sideways scroll. Dev worker restarted on this code, so Monday 2026-11-02's dev list carries the line.
   2026-11-01 — Deploy: no database change; owner merged #17 (`2754953`) and ran `scripts/live-worker-update.ps1`; live folder on `2754953`; one live `readurlist.exe`, PID 25468, started 15:12, task result 0; Vercel production build on `2754953` succeeded. Check still to come: Monday 2026-11-02's list ends with "Last week: …" and Today shows the 7-day panel.
 
+### Stage 11 — arXiv papers get their real length
+- **What gets built:** An arXiv link's reading time comes from the full paper, not the abstract page, so papers land in the long section and show their real minutes.
+- **Why:** Owner, 2026-11-01: "the length of the read is only on the abstract for arxiv links. The underlying paper is a much longer read." (Chat 2026-11-01).
+- **Depends on:** Stage 4
+- **Produces:** correct Short / Long labels and sections for arXiv papers.
+- **Terms:** Effort label
+- **Status:** in progress
+- **Changed:** 2026-11-01 — Stage added at the owner's request after the build finished. Owner chose to fetch the full paper ("Fetch the full paper", over treating all arXiv as long or tagging only) and to keep the summary from the abstract ("Abstract, length from paper"): the safety scan and the model's input stay as they are.
+- **Tasks:**
+  - [x] (a) A new arXiv link (abs, pdf or html form) is counted from the full paper's PDF; the title and summary still come from the abstract page.
+    Detail: if the PDF can't be fetched or read, the abstract's count is kept, as today. Built: `ingest/arxiv.py` (`arxiv_id`, `paper_word_count`: PDF up to 40 MB, read with `pypdf`, words counted, text not kept); `ingest_url` reads any arXiv form from its abstract page. New dependency `pypdf`. Tests: `test_arxiv.py`.
+  - [x] (b) arXiv papers already saved get their length fixed once.
+    Detail: a one-off command the owner runs on the live folder after the deploy; it waits 3 seconds between papers, as arXiv asks. Built: `readurlist-arxiv-lengths` (ready items only; a paper whose PDF fails keeps its count). Test: `test_arxiv.py::test_one_off_fix_counts_saved_papers_from_their_pdf`.
+  - [ ] (c) Deploy (you): merge to main, update and restart the live worker, run the one-off fix, then check an arXiv paper on All shows Long and its real minutes.
+    Detail: no database change; merge → `scripts/live-worker-update.ps1` → one-off fix in the live folder.
+- **How we know:** on the local database, saving an arXiv abstract link stores the paper's word count (thousands of words, Long) while the summary matches the abstract; the one-off fix turns an old arXiv item from Short to Long.
+- **Issues:** none
+- **Verified:** 2026-11-01 — `uv run pytest -q`: 136 passed. Local database, real link `https://arxiv.org/abs/1706.03762`: saved ready, title "Attention Is All You Need", abstract 346 words, stored 6,111 words → long, 27 min. Reset to 346 and ran `readurlist-arxiv-lengths`: "counted from their PDF: 2; kept the abstract count: 0"; item back to 6,111.
+
 ## Blocks
 
 ### B1 — Stage 0: no room for a dev Supabase project (2026-11-01)

pyproject.toml

@@ -14,6 +14,7 @@ dependencies = [
   "pydantic>=2.0",
   "pydantic-settings>=2.0",
   "psycopg[binary]>=3.1",
+  "pypdf>=5.0",
 ]
 
 [project.optional-dependencies]
@@ -23,6 +24,7 @@ dev = ["pytest>=8.0"]
 readurlist = "readurlist.main:main"
 readurlist-digest = "readurlist.rank.run:main"
 readurlist-effort-backfill = "readurlist.effort:main"
+readurlist-arxiv-lengths = "readurlist.ingest.arxiv:main"
 
 [build-system]
 requires = ["setuptools>=68"]

src/readurlist/ingest/__init__.py

@@ -3,6 +3,7 @@
 from readurlist.config import Settings
 from readurlist.db import Item, get_session
 from readurlist.effort import word_count
+from readurlist.ingest.arxiv import abs_url, arxiv_id, paper_word_count
 from readurlist.ingest.extract import ExtractError, FetchError, extract_article
 from readurlist.ingest.summarize import summarize_article
 from readurlist.ingest.untrusted import assess
@@ -86,8 +87,10 @@ def ingest_url(
     finally:
         session.close()
 
+    paper_id = arxiv_id(url)
     try:
-        title_hint, text = extract_article(url)
+        # An arXiv link of any form is read from its abstract page.
+        title_hint, text = extract_article(abs_url(paper_id) if paper_id else url)
     except FetchError as exc:
         return _save_stub(url=url, title=url, note=str(exc), note_clean=note_clean)
     except ExtractError as exc:
@@ -134,7 +137,7 @@ def ingest_url(
             topics=list(result.topics),
             keywords=list(result.keywords),
             extracted_text=text,
-            word_count=word_count(text),
+            word_count=(paper_id and paper_word_count(paper_id)) or word_count(text),
             priority=result.priority,
             note=note_clean,
             ingest_status=INGEST_READY,

src/readurlist/ingest/arxiv.py

@@ -0,0 +1,98 @@
+"""arXiv papers get their reading length from the full paper (build map stage 11).
+
+The abstract page gives the title and the text the summary is written from, as for
+any page. The PDF is downloaded only to count its words; its text is never stored or
+shown to the model (owner, 2026-11-01: "Abstract, length from paper").
+"""
+from __future__ import annotations
+
+import io
+import logging
+import re
+import time
+
+import httpx
+from pypdf import PdfReader
+
+from readurlist.effort import word_count
+
+logger = logging.getLogger(__name__)
+
+# New ids (2410.12345, optional version) and old ones (hep-th/9901001).
+_ARXIV = re.compile(
+    r"^https?://(?:www\.|export\.)?arxiv\.org/(?:abs|pdf|html)/"
+    r"((?:\d{4}\.\d{4,5}|[a-z\-]+(?:\.[A-Z]{2})?/\d{7})(?:v\d+)?)(?:\.pdf)?/?$"
+)
+_MAX_PDF_BYTES = 40 * 1024 * 1024
+_TIMEOUT = 60.0
+_TRANSPORT: httpx.BaseTransport | None = None  # tests swap in a fake
+
+
+def arxiv_id(url: str) -> str | None:
+    match = _ARXIV.match(url.strip())
+    return match.group(1) if match else None
+
+
+def abs_url(paper_id: str) -> str:
+    return f"https://arxiv.org/abs/{paper_id}"
+
+
+def paper_word_count(paper_id: str) -> int | None:
+    """Words in the paper's PDF, or None if it can't be fetched or read."""
+    url = f"https://arxiv.org/pdf/{paper_id}"
+    try:
+        with httpx.Client(follow_redirects=True, timeout=_TIMEOUT, transport=_TRANSPORT) as client:
+            with client.stream("GET", url) as response:
+                response.raise_for_status()
+                body = bytearray()
+                for chunk in response.iter_bytes():
+                    body += chunk
+                    if len(body) > _MAX_PDF_BYTES:
+                        logger.warning("arXiv PDF %s is over %s bytes; skipped", paper_id, _MAX_PDF_BYTES)
+                        return None
+        reader = PdfReader(io.BytesIO(bytes(body)))
+        words = sum(word_count(page.extract_text() or "") for page in reader.pages)
+    except Exception:
+        logger.warning("Could not count words in arXiv PDF %s", paper_id, exc_info=True)
+        return None
+    return words or None
+
+
+def fix_saved_papers(*, sleep=time.sleep) -> tuple[int, int]:
+    """One-off: count saved arXiv papers from their PDF. Returns (fixed, failed)."""
+    from readurlist.db import Item, get_session
+
+    session = get_session()
+    try:
+        rows = [
+            (row, paper_id)
+            for row in session.query(Item).filter(Item.ingest_status == "ready").order_by(Item.id)
+            if (paper_id := arxiv_id(row.url))
+        ]
+        fixed = failed = 0
+        for n, (row, paper_id) in enumerate(rows):
+            if n:
+                sleep(3.0)  # arXiv asks automated readers to pause between requests
+            words = paper_word_count(paper_id)
+            if words is None:
+                failed += 1
+                continue
+            row.word_count = words
+            session.commit()
+            fixed += 1
+        return fixed, failed
+    except Exception:
+        session.rollback()
+        raise
+    finally:
+        session.close()
+
+
+def main() -> None:
+    """CLI: `readurlist-arxiv-lengths` — fix the length of arXiv papers saved before stage 11."""
+    from readurlist.config import get_settings
+    from readurlist.db import init_db
+
+    init_db(get_settings().database_url)
+    fixed, failed = fix_saved_papers()
+    print(f"arXiv papers counted from their PDF: {fixed}; kept the abstract count: {failed}")

tests/test_arxiv.py

@@ -0,0 +1,181 @@
+import json
+
+import httpx
+import pytest
+
+from readurlist.config import Settings
+from readurlist.db import init_db
+from readurlist.ingest import ingest_url
+from readurlist.ingest import arxiv
+from readurlist.ingest.arxiv import abs_url, arxiv_id, paper_word_count
+
+ABSTRACT = "We study compost. " * 40  # 120 words: the abstract page alone reads as short
+
+
+def _pdf(text: str) -> bytes:
+    """A one-page PDF whose page shows `text`."""
+    stream = f"BT /F1 12 Tf 72 720 Td ({text}) Tj ET".encode()
+    objects = [
+        b"<< /Type /Catalog /Pages 2 0 R >>",
+        b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
+        b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
+        b"/Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>",
+        b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",
+        b"<< /Length " + str(len(stream)).encode() + b" >>\nstream\n" + stream + b"\nendstream",
+    ]
+    out = b"%PDF-1.4\n"
+    offsets = []
+    for n, body in enumerate(objects, 1):
+        offsets.append(len(out))
+        out += f"{n} 0 obj\n".encode() + body + b"\nendobj\n"
+    xref = len(out)
+    out += f"xref\n0 {len(objects) + 1}\n0000000000 65535 f \n".encode()
+    out += b"".join(f"{o:010d} 00000 n \n".encode() for o in offsets)
+    out += f"trailer\n<< /Size {len(objects) + 1} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n".encode()
+    return out
+
+
+@pytest.mark.parametrize(
+    ("url", "expected"),
+    [
+        ("https://arxiv.org/abs/2410.12345", "2410.12345"),
+        ("https://arxiv.org/abs/2410.12345v2", "2410.12345v2"),
+        ("https://arxiv.org/pdf/2410.12345", "2410.12345"),
+        ("https://arxiv.org/pdf/2410.12345v3.pdf", "2410.12345v3"),
+        ("https://arxiv.org/html/2410.12345v1", "2410.12345v1"),
+        ("http://export.arxiv.org/abs/2410.12345", "2410.12345"),
+        ("https://www.arxiv.org/abs/hep-th/9901001", "hep-th/9901001"),
+        ("https://example.com/abs/2410.12345", None),
+        ("https://arxiv.org/list/cs.AI/recent", None),
+    ],
+)
+def test_arxiv_id(url, expected):
+    assert arxiv_id(url) == expected
+
+
+def test_abs_url():
+    assert abs_url("2410.12345v2") == "https://arxiv.org/abs/2410.12345v2"
+
+
+def _transport(pdf: bytes | None, seen: list[str]):
+    def handler(request: httpx.Request) -> httpx.Response:
+        seen.append(str(request.url))
+        if pdf is None:
+            return httpx.Response(404)
+        return httpx.Response(200, content=pdf, headers={"content-type": "application/pdf"})
+
+    return httpx.MockTransport(handler)
+
+
+def test_paper_word_count_reads_the_pdf(monkeypatch):
+    seen: list[str] = []
+    monkeypatch.setattr(arxiv, "_TRANSPORT", _transport(_pdf("one two three four five"), seen))
+    assert paper_word_count("2410.12345") == 5
+    assert seen == ["https://arxiv.org/pdf/2410.12345"]
+
+
+def test_paper_word_count_is_none_when_the_pdf_fails(monkeypatch):
+    monkeypatch.setattr(arxiv, "_TRANSPORT", _transport(None, []))
+    assert paper_word_count("2410.12345") is None
+    monkeypatch.setattr(arxiv, "_TRANSPORT", _transport(b"not a pdf", []))
+    assert paper_word_count("2410.12345") is None
+
+
+class OneAnswerLLM:
+    def __init__(self) -> None:
+        self.prompts: list[str] = []
+
+    def complete(self, prompt, *, model, system=None, schema=None, temperature=0.3) -> str:
+        self.prompts.append(prompt)
+        return json.dumps({"title": "T", "snapshot": "Summary.", "subject": "Gardening",
+                           "topics": ["soil"], "keywords": ["compost"], "priority": 3})
+
+
+def _settings(db: str) -> Settings:
+    return Settings(telegram_bot_token="x", telegram_user_id=1, groq_api_key="x", database_url=db)
+
+
+def test_arxiv_link_takes_its_length_from_the_paper(tmp_path, monkeypatch):
+    db = f"sqlite:///{tmp_path / 't.db'}"
+    init_db(db)
+    fetched: list[str] = []
+
+    def fake_extract(url, timeout=30.0):
+        fetched.append(url)
+        return "Compost paper", ABSTRACT
+
+    monkeypatch.setattr("readurlist.ingest.extract_article", fake_extract)
+    monkeypatch.setattr("readurlist.ingest.paper_word_count", lambda paper_id: 9000)
+    llm = OneAnswerLLM()
+
+    item, created = ingest_url("https://arxiv.org/pdf/2410.12345v2", llm, _settings(db))  # type: ignore[arg-type]
+
+    assert created and item.ingest_status == "ready"
+    assert fetched == ["https://arxiv.org/abs/2410.12345v2"]  # title and summary from the abstract page
+    assert item.url == "https://arxiv.org/pdf/2410.12345v2"
+    assert item.word_count == 9000
+    assert item.extracted_text == ABSTRACT  # the paper's text isn't stored
+    assert "We study compost." in llm.prompts[0]
+
+
+def test_arxiv_link_keeps_the_abstract_count_when_the_paper_fails(tmp_path, monkeypatch):
+    db = f"sqlite:///{tmp_path / 't.db'}"
+    init_db(db)
+    monkeypatch.setattr("readurlist.ingest.extract_article", lambda url, timeout=30.0: ("T", ABSTRACT))
+    monkeypatch.setattr("readurlist.ingest.paper_word_count", lambda paper_id: None)
+
+    item, _ = ingest_url("https://arxiv.org/abs/2410.12345", OneAnswerLLM(), _settings(db))  # type: ignore[arg-type]
+
+    assert item.word_count == 120
+
+
+def test_other_links_do_not_fetch_a_paper(tmp_path, monkeypatch):
+    db = f"sqlite:///{tmp_path / 't.db'}"
+    init_db(db)
+    monkeypatch.setattr("readurlist.ingest.extract_article", lambda url, timeout=30.0: ("T", ABSTRACT))
+
+    def boom(paper_id):
+        raise AssertionError("no paper fetch for ordinary links")
+
+    monkeypatch.setattr("readurlist.ingest.paper_word_count", boom)
+    item, _ = ingest_url("https://example.com/a", OneAnswerLLM(), _settings(db))  # type: ignore[arg-type]
+    assert item.word_count == 120
+
+
+def test_one_off_fix_counts_saved_papers_from_their_pdf(tmp_path, monkeypatch):
+    from readurlist.db import Item, get_session
+    from readurlist.ingest.arxiv import fix_saved_papers
+
+    init_db(f"sqlite:///{tmp_path / 't.db'}")
+    session = get_session()
+    for url, status in [
+        ("https://arxiv.org/abs/2410.00001", "ready"),
+        ("https://arxiv.org/abs/2410.00002", "ready"),  # its PDF fails
+        ("https://arxiv.org/abs/2410.00003", "pending_body"),  # not ready: left alone
+        ("https://example.com/a", "ready"),
+    ]:
+        session.add(Item(url=url, title="T", snapshot="s", subject="a", topics=[], keywords=[],
+                         extracted_text=ABSTRACT, priority=3, ingest_status=status, word_count=120))
+    session.commit()
+    session.close()
+    counts = {"2410.00001": 8000, "2410.00002": None}
+    asked: list[str] = []
+
+    def fake_count(paper_id):
+        asked.append(paper_id)
+        return counts[paper_id]
+
+    monkeypatch.setattr(arxiv, "paper_word_count", fake_count)
+    waits: list[float] = []
+
+    fixed, failed = fix_saved_papers(sleep=waits.append)
+
+    assert (fixed, failed) == (1, 1)
+    assert asked == ["2410.00001", "2410.00002"]
+    assert waits == [3.0]  # arXiv asks for a pause between requests
+    session = get_session()
+    words = {i.url: i.word_count for i in session.query(Item)}
+    session.close()
+    assert words["https://arxiv.org/abs/2410.00001"] == 8000
+    assert words["https://arxiv.org/abs/2410.00002"] == 120
+    assert words["https://example.com/a"] == 120