Skip to content

Map: stage 12 stand-in length deployed #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
leeaky merged 1 commit into from
Nov 1, 2026
Merged

Map: stage 12 stand-in length deployed #25

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/build-map.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -343,6 +343,7 @@ Before any feature work, dev and live are separated: the live worker, bot, datab
- **Status:** done
- **Changed:** 2026-11-01 — Stage added at the owner's request after stage 11. For papers already saved, the owner chose to recheck only items shown as Short ("Only items shown as Short").
2026-11-01 — After the live one-off fix (2 papers named a PDF that couldn't be read: Nature, MIT DSpace), owner: "for any url that has a citation URL, assume long and put '>45 min' as a placeholder length"; chose a stand-in count over a true ">45 min" label (no database change). A page whose PDF tag points to a PDF that can't be downloaded or read is now stored as 45 minutes' worth of words (10,350), so it shows Long · 45 min; the one-off fix does the same and prints a fourth count. arXiv is unchanged (a withdrawn paper really is short). Tests: `test_paper_pdf.py`. Local run, `https://www.nature.com/articles/s41562-017-0064`: page 1,646 words, PDF unreadable (PdfStreamError) → stored 10,350, long, 45 min; one-off fix after a reset: back to 10,350. `uv run pytest -q`: 154 passed. Deploy: merge → `scripts/live-worker-update.ps1` → rerun `readurlist-paper-lengths` in the live folder.
2026-11-02 — Deployed: owner merged #24 (live on `e7f1830`, one live `readurlist.exe`, PID 34672, started 2026-11-01 23:54) and reran the one-off fix: "Short items counted from their PDF: 0; counted as 45 min (PDF unreadable): 4; no PDF tag: 101; kept the page count: 28". The 4 are the three saved copies of Nature s41562-017-0064 and MIT DSpace; the 28 are sites that refuse the page (HTTP 403), among them Science, ScienceDirect and SSRN.
- **Tasks:**
- [x] (a) A new link whose page names its PDF in the standard citation tag is counted from that PDF; title and summary still come from the page.
Detail: the tag is `citation_pdf_url`. Same rules as stage 11: PDF up to 40 MB, its text is counted and not kept, never shown to the model; if the PDF can't be fetched or read (some sites block downloads), the page's count is kept and one line says why. Detail: built: `extract.pdf_link` reads the tag (relative links resolved, http(s) only); `extract_article` now also returns it; `arxiv.pdf_word_count` counts any PDF with stage 11's rules; an arXiv link never falls through to its abstract page's tag. Tests: `test_paper_pdf.py`, `test_arxiv.py`.
Expand Down
Loading