Skip to content

Stage 12: papers from other sites get their real length - #22

Merged
leeaky merged 3 commits into
mainfrom
stage-12
Nov 1, 2026
Merged

leeaky merged 3 commits into
mainfrom
stage-12

Conversation

@leeaky

@leeaky leeaky commented Nov 1, 2026

Copy link
Copy Markdown
Owner

Stacked on #21 (both touch ingest/arxiv.py). Merge #21 first; this diff then shrinks to stage 12 only.

What

  • A saved page that names its PDF in the citation_pdf_url tag is counted from that PDF, with stage 11's rules (up to 40 MB, text counted not kept, never shown to the model; page count kept if the PDF fails, one log line says why).
  • arXiv links still use stage 11 only; a withdrawn arXiv paper isn't retried through its abstract page's tag.
  • One-off readurlist-paper-lengths: rechecks ready, non-arXiv items shown as Short, 2 s between pages.

Checks

  • uv run pytest -q: 154 passed.
  • Local database, https://aclanthology.org/N19-1423/: page 1,062 words, stored 10,160 → Long, 44 min; one-off fix restored it after a reset.
  • map-check readurlist: ready.

Deploy (owner): no database change. Merge → scripts/live-worker-update.ps1 → readurlist-paper-lengths in the live folder.

🤖 Generated with Claude Code

leeaky and others added 3 commits November 1, 2026 16:54
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A page naming its PDF in the citation_pdf_url tag is counted from that
PDF, with stage 11's rules. One-off readurlist-paper-lengths rechecks
saved non-arXiv items shown as Short.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
read-ur-list Ready Ready Preview Nov 1, 2026 10:04am UTC

@leeaky
leeaky merged commit d19b2c5 into main Nov 1, 2026
4 checks passed
@leeaky leeaky mentioned this pull request Nov 1, 2026

This branch was successfully deployed

1 active deployment
Preview — 5ee1a277 Deployed Nov 1, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant