diff --git a/.github/workflows/convert.yml b/.github/workflows/convert.yml index e013b65..3d58244 100644 --- a/.github/workflows/convert.yml +++ b/.github/workflows/convert.yml @@ -38,6 +38,11 @@ jobs: - name: Install dependencies run: pip install -r requirements.txt + - name: Install Calibre + run: | + sudo apt-get update + sudo apt-get install -y calibre fonts-noto-cjk + - name: Process content run: python scripts/process.py env: diff --git a/README.md b/README.md index ada0897..1026ac2 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ GitHub-hosted content shelf. Fork, upload, done. -> Fork this repo to get your own content platform on GitHub Pages. Upload PDFs or EPUBs (published as online books), Markdown documents (rendered directly), or ZIP archives (deployed as static sites). Zero server cost. +> Fork this repo to get your own content platform on GitHub Pages. Upload PDFs or EPUBs (converted into online books), Markdown documents (rendered directly), or ZIP archives (deployed as static sites). Zero server cost. ## Quick Start @@ -17,14 +17,14 @@ GitHub-hosted content shelf. Fork, upload, done. Your site is now live at `https://.github.io/gitshelf/` -### 2. Add MinerU Token (for PDF conversion) +### 2. Add MinerU Token (for PDF and EPUB conversion) 1. Register at [mineru.net](https://mineru.net) (free during beta) 2. Copy your API token 3. In your fork, go to **Settings > Secrets and variables > Actions** 4. Click **New repository secret**, name it `MINERU_TOKEN`, paste the token -> Only needed if you want to upload PDFs. EPUB, Markdown, and ZIP uploads work without this. +> Needed for book uploads. EPUB files are converted to PDF first, then processed by the same MinerU pipeline as PDFs. ### 3. Password Protection (Optional) @@ -41,7 +41,7 @@ Your site is now live at `https://.github.io/gitshelf/` ([Create one here](https://github.com/settings/tokens/new?scopes=repo&description=GitShelf)) 3. Upload a file: - **`.pdf`** — Converted to a multi-chapter book via MinerU API - - **`.epub`** — Stored as-is and read directly in the browser with TOC navigation + - **`.epub`** — Converted to PDF with Calibre, then processed through the same chapter pipeline as PDFs - **`.md`** — Rendered directly as a document - **`.zip`** — Extracted as a static site (must contain `index.html`) 4. Wait for GitHub Actions to process (progress shown in Actions tab) @@ -68,7 +68,7 @@ Your site is now live at `https://.github.io/gitshelf/` Upload content (browser → GitHub API → input/) → GitHub Actions runs scripts/process.py → .pdf: MinerU API → Markdown → Split chapters → docs/books/{id}/ - → .epub: Store original EPUB + extract TOC → docs/books/{id}/ + → .epub: Calibre → PDF → MinerU API → Markdown → Split chapters → docs/books/{id}/ → .md: Copy to docs/articles/{id}/content.md → .zip: Extract to docs/sites/{id}/ → Build manifest → GitHub Pages deploys @@ -90,7 +90,7 @@ python -m unittest discover -s tests/scripts -v # Python pipeline tests **What if MinerU stops being free?** Swap it by modifying `scripts/mineru_client.py`. Works with any PDF-to-Markdown tool. -**Can I edit converted chapters?** Yes for PDF-converted books. Edit `.md` files in `docs/books//chapters/` and commit. EPUB books are read from the original `book.epub`. +**Can I edit converted chapters?** Yes. Books uploaded as either PDF or EPUB end up as generated Markdown chapters in `docs/books//chapters/`, which you can edit and commit. **Can I upload a static site?** Yes. Package it as a `.zip` with `index.html` at the root and upload through the admin panel. diff --git a/README.zh-CN.md b/README.zh-CN.md index 810a761..2f5aeb5 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -4,7 +4,7 @@ 基于 GitHub 的内容托管平台。Fork,上传,搞定。 -> Fork 本仓库即可拥有自己的内容平台。上传 PDF 或 EPUB(发布为在线书籍)、Markdown 文档(直接渲染)、ZIP 压缩包(部署为静态站点),全部托管在 GitHub Pages 上。零服务器成本。 +> Fork 本仓库即可拥有自己的内容平台。上传 PDF 或 EPUB(转换后发布为在线书籍)、Markdown 文档(直接渲染)、ZIP 压缩包(部署为静态站点),全部托管在 GitHub Pages 上。零服务器成本。 ## 快速开始 @@ -17,14 +17,14 @@ 站点已上线:`https://.github.io/gitshelf/` -### 2. 添加 MinerU Token(用于 PDF 转换) +### 2. 添加 MinerU Token(用于 PDF 和 EPUB 转换) 1. 在 [mineru.net](https://mineru.net) 注册(测试期免费) 2. 复制 API Token 3. 在你的 Fork 中,进入 **Settings > Secrets and variables > Actions** 4. 点击 **New repository secret**,名称填 `MINERU_TOKEN`,粘贴 Token -> 仅上传 PDF 时需要。EPUB、Markdown 和 ZIP 上传无需此配置。 +> 书籍上传都需要。EPUB 会先经由 Calibre 转成 PDF,再进入和 PDF 相同的 MinerU 处理流水线。 ### 3. 密码保护(可选) @@ -41,7 +41,7 @@ ([点此创建](https://github.com/settings/tokens/new?scopes=repo&description=GitShelf)) 3. 上传文件: - **`.pdf`** — 通过 MinerU API 转换为多章节书籍 - - **`.epub`** — 原样保存,并在浏览器中直接阅读,保留目录导航 + - **`.epub`** — 先用 Calibre 转成 PDF,再复用和 PDF 相同的章节转换流程 - **`.md`** — 直接作为文档渲染展示 - **`.zip`** — 解压为静态站点(需包含 `index.html`) 4. 等待 GitHub Actions 处理完成 @@ -68,7 +68,7 @@ 上传内容(浏览器 → GitHub API → input/) → GitHub Actions 运行 scripts/process.py → .pdf: MinerU API → Markdown → 拆分章节 → docs/books/{id}/ - → .epub: 保存原始 EPUB + 提取目录 → docs/books/{id}/ + → .epub: Calibre → PDF → MinerU API → Markdown → 拆分章节 → docs/books/{id}/ → .md: 复制到 docs/articles/{id}/content.md → .zip: 解压到 docs/sites/{id}/ → 构建 manifest → GitHub Pages 部署 @@ -90,7 +90,7 @@ python -m unittest discover -s tests/scripts -v # Python 流水线测试 **MinerU 收费了怎么办?** 修改 `scripts/mineru_client.py` 即可替换,兼容任何 PDF 转 Markdown 工具。 -**可以手动编辑转换后的章节吗?** PDF 转换得到的书籍可以。编辑 `docs/books//chapters/` 下的 `.md` 文件并提交即可。EPUB 书籍则直接读取原始 `book.epub`。 +**可以手动编辑转换后的章节吗?** 可以。无论上传的是 PDF 还是 EPUB,最终都会生成 `docs/books//chapters/` 下的 Markdown 章节文件,可以直接编辑并提交。 **可以上传静态站点吗?** 可以。将站点打包为 `.zip`(根目录需包含 `index.html`),通过管理面板上传即可。 diff --git a/cli/github.js b/cli/github.js index fd8923c..0215f1e 100644 --- a/cli/github.js +++ b/cli/github.js @@ -174,11 +174,9 @@ async function listRepoTree(repo, path, token) { async function getCacheDeleteOps(repo, itemId, token) { let pdfMd5; - let epubMd5; try { const f = await readJson(repo, `docs/books/${itemId}/meta.json`, token); pdfMd5 = f.data?.pdf_md5; - epubMd5 = f.data?.epub_md5; } catch { /* skip */ } const ops = []; @@ -189,13 +187,6 @@ async function getCacheDeleteOps(repo, itemId, token) { .map((f) => ({ path: f.path, delete: true }))); } - if (epubMd5) { - const cacheFiles = await listRepoTree(repo, 'cache/epub', token); - ops.push(...cacheFiles - .filter((f) => f.name.startsWith(epubMd5)) - .map((f) => ({ path: f.path, delete: true }))); - } - return ops; } diff --git a/cli/mcp-server.mjs b/cli/mcp-server.mjs index ffc068a..a00c167 100644 --- a/cli/mcp-server.mjs +++ b/cli/mcp-server.mjs @@ -4,7 +4,7 @@ * GitShelf MCP Server * * GitShelf is a zero-cost GitHub Pages content platform. Upload PDFs or - * EPUBs (auto-converted to chapter books), Markdown (rendered as + * EPUBs (converted to PDF first, then into chapter books), Markdown (rendered as * documents), or ZIP archives (deployed as static sites). Everything is * processed by GitHub Actions and served from GitHub Pages. * @@ -42,7 +42,7 @@ const server = new McpServer( version: '0.1.3', }, { - instructions: 'GitShelf is a zero-cost GitHub Pages content platform. Upload PDFs or EPUBs (auto-converted to multi-chapter books with TOC and reader UI), Markdown files (rendered as documents), or ZIP archives (deployed as static sites). Everything is processed by GitHub Actions and served from GitHub Pages. Use these tools to browse, read, and manage GitShelf content.', + instructions: 'GitShelf is a zero-cost GitHub Pages content platform. Upload PDFs or EPUBs (EPUBs are converted to PDF first, then processed into multi-chapter books with TOC and reader UI), Markdown files (rendered as documents), or ZIP archives (deployed as static sites). Everything is processed by GitHub Actions and served from GitHub Pages. Use these tools to browse, read, and manage GitShelf content.', }, ); diff --git a/package-lock.json b/package-lock.json index 2ce46db..c5cee7a 100644 --- a/package-lock.json +++ b/package-lock.json @@ -10,7 +10,6 @@ "license": "MIT", "dependencies": { "@modelcontextprotocol/sdk": "^1.29.0", - "epubjs": "^0.3.93", "zod": "^4.3.6" }, "bin": { @@ -1279,16 +1278,6 @@ "dev": true, "license": "MIT" }, - "node_modules/@types/localforage": { - "version": "0.0.34", - "resolved": "https://registry.npmjs.org/@types/localforage/-/localforage-0.0.34.tgz", - "integrity": "sha512-tJxahnjm9dEI1X+hQSC5f2BSd/coZaqbIl1m3TCl0q9SVuC52XcXfV0XmoCU1+PmjyucuVITwoTnN8OlTbEXXA==", - "deprecated": "This is a stub types definition for localforage (https://github.com/localForage/localForage). localforage provides its own type definitions, so you don't need @types/localforage installed!", - "license": "MIT", - "dependencies": { - "localforage": "*" - } - }, "node_modules/@vitest/expect": { "version": "4.1.1", "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.1.tgz", @@ -1412,16 +1401,6 @@ "url": "https://opencollective.com/vitest" } }, - "node_modules/@xmldom/xmldom": { - "version": "0.7.13", - "resolved": "https://registry.npmjs.org/@xmldom/xmldom/-/xmldom-0.7.13.tgz", - "integrity": "sha512-lm2GW5PkosIzccsaZIz7tp8cPADSIlIHWDFTR1N0SzfinhhYgeIQjFMz4rYzanCScr3DqQLeomUDArp6MWKm+g==", - "deprecated": "this version has critical issues, please update to the latest version", - "license": "MIT", - "engines": { - "node": ">=10.0.0" - } - }, "node_modules/accepts": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/accepts/-/accepts-2.0.0.tgz", @@ -1834,23 +1813,6 @@ "node": ">=6.6.0" } }, - "node_modules/core-js": { - "version": "3.49.0", - "resolved": "https://registry.npmjs.org/core-js/-/core-js-3.49.0.tgz", - "integrity": "sha512-es1U2+YTtzpwkxVLwAFdSpaIMyQaq0PBgm3YD1W3Qpsn1NAmO3KSgZfu+oGSWVu6NvLHoHCV/aYcsE5wiB7ALg==", - "hasInstallScript": true, - "license": "MIT", - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/core-js" - } - }, - "node_modules/core-util-is": { - "version": "1.0.3", - "resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.3.tgz", - "integrity": "sha512-ZQBvi1DcpJ4GDqanjucZ2Hj3wEO5pZDS89BWbkcrvdxksJorwUDDZamX9ldFkp9aw2lmBDLgkObEA4DWNJ9FYQ==", - "license": "MIT" - }, "node_modules/cors": { "version": "2.8.6", "resolved": "https://registry.npmjs.org/cors/-/cors-2.8.6.tgz", @@ -1933,19 +1895,6 @@ "dev": true, "license": "MIT" }, - "node_modules/d": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/d/-/d-1.0.2.tgz", - "integrity": "sha512-MOqHvMWF9/9MX6nza0KgvFH4HpMU0EF5uUDXqX/BtxtU8NfB0QzRtJ8Oe/6SuS4kbhyzVJwjd97EA4PKrzJ8bw==", - "license": "ISC", - "dependencies": { - "es5-ext": "^0.10.64", - "type": "^2.7.2" - }, - "engines": { - "node": ">=0.12" - } - }, "node_modules/data-urls": { "version": "7.0.0", "resolved": "https://registry.npmjs.org/data-urls/-/data-urls-7.0.0.tgz", @@ -2200,23 +2149,6 @@ "url": "https://github.com/fb55/entities?sponsor=1" } }, - "node_modules/epubjs": { - "version": "0.3.93", - "resolved": "https://registry.npmjs.org/epubjs/-/epubjs-0.3.93.tgz", - "integrity": "sha512-c06pNSdBxcXv3dZSbXAVLE1/pmleRhOT6mXNZo6INKmvuKpYB65MwU/lO7830czCtjIiK9i+KR+3S+p0wtljrw==", - "license": "BSD-2-Clause", - "dependencies": { - "@types/localforage": "0.0.34", - "@xmldom/xmldom": "^0.7.5", - "core-js": "^3.18.3", - "event-emitter": "^0.3.5", - "jszip": "^3.7.1", - "localforage": "^1.10.0", - "lodash": "^4.17.21", - "marks-pane": "^1.0.9", - "path-webpack": "0.0.3" - } - }, "node_modules/es-define-property": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz", @@ -2275,46 +2207,6 @@ "node": ">= 0.4" } }, - "node_modules/es5-ext": { - "version": "0.10.64", - "resolved": "https://registry.npmjs.org/es5-ext/-/es5-ext-0.10.64.tgz", - "integrity": "sha512-p2snDhiLaXe6dahss1LddxqEm+SkuDvV8dnIQG0MWjyHpcMNfXKPE+/Cc0y+PhxJX3A4xGNeFCj5oc0BUh6deg==", - "hasInstallScript": true, - "license": "ISC", - "dependencies": { - "es6-iterator": "^2.0.3", - "es6-symbol": "^3.1.3", - "esniff": "^2.0.1", - "next-tick": "^1.1.0" - }, - "engines": { - "node": ">=0.10" - } - }, - "node_modules/es6-iterator": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/es6-iterator/-/es6-iterator-2.0.3.tgz", - "integrity": "sha512-zw4SRzoUkd+cl+ZoE15A9o1oQd920Bb0iOJMQkQhl3jNc03YqVjAhG7scf9C5KWRU/R13Orf588uCC6525o02g==", - "license": "MIT", - "dependencies": { - "d": "1", - "es5-ext": "^0.10.35", - "es6-symbol": "^3.1.1" - } - }, - "node_modules/es6-symbol": { - "version": "3.1.4", - "resolved": "https://registry.npmjs.org/es6-symbol/-/es6-symbol-3.1.4.tgz", - "integrity": "sha512-U9bFFjX8tFiATgtkJ1zg25+KviIXpgRvRHS8sau3GfhVzThRQrOeksPeT0BWW2MNZs1OEWJ1DPXOQMn0KKRkvg==", - "license": "ISC", - "dependencies": { - "d": "^1.0.2", - "ext": "^1.7.0" - }, - "engines": { - "node": ">=0.12" - } - }, "node_modules/escalade": { "version": "3.2.0", "resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz", @@ -2331,21 +2223,6 @@ "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==", "license": "MIT" }, - "node_modules/esniff": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/esniff/-/esniff-2.0.1.tgz", - "integrity": "sha512-kTUIGKQ/mDPFoJ0oVfcmyJn4iBDRptjNVIzwIFR7tqWXdVI9xfA2RMwY/gbSpJG3lkdWNEjLap/NqVHZiJsdfg==", - "license": "ISC", - "dependencies": { - "d": "^1.0.1", - "es5-ext": "^0.10.62", - "event-emitter": "^0.3.5", - "type": "^2.7.2" - }, - "engines": { - "node": ">=0.10" - } - }, "node_modules/estree-walker": { "version": "2.0.2", "resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-2.0.2.tgz", @@ -2362,16 +2239,6 @@ "node": ">= 0.6" } }, - "node_modules/event-emitter": { - "version": "0.3.5", - "resolved": "https://registry.npmjs.org/event-emitter/-/event-emitter-0.3.5.tgz", - "integrity": "sha512-D9rRn9y7kLPnJ+hMq7S/nhvoKwwvVJahBi2BPmx3bvbsEdK3W9ii8cBSGjP+72/LnM4n6fo3+dkCX5FeTQruXA==", - "license": "MIT", - "dependencies": { - "d": "1", - "es5-ext": "~0.10.14" - } - }, "node_modules/eventsource": { "version": "3.0.7", "resolved": "https://registry.npmjs.org/eventsource/-/eventsource-3.0.7.tgz", @@ -2464,15 +2331,6 @@ "express": ">= 4.11" } }, - "node_modules/ext": { - "version": "1.7.0", - "resolved": "https://registry.npmjs.org/ext/-/ext-1.7.0.tgz", - "integrity": "sha512-6hxeJYaL110a9b5TEJSj0gojyHQAmA2ch5Os+ySCiA1QGdS697XWY1pzsrSjqA9LDEEgdB/KypIlR59RcLuHYw==", - "license": "ISC", - "dependencies": { - "type": "^2.7.2" - } - }, "node_modules/fast-deep-equal": { "version": "3.1.3", "resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz", @@ -2805,12 +2663,6 @@ "url": "https://opencollective.com/express" } }, - "node_modules/immediate": { - "version": "3.0.6", - "resolved": "https://registry.npmjs.org/immediate/-/immediate-3.0.6.tgz", - "integrity": "sha512-XXOFtyqDjNDAQxVfYxuF7g9Il/IbWmmlQg2MYKOH8ExIT1qg6xc4zyS3HaEEATgs1btfzxq15ciUiY7gjSXRGQ==", - "license": "MIT" - }, "node_modules/indent-string": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/indent-string/-/indent-string-4.0.0.tgz", @@ -3222,18 +3074,6 @@ "node": ">=6" } }, - "node_modules/jszip": { - "version": "3.10.1", - "resolved": "https://registry.npmjs.org/jszip/-/jszip-3.10.1.tgz", - "integrity": "sha512-xXDvecyTpGLrqFrvkrUSoxxfJI5AH7U8zxxtVclpsUtMCq4JQ290LY8AW5c7Ggnr/Y/oK+bQMbqK2qmtk3pN4g==", - "license": "(MIT OR GPL-3.0-or-later)", - "dependencies": { - "lie": "~3.3.0", - "pako": "~1.0.2", - "readable-stream": "~2.3.6", - "setimmediate": "^1.0.5" - } - }, "node_modules/katex": { "version": "0.16.42", "resolved": "https://registry.npmjs.org/katex/-/katex-0.16.42.tgz", @@ -3258,15 +3098,6 @@ "dev": true, "license": "MIT" }, - "node_modules/lie": { - "version": "3.3.0", - "resolved": "https://registry.npmjs.org/lie/-/lie-3.3.0.tgz", - "integrity": "sha512-UaiMJzeWRlEujzAuw5LokY1L5ecNQYZKfmyZ9L7wDHb/p5etKaxXhohBcrw0EYby+G/NA52vRSN4N39dxHAIwQ==", - "license": "MIT", - "dependencies": { - "immediate": "~3.0.5" - } - }, "node_modules/lightningcss": { "version": "1.32.0", "resolved": "https://registry.npmjs.org/lightningcss/-/lightningcss-1.32.0.tgz", @@ -3538,30 +3369,6 @@ "uc.micro": "^2.0.0" } }, - "node_modules/localforage": { - "version": "1.10.0", - "resolved": "https://registry.npmjs.org/localforage/-/localforage-1.10.0.tgz", - "integrity": "sha512-14/H1aX7hzBBmmh7sGPd+AOMkkIrHM3Z1PAyGgZigA1H1p5O5ANnMyWzvpAETtG68/dC4pC0ncy3+PPGzXZHPg==", - "license": "Apache-2.0", - "dependencies": { - "lie": "3.1.1" - } - }, - "node_modules/localforage/node_modules/lie": { - "version": "3.1.1", - "resolved": "https://registry.npmjs.org/lie/-/lie-3.1.1.tgz", - "integrity": "sha512-RiNhHysUjhrDQntfYSfY4MU24coXXdEOgw9WGcKHNeEwffDYbF//u87M1EWaMGzuFoSbqW0C9C6lEEhDOAswfw==", - "license": "MIT", - "dependencies": { - "immediate": "~3.0.5" - } - }, - "node_modules/lodash": { - "version": "4.18.1", - "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.18.1.tgz", - "integrity": "sha512-dMInicTPVE8d1e5otfwmmjlxkZoUpiVLwyeTdUsi/Caj/gfzzblBcCE5sRHV/AsjuCmxWrte2TNGSYuCeCq+0Q==", - "license": "MIT" - }, "node_modules/lru-cache": { "version": "11.2.7", "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-11.2.7.tgz", @@ -3630,12 +3437,6 @@ "url": "https://github.com/fb55/entities?sponsor=1" } }, - "node_modules/marks-pane": { - "version": "1.0.9", - "resolved": "https://registry.npmjs.org/marks-pane/-/marks-pane-1.0.9.tgz", - "integrity": "sha512-Ahs4oeG90tbdPWwAJkAAoHg2lRR8lAs9mZXETNPO9hYg3AkjUJBKi1NQ4aaIQZVGrig7c/3NUV1jANl8rFTeMg==", - "license": "MIT" - }, "node_modules/math-intrinsics": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz", @@ -3749,12 +3550,6 @@ "node": ">= 0.6" } }, - "node_modules/next-tick": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/next-tick/-/next-tick-1.1.0.tgz", - "integrity": "sha512-CXdUiJembsNjuToQvxayPZF9Vqht7hewsvy2sOWafLvi2awflj9mOC6bHIg50orX8IJvWKY9wYQ/zB2kogPslQ==", - "license": "ISC" - }, "node_modules/node-html-parser": { "version": "6.1.13", "resolved": "https://registry.npmjs.org/node-html-parser/-/node-html-parser-6.1.13.tgz", @@ -3887,12 +3682,6 @@ "wrappy": "1" } }, - "node_modules/pako": { - "version": "1.0.11", - "resolved": "https://registry.npmjs.org/pako/-/pako-1.0.11.tgz", - "integrity": "sha512-4hLB8Py4zZce5s4yd9XzopqwVv/yGNhV1Bl8NTmCq1763HeK2+EwVTv+leGeL13Dnh2wfbqowVPXCIO0z4taYw==", - "license": "(MIT AND Zlib)" - }, "node_modules/parse5": { "version": "8.0.0", "resolved": "https://registry.npmjs.org/parse5/-/parse5-8.0.0.tgz", @@ -3934,12 +3723,6 @@ "url": "https://opencollective.com/express" } }, - "node_modules/path-webpack": { - "version": "0.0.3", - "resolved": "https://registry.npmjs.org/path-webpack/-/path-webpack-0.0.3.tgz", - "integrity": "sha512-AmeDxedoo5svf7aB3FYqSAKqMxys014lVKBzy1o/5vv9CtU7U4wgGWL1dA2o6MOzcD53ScN4Jmiq6VbtLz1vIQ==", - "license": "MIT" - }, "node_modules/pathe": { "version": "2.0.3", "resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz", @@ -4054,12 +3837,6 @@ "url": "https://github.com/chalk/ansi-styles?sponsor=1" } }, - "node_modules/process-nextick-args": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/process-nextick-args/-/process-nextick-args-2.0.1.tgz", - "integrity": "sha512-3ouUOpQhtgrbOa17J7+uxOTpITYWaGP7/AhoR3+A+/1e9skrzelGi/dXzEYyvbxubEF6Wn2ypscTKiKJFFn1ag==", - "license": "MIT" - }, "node_modules/proxy-addr": { "version": "2.0.7", "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz", @@ -4139,27 +3916,6 @@ "dev": true, "license": "MIT" }, - "node_modules/readable-stream": { - "version": "2.3.8", - "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.8.tgz", - "integrity": "sha512-8p0AUk4XODgIewSi0l8Epjs+EVnWiK7NoDIEGU0HhE7+ZyY8D1IMY7odu5lRrFXGg71L15KG8QrPmum45RTtdA==", - "license": "MIT", - "dependencies": { - "core-util-is": "~1.0.0", - "inherits": "~2.0.3", - "isarray": "~1.0.0", - "process-nextick-args": "~2.0.0", - "safe-buffer": "~5.1.1", - "string_decoder": "~1.1.1", - "util-deprecate": "~1.0.1" - } - }, - "node_modules/readable-stream/node_modules/isarray": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/isarray/-/isarray-1.0.0.tgz", - "integrity": "sha512-VLghIWNM6ELQzo7zwmcg0NmTVyWKYjvIeM83yjp0wRDTmUnrM678fQbcKBo6n2CJEF0szoG//ytg+TKla89ALQ==", - "license": "MIT" - }, "node_modules/redent": { "version": "3.0.0", "resolved": "https://registry.npmjs.org/redent/-/redent-3.0.0.tgz", @@ -4254,12 +4010,6 @@ "node": ">= 18" } }, - "node_modules/safe-buffer": { - "version": "5.1.2", - "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz", - "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g==", - "license": "MIT" - }, "node_modules/safe-regex-test": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/safe-regex-test/-/safe-regex-test-1.1.0.tgz", @@ -4386,12 +4136,6 @@ "node": ">= 0.4" } }, - "node_modules/setimmediate": { - "version": "1.0.5", - "resolved": "https://registry.npmjs.org/setimmediate/-/setimmediate-1.0.5.tgz", - "integrity": "sha512-MATJdZp8sLqDl/68LfQmbP8zKPLQNV6BIZoIgrscFDQ+RsvK/BxeDQOgyxKKoh0y/8h3BqVFnCqQ/gd+reiIXA==", - "license": "MIT" - }, "node_modules/setprototypeof": { "version": "1.2.0", "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz", @@ -4575,15 +4319,6 @@ "node": ">= 0.4" } }, - "node_modules/string_decoder": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.1.1.tgz", - "integrity": "sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg==", - "license": "MIT", - "dependencies": { - "safe-buffer": "~5.1.0" - } - }, "node_modules/strip-indent": { "version": "3.0.0", "resolved": "https://registry.npmjs.org/strip-indent/-/strip-indent-3.0.0.tgz", @@ -4724,12 +4459,6 @@ "license": "0BSD", "optional": true }, - "node_modules/type": { - "version": "2.7.3", - "resolved": "https://registry.npmjs.org/type/-/type-2.7.3.tgz", - "integrity": "sha512-8j+1QmAbPvLZow5Qpi6NCaN8FB60p/6x8/vfNqOk/hC+HuvFZhL4+WfekuhQLiqFZXOgQdrs3B+XxEmCc6b3FQ==", - "license": "ISC" - }, "node_modules/type-is": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/type-is/-/type-is-2.0.1.tgz", @@ -4801,12 +4530,6 @@ "browserslist": ">= 4.21.0" } }, - "node_modules/util-deprecate": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz", - "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==", - "license": "MIT" - }, "node_modules/vary": { "version": "1.1.2", "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz", diff --git a/package.json b/package.json index cc0c32a..6f4744f 100644 --- a/package.json +++ b/package.json @@ -56,7 +56,6 @@ }, "dependencies": { "@modelcontextprotocol/sdk": "^1.29.0", - "epubjs": "^0.3.93", "zod": "^4.3.6" } } diff --git a/scripts/build_manifest.py b/scripts/build_manifest.py index 4ad1d41..ca469d5 100644 --- a/scripts/build_manifest.py +++ b/scripts/build_manifest.py @@ -7,7 +7,6 @@ from pathlib import Path VALID_VISIBILITY = {"published", "hidden", "archived"} -EPUB_BOOK_FILENAME = "book.epub" def _utc_now_iso() -> str: @@ -174,9 +173,7 @@ def _build_book_entry(book_dir: Path) -> dict: md_files = sorted(chapters_dir.glob("*.md")) if chapters_dir.is_dir() else [] meta = _read_meta_json(book_dir / "meta.json") - source_format = str(meta.get("source_format", "")).strip() - if not source_format: - source_format = "epub" if (book_dir / EPUB_BOOK_FILENAME).exists() else "markdown-derived" + source_format = str(meta.get("source_format", "")).strip() or "markdown-derived" def _count_toc_entries(items: list[dict]) -> int: total = 0 @@ -198,8 +195,7 @@ def _count_toc_entries(items: list[dict]) -> int: book_dir.stat().st_mtime, tz=timezone.utc ).strftime("%Y-%m-%dT%H:%M:%SZ") - default_ext = ".epub" if source_format == "epub" else ".pdf" - source = str(meta.get("source", "")).strip() or f"{book_dir.name}{default_ext}" + source = str(meta.get("source", "")).strip() or f"{book_dir.name}.pdf" updated_at = str(meta.get("updated_at", "")).strip() or directory_modified_at created_at = str(meta.get("created_at", "")).strip() or updated_at diff --git a/scripts/convert.py b/scripts/convert.py index 15b8765..92e6eab 100644 --- a/scripts/convert.py +++ b/scripts/convert.py @@ -388,6 +388,8 @@ def _read_cache(md5: str) -> tuple[str, dict[str, bytes], int, list[TocEntry]] | def convert_single_pdf( pdf_path: Path, output_dir: Path, + *, + source_name: str | None = None, ) -> None: """Convert one PDF through the full pipeline. @@ -405,8 +407,9 @@ def convert_single_pdf( "book", ) title = pdf_path.stem + source_filename = str(source_name or pdf_path.name).strip() or pdf_path.name - print(f"Processing: {pdf_path.name} -> {book_id}") + print(f"Processing: {source_filename} -> {book_id}") md5 = _pdf_md5(pdf_path) cached = _read_cache(md5) @@ -455,14 +458,14 @@ def convert_single_pdf( _write_book_metadata( book_dir, book_id=book_id, - source_pdf=pdf_path.name, + source_pdf=source_filename, pdf_md5=md5, page_count=page_count, updated_at=_utc_now_iso(), ) # Clear any prior failure record for this PDF - _remove_failure(pdf_path.name, output_dir.parent) + _remove_failure(source_filename, output_dir.parent) # Delete source PDF — the cache preserves everything needed for reconvert pdf_path.unlink(missing_ok=True) diff --git a/scripts/process.py b/scripts/process.py index 00235ed..1536615 100644 --- a/scripts/process.py +++ b/scripts/process.py @@ -3,7 +3,7 @@ Handles four content types from input/: - .pdf → book (chapters via MinerU API) - - .epub → book (raw EPUB + extracted TOC metadata) + - .epub → book (convert to PDF via Calibre, then reuse the PDF pipeline) - .md → article (single markdown document) - .zip → site (static site extraction) @@ -11,18 +11,16 @@ """ import argparse -import hashlib import json import os -import posixpath import re import shutil +import subprocess import sys +import tempfile import zipfile -import xml.etree.ElementTree as ET -from dataclasses import dataclass, field from datetime import datetime, timezone -from pathlib import Path, PurePosixPath +from pathlib import Path try: from .build_manifest import build_manifest @@ -52,19 +50,7 @@ ) FAILURES_FILENAME = "failures.json" -BOOK_METADATA_FILENAME = "meta.json" -EPUB_BOOK_FILENAME = "book.epub" -EPUB_CONTAINER_PATH = "META-INF/container.xml" -EPUB_CONTAINER_NS = {"container": "urn:oasis:names:tc:opendocument:xmlns:container"} -EPUB_PACKAGE_NS = { - "opf": "http://www.idpf.org/2007/opf", - "dc": "http://purl.org/dc/elements/1.1/", -} -EPUB_NAV_NS = { - "xhtml": "http://www.w3.org/1999/xhtml", - "epub": "http://www.idpf.org/2007/ops", -} -EPUB_NCX_NS = {"ncx": "http://www.daisy.org/z3986/2005/ncx/"} +EBOOK_CONVERT_BINARY = "ebook-convert" def _utc_now_iso() -> str: @@ -76,427 +62,63 @@ def _generate_id(path: Path) -> str: return generate_book_id(path) -def _file_md5(path: Path) -> str: - """Compute MD5 hex digest for an input file.""" - h = hashlib.md5() - with open(path, "rb") as f: - for chunk in iter(lambda: f.read(8192), b""): - h.update(chunk) - return h.hexdigest() - - def detect_new_epubs(input_dir: Path) -> list[Path]: """Find .epub files in input_dir.""" return sorted(input_dir.glob("*.epub")) -def _read_existing_created_at(book_dir: Path, fallback: str) -> str: - meta_path = book_dir / BOOK_METADATA_FILENAME - if not meta_path.exists(): - return fallback - - try: - existing = json.loads(meta_path.read_text(encoding="utf-8")) - except json.JSONDecodeError: - return fallback - - return str(existing.get("created_at", "")).strip() or fallback - - -def _read_existing_epub_checksum(book_dir: Path) -> str | None: - meta_path = book_dir / BOOK_METADATA_FILENAME - if not meta_path.exists(): - return None - - try: - existing = json.loads(meta_path.read_text(encoding="utf-8")) - except json.JSONDecodeError: - return None - - checksum = str(existing.get("epub_md5", "")).strip() - return checksum or None - - -def _write_epub_metadata( - book_dir: Path, - *, - book_id: str, - source_epub: str, - epub_md5: str, - updated_at: str, - created_at: str | None = None, -) -> None: - normalized_created_at = created_at or _read_existing_created_at(book_dir, updated_at) - data = { - "id": book_id, - "type": "book", - "source": source_epub, - "source_format": "epub", - "epub_md5": epub_md5, - "created_at": normalized_created_at, - "updated_at": updated_at, - } - (book_dir / BOOK_METADATA_FILENAME).write_text( - json.dumps(data, indent=2, ensure_ascii=False) + "\n", - encoding="utf-8", - ) - - -def _slugify_epub(text: str) -> str: - normalized = text.strip().lower() - normalized = re.sub(r"[\s_]+", "-", normalized) - normalized = re.sub(r"[^\w-]+", "-", normalized, flags=re.UNICODE) - normalized = re.sub(r"-{2,}", "-", normalized).strip("-") - return normalized - - -@dataclass -class EpubTocEntry: - title: str - href: str - children: list["EpubTocEntry"] = field(default_factory=list) - - -def _normalize_epub_href(base_path: str, href: str) -> str: - value = str(href or "").strip() - if not value: - return "" - - path_part, frag = value.split("#", 1) if "#" in value else (value, "") - resolved = posixpath.normpath( - posixpath.join(posixpath.dirname(base_path), path_part) - ) - if frag: - return f"{resolved}#{frag}" - return resolved - - -def _read_epub_package_path(zf: zipfile.ZipFile) -> str: - try: - container_xml = zf.read(EPUB_CONTAINER_PATH) - except KeyError as exc: - raise ValueError("EPUB is missing META-INF/container.xml") from exc - - root = ET.fromstring(container_xml) - rootfile = root.find(".//container:rootfile", EPUB_CONTAINER_NS) - if rootfile is None: - raise ValueError("EPUB container.xml does not declare a package document") - - full_path = str(rootfile.attrib.get("full-path", "")).strip() - if not full_path: - raise ValueError("EPUB package document path is empty") - return full_path - - -def _parse_epub_nav_item( - li: ET.Element, - *, - nav_path: str, -) -> EpubTocEntry | None: - anchor = li.find("./xhtml:a", EPUB_NAV_NS) - if anchor is None: - anchor = li.find("./xhtml:span", EPUB_NAV_NS) - if anchor is None: - return None - - title = " ".join("".join(anchor.itertext()).split()) - if not title: - return None - - href = _normalize_epub_href(nav_path, anchor.attrib.get("href", "")) - children = [] - child_list = li.find("./xhtml:ol", EPUB_NAV_NS) - if child_list is not None: - for child in child_list.findall("./xhtml:li", EPUB_NAV_NS): - parsed = _parse_epub_nav_item(child, nav_path=nav_path) - if parsed is not None: - children.append(parsed) - - return EpubTocEntry(title=title, href=href, children=children) - - -def _extract_epub_nav_toc(zf: zipfile.ZipFile, nav_path: str) -> list[EpubTocEntry]: - root = ET.fromstring(zf.read(nav_path)) - toc_nav = None - for nav in root.findall(".//xhtml:nav", EPUB_NAV_NS): - nav_type = nav.attrib.get(f"{{{EPUB_NAV_NS['epub']}}}type", "") or nav.attrib.get("type", "") - if nav_type == "toc": - toc_nav = nav - break - - if toc_nav is None: - return [] - - toc_list = toc_nav.find("./xhtml:ol", EPUB_NAV_NS) - if toc_list is None: - return [] - - entries: list[EpubTocEntry] = [] - for item in toc_list.findall("./xhtml:li", EPUB_NAV_NS): - parsed = _parse_epub_nav_item(item, nav_path=nav_path) - if parsed is not None: - entries.append(parsed) - return entries - - -def _parse_epub_ncx_point(point: ET.Element, *, ncx_path: str) -> EpubTocEntry | None: - label = point.find("./ncx:navLabel/ncx:text", EPUB_NCX_NS) - content = point.find("./ncx:content", EPUB_NCX_NS) - if label is None or content is None: - return None - - title = " ".join("".join(label.itertext()).split()) - if not title: - return None - - href = _normalize_epub_href(ncx_path, content.attrib.get("src", "")) - children = [] - for child in point.findall("./ncx:navPoint", EPUB_NCX_NS): - parsed = _parse_epub_ncx_point(child, ncx_path=ncx_path) - if parsed is not None: - children.append(parsed) - - return EpubTocEntry(title=title, href=href, children=children) - - -def _extract_epub_ncx_toc(zf: zipfile.ZipFile, ncx_path: str) -> list[EpubTocEntry]: - root = ET.fromstring(zf.read(ncx_path)) - nav_map = root.find("./ncx:navMap", EPUB_NCX_NS) - if nav_map is None: - return [] - - entries: list[EpubTocEntry] = [] - for point in nav_map.findall("./ncx:navPoint", EPUB_NCX_NS): - parsed = _parse_epub_ncx_point(point, ncx_path=ncx_path) - if parsed is not None: - entries.append(parsed) - return entries - - -def _guess_epub_fallback_toc( - spine: list[str], - manifest_by_id: dict[str, dict[str, str]], - *, - package_path: str, -) -> list[EpubTocEntry]: - entries: list[EpubTocEntry] = [] - for item_id in spine: - manifest_item = manifest_by_id.get(item_id) - if not manifest_item: - continue - - href = _normalize_epub_href(package_path, manifest_item.get("href", "")) - media_type = manifest_item.get("media_type", "") - if media_type not in {"application/xhtml+xml", "text/html"}: - continue - - title = PurePosixPath(href.split("#", 1)[0]).stem.replace("_", " ").replace("-", " ").strip() - entries.append(EpubTocEntry(title=title or item_id, href=href)) - return entries - - -def _build_epub_toc_data(epub_path: Path) -> tuple[str, list[EpubTocEntry]]: - try: - with zipfile.ZipFile(epub_path, "r") as zf: - package_path = _read_epub_package_path(zf) - package_root = ET.fromstring(zf.read(package_path)) - except zipfile.BadZipFile as exc: - raise ValueError(f"Invalid EPUB file: {epub_path.name}: {exc}") from exc - - title = ( - package_root.findtext(".//dc:title", default="", namespaces=EPUB_PACKAGE_NS).strip() - or epub_path.stem +def _resolve_ebook_convert() -> str: + binary = shutil.which(EBOOK_CONVERT_BINARY) + if binary: + return binary + raise RuntimeError( + "Calibre's ebook-convert is required for EPUB uploads. " + "Install calibre and ensure 'ebook-convert' is on PATH." ) - manifest_by_id: dict[str, dict[str, str]] = {} - nav_path = "" - ncx_path = "" - for item in package_root.findall(".//opf:manifest/opf:item", EPUB_PACKAGE_NS): - item_id = str(item.attrib.get("id", "")).strip() - href = str(item.attrib.get("href", "")).strip() - media_type = str(item.attrib.get("media-type", "")).strip() - properties = str(item.attrib.get("properties", "")).strip() - if not item_id or not href: - continue - manifest_by_id[item_id] = { - "href": href, - "media_type": media_type, - "properties": properties, - } - if "nav" in properties.split(): - nav_path = _normalize_epub_href(package_path, href) - if media_type == "application/x-dtbncx+xml" and not ncx_path: - ncx_path = _normalize_epub_href(package_path, href) - - spine_element = package_root.find(".//opf:spine", EPUB_PACKAGE_NS) - spine_ids = [ - str(itemref.attrib.get("idref", "")).strip() - for itemref in package_root.findall(".//opf:spine/opf:itemref", EPUB_PACKAGE_NS) - if str(itemref.attrib.get("idref", "")).strip() +def _convert_epub_to_pdf(epub_path: Path, pdf_path: Path) -> None: + command = [ + _resolve_ebook_convert(), + str(epub_path), + str(pdf_path), ] - - if spine_element is not None: - toc_id = str(spine_element.attrib.get("toc", "")).strip() - if toc_id and toc_id in manifest_by_id: - ncx_path = _normalize_epub_href(package_path, manifest_by_id[toc_id]["href"]) - - with zipfile.ZipFile(epub_path, "r") as zf: - toc_entries: list[EpubTocEntry] - if nav_path: - toc_entries = _extract_epub_nav_toc(zf, nav_path) - elif ncx_path: - toc_entries = _extract_epub_ncx_toc(zf, ncx_path) - else: - toc_entries = [] - - if not toc_entries: - toc_entries = _guess_epub_fallback_toc(spine_ids, manifest_by_id, package_path=package_path) - - if not toc_entries: - raise ValueError(f"Could not extract a table of contents from {epub_path.name}") - - return title, toc_entries - - -def _serialize_epub_toc_entries(entries: list[EpubTocEntry]) -> list[dict]: - used: dict[str, int] = {} - - def _convert(entry: EpubTocEntry) -> dict: - title_base = _slugify_epub(entry.title) - href_base = _slugify_epub(PurePosixPath(entry.href.split("#", 1)[0]).stem) - base = title_base or href_base or "section" - count = used.get(base, 0) + 1 - used[base] = count - slug = base if count == 1 else f"{base}-{count}" - - data = { - "title": entry.title, - "slug": slug, - "href": entry.href, - } - if entry.children: - data["children"] = [_convert(child) for child in entry.children] - return data - - return [_convert(entry) for entry in entries] - - -def _generate_epub_readme(title: str, toc_entries: list[dict]) -> str: - lines = [f"# {title}", "", "## Chapters", ""] - for entry in toc_entries: - lines.append(f"- {entry['title']}") - lines.extend( - [ - "", - "---", - "", - "> **Disclaimer:** This content is provided for personal study and research " - "purposes only. All rights belong to the original authors and copyright holders. " - "This is not an authorized distribution. If you are the rights holder and wish " - "to have this content removed, please contact the repository owner.", - "", - ] - ) - return "\n".join(lines) - - -def _build_epub_book(epub_path: Path, output_dir: Path, book_id: str) -> tuple[str, list[dict]]: - """Store the original EPUB and generate lightweight TOC metadata.""" - epub_bytes = epub_path.read_bytes() - title, raw_toc = _build_epub_toc_data(epub_path) - toc_entries = _serialize_epub_toc_entries(raw_toc) - - book_dir = output_dir / book_id - if book_dir.exists(): - shutil.rmtree(book_dir) - book_dir.mkdir(parents=True, exist_ok=True) - - (book_dir / EPUB_BOOK_FILENAME).write_bytes(epub_bytes) - (book_dir / "toc.json").write_text( - json.dumps({"title": title, "children": toc_entries}, indent=2, ensure_ascii=False) + "\n", - encoding="utf-8", - ) - (book_dir / "README.md").write_text( - _generate_epub_readme(title, toc_entries), - encoding="utf-8", - ) - - return title, toc_entries + try: + result = subprocess.run( + command, + check=True, + capture_output=True, + text=True, + ) + except subprocess.CalledProcessError as exc: + detail = (exc.stderr or exc.stdout or "").strip() + if detail: + detail = detail.splitlines()[-1] + raise RuntimeError( + f"Failed to convert EPUB to PDF with calibre: {detail or exc}" + ) from exc + + if not pdf_path.exists(): + detail = (result.stderr or result.stdout or "").strip() + raise RuntimeError( + "Calibre did not produce a PDF output" + + (f": {detail.splitlines()[-1]}" if detail else ".") + ) def process_epub(epub_path: Path, output_dir: Path) -> None: - """Process a single .epub file into a raw EPUB-backed book.""" - book_id = ensure_unique_content_id(_generate_id(epub_path), output_dir.parent, "book") - md5 = _file_md5(epub_path) - book_dir = output_dir / book_id - timestamp = _utc_now_iso() - created_at = _read_existing_created_at(book_dir, timestamp) - - print(f"Processing EPUB: {epub_path.name} -> {book_id}") - - title, toc_entries = _build_epub_book(epub_path, output_dir, book_id) - print(f" Extracted EPUB TOC with {len(toc_entries)} top-level entries") - - _write_epub_metadata( - book_dir, - book_id=book_id, - source_epub=epub_path.name, - epub_md5=md5, - created_at=created_at, - updated_at=timestamp, - ) + """Process a single .epub file by converting it to PDF first.""" + print(f"Processing EPUB via Calibre: {epub_path.name}") + + with tempfile.TemporaryDirectory(prefix="gitshelf_epub_pdf_") as tmp_dir: + temp_pdf = Path(tmp_dir) / f"{epub_path.stem}.pdf" + _convert_epub_to_pdf(epub_path, temp_pdf) + convert_single_pdf(temp_pdf, output_dir, source_name=epub_path.name) epub_path.unlink(missing_ok=True) print(f" Deleted source: {epub_path.name}") -def reconvert_epub_from_cache(source_epub: str, output_dir: Path) -> None: - """Rebuild an EPUB-backed book from the stored raw EPUB source.""" - book_dir: Path | None = None - for meta_file in output_dir.glob(f"*/{BOOK_METADATA_FILENAME}"): - try: - meta = json.loads(meta_file.read_text(encoding="utf-8")) - except json.JSONDecodeError: - continue - - if ( - meta.get("source") == source_epub - and meta.get("source_format") == "epub" - ): - book_dir = meta_file.parent - break - - if book_dir is None: - raise FileNotFoundError( - f"No stored EPUB source found for {source_epub}. Re-upload the EPUB." - ) - - source_path = book_dir / EPUB_BOOK_FILENAME - if not source_path.exists(): - raise FileNotFoundError( - f"Stored EPUB missing at {source_path}. Re-upload the EPUB." - ) - - book_id = book_dir.name - md5 = _file_md5(source_path) - timestamp = _utc_now_iso() - created_at = _read_existing_created_at(book_dir, timestamp) - print(f"Reconverting EPUB from stored source: {source_epub} -> {book_id} (md5={md5})") - title, toc_entries = _build_epub_book(source_path, output_dir, book_id) - print(f" Extracted EPUB TOC with {len(toc_entries)} top-level entries") - - _write_epub_metadata( - book_dir, - book_id=book_id, - source_epub=source_epub, - epub_md5=md5, - created_at=created_at, - updated_at=timestamp, - ) - print(" Reconversion complete.") - - # --- Markdown processing --- def _count_words(text: str) -> int: @@ -791,7 +413,7 @@ def main() -> None: elif input_filename.lower().endswith(".epub"): print(f"EPUB not found, attempting reconvert from cache: {input_filename}") try: - reconvert_epub_from_cache(input_filename, books_dir) + reconvert_from_cache(input_filename, books_dir) build_manifest( books_dir=books_dir, output_path=manifest_path, @@ -838,7 +460,6 @@ def main() -> None: for epub_path in epub_jobs: try: process_epub(epub_path, books_dir) - _remove_failure(epub_path.name, args.output_dir) except Exception as exc: print(f" FAILED: {epub_path.name}: {exc}", file=sys.stderr) failures.append((epub_path, exc)) diff --git a/src/components/App.jsx b/src/components/App.jsx index 0085a57..5dfd79e 100644 --- a/src/components/App.jsx +++ b/src/components/App.jsx @@ -9,12 +9,11 @@ import { ScrollToTop } from './ScrollToTop'; import { ToastContainer } from './ToastContainer'; import { HomeView } from './HomeView'; import { ChapterReader } from './ChapterReader'; -import { EpubReader } from './EpubReader'; import { BookOverview } from './BookOverview'; import { ArticleReader } from './ArticleReader'; import { Footer } from './Footer'; import { SearchPanel } from './SearchPanel'; -import { fetchManifest, getBookSourceFormat } from '../lib/api'; +import { fetchManifest } from '../lib/api'; const AdminView = lazy(() => import('./AdminView')); @@ -27,15 +26,10 @@ export function App() { const [breadcrumb, setBreadcrumb] = useState(''); const [activeAnchor, setActiveAnchor] = useState(null); const [searchOpen, setSearchOpen] = useState(false); - const [activeBook, setActiveBook] = useState(undefined); - const [epubProgress, setEpubProgress] = useState(0); const isChapter = route.type === 'chapter'; const isBookRoute = route.type === 'book-overview' || route.type === 'chapter'; - const activeBookSourceFormat = activeBook === undefined ? null : getBookSourceFormat(activeBook); - const isEpubBook = activeBookSourceFormat === 'epub'; - const { progress: scrollProgress, showScrollTop, scrollToTop } = useScrollProgress(isChapter && !isEpubBook); - const progress = isEpubBook ? epubProgress : scrollProgress; + const { progress, showScrollTop, scrollToTop } = useScrollProgress(isChapter); const handleToggleSidebar = useCallback(() => { setSidebarOpen((prev) => !prev); @@ -54,10 +48,6 @@ export function App() { setActiveAnchor(anchor); }, []); - const handleEpubProgress = useCallback((nextProgress) => { - setEpubProgress(nextProgress); - }, []); - const handleOpenSearch = useCallback(() => setSearchOpen(true), []); const handleCloseSearch = useCallback(() => setSearchOpen(false), []); @@ -92,32 +82,6 @@ export function App() { } }, [route.type, route.siteId]); - useEffect(() => { - if (!isBookRoute) { - setActiveBook(undefined); - return undefined; - } - - let cancelled = false; - setActiveBook(undefined); - setEpubProgress(0); - fetchManifest() - .then((manifest) => { - if (cancelled) return; - const nextBook = manifest.items?.find((item) => item.id === route.bookId) || null; - setActiveBook(nextBook); - }) - .catch(() => { - if (!cancelled) { - setActiveBook(null); - } - }); - - return () => { - cancelled = true; - }; - }, [isBookRoute, route.bookId]); - // Reset sidebar state when leaving book routes useEffect(() => { if (!isBookRoute) { @@ -125,13 +89,11 @@ export function App() { setBreadcrumb(''); setSidebarOpen(false); setActiveAnchor(null); - setEpubProgress(0); } }, [isBookRoute]); const showSidebar = isBookRoute; - const showSearch = isBookRoute && activeBook !== undefined && activeBookSourceFormat !== 'epub'; - const waitingForBookFormat = route.type === 'chapter' && activeBook === undefined; + const showSearch = isBookRoute; return ( <> @@ -175,26 +137,13 @@ export function App() { /> )} {route.type === 'chapter' && ( - waitingForBookFormat ? ( -
Loading book...
- ) : isEpubBook ? ( - - ) : ( - - ) + )} {route.type === 'article' && ( diff --git a/src/components/EpubReader.jsx b/src/components/EpubReader.jsx deleted file mode 100644 index 4d2fbe5..0000000 --- a/src/components/EpubReader.jsx +++ /dev/null @@ -1,486 +0,0 @@ -import { useEffect, useRef, useState, useCallback } from 'preact/hooks'; -import { fetchToc, flattenChapters } from '../lib/api'; -import { ChapterNav } from './ChapterNav'; - -const STORAGE_PREFIX = 'gitshelf:epub-location:'; -const LOCATION_GENERATE_CHARS = 1600; - -function normalizeHref(href) { - return String(href || '').trim().replace(/^\.\//, ''); -} - -function stripFragment(href) { - return normalizeHref(href).split('#', 1)[0]; -} - -function findChapterBySlug(chapters, slug) { - return chapters.find((chapter) => chapter.slug === slug) || null; -} - -function findChapterByHref(chapters, href) { - const normalizedHref = normalizeHref(href); - if (!normalizedHref) return null; - - const exact = chapters.find((chapter) => normalizeHref(chapter.href) === normalizedHref); - if (exact) return exact; - - const sectionPath = stripFragment(normalizedHref); - return chapters.find((chapter) => stripFragment(chapter.href) === sectionPath) || null; -} - -function readStoredLocation(bookId) { - try { - const raw = localStorage.getItem(`${STORAGE_PREFIX}${bookId}`); - if (!raw) return null; - const parsed = JSON.parse(raw); - if (!parsed || typeof parsed !== 'object') return null; - return { - slug: typeof parsed.slug === 'string' ? parsed.slug : null, - cfi: typeof parsed.cfi === 'string' ? parsed.cfi : null, - }; - } catch { - return null; - } -} - -function writeStoredLocation(bookId, value) { - try { - localStorage.setItem(`${STORAGE_PREFIX}${bookId}`, JSON.stringify(value)); - } catch { - // Ignore storage failures and keep reading. - } -} - -function dispatchHashChange() { - try { - window.dispatchEvent(new HashChangeEvent('hashchange')); - } catch { - window.dispatchEvent(new Event('hashchange')); - } -} - -function replaceBookHash(bookId, slug) { - const nextHash = `#/books/${encodeURIComponent(bookId)}/${encodeURIComponent(slug)}`; - if (window.location.hash === nextHash) return; - window.history.replaceState(window.history.state, '', nextHash); - dispatchHashChange(); -} - -function buildReaderThemeCss() { - const rootStyles = getComputedStyle(document.documentElement); - const background = rootStyles.getPropertyValue('--color-bg-surface').trim(); - const text = rootStyles.getPropertyValue('--color-text-primary').trim(); - const heading = rootStyles.getPropertyValue('--color-text-heading').trim(); - const secondary = rootStyles.getPropertyValue('--color-text-secondary').trim(); - const tertiary = rootStyles.getPropertyValue('--color-text-tertiary').trim(); - const accent = rootStyles.getPropertyValue('--color-accent').trim(); - const accentHover = rootStyles.getPropertyValue('--color-accent-hover').trim(); - const border = rootStyles.getPropertyValue('--color-border').trim(); - const subtle = rootStyles.getPropertyValue('--color-bg-subtle').trim(); - const code = rootStyles.getPropertyValue('--color-bg-code').trim(); - const readingFont = rootStyles.getPropertyValue('--font-reading').trim(); - const uiFont = rootStyles.getPropertyValue('--font-ui').trim(); - const codeFont = rootStyles.getPropertyValue('--font-code').trim(); - - return ` - html, - body { - margin: 0 !important; - padding: 0 !important; - background: transparent !important; - color: ${text} !important; - font-family: ${readingFont} !important; - line-height: 1.8 !important; - -webkit-font-smoothing: antialiased; - text-rendering: optimizeLegibility; - } - - body { - padding: 24px 24px 28vh !important; - box-sizing: border-box !important; - word-break: break-word; - font-size: 17px !important; - } - - h1, h2, h3, h4, h5, h6 { - color: ${heading} !important; - font-family: ${readingFont} !important; - line-height: 1.35 !important; - margin: 1.8em 0 0.7em !important; - } - - h1 { font-size: 1.85em !important; margin-top: 0 !important; } - h2 { font-size: 1.45em !important; } - h3 { font-size: 1.2em !important; } - h4, h5, h6 { font-size: 1.05em !important; } - - p, li, blockquote, dd { - line-height: 1.8 !important; - } - - p { - margin: 0 0 1.35em !important; - } - - ul, ol { - margin: 0 0 1.35em !important; - padding-left: 1.5em !important; - } - - li + li { - margin-top: 0.35em !important; - } - - a { - color: ${accent} !important; - text-decoration-thickness: 1px !important; - text-underline-offset: 2px !important; - } - - a:hover { - color: ${accentHover} !important; - } - - blockquote { - margin: 1.8em 0 !important; - padding: 16px 24px !important; - border-left: 3px solid ${accent} !important; - background: ${subtle} !important; - color: ${secondary} !important; - } - - pre, - code, - kbd, - samp { - font-family: ${codeFont} !important; - } - - pre { - background: ${code} !important; - border: 1px solid ${border} !important; - border-radius: 8px !important; - padding: 16px !important; - overflow: auto !important; - line-height: 1.6 !important; - } - - code { - background: ${code} !important; - border-radius: 6px !important; - padding: 0.12em 0.35em !important; - } - - pre code { - background: transparent !important; - padding: 0 !important; - } - - img, - svg, - video { - max-width: 100% !important; - height: auto !important; - } - - img { - display: block !important; - margin: 1.8em auto !important; - border-radius: 10px !important; - } - - hr { - border: none !important; - border-top: 1px solid ${border} !important; - margin: 3em 0 !important; - } - - table { - width: 100% !important; - border-collapse: collapse !important; - margin: 1.6em 0 !important; - font-family: ${uiFont} !important; - font-size: 0.95em !important; - } - - th, - td { - border: 1px solid ${border} !important; - padding: 8px 12px !important; - text-align: left !important; - } - - th { - background: ${subtle} !important; - color: ${heading} !important; - } - - figcaption { - color: ${tertiary} !important; - font-family: ${uiFont} !important; - font-size: 0.9em !important; - text-align: center !important; - } - `; -} - -export function EpubReader({ - bookId, - slug, - onTocLoaded, - onActiveAnchor, - onProgressChange, - theme, -}) { - const [tocData, setTocData] = useState(null); - const [error, setError] = useState(null); - const [ready, setReady] = useState(false); - const [displaying, setDisplaying] = useState(true); - const [currentChapter, setCurrentChapter] = useState(null); - const viewerRef = useRef(null); - const bookRef = useRef(null); - const renditionRef = useRef(null); - const chaptersRef = useRef([]); - const currentSlugRef = useRef(slug); - const currentChapterRef = useRef(null); - const displayTokenRef = useRef(0); - - useEffect(() => { - currentSlugRef.current = slug; - }, [slug]); - - const applyTheme = useCallback(() => { - const rendition = renditionRef.current; - if (!rendition) return; - rendition.themes.registerCss('gitshelf-reader', buildReaderThemeCss()); - rendition.themes.select('gitshelf-reader'); - rendition.themes.override('font-size', '17px'); - }, []); - - const displayChapter = useCallback(async (nextSlug, { preferStored = false } = {}) => { - const rendition = renditionRef.current; - const chapters = chaptersRef.current; - if (!rendition || chapters.length === 0) return; - - const chapter = findChapterBySlug(chapters, nextSlug) || chapters[0]; - if (!chapter) return; - - if (currentChapterRef.current?.slug === chapter.slug) { - setCurrentChapter(chapter); - setDisplaying(false); - return; - } - - const stored = preferStored ? readStoredLocation(bookId) : null; - const target = stored?.slug === chapter.slug && stored?.cfi ? stored.cfi : chapter.href; - const token = displayTokenRef.current + 1; - displayTokenRef.current = token; - currentSlugRef.current = chapter.slug; - currentChapterRef.current = chapter; - setCurrentChapter(chapter); - setDisplaying(true); - onActiveAnchor?.(null); - - try { - await rendition.display(target); - if (displayTokenRef.current === token) { - setDisplaying(false); - } - } catch (err) { - if (displayTokenRef.current === token) { - setError(err instanceof Error ? err.message : String(err)); - setDisplaying(false); - } - } - }, [bookId, onActiveAnchor]); - - useEffect(() => { - let cancelled = false; - setError(null); - setReady(false); - setDisplaying(true); - setCurrentChapter(null); - currentChapterRef.current = null; - chaptersRef.current = []; - onProgressChange?.(0); - onActiveAnchor?.(null); - - fetchToc(bookId) - .then((toc) => { - if (cancelled) return; - setTocData(toc); - chaptersRef.current = flattenChapters(toc.children || []); - onTocLoaded(toc); - }) - .catch((err) => { - if (cancelled) return; - setError(err instanceof Error ? err.message : String(err)); - }); - - return () => { - cancelled = true; - }; - }, [bookId, onActiveAnchor, onProgressChange, onTocLoaded]); - - useEffect(() => { - if (!tocData || !viewerRef.current) return undefined; - - let cancelled = false; - let rendition = null; - let book = null; - - const handleRelocated = (location) => { - if (!location?.start) return; - - const chapters = chaptersRef.current; - const activeChapter = currentChapterRef.current || findChapterBySlug(chapters, currentSlugRef.current); - const matchedChapter = findChapterByHref(chapters, location.start.href); - const progress = Number.isFinite(location.start.percentage) - ? Math.max(0, Math.min(100, location.start.percentage * 100)) - : 0; - - onProgressChange?.(progress); - - const storedChapter = matchedChapter || activeChapter; - if (storedChapter?.slug && location.start.cfi) { - writeStoredLocation(bookId, { - slug: storedChapter.slug, - cfi: location.start.cfi, - }); - } - - if ( - matchedChapter && - matchedChapter.slug !== currentSlugRef.current && - stripFragment(matchedChapter.href) !== stripFragment(activeChapter?.href || '') - ) { - currentSlugRef.current = matchedChapter.slug; - currentChapterRef.current = matchedChapter; - setCurrentChapter(matchedChapter); - replaceBookHash(bookId, matchedChapter.slug); - } - - setDisplaying(false); - }; - - const handleRendered = (section) => { - if (!section || !tocData) return; - const matchedChapter = currentChapterRef.current || findChapterByHref(chaptersRef.current, section.href); - const titleParts = [ - matchedChapter?.title, - tocData.title || bookId, - 'GitShelf', - ].filter(Boolean); - document.title = titleParts.join(' \u00b7 '); - }; - - async function setupRendition() { - try { - const { default: ePub } = await import('epubjs'); - if (cancelled || !viewerRef.current) return; - - viewerRef.current.innerHTML = ''; - book = ePub(`./books/${encodeURIComponent(bookId)}/book.epub`); - rendition = book.renderTo(viewerRef.current, { - width: '100%', - height: '100%', - manager: 'continuous', - flow: 'scrolled-doc', - spread: 'none', - }); - - bookRef.current = book; - renditionRef.current = rendition; - rendition.on('relocated', handleRelocated); - rendition.on('rendered', handleRendered); - - applyTheme(); - await book.ready; - if (cancelled) return; - setReady(true); - - book.locations.generate(LOCATION_GENERATE_CHARS) - .then(() => rendition.reportLocation()) - .catch(() => { - // Percentage progress is optional for older or unusual EPUB files. - }); - } catch (err) { - if (cancelled) return; - setError(err instanceof Error ? err.message : String(err)); - setDisplaying(false); - } - } - - setupRendition(); - - return () => { - cancelled = true; - onProgressChange?.(0); - setReady(false); - if (rendition) { - rendition.off('relocated', handleRelocated); - rendition.off('rendered', handleRendered); - } - if (book) { - book.destroy(); - } - if (viewerRef.current) { - viewerRef.current.innerHTML = ''; - } - renditionRef.current = null; - bookRef.current = null; - }; - }, [applyTheme, bookId, onProgressChange, tocData]); - - useEffect(() => { - applyTheme(); - }, [applyTheme, theme]); - - useEffect(() => { - if (!ready || !tocData) return; - displayChapter(slug, { preferStored: true }); - }, [displayChapter, ready, slug, tocData]); - - const chapters = tocData ? flattenChapters(tocData.children || []) : []; - const currentIndex = currentChapter ? chapters.findIndex((chapter) => chapter.slug === currentChapter.slug) : -1; - - useEffect(() => { - if (currentIndex === -1) return undefined; - - const onKeyDown = (event) => { - const tag = event.target.tagName; - if (tag === 'INPUT' || tag === 'TEXTAREA' || event.target.isContentEditable) return; - if (event.key === 'ArrowLeft' && currentIndex > 0) { - window.location.hash = `#/books/${encodeURIComponent(bookId)}/${encodeURIComponent(chapters[currentIndex - 1].slug)}`; - } else if (event.key === 'ArrowRight' && currentIndex < chapters.length - 1) { - window.location.hash = `#/books/${encodeURIComponent(bookId)}/${encodeURIComponent(chapters[currentIndex + 1].slug)}`; - } - }; - - window.addEventListener('keydown', onKeyDown); - return () => window.removeEventListener('keydown', onKeyDown); - }, [bookId, chapters, currentIndex]); - - if (error) { - return
Failed to load: {error}
; - } - - if (!tocData) { - return
Loading EPUB...
; - } - - return ( -
-
-
- {displaying &&
Loading chapter...
} -
- -
- ); -} diff --git a/src/lib/api.js b/src/lib/api.js index 9f1ecdc..a70ad08 100644 --- a/src/lib/api.js +++ b/src/lib/api.js @@ -27,10 +27,6 @@ export function flattenChapters(items) { return result; } -export function getBookSourceFormat(item) { - return item?.source_format || 'markdown-derived'; -} - export function getItemDisplayTitle(item) { if (!item) return 'Untitled'; const raw = String(item.title || item.display_title || item.id || 'Untitled'); diff --git a/src/lib/github-api.js b/src/lib/github-api.js index 070de7d..2bd1fc6 100644 --- a/src/lib/github-api.js +++ b/src/lib/github-api.js @@ -302,11 +302,9 @@ async function listRepoTree(repo, path) { async function getCacheDeleteOps(repo, itemId) { let pdfMd5; - let epubMd5; try { const f = await readRepositoryJson(repo, `docs/books/${itemId}/meta.json`); pdfMd5 = f.data?.pdf_md5; - epubMd5 = f.data?.epub_md5; } catch { /* not found */ } const ops = []; @@ -317,13 +315,6 @@ async function getCacheDeleteOps(repo, itemId) { .map(f => ({ path: f.path, delete: true }))); } - if (epubMd5) { - const cacheFiles = await listRepoTree(repo, 'cache/epub'); - ops.push(...cacheFiles - .filter(f => f.name.startsWith(epubMd5)) - .map(f => ({ path: f.path, delete: true }))); - } - return ops; } diff --git a/src/style.css b/src/style.css index 4b22978..530d245 100644 --- a/src/style.css +++ b/src/style.css @@ -957,14 +957,6 @@ button { .book-overview-meta span::after { display: none; } - - .epub-reader-shell { - min-height: 58vh; - } - - .epub-reader-frame { - min-height: 420px; - } } /* ========================================================================== @@ -994,80 +986,6 @@ button { font-feature-settings: "kern" 1, "liga" 1, "onum" 1; } -.epub-reader { - display: grid; - gap: var(--space-6); -} - -.epub-reader-shell { - position: relative; - max-width: var(--content-max-width); - margin: 0 auto; - min-height: 70vh; - border: 1px solid var(--color-border); - border-radius: var(--radius-lg); - background: - linear-gradient(180deg, rgba(255, 255, 255, 0.36), rgba(255, 255, 255, 0.08)), - var(--color-bg-surface); - box-shadow: var(--shadow-subtle); - overflow: hidden; -} - -[data-theme="dark"] .epub-reader-shell { - background: - linear-gradient(180deg, rgba(255, 255, 255, 0.04), rgba(255, 255, 255, 0.01)), - var(--color-bg-surface); -} - -.epub-reader-shell--loading::after { - content: ''; - position: absolute; - inset: 0; - background: linear-gradient(180deg, rgba(248, 242, 235, 0.12), rgba(248, 242, 235, 0.02)); - pointer-events: none; -} - -[data-theme="dark"] .epub-reader-shell--loading::after { - background: linear-gradient(180deg, rgba(28, 25, 21, 0.2), rgba(28, 25, 21, 0.05)); -} - -.epub-reader-frame { - height: calc(100dvh - var(--top-bar-height) - 9rem); - min-height: 520px; -} - -.epub-reader-frame, -.epub-reader-frame > div, -.epub-reader-frame iframe { - width: 100%; - height: 100%; -} - -.epub-reader-frame iframe { - display: block; - border: none; - background: transparent; -} - -.epub-reader-loading { - position: absolute; - right: var(--space-4); - bottom: var(--space-4); - padding: var(--space-2) var(--space-3); - border-radius: 999px; - background-color: rgba(42, 34, 27, 0.78); - color: #fff; - font-family: var(--font-ui); - font-size: 12px; - letter-spacing: 0.02em; - box-shadow: var(--shadow-subtle); -} - -[data-theme="dark"] .epub-reader-loading { - background-color: rgba(243, 235, 228, 0.12); - color: var(--color-text-primary); -} - .reader-content h1, .reader-content h2, .reader-content h3, @@ -3942,11 +3860,6 @@ button { font-size: 16px; line-height: 1.7; } - - .epub-reader-frame { - height: calc(100dvh - var(--top-bar-height) - 7rem); - min-height: 460px; - } } @media (max-width: 519px) { diff --git a/tests/components/AppEpub.test.jsx b/tests/components/AppEpub.test.jsx deleted file mode 100644 index 4db7a7a..0000000 --- a/tests/components/AppEpub.test.jsx +++ /dev/null @@ -1,66 +0,0 @@ -import { describe, it, expect, vi, beforeEach } from 'vitest'; -import { render, screen, waitFor } from '@testing-library/preact'; - -vi.mock('../../src/components/EpubReader', () => ({ - EpubReader: ({ bookId, slug }) => ( -
- EPUB:{bookId}:{slug} -
- ), -})); - -import { App } from '../../src/components/App'; - -function mockFetch(responses = {}) { - return vi.fn((url) => { - const key = Object.keys(responses).find((candidate) => String(url).includes(candidate)); - if (!key) { - return Promise.resolve({ - ok: false, - status: 404, - json: () => Promise.resolve({}), - text: () => Promise.resolve('Not found'), - }); - } - - const body = responses[key]; - return Promise.resolve({ - ok: true, - status: 200, - json: () => Promise.resolve(body), - text: () => Promise.resolve(typeof body === 'string' ? body : JSON.stringify(body)), - }); - }); -} - -beforeEach(() => { - global.fetch = undefined; - localStorage.clear(); - window.location.hash = '#/books/demo-book/chapter-1'; -}); - -describe('App EPUB routing', () => { - it('uses the EPUB reader for raw epub books instead of markdown chapters', async () => { - const fetchMock = mockFetch({ - 'manifest.json': { - items: [ - { - id: 'demo-book', - type: 'book', - title: 'Demo EPUB', - source_format: 'epub', - chapters_count: 2, - }, - ], - }, - }); - global.fetch = fetchMock; - - render(); - - expect(await screen.findByTestId('epub-reader')).toHaveTextContent('EPUB:demo-book:chapter-1'); - expect(screen.queryByLabelText(/Search book/i)).not.toBeInTheDocument(); - await waitFor(() => expect(fetchMock).toHaveBeenCalled()); - expect(fetchMock.mock.calls.some(([url]) => String(url).includes('/chapters/'))).toBe(false); - }); -}); diff --git a/tests/components/EpubReader.test.jsx b/tests/components/EpubReader.test.jsx deleted file mode 100644 index e7fbb70..0000000 --- a/tests/components/EpubReader.test.jsx +++ /dev/null @@ -1,109 +0,0 @@ -import { describe, it, expect, vi, beforeEach } from 'vitest'; -import { render, waitFor } from '@testing-library/preact'; -import { EpubReader } from '../../src/components/EpubReader'; - -const renditionDisplayMock = vi.fn(() => Promise.resolve()); -const renditionOnMock = vi.fn(); -const renditionOffMock = vi.fn(); -const renditionReportLocationMock = vi.fn(() => Promise.resolve()); -const renditionMock = { - display: renditionDisplayMock, - on: renditionOnMock, - off: renditionOffMock, - reportLocation: renditionReportLocationMock, - themes: { - registerCss: vi.fn(), - select: vi.fn(), - override: vi.fn(), - }, -}; - -const bookDestroyMock = vi.fn(); -const bookLocationsGenerateMock = vi.fn(() => Promise.resolve()); -const bookRenderToMock = vi.fn(() => renditionMock); -const epubFactoryMock = vi.fn(() => ({ - renderTo: bookRenderToMock, - ready: Promise.resolve(), - locations: { - generate: bookLocationsGenerateMock, - }, - destroy: bookDestroyMock, -})); - -vi.mock('epubjs', () => ({ - default: epubFactoryMock, -})); - -function mockFetch(responses = {}) { - return vi.fn((url) => { - const key = Object.keys(responses).find((candidate) => String(url).includes(candidate)); - if (!key) { - return Promise.resolve({ - ok: false, - status: 404, - json: () => Promise.resolve({}), - text: () => Promise.resolve('Not found'), - }); - } - - const body = responses[key]; - return Promise.resolve({ - ok: true, - status: 200, - json: () => Promise.resolve(body), - text: () => Promise.resolve(typeof body === 'string' ? body : JSON.stringify(body)), - }); - }); -} - -beforeEach(() => { - localStorage.clear(); - renditionDisplayMock.mockClear(); - renditionOnMock.mockClear(); - renditionOffMock.mockClear(); - renditionReportLocationMock.mockClear(); - bookDestroyMock.mockClear(); - bookLocationsGenerateMock.mockClear(); - bookRenderToMock.mockClear(); - epubFactoryMock.mockClear(); - - global.fetch = mockFetch({ - 'books/demo-book/toc.json': { - title: 'Demo EPUB', - children: [ - { title: 'Chapter One', slug: 'chapter-1', href: 'OPS/chapter-1.xhtml' }, - { title: 'Chapter Two', slug: 'chapter-2', href: 'OPS/chapter-2.xhtml' }, - ], - }, - }); -}); - -describe('EpubReader', () => { - it('prefers a stored CFI when reopening the same EPUB chapter', async () => { - localStorage.setItem( - 'gitshelf:epub-location:demo-book', - JSON.stringify({ - slug: 'chapter-1', - cfi: 'epubcfi(/6/2[chapter-1]!/4/2/14)', - }), - ); - - render( - {}} - onActiveAnchor={() => {}} - onProgressChange={() => {}} - theme="light" - />, - ); - - await waitFor(() => expect(epubFactoryMock).toHaveBeenCalledWith('./books/demo-book/book.epub')); - await waitFor(() => - expect(renditionDisplayMock).toHaveBeenCalledWith('epubcfi(/6/2[chapter-1]!/4/2/14)'), - ); - expect(bookRenderToMock).toHaveBeenCalled(); - expect(bookLocationsGenerateMock).toHaveBeenCalled(); - }); -}); diff --git a/tests/scripts/test_build_manifest.py b/tests/scripts/test_build_manifest.py index 2f88c04..4f68731 100644 --- a/tests/scripts/test_build_manifest.py +++ b/tests/scripts/test_build_manifest.py @@ -23,18 +23,14 @@ def _create_book( source: str | None = None, source_format: str | None = None, toc_children: list[dict] | None = None, - raw_epub: bool = False, ) -> None: book_dir = books_dir / book_id book_dir.mkdir(parents=True, exist_ok=True) _write_json(book_dir / "toc.json", {"title": title, "children": toc_children or []}) - if raw_epub: - (book_dir / "book.epub").write_bytes(b"epub-bytes") - else: - chapters_dir = book_dir / "chapters" - chapters_dir.mkdir(parents=True, exist_ok=True) - (chapters_dir / "01-intro.md").write_text("# Intro\n\nhello world\n", encoding="utf-8") + chapters_dir = book_dir / "chapters" + chapters_dir.mkdir(parents=True, exist_ok=True) + (chapters_dir / "01-intro.md").write_text("# Intro\n\nhello world\n", encoding="utf-8") if source or source_format: data = { @@ -167,7 +163,7 @@ def test_create_default_metadata_and_source_fallback(self) -> None: self.assertEqual(manifest["items"][0]["type"], "book") self.assertEqual(manifest["items"][0]["source_format"], "markdown-derived") - def test_raw_epub_books_use_toc_counts_and_epub_source_format(self) -> None: + def test_books_can_keep_original_epub_source_name_after_conversion(self) -> None: with tempfile.TemporaryDirectory() as tmp_dir: root = Path(tmp_dir) books_dir = root / "docs" / "books" @@ -182,25 +178,25 @@ def test_raw_epub_books_use_toc_counts_and_epub_source_format(self) -> None: book_id="epub-book", title="EPUB Book", source="epub-book.epub", - source_format="epub", - raw_epub=True, + source_format="markdown-derived", toc_children=[ { "title": "Chapter One", "slug": "chapter-one", - "href": "OPS/chapter-1.xhtml", + "href": "chapter-1", "children": [ { "title": "Section One", "slug": "section-one", - "href": "OPS/chapter-1.xhtml#section-1", + "href": "chapter-1", + "anchor": "section-1", } ], }, { "title": "Chapter Two", "slug": "chapter-two", - "href": "OPS/chapter-2.xhtml", + "href": "chapter-2", }, ], ) @@ -215,10 +211,10 @@ def test_raw_epub_books_use_toc_counts_and_epub_source_format(self) -> None: ) manifest = json.loads(manifest_path.read_text(encoding="utf-8")) - self.assertEqual(manifest["items"][0]["chapters_count"], 3) - self.assertIsNone(manifest["items"][0]["word_count"]) + self.assertEqual(manifest["items"][0]["chapters_count"], 1) + self.assertEqual(manifest["items"][0]["word_count"], 4) self.assertEqual(manifest["items"][0]["source"], "epub-book.epub") - self.assertEqual(manifest["items"][0]["source_format"], "epub") + self.assertEqual(manifest["items"][0]["source_format"], "markdown-derived") def test_invalid_metadata_fails_loudly(self) -> None: with tempfile.TemporaryDirectory() as tmp_dir: diff --git a/tests/scripts/test_process.py b/tests/scripts/test_process.py index 09da349..7c9653d 100644 --- a/tests/scripts/test_process.py +++ b/tests/scripts/test_process.py @@ -1,167 +1,79 @@ -import hashlib -import json -import sys +import os import tempfile import unittest -import zipfile from pathlib import Path from unittest.mock import patch REPO_ROOT = Path(__file__).resolve().parents[2] +import sys + sys.path.insert(0, str(REPO_ROOT / "scripts")) import process -def _build_sample_epub(path: Path, *, title: str = "Sample EPUB") -> bytes: - path.parent.mkdir(parents=True, exist_ok=True) - with zipfile.ZipFile(path, "w") as zf: - zf.writestr("mimetype", "application/epub+zip") - zf.writestr( - "META-INF/container.xml", - """ - - - - - -""", - ) - zf.writestr( - "OEBPS/content.opf", - f""" - - - urn:uuid:test-book - {title} - - - - - - - - - - - -""", - ) - zf.writestr( - "OEBPS/nav.xhtml", - """ - - TOC - - - - -""", - ) - zf.writestr( - "OEBPS/chapter-1.xhtml", - """

Chapter One

Section One

Hello world.

""", - ) - zf.writestr( - "OEBPS/chapter-2.xhtml", - """

Chapter Two

More text.

""", - ) - return path.read_bytes() - - class EpubProcessingTest(unittest.TestCase): - def test_process_epub_stores_original_file_and_generated_toc(self) -> None: + def test_process_epub_converts_to_pdf_and_reuses_pdf_pipeline(self) -> None: with tempfile.TemporaryDirectory() as tmp_dir: root = Path(tmp_dir) books_dir = root / "docs" / "books" books_dir.mkdir(parents=True, exist_ok=True) epub_path = root / "input" / "sample.epub" - epub_bytes = _build_sample_epub(epub_path) - expected_md5 = hashlib.md5(epub_bytes).hexdigest() + epub_path.parent.mkdir(parents=True, exist_ok=True) + epub_path.write_bytes(b"epub-bytes") + + converted_pdfs: list[Path] = [] - with patch.object(process, "_utc_now_iso", return_value="2026-04-12T12:00:00Z"): - process.process_epub(epub_path, books_dir) + def fake_convert_epub_to_pdf(source: Path, dest: Path) -> None: + self.assertEqual(source, epub_path) + dest.write_bytes(b"%PDF-1.4\n") + converted_pdfs.append(dest) + + with patch.object(process, "_convert_epub_to_pdf", side_effect=fake_convert_epub_to_pdf) as convert_mock: + with patch.object(process, "convert_single_pdf") as convert_pdf_mock: + process.process_epub(epub_path, books_dir) - book_dir = books_dir / "sample" self.assertFalse(epub_path.exists()) - self.assertFalse((book_dir / "chapters").exists()) - self.assertTrue((book_dir / "book.epub").exists()) - self.assertEqual((book_dir / "book.epub").read_bytes(), epub_bytes) - - toc = json.loads((book_dir / "toc.json").read_text(encoding="utf-8")) - self.assertEqual(toc["title"], "Sample EPUB") - self.assertEqual(toc["children"][0]["title"], "Chapter One") - self.assertEqual(toc["children"][0]["href"], "OEBPS/chapter-1.xhtml") - self.assertEqual(toc["children"][0]["children"][0]["title"], "Section One") - self.assertEqual( - toc["children"][0]["children"][0]["href"], - "OEBPS/chapter-1.xhtml#section-1", - ) - self.assertEqual(toc["children"][1]["title"], "Chapter Two") - - readme = (book_dir / "README.md").read_text(encoding="utf-8") - self.assertIn("# Sample EPUB", readme) - self.assertIn("- Chapter One", readme) - - meta = json.loads((book_dir / "meta.json").read_text(encoding="utf-8")) - self.assertEqual(meta["source"], "sample.epub") - self.assertEqual(meta["source_format"], "epub") - self.assertEqual(meta["epub_md5"], expected_md5) - self.assertEqual(meta["created_at"], "2026-04-12T12:00:00Z") - self.assertEqual(meta["updated_at"], "2026-04-12T12:00:00Z") - - def test_reconvert_epub_from_stored_source_preserves_created_at(self) -> None: + convert_mock.assert_called_once() + convert_pdf_mock.assert_called_once() + + args, kwargs = convert_pdf_mock.call_args + self.assertEqual(args[1], books_dir) + self.assertEqual(kwargs["source_name"], "sample.epub") + self.assertEqual(args[0].suffix, ".pdf") + self.assertEqual(args[0].name, "sample.pdf") + self.assertGreater(len(converted_pdfs), 0) + + def test_main_reconverts_missing_epub_via_pdf_cache(self) -> None: with tempfile.TemporaryDirectory() as tmp_dir: root = Path(tmp_dir) - books_dir = root / "docs" / "books" - book_dir = books_dir / "cached-book" - book_dir.mkdir(parents=True, exist_ok=True) - - epub_bytes = _build_sample_epub(book_dir / "book.epub", title="Stored EPUB") - expected_md5 = hashlib.md5(epub_bytes).hexdigest() - (book_dir / "meta.json").write_text( - json.dumps( - { - "id": "cached-book", - "type": "book", - "source": "cached.epub", - "source_format": "epub", - "epub_md5": expected_md5, - "created_at": "2026-04-10T08:00:00Z", - "updated_at": "2026-04-10T08:00:00Z", - }, - indent=2, - ) - + "\n", - encoding="utf-8", - ) - - with patch.object(process, "_utc_now_iso", return_value="2026-04-12T12:00:00Z"): - process.reconvert_epub_from_cache("cached.epub", books_dir) - - self.assertTrue((book_dir / "book.epub").exists()) - self.assertEqual((book_dir / "book.epub").read_bytes(), epub_bytes) - - toc = json.loads((book_dir / "toc.json").read_text(encoding="utf-8")) - self.assertEqual([entry["title"] for entry in toc["children"]], ["Chapter One", "Chapter Two"]) - - meta = json.loads((book_dir / "meta.json").read_text(encoding="utf-8")) - self.assertEqual(meta["source"], "cached.epub") - self.assertEqual(meta["source_format"], "epub") - self.assertEqual(meta["epub_md5"], expected_md5) - self.assertEqual(meta["created_at"], "2026-04-10T08:00:00Z") - self.assertEqual(meta["updated_at"], "2026-04-12T12:00:00Z") + input_dir = root / "input" + output_dir = root / "docs" + books_dir = output_dir / "books" + books_dir.mkdir(parents=True, exist_ok=True) + + env = os.environ.copy() + env["INPUT_FILENAME"] = "cached.epub" + + with patch.dict(os.environ, env, clear=True): + with patch.object(process, "reconvert_from_cache") as reconvert_mock: + with patch.object(process, "build_manifest") as build_manifest_mock: + argv = sys.argv[:] + try: + sys.argv = [ + "process.py", + "--input-dir", + str(input_dir), + "--output-dir", + str(output_dir), + ] + process.main() + finally: + sys.argv = argv + + reconvert_mock.assert_called_once_with("cached.epub", books_dir) + build_manifest_mock.assert_called_once() if __name__ == "__main__":