Skip to content

feat: reuse a source across notebooks from a library - #217

Merged
mrsibe merged 5 commits into
mainfrom
feat/library-source-reuse
Oct 2, 2026
Merged

mrsibe merged 5 commits into
mainfrom
feat/library-source-reuse

Conversation

@mrsibe

@mrsibe mrsibe commented Oct 2, 2026

Copy link
Copy Markdown
Owner

What does this PR do?

A source is now a library-level entity and a notebook holds a membership of it, so one imported paper can be used by two notebooks without being stored or embedded twice.

  • library_sources holds the durable snapshot: canonical text, parse structure, original URI and the library-owned copy of the file.
  • documents stays the per-notebook membership. Existing rows keep their ids, so every recorded citation, excerpt and retrieval scope keeps resolving.
  • "From library" in the source panel adds a snapshot that another notebook already has, showing whether its index can be reused.
  • Removing a source removes that notebook's membership and index only. The library snapshot and its file stay, and are only deleted by an explicit, confirmed action on a snapshot no notebook uses.

Why?

Sources belonged to exactly one notebook, so reading the same paper in two contexts meant importing it twice, storing it twice and embedding it twice — with the two copies drifting as soon as one was re-indexed. This is the storage and indexing half of #99 (the part that had to be decided before any UI existed); the design is recorded in docs/adr/0002-library-source-reuse.md.

Two answers the issue asked for, in writing:

  • Is a source embedded once per notebook, or once globally? Once per notebook. A notebook owns its vector space, so chunks, provenance and vectors stay per membership; a reuse copies a compatible donor's index instead of re-embedding.
  • What happens when A uses e5-small and B something else? The membership lands pending and says indexing is required. Nothing is embedded automatically, and no existing vectors are cleared.

Related issue

Fixes #99

What changed?

  • Schema + migration (0023_sleepy_luckman): new library_sources table, nullable documents.source_id, and a (notebook_id, source_id) unique index. The migration backfills one snapshot per existing document, one-to-one — unrelated sources are deliberately not merged by title or path.
  • src/main/services/librarySources.ts (new, Electron-free): list, attach and permanent-delete, plus the synchronous clone of blocks, chunks, chunk↔block mappings, embedding metadata, vectors and full-text rows. resolveReuse() is the single reuse decision, shared by the listing and the attach, so what the picker promises is what attaching does.
  • KnowledgeService: snapshots are created after parsing/content is known and before indexing; file refresh is copy-on-write so a shared snapshot's text and page offsets can never move under another notebook's citations; deleteDocument is now detach-only.
  • deleteNotebook: no longer unlinks files, which a shared snapshot may still own.
  • IPC/preload/UI: knowledge:list-library-sources, knowledge:attach-library-source, knowledge:delete-library-source; a "From library" picker that distinguishes ready-to-reuse from needs-indexing; deletion copy changed to "Remove from notebook"; EN and zh-CN strings. The source-add menu now only disables the import actions that need an embedding model, not the whole menu.
  • Folder watching: a reused membership is snapshot-only and is not enrolled in mutable-file reconciliation, so an unchanged shared file is never re-embedded by a watcher.
  • Docs: the ADR, plus the source-provenance diagram in docs/architecture.md.

How was this tested?

  • npm run typecheck, npm test (555 tests), npm run build, npm run check:design, npm run check:version.
  • npm run build:unpack + npm run smoke:packaged — 31 checks pass, including two new packaged-service checks that drive the real KnowledgeService: import into notebook A, attach to B with an embedding backend that throws if called, retrieve from B, compare page/span provenance, read the reused source through knownote-doc://, then refresh A (copy-on-write, A's shared file unchanged and B's text unchanged), remove A's membership (B still retrieves), delete notebook A (file intact), and confirm that permanent deletion is refused while attached and without confirmation.
  • New regression tests: test/librarySourceReuse.test.ts (real SQLite + sqlite-vec: clone with preserved provenance, empty-target space adoption, space mismatch → pending, incompatible target vectors untouched, idempotent attach, incomplete donor coverage → pending, half-refreshed donor not reused, clone writes FTS rows, unparsed snapshot visible but unusable, listing and attaching agree), test/librarySourceBackfill.test.ts (runs the real migration statements, including idempotency), test/librarySourceGuards.test.ts (source-level direction checks, now parsed with the TypeScript AST so they inspect method bodies rather than an object return type), test/librarySourceWatch.test.ts (watcher ignores snapshot-only memberships), test/librarySourceUi.test.ts (IPC validation + picker/locale contracts).
  • Packaged-renderer check via the DevTools protocol: window.api.knowledge.{listLibrarySources,attachLibrarySource,deleteLibrarySource,deleteDocument} are all present in the packaged preload, the renderer mounts, and the packaged artifact contains both locales' new strings. This environment has no GUI screenshot tooling (headless WSL, no scrot/xwd/import), so the screenshots below are omitted rather than fabricated; the UI itself was verified by typecheck, ESLint, the build and the packaged-artifact checks above.

Screenshots / recordings

Not available in this environment (no screenshot tooling); see the verification note above.

Checklist

  • I have reviewed my own changes.
  • npm run typecheck passes.
  • npm run build passes.
  • I have tested the affected user workflow.
  • I have not included unrelated changes.
  • I have updated documentation when necessary.

Desktop / build changes

  • Not applicable
  • npm run build:unpack passes.
  • npm run smoke:packaged passes.

Known limitations

  • A notebook on a remote embedding connection can reuse a matching donor's vectors only when the exact persisted space identity matches; otherwise the membership is pending and needs an explicit re-index.
  • A membership's derived index is still replaced non-atomically (the pre-existing #95 limitation): a crash during replacement, not a failure before it, can leave a membership without a complete index.
  • Files left behind by a crash between copying a file and writing its snapshot row are not reachable by the delete-unused path.

@github-actions github-actions Bot added the enhancement New feature or request label Oct 2, 2026
@mrsibe
mrsibe merged commit afbd1a5 into main Oct 2, 2026
4 checks passed
@mrsibe
mrsibe deleted the feat/library-source-reuse branch October 2, 2026 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feat] Cross-notebook source reuse (library-level sources)

1 participant