Skip to content

feat(corpus): export / import only selected entries (#104) - #105

Merged
gangtao merged 1 commit into
mainfrom
feat/transfer-entries
Nov 12, 2026
Merged

gangtao merged 1 commit into
mainfrom
feat/transfer-entries

Conversation

@gangtao

@gangtao gangtao commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Closes #104.

What

Export and import can now be limited to selected corpus entries, in every surface:

Surface Export Import
Library transfer.export_bundle(entries=[...]) transfer.import_bundle(entries=[...])
CLI tpk export -o DIR -e KEY [-e KEY…] tpk import DIR -e KEY [--replace]
API GET /api/corpus/export?entries=KEY&entries=KEY repeated entries form field on POST /api/corpus/import
Manage tab Export corpus opens a modal: Whole corpus / Selected entries with a checkbox list Import corpus reads manifest.json out of the chosen archive in the browser and offers the bundle's entries as a pick list

How

  • Export filters each stream on its entry key (repo on nodes/edges/ingest_log; name@ref on kg_repos), the same identity delete_entry(purge=True) uses. The manifest always records entries and partial; bundle_version stays 1, the fields are additive.
  • Import keeps the engine-side serialization: the bundle is loaded into staging streams under a throwaway prefix (db.ensure_graph_streams / drop_graph_streams, split out of ensure_schema), the staged rows are waited on until readable, then copied with INSERT … SELECT … WHERE has(keys, repo). Targets are untouched until every stream has staged, so a failed stage leaves the graph as it was. The returned manifest reports rows actually copied, which is what the job record and the CLI print.
  • Replace with a selection means replace those entries: purge their registry row, nodes and edges first, nothing else. Whole-bundle replace keeps the drop-and-recreate semantics and its red warning; the UI labels the two differently.
  • Browser-side manifest peek (web/src/bundle.ts): pack_bundle writes manifest.json as the first tar member, so DecompressionStream plus a 512-byte tar-header parse reaches it without inflating the rest. No second upload, no staged uploads to garbage-collect. If the peek fails (older bundle, older browser) the modal says so and imports whole.
  • Unknown keys: 422 on both API routes, BadParameter on the CLI. Import can only check against bundles that list entries.
  • Still admin-only.

Tests

  • tests/test_transfer.py: full-export manifest lists entries; filtered export filters all four streams and round-trips; empty selection; partial import loads only the selection (and preserves ingest-log order); partial replace purges only the selected entry; unknown key loads nothing; staging streams are dropped.
  • tests/test_api.py: export with entries (partial filename, manifest, 422 unknown); import with entries (422 before any job for a key the bundle lacks, job record carries entries, other entries untouched, scratch cleaned).
  • tests/test_cli.py: -e on both commands incl. rejections. tests/test_jobmanager.py: entries passed through and recorded.
  • Full suite on timeplusd 3.2.15: 384 passed, 10 skipped, 2 failed — both pre-existing on main (test_ingest_repo_passes_extraction_and_backend, test_main_reports_a_connect_failure_in_one_actionable_line). The new transfer tests also pass against the OSS proton backend (all-in-one image, TPK_DB_BACKEND=proton), tombstones included.
  • Browser pass on a local tpk serve with seven entries: exported alpha@v1 + gamma@main (download tpk-corpus-…-partial.tar.gz, manifest partial: true, 6 nodes); deleted alpha@v1 with purge; Import modal showed "Bundle (partial export) … 2 entries, 6 nodes", picked alpha@v1 with Replace the selected entries; job finished ok · kg_nodes 4 … · alpha@v1, alpha@v1 back with 4 nodes and status ok, beta/gamma untouched, scratch dir empty, no staging streams left.

🤖 Generated with Claude Code

Both transfer directions take an entry filter, end to end:

- transfer.export_bundle(entries=...) filters every stream on its entry
  key (repo column; name@ref on kg_repos); the manifest now records
  `entries` and `partial`. transfer.import_bundle(entries=...) stages the
  bundle under a throwaway stream prefix and copies the selected rows
  engine-side (INSERT ... SELECT WHERE has(keys, repo)), so the target's
  other entries are never touched; with entries, replace means purge
  just those entries first. db.ensure_graph_streams / drop_graph_streams
  split out of ensure_schema for the staging.
- CLI: tpk export/import --entry/-e (repeatable), unknown keys rejected.
- API: GET /api/corpus/export?entries=..., repeated `entries` form field
  on POST /api/corpus/import (422 for keys the bundle lacks); import jobs
  carry `entries` and report rows actually loaded.
- Manage tab: Export corpus opens a modal (whole / selected entries);
  the Import modal reads manifest.json out of the chosen .tar.gz in the
  browser (first tar member; DecompressionStream) and offers the
  bundle's entries as a pick list, with "Replace the selected entries"
  wording instead of the whole-graph warning when a selection is active.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@gangtao

gangtao commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

End-to-end run on docker compose (branch image built from 3b20270)

Stack: docker compose build app && docker compose up -d (timeplusd 3.2.15 backend). Driver: the #102 e2e script extended with a #104 section. 59 of 59 checks pass (one assertion in the driver initially expected the manifest to list only the two e2e entries; the compose stack also seeds the four repos.toml entries, so the manifest correctly listed six, and the check was corrected to containment and re-run).

The #104 section, after ingesting proton-python-driver at develop (A) and v0.3.0 (B), 2694 nodes each:

Check Result
full export manifest lists A and B, partial: false pass
GET /api/corpus/export?entries=B → 200, …-partial.tar.gz, manifest entries: [B], nodes 2694, kg_repos rows 1 pass
export unknown key → 422; import with a key the bundle lacks → 422, no job created pass
tpk export -e B prints entries (partial): B; unknown key rejected pass
purge B; import B from an A-only bundle → 422 pass
POST /api/corpus/import with entries=B, replace=true → job ok, job entries: [B], loaded rows 2694 pass
B restored: node_count 2694, last_status ok; A untouched (2694) pass
no imp* staging streams left in timeplusd; transfer scratch dir empty pass
tpk import -e ghost rejected; tpk import -e B --replace loads; B node_count 2694 pass
chat afterwards answers from the graph, citing connection.py pass

All 31 #102 checks (export shape, CLI interop, admin-only, bad archives, 409 during reindex, purge + whole replace-import, explore parity) still pass on this branch.

@gangtao
gangtao merged commit 4e21cc9 into main Nov 12, 2026
2 checks passed
@gangtao
gangtao deleted the feat/transfer-entries branch November 12, 2026 21:34
@gangtao

gangtao commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Release validation: v0.0.9 images

Same 59-check driver as the branch run above, against the published images (workflow run 43311685059, both tags multi-arch on Docker Hub).

Image Backend /healthz Checks
timeplus/tpk-app:0.0.9 on the compose timeplusd timeplusd 3.2.15 0.0.9 / 4e21cc9 59 / 59 pass
timeplus/tpk:0.0.9 all-in-one bundled proton 0.0.9 / 4e21cc9 59 / 59 pass

Both runs: ingest, export shape, CLI interop, admin-only enforcement, bad-archive rejection, 409 during reindex, purge + whole replace-import with identical explore results, then the #104 section (second entry, full/partial manifests, 422 paths, tpk export -e / tpk import -e, purge + partial replace-import leaving the other entry untouched, no staging streams left, scratch dir empty), and a chat answer citing connection.py from the imported graph.

One driver change for the all-in-one run: the "no staging streams left" check now queries the DB over HTTP instead of the timeplusd client binary, which the proton-based image does not ship.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Export / import only selected corpus entries

1 participant