Repository navigation
feat(corpus): export / import only selected entries (#104) - #105
Conversation
Both transfer directions take an entry filter, end to end: - transfer.export_bundle(entries=...) filters every stream on its entry key (repo column; name@ref on kg_repos); the manifest now records `entries` and `partial`. transfer.import_bundle(entries=...) stages the bundle under a throwaway stream prefix and copies the selected rows engine-side (INSERT ... SELECT WHERE has(keys, repo)), so the target's other entries are never touched; with entries, replace means purge just those entries first. db.ensure_graph_streams / drop_graph_streams split out of ensure_schema for the staging. - CLI: tpk export/import --entry/-e (repeatable), unknown keys rejected. - API: GET /api/corpus/export?entries=..., repeated `entries` form field on POST /api/corpus/import (422 for keys the bundle lacks); import jobs carry `entries` and report rows actually loaded. - Manage tab: Export corpus opens a modal (whole / selected entries); the Import modal reads manifest.json out of the chosen .tar.gz in the browser (first tar member; DecompressionStream) and offers the bundle's entries as a pick list, with "Replace the selected entries" wording instead of the whole-graph warning when a selection is active. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
End-to-end run on docker compose (branch image built from 3b20270)Stack: The #104 section, after ingesting
All 31 #102 checks (export shape, CLI interop, admin-only, bad archives, 409 during reindex, purge + whole replace-import, explore parity) still pass on this branch. |
Release validation: v0.0.9 imagesSame 59-check driver as the branch run above, against the published images (workflow run 43311685059, both tags multi-arch on Docker Hub).
Both runs: ingest, export shape, CLI interop, admin-only enforcement, bad-archive rejection, 409 during reindex, purge + whole replace-import with identical explore results, then the #104 section (second entry, full/partial manifests, 422 paths, One driver change for the all-in-one run: the "no staging streams left" check now queries the DB over HTTP instead of the |
Closes #104.
What
Export and import can now be limited to selected corpus entries, in every surface:
transfer.export_bundle(entries=[...])transfer.import_bundle(entries=[...])tpk export -o DIR -e KEY [-e KEY…]tpk import DIR -e KEY [--replace]GET /api/corpus/export?entries=KEY&entries=KEYentriesform field onPOST /api/corpus/importmanifest.jsonout of the chosen archive in the browser and offers the bundle's entries as a pick listHow
repoon nodes/edges/ingest_log;name@refonkg_repos), the same identitydelete_entry(purge=True)uses. The manifest always recordsentriesandpartial;bundle_versionstays 1, the fields are additive.db.ensure_graph_streams/drop_graph_streams, split out ofensure_schema), the staged rows are waited on until readable, then copied withINSERT … SELECT … WHERE has(keys, repo). Targets are untouched until every stream has staged, so a failed stage leaves the graph as it was. The returned manifest reports rows actually copied, which is what the job record and the CLI print.web/src/bundle.ts):pack_bundlewritesmanifest.jsonas the first tar member, soDecompressionStreamplus a 512-byte tar-header parse reaches it without inflating the rest. No second upload, no staged uploads to garbage-collect. If the peek fails (older bundle, older browser) the modal says so and imports whole.BadParameteron the CLI. Import can only check against bundles that listentries.Tests
tests/test_transfer.py: full-export manifest lists entries; filtered export filters all four streams and round-trips; empty selection; partial import loads only the selection (and preserves ingest-log order); partial replace purges only the selected entry; unknown key loads nothing; staging streams are dropped.tests/test_api.py: export withentries(partial filename, manifest, 422 unknown); import withentries(422 before any job for a key the bundle lacks, job record carriesentries, other entries untouched, scratch cleaned).tests/test_cli.py:-eon both commands incl. rejections.tests/test_jobmanager.py: entries passed through and recorded.test_ingest_repo_passes_extraction_and_backend,test_main_reports_a_connect_failure_in_one_actionable_line). The new transfer tests also pass against the OSS proton backend (all-in-one image,TPK_DB_BACKEND=proton), tombstones included.tpk servewith seven entries: exportedalpha@v1+gamma@main(downloadtpk-corpus-…-partial.tar.gz, manifestpartial: true, 6 nodes); deletedalpha@v1with purge; Import modal showed "Bundle (partial export) … 2 entries, 6 nodes", pickedalpha@v1with Replace the selected entries; job finishedok · kg_nodes 4 … · alpha@v1,alpha@v1back with 4 nodes and status ok,beta/gammauntouched, scratch dir empty, no staging streams left.🤖 Generated with Claude Code