Repository navigation
feat(corpus): Export / Import corpus buttons in the Manage tab (#102) - #103
Conversation
CLI-only export/import (#29) needed kubectl exec + kubectl cp in production. Admins now get two Manage-tab buttons backed by: - GET /api/corpus/export -> one .tar.gz of the CLI bundle layout (FileResponse streamed from scratch, cleaned after send) - POST /api/corpus/import -> multipart upload + replace flag; the archive is validated in full before anything is written (flat regular files only, manifest present, nothing unlisted), then loaded as a JobManager job shown in the ingest panel with per-stream row counts. 409 while any UI job is active; 413 over TPK_IMPORT_MAX_MB. Both are admin-only: a corpus:manage role may be scoped to a subset of entries but the bundle is the whole graph. transfer: pack_bundle/unpack_bundle + BadBundle; export blanks the dev-mode `path` of GitHub-sourced entries (meaningless on another host). JobManager: `kind` field, submit_import, has_active, finished_at in a finally. k8s: `transfer` emptyDir (4Gi) + TPK_TRANSFER_DIR in all three manifests. Docs + README config rows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s import Found in the docker-compose end-to-end run: every imported kg_ingest_log row landed on one insert timestamp, so ingest.latest_status (arg_max by _tp_time) could not tell `ok` from `started` -- an imported entry showed "started" and would flip to "stale" after the stale window, and run durations collapsed to 0. _tp_time is insertable on append-only streams (verified on timeplusd 3.2.15), so export/import now keep it for TIME_ORDERED_STREAMS (kg_ingest_log only; keyed streams upsert by PK and need no order). Old bundles without the column still load; new bundles need this version. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
End-to-end run on a local docker-compose deployment (2026-11-12)Fresh
Defect found and fixed (second commit)Right after an import the entry's Last ingest read Not a code issue, but worth knowing
|
Release validation — v0.0.8 images from Docker Hub (2026-11-12)Same end-to-end flow as above, scripted (31 checks), run against the published images, each on a fresh compose stack:
Both report Two notes, neither from this PR:
|
Closes #102.
What
Admins get Export corpus and Import corpus next to Add corpus entry in the Manage tab, backed by two new routes over the existing
tpk export/tpk importbundle (#29):GET /api/corpus/export— runsexport_bundleinto scratch, packs it as onetpk-corpus-<stamp>.tar.gz(same layout as the CLI bundle, sotpk importreads it unpacked), streams it back withFileResponse, cleans scratch after send.?format=Nativesupported.POST /api/corpus/import— multipart upload +replaceform field. Validates the whole archive before writing anything (gzipped tar, flat regular files only,manifest.jsonpresent, every file listed in the manifest and nothing else), then submits a JobManager job. The job runsimport_bundle, records per-stream row counts, and deletes the unpacked dir whatever the outcome. 409 while any UI-started job is queued/running; 413 overTPK_IMPORT_MAX_MB(default 2048).Import progress and result show in the existing ingest-history panel (
corpus import · ok · kg_nodes N · kg_edges N …, or the failure text). Import modal has a Replace existing graph checkbox with a red warning and a danger-styled submit.Decisions (from the issue)
corpus:manage— a manager role may be scoped to a subset of entries; the bundle is the whole graph.tpk ingestrun from the CLI/kubectl execis a separate process the server can't see; documented.TPK_TRANSFER_DIR([server].transfer_dir, default system temp). All three k8s manifests gain atransferemptyDir (4GisizeLimit) mounted at/opt/tpk/transfer.path(follow-up comment on Manage tab: Export / Import corpus buttons (UI + API for tpk export/import) #102) — export now emits an emptypathfor entries that also have a GitHub slug, so the target's Manage tab doesn't show a dead local path. Dev-mode path-only entries keep theirs.Tests
tests/test_transfer.py: pack/unpack round-trip; rejects path traversal, symlinks/dirs, unlisted + missing files, no manifest, non-archive — each asserting nothing was written; export path-stripping round-trip.tests/test_jobmanager.py: import job ok path (streams, counts, cleanup,has_activefalse after) and failure path (error recorded, cleanup).tests/test_api.py: export download is a valid archive matching DB counts + scratch cleaned; upload → job → ok → rows back in DB; bad/traversal archive → 400 with no job and no files; 409 during a running ingest then 200 after; 413 size cap; 401/403 for anon and a fullcorpus:managenon-admin.Full suite locally against timeplusd 3.2.15: 374 passed, 10 skipped, 1 failed —
test_ingest_repo_passes_extraction_and_backendfails identically onmain(stubfake_run_graphifylacks theon_linekwarg), unrelated to this PR.Browser check (Playwright against
tpk serve+ builtweb/dist): export downloaded a 6-member archive, import with replace ran as a job and landed in the panel asok · kg_nodes 0 · kg_edges 0 · kg_repos 4 · kg_ingest_log 0, scratch dir empty afterwards.Deploy note
Prod needs the manifest change (new emptyDir + env) with the next release, or the export/import scratch lands in the container's writable layer under
/tmp.🤖 Generated with Claude Code