Skip to content

feat(corpus): Export / Import corpus buttons in the Manage tab (#102) - #103

Merged
gangtao merged 2 commits into
mainfrom
feat/corpus-export-import-ui
Nov 12, 2026
Merged

gangtao merged 2 commits into
mainfrom
feat/corpus-export-import-ui

Conversation

@gangtao

@gangtao gangtao commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Closes #102.

What

Admins get Export corpus and Import corpus next to Add corpus entry in the Manage tab, backed by two new routes over the existing tpk export / tpk import bundle (#29):

  • GET /api/corpus/export — runs export_bundle into scratch, packs it as one tpk-corpus-<stamp>.tar.gz (same layout as the CLI bundle, so tpk import reads it unpacked), streams it back with FileResponse, cleans scratch after send. ?format=Native supported.
  • POST /api/corpus/import — multipart upload + replace form field. Validates the whole archive before writing anything (gzipped tar, flat regular files only, manifest.json present, every file listed in the manifest and nothing else), then submits a JobManager job. The job runs import_bundle, records per-stream row counts, and deletes the unpacked dir whatever the outcome. 409 while any UI-started job is queued/running; 413 over TPK_IMPORT_MAX_MB (default 2048).

Import progress and result show in the existing ingest-history panel (corpus import · ok · kg_nodes N · kg_edges N …, or the failure text). Import modal has a Replace existing graph checkbox with a red warning and a danger-styled submit.

Decisions (from the issue)

  • Admin-only, not corpus:manage — a manager role may be scoped to a subset of entries; the bundle is the whole graph.
  • Concurrency — the API refuses an import while any UI job is active, and the single worker serialises what does get queued. A tpk ingest run from the CLI/kubectl exec is a separate process the server can't see; documented.
  • Disk — TPK_TRANSFER_DIR ([server].transfer_dir, default system temp). All three k8s manifests gain a transfer emptyDir (4Gi sizeLimit) mounted at /opt/tpk/transfer.
  • Stale path (follow-up comment on Manage tab: Export / Import corpus buttons (UI + API for tpk export/import) #102) — export now emits an empty path for entries that also have a GitHub slug, so the target's Manage tab doesn't show a dead local path. Dev-mode path-only entries keep theirs.

Tests

  • tests/test_transfer.py: pack/unpack round-trip; rejects path traversal, symlinks/dirs, unlisted + missing files, no manifest, non-archive — each asserting nothing was written; export path-stripping round-trip.
  • tests/test_jobmanager.py: import job ok path (streams, counts, cleanup, has_active false after) and failure path (error recorded, cleanup).
  • tests/test_api.py: export download is a valid archive matching DB counts + scratch cleaned; upload → job → ok → rows back in DB; bad/traversal archive → 400 with no job and no files; 409 during a running ingest then 200 after; 413 size cap; 401/403 for anon and a full corpus:manage non-admin.

Full suite locally against timeplusd 3.2.15: 374 passed, 10 skipped, 1 failed — test_ingest_repo_passes_extraction_and_backend fails identically on main (stub fake_run_graphify lacks the on_line kwarg), unrelated to this PR.

Browser check (Playwright against tpk serve + built web/dist): export downloaded a 6-member archive, import with replace ran as a job and landed in the panel as ok · kg_nodes 0 · kg_edges 0 · kg_repos 4 · kg_ingest_log 0, scratch dir empty afterwards.

Deploy note

Prod needs the manifest change (new emptyDir + env) with the next release, or the export/import scratch lands in the container's writable layer under /tmp.

🤖 Generated with Claude Code

gangtao and others added 2 commits November 11, 2026 13:13
CLI-only export/import (#29) needed kubectl exec + kubectl cp in
production. Admins now get two Manage-tab buttons backed by:

- GET  /api/corpus/export  -> one .tar.gz of the CLI bundle layout
  (FileResponse streamed from scratch, cleaned after send)
- POST /api/corpus/import  -> multipart upload + replace flag; the
  archive is validated in full before anything is written (flat regular
  files only, manifest present, nothing unlisted), then loaded as a
  JobManager job shown in the ingest panel with per-stream row counts.
  409 while any UI job is active; 413 over TPK_IMPORT_MAX_MB.

Both are admin-only: a corpus:manage role may be scoped to a subset of
entries but the bundle is the whole graph.

transfer: pack_bundle/unpack_bundle + BadBundle; export blanks the
dev-mode `path` of GitHub-sourced entries (meaningless on another host).
JobManager: `kind` field, submit_import, has_active, finished_at in a
finally. k8s: `transfer` emptyDir (4Gi) + TPK_TRANSFER_DIR in all three
manifests. Docs + README config rows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s import

Found in the docker-compose end-to-end run: every imported kg_ingest_log
row landed on one insert timestamp, so ingest.latest_status (arg_max by
_tp_time) could not tell `ok` from `started` -- an imported entry showed
"started" and would flip to "stale" after the stale window, and run
durations collapsed to 0.

_tp_time is insertable on append-only streams (verified on timeplusd
3.2.15), so export/import now keep it for TIME_ORDERED_STREAMS
(kg_ingest_log only; keyed streams upsert by PK and need no order). Old
bundles without the column still load; new bundles need this version.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@gangtao

gangtao commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

End-to-end run on a local docker-compose deployment (2026-11-12)

Fresh docker compose down -v && up -d on the branch image (timeplusd 3.2.15, app-only container). Everything driven through the public API and the built web UI (Playwright), not through test fixtures.

Step Result
Add timeplus-io/proton-python-driver@develop (code-only) + ingest via UI job ok · 2694 nodes · 7175 edges · 10 s
Export corpus via API 200, application/gzip, 209 KB, 5 members; manifest rows == live counts; container scratch empty afterwards
tpk import (CLI, inside container) of the UI-exported archive loads unchanged — bundle layout is identical
tpk export (CLI) → packed → later uploaded through the UI loads — CLI→UI direction works too
Non-admin with full corpus:manage 403 admin required on both routes; anonymous 401; the same user still gets 200 on /api/repos
Garbage file / path-traversal archive upload 400 with the specific reason, no job created, no file written
Upload while a reindex is running 409; 200 once it finishes
Delete entry with purge → graph empty (DB counts 0, explore search 0 results) wiped
Import corpus in the Manage tab (file picker, Replace existing graph, danger submit) job row corpus import · ok · kg_nodes 2694 · kg_edges 7175 · kg_repos 5 · kg_ingest_log … in the panel
After import /api/repos 2694 nodes; explore search for connect returns the identical 5 results as before the wipe; DB counts match
Chat over the imported graph agent used search_entities ×2 + read_source and answered with a citation to proton_driver/connection.py:311

Defect found and fixed (second commit)

Right after an import the entry's Last ingest read started instead of ok. The bundle didn't carry _tp_time for the append-only kg_ingest_log, so every imported row got one insert timestamp and latest_status's arg_max-by-time became a coin flip (and run durations collapsed to 0; after the stale window the UI would have shown stale). _tp_time is insertable on append-only streams (verified on 3.2.15), so the bundle now keeps it for kg_ingest_log. Re-ran the wipe → replace-import cycle on the rebuilt image: status ok, original timestamp and 10.8 s duration preserved. Regression test added (test_ingest_log_order_survives_import). Compatibility: old bundles still load; a bundle from this version needs this version or newer.

Not a code issue, but worth knowing

  • The tests/test_transfer.py / test_api.py DB helpers address streams without a database name, so they only run with TIMEPLUS_DATABASE=default (fine on the compose server as the tpk user).
  • The chat check needed a working LLM key: the compose .env Anthropic gateway key (Sept) now returns 401. I ran the app container with the key from the k8s secret for that one check; .env itself is unchanged.

@gangtao
gangtao merged commit f79fe87 into main Nov 12, 2026
2 checks passed
@gangtao

gangtao commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Release validation — v0.0.8 images from Docker Hub (2026-11-12)

Same end-to-end flow as above, scripted (31 checks), run against the published images, each on a fresh compose stack:

Image Backend Result
timeplus/tpk-app:0.0.8 (docker-compose.yml) timeplusd 3.2.15 31/31 pass
timeplus/tpk:0.0.8 all-in-one (docker-compose.allinone.yml) bundled proton 31/31 pass

Both report {"version":"0.0.8","commit":"f79fe87"} on /healthz. Checks cover: UI ingest → export (archive members, manifest rows == live counts, _tp_time carried for the log, scratch cleaned) → tpk import of the UI archive and tpk export → upload of a CLI archive → admin-only (403 for a full corpus:manage role, 401 anonymous) → garbage / path-traversal archives rejected with no job and no file → 409 while a reindex runs → purge to empty → replace-import restores identical node/edge counts, ok status, and identical explore results → chat over the imported graph cites proton_driver/connection.py. Browser pass on both images: Export downloads the archive, Import modal renders, import jobs show in the panel.

Two notes, neither from this PR:

  • On the proton backend a fresh login session takes ~2 s to become readable, so a script that calls the API immediately after /auth/login gets 401 (the web UI is unaffected). Worth a small retry in get_session or a note in the API docs.
  • tests/test_ingest.py::test_ingest_repo_passes_extraction_and_backend is still broken on main (stub lacks the on_line kwarg).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Manage tab: Export / Import corpus buttons (UI + API for tpk export/import)

1 participant