Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -548,6 +548,10 @@ serialization (the engine encodes/decodes; no extra dependency):
tpk import corpus-dump/ # load into this environment (upsert)
tpk import corpus-dump/ --replace # reset the target streams first (drop stale rows)

tpk export --out one/ -e proton@v1.6.0 -e docs@main # only these entries (#104)
tpk import corpus-dump/ -e docs@main # only this entry out of a bundle
tpk import corpus-dump/ -e docs@main --replace # ...purging its existing rows first

The bundle covers the expensive/portable streams — `kg_nodes`, `kg_edges`,
`kg_repos`, `kg_ingest_log` — plus a `manifest.json` (format, per-stream
columns + row counts, `created_at`). **Auth streams
Expand All @@ -569,6 +573,16 @@ all exist in the target deployment is rejected up front (schema drift between
versions), before anything is written. Bundle files stream to/from disk, so a
300k-node graph never lands wholly in memory.

`--entry KEY` / `-e KEY` (repeatable, #104) limits either direction to the
named corpus entries (`name@ref`, or bare `name` for a local-path entry).
The manifest always lists the entries a bundle holds (`entries`) and whether
it was filtered (`partial`). A partial import stages the bundle in throwaway
streams and copies only the matching rows, so the target's other entries are
never touched; with `--entry`, `--replace` purges just those entries'
registry row, nodes and edges before loading (the append-only ingest log keeps
its history). Unknown keys are rejected before anything is written, on both
sides.

Streams under `TPK_STREAM_PREFIX` are exported/imported when that env var is
set, matching the rest of the CLI.

Expand Down
1 change: 1 addition & 0 deletions docs/FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ Manage what the graph indexes at runtime, without editing files or restarting.
- **Release-upgrade workflow:** add the new ref → ingest → verify → flip enabled; a legacy migration re-keys old bare-name rows on first ingest.
- **Export / import (issue #29).** `tpk export` dumps the ingested graph + corpus registry to a portable bundle (proton `FORMAT Parquet`, one file per stream + manifest); `tpk import` loads it into a new environment — no re-ingest, no LLM calls. Idempotent upsert by default (`--replace` to reset); auth streams excluded. The bundle carries each entry's registry row (source, ref, visibility, extraction mode, description) and the ingest history; a dev-mode local `path` is blanked for GitHub-sourced entries, since it is meaningless on another host. The append-only ingest log is exported with its `_tp_time`, so run order (and therefore each entry's `ok`/`started`/`stale` status and run durations) survives an import; a bundle written by an older `tpk` still loads, but one written by this version needs this version or newer to import.
- **Export / import from the Manage tab (issue #102).** Admins get **Export corpus** (one `.tar.gz` download of the same bundle, `GET /api/corpus/export`) and **Import corpus** (upload the archive, optional *replace*; `POST /api/corpus/import`). The archive is validated in full before anything is written — flat regular files only, a manifest, nothing the manifest doesn't list — then loaded as a job shown in the ingest panel with per-stream row counts. An import is refused (409) while any UI-started ingest or import job is active, so it never drops a stream such a job is writing into (a `tpk ingest` run from the CLI or `kubectl exec` is a separate process the server cannot see — don't import during one). Admin-only rather than `corpus:manage`, because a manager role may be scoped to a subset of entries and the bundle is the whole graph. Scratch space: `TPK_TRANSFER_DIR` (an `emptyDir` in the k8s manifests); upload cap `TPK_IMPORT_MAX_MB` (default 2048).
- **Export / import only selected entries (issue #104).** Both directions take an entry filter, end to end: `tpk export -e KEY` / `tpk import -e KEY` (repeatable), `GET /api/corpus/export?entries=KEY&…`, a repeated `entries` form field on `POST /api/corpus/import`, and pickers in the Manage tab. **Export corpus** now opens a modal (*Whole corpus* / *Selected entries*); the **Import corpus** modal reads `manifest.json` out of the chosen `.tar.gz` in the browser (it is the first tar member; `DecompressionStream` plus a tar-header parse, no second upload) and offers the bundle's entries as a pick list, falling back to whole-bundle import for bundles or browsers without one. The manifest carries `entries` and `partial`. A partial import stages the bundle under a throwaway stream prefix and copies the selected rows engine-side (`INSERT … SELECT … WHERE has(keys, repo)`), so the other entries in the target are never touched; the targets are not written until every staged stream is readable. With a selection, *replace* means **replace those entries** (purge their registry row, nodes and edges first, as delete-with-purge does; ingest history is append-only and kept), and the UI labels it that way without the red whole-graph warning. Unknown keys get a 422 (import can only check them against bundles that list `entries`; older bundles import whole). Still admin-only.

## 7. Authentication & role-based access (issue #8)

Expand Down
54 changes: 42 additions & 12 deletions src/tpk/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -208,10 +208,14 @@ def _submit(self, kind: str, key: str, payload) -> str:
def submit(self, cfg: RepoConfig) -> str:
return self._submit("ingest", entry_key(cfg), cfg)

def submit_import(self, bundle_dir, replace: bool = False) -> str:
def submit_import(self, bundle_dir, replace: bool = False, entries=None) -> str:
"""Load an already-unpacked bundle directory; the worker deletes the
directory when done, whatever the outcome."""
return self._submit("import", IMPORT_ENTRY_KEY, (Path(bundle_dir), replace))
directory when done, whatever the outcome. `entries` (#104) restricts
the load to those entry keys."""
job_id = self._submit("import", IMPORT_ENTRY_KEY, (Path(bundle_dir), replace, entries))
with self._jobs_lock:
self.jobs[job_id]["entries"] = list(entries) if entries is not None else None
return job_id

def has_active(self) -> bool:
with self._jobs_lock:
Expand All @@ -225,15 +229,19 @@ def get(self, job_id: str) -> dict | None:
with self._jobs_lock:
return self.jobs.get(job_id)

def _run_import(self, rec: dict, bundle_dir: Path, replace: bool):
def _run_import(self, rec: dict, bundle_dir: Path, replace: bool, entries=None):
try:
with self._jobs_lock:
rec["phase"] = "import"
rec["message"] = "replacing graph streams" if replace else "loading bundle"
if entries is not None:
rec["message"] = (f"{'replacing' if replace else 'loading'} "
f"{len(entries)} selected entr{'y' if len(entries) == 1 else 'ies'}")
else:
rec["message"] = "replacing graph streams" if replace else "loading bundle"
rec["updated_at"] = datetime.now(timezone.utc).isoformat()
client = db.get_client(Settings.from_env())
manifest = transfer.import_bundle(client, bundle_dir, prefix=self.prefix,
replace=replace)
replace=replace, entries=entries)
streams = {name: info["rows"] for name, info in manifest["streams"].items()}
with self._jobs_lock:
rec["streams"] = streams
Expand Down Expand Up @@ -449,18 +457,29 @@ def api_ingest_log(limit: int = Query(50, ge=1, le=200), entry: str | None = Non
# entry included) and an import rewrites all of it.

@router.get("/corpus/export")
def export_corpus(format: str = Query("Parquet"), actor: User = Depends(auth.require_admin)):
def export_corpus(format: str = Query("Parquet"),
entries: list[str] = Query(default=[]),
actor: User = Depends(auth.require_admin)):
"""Dump the graph + corpus registry and stream it back as one
`.tar.gz` -- the CLI bundle layout, so `tpk import` reads it unpacked."""
`.tar.gz` -- the CLI bundle layout, so `tpk import` reads it unpacked.
`entries` (repeatable, #104) limits the bundle to those entry keys."""
if format not in transfer.FORMATS:
raise HTTPException(422, "format must be Parquet|Native")
client = _client()
selected = sorted(set(entries)) if entries else None
if selected:
unknown = sorted(set(selected) - set(transfer.list_entry_keys(client, prefix=prefix)))
if unknown:
raise HTTPException(422, f"unknown corpus entries: {', '.join(unknown)}")
root = transfer_dir()
root.mkdir(parents=True, exist_ok=True)
work = Path(tempfile.mkdtemp(prefix="tpk-export-", dir=root))
try:
transfer.export_bundle(_client(), work / "bundle", prefix=prefix, fmt=format)
transfer.export_bundle(client, work / "bundle", prefix=prefix, fmt=format,
entries=selected)
stamp = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
name = f"tpk-corpus-{stamp}{transfer.ARCHIVE_SUFFIX}"
scope = "-partial" if selected is not None else ""
name = f"tpk-corpus-{stamp}{scope}{transfer.ARCHIVE_SUFFIX}"
archive = transfer.pack_bundle(work / "bundle", work / name)
shutil.rmtree(work / "bundle", ignore_errors=True)
except Exception:
Expand All @@ -473,10 +492,14 @@ def export_corpus(format: str = Query("Parquet"), actor: User = Depends(auth.req

@router.post("/corpus/import")
def import_corpus(file: UploadFile = File(...), replace: bool = Form(False),
entries: list[str] = Form(default=[]),
actor: User = Depends(auth.require_admin)):
"""Upload a bundle archive and load it as a background job (poll
`/api/jobs/{id}`). The archive is validated in full before anything
is written; a load is refused while any job is queued or running."""
is written; a load is refused while any job is queued or running.
`entries` (repeatable form field, #104) loads only those entry keys;
with `replace`, only those entries' existing rows are purged first."""
selected = sorted(set(entries)) if entries else None
if jobs.has_active():
raise HTTPException(409, "an ingest or import job is already running; retry when it finishes")
root = transfer_dir()
Expand All @@ -499,6 +522,11 @@ def import_corpus(file: UploadFile = File(...), replace: bool = Form(False),
except transfer.BadBundle as exc:
raise HTTPException(400, f"invalid bundle: {exc}")
archive.unlink()
listed = manifest.get("entries")
if selected and listed is not None:
unknown = sorted(set(selected) - set(listed))
if unknown:
raise HTTPException(422, f"bundle does not contain: {', '.join(unknown)}")
except Exception:
shutil.rmtree(work, ignore_errors=True)
raise
Expand All @@ -508,9 +536,11 @@ def import_corpus(file: UploadFile = File(...), replace: bool = Form(False),
if jobs.has_active():
shutil.rmtree(work, ignore_errors=True)
raise HTTPException(409, "an ingest or import job is already running; retry when it finishes")
job_id = jobs.submit_import(work, replace=replace)
job_id = jobs.submit_import(work, replace=replace, entries=selected)
return {"job_id": job_id,
"streams": {k: v["rows"] for k, v in manifest["streams"].items()},
"entries": selected, "bundle_entries": manifest.get("entries"),
"partial": manifest.get("partial", False),
"format": manifest.get("format"), "created_at": manifest.get("created_at")}

@router.get("/jobs")
Expand Down
38 changes: 34 additions & 4 deletions src/tpk/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -149,33 +149,63 @@ def status():
def export(
out: Path = typer.Option(..., "--out", "-o", help="Output bundle directory"),
fmt: str = typer.Option("Parquet", "--format", help="Bundle format: Parquet or Native"),
entry: list[str] = typer.Option(
None, "--entry", "-e",
help="Only this corpus entry (name@ref, or name for a local-path entry); repeatable. "
"Default: the whole corpus.",
),
):
"""Export the ingested graph + corpus registry to a portable bundle."""
from tpk import transfer

client = db.get_client(Settings.from_env())
prefix = setting("TPK_STREAM_PREFIX", "db", "stream_prefix", "")
manifest = transfer.export_bundle(client, out, prefix=prefix, fmt=fmt)
entries = list(entry) if entry else None
if entries:
known = set(transfer.list_entry_keys(client, prefix=prefix))
unknown = sorted(set(entries) - known)
if unknown:
raise typer.BadParameter(f"unknown corpus entries: {', '.join(unknown)}")
manifest = transfer.export_bundle(client, out, prefix=prefix, fmt=fmt, entries=entries)
for stream, info in manifest["streams"].items():
typer.echo(f"{stream:16s} {info['rows']:>9} rows -> {info['file']}")
scope = "partial" if manifest.get("partial") else "whole corpus"
typer.echo(f"entries ({scope}): {', '.join(manifest['entries']) or '(none)'}")
typer.echo(f"bundle written to {out}")


@app.command(name="import")
def import_bundle(
src: Path = typer.Argument(..., help="Bundle directory to load"),
replace: bool = typer.Option(
False, "--replace", help="Reset target streams before loading (else upsert)"
False, "--replace",
help="Reset target streams before loading (else upsert). With --entry: purge just "
"those entries' existing rows first.",
),
entry: list[str] = typer.Option(
None, "--entry", "-e",
help="Only load this entry from the bundle (name@ref); repeatable. Default: the whole bundle.",
),
):
"""Load a corpus bundle into this environment — no re-ingest."""
from tpk import transfer

client = db.get_client(Settings.from_env())
prefix = setting("TPK_STREAM_PREFIX", "db", "stream_prefix", "")
manifest = transfer.import_bundle(client, src, prefix=prefix, replace=replace)
entries = list(entry) if entry else None
if entries:
listed = transfer.load_manifest(src).get("entries")
if listed is not None:
unknown = sorted(set(entries) - set(listed))
if unknown:
raise typer.BadParameter(
f"bundle does not contain: {', '.join(unknown)} (it has: {', '.join(listed) or 'none'})")
manifest = transfer.import_bundle(client, src, prefix=prefix, replace=replace, entries=entries)
for stream, info in manifest["streams"].items():
typer.echo(f"{stream:16s} loaded {info['rows']:>9} rows")
typer.echo(f"{stream:16s} loaded {info['rows']:>9} rows"
+ (" (filtered)" if entries else ""))
if entries:
typer.echo(f"entries loaded: {', '.join(entries)}")
typer.echo("import complete")


Expand Down
31 changes: 25 additions & 6 deletions src/tpk/db.py
Original file line number Diff line number Diff line change
Expand Up @@ -144,12 +144,13 @@ def _add_column_if_missing(client, stream: str, column: str, col_type: str) -> N
pass


def ensure_schema(client, prefix: str = "") -> None:
# All tpk streams live under a dedicated database (default `tpk`, #58);
# create it first so the qualified DDL below resolves. CREATE DATABASE is a
# global statement (works from the default-database session get_client uses)
# and is a no-op if it already exists, on both timeplusd and proton.
client.command(f"CREATE DATABASE IF NOT EXISTS {database()}")
GRAPH_STREAMS = ("kg_nodes", "kg_edges", "kg_ingest_log", "kg_repos")


def ensure_graph_streams(client, prefix: str = "") -> None:
"""DDL for the four graph/corpus streams only (`GRAPH_STREAMS`). Split out
of `ensure_schema` so a corpus import can stage a bundle under a
throwaway prefix (#104) without also creating the auth streams."""
# Keyed (upsertable, latest-state) streams -- MUTABLE on timeplusd,
# versioned_kv + `deleted` tombstone on proton (see _keyed_stream).
client.command(_keyed_stream(prefix, "kg_nodes", [
Expand Down Expand Up @@ -181,6 +182,24 @@ def ensure_schema(client, prefix: str = "") -> None:
"enabled bool", "visibility string", "extraction string",
"description string", "updated_at datetime64(3, 'UTC')",
], pk="name, ref"))


def drop_graph_streams(client, prefix: str) -> None:
"""Drop the four graph streams under `prefix` (staging cleanup, #104).
Refuses an empty prefix, like `drop_schema`."""
if not prefix:
raise ValueError("refusing to drop unprefixed (production) streams")
for name in GRAPH_STREAMS:
client.command(f"DROP STREAM IF EXISTS {qualified(name, prefix)}")


def ensure_schema(client, prefix: str = "") -> None:
# All tpk streams live under a dedicated database (default `tpk`, #58);
# create it first so the qualified DDL below resolves. CREATE DATABASE is a
# global statement (works from the default-database session get_client uses)
# and is a no-op if it already exists, on both timeplusd and proton.
client.command(f"CREATE DATABASE IF NOT EXISTS {database()}")
ensure_graph_streams(client, prefix)
client.command(_keyed_stream(prefix, "kg_users", [
"username string", "password_hash string", "role string",
"must_change_password bool", "disabled bool",
Expand Down
Loading
Loading