Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,8 @@ export the matching env var — whichever suits your deployment. Secrets are
| `TPK_AGENT_MODEL` | `[agent].model` | per-provider | Chat model override |
| `TPK_AGENT_REASONING_EFFORT` | `[agent].reasoning_effort` | unset | OpenAI reasoning effort (e.g. `low`) |
| `TPK_CHECKOUT_DIR` | `[server].checkout_dir` | `~/.tpk/checkouts` | GitHub checkout cache |
| `TPK_TRANSFER_DIR` | `[server].transfer_dir` | system temp dir | Scratch space for corpus export/import through the UI (#102) |
| `TPK_IMPORT_MAX_MB` | `[server].import_max_mb` | `2048` | Upload cap for `POST /api/corpus/import` (`0` = unlimited) |
| `TPK_SESSION_TTL` | `[server].session_ttl` | `86400` | Login session lifetime (seconds) |
| `TPK_CHAT_AUDIT` | `[server].chat_audit` | `true` | Chat Q&A auditing (`false`/`0` disables) |
| `TPK_DAILY_TOKEN_LIMIT` | `[server].daily_token_limit` | `500000` | Global fallback per-user daily token budget for non-admins (`0` = unlimited; a per-user or role `daily_token_limit` wins) |
Expand Down Expand Up @@ -125,7 +127,8 @@ the connecting user needs `CREATE DATABASE` (first run) plus read/write on `tpk`
**Upgrading an existing deployment:** streams created before this change live in
`default` and are **not** moved automatically — after upgrading, tpk looks in
`tpk` and finds an empty graph. Either re-ingest (`tpk ingest`), or carry data
over with `tpk export` (old version) → `tpk import` (new version). Point
over with `tpk export` (old version) → `tpk import` (new version), or from the
Manage tab as an admin (**Export corpus** → **Import corpus**, #102). Point
`TIMEPLUS_DATABASE=default` to keep using the old location instead.

## Docker compose (quickest start)
Expand Down
4 changes: 4 additions & 0 deletions deploy/k8s/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,10 @@ spec:
- **Backups.** Snapshot the DB data PVC (`/var/lib/proton` or
`/var/lib/timeplusd`), or use `tpk export` to write a portable bundle you can
`tpk import` elsewhere.
The same bundle is one click away in the Manage tab for an admin (**Export
corpus** downloads a `.tar.gz`, **Import corpus** uploads one); it is staged
in the `transfer` emptyDir (`TPK_TRANSFER_DIR=/opt/tpk/transfer`, 4Gi
`sizeLimit` — raise it if a packed bundle is bigger).
- **All-in-one disk guard (dev tuning).** The `timeplus/tpk` image bakes in the
dev override `deploy/timeplusd-dev/small-segments.yaml`, which — besides
disabling nativelog preallocation — raises proton's disk-usage write guard to
Expand Down
11 changes: 11 additions & 0 deletions deploy/k8s/allinone.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,11 @@ spec:
secretKeyRef:
name: tpk-secrets
key: TIMEPLUS_PASSWORD
# Scratch space for corpus export/import through the UI (#102):
# an emptyDir sized for a packed bundle, so a big download/upload
# never fills the container's writable layer.
- name: TPK_TRANSFER_DIR
value: /opt/tpk/transfer
# Optional chat-agent overrides (defaults inferred from the keys below).
# - name: TPK_AGENT_PROVIDER
# value: "anthropic"
Expand Down Expand Up @@ -137,6 +142,12 @@ spec:
mountPath: /var/lib/proton
- name: checkouts # cached github repo checkouts
mountPath: /opt/tpk/.checkouts
- name: transfer # export/import scratch (#102)
mountPath: /opt/tpk/transfer
volumes:
- name: transfer
emptyDir:
sizeLimit: 4Gi
volumeClaimTemplates:
- metadata:
name: data
Expand Down
10 changes: 10 additions & 0 deletions deploy/k8s/app-only.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,11 @@ spec:
# The DB is already up; a short connect-retry covers transient blips.
- name: TPK_DB_WAIT_SECONDS
value: "120"
# Scratch space for corpus export/import through the UI (#102):
# an emptyDir sized for a packed bundle, so a big download/upload
# never fills the container's writable layer.
- name: TPK_TRANSFER_DIR
value: /opt/tpk/transfer
# Optional chat-agent overrides:
# - name: TPK_AGENT_PROVIDER
# value: "anthropic"
Expand Down Expand Up @@ -133,10 +138,15 @@ spec:
volumeMounts:
- name: checkouts
mountPath: /opt/tpk/.checkouts
- name: transfer # export/import scratch (#102)
mountPath: /opt/tpk/transfer
volumes:
- name: checkouts
persistentVolumeClaim:
claimName: tpk-checkouts
- name: transfer
emptyDir:
sizeLimit: 4Gi
---
# Access service for the chat agent + web UI — target from Ingress / port-forward.
apiVersion: v1
Expand Down
10 changes: 10 additions & 0 deletions deploy/k8s/enterprise.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,11 @@ spec:
# app CrashLoopBackOff-ing on a slow first boot (cold image/PVC).
- name: TPK_DB_WAIT_SECONDS
value: "300"
# Scratch space for corpus export/import through the UI (#102):
# an emptyDir sized for a packed bundle, so a big download/upload
# never fills the container's writable layer.
- name: TPK_TRANSFER_DIR
value: /opt/tpk/transfer
# Optional chat-agent overrides:
# - name: TPK_AGENT_PROVIDER
# value: "anthropic"
Expand Down Expand Up @@ -288,10 +293,15 @@ spec:
volumeMounts:
- name: checkouts
mountPath: /opt/tpk/.checkouts
- name: transfer # export/import scratch (#102)
mountPath: /opt/tpk/transfer
volumes:
- name: checkouts
persistentVolumeClaim:
claimName: tpk-checkouts
- name: transfer
emptyDir:
sizeLimit: 4Gi
---
# Access service for the chat agent + web UI — target from Ingress / port-forward.
apiVersion: v1
Expand Down
3 changes: 2 additions & 1 deletion docs/FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,8 @@ Manage what the graph indexes at runtime, without editing files or restarting.
- **Background jobs.** Add / reindex operations run on a background worker; job status is pollable.
- **Manage tab** in the web UI drives all of it (add, toggle, reindex, delete), and shows the **ingest history** — every run, whether started from the UI, `tpk ingest` or `kubectl exec`: status (`running` / `ok` / `failed` with the error / `stale` for a run that never wrote its end row), nodes, edges, commit, duration. Backed by the append-only `kg_ingest_log` stream via `GET /api/ingest-log`.
- **Release-upgrade workflow:** add the new ref → ingest → verify → flip enabled; a legacy migration re-keys old bare-name rows on first ingest.
- **Export / import (issue #29).** `tpk export` dumps the ingested graph + corpus registry to a portable bundle (proton `FORMAT Parquet`, one file per stream + manifest); `tpk import` loads it into a new environment — no re-ingest, no LLM calls. Idempotent upsert by default (`--replace` to reset); auth streams excluded.
- **Export / import (issue #29).** `tpk export` dumps the ingested graph + corpus registry to a portable bundle (proton `FORMAT Parquet`, one file per stream + manifest); `tpk import` loads it into a new environment — no re-ingest, no LLM calls. Idempotent upsert by default (`--replace` to reset); auth streams excluded. The bundle carries each entry's registry row (source, ref, visibility, extraction mode, description) and the ingest history; a dev-mode local `path` is blanked for GitHub-sourced entries, since it is meaningless on another host. The append-only ingest log is exported with its `_tp_time`, so run order (and therefore each entry's `ok`/`started`/`stale` status and run durations) survives an import; a bundle written by an older `tpk` still loads, but one written by this version needs this version or newer to import.
- **Export / import from the Manage tab (issue #102).** Admins get **Export corpus** (one `.tar.gz` download of the same bundle, `GET /api/corpus/export`) and **Import corpus** (upload the archive, optional *replace*; `POST /api/corpus/import`). The archive is validated in full before anything is written — flat regular files only, a manifest, nothing the manifest doesn't list — then loaded as a job shown in the ingest panel with per-stream row counts. An import is refused (409) while any UI-started ingest or import job is active, so it never drops a stream such a job is writing into (a `tpk ingest` run from the CLI or `kubectl exec` is a separate process the server cannot see — don't import during one). Admin-only rather than `corpus:manage`, because a manager role may be scoped to a subset of entries and the bundle is the whole graph. Scratch space: `TPK_TRANSFER_DIR` (an `emptyDir` in the k8s manifests); upload cap `TPK_IMPORT_MAX_MB` (default 2048).

## 7. Authentication & role-based access (issue #8)

Expand Down
157 changes: 146 additions & 11 deletions src/tpk/api.py
Original file line number Diff line number Diff line change
@@ -1,19 +1,24 @@
"""Management API: corpus CRUD + background ingest jobs."""
"""Management API: corpus CRUD + background ingest jobs + corpus
export/import transport (#102)."""

import queue
import re
import shutil
import tempfile
import threading
import uuid
from datetime import datetime, timezone
from pathlib import Path

from fastapi import APIRouter, Depends, HTTPException, Query
from fastapi import APIRouter, Depends, File, Form, HTTPException, Query, UploadFile
from fastapi.responses import FileResponse
from pydantic import BaseModel
from starlette.background import BackgroundTask

import tpk.auth as auth_mod
from tpk import corpus, db
from tpk import corpus, db, transfer
from tpk.auth import AuthLayer, User
from tpk.config import RepoConfig, Settings, config_path, entry_key, load_llm
from tpk.config import RepoConfig, Settings, config_path, entry_key, load_llm, setting
from tpk import ingest as ingest_mod
from tpk.ingest import ingest_repo

Expand Down Expand Up @@ -139,8 +144,33 @@ def _validate(body: AddRepo) -> RepoConfig:
)


def transfer_dir() -> Path:
"""Scratch space for export bundles and uploaded import archives (#102):
env > [server].transfer_dir > the system temp dir. In Kubernetes point it
at an emptyDir sized for the bundle (see deploy/k8s)."""
return Path(setting("TPK_TRANSFER_DIR", "server", "transfer_dir",
tempfile.gettempdir())).expanduser()


def import_max_bytes() -> int:
"""Upload cap for `POST /api/corpus/import`, in MB (0 = unlimited)."""
mb = setting("TPK_IMPORT_MAX_MB", "server", "import_max_mb", 2048, cast=int)
return int(mb) * 1024 * 1024


IMPORT_ENTRY_KEY = "corpus import"


class JobManager:
"""One worker thread; jobs use their own DB client (never the KG's)."""
"""One worker thread; jobs use their own DB client (never the KG's).

Two job kinds share the queue: `ingest` (one corpus entry) and `import`
(a whole bundle, #102). Running them on the same single worker is what
serialises an import against ingests already queued; the API refuses to
*submit* an import while anything is active (409), so an import never
drops a stream a UI-started ingest is writing into. (A `tpk ingest`
run from the CLI or `kubectl exec` is a separate process this manager
cannot see -- don't import while one of those is running.)"""

def __init__(self, prefix: str = ""):
self.prefix = prefix
Expand All @@ -160,21 +190,33 @@ def _ensure_worker(self):
threading.Thread(target=self._run, daemon=True).start()
self._started = True

def submit(self, cfg: RepoConfig) -> str:
def _submit(self, kind: str, key: str, payload) -> str:
job_id = uuid.uuid4().hex[:12]
rec = {
"id": job_id, "entry_key": entry_key(cfg), "status": "queued",
"id": job_id, "kind": kind, "entry_key": key, "status": "queued",
"phase": None, "message": "", "nodes": 0, "edges": 0, "error": None,
"submitted_at": datetime.now(timezone.utc).isoformat(),
"updated_at": None,
"finished_at": None,
}
with self._jobs_lock:
self.jobs[job_id] = rec
self._q.put((job_id, cfg))
self._q.put((job_id, kind, payload))
self._ensure_worker()
return job_id

def submit(self, cfg: RepoConfig) -> str:
return self._submit("ingest", entry_key(cfg), cfg)

def submit_import(self, bundle_dir, replace: bool = False) -> str:
"""Load an already-unpacked bundle directory; the worker deletes the
directory when done, whatever the outcome."""
return self._submit("import", IMPORT_ENTRY_KEY, (Path(bundle_dir), replace))

def has_active(self) -> bool:
with self._jobs_lock:
return any(j["status"] in ("queued", "running") for j in self.jobs.values())

def snapshot(self) -> list[dict]:
with self._jobs_lock:
return list(self.jobs.values())
Expand All @@ -183,13 +225,35 @@ def get(self, job_id: str) -> dict | None:
with self._jobs_lock:
return self.jobs.get(job_id)

def _run_import(self, rec: dict, bundle_dir: Path, replace: bool):
try:
with self._jobs_lock:
rec["phase"] = "import"
rec["message"] = "replacing graph streams" if replace else "loading bundle"
rec["updated_at"] = datetime.now(timezone.utc).isoformat()
client = db.get_client(Settings.from_env())
manifest = transfer.import_bundle(client, bundle_dir, prefix=self.prefix,
replace=replace)
streams = {name: info["rows"] for name, info in manifest["streams"].items()}
with self._jobs_lock:
rec["streams"] = streams
rec["nodes"] = streams.get("kg_nodes", 0)
rec["edges"] = streams.get("kg_edges", 0)
rec["status"] = "ok"
finally:
shutil.rmtree(bundle_dir, ignore_errors=True)

def _run(self):
while True:
job_id, cfg = self._q.get()
job_id, kind, payload = self._q.get()
with self._jobs_lock:
rec = self.jobs[job_id]
rec["status"] = "running"
try:
if kind == "import":
self._run_import(rec, *payload)
continue
cfg = payload
client = db.get_client(Settings.from_env())
llm = load_llm(REPOS_TOML) if REPOS_TOML.exists() else None
backend = None if (llm is None or llm.backend == "auto") else llm.backend
Expand Down Expand Up @@ -220,8 +284,9 @@ def _on_progress(p, _rec=rec):
with self._jobs_lock:
rec["status"] = "failed"
rec["error"] = f"{type(exc).__name__}: {exc}"
with self._jobs_lock:
rec["finished_at"] = datetime.now(timezone.utc).isoformat()
finally:
with self._jobs_lock:
rec["finished_at"] = datetime.now(timezone.utc).isoformat()


def create_api_router(prefix: str = "", auth=None) -> APIRouter:
Expand Down Expand Up @@ -378,6 +443,76 @@ def api_ingest_log(limit: int = Query(50, ge=1, le=200), entry: str | None = Non
for runs that have not written their end row."""
return {"runs": ingest_mod.list_runs(_client(), prefix=prefix, limit=limit, entry=entry)}

# -- corpus export / import transport (#102) ---------------------------
# Admin-only, not corpus:manage: a non-admin manager is role-scoped to a
# subset of entries, but a bundle is the WHOLE graph (every internal
# entry included) and an import rewrites all of it.

@router.get("/corpus/export")
def export_corpus(format: str = Query("Parquet"), actor: User = Depends(auth.require_admin)):
"""Dump the graph + corpus registry and stream it back as one
`.tar.gz` -- the CLI bundle layout, so `tpk import` reads it unpacked."""
if format not in transfer.FORMATS:
raise HTTPException(422, "format must be Parquet|Native")
root = transfer_dir()
root.mkdir(parents=True, exist_ok=True)
work = Path(tempfile.mkdtemp(prefix="tpk-export-", dir=root))
try:
transfer.export_bundle(_client(), work / "bundle", prefix=prefix, fmt=format)
stamp = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
name = f"tpk-corpus-{stamp}{transfer.ARCHIVE_SUFFIX}"
archive = transfer.pack_bundle(work / "bundle", work / name)
shutil.rmtree(work / "bundle", ignore_errors=True)
except Exception:
shutil.rmtree(work, ignore_errors=True)
raise
# FileResponse streams from disk; the scratch dir goes once the body
# has been sent.
return FileResponse(archive, media_type="application/gzip", filename=name,
background=BackgroundTask(shutil.rmtree, work, ignore_errors=True))

@router.post("/corpus/import")
def import_corpus(file: UploadFile = File(...), replace: bool = Form(False),
actor: User = Depends(auth.require_admin)):
"""Upload a bundle archive and load it as a background job (poll
`/api/jobs/{id}`). The archive is validated in full before anything
is written; a load is refused while any job is queued or running."""
if jobs.has_active():
raise HTTPException(409, "an ingest or import job is already running; retry when it finishes")
root = transfer_dir()
root.mkdir(parents=True, exist_ok=True)
work = Path(tempfile.mkdtemp(prefix="tpk-import-", dir=root))
try:
limit = import_max_bytes()
# Dot-prefixed so no archive member (those are rejected) can
# name it; the bundle files unpack beside it into `work`.
archive = work / ".upload.tar.gz"
size = 0
with open(archive, "wb") as fh:
while chunk := file.file.read(1024 * 1024):
size += len(chunk)
if limit and size > limit:
raise HTTPException(413, f"archive exceeds the {limit // (1024 * 1024)} MB upload limit (TPK_IMPORT_MAX_MB)")
fh.write(chunk)
try:
manifest = transfer.unpack_bundle(archive, work)
except transfer.BadBundle as exc:
raise HTTPException(400, f"invalid bundle: {exc}")
archive.unlink()
except Exception:
shutil.rmtree(work, ignore_errors=True)
raise
# Re-check under the submit: has_active() above and the submit are
# not atomic, but the single worker still serialises whatever lands
# in the queue, so the race only costs the 409 courtesy, not safety.
if jobs.has_active():
shutil.rmtree(work, ignore_errors=True)
raise HTTPException(409, "an ingest or import job is already running; retry when it finishes")
job_id = jobs.submit_import(work, replace=replace)
return {"job_id": job_id,
"streams": {k: v["rows"] for k, v in manifest["streams"].items()},
"format": manifest.get("format"), "created_at": manifest.get("created_at")}

@router.get("/jobs")
def list_jobs(actor: User = Depends(auth.require_cap(auth_mod.CAP_CORPUS_VIEW))):
return sorted(jobs.snapshot(), key=lambda j: j["submitted_at"], reverse=True)[:50]
Expand Down
Loading
Loading