From c588ad95e2d83f7d220d281e7e4098ba5b1611e0 Mon Sep 17 00:00:00 2001 From: Jakub Dzikowski Date: Fri, 4 Sep 2026 15:37:54 +0200 Subject: [PATCH 01/60] Docs: fold leftover Unreleased notes into 0.14.0 The 0.14.0 stamp left follow-up work in Unreleased. Move the change list into 0.14.0 and keep only the Summary lines that are not already in that section. Co-authored-by: Cursor --- CHANGELOG.md | 36 +++++++++++++----------------------- 1 file changed, 13 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7dd5fa58..fba39767 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,15 +2,19 @@ ## Summary -- **CLI:** `claude` no longer warns when `ANTHROPIC_API_KEY` and `CLAUDE_CODE_OAUTH_TOKEN` are unset. A stored Claude CLI login is the host path. -- **Language:** compact `return match … { … }` and `return prompt …` parse as those expressions. A bad `return` is `E_PARSE`, not a shell line. -- **Language:** using a `const` before its declaration is `E_VALIDATE` in a `${…}` interpolation, a bare `run` / `prompt` call argument, and an `if` / `match` subject, not only in a `run` / `prompt` target. -- **Language:** nested `const` / `script` / `def` / named `prompt` declared inside an `if` / `else` / `for` / `catch` / `recover` body are block-scoped to that body. A reference after the branch is `E_VALIDATE`, not a name that validates and then interpolates empty at runtime when the branch was not taken. -- **Language:** a nested `def` may `run` itself. Self-recursion validates and is bounded only by the runtime recursion-depth cap of 256; a `run` of a sibling nested `def` declared later in the same body stays `E_VALIDATE` (declarations are sequential, not hoisted). -- **Docs:** Spawn-env and journal rules are guardrails, not a sandbox. [Why Jaiph](docs/why-jaiph.md) states the limit. [Deploy jaiph](docs/deploy.md) has the dedicated-user / two-run operator recipe. -- **Docs:** [Pass a host key to a script](docs/script-env.md) is the operator recipe for sterile script env, `use`, and `--env`. -- **Editors:** VS Code and Zed now scope a named prompt call site (`prompt analyze(log)`) as a function and highlight the field on a dotted `if` subject (`if answer.risk == "ok"`). Neither editor paints the invalid reverse arrow `<-` as a send operator. -- **Security — spawn env:** `--env` values no longer sit on the runner (workflow-leader) process environment. Jaiph builds the runner env from an allowlist (process basics, `JAIPH_*` control keys, backend credentials), so an ungranted host key is absent from it, and a granted value reaches only a subprocess whose declaration `use`s the key. +## All changes + +# 0.14.0 + +## Summary + +- **Language:** `def` and `run` replace `workflow` / `rule` / `ensure`. Names are private unless `export`. `jaiph run` needs `export def main`. +- **Runtime:** No first-party Docker sandbox. Runs execute on the host. Isolate with your own container or CI runner ([Deploy jaiph](docs/deploy.md)). +- **Scripts:** Sterile env. A script sees process basics, `JAIPH_*` contract keys, and host keys listed in `use` and granted with `--env`. Host presence alone is not a grant. `--env` values do not sit on the runner; a granted key reaches only a subprocess whose declaration `use`s it. +- **Prompts:** Named, reusable `prompt name(params) [use KEY] = "…"`. A def may declare a local `script`, `def`, `prompt`, or `const`. Nested decls in `if` / `for` / `catch` / `recover` are block-scoped. A nested `def` may `run` itself. +- **Channels:** `send payload -> channel`. The old `channel <- payload` form is gone. +- **CLI:** `--env KEY` names a missing host value and how to pass it. `use` / `--env` cannot name the audit-chain key or journal path. +- **Editors:** VS Code, Zed, and the docs highlighter cover `use`, `import script`, and named prompts. ## All changes @@ -31,20 +35,6 @@ - **Fix — Language:** sequential `const` visibility now rejects use-before-declaration in every position, not only a `run` / `prompt` target. A `${…}` interpolation, a bare `run` / `prompt` call argument, and an `if` / `match` subject that names a `const` declared later in the same def are `E_VALIDATE` (unknown identifier). A nested `def` or named `prompt` body sees an enclosing `const` only when it was declared before that nested declaration; a `${…}` of a later enclosing `const` is `E_VALIDATE`, while the enclosing def's params and module-level `const`s stay visible. Runtime interpolation of a genuinely missing variable is still empty — the change is compile-time rejection. Previously these forms compiled and interpolated empty at runtime. Docs: [Language — `const`](docs/language.md#const--bind-a-value), [Language — Nested declarations](docs/language.md#nested-declarations). Tests: `src/transpile/validate-nested-decl.test.ts`, `e2e/tests/148_nested_decls.sh`. -# 0.14.0 - -## Summary - -- **Language:** `def` and `run` replace `workflow` / `rule` / `ensure`. Names are private unless `export`. `jaiph run` needs `export def main`. -- **Runtime:** No first-party Docker sandbox. Runs execute on the host. Isolate with your own container or CI runner ([Deploy jaiph](docs/deploy.md)). -- **Scripts:** Sterile env. A script sees process basics, `JAIPH_*` contract keys, and host keys listed in `use` and granted with `--env`. Host presence alone is not a grant. -- **Prompts:** Named, reusable `prompt name(params) [use KEY] = "…"`. A def may declare a local `script`, `def`, `prompt`, or `const`. -- **Channels:** `send payload -> channel`. The old `channel <- payload` form is gone. -- **CLI:** `--env KEY` names a missing host value and how to pass it. `use` / `--env` cannot name the audit-chain key or journal path. -- **Editors:** VS Code, Zed, and the docs highlighter cover `use`, `import script`, and named prompts. - -## All changes - - **UX — CLI:** a bare `--env KEY` whose value is unset on the host now says Jaiph requires that key and how to pass it (`--env KEY` or `--env KEY=VALUE`), instead of `no value given and KEY is not set on the host`. - **Fix — Reserved keys:** `use` and `--env` reject `JAIPH_CHAIN_KEY` and `JAIPH_RUN_SUMMARY_FILE` (`E_ENV_RESERVED`). Those keys stay with the runner; a `use` grant can no longer put them on a script or agent after the prompt scrub. - **Docs:** operator recipe for `run async` at [Run work concurrently](docs/async.md). The value model stays at [Async Handles](docs/spec-async-handles.md). From fcb547a8821a3bae19abf0402eaa80b0710c7bc8 Mon Sep 17 00:00:00 2001 From: Jakub Dzikowski Date: Thu, 10 Sep 2026 13:14:59 +0200 Subject: [PATCH 02/60] Feat: flatten serve paths, publish a runtime image, and add the product-owner API Invoke is POST /{name} with no /v1 prefix. Releases push ghcr.io//jaiph-runtime. The product-owner defs are the sole queue writers; pick prefers the first available task. Co-authored-by: Cursor --- .github/workflows/release.yml | 89 ++++ .gitignore | 3 + .jaiph/libs/jaiphlang/queue.jh | 49 +- .jaiph/libs/jaiphlang/queue.py | 480 +++++++++++++++----- .jaiph/libs/jaiphlang/queue_selftest.jh | 7 + .jaiph/libs/jaiphlang/queue_test.py | 114 +++++ .jaiph/product_owner.jh | 283 ++++++++++++ .jaiph/product_owner.test.jh | 154 +++++++ .jaiph/queue_ops.test.jh | 8 + CHANGELOG.md | 7 + DONE.md | 4 + QUEUE.md | 116 ++++- docs/architecture.md | 4 +- docs/build-jaiph-dev-image.sh | 50 ++ docs/cli.md | 28 +- docs/contributing.md | 16 +- docs/deploy.md | 24 +- docs/deploy/k8s.yaml | 16 +- docs/env-vars.md | 2 +- docs/serve.md | 72 +-- e2e/tests/147_serve_http_api.sh | 36 +- e2e/tests/152_shell_injection_serve.sh | 6 +- integration/exec-policy.test.ts | 2 +- integration/otlp-export.test.ts | 4 +- integration/release-workflow.test.ts | 30 +- integration/sentry-export.test.ts | 2 +- integration/serve-auth.test.ts | 28 +- integration/serve-restart.test.ts | 22 +- integration/serve-server.test.ts | 90 ++-- runtime/.gitignore | 3 + runtime/Dockerfile | 23 + src/cli/commands/serve.ts | 90 ++-- src/cli/serve/auth.ts | 3 +- src/cli/serve/handler.test.ts | 197 ++++---- src/cli/serve/handler.ts | 66 +-- src/cli/serve/openapi.test.ts | 13 +- src/cli/serve/openapi.ts | 29 +- src/cli/serve/server.test.ts | 22 +- src/cli/serve/types.ts | 2 +- src/cli/shared/serve-bootstrap.ts | 8 +- src/cli/shared/usage.ts | 15 +- src/cli/shared/workflow-call-exec.ts | 2 +- src/runtime/kernel/emit.ts | 2 +- src/runtime/kernel/redact.ts | 2 +- src/runtime/kernel/runtime-event-emitter.ts | 2 +- start-product-owner.sh | 60 +++ 46 files changed, 1774 insertions(+), 511 deletions(-) create mode 100644 .jaiph/libs/jaiphlang/queue_selftest.jh create mode 100644 .jaiph/libs/jaiphlang/queue_test.py create mode 100644 .jaiph/product_owner.jh create mode 100644 .jaiph/product_owner.test.jh create mode 100644 .jaiph/queue_ops.test.jh create mode 100644 DONE.md create mode 100755 docs/build-jaiph-dev-image.sh create mode 100644 runtime/.gitignore create mode 100644 runtime/Dockerfile create mode 100755 start-product-owner.sh diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 87d714a6..9f083611 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -149,6 +149,95 @@ jobs: bash ../scripts/release-version-check.sh \ "${{ steps.meta.outputs.channel }}" "${{ steps.meta.outputs.tag }}" "${got}" + publish-image: + name: Publish GHCR runner image + needs: [build, sanity-windows] + runs-on: ubuntu-latest + permissions: + contents: read + packages: write + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Resolve tag and channel + id: meta + run: | + set -euo pipefail + case "${GITHUB_REF}" in + refs/tags/v*) + tag="${GITHUB_REF_NAME}"; channel="stable" ;; + refs/heads/nightly) + tag="nightly"; channel="nightly" ;; + *) + echo "Unsupported ref for release: ${GITHUB_REF}" >&2; exit 1 ;; + esac + echo "tag=${tag}" >> "${GITHUB_OUTPUT}" + echo "channel=${channel}" >> "${GITHUB_OUTPUT}" + + - name: Resolve image tags + id: image + run: | + set -euo pipefail + owner="$(printf '%s' "${GITHUB_REPOSITORY_OWNER}" | tr '[:upper:]' '[:lower:]')" + image="ghcr.io/${owner}/jaiph-runtime" + echo "image=${image}" >> "${GITHUB_OUTPUT}" + case "${{ steps.meta.outputs.channel }}" in + stable) + tag="${{ steps.meta.outputs.tag }}" + ver="${tag#v}" + echo "tags=${image}:${ver},${image}:${tag},${image}:latest" >> "${GITHUB_OUTPUT}" + ;; + nightly) + echo "tags=${image}:nightly" >> "${GITHUB_OUTPUT}" + ;; + *) + echo "Unsupported channel: ${{ steps.meta.outputs.channel }}" >&2 + exit 1 + ;; + esac + + - name: Download linux binary artifacts + uses: actions/download-artifact@v4 + with: + pattern: jaiph-linux-* + path: runtime + merge-multiple: true + + - name: Prepare linux binaries + working-directory: runtime + run: | + set -euo pipefail + test -f jaiph-linux-x64 + test -f jaiph-linux-arm64 + chmod +x jaiph-linux-x64 jaiph-linux-arm64 + + - name: Set up QEMU + uses: docker/setup-qemu-action@v3 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Log in to GHCR + uses: docker/login-action@v3 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Build and push + uses: docker/build-push-action@v6 + with: + context: runtime + file: runtime/Dockerfile + platforms: linux/amd64,linux/arm64 + push: true + tags: ${{ steps.image.outputs.tags }} + labels: | + org.opencontainers.image.source=https://github.com/${{ github.repository }} + org.opencontainers.image.revision=${{ github.sha }} + org.opencontainers.image.version=${{ steps.meta.outputs.tag }} + release: name: Publish release assets needs: [build, sanity-windows] diff --git a/.gitignore b/.gitignore index b39a8cec..dcd068b1 100644 --- a/.gitignore +++ b/.gitignore @@ -34,7 +34,10 @@ jaiph.key *.swp *.vsix +.jaiph/queue-state.md +.jaiph/queue-state.md.lock .jaiph/runs/ +.jaiph/product-owner/ .jaiph/*.jaiph.map .jaiph/tmp/ .jaiph/scripts/ diff --git a/.jaiph/libs/jaiphlang/queue.jh b/.jaiph/libs/jaiphlang/queue.jh index a384927c..9fd3a571 100755 --- a/.jaiph/libs/jaiphlang/queue.jh +++ b/.jaiph/libs/jaiphlang/queue.jh @@ -1,80 +1,87 @@ #!/usr/bin/env jaiph # -# Task/queue management for QUEUE.md. -# Reads/modifies ${JAIPH_WORKSPACE:-.}/QUEUE.md. +# Task/queue management. +# Canonical store: ${JAIPH_QUEUE_STATE} or ${JAIPH_WORKSPACE:-.}/.jaiph/queue-state.md +# QUEUE.md and DONE.md are generated views (never parsed after bootstrap). # Tags are #hashtags in ## headers (e.g. ## My Task #dev-ready). # # CLI usage: # jaiph .jaiph/libs/jaiphlang/queue.jh headers -# jaiph .jaiph/libs/jaiphlang/queue.jh get dev-ready +# jaiph .jaiph/libs/jaiphlang/queue.jh get_available # jaiph .jaiph/libs/jaiphlang/queue.jh json -# jaiph .jaiph/libs/jaiphlang/queue.jh add_from_file path/to/tasks.md # import script "./queue.py" as queue -# Dispatches CLI arguments to the queue Python script. export def main(cmd, arg1, arg2) { const result = run queue(cmd, arg1, arg2) log result } -# Append ## tasks from a markdown file into QUEUE.md. Titles already present -# are skipped. Missing #dev-ready tags are added automatically. export def add_tasks_from_file(path) { run queue("add_from_file", path) } -# Returns the full text block (header + body) of the first task. +export def add_task_from_file(path) { + run queue("add", path) +} + export def get_first_task() { return run queue("get") } -# Returns the first task whose header carries the given tag. export def next_task(tag) { return run queue("get", tag) } -# Returns the full text block of a task identified by its title. -# Accepts both clean titles and titles with #tags — tags are stripped -# before matching so callers don't need to know the exact tag set. +export def next_available() { + return run queue("get_available") +} + export def get_task_by_header(header) { return run queue("get_by_header", header) } -# Returns all task titles as newline-separated text (without ## prefix). -# Pass an optional tag to filter (e.g. "dev-ready"). export def get_all_task_headers() { return run queue("headers") } -# Adds #dev-ready to the header of the task matching the given title. export def mark_task_dev_ready(header) { run queue("mark", header, "dev-ready") } -# Removes the task matching the given title from the queue file. +export def mark_task_in_progress(header) { + run queue("mark", header, "in-progress") +} + +export def unmark_task(header, tag) { + run queue("unmark", header, tag) +} + export def remove_completed_task(header) { run queue("complete_by_header", header) } -# Replaces the body (markdown below the ## line) for the task matching the title. -# `bodyPath` must be a UTF-8 file; header line and tags are unchanged. +export def archive_to_done(header, notesPath) { + run queue("archive_to_done", header, notesPath) +} + export def set_task_description_from_file(header, bodyPath) { run queue("set_description", header, bodyPath) } -# Passes when the queue has at least one task. +export def state_json() { + return run queue("json") +} + export def has_tasks() { run queue("get") } -# Passes when the task text has #dev-ready on its header line. export def task_is_dev_ready(task) { run queue("has_tag", task, "dev-ready") } -# Passes only when every task in the queue carries #dev-ready. export def all_dev_ready() { run queue("check_all_tagged", "dev-ready") } diff --git a/.jaiph/libs/jaiphlang/queue.py b/.jaiph/libs/jaiphlang/queue.py index 17ccb8bf..0751c862 100755 --- a/.jaiph/libs/jaiphlang/queue.py +++ b/.jaiph/libs/jaiphlang/queue.py @@ -1,8 +1,52 @@ #!/usr/bin/env python3 -import sys, os, re, json +import fcntl, json, os, re, sys, time + +DONE_SEP = "\n\n" + +QUEUE_VIEW_PREAMBLE = """# Jaiph Improvement Queue (Hard Rewrite Track) + +This file is a generated view. Do not edit it. Do not agent-edit it. +Use the product-owner defs (`propose_task`, `update_task`, `pick_task`, +`report_completed_task`, `task_details`) via `jaiph serve` / MCP. + +Process rules: + +1. `pick_task` prefers the first `#dev-ready` task that is not `#in-progress`. + The product owner may claim a later available task instead. +2. The first `##` section is the preferred next task in this view. +3. `#dev-ready` means ready to implement. `#in-progress` means claimed by `pick_task`. +4. Runtime mutations go through the product-owner defs only. +5. Every task must be standalone: no hidden assumptions, no "read prior task". +6. Hard rewrite semantics: breaking changes are allowed unless a task says otherwise. +7. Acceptance criteria are non-negotiable. A task is not done until every + acceptance bullet is verified by a test that fails when the contract is violated. +""" + +DONE_VIEW_PREAMBLE = """# Done + +Append-only archive. Generated view. Do not edit. +Each section was accepted by the product-owner `report_completed_task` def. +""" + + +def workspace_root(): + return os.environ.get("JAIPH_WORKSPACE", ".") + + +def view_queue_path(): + return os.path.join(workspace_root(), "QUEUE.md") + + +def view_done_path(): + return os.path.join(workspace_root(), "DONE.md") + + +def state_path(): + env = os.environ.get("JAIPH_QUEUE_STATE") + if env: + return env + return os.path.join(workspace_root(), ".jaiph", "queue-state.md") -def queue_path(): - return os.path.join(os.environ.get("JAIPH_WORKSPACE", "."), "QUEUE.md") def clean_header(h): h = h.strip() @@ -10,11 +54,10 @@ def clean_header(h): h = h[3:] return re.sub(r"\s*#[A-Za-z0-9_-]+", "", h).strip() -def parse_queue(path): - if not os.path.isfile(path): + +def parse_queue_text(text): + if not text or not text.strip(): return {"description": "", "tasks": []} - with open(path) as f: - text = f.read() lines = text.split("\n") desc_lines, tasks, current = [], [], None for line in lines: @@ -37,25 +80,42 @@ def parse_queue(path): tasks.append(current) return {"description": "\n".join(desc_lines).strip(), "tasks": tasks} -def write_queue(path, q): + +def parse_queue(path): + if not os.path.isfile(path): + return {"description": "", "tasks": []} + with open(path, encoding="utf-8") as f: + return parse_queue_text(f.read()) + + +def emit_tasks(tasks): lines = [] - if q["description"]: - lines.append(q["description"]) - lines.append("") - for t in q["tasks"]: + for t in tasks: tag_s = " " + " ".join(f"#{x}" for x in t["tags"]) if t["tags"] else "" lines.append(f"## {t['title']}{tag_s}") - if t["description"]: + if t.get("description"): lines.append("") lines.append(t["description"]) lines.append("") - with open(path, "w") as f: + return lines + + +def write_markdown(path, preamble, tasks): + lines = [] + if preamble: + lines.append(preamble.rstrip()) + lines.append("") + lines.extend(emit_tasks(tasks)) + os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True) + with open(path, "w", encoding="utf-8") as f: f.write("\n".join(lines).rstrip() + "\n") + def fmt_task(t): tag_s = " " + " ".join(f"#{x}" for x in t["tags"]) if t["tags"] else "" h = f"## {t['title']}{tag_s}" - return f"{h}\n\n{t['description']}" if t["description"] else h + return f"{h}\n\n{t['description']}" if t.get("description") else h + def find_task(tasks, header): needle = clean_header(header) @@ -64,59 +124,170 @@ def find_task(tasks, header): return i return -1 + +def empty_state(): + return {"description": "", "tasks": [], "done": []} + + +def parse_state_text(text): + if DONE_SEP in text: + live, done_text = text.split(DONE_SEP, 1) + else: + live, done_text = text, "" + q = parse_queue_text(live) + d = parse_queue_text(done_text) + return {"description": q["description"], "tasks": q["tasks"], "done": d["tasks"]} + + +def read_state_file(path): + if not os.path.isfile(path): + return None + with open(path, encoding="utf-8") as f: + return parse_state_text(f.read()) + + +def write_state_file(path, st): + lines = [] + if st.get("description"): + lines.append(st["description"]) + lines.append("") + lines.extend(emit_tasks(st["tasks"])) + body = "\n".join(lines).rstrip() + if st.get("done"): + done_body = "\n".join(emit_tasks(st["done"])).rstrip() + body = body + DONE_SEP + done_body + os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True) + with open(path, "w", encoding="utf-8") as f: + f.write(body.rstrip() + "\n") + + +def dump_views(st): + write_markdown(view_queue_path(), QUEUE_VIEW_PREAMBLE, st["tasks"]) + stripped = [] + for t in st["done"]: + stripped.append({ + "title": t["title"], + "tags": [], + "description": t.get("description") or "", + }) + write_markdown(view_done_path(), DONE_VIEW_PREAMBLE, stripped) + + +def bootstrap_state(): + path = state_path() + existing = read_state_file(path) + if existing is not None: + return existing + view = view_queue_path() + if os.path.isfile(view): + q = parse_queue(view) + st = {"description": q["description"], "tasks": q["tasks"], "done": []} + else: + st = empty_state() + write_state_file(path, st) + dump_views(st) + return st + + +def with_lock(fn): + path = state_path() + os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True) + lock_path = path + ".lock" + with open(lock_path, "a+", encoding="utf-8") as lf: + fcntl.lockf(lf, fcntl.LOCK_EX) + try: + return fn() + finally: + fcntl.lockf(lf, fcntl.LOCK_UN) + + +def load_state(): + return bootstrap_state() + + +def save_state(st): + write_state_file(state_path(), st) + dump_views(st) + + def cmd_get(args): - tag = args[0] if args else None - q = parse_queue(queue_path()) - for t in q["tasks"]: - if tag is None or tag in t["tags"]: - print(fmt_task(t)) - return - sys.exit(1) + def go(): + tag = args[0] if args else None + st = load_state() + for t in st["tasks"]: + if tag is None or tag in t["tags"]: + print(fmt_task(t)) + return + sys.exit(1) + with_lock(go) + + +def cmd_get_available(args): + def go(): + st = load_state() + for t in st["tasks"]: + if "dev-ready" in t["tags"] and "in-progress" not in t["tags"]: + print(fmt_task(t)) + return + print("no #dev-ready task is available", file=sys.stderr) + sys.exit(1) + with_lock(go) + def cmd_get_by_header(args): if not args: print("get_by_header: header required", file=sys.stderr) sys.exit(1) - q = parse_queue(queue_path()) - i = find_task(q["tasks"], args[0]) - if i < 0: - print(f"task not found: {args[0]}", file=sys.stderr) - sys.exit(1) - print(fmt_task(q["tasks"][i])) + def go(): + st = load_state() + i = find_task(st["tasks"], args[0]) + if i < 0: + print(f"task not found: {args[0]}", file=sys.stderr) + sys.exit(1) + print(fmt_task(st["tasks"][i])) + with_lock(go) + def cmd_headers(args): - tag = args[0] if args else None - q = parse_queue(queue_path()) - for t in q["tasks"]: - if tag is None or tag in t["tags"]: - print(t["title"]) + def go(): + tag = args[0] if args else None + st = load_state() + for t in st["tasks"]: + if tag is None or tag in t["tags"]: + print(t["title"]) + with_lock(go) + def cmd_complete(args): tag = args[0] if args else None - path = queue_path() - q = parse_queue(path) - for i, t in enumerate(q["tasks"]): - if tag is None or tag in t["tags"]: - removed = q["tasks"].pop(i) - write_queue(path, q) - print(f"Completed: {removed['title']}") - return - print("No matching task found", file=sys.stderr) - sys.exit(1) + def go(): + st = load_state() + for i, t in enumerate(st["tasks"]): + if tag is None or tag in t["tags"]: + removed = st["tasks"].pop(i) + save_state(st) + print(f"Completed: {removed['title']}") + return + print("No matching task found", file=sys.stderr) + sys.exit(1) + with_lock(go) + def cmd_complete_by_header(args): if not args: print("complete_by_header: header required", file=sys.stderr) sys.exit(1) - path = queue_path() - q = parse_queue(path) - i = find_task(q["tasks"], args[0]) - if i < 0: - print(f"task not found: {args[0]}", file=sys.stderr) - sys.exit(1) - q["tasks"].pop(i) - write_queue(path, q) - print(f"Completed: {args[0]}") + def go(): + st = load_state() + i = find_task(st["tasks"], args[0]) + if i < 0: + print(f"task not found: {args[0]}", file=sys.stderr) + sys.exit(1) + st["tasks"].pop(i) + save_state(st) + print(f"Completed: {args[0]}") + with_lock(go) + def cmd_set_description(args): if len(args) < 2: @@ -128,44 +299,70 @@ def cmd_set_description(args): sys.exit(1) with open(body_path, encoding="utf-8") as f: body = f.read() - qpath = queue_path() - q = parse_queue(qpath) - i = find_task(q["tasks"], header) - if i < 0: - print(f"task not found: {header}", file=sys.stderr) - sys.exit(1) - q["tasks"][i]["description"] = body.rstrip() - write_queue(qpath, q) - print(f"Updated description: {q['tasks'][i]['title']}") + def go(): + st = load_state() + i = find_task(st["tasks"], header) + if i < 0: + print(f"task not found: {header}", file=sys.stderr) + sys.exit(1) + st["tasks"][i]["description"] = body.rstrip() + save_state(st) + print(f"Updated description: {st['tasks'][i]['title']}") + with_lock(go) + def cmd_mark(args): if len(args) < 2: print("mark: header and tag required", file=sys.stderr) sys.exit(1) header, tag = args[0], args[1] - path = queue_path() - q = parse_queue(path) - i = find_task(q["tasks"], header) - if i < 0: - print(f"task not found: {header}", file=sys.stderr) - sys.exit(1) - t = q["tasks"][i] - if tag not in t["tags"]: - t["tags"].append(tag) - write_queue(path, q) - print(f"Marked #{tag}: {t['title']}") + def go(): + st = load_state() + i = find_task(st["tasks"], header) + if i < 0: + print(f"task not found: {header}", file=sys.stderr) + sys.exit(1) + t = st["tasks"][i] + if tag not in t["tags"]: + t["tags"].append(tag) + save_state(st) + print(f"Marked #{tag}: {t['title']}") + with_lock(go) + + +def cmd_unmark(args): + if len(args) < 2: + print("unmark: header and tag required", file=sys.stderr) + sys.exit(1) + header, tag = args[0], args[1] + def go(): + st = load_state() + i = find_task(st["tasks"], header) + if i < 0: + print(f"task not found: {header}", file=sys.stderr) + sys.exit(1) + t = st["tasks"][i] + if tag in t["tags"]: + t["tags"] = [x for x in t["tags"] if x != tag] + save_state(st) + print(f"Unmarked #{tag}: {t['title']}") + with_lock(go) + def cmd_check_all_tagged(args): if not args: print("check_all_tagged: tag required", file=sys.stderr) sys.exit(1) tag = args[0] - q = parse_queue(queue_path()) - if not q["tasks"]: - sys.exit(1) - for t in q["tasks"]: - if tag not in t["tags"]: + def go(): + st = load_state() + if not st["tasks"]: sys.exit(1) + for t in st["tasks"]: + if tag not in t["tags"]: + sys.exit(1) + with_lock(go) + def cmd_has_tag(args): if len(args) < 2: @@ -175,40 +372,26 @@ def cmd_has_tag(args): if f"#{args[1]}" not in first_line: sys.exit(1) + def cmd_json(args): - print(json.dumps(parse_queue(queue_path()), indent=2)) + def go(): + print(json.dumps(load_state(), indent=2)) + with_lock(go) -def cmd_add_from_file(args): - """Append tasks from a markdown file. Skips titles that already exist. - The file is parsed like QUEUE.md (## Title #tags + body). Existing titles - in QUEUE.md are left untouched. Returns how many tasks were added. - """ - if not args: - print("add_from_file: path required", file=sys.stderr) - sys.exit(1) - src = args[0] - if not os.path.isfile(src): - print(f"add_from_file: file not found: {src}", file=sys.stderr) - sys.exit(1) - incoming = parse_queue(src) - if not incoming["tasks"]: - print("Added 0 tasks (file had no ## sections)") - return - path = queue_path() - q = parse_queue(path) - existing = {t["title"] for t in q["tasks"]} +def append_tasks_from_parsed(incoming, force_dev_ready): + st = load_state() + existing = {t["title"] for t in st["tasks"]} added = 0 skipped = 0 for t in incoming["tasks"]: if t["title"] in existing: skipped += 1 continue - # Overnight / engineer loops require #dev-ready on the header. tags = list(t["tags"]) - if "dev-ready" not in tags: + if force_dev_ready and "dev-ready" not in tags: tags.append("dev-ready") - q["tasks"].append({ + st["tasks"].append({ "title": t["title"], "tags": tags, "description": t["description"], @@ -216,16 +399,101 @@ def cmd_add_from_file(args): existing.add(t["title"]) added += 1 if added: - write_queue(path, q) + save_state(st) print(f"Added {added} tasks" + (f" (skipped {skipped} existing)" if skipped else "")) + +def cmd_add_from_file(args): + if not args: + print("add_from_file: path required", file=sys.stderr) + sys.exit(1) + src = args[0] + if not os.path.isfile(src): + print(f"add_from_file: file not found: {src}", file=sys.stderr) + sys.exit(1) + incoming = parse_queue(src) + if not incoming["tasks"]: + print("Added 0 tasks (file had no ## sections)") + return + def go(): + append_tasks_from_parsed(incoming, True) + with_lock(go) + + +def cmd_add(args): + if not args: + print("add: path required", file=sys.stderr) + sys.exit(1) + src = args[0] + if not os.path.isfile(src): + print(f"add: file not found: {src}", file=sys.stderr) + sys.exit(1) + incoming = parse_queue(src) + if not incoming["tasks"]: + print("Added 0 tasks (file had no ## sections)") + return + def go(): + append_tasks_from_parsed(incoming, False) + with_lock(go) + + +def cmd_archive_to_done(args): + if not args: + print("archive_to_done: header required", file=sys.stderr) + sys.exit(1) + header = args[0] + notes = "" + if len(args) >= 2 and args[1] and os.path.isfile(args[1]): + with open(args[1], encoding="utf-8") as f: + notes = f.read().strip() + def go(): + st = load_state() + i = find_task(st["tasks"], header) + if i < 0: + print(f"task not found: {header}", file=sys.stderr) + sys.exit(1) + t = st["tasks"].pop(i) + tags = [x for x in t["tags"] if x not in ("dev-ready", "in-progress")] + parts = [f"Completed: {time.strftime('%Y-%m-%d')}"] + if notes: + parts.extend(["", "### PO notes", "", notes]) + if t.get("description"): + parts.extend(["", t["description"]]) + st["done"].append({ + "title": t["title"], + "tags": tags, + "description": "\n".join(parts).strip(), + }) + save_state(st) + print(f"Archived: {t['title']}") + with_lock(go) + + +def cmd_dump_views(args): + def go(): + st = load_state() + dump_views(st) + print("Dumped views") + with_lock(go) + + cmds = { - "get": cmd_get, "get_by_header": cmd_get_by_header, - "headers": cmd_headers, "complete": cmd_complete, - "complete_by_header": cmd_complete_by_header, "mark": cmd_mark, + "get": cmd_get, + "get_available": cmd_get_available, + "get_by_header": cmd_get_by_header, + "headers": cmd_headers, + "complete": cmd_complete, + "complete_by_header": cmd_complete_by_header, + "mark": cmd_mark, + "unmark": cmd_unmark, "set_description": cmd_set_description, - "has_tag": cmd_has_tag, "check_all_tagged": cmd_check_all_tagged, - "json": cmd_json, "add_from_file": cmd_add_from_file, + "has_tag": cmd_has_tag, + "check_all_tagged": cmd_check_all_tagged, + "json": cmd_json, + "add_from_file": cmd_add_from_file, + "add": cmd_add, + "archive_to_done": cmd_archive_to_done, + "dump_views": cmd_dump_views, } argv = [a for a in sys.argv[1:] if a] diff --git a/.jaiph/libs/jaiphlang/queue_selftest.jh b/.jaiph/libs/jaiphlang/queue_selftest.jh new file mode 100644 index 00000000..cc8df729 --- /dev/null +++ b/.jaiph/libs/jaiphlang/queue_selftest.jh @@ -0,0 +1,7 @@ +#!/usr/bin/env jaiph + +import script "./queue_test.py" as queue_selftest + +export def main() { + return run queue_selftest() +} diff --git a/.jaiph/libs/jaiphlang/queue_test.py b/.jaiph/libs/jaiphlang/queue_test.py new file mode 100644 index 00000000..23dbc9f6 --- /dev/null +++ b/.jaiph/libs/jaiphlang/queue_test.py @@ -0,0 +1,114 @@ +#!/usr/bin/env python3 +"""Fixture tests for queue.py state + views. Run from any cwd.""" +import os, subprocess, sys, tempfile, textwrap + +HERE = os.path.dirname(os.path.abspath(__file__)) +_LOCAL = os.path.join(HERE, "queue.py") +_WS = os.path.join(os.environ.get("JAIPH_WORKSPACE", "."), ".jaiph", "libs", "jaiphlang", "queue.py") +QUEUE_PY = _LOCAL if os.path.isfile(_LOCAL) else _WS + + +def run_cmd(env, *args): + r = subprocess.run( + [sys.executable, QUEUE_PY, *args], + cwd=env["JAIPH_WORKSPACE"], + env={**os.environ, **env}, + capture_output=True, + text=True, + ) + return r + + +def main(): + with tempfile.TemporaryDirectory() as tmp: + ws = os.path.join(tmp, "ws") + os.makedirs(os.path.join(ws, ".jaiph")) + state = os.path.join(tmp, "hidden", "queue-state.md") + os.makedirs(os.path.dirname(state)) + env = {"JAIPH_WORKSPACE": ws, "JAIPH_QUEUE_STATE": state} + + qview = os.path.join(ws, "QUEUE.md") + with open(qview, "w", encoding="utf-8") as f: + f.write(textwrap.dedent("""\ + # Old preamble + + ## First #dev-ready + + Do the first thing. + + ## Second #dev-ready + + Do the second thing. + """)) + + r = run_cmd(env, "get_available") + assert r.returncode == 0, r.stderr + assert "First" in r.stdout + assert os.path.isfile(state), "bootstrap must write state" + + r = run_cmd(env, "mark", "First", "in-progress") + assert r.returncode == 0, r.stderr + + r = run_cmd(env, "get_available") + assert r.returncode == 0, r.stderr + assert "Second" in r.stdout + assert "First" not in r.stdout.split("\n")[0] + + with open(qview, "w", encoding="utf-8") as f: + f.write("# corrupted by agent\n") + + r = run_cmd(env, "get_by_header", "First") + assert r.returncode == 0, r.stderr + assert "Do the first thing" in r.stdout + + notes = os.path.join(tmp, "notes.md") + with open(notes, "w", encoding="utf-8") as f: + f.write("looks good") + r = run_cmd(env, "archive_to_done", "First", notes) + assert r.returncode == 0, r.stderr + + r = run_cmd(env, "get_by_header", "First") + assert r.returncode != 0 + + done = open(os.path.join(ws, "DONE.md"), encoding="utf-8").read() + assert "First" in done + assert "looks good" in done + assert "corrupted by agent" not in open(qview, encoding="utf-8").read() + assert "Second" in open(qview, encoding="utf-8").read() + + addf = os.path.join(tmp, "add.md") + with open(addf, "w", encoding="utf-8") as f: + f.write("## Third\n\nNo ready tag.\n") + r = run_cmd(env, "add", addf) + assert r.returncode == 0, r.stderr + r = run_cmd(env, "get_by_header", "Third") + assert r.returncode == 0 + assert "#dev-ready" not in r.stdout.split("\n")[0] + + r = run_cmd(env, "archive_to_done", "Second") + assert r.returncode == 0 + done2 = open(os.path.join(ws, "DONE.md"), encoding="utf-8").read() + assert "First" in done2 and "Second" in done2 + + r = run_cmd(env, "unmark", "Third", "in-progress") + assert r.returncode == 0, r.stderr + + other_state = os.path.join(tmp, "other-state.md") + env2 = {**env, "JAIPH_QUEUE_STATE": other_state} + with open(os.path.join(ws, "QUEUE.md"), "w", encoding="utf-8") as f: + f.write("## OnlyInOverride #dev-ready\n\nbody\n") + # Override path is empty: bootstrap from current QUEUE.md view. + r = run_cmd(env2, "get_available") + assert r.returncode == 0, r.stderr + assert "OnlyInOverride" in r.stdout + + print("queue_test.py: ok") + + +if __name__ == "__main__": + try: + main() + except Exception: + import traceback + traceback.print_exc() + sys.exit(1) diff --git a/.jaiph/product_owner.jh b/.jaiph/product_owner.jh new file mode 100644 index 00000000..6d2851e7 --- /dev/null +++ b/.jaiph/product_owner.jh @@ -0,0 +1,283 @@ +#!/usr/bin/env jaiph + +# +# Product owner — sole runtime writer of the task queue. +# Serve (HTTP + MCP on one port): +# ./start-product-owner.sh +# or: jaiph serve .jaiph/product_owner.jh +# Claude creds (ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN) come from +# the process env. They are not `use` keys — do not --env them. +# +import "jaiphlang/queue" as queue +import "./lib_common.jh" as common + +config { + agent.backend = "claude" + agent.model = "sonnet" + agent.claude_flags = "--permission-mode bypassPermissions" +} + +const po_file_rule = """ + You are given the queue as text in this prompt. That text is complete. + Do not read or write QUEUE.md, DONE.md, .jaiph/queue-state.md, + .jaiph/product-owner/, or any other queue file. Do not launch nested + jaiph workflows or agent sessions. +""" + +script first_line_task = ``` + local line + line="$(printf '%s\n' "$1" | awk 'NR==1 { print; exit }')" + line="${line#\#\# }" + line="$(printf '%s\n' "$line" | sed 's/ *#[A-Za-z0-9_-]*//g')" + printf '%s\n' "$line" +``` + +script git_evidence = ``` + git log -5 --oneline 2>/dev/null || true + echo "---" + git status --porcelain 2>/dev/null || true + echo "---" + git log -1 --format=fuller 2>/dev/null || true + echo "---" + git diff --stat HEAD~1 2>/dev/null || true +``` + +script format_task_file = ``` + title="$1" + tags="$2" + body="$3" + dest="$4" + out="## $title" + for t in $tags; do + t="${t#\#}" + [ -n "$t" ] && out="$out #$t" + done + { + printf '%s\n\n' "$out" + printf '%s\n' "$body" + } > "$dest" +``` + +# Claim one available task. Prefers the first #dev-ready that is not +# #in-progress; may claim a later one. HTTP 200 either way — empty queue is fail. +export def pick_task() { + const preferred = run queue.next_available() + const state = run queue.state_json() + const result = prompt """ + ${po_file_rule} + + You are the product owner. Pick one available task for an implementer. + + Available means #dev-ready and not #in-progress. Prefer the first + such task (given below). Pick a later available task only with a + reason: blocked preferred task, sequencing, or a dependency. + Do not invent a title. Do not pick #in-progress. + + Preferred (first available): + ${preferred} + + Full queue state (JSON): + ${state} + + Respond with title (exact task title, no #tags) and notes + (empty if you took the preferred task; required if you skipped it). + """ + returns "{ title: string, notes: string }" + + run common.arg_nonempty("${result.title}") catch (err) { + fail "pick_task requires a title" + } + const task = run queue.get_task_by_header("${result.title}") + const header = run first_line_task(task) + run queue.mark_task_in_progress(header) + return run queue.get_task_by_header(header) +} + +# Propose a task from a brief or markdown spec. Reject leaves the queue unchanged. +# Verdict is in the return text (accepted: / rejected:). HTTP stays 200. +# fail is only a broken call (empty spec, accepted with no title). +export def propose_task(spec) { + run common.arg_nonempty(spec) catch (err) { + fail "propose_task requires a non-empty spec" + } + const state = run queue.state_json() + const result = prompt """ + ${po_file_rule} + + You are the product owner. Decide whether to accept this proposal + onto the queue. + + Rules: standalone task, testable acceptance, no title clash, no + "see prior task", no stealing scope from an existing task. + Set tags to #dev-ready only if the spec is implementable as-is. + Otherwise leave tags empty. + + Current queue state (JSON): + ${state} + + Spec: + ${spec} + + Respond with verdict accepted or rejected, title, body (markdown + below the heading, no ## heading line), and tags (space-separated + tag names without #, or empty). + """ + returns "{ verdict: string, title: string, body: string, tags: string }" + + run common.str_equals("${result.verdict}", "accepted") catch (err) { + return "rejected: ${result.body}" + } + run common.arg_nonempty("${result.title}") catch (err) { + fail "accepted propose_task requires a title" + } + const tmpdir = run common.jaiph_tmp_dir() + run common.mkdir_p_simple(tmpdir) + const spec_file = "${tmpdir}/po_propose_task.md" + run format_task_file("${result.title}", "${result.tags}", "${result.body}", spec_file) + run queue.add_task_from_file(spec_file) + return "accepted: ${result.title}" +} + +# Report a claimed task as done. Archive or send back (needs-work). +# Verdict is in the return text. HTTP stays 200. fail is a broken call. +export def report_completed_task(header, notes) { + run common.arg_nonempty(header) catch (err) { + fail "report_completed_task requires a task header" + } + const task = run queue.get_task_by_header(header) + const state = run queue.state_json() + const evidence = run git_evidence() + const result = prompt """ + ${po_file_rule} + + You are the product owner. Decide if this task is done. + + Rough check only: git evidence vs acceptance criteria. Look for + missing behavior, missing tests, or an unmerged main. + + Task: + ${task} + + Caller notes: + ${notes} + + Git evidence: + ${evidence} + + Full queue state (JSON): + ${state} + + Respond with verdict accepted or needs-work, and notes + (empty if accepted and nothing to record; required if needs-work). + """ + returns "{ verdict: string, notes: string }" + + const tmpdir = run common.jaiph_tmp_dir() + run common.mkdir_p_simple(tmpdir) + const notes_file = "${tmpdir}/po_complete_notes.md" + run common.str_equals("${result.verdict}", "accepted") catch (err) { + run common.arg_nonempty("${result.notes}") catch (err) { + fail "needs-work requires notes" + } + const rest = run common.rest_lines_str(task) + const updated = "${rest}\n\n### PO notes\n\n${result.notes}\n" + run common.save_string_to_file(notes_file, updated) + const clean = run first_line_task(task) + run queue.set_task_description_from_file(clean, notes_file) + return "needs-work: ${result.notes}" + } + run common.save_string_to_file(notes_file, "${result.notes}") + const title = run first_line_task(task) + run queue.archive_to_done(title, notes_file) + return "accepted: ${title}" +} + +# Answer a question about a task. Persist Q and A on the task body. +export def task_details(header, question) { + run common.arg_nonempty(header) catch (err) { + fail "task_details requires a task header" + } + run common.arg_nonempty(question) catch (err) { + fail "task_details requires a question" + } + const task = run queue.get_task_by_header(header) + const state = run queue.state_json() + const result = prompt """ + ${po_file_rule} + + You are the product owner. Answer the question from the queue + state and the task text. You may use the codebase if needed. + + Task: + ${task} + + Question: + ${question} + + Full queue state (JSON): + ${state} + + Respond with verdict ok and notes equal to the answer. + """ + returns "{ verdict: string, notes: string }" + + run common.arg_nonempty("${result.notes}") catch (err) { + fail "task_details requires an answer" + } + const tmpdir = run common.jaiph_tmp_dir() + run common.mkdir_p_simple(tmpdir) + const body_file = "${tmpdir}/po_details_body.md" + const rest = run common.rest_lines_str(task) + const updated = "${rest}\n\n### PO note\n\nQ: ${question}\n\nA: ${result.notes}\n" + run common.save_string_to_file(body_file, updated) + const title = run first_line_task(task) + run queue.set_task_description_from_file(title, body_file) + return "${result.notes}" +} + +# Apply a requested change to a task after validation. Reject leaves the file unchanged. +# Verdict is in the return text. HTTP stays 200. +export def update_task(header, request) { + run common.arg_nonempty(header) catch (err) { + fail "update_task requires a task header" + } + run common.arg_nonempty(request) catch (err) { + fail "update_task requires a request" + } + const task = run queue.get_task_by_header(header) + const state = run queue.state_json() + const result = prompt """ + ${po_file_rule} + + You are the product owner. Decide whether to apply this change + to the task. Keep the task standalone, with testable acceptance, + and do not steal scope from other tasks. + + Task: + ${task} + + Request: + ${request} + + Full queue state (JSON): + ${state} + + If accepted, body is the full new task body (markdown below the + heading; no ## heading line). If rejected, body is the reason. + """ + returns "{ verdict: string, body: string }" + + run common.str_equals("${result.verdict}", "accepted") catch (err) { + return "rejected: ${result.body}" + } + run common.arg_nonempty("${result.body}") catch (err) { + fail "accepted update_task requires a body" + } + const tmpdir = run common.jaiph_tmp_dir() + run common.mkdir_p_simple(tmpdir) + const body_file = "${tmpdir}/po_update_body.md" + run common.save_string_to_file(body_file, "${result.body}") + const title = run first_line_task(task) + run queue.set_task_description_from_file(title, body_file) + return "accepted: ${title}" +} diff --git a/.jaiph/product_owner.test.jh b/.jaiph/product_owner.test.jh new file mode 100644 index 00000000..03ab1b9b --- /dev/null +++ b/.jaiph/product_owner.test.jh @@ -0,0 +1,154 @@ +#!/usr/bin/env jaiph + +import "./product_owner.jh" as po +import "jaiphlang/queue" as queue + +test "pick_task claims the preferred first available task" { + mock def queue.next_available() { + return "## First #dev-ready\n\nDo first." + } + mock def queue.state_json() { + return "{}" + } + mock def queue.mark_task_in_progress() { + return "" + } + mock def queue.get_task_by_header() { + return "## First #dev-ready #in-progress\n\nDo first." + } + mock prompt { + _ => "{\"title\":\"First\",\"notes\":\"\"}" + } + const out = run po.pick_task() + expect_contain out "First" + expect_contain out "in-progress" +} + +test "pick_task may claim a later available task" { + mock def queue.next_available() { + return "## First #dev-ready\n\nDo first." + } + mock def queue.state_json() { + return "{}" + } + mock def queue.mark_task_in_progress() { + return "" + } + mock def queue.get_task_by_header() { + return "## Second #dev-ready #in-progress\n\nDo second." + } + mock prompt { + _ => "{\"title\":\"Second\",\"notes\":\"First is blocked on a merge\"}" + } + const out = run po.pick_task() + expect_contain out "Second" + expect_contain out "in-progress" +} + +test "pick_task fails when the queue has no available task" { + mock def queue.next_available() { + fail "no #dev-ready task is available" + } + const out = run po.pick_task() allow_failure + expect_contain out "no #dev-ready task is available" +} + +test "propose_task reject does not write" { + mock def queue.state_json() { + return "{}" + } + mock def queue.add_task_from_file() { + fail "SHOULD_NOT_WRITE" + } + mock prompt { + _ => "{\"verdict\":\"rejected\",\"title\":\"\",\"body\":\"needs acceptance\",\"tags\":\"\"}" + } + const out = run po.propose_task("vague idea") + expect_contain out "rejected" +} + +test "propose_task accept writes via add_task_from_file" { + mock def queue.state_json() { + return "{}" + } + mock def queue.add_task_from_file() { + return "wrote" + } + mock prompt { + _ => "{\"verdict\":\"accepted\",\"title\":\"New Task\",\"body\":\"Do it.\",\"tags\":\"dev-ready\"}" + } + const out = run po.propose_task("## New Task\n\nDo it.") + expect_contain out "accepted: New Task" +} + +test "report_completed_task needs-work does not archive" { + mock def queue.get_task_by_header() { + return "## Claimed #dev-ready #in-progress\n\nBuild it." + } + mock def queue.state_json() { + return "{}" + } + mock def queue.archive_to_done() { + fail "SHOULD_NOT_ARCHIVE" + } + mock def queue.set_task_description_from_file() { + return "updated" + } + mock prompt { + _ => "{\"verdict\":\"needs-work\",\"notes\":\"merge main\"}" + } + const out = run po.report_completed_task("Claimed", "") + expect_contain out "needs-work" + expect_contain out "merge main" +} + +test "report_completed_task accepted archives" { + mock def queue.get_task_by_header() { + return "## Claimed #dev-ready #in-progress\n\nBuild it." + } + mock def queue.state_json() { + return "{}" + } + mock def queue.archive_to_done() { + return "archived" + } + mock prompt { + _ => "{\"verdict\":\"accepted\",\"notes\":\"ok\"}" + } + const out = run po.report_completed_task("Claimed", "done") + expect_contain out "accepted: Claimed" +} + +test "update_task reject does not write" { + mock def queue.get_task_by_header() { + return "## Claimed #dev-ready\n\nBuild it." + } + mock def queue.state_json() { + return "{}" + } + mock def queue.set_task_description_from_file() { + fail "SHOULD_NOT_WRITE" + } + mock prompt { + _ => "{\"verdict\":\"rejected\",\"body\":\"steals later task\"}" + } + const out = run po.update_task("Claimed", "also do the next task") + expect_contain out "rejected" +} + +test "task_details persists an answer" { + mock def queue.get_task_by_header() { + return "## Claimed #dev-ready\n\nBuild it." + } + mock def queue.state_json() { + return "{}" + } + mock def queue.set_task_description_from_file() { + return "updated" + } + mock prompt { + _ => "{\"verdict\":\"ok\",\"notes\":\"use the public entry\"}" + } + const out = run po.task_details("Claimed", "which file?") + expect_contain out "use the public entry" +} diff --git a/.jaiph/queue_ops.test.jh b/.jaiph/queue_ops.test.jh new file mode 100644 index 00000000..28a3de81 --- /dev/null +++ b/.jaiph/queue_ops.test.jh @@ -0,0 +1,8 @@ +#!/usr/bin/env jaiph + +import "./libs/jaiphlang/queue_selftest.jh" as qs + +test "queue state ignores QUEUE.md after bootstrap and archives to DONE" { + const out = run qs.main() + expect_contain out "queue_test.py: ok" +} diff --git a/CHANGELOG.md b/CHANGELOG.md index fba39767..64255c83 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,13 @@ ## All changes +- **Factory — product owner:** `.jaiph/product_owner.jh` exposes `propose_task`, `pick_task`, `report_completed_task`, `task_details`, and `update_task` for `jaiph serve` / MCP. `pick_task` prefers the first `#dev-ready` task that is not `#in-progress` and may claim a later one. A reject or needs-work is a successful return (`accepted:` / `rejected:` / `needs-work:`), not an HTTP error. Canonical queue state is `${JAIPH_QUEUE_STATE}` or `.jaiph/queue-state.md`. `QUEUE.md` and `DONE.md` are generated views. `./start-product-owner.sh` runs serve in Docker with volume `jaiph-po-state` at `.jaiph/product-owner/` (queue + runs); host `.jaiph/runs` is tmpfs-masked and forwards `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN` (no `--env`; those are backend credentials, not `use` grants). Default is `--allow-anonymous`; set `JAIPH_SERVE_TOKEN` to require a bearer. Tests: `.jaiph/product_owner.test.jh`, `.jaiph/queue_ops.test.jh`. +- **CLI — `jaiph serve`:** `--allow-anonymous` also permits a non-loopback bind. Without the flag, a bind with no token and no OIDC is still a startup error. Docs: [Serve defs over HTTP](docs/serve.md), [CLI](docs/cli.md). Tests: `integration/serve-server.test.ts`. +- **Breaking — CLI — `jaiph serve`:** HTTP paths drop `/v1`. Invoke is `POST /{name}`. List and inspect are `GET /defs`, `GET /runs`, `GET /runs/{id}` (events, artifacts, cancel stay under `/runs/{id}/…`). `Location` is `/runs/{id}`. `POST /v1/defs/{name}/runs` and every `/v1/…` path are gone. OpenAPI and `/docs` match. Docs: [Serve defs over HTTP](docs/serve.md), [CLI](docs/cli.md). Tests: `src/cli/serve/handler.test.ts`, `src/cli/serve/openapi.test.ts`. +- **CLI — `jaiph serve` / `jaiph mcp`:** startup logs `loading module graph…` / `module graph ready in Nms` and `reconstructed N run(s) … in Nms`. Operator log writes with `writeSync` so Docker (no TTY) does not buffer the lines. Docs: [CLI](docs/cli.md). +- **Docs:** `./docs/build-jaiph-dev-image.sh` compiles the linux standalones and tags a local `ghcr.io/jaiphlang/jaiph-runtime`. `docs/install-from-local.sh` still installs the host binary only. +- **Release:** each `v*` tag and `nightly` push publishes `ghcr.io//jaiph-runtime` (`:` / `:latest` on stable, `:nightly` on nightly) from `runtime/Dockerfile`. The image has `jaiph`, `python3`, `git`, and `curl` as uid `10001`. It is not a GitHub Release asset. Docs: [Deploy jaiph](docs/deploy.md). Tests: `integration/release-workflow.test.ts`. + # 0.14.0 ## Summary diff --git a/DONE.md b/DONE.md new file mode 100644 index 00000000..f85b8a6f --- /dev/null +++ b/DONE.md @@ -0,0 +1,4 @@ +# Done + +Append-only archive. Generated view. Do not edit. +Each section was accepted by the product-owner `report_completed_task` def. diff --git a/QUEUE.md b/QUEUE.md index 31305541..31b53104 100644 --- a/QUEUE.md +++ b/QUEUE.md @@ -1,15 +1,109 @@ # Jaiph Improvement Queue (Hard Rewrite Track) +This file is a generated view. Do not edit it. Do not agent-edit it. +Use the product-owner defs (`propose_task`, `update_task`, `pick_task`, +`report_completed_task`, `task_details`) via `jaiph serve` / MCP. + Process rules: -1. Tasks are executed top-to-bottom. -2. The first `##` section is always the current task. -3. Task that is ready for implementation is marked with `#dev-ready` at the end of the header. -4. When a task is completed, **orchestration** removes that section (`queue.remove_completed_task` in `.jaiph/engineer.jh`). Agents and humans implementing a task must **not** edit `QUEUE.md` to delete or rewrite the current task — leave queue updates to the workflow. -5. Every task must be standalone: no hidden assumptions, no "read prior task" dependency. -6. This queue assumes **hard rewrite semantics**: - * breaking changes are allowed, - * backward compatibility is **not** a design goal unless a task explicitly says otherwise. -7. **Acceptance criteria are non-negotiable.** A task is not done until every acceptance bullet is verified by a test that fails when the contract is violated. "It works on my machine" or "the existing tests pass" is not acceptance. - -*** +1. `pick_task` prefers the first `#dev-ready` task that is not `#in-progress`. + The product owner may claim a later available task instead. +2. The first `##` section is the preferred next task in this view. +3. `#dev-ready` means ready to implement. `#in-progress` means claimed by `pick_task`. +4. Runtime mutations go through the product-owner defs only. +5. Every task must be standalone: no hidden assumptions, no "read prior task". +6. Hard rewrite semantics: breaking changes are allowed unless a task says otherwise. +7. Acceptance criteria are non-negotiable. A task is not done until every + acceptance bullet is verified by a test that fails when the contract is violated. + +## Point ensure_ci_passes at the existing step capture; do not pass recover bytes through argv #dev-ready + +Context: `.jaiph/ensure_ci_passes.jh` runs `npm_run_test_ci` under `recover (failure)`. On failure the recover body calls `common.save_string_to_file(ci_log_file, failure)`. `save_string_to_file` (`.jaiph/lib_common.jh`) takes content as `$2` / `sys.argv[2]`. Scripts have no stdin channel. The same bytes are already on disk as the failed step's stdout capture under `JAIPH_RUN_DIR` (`NNNNNN-script__npm_run_test_ci.out`, written incrementally by `executeManagedStep` in `src/runtime/kernel/node-workflow-runtime.ts`). + +Problem: a CI log larger than the OS spawn argument limit (macOS `ARG_MAX` is about 1 MB for args + environment; jaiph's sterile env still occupies part of that) never reaches the recover prompt. The write is redundant: recover rematerializes a file that already exists. Playwright `test:ci` output around 1.3 MB is a normal case, not an edge case. + +Remediation — implement exactly this: + +1. In `.jaiph/ensure_ci_passes.jh`, delete the `run common.save_string_to_file(ci_log_file, failure)` call. Do not pass the `failure` binding into any script argument. +2. Resolve `JAIPH_RUN_DIR` with a tiny script that prints only that env value (no content argv). Point the recover prompt at the existing capture: `${run_dir}/*-script__npm_run_test_ci.out`. Tell the agent to `tail` that file. Drop the `.jaiph/tmp/ensure_ci_passes.last.log` copy, or keep a dest path only if a script copies via `cp` from the capture glob (path argv only). +3. Replace `assert_nonempty_file_or_fail(ci_log_file)` with a check that the capture glob exists and is non-empty, still using only path argv. +4. Do not change recover binding semantics, `save_string_to_file`, or the runtime in this task. + +### Acceptance criteria + +- `.jaiph/ensure_ci_passes.jh` has no `save_string_to_file` (or other script) call whose argument is the recover binding. +- The recover prompt names the `JAIPH_RUN_DIR` capture (`*-script__npm_run_test_ci.out`) as the log to read. +- A script in that recover body receives at most short paths / names as argv, never the failed step's stdout. +- `npm run build` and `npm test` pass. + +## Spawn and exec failures are failed steps; always emit STEP_END and RUN_END #dev-ready + +Context: `spawnAndCapture` (`src/runtime/kernel/node-workflow-runtime.ts`) calls `_scriptSpawn.spawn(command, args, …)` inside a Promise executor with no try/catch. The `'error'` handler settles `status: 1`, but a synchronous throw from `spawn` (including `E2BIG` when argv + env exceed `ARG_MAX`) rejects the Promise. `executeManagedStep` awaits `fn(stepIo)` and only `finally`-stops the idle watchdog; on throw it never writes `STEP_END`. `runRoot` awaits `executeDef` then emits `RUN_END`; on throw it never emits `RUN_END`. `runWorkflowRunner` (`.catch` in `src/runtime/kernel/node-workflow-runner.ts`) prints `jaiph node runner: …` and `process.exit(1)` with no journal close. Observed: `STEP_START` for the oversized script, empty `.out`/`.err`, no `STEP_END`, no `RUN_END`, heartbeat stops. `recover_limit` never applies because recover never sees a failed step. + +Problem: `run save_string_to_file(path, big)` after a large script failure (CI log ≳ 1 MB on macOS) aborts the jaiph process instead of failing the step. From outside the run vanished; the child never started. + +Remediation — implement exactly this: + +1. Wrap `spawn` in `spawnAndCapture` so a synchronous throw settles the same way as the `'error'` event: `status: 1`, stderr text, Promise resolves (never rejects). +2. Map `E2BIG` (throw or `'error'`) to a stable diagnostic on stderr: `E_ARGV_TOO_LARGE: bytes (ARG_MAX …)` (include the attempted argv+env size). Other spawn failures stay `status: 1` with `errText` (keep the existing ENOENT-interpreter message). +3. `executeManagedStep`: if `fn` throws, convert to `StepResult` `{ status: 1, output: "", error: }` and still write capture files + `STEP_END`. `result` must always be defined before emit. +4. `runRoot`: emit `RUN_END` and stop the heartbeat in a `finally`, including when `executeDef` throws. A vanished process is not a handleable error. +5. Do not add a new language form. Do not change recover binding contents. + +### Acceptance criteria + +- Unit test (spawn seam `_scriptSpawn`): `spawn` throwing `E2BIG` resolves the script step as `status: 1`; stderr matches `E_ARGV_TOO_LARGE` and includes a byte count; the Promise does not reject. +- Unit test: `spawn` emitting `'error'` with `code: "E2BIG"` is the same failed-step contract (not a thrown run). +- Unit test: `executeManagedStep` / `runRoot` still emit `STEP_END` (status 1) and `RUN_END` when the inner `fn` throws a generic `Error`. Today's control flow (no `STEP_END` / `RUN_END` on throw) must fail this test. +- A `run script(huge)` whose argv would exceed `ARG_MAX` does not reject `runRoot`. After the step, `run_summary.jsonl` contains `STEP_END` for that script and a terminal `RUN_END`. Prefer the spawn mock; a live ≳1 MB argv is optional and must not be the only coverage. +- `recover` on a later step still runs when the oversized spawn is itself the failed `run` (the step ends `status: 1`, so `recover_limit` applies). Add a runtime or e2e test that proves recover body runs after a mocked `E2BIG`. +- `npm run build`, `npm test`, and `npm run test:e2e` pass. + +## recover and catch bind the failed step capture path, not the output bytes #dev-ready + +Context: `runRecoverBody` (`src/runtime/kernel/node-workflow-runtime.ts`) sets the recover/catch binding to `` `${lastResult.output}${lastResult.error}` `` — the full merged stdout+stderr string. Docs (`docs/language.md` § catch/recover, `docs/jaiph-skill.md`, `docs/grammar.md`) say the same. The failed step already has those bytes on disk: `executeManagedStep` writes `JAIPH_RUN_DIR/NNNNNN-__.out` and `.err` incrementally, then rewrites them at `STEP_END`. Call sites such as `.jaiph/ensure_ci_passes.jh` then pass that string into a script as argv (`save_string_to_file(path, failure)`), which hits `ARG_MAX` on a ~1 MB+ CI log. `run foo() > file` is `E_PARSE` (`src/parse/core.ts`); there is no `capture_to` form. Leave that ban in place. + +Problem: the recover binding duplicates a file the runtime already wrote, then forces the next script to put those bytes on `execve`. The prompt already tells the agent to read a file. The binding should be that file's path. + +Remediation — implement exactly this (breaking): + +1. `catch (name)` and `recover (name)` bind `name` to the **absolute path** of the failed step's stdout capture (`out_file` / `NNNNNN-*.out` under `JAIPH_RUN_DIR`). Stderr stays in the sibling `.err` (same seq prefix). Do not concatenate stdout+stderr into the binding and do not copy the capture to a second file unless a test needs a merged witness — prefer `cp`/`cat` in the test script from the bound path. +2. Plumb `outFile` (and `errFile` if needed) on `StepResult` from `executeManagedStep` so every `run` target that can carry `catch`/`recover` (named script, inline script, def, async branch) supplies a real path. A spawn that produced no stdout still binds the `.out` path (file exists; may be empty). Spawn diagnostics live in `.err`. +3. Update docs to state the binding is a path: `docs/language.md` (catch and recover), `docs/jaiph-skill.md` (Failure handling), `docs/grammar.md` if it claims "merged stdout+stderr". Examples that interpolate `${err}` as log *content* should treat it as a path (`logerr "failed; see ${err}"` is fine). +4. Update in-repo callers that treat the binding as content: `e2e/tests/101_ensure_recover_output_contract.sh` (and any sibling that `printf`s `$1` as the payload), `.jaiph/ensure_ci_passes.jh` if it still forwards the binding to a script. `.jaiph/gh_ci_passes.jh` `log "… ${failure}"` becomes a path line — acceptable. `examples/recover_loop.jh` does not use the binding as content. +5. Do not add `run foo() > file` or `capture_to`. Do not add a stdin clause in this task. + +### Acceptance criteria + +- Runtime test: `run failing_script() catch (failure) { … }` binds `failure` to an absolute path; `readFileSync(failure)` equals the script's stdout; sibling `.err` equals stderr. Today's "binding === Hello\\nOops" content contract must fail and be rewritten. +- Runtime or e2e: a script whose stdout is > 1 MB fails; the recover/catch binding is a path whose file size matches that stdout; no recover-body script receives the bytes as argv. +- `e2e/tests/101_ensure_recover_output_contract.sh` (and `102_engineer_recover_contract.sh` if it assumes content) assert path + file contents, not the binding string itself being the log. +- Docs listed above say the binding is the capture path, not merged stdout+stderr text. +- `npm run build`, `npm test`, and `npm run test:e2e` pass. + +## Scripts accept a large payload on stdin; save_string_to_file reads stdin #dev-ready + +Context: Script arguments are argv only (`$1` / `sys.argv`). That is the documented contract (`docs/jaiph-skill.md`, `docs/language.md`). `run foo() > file` is `E_PARSE`. `.jaiph/lib_common.jh` `save_string_to_file` already documents that content travels through argv and is subject to `ARG_MAX` (~1 MB on macOS). `spawnAndCapture` uses `stdio: ["ignore", "pipe", "pipe"]` — stdin is discarded. Recover path-binding does not help `run process(huge_json)` or any other large string that is not already a capture file. + +Problem: any `run script(big)` whose encoded argv + env exceeds the OS limit fails at `execve`. There is no supported way to hand a large string to a script. + +Remediation — implement exactly this: + +1. Language: optional `stdin ` on a standalone `run` of a **script** (named or inline), after `()` and before `catch` / `recover`: + + `run save_string_to_file(path) stdin content` + + `stdin` is rejected (`E_PARSE` / `E_VALIDATE`) on `run` of a def, on `run async`, and on `run` without a script target. `` is a normal string expr (bare ident, quoted, interpolation). Trailing `>` / `>>` / `|` / `&` stay `E_PARSE`. +2. Runtime: spawn that step with stdin piped; write the evaluated expr bytes (UTF-8) to the child's stdin and end the stream. Those bytes must not appear in argv. `spawnAndCapture` grows a stdin parameter; default remains `ignore` when the clause is absent. +3. Change `.jaiph/lib_common.jh` `save_string_to_file` to `path = sys.argv[1]; content = sys.stdin.read()`. Update every in-repo call to `run common.save_string_to_file(path) stdin content` (`.jaiph/architect_review.jh` and any other caller). Comment the new contract; drop the ARG_MAX warning for this helper. +4. Docs: `docs/language.md` (`run` / scripts), `docs/jaiph-skill.md` (arguments), `docs/grammar.md` (`run_stmt`). Editor grammars (VS Code TextMate, Zed/Tree-sitter) must highlight `stdin` as a `run` clause, not as a shell redirect. +5. Keep argv as the default small-arg channel. Do not auto-promote large argv to stdin. + +### Acceptance criteria + +- Parse/validate tests: `run foo(a) stdin body` is accepted when `foo` is a script; `run someDef() stdin x` is `E_VALIDATE` (or `E_PARSE` if you reject earlier); `run foo() > file` remains `E_PARSE`; `run async foo() stdin x` is rejected. +- Runtime test: `run echo_stdin() stdin payload` with `script echo_stdin = \`cat\`` returns `payload`; spawn argv does not contain `payload` (assert via `_scriptSpawn` spy). +- Runtime test: a stdin payload larger than 1 MB is written in full and the step exits 0. Today's argv path cannot pass this. +- `save_string_to_file` / architect_review call sites compile under the new helper contract (path argv + stdin body). +- Formatter round-trips `run name(args) stdin expr`. +- `npm run build`, `npm test`, `npm run test:e2e`, and editor grammar tests (`plugins/vscode`, `plugins/zed` as already wired) pass. diff --git a/docs/architecture.md b/docs/architecture.md index 2ca89a87..ba507d1b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -183,7 +183,7 @@ Every line written to `run_summary.jsonl` by `RuntimeEventEmitter` carries a `pr **Key storage outside the run directory (finding M-3).** The key is **not** written into the run directory, which is agent-writable (`$JAIPH_RUN_DIR` for script steps). Storing it there let a workflow's first script step `mkdir "$JAIPH_RUN_DIR/.chain-key"` to squat the path so the host's write threw and was swallowed, then rewrite the journal freely with no integrity failure surfaced. Instead the key lives in an operator-side store: `resolveAuditKeyStore` (`emit.ts`), default `~/.jaiph/audit-keys`, override `JAIPH_AUDIT_KEY_DIR`. Each run gets one entry directory `/` holding the secret `key` file; the directory's existence is the durable "this run was keyed" marker. Persistence is a **hard error**: `writeChainKey` no longer swallows failures, and it creates the marker directory before the key, so even a partial write leaves the run marked keyed-but-keyless (which fails closed below) rather than silently unverifiable. -**Verification at read/export boundaries.** `verifyRunSummaryChain(filePath, key, opts?)` walks each line, checks `prev_hash` against the recomputed keyed digest, and returns `{ ok: false, error }` at the first broken link (a missing/unreadable journal is a failure, not a silent pass). With `opts.requireTerminal` it additionally requires the journal to **end with the `RUN_END` terminal marker** (`TERMINAL_EVENT_TYPE`) — the chain commits to prefix integrity but not to length, so deleting the last *K* lines of a completed journal leaves a shorter-but-valid chain that would otherwise verify; requiring the terminal marker rejects any post-terminal tail truncation (finding L-3). `verifyRunJournal(runDir)` wraps it (always with `requireTerminal`, since a key is persisted only once the run is terminal): it looks up the run's store entry and returns `{ verified: false, ok: true }` when the run has **no** entry (an unkeyed/legacy run that cannot be verified — never blocked), `{ verified: true, ok: false }` when the run **was** keyed but the key is missing at verification time (**fail closed** — a keyed run whose key vanished must not downgrade to "not verified" and let a tampered journal through), or `{ verified: true, ok }` with the chain result otherwise. Every read/export boundary hard-fails when `verified && !ok`: run listing (`loadPersistedRuns` marks the run `failed` with `TAMPERED_RESULT_TEXT`), `GET /v1/runs/{id}/events` (`409 E_TAMPERED`), and OTLP/Sentry export (skip + warn, never POST a tampered journal). +**Verification at read/export boundaries.** `verifyRunSummaryChain(filePath, key, opts?)` walks each line, checks `prev_hash` against the recomputed keyed digest, and returns `{ ok: false, error }` at the first broken link (a missing/unreadable journal is a failure, not a silent pass). With `opts.requireTerminal` it additionally requires the journal to **end with the `RUN_END` terminal marker** (`TERMINAL_EVENT_TYPE`) — the chain commits to prefix integrity but not to length, so deleting the last *K* lines of a completed journal leaves a shorter-but-valid chain that would otherwise verify; requiring the terminal marker rejects any post-terminal tail truncation (finding L-3). `verifyRunJournal(runDir)` wraps it (always with `requireTerminal`, since a key is persisted only once the run is terminal): it looks up the run's store entry and returns `{ verified: false, ok: true }` when the run has **no** entry (an unkeyed/legacy run that cannot be verified — never blocked), `{ verified: true, ok: false }` when the run **was** keyed but the key is missing at verification time (**fail closed** — a keyed run whose key vanished must not downgrade to "not verified" and let a tampered journal through), or `{ verified: true, ok }` with the chain result otherwise. Every read/export boundary hard-fails when `verified && !ok`: run listing (`loadPersistedRuns` marks the run `failed` with `TAMPERED_RESULT_TEXT`), `GET /runs/{id}/events` (`409 E_TAMPERED`), and OTLP/Sentry export (skip + warn, never POST a tampered journal). **Scope of the guarantee.** A workflow script step cannot read the key or the journal path from its env, and cannot alter the journal in any way that verifies — any rewrite or omitted line is rejected, and any truncation *during* the run is caught because the kernel keeps appending under the pre-truncation head. A *post-run* clean truncation of a completed journal's tail is also rejected: the terminal-marker check (finding L-3) fails a keyed journal that no longer ends with `RUN_END`. Because a `.jh` host run and its `script` steps execute under the same OS user, a hash chain still cannot defend against a post-run same-user process that deletes the run's store entry, which makes the run unverifiable (`verified:false`) rather than a detectable tamper. A workflow cannot read the key from its env, and the key store lives outside the run directory. @@ -199,7 +199,7 @@ Before `RuntimeEventEmitter` writes an event line to `run_summary.jsonl`, it red The rule covers backend API keys such as `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, and `CURSOR_API_KEY`. -The same credential rule lives in one shared helper, **`redactCredentials`** (`src/runtime/kernel/redact.ts`). The helper is also the redaction boundary for returned call results. `composeResult` (`src/cli/shared/workflow-call.ts`) redacts a failed call's diagnostic capture (the failed-step detail, the raw stderr and stdout, and the collected `log` messages) before it becomes `jaiph serve`'s `result_text` or a `jaiph mcp` tool result. A successful workflow's return value is intentional API output rather than diagnostic capture, so it is returned verbatim. The journal that `redactCredentials` produces is what the OTLP export (`otlp.ts`), the Sentry export (`sentry.ts`), and `GET /v1/runs/{id}/events` (`handler.ts`) read back verbatim, so broadening the rule tightens all four surfaces at once. +The same credential rule lives in one shared helper, **`redactCredentials`** (`src/runtime/kernel/redact.ts`). The helper is also the redaction boundary for returned call results. `composeResult` (`src/cli/shared/workflow-call.ts`) redacts a failed call's diagnostic capture (the failed-step detail, the raw stderr and stdout, and the collected `log` messages) before it becomes `jaiph serve`'s `result_text` or a `jaiph mcp` tool result. A successful workflow's return value is intentional API output rather than diagnostic capture, so it is returned verbatim. The journal that `redactCredentials` produces is what the OTLP export (`otlp.ts`), the Sentry export (`sentry.ts`), and `GET /runs/{id}/events` (`handler.ts`) read back verbatim, so broadening the rule tightens all four surfaces at once. **Explicit non-guarantee.** Redaction is literal-substring replacement of the value and the base64 / base64url / hex / URL-encoded encodings listed above. A secret transformed some other way — split across two output chunks, JSON-string-escaped, gzipped, re-chunked, or embedded as the password inside an opaque connection string (e.g. a `DATABASE_URL`, whose key name does not itself look like a credential) — is **not** guaranteed to be redacted. Beyond the two redaction boundaries (journal copies and returned call results), redaction is not applied at all: the per-step raw capture files (`%06d-.out` / `.err`) are streamed to disk verbatim. Treat them, and the run directory as a whole, as sensitive. Redaction also covers only the durable journal copy of each event, not the live `__JAIPH_EVENT__` progress stream on the runner's stderr that the progress UI and hooks read, so a hook that reads a `LOG` or `STEP_END` line off that stream sees the field as the workflow authored it. diff --git a/docs/build-jaiph-dev-image.sh b/docs/build-jaiph-dev-image.sh new file mode 100755 index 00000000..8e6f1679 --- /dev/null +++ b/docs/build-jaiph-dev-image.sh @@ -0,0 +1,50 @@ +#!/usr/bin/env bash +# Build a local linux runner image from this clone. +# docs/install-from-local.sh does not do this — that script installs the host +# binary only. This compiles the linux standalones and docker-builds +# runtime/Dockerfile. +# +# Requires: npm, bun, docker +# +# Usage: +# ./docs/build-jaiph-dev-image.sh +# JAIPH_DEV_IMAGE=my/jaiph ./docs/build-jaiph-dev-image.sh +# +# Tags ${JAIPH_DEV_IMAGE:-ghcr.io/jaiphlang/jaiph-runtime} and :latest +# for linux/${host-arch}. The Dockerfile COPYs both linux binaries. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)" +IMAGE="${JAIPH_DEV_IMAGE:-ghcr.io/jaiphlang/jaiph-runtime}" + +case "$(uname -m)" in + arm64|aarch64) DOCKER_ARCH=arm64 ;; + x86_64|amd64) DOCKER_ARCH=amd64 ;; + *) + echo "build-jaiph-dev-image: unsupported uname -m: $(uname -m)" >&2 + exit 1 + ;; +esac + +for cmd in npm bun docker; do + if ! command -v "${cmd}" >/dev/null 2>&1; then + echo "build-jaiph-dev-image: ${cmd} is required" >&2 + exit 1 + fi +done + +cd "${REPO_ROOT}" +npm run build +bun build --compile --target=bun-linux-arm64 ./src/cli.ts --outfile runtime/jaiph-linux-arm64 +bun build --compile --target=bun-linux-x64 ./src/cli.ts --outfile runtime/jaiph-linux-x64 +chmod +x runtime/jaiph-linux-arm64 runtime/jaiph-linux-x64 + +docker build --platform "linux/${DOCKER_ARCH}" \ + -f runtime/Dockerfile \ + -t "${IMAGE}:latest" \ + -t "${IMAGE}" \ + runtime/ + +echo "Tagged ${IMAGE} and ${IMAGE}:latest (linux/${DOCKER_ARCH})" diff --git a/docs/cli.md b/docs/cli.md index b6aaf970..da8b0f20 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -381,16 +381,16 @@ jaiph serve [--host ] [--port ] [--workspace ] [--allow-anonymous] | Flag | Argument | Effect | |---|---|---| -| `--host` | `` | Listen address (default `127.0.0.1`). Binding a non-loopback host with no authentication (neither `JAIPH_SERVE_TOKEN` nor OIDC configured) aborts startup, even with `--allow-anonymous`. | +| `--host` | `` | Listen address (default `127.0.0.1`). Binding with no authentication (neither `JAIPH_SERVE_TOKEN` nor OIDC) aborts startup unless `--allow-anonymous` is passed. | | `--port` | `` | Listen port (default `5247`). `0` picks a free port. | -| `--allow-anonymous` | — | Explicit opt-in to run open with no authentication on loopback. Without it, a loopback bind with no `JAIPH_SERVE_TOKEN` and no OIDC aborts startup, because anonymous mode authorizes every local principal with all capabilities over all runs (loopback guards the network, not other local users — finding M-2). For a single-user workstation only; shared hosts must set `JAIPH_SERVE_TOKEN` or configure OIDC. When passed, the server prints a startup warning that it is open to all local principals. Ignored (no-op) when a token or OIDC is configured, and it never permits a non-loopback bind. | +| `--allow-anonymous` | — | Explicit opt-in to run open with no authentication. Without it, any bind with no `JAIPH_SERVE_TOKEN` and no OIDC aborts startup. Anonymous mode authorizes every caller with all capabilities over all runs. The flag permits loopback and non-loopback binds (Docker must listen on `0.0.0.0` for published ports). When passed, the server prints a startup warning. Ignored (no-op) when a token or OIDC is configured. Shared or network-exposed hosts must set `JAIPH_SERVE_TOKEN` or configure OIDC. | | `--workspace` | `` | Workspace root for import resolution (default: auto-detected). | | `--env` | `KEY=VALUE` or `KEY` | Same per-key passthrough as `jaiph run --env`, resolved once at startup and applied to every run for the server's lifetime. | | `-h`, `--help` | — | Print the subcommand usage and exit `0`. | Flags that belong to another command (for example `--raw` or `--target`) are usage errors naming the owning command — never silently ignored. Precedence across layers is the shared execution-policy order: CLI flags > `JAIPH_*` env vars > module config metadata > defaults (see [Environment variables — Precedence](env-vars.md#precedence)). -Startup mirrors `jaiph mcp`: graph load + `collectDiagnostics` (diagnostics to stderr, exit `1`), credential pre-flight as warnings, and a host-execution notice. All logs go to stderr. Startup prints a line with the listen URL, the `/docs` and `/mcp` URLs, and the exposed-def count, followed by an authentication-mode line, a memory-bounds line, and, when terminal runs were rebuilt from disk, a line reporting how many were reconstructed. Per-run operator lines (a start line `Running … run_id=` and an end line `Finished … status=… elapsed_ms=…`) and the optional log mirror follow the same stderr-only operator-log contract as `jaiph mcp`, and HTTP response bodies stay API payloads. See [Operator log (stderr)](#operator-log-stderr) above, and `JAIPH_SERVER_LOG` and `JAIPH_SERVER_LOG_RUNS` in [Environment variables](env-vars.md). +Startup mirrors `jaiph mcp`: graph load + `collectDiagnostics` (diagnostics to stderr, exit `1`), credential pre-flight as warnings, and a host-execution notice. All logs go to stderr. Startup prints timed lines for module-graph load and run reconstruction (`… in Nms`), then a line with the listen URL, the `/docs` and `/mcp` URLs, and the exposed-def count, followed by an authentication-mode line and a memory-bounds line. Per-run operator lines (a start line `Running … run_id=` and an end line `Finished … status=… elapsed_ms=…`) and the optional log mirror follow the same stderr-only operator-log contract as `jaiph mcp`, and HTTP response bodies stay API payloads. See [Operator log (stderr)](#operator-log-stderr) above, and `JAIPH_SERVER_LOG` and `JAIPH_SERVER_LOG_RUNS` in [Environment variables](env-vars.md). ### Endpoints @@ -403,23 +403,23 @@ The **Cap.** column names the capability an authenticated principal must hold to | `GET /openapi.json` | none | OpenAPI 3.1 document, regenerated per request (hot reload needs no cache invalidation). `404` when `JAIPH_SERVE_EXPOSE_DOCS=false`. | | `GET /docs` | none | Self-contained Swagger UI shell. The pinned `swagger-ui-dist` assets are embedded in the binary and served from same-origin `/docs/*` paths, so it needs no browser internet access. `404` when `JAIPH_SERVE_EXPOSE_DOCS=false`. | | `GET /docs/swagger-ui-bundle.js`, `GET /docs/swagger-ui.css` | none | The embedded Swagger UI assets, each stamped with a `sha384` Subresource Integrity hash over the served bytes. `404` when `JAIPH_SERVE_EXPOSE_DOCS=false`. | -| `GET /v1/defs` | `inspect` | `{defs: [{name, description, params}]}`. | -| `POST /v1/defs/{name}/runs` | `invoke` | Start a run. Default `202` + `Location: /v1/runs/{id}`; `?wait=true` blocks for the terminal `200`. Send an `Idempotency-Key` header (scoped to the authenticated principal + def) to make retries safe: an identical repeat returns the original run (`200`, no second spawn); a reused key with different arguments is `409 E_IDEMPOTENCY_CONFLICT` and spawns nothing. | -| `GET /v1/runs` | `inspect` | Runs started by this process **plus runs reconstructed from disk on restart**, newest first, scoped to the caller's own runs (all runs for a static/open principal). Paginated: `?limit` (default `100`, clamped to `1000`), `?offset` (default `0`). Response is `{runs, total, limit, offset}` and never unbounded. | -| `GET /v1/runs/{id}` | `inspect` | The run object. `404` unknown (a run the principal does not own is indistinguishable from nonexistent). | -| `GET /v1/runs/{id}/events` | `inspect` | The run's `run_summary.jsonl`. Default `application/x-ndjson` snapshot, streamed from disk (never buffered whole); `Accept: text/event-stream` replays then follows it live, closing with `event: end` when terminal. The snapshot mode first verifies the journal's keyed integrity chain and returns `409 E_TAMPERED` when the chain does not verify (see [Architecture — Keyed hash chain](architecture.md#hash-chain)). Served verbatim (already credential-redacted); raw capture files are never exposed. `404` unknown. | -| `GET /v1/runs/{id}/artifacts` | `inspect` | `{artifacts: [{path, size, mtime}]}` for files published under the run's `artifacts/` (empty when none). `404` unknown. | -| `GET /v1/runs/{id}/artifacts/{path}` | `inspect` | Download one published file (`application/octet-stream`), streamed with backpressure — never buffered whole, so an arbitrarily large file costs no server memory and a client disconnect closes the file. Traversal-proof — `..`, absolute paths, and escaping symlinks are `404`. `413 E_ARTIFACT_TOO_LARGE` when the file exceeds `JAIPH_SERVE_MAX_ARTIFACT_BYTES`. | -| `POST /v1/runs/{id}/cancel` | `cancel` | `202`; the run reaches `cancelled`. `409` if already terminal. | +| `GET /defs` | `inspect` | `{defs: [{name, description, params}]}`. | +| `POST /{name}` | `invoke` | Start a run. Default `202` + `Location: /runs/{id}`; `?wait=true` blocks for the terminal `200`. Send an `Idempotency-Key` header (scoped to the authenticated principal + def) to make retries safe: an identical repeat returns the original run (`200`, no second spawn); a reused key with different arguments is `409 E_IDEMPOTENCY_CONFLICT` and spawns nothing. | +| `GET /runs` | `inspect` | Runs started by this process **plus runs reconstructed from disk on restart**, newest first, scoped to the caller's own runs (all runs for a static/open principal). Paginated: `?limit` (default `100`, clamped to `1000`), `?offset` (default `0`). Response is `{runs, total, limit, offset}` and never unbounded. | +| `GET /runs/{id}` | `inspect` | The run object. `404` unknown (a run the principal does not own is indistinguishable from nonexistent). | +| `GET /runs/{id}/events` | `inspect` | The run's `run_summary.jsonl`. Default `application/x-ndjson` snapshot, streamed from disk (never buffered whole); `Accept: text/event-stream` replays then follows it live, closing with `event: end` when terminal. The snapshot mode first verifies the journal's keyed integrity chain and returns `409 E_TAMPERED` when the chain does not verify (see [Architecture — Keyed hash chain](architecture.md#hash-chain)). Served verbatim (already credential-redacted); raw capture files are never exposed. `404` unknown. | +| `GET /runs/{id}/artifacts` | `inspect` | `{artifacts: [{path, size, mtime}]}` for files published under the run's `artifacts/` (empty when none). `404` unknown. | +| `GET /runs/{id}/artifacts/{path}` | `inspect` | Download one published file (`application/octet-stream`), streamed with backpressure — never buffered whole, so an arbitrarily large file costs no server memory and a client disconnect closes the file. Traversal-proof — `..`, absolute paths, and escaping symlinks are `404`. `413 E_ARTIFACT_TOO_LARGE` when the file exceeds `JAIPH_SERVE_MAX_ARTIFACT_BYTES`. | +| `POST /runs/{id}/cancel` | `cancel` | `202`; the run reaches `cancelled`. `409` if already terminal. | The run object is `{run_id, def, status, started_at, ended_at, exit_status, signal, result_text, run_dir, principal, correlation_id}` where `status` is `running` \| `succeeded` \| `failed` \| `cancelled` \| `interrupted`. `principal` is the audit subject that created the run (`anonymous`/`operator` in open/static mode, the token `sub` or `client_id` in OIDC mode — never a token) and `correlation_id` is the request id attached at create time; both are `null` when unset. `interrupted` is the terminal state a run is reconciled to after a process death caught it mid-flight — its outcome is unknown, so it is neither `succeeded` nor `failed`, but it is never reported as permanently `running`. **A def failure is not an HTTP error** — the run object reports `status: "failed"` with the same failure narrative `jaiph mcp` returns, over HTTP `200`/`202`. Errors use `{error: {code, message}}` with `400 E_BAD_ARGS`, `401 E_UNAUTHORIZED` (missing or invalid static token; in OIDC mode, a request with no bearer token, or a verified token that carries neither `sub` nor `client_id`), `401 E_TOKEN_EXPIRED` / `401 E_TOKEN_INVALID` (OIDC token expired, or bad audience/issuer/key/signature/algorithm), `403 E_FORBIDDEN` (principal lacks the required capability), `404 E_NOT_FOUND`, `409 E_RUN_TERMINAL`, `409 E_IDEMPOTENCY_CONFLICT` (idempotency key reused with different arguments), `409 E_TAMPERED` (the run's journal failed its keyed integrity chain), `413 E_BODY_TOO_LARGE` (1 MiB request-body cap), `413 E_ARTIFACT_TOO_LARGE` (artifact download over `JAIPH_SERVE_MAX_ARTIFACT_BYTES`), `415` (non-`application/json` body), `429 E_TOO_MANY_RUNS`, and `503 E_AUTH_UNAVAILABLE` (OIDC identity provider / JWKS unreachable). -Each run's public record is persisted beside its journal as `run.json` when it finishes, and reconstructed into the registry on startup — so `GET /v1/runs`, `/v1/runs/{id}`, `/events`, and `/artifacts` keep working for pre-restart terminal runs, and idempotency keys survive a restart. `jaiph serve` is a **single-replica** service: the run registry, concurrency cap, and idempotency index are per-process and not shared across replicas — run two behind one load balancer and each has its own view. See [Serve — deployment topology](serve.md#deployment-topology). +Each run's public record is persisted beside its journal as `run.json` when it finishes, and reconstructed into the registry on startup — so `GET /runs`, `/runs/{id}`, `/events`, and `/artifacts` keep working for pre-restart terminal runs, and idempotency keys survive a restart. `jaiph serve` is a **single-replica** service: the run registry, concurrency cap, and idempotency index are per-process and not shared across replicas — run two behind one load balancer and each has its own view. See [Serve — deployment topology](serve.md#deployment-topology). ### Auth and limits -- **Authentication** has two production modes (credentials come from the environment, never argv) plus an anonymous mode that is an explicit opt-in for a single-user workstation (`--allow-anonymous`). **Static single-operator token:** `JAIPH_SERVE_TOKEN` is a shared secret required on every `/v1/*` and `/mcp` request (`Authorization: Bearer `, constant-time compared). It is a fail-closed gate for **one operator** — no per-user identity, revocation, or per-action authorization; the operator holds every capability and sees every run — not multi-tenant authentication. **OIDC/JWT (multi-tenant):** set `JAIPH_SERVE_OIDC_ISSUER` + `JAIPH_SERVE_OIDC_AUDIENCE` (takes precedence over the static token; setting only one is a startup error) to verify bearer JWTs against the issuer's JWKS (discovered from `/.well-known/openid-configuration`, or set `JAIPH_SERVE_OIDC_JWKS_URI`) with a maintained JWT library — signature, `exp`/`nbf`, `aud`, `iss`, `kid`, and an explicit allowlist of asymmetric signing algorithms (RSA, ECDSA, and EdDSA families; symmetric algorithms, `alg: none`, and `ES256K` are rejected). Each token is authorized by OAuth scopes: `jaiph:invoke` (run), `jaiph:inspect` (read defs/runs/events/artifacts, MCP `tools/list`), `jaiph:cancel` (cancel a run); a missing capability is `403 E_FORBIDDEN`, and a principal (the token `sub`, or `client_id` for `sub`-less machine tokens; a verified token with neither is `401 E_UNAUTHORIZED`) may inspect or cancel **only the runs it created**. The authenticated subject and the request's correlation id (`X-Correlation-Id` / `X-Request-Id`, else a generated UUID) attach to run metadata, the invoke/cancel audit log lines, OTLP resource attributes, and Sentry tags — never a token or a claim value. -- Binding a non-loopback `--host` with **no** authentication is a startup error, and `--allow-anonymous` does not lift it. On loopback with no token or OIDC, startup is also refused unless you pass `--allow-anonymous` — anonymous mode makes every caller the `anonymous` principal with all capabilities over all runs, so it is for a single-user workstation only and prints a startup warning when enabled. +- **Authentication** has two production modes (credentials come from the environment, never argv) plus an anonymous mode that is an explicit opt-in for a single-user workstation (`--allow-anonymous`). **Static single-operator token:** `JAIPH_SERVE_TOKEN` is a shared secret required on every REST and `/mcp` request (`Authorization: Bearer `, constant-time compared). It is a fail-closed gate for **one operator** — no per-user identity, revocation, or per-action authorization; the operator holds every capability and sees every run — not multi-tenant authentication. **OIDC/JWT (multi-tenant):** set `JAIPH_SERVE_OIDC_ISSUER` + `JAIPH_SERVE_OIDC_AUDIENCE` (takes precedence over the static token; setting only one is a startup error) to verify bearer JWTs against the issuer's JWKS (discovered from `/.well-known/openid-configuration`, or set `JAIPH_SERVE_OIDC_JWKS_URI`) with a maintained JWT library — signature, `exp`/`nbf`, `aud`, `iss`, `kid`, and an explicit allowlist of asymmetric signing algorithms (RSA, ECDSA, and EdDSA families; symmetric algorithms, `alg: none`, and `ES256K` are rejected). Each token is authorized by OAuth scopes: `jaiph:invoke` (run), `jaiph:inspect` (read defs/runs/events/artifacts, MCP `tools/list`), `jaiph:cancel` (cancel a run); a missing capability is `403 E_FORBIDDEN`, and a principal (the token `sub`, or `client_id` for `sub`-less machine tokens; a verified token with neither is `401 E_UNAUTHORIZED`) may inspect or cancel **only the runs it created**. The authenticated subject and the request's correlation id (`X-Correlation-Id` / `X-Request-Id`, else a generated UUID) attach to run metadata, the invoke/cancel audit log lines, OTLP resource attributes, and Sentry tags — never a token or a claim value. +- With **no** authentication, startup is refused unless you pass `--allow-anonymous`. The flag makes every caller the `anonymous` principal with all capabilities over all runs, and it permits loopback and non-loopback binds. The server prints a startup warning. Shared or network-exposed hosts must set `JAIPH_SERVE_TOKEN` or configure OIDC. - `JAIPH_SERVE_EXPOSE_DOCS` (default `true`) controls whether `/docs` and `/openapi.json` are served; set `false` (or `0`) to return `404` for both and hide the API surface. `/healthz` is always open and credential-free (liveness/readiness only — no tokens or sensitive detail). - `JAIPH_SERVE_MAX_CONCURRENT` (default `4`) caps simultaneous runs; requests beyond it get `429`. - `JAIPH_SERVE_MAX_ARTIFACT_BYTES` (default `0` = no cap) refuses artifact downloads larger than the limit with `413`. Downloads stream with backpressure regardless, so the default keeps server memory bounded no matter the file size; set a finite cap only to reject oversized downloads outright. diff --git a/docs/contributing.md b/docs/contributing.md index b2c1b42c..8c7aea42 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -38,9 +38,15 @@ jaiph --version jaiph --help ``` -The script builds the self-contained standalone binary via `docs/install` (`npm ci` when a lockfile is present, else `npm install`, plus `npm run build:standalone`, including uncommitted changes) and installs `dist/jaiph` to `~/.local/bin` by default (or `JAIPH_BIN_DIR` if set). +The script builds the self-contained standalone binary via `docs/install` (`npm ci` when a lockfile is present, else `npm install`, plus `npm run build:standalone`, including uncommitted changes) and installs `dist/jaiph` to `~/.local/bin` by default (or `JAIPH_BIN_DIR` if set). It does **not** build the linux runner image. -**From-source prerequisites:** **`npm`** and **[Bun](https://bun.sh)**. +To build a local `ghcr.io/jaiphlang/jaiph-runtime` (linux binaries + `runtime/Dockerfile`): + +```bash +./docs/build-jaiph-dev-image.sh +``` + +**From-source prerequisites:** **`npm`** and **[Bun](https://bun.sh)**. The image script also needs **Docker**. ## Developing in the repository @@ -186,7 +192,7 @@ Tests that span multiple modules, require subprocess/PTY harnesses, exercise pro | `integration/signal-lifecycle.test.ts` | Acceptance | After SIGINT/SIGTERM, verifies `jaiph run` exits within a time bound and leaves no stale child processes | | `integration/subcommand-help.test.ts` | Integration | `--help` / usage text for CLI subcommands | | `integration/mcp-server.test.ts` | Integration | Drives a live `jaiph mcp` stdio JSON-RPC session — `initialize` / `tools/list` / `tools/call`, the `--mcp` alias, hot-reload `list_changed`, `--env` forwarding and `E_ENV_*` pre-flight, compile diagnostics to stderr, progress-token streaming, and `notifications/cancelled` | -| `integration/serve-server.test.ts` | Integration | `jaiph serve` HTTP contract — synchronous (`wait=true`) and async (`202` + `Location` polling) runs, a failing workflow returned as HTTP 200 with status `failed`, hot reload surfacing a new workflow, bearer auth required on `/v1/*` while `/healthz` / `/openapi.json` / `/docs` stay open, and refusing to bind a non-loopback host without `JAIPH_SERVE_TOKEN` | +| `integration/serve-server.test.ts` | Integration | `jaiph serve` HTTP contract — synchronous (`wait=true`) and async (`202` + `Location` polling) runs, a failing workflow returned as HTTP 200 with status `failed`, hot reload surfacing a new workflow, bearer auth required on REST while `/healthz` / `/openapi.json` / `/docs` stay open, refusing to bind without auth unless `--allow-anonymous`, and `--allow-anonymous` permitting a non-loopback bind | | `integration/serve-auth.test.ts` | Integration | `jaiph serve` OIDC/JWT auth against a local JWKS server — token validity matrix (valid, expired, wrong audience, wrong issuer, unknown key, insufficient scope, missing), per-principal capabilities and run ownership, and audited invoke / cancel | | `integration/serve-restart.test.ts` | Integration | `jaiph serve` run recovery and idempotency across a real process restart | | `integration/otlp-export.test.ts` | Integration | OTLP trace export — a run with OTLP env sends exactly one well-formed POST to `/v1/traces`, and delivery is detached so a hanging collector does not block the terminal result | @@ -221,11 +227,11 @@ jaiph run .jaiph/prepare_release.jh # next patch from package.json The workflow refuses to start when the git tree is dirty or when `v` already exists, then bumps `package.json` + `package-lock.json` via `npm version X.Y.Z --no-git-tag-version --allow-same-version`, refreshes the hardcoded release ref in `docs/install` and `docs/install.ps1` (kept in lockstep), runs **`npm run build`** (rebuilding **`dist/`**), asserts **`node dist/src/cli.js --version`** matches the new version, and runs **`npm run registry:build`** to regenerate **`docs/registry`**. It creates **no commits, no tags, and no `git add`** — review the diff (`git diff`), stage the changes, commit, then `git tag v` and push branch + tag yourself. The CLI version is single-sourced from `package.json`'s `version` field (codegen'd into `src/version.ts` by `npm run embed-assets`). -Pushing a **`v*`** tag triggers the standalone-binary release — `.github/workflows/release.yml` cross-compiles the Bun-compiled standalone binary for five targets via `oven-sh/setup-bun` and `bun build --compile --target=…`, generates a `SHA256SUMS` file, signs it with minisign (if `MINISIGN_SECRET_KEY` is set) to produce `SHA256SUMS.minisig`, runs a Linux x64 sanity gate and a `windows-latest` Windows x64 sanity gate (`--version` must equal `jaiph ` for stable tags — both delegate to `scripts/release-version-check.sh`), and uploads all seven assets to the GitHub Release for the tag (creating it if needed). The Windows gate is a required dependency of the publish job, so a version mismatch there fails the whole release. The release job waits for the `CI` workflow on the same SHA to succeed before publishing. Re-runs are available via `workflow_dispatch`. +Pushing a **`v*`** tag triggers the standalone-binary release — `.github/workflows/release.yml` cross-compiles the Bun-compiled standalone binary for five targets via `oven-sh/setup-bun` and `bun build --compile --target=…`, generates a `SHA256SUMS` file, signs it with minisign (if `MINISIGN_SECRET_KEY` is set) to produce `SHA256SUMS.minisig`, runs a Linux x64 sanity gate and a `windows-latest` Windows x64 sanity gate (`--version` must equal `jaiph ` for stable tags — both delegate to `scripts/release-version-check.sh`), and uploads all seven assets to the GitHub Release for the tag (creating it if needed). A sibling job builds `runtime/Dockerfile` for `linux/amd64` and `linux/arm64` and pushes `ghcr.io//jaiph-runtime` (`:`, `:v`, `:latest` on stable; `:nightly` on nightly). The image is not a GitHub Release asset and is not listed in `SHA256SUMS`. The Windows gate is a required dependency of both publish jobs, so a version mismatch there fails the whole release. The release job waits for the `CI` workflow on the same SHA to succeed before publishing. Re-runs are available via `workflow_dispatch`. Pushes to the **`nightly`** branch follow the same matrix and upload to a **rolling prerelease** tagged `nightly` (`gh release upload nightly --clobber`), so `jaiph use nightly` keeps working under the binary installer. -Pushing a **`v*`** ref does **not** run any npm publish step from this repository — `.github/workflows/` contains `ci.yml` (push CI), `release.yml` (standalone binaries; see above), `nightly-engineer.yml` (optional manual engineer run), the path-filtered editor-plugin jobs `vscode-plugin.yml` and `zed-plugin.yml` (each builds and tests its extension under `plugins/` only when that extension's tree changes), and the path-filtered `setup-jaiph-action.yml` (installs the nightly release through the `actions/setup-jaiph` composite action on Linux and macOS, only when that action or `docs/install` changes), and **none publishes to npm**. If you are preparing a release that includes the **npm** package, coordinate version bumps, registry publish, and smoke checks with the maintainers — that flow is intentionally outside this repo's workflows. +Pushing a **`v*`** ref does **not** run any npm publish step from this repository — `.github/workflows/` contains `ci.yml` (push CI), `release.yml` (standalone binaries and the GHCR runner image; see above), `nightly-engineer.yml` (optional manual engineer run), the path-filtered editor-plugin jobs `vscode-plugin.yml` and `zed-plugin.yml` (each builds and tests its extension under `plugins/` only when that extension's tree changes), and the path-filtered `setup-jaiph-action.yml` (installs the nightly release through the `actions/setup-jaiph` composite action on Linux and macOS, only when that action or `docs/install` changes), and **none publishes to npm**. If you are preparing a release that includes the **npm** package, coordinate version bumps, registry publish, and smoke checks with the maintainers — that flow is intentionally outside this repo's workflows. #### Release asset naming contract diff --git a/docs/deploy.md b/docs/deploy.md index 6df5e745..ed876902 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -6,9 +6,11 @@ diataxis: how-to # Deploy jaiph in your own image or pod -Jaiph executes programs on the host. It does not ship a runtime image or a Docker sandbox driver. Isolation is an outer concern: wrap `jaiph` in a container, a Kubernetes pod, a CI runner, or another sandbox you already operate. +Jaiph executes programs on the host. It does not ship a Docker sandbox driver. Isolation is an outer concern: wrap `jaiph` in a container, a Kubernetes pod, a CI runner, or another sandbox you already operate. -A complete, apply-ready Kubernetes example lives at [`docs/deploy/k8s.yaml`](https://github.com/jaiphlang/jaiph/blob/main/docs/deploy/k8s.yaml). It assumes **your** image already has `jaiph` on `PATH`. Build that image yourself (install the CLI, any agent backends your programs use, and whatever toolchain the scripts need). Jaiph does not publish a first-party runner image. +Each release publishes a lean runner image to GHCR: `ghcr.io/jaiphlang/jaiph-runtime` (`:`, `:v`, and `:latest` on a stable tag; `:nightly` on the rolling nightly). It has `jaiph`, `python3`, `git`, and `curl` on `PATH`, running as uid `10001`. It is not a sandbox and not a toolchain image. Layer agent CLIs (`claude`, `cursor-agent`, `codex`) and any script dependencies yourself. From a clone, `./docs/build-jaiph-dev-image.sh` builds the same image locally (`docs/install-from-local.sh` does not). + +A complete, apply-ready Kubernetes example lives at [`docs/deploy/k8s.yaml`](https://github.com/jaiphlang/jaiph/blob/main/docs/deploy/k8s.yaml). The example uses the published image. Pin a version tag in production. ## Isolation is yours @@ -33,12 +35,12 @@ This isolates Jaiph from **you**. It does not isolate the agent from a script in ## Run one program in a container you own -Mount your working directory and run the CLI as the container command. Replace `your-registry/your-jaiph-image` with an image you built: +Mount your working directory and run the CLI as the container command. The published image has `jaiph` on `PATH`; add the agent CLI your program uses if the entry file has a `prompt` step: ```bash -# claude backend (Anthropic) +# claude backend (Anthropic) — image must also have `claude` on PATH docker run --rm -e ANTHROPIC_API_KEY -v "$PWD":/work -w /work \ - your-registry/your-jaiph-image jaiph run flow.jh + ghcr.io/jaiphlang/jaiph-runtime:latest jaiph run flow.jh ``` The credential env var depends on the backend the entry file selects: @@ -46,11 +48,11 @@ The credential env var depends on the backend the entry file selects: ```bash # cursor backend docker run --rm -e CURSOR_API_KEY -v "$PWD":/work -w /work \ - your-registry/your-jaiph-image jaiph run flow.jh + ghcr.io/jaiphlang/jaiph-runtime:latest jaiph run flow.jh # codex backend (OpenAI HTTP API) docker run --rm -e OPENAI_API_KEY -v "$PWD":/work -w /work \ - your-registry/your-jaiph-image jaiph run flow.jh + ghcr.io/jaiphlang/jaiph-runtime:latest jaiph run flow.jh ``` `-e ANTHROPIC_API_KEY` with no `=value` forwards the value from your shell environment. The `claude` backend also accepts `CLAUDE_CODE_OAUTH_TOKEN` in place of `ANTHROPIC_API_KEY`. A program with no `prompt` step needs no credential at all. Run artifacts land under `/work/.jaiph/runs/` when `/work` is your bind-mounted directory. @@ -65,12 +67,12 @@ Use a container image when you want a preinstalled toolchain. Fit the one-shot ` - name: Run jaiph run: | docker run --rm -e ANTHROPIC_API_KEY -v "$PWD":/work -w /work \ - your-registry/your-jaiph-image jaiph run flow.jh + ghcr.io/jaiphlang/jaiph-runtime:latest jaiph run flow.jh ``` ## Kubernetes -Create the `jaiph-credentials` Secret out-of-band first, then apply the example manifest. Edit the Deployment `image:` to your image before apply: +Create the `jaiph-credentials` Secret out-of-band first, then apply the example manifest. Pin `image:` to a version tag before production apply: ```bash kubectl create secret generic jaiph-credentials \ @@ -83,11 +85,11 @@ The Deployment references the Secret as a required `envFrom`, so a missing Secre The manifest runs `jaiph serve --host 0.0.0.0` as a long-lived HTTP runner (see [Serve defs over HTTP](serve.md)), with `JAIPH_SERVE_TOKEN` sourced from the Secret and liveness and readiness probes on `GET /healthz`, which stays open and needs no bearer token. The same Service port serves both the REST and OpenAPI API and MCP Streamable HTTP at `POST /mcp`. The example sets: -- **Pod hardening by default.** `runAsNonRoot`, `allowPrivilegeEscalation: false`, all capabilities dropped, the `RuntimeDefault` seccomp profile, `readOnlyRootFilesystem: true`, and `automountServiceAccountToken: false`. Programs never talk to the Kubernetes API, so they get no API credential to leak. Set `runAsUser` / `runAsGroup` to match the user in **your** image. +- **Pod hardening by default.** `runAsNonRoot`, `allowPrivilegeEscalation: false`, all capabilities dropped, the `RuntimeDefault` seccomp profile, `readOnlyRootFilesystem: true`, and `automountServiceAccountToken: false`. Programs never talk to the Kubernetes API, so they get no API credential to leak. `runAsUser` / `runAsGroup` are `10001`, matching the published image. - **Writable mounts only where required.** Program sources stay read-only as a ConfigMap at `/work`. Run artifacts go to a dedicated `emptyDir` at `/jaiph/runs` set by `JAIPH_RUNS_DIR`. Two more `emptyDir` volumes cover `/tmp` and a writable `$HOME`. - **Single replica by design.** The manifest pins `replicas: 1` with a `Recreate` strategy. `jaiph serve` holds its run registry, concurrency cap, and idempotency index in process with no shared store, so running more than one replica is not supported. Scale vertically with more resources and `JAIPH_SERVE_MAX_CONCURRENT`, not by adding replicas. See [Serve, deployment topology](serve.md#deployment-topology). - **TLS at the ingress.** `jaiph serve` speaks plain HTTP. The Service stays `ClusterIP`, and you terminate TLS at an Ingress or gateway in front of it. Do not expose the token-guarded API to the internet without TLS. -- **Authentication.** Binding `0.0.0.0` with no authentication is a startup error by design, so the Secret is mandatory. `JAIPH_SERVE_TOKEN` is the single-operator shared secret shown here. For multiple company users, configure OIDC or JWT instead. See [Authenticate and authorize](serve.md#7-authenticate-and-authorize). +- **Authentication.** The example does not pass `--allow-anonymous`, so the Secret is mandatory. `JAIPH_SERVE_TOKEN` is the single-operator shared secret shown here. For multiple company users, configure OIDC or JWT instead. See [Authenticate and authorize](serve.md#7-authenticate-and-authorize). Isolation is the pod boundary. There is no jaiph-managed sandbox inside. diff --git a/docs/deploy/k8s.yaml b/docs/deploy/k8s.yaml index 15609326..26114867 100644 --- a/docs/deploy/k8s.yaml +++ b/docs/deploy/k8s.yaml @@ -1,9 +1,9 @@ # Jaiph on Kubernetes — long-lived HTTP runner. # -# Wrap jaiph in an image you own: this manifest does not ship a first-party -# runtime image. Set `image:` to a registry image that has `jaiph` on PATH -# (and any agent backends / toolchain your workflows need). Isolation is the -# pod boundary. See docs/deploy.md. +# Default image is the published runner (`ghcr.io/jaiphlang/jaiph-runtime`). +# It has `jaiph` on PATH as uid 10001. Layer agent CLIs and script +# dependencies in a derived image if your programs need them. Isolation is +# the pod boundary. See docs/deploy.md. # # Credentials are NOT part of this manifest. Create the `jaiph-credentials` # Secret out-of-band before applying — the Deployment references it as a @@ -74,8 +74,8 @@ spec: # (and the agent backends they spawn) an API credential. automountServiceAccountToken: false securityContext: - # Set UID/GID to the non-root user in YOUR image; fsGroup makes the - # emptyDir volumes below group-writable for it. + # UID/GID match the published runner image (uid 10001). fsGroup + # makes the emptyDir volumes below group-writable for it. runAsNonRoot: true runAsUser: 10001 runAsGroup: 10001 @@ -84,8 +84,8 @@ spec: type: RuntimeDefault containers: - name: jaiph - # Replace with an image you built that has `jaiph` on PATH. - image: your-registry/your-jaiph-image:tag + # Pin a version tag in production (`:0.15.0`, not `:latest`). + image: ghcr.io/jaiphlang/jaiph-runtime:latest command: ["jaiph", "serve", "--host", "0.0.0.0", "/work/tools.jh"] securityContext: allowPrivilegeEscalation: false diff --git a/docs/env-vars.md b/docs/env-vars.md index 5532bdbb..123a4d13 100644 --- a/docs/env-vars.md +++ b/docs/env-vars.md @@ -96,7 +96,7 @@ The long-lived servers `jaiph serve` and `jaiph mcp` resolve the effective env o | `JAIPH_SERVE_OIDC_JWKS_URI` | host | string | — | — | `jaiph serve` — explicit JWKS URI for OIDC token verification. Optional; when unset the JWKS URI is discovered from `JAIPH_SERVE_OIDC_ISSUER`'s OpenID configuration document. | | `JAIPH_SERVE_RETAIN_AGE_SEC` | host | int | `86400` (24h) | — | `jaiph serve` — max age (seconds, from `ended_at`) of a completed run kept in the in-memory registry; older terminal records are evicted. `0` disables age eviction. Active runs are never evicted; durable `.jaiph/runs` artifacts are unaffected. Must be `>= 0`. | | `JAIPH_SERVE_RETAIN_RUNS` | host | int | `500` | — | `jaiph serve` — max completed runs kept in the in-memory registry; beyond it the oldest terminal records are evicted first. Active runs are never evicted; durable `.jaiph/runs` artifacts are unaffected. Must be a positive integer. | -| `JAIPH_SERVE_TOKEN` | host | string | — | — | `jaiph serve` — static **single-operator** bearer token required on every `/v1/*` and `/mcp` request (constant-time compared). This is a shared-secret gate, **not** multi-tenant authentication: there is no per-user identity, revocation, or per-action authorization — the one operator holds every capability. For those, use OIDC (`JAIPH_SERVE_OIDC_ISSUER` + `JAIPH_SERVE_OIDC_AUDIENCE`), which takes precedence. Unset leaves `/v1/*` open on loopback; binding a non-loopback `--host` with no auth is a startup error. `/healthz` is always unauthenticated; `/docs` + `/openapi.json` follow `JAIPH_SERVE_EXPOSE_DOCS`. The whole `JAIPH_SERVE_*` family is host-only and is excluded from the forwarding allowlist and the prompt scrub, so a program the server runs never sees this token. | +| `JAIPH_SERVE_TOKEN` | host | string | — | — | `jaiph serve` — static **single-operator** bearer token required on every REST and `/mcp` request (constant-time compared). This is a shared-secret gate, **not** multi-tenant authentication: there is no per-user identity, revocation, or per-action authorization — the one operator holds every capability. For those, use OIDC (`JAIPH_SERVE_OIDC_ISSUER` + `JAIPH_SERVE_OIDC_AUDIENCE`), which takes precedence. Unset leaves the API open only with `--allow-anonymous` (loopback or non-loopback). Without the flag, any bind with no token and no OIDC is a startup error. `/healthz` is always unauthenticated; `/docs` + `/openapi.json` follow `JAIPH_SERVE_EXPOSE_DOCS`. The whole `JAIPH_SERVE_*` family is host-only and is excluded from the forwarding allowlist and the prompt scrub, so a program the server runs never sees this token. | | `JAIPH_SERVER_LOG` | host | string (`debug`) | — | — | `jaiph mcp` and `jaiph serve` operator-log verbosity. Set it to `debug` to print the servers' `debug` diagnostic lines on **stderr**. Any other value keeps only `info`, `warn`, and `error`. It never affects MCP stdout (JSON-RPC only) or HTTP response bodies (API payloads only). It is not a logging framework, only a thin stderr line writer, with no winston or pino. | | `JAIPH_SERVER_LOG_RUNS` | host | bool | — | — | `jaiph mcp` and `jaiph serve`. When set to `1` or `true`, mirror each run's `log`, `logwarn`, and `logerr` events to the operator log on **stderr**. Each mirrored line is colored by level, carries `run_id=`, and uses the same depth and async-branch subscript indent as the `jaiph run` tree. Mirroring is off by default, so an MCP host is not flooded and the tool-result text is not repeated. Mirrored lines go through the same credential redaction as the durable run journal, so a secret is never printed to stderr. | | `JAIPH_SITE` | host | string | `https://jaiph.org` | — | Base URL `jaiph use` (and the `docs/run` / `docs/init` bootstraps) fetch the install script and its `install.sha256` from. The default install path verifies the fetched script against the published checksum before executing it (finding M-11). | diff --git a/docs/serve.md b/docs/serve.md index 93039354..55ff3cc1 100644 --- a/docs/serve.md +++ b/docs/serve.md @@ -10,7 +10,7 @@ This guide turns a `.jh` file into an HTTP API. `jaiph serve ./tools.jh` exposes It reuses the same compile-time validation, host execution, and `.jaiph/runs/` artifacts as [`jaiph run`](cli.md#jaiph-run), and the same exposure rules as [`jaiph mcp`](mcp.md). `jaiph mcp` binds the server to a stdio parent on the same machine, and `jaiph serve` instead makes the defs reachable over the network. -> **Security.** An exposed def is arbitrary shell that anyone who can reach the port can run, and that is the point of serving it. A request argument that binds to a def parameter is shell-quoted before it reaches any shell step, so a caller cannot use an argument value to inject extra shell commands, though the caller can still run whatever the exposed def itself does. When a token is set, the caller must also hold the token. With no token or OIDC the server has no auth at all, so it refuses to start unless you pass `--allow-anonymous`, which is for a single-user workstation only — on a shared host any local user could invoke the defs (see [Authenticate and authorize](#7-authenticate-and-authorize)). For anything beyond a single-user loopback, configure authentication, put the server behind a reverse proxy or ingress that terminates TLS, and treat the run directory as sensitive. The process itself speaks plain HTTP. For authentication, use a static `JAIPH_SERVE_TOKEN` for a single operator, or OIDC/JWT for multiple users (see [Authenticate and authorize](#7-authenticate-and-authorize)). +> **Security.** An exposed def is arbitrary shell that anyone who can reach the port can run, and that is the point of serving it. A request argument that binds to a def parameter is shell-quoted before it reaches any shell step, so a caller cannot use an argument value to inject extra shell commands, though the caller can still run whatever the exposed def itself does. When a token is set, the caller must also hold the token. With no token or OIDC the server has no auth at all, so it refuses to start unless you pass `--allow-anonymous` (see [Authenticate and authorize](#7-authenticate-and-authorize)). For anything reachable beyond a single operator, configure authentication, put the server behind a reverse proxy or ingress that terminates TLS, and treat the run directory as sensitive. The process itself speaks plain HTTP. For authentication, use a static `JAIPH_SERVE_TOKEN` for a single operator, or OIDC/JWT for multiple users (see [Authenticate and authorize](#7-authenticate-and-authorize)). ## Prerequisites @@ -29,22 +29,22 @@ The defaults are `--host 127.0.0.1` and `--port 5247`. All logs go to stderr. St ## 2. Discover the defs ```bash -curl -s http://127.0.0.1:5247/v1/defs | jq +curl -s http://127.0.0.1:5247/defs | jq ``` -`GET /v1/defs` returns `{"defs": [{name, description, params}, ...]}`, one entry per exposed def, and it needs the `inspect` capability. `GET /openapi.json` returns the full OpenAPI 3.1 document, with one path per def. Each path carries the def's `#`-comment description and its parameters as a JSON request-body schema. `GET /healthz` is a liveness and readiness probe that needs no authentication. +`GET /defs` returns `{"defs": [{name, description, params}, ...]}`, one entry per exposed def, and it needs the `inspect` capability. `GET /openapi.json` returns the full OpenAPI 3.1 document, with one path per def. Each path carries the def's `#`-comment description and its parameters as a JSON request-body schema. `GET /healthz` is a liveness and readiness probe that needs no authentication. ## 3. Invoke a def -A run is a durable resource. `POST /v1/defs/{name}/runs` starts one. The request body is a JSON object of the def's parameters, and every value is a string. +A run is a durable resource. `POST /{name}` starts one. The request body is a JSON object of the def's parameters, and every value is a string. ```bash # Async: 202 + a Location header pointing at the run resource. -curl -si -X POST http://127.0.0.1:5247/v1/defs/greet/runs \ +curl -si -X POST http://127.0.0.1:5247/greet \ -H 'content-type: application/json' -d '{"name":"world"}' # Synchronous: block until the run is terminal, then return the final object. -curl -s -X POST 'http://127.0.0.1:5247/v1/defs/greet/runs?wait=true' \ +curl -s -X POST 'http://127.0.0.1:5247/greet?wait=true' \ -H 'content-type: application/json' -d '{"name":"world"}' | jq ``` @@ -52,19 +52,19 @@ The run object has these fields: `run_id`, `def`, `status`, `started_at`, `ended `result_text` is the same content an MCP client sees, which is the def's `return` value or its failure narrative. A failure narrative is credential-redacted the same way as the event journal, so the value of any env var whose name looks like a credential becomes `[REDACTED]`. The rule now covers many common secret names, e.g. `*_API_KEY`, `*_SECRET`, `*_PASSWORD`, `AWS_SECRET_ACCESS_KEY`, and `*_PRIVATE_KEY`, and it also redacts the base64, hex, and URL-encoded forms of each value. See [Architecture — Secret redaction](architecture.md#secret-redaction) for the full rule and its limits. The `return` value of a successful run is intended API output, so it is returned verbatim. -A def failure is not an HTTP error. A failed run comes back `200` or `202` with `status: "failed"` and a `run dir:` pointer in `result_text`. Poll `GET /v1/runs/{id}` for an async run. List runs with `GET /v1/runs`, which returns them newest first and paginated. `?limit=` defaults to 100 and is clamped to 1000, and `?offset=` skips that many records. The listing response carries `{runs, total, limit, offset}`. Stop a run with `POST /v1/runs/{id}/cancel`. +A def failure is not an HTTP error. A failed run comes back `200` or `202` with `status: "failed"` and a `run dir:` pointer in `result_text`. Poll `GET /runs/{id}` for an async run. List runs with `GET /runs`, which returns them newest first and paginated. `?limit=` defaults to 100 and is clamped to 1000, and `?offset=` skips that many records. The listing response carries `{runs, total, limit, offset}`. Stop a run with `POST /runs/{id}/cancel`. ## 4. Watch a run as it executes -`GET /v1/runs/{id}/events` streams the run's durable event journal (`run_summary.jsonl`), which is the same timeline the CLI builds its progress tree from. It has two modes. +`GET /runs/{id}/events` streams the run's durable event journal (`run_summary.jsonl`), which is the same timeline the CLI builds its progress tree from. It has two modes. ```bash # Snapshot (default): the whole journal as newline-delimited JSON, then close. -curl -s http://127.0.0.1:5247/v1/runs/$ID/events +curl -s http://127.0.0.1:5247/runs/$ID/events # Live: Server-Sent Events. Replays the journal so far, then follows it as the # run appends, and closes with an `event: end` when the run is terminal. -curl -sN -H 'accept: text/event-stream' http://127.0.0.1:5247/v1/runs/$ID/events +curl -sN -H 'accept: text/event-stream' http://127.0.0.1:5247/runs/$ID/events ``` Each SSE message is a `data:` line that carries one raw journal line, such as `RUN_START`, `STEP_START`, `STEP_END`, `LOG*`, `PROMPT_*`, or `RUN_END`. A `:ka` comment every 15 seconds keeps proxies from idling the connection out. Connect while the run is still going to watch it step by step, or connect after it finishes for a full replay followed by an immediate `event: end`. Add `-H 'authorization: Bearer '` when a token is set. The `-N` flag on `curl` disables buffering, so events surface as they arrive. @@ -79,10 +79,10 @@ A def can publish files to `$JAIPH_ARTIFACTS_DIR` (see [artifacts](artifacts.md) ```bash # List published files: {"artifacts": [{path, size, mtime}, ...]} (empty when the run made none). -curl -s http://127.0.0.1:5247/v1/runs/$ID/artifacts | jq +curl -s http://127.0.0.1:5247/runs/$ID/artifacts | jq # Download one by its relative path (application/octet-stream). -curl -s http://127.0.0.1:5247/v1/runs/$ID/artifacts/report.txt -o report.txt +curl -s http://127.0.0.1:5247/runs/$ID/artifacts/report.txt -o report.txt ``` The download path is resolved strictly inside the run's `artifacts/` directory and is safe against path traversal. A `..` segment, an absolute path, or a symlink pointing outside the directory all return `404` without touching the target file. @@ -97,11 +97,11 @@ Open `http://127.0.0.1:5247/docs` in a browser to get a live form for every def. `jaiph serve` has two production authentication modes, plus an anonymous mode that is an explicit opt-in for a single-user workstation. Credentials come from the environment, never from argv, because argv leaks into process listings. In every mode, `/healthz` stays open and needs no credentials. `/docs` and `/openapi.json` also stay open, unless `JAIPH_SERVE_EXPOSE_DOCS=false` hides them behind a `404`. -**Static single-operator token.** `JAIPH_SERVE_TOKEN` is a shared secret required on every `/v1/*` and `/mcp` request, sent as `Authorization: Bearer ` and compared in constant time. It is a fail-closed gate for one operator. There is no per-user identity, no revocation, and no per-action authorization, so the one operator holds every capability and sees every run. Use it for a single trusted caller, not for several people in a company. The token stays on the host and never reaches a run process, so a def the server runs cannot read it and use it to call the API back as the operator. +**Static single-operator token.** `JAIPH_SERVE_TOKEN` is a shared secret required on every REST and `/mcp` request, sent as `Authorization: Bearer ` and compared in constant time. It is a fail-closed gate for one operator. There is no per-user identity, no revocation, and no per-action authorization, so the one operator holds every capability and sees every run. Use it for a single trusted caller, not for several people in a company. The token stays on the host and never reaches a run process, so a def the server runs cannot read it and use it to call the API back as the operator. ```bash JAIPH_SERVE_TOKEN=secret jaiph serve --host 0.0.0.0 --port 8080 ./tools.jh -curl -s http://host:8080/v1/defs -H 'authorization: Bearer secret' | jq +curl -s http://host:8080/defs -H 'authorization: Bearer secret' | jq ``` **OIDC/JWT for multiple users.** Set `JAIPH_SERVE_OIDC_ISSUER` and `JAIPH_SERVE_OIDC_AUDIENCE`. OIDC takes precedence over the static token when both are set. Bearer tokens are verified against the issuer's JWKS, which is the set of public signing keys. Jaiph discovers the key set from `/.well-known/openid-configuration`, or you can set `JAIPH_SERVE_OIDC_JWKS_URI` explicitly. A maintained JWT library (`jose`) does the cryptographic checks: the signature, the `exp` and `nbf` times, the `aud` and `iss` claims, and key selection by `kid`. Jaiph also pins an explicit allowlist of asymmetric signing algorithms, covering the RSA, ECDSA, and EdDSA families that standard OIDC providers sign with, and it excludes symmetric algorithms, `alg: none`, and the non-recommended secp256k1 curve (`ES256K`). A token whose header names an algorithm outside the allowlist is rejected even when its signing key is in the JWKS, so a future key-type or JWKS change cannot make an algorithm-confusion or `alg: none` forgery reachable. A verification failure is a `401`, with `E_TOKEN_EXPIRED` for an expired token and `E_TOKEN_INVALID` for a bad audience, issuer, key, signature, or signing algorithm. An unreachable identity provider is a `503` (`E_AUTH_UNAVAILABLE`). @@ -110,9 +110,9 @@ Each token is authorized by three OAuth scopes. Request them in the `scope` clai | Scope | Grants | | --- | --- | -| `jaiph:invoke` | `POST /v1/defs/{name}/runs` and MCP `tools/call` | -| `jaiph:inspect` | `GET /v1/defs`, `/v1/runs`, a run, its events and artifacts; MCP `tools/list` | -| `jaiph:cancel` | `POST /v1/runs/{id}/cancel` | +| `jaiph:invoke` | `POST /{name}` and MCP `tools/call` | +| `jaiph:inspect` | `GET /defs`, `/runs`, a run, its events and artifacts; MCP `tools/list` | +| `jaiph:cancel` | `POST /runs/{id}/cancel` | A missing capability is a `403` (`E_FORBIDDEN`). A principal is identified by the token `sub`, falling back to `client_id` for `sub`-less machine tokens (OAuth2 client-credentials); a verified token carrying neither is rejected `401 E_UNAUTHORIZED` rather than sharing one identity, so distinct callers never share a run-visibility bucket or idempotency namespace. A principal may inspect or cancel only the runs it created. Another principal's run returns `404`, so it looks the same as a run that does not exist. The authenticated `sub` and the request's correlation id are attached to three places: each run's metadata (`principal` and `correlation_id` on the run object), the invoke and cancel audit log lines, and the OTLP resource attributes and Sentry tags. The correlation id comes from an `X-Correlation-Id` or `X-Request-Id` header, or a generated UUID when neither is present. Jaiph never attaches the token or a claim value. @@ -120,16 +120,16 @@ A missing capability is a `403` (`E_FORBIDDEN`). A principal is identified by th JAIPH_SERVE_OIDC_ISSUER=https://issuer.example \ JAIPH_SERVE_OIDC_AUDIENCE=jaiph-serve \ jaiph serve --host 0.0.0.0 --port 8080 ./tools.jh -curl -s http://host:8080/v1/runs -H "authorization: Bearer $JWT" | jq +curl -s http://host:8080/runs -H "authorization: Bearer $JWT" | jq ``` -**Anonymous mode is single-user-workstation only.** With no `JAIPH_SERVE_TOKEN` and no OIDC configured, the server has no authentication: every `/v1/*` and `/mcp` request is authorized as an anonymous principal that holds every capability over every run. Loopback is a boundary against the network, not against other local users, so on a shared or multi-user host any other local user or process can invoke the exposed defs and read every run's artifacts. Because of that, anonymous mode is not the default — it is refused at startup unless you pass `--allow-anonymous` to opt in explicitly. Even with the flag, a non-loopback bind stays a startup error; the flag only permits a loopback bind. When you do pass it, the server prints a prominent startup warning that it is open to all local principals. Use anonymous mode only on a single-user workstation; any shared host must set `JAIPH_SERVE_TOKEN` or configure OIDC. +**Anonymous mode is an explicit opt-in.** With no `JAIPH_SERVE_TOKEN` and no OIDC configured, the server has no authentication: every REST and `/mcp` request is authorized as an anonymous principal that holds every capability over every run. Startup is refused unless you pass `--allow-anonymous`. The flag also permits a non-loopback bind — required inside Docker, where the process must listen on `0.0.0.0` for a published port. When you pass it, the server prints a prominent startup warning. On a shared host or a port reachable from the network, set `JAIPH_SERVE_TOKEN` or configure OIDC. -Binding a non-loopback `--host` with no authentication, meaning neither the token nor OIDC, is a startup error, and `--allow-anonymous` does not lift it. The server refuses to expose unauthenticated arbitrary shell over the network. Cap simultaneous runs with `JAIPH_SERVE_MAX_CONCURRENT`, which defaults to `4`. A request beyond the cap gets `429`. See [Environment variables](env-vars.md) for all of these. +Cap simultaneous runs with `JAIPH_SERVE_MAX_CONCURRENT`, which defaults to `4`. A request beyond the cap gets `429`. See [Environment variables](env-vars.md) for all of these. ## 8. Connect an MCP client over HTTP -The same process also speaks MCP [Streamable HTTP](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#streamable-http) at `POST /mcp`, which is the network sibling of [`jaiph mcp`](mcp.md) stdio. It exposes the same tools, with identical [exposure rules](mcp.md#3-choose-which-defs-are-exposed) and comment-derived descriptions. It runs them through the same run registry, concurrency cap, and hot reload as the REST API. When a token is set, it sits behind the same bearer authentication as `/v1/*`. A single deployment serves REST clients, browsers, and MCP agents at once, with no second process. +The same process also speaks MCP [Streamable HTTP](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#streamable-http) at `POST /mcp`, which is the network sibling of [`jaiph mcp`](mcp.md) stdio. It exposes the same tools, with identical [exposure rules](mcp.md#3-choose-which-defs-are-exposed) and comment-derived descriptions. It runs them through the same run registry, concurrency cap, and hot reload as the REST API. When a token is set, it sits behind the same bearer authentication as the REST API. A single deployment serves REST clients, browsers, and MCP agents at once, with no second process. ```bash # One JSON-RPC message per POST: initialize, then tools/list, then tools/call. @@ -144,8 +144,8 @@ curl -s -X POST http://127.0.0.1:5247/mcp -H 'content-type: application/json' \ - **One JSON-RPC message per POST.** A request (`initialize`, `tools/list`, or `tools/call`) returns its reply as a single `application/json` object. A notification (`notifications/initialized` or `notifications/cancelled`) returns `202 Accepted` with no body. `GET` and `DELETE` on `/mcp` return `405`, because this endpoint offers no server-initiated stream. - **Progress streaming.** Send `Accept: text/event-stream` on a `tools/call` and include a `params._meta.progressToken`. You then receive the run's step boundaries as `notifications/progress` SSE frames, followed by the result frame. It uses the same progress model as [`jaiph mcp`](mcp.md#7-stream-progress-and-cancel-a-long-call). Without that `Accept` header, the call returns a single JSON result and progress is dropped. -- **Same run inspection.** Every `tools/call` is a run like any other. It appears in `GET /v1/runs`, streams at `GET /v1/runs/{id}/events`, and can be cancelled with `POST /v1/runs/{id}/cancel`, all in the same registry the REST endpoint populates. A client that hangs up a streaming call cancels the run, which tears down the child process tree, the same as an MCP `notifications/cancelled`. -- **Authentication.** `POST /mcp` sits behind the same authentication boundary as `/v1/*` (see [Authenticate and authorize](#7-authenticate-and-authorize)). Every request needs a bearer token, either static or OIDC/JWT, and an unauthenticated call gets `401`. Capabilities apply per method, so `tools/call` needs `invoke` and `tools/list` needs `inspect`. A principal without the needed scope gets a JSON-RPC error and nothing spawns, rather than an HTTP `403`. The authenticated principal and correlation id are attached to every run an MCP `tools/call` creates, exactly as for a REST run. +- **Same run inspection.** Every `tools/call` is a run like any other. It appears in `GET /runs`, streams at `GET /runs/{id}/events`, and can be cancelled with `POST /runs/{id}/cancel`, all in the same registry the REST endpoint populates. A client that hangs up a streaming call cancels the run, which tears down the child process tree, the same as an MCP `notifications/cancelled`. +- **Authentication.** `POST /mcp` sits behind the same authentication boundary as the REST API (see [Authenticate and authorize](#7-authenticate-and-authorize)). Every request needs a bearer token, either static or OIDC/JWT, and an unauthenticated call gets `401`. Capabilities apply per method, so `tools/call` needs `invoke` and `tools/list` needs `inspect`. A principal without the needed scope gets a JSON-RPC error and nothing spawns, rather than an HTTP `403`. The authenticated principal and correlation id are attached to every run an MCP `tools/call` creates, exactly as for a REST run. Point any Streamable HTTP MCP client at `http(s):///mcp`. Use `jaiph mcp` for a stdio client on the same machine, and use `jaiph serve` when MCP and REST clients both need to reach the defs over the network. @@ -155,34 +155,34 @@ The concurrency cap limits active children, not process memory. A long-lived ser - **Per-run output caps.** `JAIPH_SERVE_MAX_OUTPUT_BYTES` (default 1 MiB) caps collected stdout, stderr, log output, and the resident `result_text`, each independently. Output beyond a cap is dropped and replaced with a fixed `[jaiph: output truncated — exceeded the configured byte cap]` marker. A run that emits gigabytes therefore still costs bounded memory and returns a result that describes what happened. - **Completed-run retention.** The in-memory run registry keeps at most `JAIPH_SERVE_RETAIN_RUNS` completed runs (default 500), and it drops any completed run older than `JAIPH_SERVE_RETAIN_AGE_SEC` (default 24 hours). The oldest terminal records evict first, and active runs are never evicted. -- **Bounded listing.** `GET /v1/runs` is paginated with `?limit=` and `?offset=`. The default page is 100 and the hard maximum is 1000, so the endpoint can never return an unbounded response. +- **Bounded listing.** `GET /runs` is paginated with `?limit=` and `?offset=`. The default page is 100 and the hard maximum is 1000, so the endpoint can never return an unbounded response. -Eviction is in-memory only. Dropping a run from the registry does not delete its durable `run_summary.jsonl` journal or its published `artifacts/`. Both persist on disk under `JAIPH_RUNS_DIR`, which is an `emptyDir` or a PVC under Kubernetes, and pruning them is the operator's job. Once a run is evicted, its API endpoints (`GET /v1/runs/{id}`, `/events`, and `/artifacts`) return `404`, so read the durable artifacts from the filesystem instead. +Eviction is in-memory only. Dropping a run from the registry does not delete its durable `run_summary.jsonl` journal or its published `artifacts/`. Both persist on disk under `JAIPH_RUNS_DIR`, which is an `emptyDir` or a PVC under Kubernetes, and pruning them is the operator's job. Once a run is evicted, its API endpoints (`GET /runs/{id}`, `/events`, and `/artifacts`) return `404`, so read the durable artifacts from the filesystem instead. ## Restart-safe and retry-safe The run registry is in memory, but `jaiph serve` rebuilds it from disk on startup, so a restart does not lose run data. -- **Durable run records.** When a run finishes, `jaiph serve` writes its public record (`run.json`) atomically beside its journal in the run directory. On startup `jaiph serve` scans `JAIPH_RUNS_DIR` and reloads every `run.json`, so `GET /v1/runs`, `GET /v1/runs/{id}`, `/events`, and `/artifacts` keep answering for terminal runs that finished before the restart. As it reloads each run, `jaiph serve` verifies the run's keyed integrity chain. A run whose chain does not verify is loaded with status `failed` and a result that says the journal failed integrity verification, so a rewritten or truncated journal is surfaced as a failure rather than trusted. A run with no persisted key cannot be verified and is loaded unchanged. See [Architecture — Keyed hash chain](architecture.md#hash-chain). +- **Durable run records.** When a run finishes, `jaiph serve` writes its public record (`run.json`) atomically beside its journal in the run directory. On startup `jaiph serve` scans `JAIPH_RUNS_DIR` and reloads every `run.json`, so `GET /runs`, `GET /runs/{id}`, `/events`, and `/artifacts` keep answering for terminal runs that finished before the restart. As it reloads each run, `jaiph serve` verifies the run's keyed integrity chain. A run whose chain does not verify is loaded with status `failed` and a result that says the journal failed integrity verification, so a rewritten or truncated journal is surfaced as a failure rather than trusted. A run with no persisted key cannot be verified and is loaded unchanged. See [Architecture — Keyed hash chain](architecture.md#hash-chain). - **Interrupted runs are reconciled.** A run that was still `running` when the process died has a journal but no `run.json`. On startup `jaiph serve` reconciles it into the explicit terminal status `interrupted`. Its real outcome is unknown, so it is neither `succeeded` nor `failed`, but it is never reported as permanently `running`. Jaiph persists the reconciliation, so it stays stable across further restarts. -- **Idempotent run creation.** Send an `Idempotency-Key` request header on `POST /v1/defs/{name}/runs`. The key is scoped to the authenticated principal and the def. Repeating the request with the same key and identical arguments returns the original run (`200`) and starts nothing. Reusing the key with different arguments is a `409 E_IDEMPOTENCY_CONFLICT` and, again, spawns nothing. A client that retries an expensive run after a network blip or a server restart therefore never doubles it. The key and run mapping is stored in the durable record, so it survives a restart too. An idempotency key is remembered only as long as its run is retained in the registry. Once the retention bounds above evict a run, its key is forgotten, and a fresh request with that key starts a new run. +- **Idempotent run creation.** Send an `Idempotency-Key` request header on `POST /{name}`. The key is scoped to the authenticated principal and the def. Repeating the request with the same key and identical arguments returns the original run (`200`) and starts nothing. Reusing the key with different arguments is a `409 E_IDEMPOTENCY_CONFLICT` and, again, spawns nothing. A client that retries an expensive run after a network blip or a server restart therefore never doubles it. The key and run mapping is stored in the durable record, so it survives a restart too. An idempotency key is remembered only as long as its run is retained in the registry. Once the retention bounds above evict a run, its key is forgotten, and a fresh request with that key starts a new run. ```bash # The same key + same args returns the original run and spawns nothing. KEY=$(uuidgen) -curl -s -X POST 'http://127.0.0.1:5247/v1/defs/greet/runs?wait=true' \ +curl -s -X POST 'http://127.0.0.1:5247/greet?wait=true' \ -H 'content-type: application/json' -H "Idempotency-Key: $KEY" -d '{"name":"ok"}' | jq .run_id -curl -s -X POST 'http://127.0.0.1:5247/v1/defs/greet/runs?wait=true' \ +curl -s -X POST 'http://127.0.0.1:5247/greet?wait=true' \ -H 'content-type: application/json' -H "Idempotency-Key: $KEY" -d '{"name":"ok"}' | jq .run_id # same id # The same key + different args is a 409 conflict (spawns nothing). -curl -s -o /dev/null -w '%{http_code}\n' -X POST 'http://127.0.0.1:5247/v1/defs/greet/runs' \ +curl -s -o /dev/null -w '%{http_code}\n' -X POST 'http://127.0.0.1:5247/greet' \ -H 'content-type: application/json' -H "Idempotency-Key: $KEY" -d '{"name":"changed"}' # 409 ``` ## Deployment topology -`jaiph serve` is a single-replica service. Its run registry, in-flight concurrency cap, and idempotency index are per-process, with no shared store and no coordination between replicas. Running two or more replicas behind a load balancer is not supported. Each replica would see only its own runs, so `GET /v1/runs/{id}` would return `404` for a run another replica started. Each replica would also enforce `JAIPH_SERVE_MAX_CONCURRENT` on its own and keep a separate idempotency index, so the same `Idempotency-Key` could start one run per replica. Restart safety and retry safety hold within a single long-lived process that owns one `JAIPH_RUNS_DIR`. +`jaiph serve` is a single-replica service. Its run registry, in-flight concurrency cap, and idempotency index are per-process, with no shared store and no coordination between replicas. Running two or more replicas behind a load balancer is not supported. Each replica would see only its own runs, so `GET /runs/{id}` would return `404` for a run another replica started. Each replica would also enforce `JAIPH_SERVE_MAX_CONCURRENT` on its own and keep a separate idempotency index, so the same `Idempotency-Key` could start one run per replica. Restart safety and retry safety hold within a single long-lived process that owns one `JAIPH_RUNS_DIR`. Deploy exactly one replica. The [Kubernetes manifest](deploy.md#kubernetes) pins `replicas: 1` for this reason. Scale it vertically, with more CPU and memory and a higher `JAIPH_SERVE_MAX_CONCURRENT`, rather than horizontally. Point `JAIPH_RUNS_DIR` at a durable volume, a PVC rather than an `emptyDir`, if runs and their idempotency keys must survive pod replacement. Keep the pod a single instance with the `Recreate` strategy, so a rollout hands the runs directory to exactly one successor. @@ -190,12 +190,12 @@ Execution follows the same contract as [`jaiph run`](cli.md#jaiph-run): every ru ## Reverse-proxy and ingress requirements -`jaiph serve` speaks plain HTTP and holds long-lived streaming connections, both the SSE stream at `GET /v1/runs/{id}/events` and MCP progress at `POST /mcp` with `Accept: text/event-stream`. Front it with a reverse proxy or ingress that is configured for streaming and TLS, not only for request and response: +`jaiph serve` speaks plain HTTP and holds long-lived streaming connections, both the SSE stream at `GET /runs/{id}/events` and MCP progress at `POST /mcp` with `Accept: text/event-stream`. Front it with a reverse proxy or ingress that is configured for streaming and TLS, not only for request and response: -- **Disable response buffering on the streaming routes.** A proxy that buffers the whole response defeats live streaming, because clients see nothing until the run ends. On nginx, set `proxy_buffering off;` (or the `X-Accel-Buffering: no` header) on `/v1/runs/*/events` and `/mcp`. On Envoy or another ingress, disable response buffering for those paths. The server already sends `Cache-Control: no-cache` and a `:ka` keep-alive comment every 15 seconds on SSE, to keep intermediaries from idling the connection out. +- **Disable response buffering on the streaming routes.** A proxy that buffers the whole response defeats live streaming, because clients see nothing until the run ends. On nginx, set `proxy_buffering off;` (or the `X-Accel-Buffering: no` header) on `/runs/*/events` and `/mcp`. On Envoy or another ingress, disable response buffering for those paths. The server already sends `Cache-Control: no-cache` and a `:ka` keep-alive comment every 15 seconds on SSE, to keep intermediaries from idling the connection out. - **Raise read and idle timeouts to cover the longest run.** A `tools/call` or a `?wait=true` REST run blocks the connection until the def finishes, and an SSE follow stays open for the whole run. Set the proxy's upstream read timeout above your slowest def (nginx `proxy_read_timeout`, or a cloud load balancer's idle timeout), or those clients get cut off mid-run. Use `HTTP/1.1` on the streaming hops, rather than a buffered `HTTP/2` translation that coalesces frames. - **Terminate TLS at the proxy.** The process serves cleartext, so put HTTPS at the ingress or gateway in front of it, using cert-manager, a cloud load balancer, or a service mesh. Keep the app port private, on loopback or a `ClusterIP` Service (see [Deploy](deploy.md)). Never expose the token-guarded API to the internet without TLS, because the bearer token would travel in the clear. -- **Preserve and require authentication end to end.** Forward the `Authorization` header unchanged, and do not strip it. Terminate untrusted traffic at the proxy only if the proxy itself authenticates. Jaiph's own bearer check guards `/v1/*` and `/mcp`, while `/healthz`, `/openapi.json`, and `/docs` stay open for probes and discovery. If the proxy adds its own authentication, keep `JAIPH_SERVE_TOKEN` set anyway, so a proxy misconfiguration can never expose unauthenticated shell. +- **Preserve and require authentication end to end.** Forward the `Authorization` header unchanged, and do not strip it. Terminate untrusted traffic at the proxy only if the proxy itself authenticates. Jaiph's own bearer check guards the REST API and `/mcp`, while `/healthz`, `/openapi.json`, and `/docs` stay open for probes and discovery. If the proxy adds its own authentication, keep `JAIPH_SERVE_TOKEN` set anyway, so a proxy misconfiguration can never expose unauthenticated shell. ## Verification @@ -204,12 +204,12 @@ Execution follows the same contract as [`jaiph run`](cli.md#jaiph-run): every ru curl -s http://127.0.0.1:5247/healthz | jq -e '.status == "ok"' # A synchronous run round-trips its return value with a durable run dir. -curl -s -X POST 'http://127.0.0.1:5247/v1/defs/greet/runs?wait=true' \ +curl -s -X POST 'http://127.0.0.1:5247/greet?wait=true' \ -H 'content-type: application/json' -d '{"name":"ok"}' \ | jq -e '.status == "succeeded" and (.run_dir | length > 0)' # The run listing is bounded: a hostile limit is clamped to at most 1000 records. -curl -s 'http://127.0.0.1:5247/v1/runs?limit=100000' \ +curl -s 'http://127.0.0.1:5247/runs?limit=100000' \ | jq -e '.limit == 1000 and (.runs | length) <= 1000' ``` diff --git a/e2e/tests/147_serve_http_api.sh b/e2e/tests/147_serve_http_api.sh index affba131..cdbfd5b9 100755 --- a/e2e/tests/147_serve_http_api.sh +++ b/e2e/tests/147_serve_http_api.sh @@ -89,7 +89,7 @@ openapi_fields="$(printf '%s' "${openapi}" | python3 -c ' import json, sys d = json.load(sys.stdin) print(d["openapi"]) -print("yes" if "/v1/defs/greet/runs" in d["paths"] else "no") +print("yes" if "/greet" in d["paths"] else "no") ')" { read -r p_openapi_version @@ -125,7 +125,7 @@ docs_css_code="$(curl -s -o "${TEST_DIR}/swagger-ui.css" -w '%{http_code}' "${ba e2e::assert_equals "${docs_css_code}" "200" "the embedded swagger-ui stylesheet is served same-origin" # --- POST greet ?wait=true → succeeded, return value round-trips --- -greet="$(curl -s -X POST "${base}/v1/defs/greet/runs?wait=true" \ +greet="$(curl -s -X POST "${base}/greet?wait=true" \ -H 'content-type: application/json' -d '{"name":"world"}')" greet_fields="$(printf '%s' "${greet}" | python3 -c ' import json, sys @@ -146,62 +146,62 @@ e2e::assert_equals "${p_greet_has_rundir}" "yes" "greet run object carries a run # --- POST boom ?wait=true → HTTP 200 with status failed (not an HTTP error) --- boom_body="${TEST_DIR}/boom_body.json" boom_code="$(curl -s -o "${boom_body}" -w '%{http_code}' -X POST \ - "${base}/v1/defs/boom/runs?wait=true" -H 'content-type: application/json' -d '{}')" + "${base}/boom?wait=true" -H 'content-type: application/json' -d '{}')" e2e::assert_equals "${boom_code}" "200" "a failing workflow is HTTP 200 (workflow failure is not an HTTP error)" boom_status="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["status"])' "${boom_body}")" e2e::assert_equals "${boom_status}" "failed" "boom run status is failed" -# --- GET /v1/defs lists all exposed workflows --- -workflows="$(curl -s "${base}/v1/defs" | python3 -c ' +# --- GET /defs lists all exposed workflows --- +workflows="$(curl -s "${base}/defs" | python3 -c ' import json, sys names = sorted(w["name"] for w in json.load(sys.stdin)["defs"]) print(",".join(names)) ')" -e2e::assert_equals "${workflows}" "boom,greet,make_artifact" "GET /v1/defs lists all workflows" +e2e::assert_equals "${workflows}" "boom,greet,make_artifact" "GET /defs lists all workflows" -# --- GET /v1/runs is paginated (bounded listing) --- +# --- GET /runs is paginated (bounded listing) --- # Two runs exist by now (greet, boom); ?limit=1 must return exactly one record, # echo the requested limit, and report the full total so the response can never # be unbounded. Field-level checks: run objects carry volatile run_dir/timestamps. -runs_page="$(curl -s "${base}/v1/runs?limit=1" | python3 -c ' +runs_page="$(curl -s "${base}/runs?limit=1" | python3 -c ' import json, sys d = json.load(sys.stdin) print("%d,%d,%d" % (len(d["runs"]), d["limit"], d["total"])) ')" -e2e::assert_equals "${runs_page}" "1,1,2" "GET /v1/runs?limit=1 returns a bounded page with a total count" +e2e::assert_equals "${runs_page}" "1,1,2" "GET /runs?limit=1 returns a bounded page with a total count" -# --- GET /v1/runs/{id}/events (NDJSON) mirrors the durable journal --- +# --- GET /runs/{id}/events (NDJSON) mirrors the durable journal --- # The greet run above returned a run object; re-run it capturing the id, then # byte-compare the events endpoint against the on-disk run_summary.jsonl. -run_json="$(curl -s -X POST "${base}/v1/defs/greet/runs?wait=true" \ +run_json="$(curl -s -X POST "${base}/greet?wait=true" \ -H 'content-type: application/json' -d '{"name":"events"}')" run_id="$(printf '%s' "${run_json}" | python3 -c 'import json,sys; print(json.load(sys.stdin)["run_id"])')" run_dir="$(printf '%s' "${run_json}" | python3 -c 'import json,sys; print(json.load(sys.stdin)["run_dir"])')" events_body="${TEST_DIR}/events.ndjson" -curl -s "${base}/v1/runs/${run_id}/events" -o "${events_body}" +curl -s "${base}/runs/${run_id}/events" -o "${events_body}" # Full-content equality: the NDJSON stream is the journal, verbatim. e2e::assert_equals "$(cat "${events_body}")" "$(cat "${run_dir}/run_summary.jsonl")" \ "GET events (NDJSON) byte-matches the run's run_summary.jsonl" # --- artifacts: list + byte-identical download, traversal is rejected --- -art_json="$(curl -s -X POST "${base}/v1/defs/make_artifact/runs?wait=true" \ +art_json="$(curl -s -X POST "${base}/make_artifact?wait=true" \ -H 'content-type: application/json' -d '{}')" art_id="$(printf '%s' "${art_json}" | python3 -c 'import json,sys; print(json.load(sys.stdin)["run_id"])')" -art_list="$(curl -s "${base}/v1/runs/${art_id}/artifacts" | python3 -c ' +art_list="$(curl -s "${base}/runs/${art_id}/artifacts" | python3 -c ' import json, sys print(",".join(a["path"] for a in json.load(sys.stdin)["artifacts"])) ')" e2e::assert_equals "${art_list}" "result.txt" "GET artifacts lists the published file" art_file="${TEST_DIR}/downloaded.txt" -curl -s "${base}/v1/runs/${art_id}/artifacts/result.txt" -o "${art_file}" +curl -s "${base}/runs/${art_id}/artifacts/result.txt" -o "${art_file}" e2e::assert_equals "$(cat "${art_file}")" "artifact-payload" "artifact downloads byte-identically" # A URL-encoded `..` traversal escaping artifacts/ is a 404 (no bytes served). trav_code="$(curl -s -o /dev/null -w '%{http_code}' \ - "${base}/v1/runs/${art_id}/artifacts/%2e%2e%2frun_summary.jsonl")" + "${base}/runs/${art_id}/artifacts/%2e%2e%2frun_summary.jsonl")" e2e::assert_equals "${trav_code}" "404" "artifact path traversal is rejected with 404" # --- MCP Streamable HTTP (POST /mcp) on the SAME process --- @@ -239,7 +239,7 @@ e2e::assert_equals "${p_mcp_iserror}" "false" "POST /mcp tools/call reports isEr # The MCP call is a first-class run: it is the newest entry in the shared REST # registry, succeeded, and carries the same result_text the MCP client saw. -newest="$(curl -s "${base}/v1/runs?limit=1" | python3 -c ' +newest="$(curl -s "${base}/runs?limit=1" | python3 -c ' import json, sys r = json.load(sys.stdin)["runs"][0] print(r["def"]) @@ -251,6 +251,6 @@ print(r["result_text"]) read -r p_newest_status read -r p_newest_text } <<< "${newest}" -e2e::assert_equals "${p_newest_wf}" "greet" "the MCP call appears in the shared /v1/runs registry" +e2e::assert_equals "${p_newest_wf}" "greet" "the MCP call appears in the shared /runs registry" e2e::assert_equals "${p_newest_status}" "succeeded" "the MCP-initiated run status is succeeded" e2e::assert_equals "${p_newest_text}" "hello mcp" "the MCP-initiated run's result_text matches the tool result" diff --git a/e2e/tests/152_shell_injection_serve.sh b/e2e/tests/152_shell_injection_serve.sh index e6cf841e..bda015b9 100755 --- a/e2e/tests/152_shell_injection_serve.sh +++ b/e2e/tests/152_shell_injection_serve.sh @@ -67,14 +67,14 @@ fi base="http://127.0.0.1:${port}" # --- $(id) command substitution is not evaluated --- -run_json="$(curl -s -X POST "${base}/v1/defs/greet_shell/runs?wait=true" \ +run_json="$(curl -s -X POST "${base}/greet_shell?wait=true" \ -H 'content-type: application/json' -d '{"name":"$(id)"}')" run_id="$(printf '%s' "${run_json}" | python3 -c 'import json,sys; print(json.load(sys.stdin)["run_id"])')" run_status="$(printf '%s' "${run_json}" | python3 -c 'import json,sys; print(json.load(sys.stdin)["status"])')" e2e::assert_equals "${run_status}" "succeeded" "greet_shell run completes" art_file="${TEST_DIR}/downloaded_greeting.txt" -curl -s "${base}/v1/runs/${run_id}/artifacts/greeting.txt" -o "${art_file}" +curl -s "${base}/runs/${run_id}/artifacts/greeting.txt" -o "${art_file}" # Full-content equality: the $(id) text survives literally (shell-quoted), so # the byte content is fixed. If the substitution had run, this would contain the # host's `uid=…` and the equality would fail. @@ -84,7 +84,7 @@ e2e::assert_equals "$(cat "${art_file}")" 'Hello $\(id\)' \ # --- $(touch marker) does not create a file --- marker="${TEST_DIR}/pwned.txt" rm -f "${marker}" -curl -s -X POST "${base}/v1/defs/greet_shell/runs?wait=true" \ +curl -s -X POST "${base}/greet_shell?wait=true" \ -H 'content-type: application/json' \ -d "{\"name\":\"\$(touch ${marker})\"}" >/dev/null if [[ -f "${marker}" ]]; then diff --git a/integration/exec-policy.test.ts b/integration/exec-policy.test.ts index 9cda3a74..288959b7 100644 --- a/integration/exec-policy.test.ts +++ b/integration/exec-policy.test.ts @@ -101,7 +101,7 @@ async function runServeMode(ws: string, fixture: string, flags: string[], env: N return { exitCode: started.exitCode ?? 1, stderr: stderrBuf }; } try { - const res = await fetch(`${started.baseUrl}/v1/defs/probe_and_write/runs?wait=true`, { + const res = await fetch(`${started.baseUrl}/probe_and_write?wait=true`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({}), diff --git a/integration/otlp-export.test.ts b/integration/otlp-export.test.ts index ce90385c..f1ad9b2e 100644 --- a/integration/otlp-export.test.ts +++ b/integration/otlp-export.test.ts @@ -522,7 +522,7 @@ test("jaiph serve: an HTTP run exports exactly one trace via the shared call lay OTEL_EXPORTER_OTLP_ENDPOINT: `http://127.0.0.1:${collector.port}`, }); try { - const res = await fetch(`${serve.baseUrl}/v1/defs/greet/runs?wait=true`, { + const res = await fetch(`${serve.baseUrl}/greet?wait=true`, { method: "POST", headers: { "content-type": "application/json", authorization: `Bearer ${SERVE_TOKEN}` }, body: JSON.stringify({ name: "x" }), @@ -562,7 +562,7 @@ test("jaiph serve: an unreachable collector cannot delay a terminal ?wait=true r }); try { const started = Date.now(); - const res = await fetch(`${serve.baseUrl}/v1/defs/greet/runs?wait=true`, { + const res = await fetch(`${serve.baseUrl}/greet?wait=true`, { method: "POST", headers: { "content-type": "application/json", authorization: `Bearer ${SERVE_TOKEN}` }, body: JSON.stringify({ name: "x" }), diff --git a/integration/release-workflow.test.ts b/integration/release-workflow.test.ts index f8c2fe94..8b009630 100644 --- a/integration/release-workflow.test.ts +++ b/integration/release-workflow.test.ts @@ -22,6 +22,7 @@ import { tmpdir } from "node:os"; const REPO_ROOT = process.cwd(); const RELEASE_YML = readFileSync(join(REPO_ROOT, ".github/workflows/release.yml"), "utf8"); +const RUNNER_DOCKERFILE = readFileSync(join(REPO_ROOT, "runtime/Dockerfile"), "utf8"); const CONTRIBUTING = readFileSync(join(REPO_ROOT, "docs/contributing.md"), "utf8"); const INSTALLER = readFileSync(join(REPO_ROOT, "docs/install"), "utf8"); const INSTALLER_PS = readFileSync(join(REPO_ROOT, "docs/install.ps1"), "utf8"); @@ -142,7 +143,7 @@ test("version sanity gate only requires a version-shaped banner for nightly", () }); test("a windows-latest job runs the .exe --version through the shared gate and blocks publish", () => { - const job = sliceBetween(RELEASE_YML, "sanity-windows:", "\n release:"); + const job = sliceBetween(RELEASE_YML, "sanity-windows:", "\n publish-image:"); assert.match(job, /runs-on:\s*windows-latest/); assert.match(job, /jaiph-windows-x64\.exe --version/); assert.match(job, /release-version-check\.sh/, "windows gate delegates to the shared script"); @@ -286,3 +287,30 @@ test("release fails closed instead of publishing unsigned when MINISIGN_SECRET_K assert.match(signStep, /\n\s*exit 1\n/, "aborts the release job when the secret is unset"); assert.doesNotMatch(signStep, /skipping detached signature/i, "no silent skip that would publish unsigned"); }); + +// ── Acceptance 6: GHCR runner image (not a GitHub Release asset) ────────────── + +test("release publishes a GHCR runner image without adding it to SHA256SUMS", () => { + const job = sliceBetween(RELEASE_YML, " publish-image:", "\n release:"); + assert.match(job, /packages:\s*write/, "image job can push to GHCR"); + assert.match(job, /ghcr\.io/, "pushes to GHCR"); + assert.match(job, /jaiph-runtime/, "image name is jaiph-runtime"); + assert.match(job, /linux\/amd64,linux\/arm64/, "multi-arch linux"); + assert.match(job, /needs:\s*\[build, sanity-windows\]/, "image job waits for the windows gate"); + assert.doesNotMatch(job, /gh release/, "image is not a GitHub Release asset"); + const shaLine = RELEASE_YML.split("\n").find((l) => l.includes("sha256sum ") && l.includes("SHA256SUMS")); + assert.ok(shaLine, "found the sha256sum generation line"); + assert.doesNotMatch(shaLine!, /docker|jaiph-runtime|\.tar/, "SHA256SUMS stays binaries-only"); +}); + +test("runner Dockerfile installs the matching linux binary as uid 10001", () => { + assert.match(RUNNER_DOCKERFILE, /FROM ubuntu:24\.04/); + assert.match(RUNNER_DOCKERFILE, /COPY jaiph-linux-x64/); + assert.match(RUNNER_DOCKERFILE, /COPY jaiph-linux-arm64/); + assert.match(RUNNER_DOCKERFILE, /TARGETARCH/); + assert.match(RUNNER_DOCKERFILE, /useradd[^\n]*10001/); + assert.match(RUNNER_DOCKERFILE, /USER jaiph/); + assert.match(RUNNER_DOCKERFILE, /python3/); + assert.match(RUNNER_DOCKERFILE, /\bgit\b/); + assert.doesNotMatch(RUNNER_DOCKERFILE, /openjdk|golang|rustup|npm install/i); +}); diff --git a/integration/sentry-export.test.ts b/integration/sentry-export.test.ts index a15a5df8..310e9974 100644 --- a/integration/sentry-export.test.ts +++ b/integration/sentry-export.test.ts @@ -265,7 +265,7 @@ test("jaiph serve: a failed HTTP run delivers exactly one Sentry event via the s writeFileSync(jh, ['script boom = `echo "step output"; exit 3`', "# Fails on purpose.", "export def crash() {", " run boom()", "}", ""].join("\n")); const serve = await startServe(jh, root, { ...baseEnv(join(root, ".jaiph/runs")), SENTRY_DSN: dsn(sentry.port) }); try { - const res = await fetch(`${serve.baseUrl}/v1/defs/crash/runs?wait=true`, { + const res = await fetch(`${serve.baseUrl}/crash?wait=true`, { method: "POST", headers: { "content-type": "application/json", authorization: `Bearer ${SERVE_TOKEN}` }, body: "{}", diff --git a/integration/serve-auth.test.ts b/integration/serve-auth.test.ts index a37c075f..42672d7a 100644 --- a/integration/serve-auth.test.ts +++ b/integration/serve-auth.test.ts @@ -173,7 +173,7 @@ test("jaiph serve OIDC: token validity matrix (valid, expired, wrong-aud, wrong- }), ); const post = (token?: string): Promise => - fetch(`${srv.baseUrl}/v1/defs/greet/runs?wait=true`, { + fetch(`${srv.baseUrl}/greet?wait=true`, { method: "POST", headers: { "content-type": "application/json", ...(token ? bearer(token) : {}) }, body: JSON.stringify({ name: "x" }), @@ -249,7 +249,7 @@ test("jaiph serve OIDC: capabilities are separate, runs are per-principal, and i const noCancelTok = await idp.sign({ sub: "carol", scope: "jaiph:invoke jaiph:inspect" }); // Alice runs greet to completion. - const created = await fetch(`${srv.baseUrl}/v1/defs/greet/runs?wait=true`, { + const created = await fetch(`${srv.baseUrl}/greet?wait=true`, { method: "POST", headers: { "content-type": "application/json", ...bearer(aliceTok) }, body: JSON.stringify({ name: "world" }), @@ -260,23 +260,23 @@ test("jaiph serve OIDC: capabilities are separate, runs are per-principal, and i assert.equal(run.principal, "alice", "the run records its creating principal"); // Bob (valid, fully-scoped) cannot see or cancel Alice's run. - assert.equal((await fetch(`${srv.baseUrl}/v1/runs/${run.run_id}`, { headers: bearer(bobTok) })).status, 404); - assert.equal((await fetch(`${srv.baseUrl}/v1/runs/${run.run_id}/events`, { headers: bearer(bobTok) })).status, 404); - const bobList = await (await fetch(`${srv.baseUrl}/v1/runs`, { headers: bearer(bobTok) })).json(); + assert.equal((await fetch(`${srv.baseUrl}/runs/${run.run_id}`, { headers: bearer(bobTok) })).status, 404); + assert.equal((await fetch(`${srv.baseUrl}/runs/${run.run_id}/events`, { headers: bearer(bobTok) })).status, 404); + const bobList = await (await fetch(`${srv.baseUrl}/runs`, { headers: bearer(bobTok) })).json(); assert.equal(bobList.total, 0, "bob's listing does not include alice's run"); // Alice sees her own run. - assert.equal((await fetch(`${srv.baseUrl}/v1/runs/${run.run_id}`, { headers: bearer(aliceTok) })).status, 200); + assert.equal((await fetch(`${srv.baseUrl}/runs/${run.run_id}`, { headers: bearer(aliceTok) })).status, 200); // A principal without jaiph:cancel cannot cancel even its own in-flight run. - const slowStart = await fetch(`${srv.baseUrl}/v1/defs/slow/runs`, { + const slowStart = await fetch(`${srv.baseUrl}/slow`, { method: "POST", headers: { "content-type": "application/json", ...bearer(noCancelTok) }, body: "{}", }); assert.equal(slowStart.status, 202); const slowId = (await slowStart.json()).run_id; - const cancelDenied = await fetch(`${srv.baseUrl}/v1/runs/${slowId}/cancel`, { method: "POST", headers: bearer(noCancelTok) }); + const cancelDenied = await fetch(`${srv.baseUrl}/runs/${slowId}/cancel`, { method: "POST", headers: bearer(noCancelTok) }); assert.equal(cancelDenied.status, 403); assert.equal((await cancelDenied.json()).error.code, "E_FORBIDDEN"); @@ -314,7 +314,7 @@ test("jaiph serve OIDC: sub-less machine tokens get distinct client_id identitie const clientB = await idp.sign({ scope: fullScope, clientId: "service-b" }); // Client A runs greet; the run records client A's identity, never the shared "unknown". - const created = await fetch(`${srv.baseUrl}/v1/defs/greet/runs?wait=true`, { + const created = await fetch(`${srv.baseUrl}/greet?wait=true`, { method: "POST", headers: { "content-type": "application/json", ...bearer(clientA) }, body: JSON.stringify({ name: "world" }), @@ -326,17 +326,17 @@ test("jaiph serve OIDC: sub-less machine tokens get distinct client_id identitie assert.notEqual(run.principal, "unknown", "no principal collapses onto the shared constant"); // Client B (a distinct sub-less token) cannot enumerate or cancel client A's run. - assert.equal((await fetch(`${srv.baseUrl}/v1/runs/${run.run_id}`, { headers: bearer(clientB) })).status, 404); - const bList = await (await fetch(`${srv.baseUrl}/v1/runs`, { headers: bearer(clientB) })).json(); + assert.equal((await fetch(`${srv.baseUrl}/runs/${run.run_id}`, { headers: bearer(clientB) })).status, 404); + const bList = await (await fetch(`${srv.baseUrl}/runs`, { headers: bearer(clientB) })).json(); assert.equal(bList.total, 0, "client B's listing does not include client A's run"); - const cancelDenied = await fetch(`${srv.baseUrl}/v1/runs/${run.run_id}/cancel`, { method: "POST", headers: bearer(clientB) }); + const cancelDenied = await fetch(`${srv.baseUrl}/runs/${run.run_id}/cancel`, { method: "POST", headers: bearer(clientB) }); assert.equal(cancelDenied.status, 404, "client B cannot cancel client A's run"); // Client A still sees its own run. - assert.equal((await fetch(`${srv.baseUrl}/v1/runs/${run.run_id}`, { headers: bearer(clientA) })).status, 200); + assert.equal((await fetch(`${srv.baseUrl}/runs/${run.run_id}`, { headers: bearer(clientA) })).status, 200); // A verified token with neither `sub` nor `client_id` is rejected — never bucketed together. - const anon = await fetch(`${srv.baseUrl}/v1/defs/greet/runs?wait=true`, { + const anon = await fetch(`${srv.baseUrl}/greet?wait=true`, { method: "POST", headers: { "content-type": "application/json", ...bearer(await idp.sign({ scope: fullScope })) }, body: JSON.stringify({ name: "x" }), diff --git a/integration/serve-restart.test.ts b/integration/serve-restart.test.ts index 9fcbd581..a1c88188 100644 --- a/integration/serve-restart.test.ts +++ b/integration/serve-restart.test.ts @@ -77,7 +77,7 @@ function stop(child: ChildProcess, signal: NodeJS.Signals = "SIGTERM"): Promise< } async function getRun(baseUrl: string, id: string): Promise { - const res = await fetch(`${baseUrl}/v1/runs/${id}`); + const res = await fetch(`${baseUrl}/runs/${id}`); return { status: res.status, body: res.status === 200 ? await res.json() : null }; } @@ -93,7 +93,7 @@ test("jaiph serve: recovery + idempotency survive a real process restart", async try { // 1) A completed run with an idempotency key — the record we expect to // reconstruct after restart. - const created = await fetch(`${srv1.baseUrl}/v1/defs/greet/runs?wait=true`, { + const created = await fetch(`${srv1.baseUrl}/greet?wait=true`, { method: "POST", headers: { "content-type": "application/json", "idempotency-key": "idem-1" }, body: JSON.stringify({ name: "world" }), @@ -110,7 +110,7 @@ test("jaiph serve: recovery + idempotency survive a real process restart", async await delay(1200); // 2) A long run started async and left in flight, so a hard kill interrupts it. - const longRes = await fetch(`${srv1.baseUrl}/v1/defs/longflow/runs`, { + const longRes = await fetch(`${srv1.baseUrl}/longflow`, { method: "POST", headers: { "content-type": "application/json" }, body: "{}", @@ -125,7 +125,7 @@ test("jaiph serve: recovery + idempotency survive a real process restart", async // journal is what we want. Break once the journal is on disk. } // Confirm the journal exists via the events endpoint resolving a dir. - const ev = await fetch(`${srv1.baseUrl}/v1/runs/${longId}/events`); + const ev = await fetch(`${srv1.baseUrl}/runs/${longId}/events`); if (ev.status === 200 && (await ev.text()).includes("RUN_START")) break; await delay(100); } @@ -145,11 +145,11 @@ test("jaiph serve: recovery + idempotency survive a real process restart", async assert.equal(reloaded.body.result_text, "hi world"); // events + artifacts work for the reloaded run. - const ev = await fetch(`${srv2.baseUrl}/v1/runs/${terminalId}/events`); + const ev = await fetch(`${srv2.baseUrl}/runs/${terminalId}/events`); assert.equal(ev.status, 200); const journal = readFileSync(join(reloaded.body.run_dir, "run_summary.jsonl")); assert.deepEqual(Buffer.from(await ev.arrayBuffer()), journal, "NDJSON events byte-match the journal after restart"); - const arts = await fetch(`${srv2.baseUrl}/v1/runs/${terminalId}/artifacts`); + const arts = await fetch(`${srv2.baseUrl}/runs/${terminalId}/artifacts`); assert.equal(arts.status, 200); // (b) The interrupted run is reconciled out of `running`. @@ -159,26 +159,26 @@ test("jaiph serve: recovery + idempotency survive a real process restart", async // (c) Idempotency survives: same key + same args returns the ORIGINAL run, // spawning nothing new. - const before = (await (await fetch(`${srv2.baseUrl}/v1/runs`)).json()).total; - const replay = await fetch(`${srv2.baseUrl}/v1/defs/greet/runs?wait=true`, { + const before = (await (await fetch(`${srv2.baseUrl}/runs`)).json()).total; + const replay = await fetch(`${srv2.baseUrl}/greet?wait=true`, { method: "POST", headers: { "content-type": "application/json", "idempotency-key": "idem-1" }, body: JSON.stringify({ name: "world" }), }); assert.equal(replay.status, 200); assert.equal((await replay.json()).run_id, terminalId, "same key + args returns the reconstructed original run"); - const after = (await (await fetch(`${srv2.baseUrl}/v1/runs`)).json()).total; + const after = (await (await fetch(`${srv2.baseUrl}/runs`)).json()).total; assert.equal(after, before, "no new run was spawned by the idempotent replay"); // (d) Same key + changed args is a conflict that never spawns. - const conflict = await fetch(`${srv2.baseUrl}/v1/defs/greet/runs?wait=true`, { + const conflict = await fetch(`${srv2.baseUrl}/greet?wait=true`, { method: "POST", headers: { "content-type": "application/json", "idempotency-key": "idem-1" }, body: JSON.stringify({ name: "changed" }), }); assert.equal(conflict.status, 409); assert.equal((await conflict.json()).error.code, "E_IDEMPOTENCY_CONFLICT"); - const afterConflict = (await (await fetch(`${srv2.baseUrl}/v1/runs`)).json()).total; + const afterConflict = (await (await fetch(`${srv2.baseUrl}/runs`)).json()).total; assert.equal(afterConflict, before, "the conflicting request spawned nothing"); } finally { await stop(srv2.child, "SIGTERM"); diff --git a/integration/serve-server.test.ts b/integration/serve-server.test.ts index 00e969f5..3511c0dd 100644 --- a/integration/serve-server.test.ts +++ b/integration/serve-server.test.ts @@ -140,7 +140,7 @@ function delay(ms: number): Promise { async function pollRun(baseUrl: string, id: string, timeoutMs = 20_000): Promise { const start = Date.now(); for (;;) { - const res = await fetch(`${baseUrl}/v1/runs/${id}`); + const res = await fetch(`${baseUrl}/runs/${id}`); assert.equal(res.status, 200); const run = await res.json(); if (run.status !== "running") return run; @@ -155,7 +155,7 @@ test("jaiph serve: wait=true round-trips a workflow return value as succeeded", writeFileSync(jh, BASE_FIXTURE); const srv = await startServe(jh, root, serveEnv(join(root, ".jaiph/runs"))); try { - const res = await fetch(`${srv.baseUrl}/v1/defs/greet/runs?wait=true`, { + const res = await fetch(`${srv.baseUrl}/greet?wait=true`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ name: "world" }), @@ -177,7 +177,7 @@ test("jaiph serve: async POST returns 202 + Location and polling reaches the sam writeFileSync(jh, BASE_FIXTURE); const srv = await startServe(jh, root, serveEnv(join(root, ".jaiph/runs"))); try { - const res = await fetch(`${srv.baseUrl}/v1/defs/greet/runs`, { + const res = await fetch(`${srv.baseUrl}/greet`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ name: "async" }), @@ -186,7 +186,7 @@ test("jaiph serve: async POST returns 202 + Location and polling reaches the sam const location = res.headers.get("location"); const started = await res.json(); assert.equal(started.status, "running"); - assert.equal(location, `/v1/runs/${started.run_id}`); + assert.equal(location, `/runs/${started.run_id}`); const run = await pollRun(srv.baseUrl, started.run_id); assert.equal(run.status, "succeeded"); @@ -204,7 +204,7 @@ test("jaiph serve: a failing workflow is HTTP 200 with status failed (not an HTT writeFileSync(jh, BASE_FIXTURE); const srv = await startServe(jh, root, serveEnv(join(root, ".jaiph/runs"))); try { - const res = await fetch(`${srv.baseUrl}/v1/defs/boom/runs?wait=true`, { + const res = await fetch(`${srv.baseUrl}/boom?wait=true`, { method: "POST", headers: { "content-type": "application/json" }, body: "{}", @@ -229,7 +229,7 @@ test("jaiph serve: hot reload surfaces a new workflow and a pre-reload run still const srv = await startServe(jh, root, serveEnv(join(root, ".jaiph/runs"))); try { // Start a slow run against the current generation, then reload. - const startRes = await fetch(`${srv.baseUrl}/v1/defs/slow/runs`, { + const startRes = await fetch(`${srv.baseUrl}/slow`, { method: "POST", headers: { "content-type": "application/json" }, body: "{}", @@ -243,12 +243,12 @@ test("jaiph serve: hot reload surfaces a new workflow and a pre-reload run still const start = Date.now(); for (;;) { const doc = await (await fetch(`${srv.baseUrl}/openapi.json`)).json(); - if (doc.paths["/v1/defs/extra/runs"]) break; + if (doc.paths["/extra"]) break; if (Date.now() - start > 15_000) throw new Error("reload did not surface the new workflow in /openapi.json"); await delay(200); } - const wf = await (await fetch(`${srv.baseUrl}/v1/defs`)).json(); - assert.ok(wf.defs.some((w: any) => w.name === "extra"), "/v1/defs lists the new workflow"); + const wf = await (await fetch(`${srv.baseUrl}/defs`)).json(); + assert.ok(wf.defs.some((w: any) => w.name === "extra"), "/defs lists the new workflow"); // The run started before the reload still finishes successfully (its // generation's scripts dir survives until it completes — refcounted). @@ -261,15 +261,15 @@ test("jaiph serve: hot reload surfaces a new workflow and a pre-reload run still } }); -test("jaiph serve: with a token, /v1/* needs the bearer while /healthz, /openapi.json, /docs stay open", async () => { +test("jaiph serve: with a token, REST needs the bearer while /healthz, /openapi.json, /docs stay open", async () => { const root = mkdtempSync(join(tmpdir(), "jaiph-serve-auth-")); const jh = join(root, "tools.jh"); writeFileSync(jh, BASE_FIXTURE); const srv = await startServe(jh, root, serveEnv(join(root, ".jaiph/runs"), { JAIPH_SERVE_TOKEN: "s3cret" })); try { - assert.equal((await fetch(`${srv.baseUrl}/v1/defs`)).status, 401); - assert.equal((await fetch(`${srv.baseUrl}/v1/defs`, { headers: { authorization: "Bearer wrong" } })).status, 401); - assert.equal((await fetch(`${srv.baseUrl}/v1/defs`, { headers: { authorization: "Bearer s3cret" } })).status, 200); + assert.equal((await fetch(`${srv.baseUrl}/defs`)).status, 401); + assert.equal((await fetch(`${srv.baseUrl}/defs`, { headers: { authorization: "Bearer wrong" } })).status, 401); + assert.equal((await fetch(`${srv.baseUrl}/defs`, { headers: { authorization: "Bearer s3cret" } })).status, 200); for (const path of ["/healthz", "/openapi.json", "/docs"]) { assert.equal((await fetch(`${srv.baseUrl}${path}`)).status, 200, `${path} is open without a token`); @@ -295,12 +295,34 @@ test("jaiph serve: binding a non-loopback host without JAIPH_SERVE_TOKEN exits 1 }); assert.equal(result.status, 1, `expected exit 1, got ${result.status}\n${result.stderr}`); assert.match(result.stderr, /JAIPH_SERVE_TOKEN/); + assert.match(result.stderr, /--allow-anonymous/, "the error names the public opt-in"); assert.doesNotMatch(result.stderr, /listening on/, "must not bind before failing"); } finally { rmSync(root, { recursive: true, force: true }); } }); +test("jaiph serve: --allow-anonymous permits a non-loopback bind and warns it is open on the network", async () => { + const root = mkdtempSync(join(tmpdir(), "jaiph-serve-noloop-anon-")); + const jh = join(root, "tools.jh"); + writeFileSync(jh, BASE_FIXTURE); + const env = serveEnv(join(root, ".jaiph/runs")); + delete env.JAIPH_SERVE_TOKEN; + const srv = await startServe(jh, root, env, ["--host", "0.0.0.0"]); + try { + assert.match( + srv.stderr(), + /WARNING --allow-anonymous.*anyone who can reach the port/s, + "startup warns that a non-loopback anonymous bind is open on the network", + ); + const health = await fetch(`${srv.baseUrl}/healthz`); + assert.equal(health.status, 200); + } finally { + await srv.close(); + rmSync(root, { recursive: true, force: true }); + } +}); + test("jaiph serve: loopback with no auth and no --allow-anonymous exits 1 before listening (finding M-2)", () => { const root = mkdtempSync(join(tmpdir(), "jaiph-serve-anon-refuse-")); try { @@ -342,7 +364,7 @@ test("jaiph serve: --allow-anonymous starts, warns about open auth, and serves a /WARNING --allow-anonymous.*open to ALL local principals/s, "startup warns that the server is open to all local principals", ); - const created = await fetch(`${srv.baseUrl}/v1/defs/greet/runs?wait=true`, { + const created = await fetch(`${srv.baseUrl}/greet?wait=true`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ name: "world" }), @@ -378,7 +400,7 @@ test("jaiph serve: SSE events replay RUN_START, stream a STEP_END mid-run, then const srv = await startServe(jh, root, serveEnv(join(root, ".jaiph/runs"))); try { // Start async, then connect the event stream while the run is still going. - const startRes = await fetch(`${srv.baseUrl}/v1/defs/watchable/runs`, { + const startRes = await fetch(`${srv.baseUrl}/watchable`, { method: "POST", headers: { "content-type": "application/json" }, body: "{}", @@ -386,7 +408,7 @@ test("jaiph serve: SSE events replay RUN_START, stream a STEP_END mid-run, then assert.equal(startRes.status, 202); const runId = (await startRes.json()).run_id; - const evRes = await fetch(`${srv.baseUrl}/v1/runs/${runId}/events`, { headers: { accept: "text/event-stream" } }); + const evRes = await fetch(`${srv.baseUrl}/runs/${runId}/events`, { headers: { accept: "text/event-stream" } }); assert.equal(evRes.status, 200); assert.match(evRes.headers.get("content-type") ?? "", /text\/event-stream/); @@ -432,7 +454,7 @@ test("jaiph serve: NDJSON events on a terminal run byte-match the journal; unkno const srv = await startServe(jh, root, serveEnv(join(root, ".jaiph/runs"), { JAIPH_SERVE_TOKEN: "t0ken" })); const auth = { authorization: "Bearer t0ken" }; try { - const runRes = await fetch(`${srv.baseUrl}/v1/defs/greet/runs?wait=true`, { + const runRes = await fetch(`${srv.baseUrl}/greet?wait=true`, { method: "POST", headers: { "content-type": "application/json", ...auth }, body: JSON.stringify({ name: "nd" }), @@ -440,16 +462,16 @@ test("jaiph serve: NDJSON events on a terminal run byte-match the journal; unkno const run = await runRes.json(); assert.equal(run.status, "succeeded"); - const ev = await fetch(`${srv.baseUrl}/v1/runs/${run.run_id}/events`, { headers: auth }); + const ev = await fetch(`${srv.baseUrl}/runs/${run.run_id}/events`, { headers: auth }); assert.equal(ev.status, 200); assert.match(ev.headers.get("content-type") ?? "", /application\/x-ndjson/); const body = Buffer.from(await ev.arrayBuffer()); assert.deepEqual(body, readFileSync(join(run.run_dir, "run_summary.jsonl")), "NDJSON is byte-identical to the journal"); // Unknown run id → 404. - assert.equal((await fetch(`${srv.baseUrl}/v1/runs/does-not-exist/events`, { headers: auth })).status, 404); + assert.equal((await fetch(`${srv.baseUrl}/runs/does-not-exist/events`, { headers: auth })).status, 404); // Unauthenticated → 401. - assert.equal((await fetch(`${srv.baseUrl}/v1/runs/${run.run_id}/events`)).status, 401); + assert.equal((await fetch(`${srv.baseUrl}/runs/${run.run_id}/events`)).status, 401); } finally { await srv.close(); rmSync(root, { recursive: true, force: true }); @@ -463,7 +485,7 @@ test("jaiph serve: a credential echoed by a run is [REDACTED] in the event strea const secret = "supersecretvalue123"; const srv = await startServe(jh, root, serveEnv(join(root, ".jaiph/runs")), ["--env", `LEAK_API_KEY=${secret}`]); try { - const runRes = await fetch(`${srv.baseUrl}/v1/defs/leak_secret/runs?wait=true`, { + const runRes = await fetch(`${srv.baseUrl}/leak_secret?wait=true`, { method: "POST", headers: { "content-type": "application/json" }, body: "{}", @@ -471,7 +493,7 @@ test("jaiph serve: a credential echoed by a run is [REDACTED] in the event strea const run = await runRes.json(); assert.equal(run.status, "succeeded", `run failed: ${JSON.stringify(run)}`); - const ev = await fetch(`${srv.baseUrl}/v1/runs/${run.run_id}/events`); + const ev = await fetch(`${srv.baseUrl}/runs/${run.run_id}/events`); const journal = await ev.text(); assert.ok(!journal.includes(secret), "the raw credential value must not appear in the event stream"); assert.ok(journal.includes("[REDACTED]"), "the redaction marker is present where the value was"); @@ -488,7 +510,7 @@ test("jaiph serve: a credential passed as a step param is [REDACTED] in the even const secret = "supersecretparamvalue123"; const srv = await startServe(jh, root, serveEnv(join(root, ".jaiph/runs")), ["--env", `LEAK_API_KEY=${secret}`]); try { - const runRes = await fetch(`${srv.baseUrl}/v1/defs/leak_param/runs?wait=true`, { + const runRes = await fetch(`${srv.baseUrl}/leak_param?wait=true`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ token: secret }), @@ -496,7 +518,7 @@ test("jaiph serve: a credential passed as a step param is [REDACTED] in the even const run = await runRes.json(); assert.equal(run.status, "succeeded", `run failed: ${JSON.stringify(run)}`); - const ev = await fetch(`${srv.baseUrl}/v1/runs/${run.run_id}/events`); + const ev = await fetch(`${srv.baseUrl}/runs/${run.run_id}/events`); const journal = await ev.text(); assert.ok(!journal.includes(secret), "the raw credential value must not appear in step params in the event stream"); assert.ok(journal.includes("[REDACTED]"), "the redaction marker is present where the value was"); @@ -506,14 +528,14 @@ test("jaiph serve: a credential passed as a step param is [REDACTED] in the even } }); -test("jaiph serve: a failing run's result_text is [REDACTED] via wait=true and GET /v1/runs/{id}", async () => { +test("jaiph serve: a failing run's result_text is [REDACTED] via wait=true and GET /runs/{id}", async () => { const root = mkdtempSync(join(tmpdir(), "jaiph-serve-redact-fail-")); const jh = join(root, "tools.jh"); writeFileSync(jh, REDACT_FIXTURE); const secret = "supersecretvalue123"; const srv = await startServe(jh, root, serveEnv(join(root, ".jaiph/runs")), ["--env", `LEAK_API_KEY=${secret}`]); try { - const res = await fetch(`${srv.baseUrl}/v1/defs/leak_and_fail/runs?wait=true`, { + const res = await fetch(`${srv.baseUrl}/leak_and_fail?wait=true`, { method: "POST", headers: { "content-type": "application/json" }, body: "{}", @@ -525,9 +547,9 @@ test("jaiph serve: a failing run's result_text is [REDACTED] via wait=true and G assert.ok(run.result_text.includes("[REDACTED]"), "diagnostic context is retained with the marker"); assert.match(run.result_text, /failed step/, "failed-step diagnostics survive redaction"); - const again = await (await fetch(`${srv.baseUrl}/v1/runs/${run.run_id}`)).json(); + const again = await (await fetch(`${srv.baseUrl}/runs/${run.run_id}`)).json(); assert.equal(again.status, "failed"); - assert.ok(!again.result_text.includes(secret), "GET /v1/runs/{id} result_text must not contain the credential"); + assert.ok(!again.result_text.includes(secret), "GET /runs/{id} result_text must not contain the credential"); assert.ok(again.result_text.includes("[REDACTED]"), "the marker persists on the durable run object"); } finally { await srv.close(); @@ -541,7 +563,7 @@ test("jaiph serve: artifacts round-trip — list then byte-identical download, t writeFileSync(jh, BASE_FIXTURE); const srv = await startServe(jh, root, serveEnv(join(root, ".jaiph/runs"))); try { - const runRes = await fetch(`${srv.baseUrl}/v1/defs/make_artifact/runs?wait=true`, { + const runRes = await fetch(`${srv.baseUrl}/make_artifact?wait=true`, { method: "POST", headers: { "content-type": "application/json" }, body: "{}", @@ -549,14 +571,14 @@ test("jaiph serve: artifacts round-trip — list then byte-identical download, t const run = await runRes.json(); assert.equal(run.status, "succeeded", `run failed: ${JSON.stringify(run)}`); - const list = await (await fetch(`${srv.baseUrl}/v1/runs/${run.run_id}/artifacts`)).json(); + const list = await (await fetch(`${srv.baseUrl}/runs/${run.run_id}/artifacts`)).json(); assert.deepEqual( list.artifacts.map((a: any) => a.path), ["result.txt"], "the published file is listed", ); - const dl = await fetch(`${srv.baseUrl}/v1/runs/${run.run_id}/artifacts/result.txt`); + const dl = await fetch(`${srv.baseUrl}/runs/${run.run_id}/artifacts/result.txt`); assert.equal(dl.status, 200); assert.match(dl.headers.get("content-type") ?? "", /application\/octet-stream/); assert.match(dl.headers.get("content-disposition") ?? "", /filename="result\.txt"/); @@ -565,7 +587,7 @@ test("jaiph serve: artifacts round-trip — list then byte-identical download, t // Traversal battery: encoded `..`, `%2e%2e`, and an absolute path all 404, // and the run's own run_summary.jsonl (outside artifacts/) is unreachable. for (const escape of ["..%2Frun_summary.jsonl", "%2e%2e%2frun_summary.jsonl", "%2Fetc%2Fpasswd"]) { - const res = await fetch(`${srv.baseUrl}/v1/runs/${run.run_id}/artifacts/${escape}`); + const res = await fetch(`${srv.baseUrl}/runs/${run.run_id}/artifacts/${escape}`); assert.equal(res.status, 404, `traversal ${escape} → 404`); } } finally { @@ -590,7 +612,7 @@ test( writeFileSync(jh, BASE_FIXTURE); const srv = await startServe(jh, root, serveEnv(join(root, ".jaiph/runs"))); try { - const runRes = await fetch(`${srv.baseUrl}/v1/defs/make_artifact/runs?wait=true`, { + const runRes = await fetch(`${srv.baseUrl}/make_artifact?wait=true`, { method: "POST", headers: { "content-type": "application/json" }, body: "{}", @@ -606,7 +628,7 @@ test( ftruncateSync(fd, SIZE); closeSync(fd); - const dl = await fetch(`${srv.baseUrl}/v1/runs/${run.run_id}/artifacts/big.bin`); + const dl = await fetch(`${srv.baseUrl}/runs/${run.run_id}/artifacts/big.bin`); assert.equal(dl.status, 200); assert.equal(dl.headers.get("content-length"), String(SIZE)); diff --git a/runtime/.gitignore b/runtime/.gitignore new file mode 100644 index 00000000..9faa1e47 --- /dev/null +++ b/runtime/.gitignore @@ -0,0 +1,3 @@ +# Release / local docker context copies. Do not commit binaries. +jaiph-linux-x64 +jaiph-linux-arm64 diff --git a/runtime/Dockerfile b/runtime/Dockerfile new file mode 100644 index 00000000..152703de --- /dev/null +++ b/runtime/Dockerfile @@ -0,0 +1,23 @@ +# Lean runner image for `docker run` / Kubernetes. +# Not a sandbox and not a toolchain image. Layer agent CLIs (claude, cursor, +# codex) and any script dependencies yourself. +FROM ubuntu:24.04 + +RUN apt-get update && apt-get install -y --no-install-recommends \ + ca-certificates curl git python3 \ + && rm -rf /var/lib/apt/lists/* + +ARG TARGETARCH +COPY jaiph-linux-x64 /tmp/jaiph-linux-x64 +COPY jaiph-linux-arm64 /tmp/jaiph-linux-arm64 +RUN set -eu; \ + if [ "$TARGETARCH" = "arm64" ]; then src=/tmp/jaiph-linux-arm64; \ + else src=/tmp/jaiph-linux-x64; fi; \ + install -m 0755 "$src" /usr/local/bin/jaiph; \ + rm -f /tmp/jaiph-linux-x64 /tmp/jaiph-linux-arm64; \ + jaiph --version + +# Matches docs/deploy/k8s.yaml runAsUser 10001. +RUN useradd --uid 10001 --create-home --shell /bin/bash jaiph +USER jaiph +WORKDIR /work diff --git a/src/cli/commands/serve.ts b/src/cli/commands/serve.ts index 45721fe8..8ff6865f 100644 --- a/src/cli/commands/serve.ts +++ b/src/cli/commands/serve.ts @@ -49,34 +49,34 @@ const SERVE_USAGE = "when it is the only export, named after the file's basename. Descriptions come\n" + "from the `#` comment lines above each def. Sources are re-validated on change.\n\n" + "Endpoints: GET /docs (self-contained Swagger UI — assets embedded, no browser internet\n" + - "access needed), GET /openapi.json, GET /healthz, GET /v1/defs,\n" + - "POST /v1/defs/{name}/runs (async 202 or ?wait=true for 200), GET /v1/runs,\n" + - "GET /v1/runs/{id}, GET /v1/runs/{id}/events (NDJSON, or SSE with Accept: text/event-stream),\n" + - "GET /v1/runs/{id}/artifacts, GET /v1/runs/{id}/artifacts/{path}, POST /v1/runs/{id}/cancel.\n" + + "access needed), GET /openapi.json, GET /healthz, GET /defs,\n" + + "POST /{name} (async 202 or ?wait=true for 200), GET /runs,\n" + + "GET /runs/{id}, GET /runs/{id}/events (NDJSON, or SSE with Accept: text/event-stream),\n" + + "GET /runs/{id}/artifacts, GET /runs/{id}/artifacts/{path}, POST /runs/{id}/cancel.\n" + "MCP clients: POST /mcp speaks MCP Streamable HTTP over the same workflows, run\n" + "registry, concurrency cap, and auth — the network sibling of `jaiph mcp` stdio.\n\n" + - "Auth: JAIPH_SERVE_TOKEN sets a static single-operator bearer required on every /v1/* and\n" + + "Auth: JAIPH_SERVE_TOKEN sets a static single-operator bearer required on every REST and\n" + "/mcp request — single-operator, not multi-tenant. For per-user identity and authorization,\n" + "configure OIDC/JWT with JAIPH_SERVE_OIDC_ISSUER + JAIPH_SERVE_OIDC_AUDIENCE (JWKS discovered\n" + "from the issuer, or set JAIPH_SERVE_OIDC_JWKS_URI). OIDC tokens are authorized by scope —\n" + "jaiph:invoke (run), jaiph:inspect (read runs/artifacts), jaiph:cancel — and a principal may\n" + "inspect/cancel only its own runs. /healthz is always open and credential-free; /docs and\n" + - "/openapi.json are open unless JAIPH_SERVE_EXPOSE_DOCS=false. Binding a non-loopback host with\n" + - "no auth is a startup error. With no JAIPH_SERVE_TOKEN and no OIDC, even a loopback bind is a\n" + - "startup error unless --allow-anonymous is passed: anonymous mode authorizes every local\n" + - "principal with all capabilities over all runs, so it is for a single-user workstation only —\n" + - "shared hosts must set JAIPH_SERVE_TOKEN or configure OIDC. Cap concurrent runs with\n" + + "/openapi.json are open unless JAIPH_SERVE_EXPOSE_DOCS=false. With no JAIPH_SERVE_TOKEN and no\n" + + "OIDC, startup is refused unless --allow-anonymous is passed: anonymous mode authorizes every\n" + + "caller with all capabilities over all runs. The flag also permits a non-loopback bind (needed\n" + + "inside Docker). Shared or network-exposed hosts must set JAIPH_SERVE_TOKEN or configure OIDC.\n" + + "Cap concurrent runs with\n" + "JAIPH_SERVE_MAX_CONCURRENT (default 4). Bound memory with JAIPH_SERVE_MAX_OUTPUT_BYTES\n" + "(per-run stdout/stderr/log/result cap, default 1 MiB), JAIPH_SERVE_RETAIN_RUNS\n" + "(completed runs kept in memory, default 500), and JAIPH_SERVE_RETAIN_AGE_SEC\n" + - "(max completed-run age, default 86400; 0 disables). GET /v1/runs is paginated\n" + + "(max completed-run age, default 86400; 0 disables). GET /runs is paginated\n" + "(?limit default 100, max 1000; ?offset). Artifact downloads stream with\n" + "backpressure; JAIPH_SERVE_MAX_ARTIFACT_BYTES (default 0 = no cap) refuses\n" + "larger files with 413.\n\n" + " --host listen address (default: 127.0.0.1)\n" + " --port listen port (default: 5247)\n" + - " --allow-anonymous run open with no auth on loopback (single-user workstation only; every\n" + - " local user gets all capabilities over all runs). Ignored when\n" + + " --allow-anonymous run open with no auth (every caller gets all capabilities over all\n" + + " runs). Permits loopback and non-loopback binds. Ignored when\n" + " JAIPH_SERVE_TOKEN or OIDC is set.\n" + " --workspace workspace root for import resolution (default: auto-detect)\n" + " --env KEY=VALUE define KEY in every run's env (repeatable); --env KEY forwards the host value.\n" + @@ -122,42 +122,46 @@ export async function runServe(rest: string[]): Promise { : { token }; const authenticator = createAuthenticator(authConfig); - // Fail closed on exposure when no auth is configured. A non-loopback bind is - // always refused. Even loopback is refused unless the operator explicitly - // opts in with --allow-anonymous: in mode "none" every /v1/* and /mcp request - // is authorized as an anonymous principal holding all capabilities over all - // runs, and loopback is a boundary against the network, not against other - // local users — on a shared host any local user or process could invoke - // workflows and read every run's artifacts (finding M-2). All decided before - // any socket is opened. + // Fail closed when no auth is configured, unless the operator opts in with + // --allow-anonymous. In mode "none" every REST and /mcp request is + // authorized as an anonymous principal holding all capabilities over all + // runs. The flag also permits a non-loopback bind (Docker must listen on + // 0.0.0.0 for published ports). Decided before any socket is opened. if (!authenticator.enabled) { - if (!isLoopbackHost(host)) { - process.stderr.write( - `jaiph serve: refusing to bind non-loopback host "${host}" without authentication ` + - "(every /v1/* endpoint would be unauthenticated arbitrary shell). Set JAIPH_SERVE_TOKEN or configure " + - "OIDC (JAIPH_SERVE_OIDC_ISSUER + JAIPH_SERVE_OIDC_AUDIENCE) and retry.\n", - ); - return 1; - } if (!allowAnonymous) { + if (!isLoopbackHost(host)) { + process.stderr.write( + `jaiph serve: refusing to bind non-loopback host "${host}" without authentication ` + + "(every REST endpoint would be unauthenticated arbitrary shell). Set JAIPH_SERVE_TOKEN or configure " + + "OIDC (JAIPH_SERVE_OIDC_ISSUER + JAIPH_SERVE_OIDC_AUDIENCE), or pass --allow-anonymous to run open.\n", + ); + return 1; + } process.stderr.write( `jaiph serve: refusing to start on loopback host "${host}" with no authentication. ` + - "In anonymous mode every /v1/* and /mcp request is authorized as an anonymous principal with all " + + "In anonymous mode every REST and /mcp request is authorized as an anonymous principal with all " + "capabilities over all runs, so on a shared or multi-user host any other local user could invoke " + "workflows and read every run's artifacts (loopback guards the network, not other local users). " + "Set JAIPH_SERVE_TOKEN or configure OIDC (JAIPH_SERVE_OIDC_ISSUER + JAIPH_SERVE_OIDC_AUDIENCE), or " + - "pass --allow-anonymous to run open on a single-user workstation.\n", + "pass --allow-anonymous to run open.\n", ); return 1; } - // --allow-anonymous on loopback: warn loudly, before binding, that every - // local principal holds all capabilities over all runs (finding M-2). - process.stderr.write( - "jaiph serve: WARNING --allow-anonymous — no authentication configured. The server is open to ALL " + - "local principals: every /v1/* and /mcp request is authorized as an anonymous principal with all " + - "capabilities over all runs. Use this only on a single-user workstation; set JAIPH_SERVE_TOKEN or " + - "configure OIDC on any shared or multi-user host.\n", - ); + if (!isLoopbackHost(host)) { + process.stderr.write( + "jaiph serve: WARNING --allow-anonymous — no authentication configured. The server is bound on a " + + "non-loopback address and is open to anyone who can reach the port: every REST and /mcp request " + + "is authorized as an anonymous principal with all capabilities over all runs. Set JAIPH_SERVE_TOKEN " + + "or configure OIDC unless you intend this.\n", + ); + } else { + process.stderr.write( + "jaiph serve: WARNING --allow-anonymous — no authentication configured. The server is open to ALL " + + "local principals: every REST and /mcp request is authorized as an anonymous principal with all " + + "capabilities over all runs. Use this only on a single-user workstation; set JAIPH_SERVE_TOKEN or " + + "configure OIDC on any shared or multi-user host.\n", + ); + } } // Hide the API surface (/docs + /openapi.json) with JAIPH_SERVE_EXPOSE_DOCS=false. @@ -221,10 +225,12 @@ export async function runServe(rest: string[]): Promise { // keeps list/get/events/artifacts and idempotency working for prior runs. let initialRuns: ReturnType = []; try { + log(`jaiph serve: reconstructing runs from ${hostRunsRoot}...`); + const reconstructStarted = Date.now(); initialRuns = loadPersistedRuns(hostRunsRoot, new Date().toISOString()); - if (initialRuns.length > 0) { - log(`jaiph serve: reconstructed ${initialRuns.length} run(s) from ${hostRunsRoot}`); - } + log( + `jaiph serve: reconstructed ${initialRuns.length} run(s) from ${hostRunsRoot} in ${Date.now() - reconstructStarted}ms`, + ); } catch (err) { log(`jaiph serve: could not reconstruct prior runs: ${errText(err)}`); } diff --git a/src/cli/serve/auth.ts b/src/cli/serve/auth.ts index d2a8ffe5..8592c81f 100644 --- a/src/cli/serve/auth.ts +++ b/src/cli/serve/auth.ts @@ -8,8 +8,7 @@ import { createRemoteJWKSet, jwtVerify, errors as joseErrors, type JWTPayload, t * - **none** — no `JAIPH_SERVE_TOKEN` and no OIDC config. Every caller is the * `anonymous` principal with all capabilities. This mode is an explicit * single-user opt-in: `jaiph serve` refuses to start in it unless - * `--allow-anonymous` is passed, and only ever on loopback (binding a - * non-loopback host in this mode is a startup error regardless of the flag). + * `--allow-anonymous` is passed. The flag also permits a non-loopback bind. * - **static** — `JAIPH_SERVE_TOKEN` is a single shared secret. It is a * **single-operator** gate: the one `operator` principal holds every * capability and can inspect/cancel every run. It is NOT multi-tenant diff --git a/src/cli/serve/handler.test.ts b/src/cli/serve/handler.test.ts index 57eb5ba4..80425259 100644 --- a/src/cli/serve/handler.test.ts +++ b/src/cli/serve/handler.test.ts @@ -158,22 +158,22 @@ test("wrong method on a known path is 405", async () => { // === auth matrix (token set) === -test("auth matrix: /v1/* requires the bearer token when JAIPH_SERVE_TOKEN is set", async () => { +test("auth matrix: REST paths require the bearer token when JAIPH_SERVE_TOKEN is set", async () => { const h = makeHandler({ token: "secret" }); - const none = await h.handleRequest(req("GET", "/v1/defs")); + const none = await h.handleRequest(req("GET", "/defs")); assert.equal(none.status, 401); assert.equal(bodyJson(none).error.code, "E_UNAUTHORIZED"); - const wrong = await h.handleRequest(req("GET", "/v1/defs", { headers: { authorization: "Bearer nope" } })); + const wrong = await h.handleRequest(req("GET", "/defs", { headers: { authorization: "Bearer nope" } })); assert.equal(wrong.status, 401); - const right = await h.handleRequest(req("GET", "/v1/defs", { headers: { authorization: "Bearer secret" } })); + const right = await h.handleRequest(req("GET", "/defs", { headers: { authorization: "Bearer secret" } })); assert.equal(right.status, 200); assert.deepEqual(bodyJson(right).defs, [{ name: "build", description: "Builds the target.", params: ["target"] }]); }); -test("with no token configured, /v1/* is open (loopback default)", async () => { - const res = await makeHandler().handleRequest(req("GET", "/v1/defs")); +test("with no token configured, REST paths are open (loopback default)", async () => { + const res = await makeHandler().handleRequest(req("GET", "/defs")); assert.equal(res.status, 200); }); @@ -210,7 +210,7 @@ function tokenAuth(map: Record): Authenticator { test("authz: a principal without the invoke capability cannot create a run (403)", async () => { const h = makeHandler({ authenticator: principalAuth(principal("alice", ["inspect", "cancel"])) }); const res = await h.handleRequest( - req("POST", "/v1/defs/build/runs", { headers: { "content-type": "application/json" }, body: JSON.stringify({ target: "x" }) }), + req("POST", "/build", { headers: { "content-type": "application/json" }, body: JSON.stringify({ target: "x" }) }), ); assert.equal(res.status, 403); assert.equal(bodyJson(res).error.code, "E_FORBIDDEN"); @@ -218,9 +218,9 @@ test("authz: a principal without the invoke capability cannot create a run (403) test("authz: a principal without the inspect capability cannot list or read runs (403)", async () => { const h = makeHandler({ authenticator: principalAuth(principal("alice", ["invoke"])) }); - assert.equal((await h.handleRequest(req("GET", "/v1/runs"))).status, 403); - assert.equal((await h.handleRequest(req("GET", "/v1/defs"))).status, 403); - assert.equal((await h.handleRequest(req("GET", "/v1/runs/whatever"))).status, 403); + assert.equal((await h.handleRequest(req("GET", "/runs"))).status, 403); + assert.equal((await h.handleRequest(req("GET", "/defs"))).status, 403); + assert.equal((await h.handleRequest(req("GET", "/runs/whatever"))).status, 403); }); test("authz: a principal without the cancel capability cannot cancel a run it owns (403)", async () => { @@ -230,9 +230,9 @@ test("authz: a principal without the cancel capability cannot cancel a run it ow authenticator: principalAuth(p), callTool: () => new Promise(() => {}), }); - const start = bodyJson(await h.handleRequest(req("POST", "/v1/defs/ping/runs"))); + const start = bodyJson(await h.handleRequest(req("POST", "/ping"))); assert.equal(start.status, "running"); - const cancel = await h.handleRequest(req("POST", `/v1/runs/${start.run_id}/cancel`)); + const cancel = await h.handleRequest(req("POST", `/runs/${start.run_id}/cancel`)); assert.equal(cancel.status, 403); assert.equal(bodyJson(cancel).error.code, "E_FORBIDDEN"); }); @@ -247,21 +247,21 @@ test("authz: a principal cannot inspect or cancel another principal's runs (404, }); const aliceHdr = { authorization: "Bearer alice-tok" }; const bobHdr = { authorization: "Bearer bob-tok" }; - const run = bodyJson(await h.handleRequest(req("POST", "/v1/defs/ping/runs?wait=true", { headers: aliceHdr }))); + const run = bodyJson(await h.handleRequest(req("POST", "/ping?wait=true", { headers: aliceHdr }))); assert.equal(run.principal, "alice", "the run records its creating principal"); // Bob cannot see or cancel Alice's run — indistinguishable from nonexistent. - assert.equal((await h.handleRequest(req("GET", `/v1/runs/${run.run_id}`, { headers: bobHdr }))).status, 404); - assert.equal((await h.handleRequest(req("GET", `/v1/runs/${run.run_id}/events`, { headers: bobHdr }))).status, 404); - assert.equal((await h.handleRequest(req("GET", `/v1/runs/${run.run_id}/artifacts`, { headers: bobHdr }))).status, 404); - assert.equal((await h.handleRequest(req("POST", `/v1/runs/${run.run_id}/cancel`, { headers: bobHdr }))).status, 404); + assert.equal((await h.handleRequest(req("GET", `/runs/${run.run_id}`, { headers: bobHdr }))).status, 404); + assert.equal((await h.handleRequest(req("GET", `/runs/${run.run_id}/events`, { headers: bobHdr }))).status, 404); + assert.equal((await h.handleRequest(req("GET", `/runs/${run.run_id}/artifacts`, { headers: bobHdr }))).status, 404); + assert.equal((await h.handleRequest(req("POST", `/runs/${run.run_id}/cancel`, { headers: bobHdr }))).status, 404); // Bob's listing excludes it entirely. - assert.equal(bodyJson(await h.handleRequest(req("GET", "/v1/runs", { headers: bobHdr }))).total, 0); + assert.equal(bodyJson(await h.handleRequest(req("GET", "/runs", { headers: bobHdr }))).total, 0); // Alice sees her own run. - const aliceGet = await h.handleRequest(req("GET", `/v1/runs/${run.run_id}`, { headers: aliceHdr })); + const aliceGet = await h.handleRequest(req("GET", `/runs/${run.run_id}`, { headers: aliceHdr })); assert.equal(aliceGet.status, 200); - assert.equal(bodyJson(await h.handleRequest(req("GET", "/v1/runs", { headers: aliceHdr }))).total, 1); + assert.equal(bodyJson(await h.handleRequest(req("GET", "/runs", { headers: aliceHdr }))).total, 1); }); test("docs exposure can be disabled (404) while /healthz stays open", async () => { @@ -288,11 +288,11 @@ test("audit: invoke and cancel log the principal + correlation, never the bearer }, }); const hdr = { authorization: `Bearer ${TOKEN}`, "x-correlation-id": "cid-1" }; - const start = bodyJson(await h.handleRequest(req("POST", "/v1/defs/ping/runs", { headers: hdr }))); + const start = bodyJson(await h.handleRequest(req("POST", "/ping", { headers: hdr }))); assert.equal(start.principal, "alice", "the public run object carries the audit principal"); assert.equal(start.correlation_id, "cid-1"); - await h.handleRequest(req("POST", `/v1/runs/${start.run_id}/cancel`, { headers: { authorization: `Bearer ${TOKEN}` } })); + await h.handleRequest(req("POST", `/runs/${start.run_id}/cancel`, { headers: { authorization: `Bearer ${TOKEN}` } })); await runPromise.catch(() => {}); // Invoke audit rides the operator start banner emitted by the executor @@ -322,7 +322,7 @@ test("authz: an MCP SSE tools/call is owned and audited by the authenticated pri await res.stream!(fakeTarget()); // The run the deferred SSE stream created carries alice's identity — proof the // request context is re-established inside the stream body. - const listed = bodyJson(await h.handleRequest(req("GET", "/v1/runs", { headers: { authorization: "Bearer tok" } }))); + const listed = bodyJson(await h.handleRequest(req("GET", "/runs", { headers: { authorization: "Bearer tok" } }))); assert.equal(listed.total, 1); assert.equal(listed.runs[0].principal, "alice"); }); @@ -354,9 +354,24 @@ test("authz: MCP tools/call without invoke is a JSON-RPC authorization error and // === POST run: validation === -test("POST run for an unknown def is 404", async () => { +test("POST /{name} starts a run", async () => { + const h = makeHandler({ tools: [NOARG_TOOL] }); + const res = await h.handleRequest(req("POST", "/ping?wait=true")); + assert.equal(res.status, 200); + assert.equal(bodyJson(res).status, "succeeded"); +}); + +test("POST /v1/{name} and POST /defs/{name}/runs are 404", async () => { + const h = makeHandler({ tools: [NOARG_TOOL] }); + const prefixed = await h.handleRequest(req("POST", "/v1/ping?wait=true")); + assert.equal(prefixed.status, 404); + const nested = await h.handleRequest(req("POST", "/defs/ping/runs?wait=true")); + assert.equal(nested.status, 404); +}); + +test("POST /{name} for an unknown def is 404", async () => { const res = await makeHandler().handleRequest( - req("POST", "/v1/defs/nope/runs", { headers: { "content-type": "application/json" }, body: "{}" }), + req("POST", "/nope", { headers: { "content-type": "application/json" }, body: "{}" }), ); assert.equal(res.status, 404); assert.equal(bodyJson(res).error.code, "E_NOT_FOUND"); @@ -364,7 +379,7 @@ test("POST run for an unknown def is 404", async () => { test("POST run with a missing required param is 400", async () => { const res = await makeHandler().handleRequest( - req("POST", "/v1/defs/build/runs", { headers: { "content-type": "application/json" }, body: "{}" }), + req("POST", "/build", { headers: { "content-type": "application/json" }, body: "{}" }), ); assert.equal(res.status, 400); assert.equal(bodyJson(res).error.code, "E_BAD_ARGS"); @@ -373,7 +388,7 @@ test("POST run with a missing required param is 400", async () => { test("POST run with a non-string param is 400", async () => { const res = await makeHandler().handleRequest( - req("POST", "/v1/defs/build/runs", { headers: { "content-type": "application/json" }, body: JSON.stringify({ target: 5 }) }), + req("POST", "/build", { headers: { "content-type": "application/json" }, body: JSON.stringify({ target: 5 }) }), ); assert.equal(res.status, 400); assert.match(bodyJson(res).error.message, /target/); @@ -381,7 +396,7 @@ test("POST run with a non-string param is 400", async () => { test("POST run with an unexpected param key is 400", async () => { const res = await makeHandler().handleRequest( - req("POST", "/v1/defs/build/runs", { + req("POST", "/build", { headers: { "content-type": "application/json" }, body: JSON.stringify({ target: "x", bogus: "y" }), }), @@ -392,7 +407,7 @@ test("POST run with an unexpected param key is 400", async () => { test("POST run with a non-JSON content type is 415", async () => { const res = await makeHandler().handleRequest( - req("POST", "/v1/defs/build/runs", { headers: { "content-type": "text/plain" }, body: "target=x" }), + req("POST", "/build", { headers: { "content-type": "text/plain" }, body: "target=x" }), ); assert.equal(res.status, 415); assert.equal(bodyJson(res).error.code, "E_UNSUPPORTED_MEDIA_TYPE"); @@ -400,7 +415,7 @@ test("POST run with a non-JSON content type is 415", async () => { test("POST run past the body cap is 413", async () => { const res = await makeHandler().handleRequest( - req("POST", "/v1/defs/build/runs", { headers: { "content-type": "application/json" }, bodyTooLarge: true }), + req("POST", "/build", { headers: { "content-type": "application/json" }, bodyTooLarge: true }), ); assert.equal(res.status, 413); assert.equal(bodyJson(res).error.code, "E_BODY_TOO_LARGE"); @@ -410,12 +425,12 @@ test("concurrency cap returns 429 beyond the limit", async () => { // callTool never resolves, so the first run stays in-flight and the second is // rejected by the cap of 1. const h = makeHandler({ maxConcurrent: 1, callTool: () => new Promise(() => {}) }); - const first = await h.handleRequest(req("POST", "/v1/defs/ping/runs")); + const first = await h.handleRequest(req("POST", "/ping")); // ping has no params; empty body is fine. const hp = makeHandler({ maxConcurrent: 1, tools: [NOARG_TOOL], callTool: () => new Promise(() => {}) }); - const a = await hp.handleRequest(req("POST", "/v1/defs/ping/runs")); + const a = await hp.handleRequest(req("POST", "/ping")); assert.equal(a.status, 202); - const b = await hp.handleRequest(req("POST", "/v1/defs/ping/runs")); + const b = await hp.handleRequest(req("POST", "/ping")); assert.equal(b.status, 429); assert.equal(bodyJson(b).error.code, "E_TOO_MANY_RUNS"); void first; @@ -425,11 +440,11 @@ test("concurrency cap returns 429 beyond the limit", async () => { test("async POST returns 202 with a Location header and a running run object", async () => { const h = makeHandler({ tools: [NOARG_TOOL], callTool: () => new Promise(() => {}) }); - const res = await h.handleRequest(req("POST", "/v1/defs/ping/runs")); + const res = await h.handleRequest(req("POST", "/ping")); assert.equal(res.status, 202); const body = bodyJson(res); assert.equal(body.status, "running"); - assert.equal(res.headers.location, `/v1/runs/${body.run_id}`); + assert.equal(res.headers.location, `/runs/${body.run_id}`); }); test("?wait=true returns 200 with the terminal run object and result_text", async () => { @@ -437,7 +452,7 @@ test("?wait=true returns 200 with the terminal run object and result_text", asyn tools: [NOARG_TOOL], callTool: async () => ({ text: "hello world", isError: false, exitStatus: 0, runDir: "/runs/x" }), }); - const res = await h.handleRequest(req("POST", "/v1/defs/ping/runs?wait=true")); + const res = await h.handleRequest(req("POST", "/ping?wait=true")); assert.equal(res.status, 200); const body = bodyJson(res); assert.equal(body.status, "succeeded"); @@ -450,7 +465,7 @@ test("a workflow failure is not an HTTP error: 200 with status failed and exit_s tools: [NOARG_TOOL], callTool: async () => ({ text: "workflow ping failed (exit 1)\n\nrun dir: /runs/x", isError: true, exitStatus: 1, runDir: "/runs/x" }), }); - const res = await h.handleRequest(req("POST", "/v1/defs/ping/runs?wait=true")); + const res = await h.handleRequest(req("POST", "/ping?wait=true")); assert.equal(res.status, 200); const body = bodyJson(res); assert.equal(body.status, "failed"); @@ -462,7 +477,7 @@ test("a workflow failure is not an HTTP error: 200 with status failed and exit_s /** A create request carrying an Idempotency-Key header and JSON args. */ function idemReq(target: string, key: string, headers?: Record): ServeRequest { - return req("POST", "/v1/defs/build/runs?wait=true", { + return req("POST", "/build?wait=true", { headers: { "content-type": "application/json", "idempotency-key": key, ...headers }, body: JSON.stringify({ target }), }); @@ -511,7 +526,7 @@ test("idempotency: distinct keys, workflows, and principals are independent", as const r2 = bodyJson(await h.handleRequest(idemReq("app", "k2", auth))); assert.notEqual(r2.run_id, r1.run_id); // Same key but different workflow → new run (scope includes the workflow). - const otherWf = req("POST", "/v1/defs/other_wf/runs?wait=true", { + const otherWf = req("POST", "/other_wf?wait=true", { headers: { "content-type": "application/json", "idempotency-key": "k", ...auth }, body: JSON.stringify({ target: "app" }), }); @@ -557,7 +572,7 @@ test("initialRuns seed the registry so a restarted server serves prior terminal callTool: async () => ({ text: "fresh", isError: false, exitStatus: 0, runDir: "/runs/x" }), }); // GET works for the pre-restart run. - const got = bodyJson(await h.handleRequest(req("GET", "/v1/runs/prior-1"))); + const got = bodyJson(await h.handleRequest(req("GET", "/runs/prior-1"))); assert.equal(got.status, "succeeded"); assert.equal(got.result_text, "built earlier"); // Its idempotency key is honored across the restart: a matching replay @@ -574,7 +589,7 @@ test("persistRun fires with the terminal record at finalize", async () => { persistRun: (r) => persisted.push({ ...r }), callTool: async () => ({ text: "ok", isError: false, exitStatus: 0, runDir: "/runs/x" }), }); - await h.handleRequest(req("POST", "/v1/defs/ping/runs?wait=true")); + await h.handleRequest(req("POST", "/ping?wait=true")); assert.equal(persisted.length, 1); assert.equal(persisted[0].status, "succeeded"); assert.equal(persisted[0].run_dir, "/runs/x"); @@ -582,14 +597,14 @@ test("persistRun fires with the terminal record at finalize", async () => { // === run inspection === -test("GET /v1/runs/{id} for an unknown id is 404, and lists newest first", async () => { +test("GET /runs/{id} for an unknown id is 404, and lists newest first", async () => { const h = makeHandler({ tools: [NOARG_TOOL], callTool: async () => ({ text: "ok", isError: false, exitStatus: 0 }) }); - const notFound = await h.handleRequest(req("GET", "/v1/runs/does-not-exist")); + const notFound = await h.handleRequest(req("GET", "/runs/does-not-exist")); assert.equal(notFound.status, 404); - await h.handleRequest(req("POST", "/v1/defs/ping/runs?wait=true")); - await h.handleRequest(req("POST", "/v1/defs/ping/runs?wait=true")); - const list = bodyJson(await h.handleRequest(req("GET", "/v1/runs"))); + await h.handleRequest(req("POST", "/ping?wait=true")); + await h.handleRequest(req("POST", "/ping?wait=true")); + const list = bodyJson(await h.handleRequest(req("GET", "/runs"))); assert.equal(list.runs.length, 2); // newest first: run-1 then run-0 assert.equal(list.runs[0].run_id, "run-1"); @@ -601,12 +616,12 @@ test("GET /v1/runs/{id} for an unknown id is 404, and lists newest first", async test("count retention evicts only the oldest terminal records, keeping the newest", async () => { const h = makeHandler({ tools: [NOARG_TOOL], retainRuns: 2, callTool: async () => ({ text: "ok", isError: false, exitStatus: 0 }) }); for (let i = 0; i < 5; i += 1) { - await h.handleRequest(req("POST", "/v1/defs/ping/runs?wait=true")); + await h.handleRequest(req("POST", "/ping?wait=true")); } // 5 completed, retain 2 → only run-3 and run-4 (newest) survive. const ids = [...h.runs.keys()].sort(); assert.deepEqual(ids, ["run-3", "run-4"]); - const notFound = await h.handleRequest(req("GET", "/v1/runs/run-0")); + const notFound = await h.handleRequest(req("GET", "/runs/run-0")); assert.equal(notFound.status, 404, "evicted run is gone from the registry"); }); @@ -623,12 +638,12 @@ test("retention never evicts an active run even when the terminal budget is exce return Promise.resolve({ text: "ok", isError: false, exitStatus: 0 }); }, }); - const active = bodyJson(await h.handleRequest(req("POST", "/v1/defs/ping/runs"))); + const active = bodyJson(await h.handleRequest(req("POST", "/ping"))); assert.equal(active.status, "running"); // Complete three more runs; with retainRuns=1 they churn, but the active run // must never be evicted. for (let i = 0; i < 3; i += 1) { - await h.handleRequest(req("POST", "/v1/defs/ping/runs?wait=true")); + await h.handleRequest(req("POST", "/ping?wait=true")); } assert.ok(h.runs.has(active.run_id), "the running run survives eviction"); releaseActive({ text: "ok", isError: false, exitStatus: 0 }); @@ -644,40 +659,40 @@ test("age retention evicts a completed run once it is older than the window", as callTool: async () => ({ text: "ok", isError: false, exitStatus: 0 }), }); // First run ends at T0. - const first = bodyJson(await h.handleRequest(req("POST", "/v1/defs/ping/runs?wait=true"))); + const first = bodyJson(await h.handleRequest(req("POST", "/ping?wait=true"))); assert.ok(h.runs.has(first.run_id)); // Advance the clock 2 minutes; a new completion triggers age eviction of the // now-stale first run (ended > 60s ago). clock = "2026-07-24T00:02:00.000Z"; - await h.handleRequest(req("POST", "/v1/defs/ping/runs?wait=true")); + await h.handleRequest(req("POST", "/ping?wait=true")); assert.ok(!h.runs.has(first.run_id), "the stale completed run was evicted"); }); -// === /v1/runs pagination (bounded listing) === +// === /runs pagination (bounded listing) === -test("GET /v1/runs paginates: bounded default, stable newest-first order, total count", async () => { +test("GET /runs paginates: bounded default, stable newest-first order, total count", async () => { const h = makeHandler({ tools: [NOARG_TOOL], callTool: async () => ({ text: "ok", isError: false, exitStatus: 0 }) }); for (let i = 0; i < 5; i += 1) { - await h.handleRequest(req("POST", "/v1/defs/ping/runs?wait=true")); + await h.handleRequest(req("POST", "/ping?wait=true")); } - const page = bodyJson(await h.handleRequest(req("GET", "/v1/runs?limit=2"))); + const page = bodyJson(await h.handleRequest(req("GET", "/runs?limit=2"))); assert.equal(page.total, 5, "total reflects the full registry"); assert.equal(page.limit, 2); assert.equal(page.offset, 0); assert.deepEqual(page.runs.map((r: any) => r.run_id), ["run-4", "run-3"], "newest first"); - const next = bodyJson(await h.handleRequest(req("GET", "/v1/runs?limit=2&offset=2"))); + const next = bodyJson(await h.handleRequest(req("GET", "/runs?limit=2&offset=2"))); assert.deepEqual(next.runs.map((r: any) => r.run_id), ["run-2", "run-1"], "stable next page"); }); -test("GET /v1/runs clamps limit to the maximum and cannot return an unbounded response", async () => { +test("GET /runs clamps limit to the maximum and cannot return an unbounded response", async () => { const h = makeHandler({ tools: [NOARG_TOOL], callTool: async () => ({ text: "ok", isError: false, exitStatus: 0 }) }); - await h.handleRequest(req("POST", "/v1/defs/ping/runs?wait=true")); - const page = bodyJson(await h.handleRequest(req("GET", "/v1/runs?limit=999999"))); + await h.handleRequest(req("POST", "/ping?wait=true")); + const page = bodyJson(await h.handleRequest(req("GET", "/runs?limit=999999"))); assert.equal(page.limit, 1000, "limit clamped to MAX_RUNS_PAGE"); // A malformed limit falls back to the bounded default rather than being unbounded. - const bad = bodyJson(await h.handleRequest(req("GET", "/v1/runs?limit=abc"))); + const bad = bodyJson(await h.handleRequest(req("GET", "/runs?limit=abc"))); assert.equal(bad.limit, 100); }); @@ -700,28 +715,28 @@ test("cancel: 202 then terminal cancelled, invoking child + container teardown", return runPromise; }, }); - const start = await h.handleRequest(req("POST", "/v1/defs/ping/runs")); + const start = await h.handleRequest(req("POST", "/ping")); assert.equal(start.status, 202); const runId = bodyJson(start).run_id; - const cancel = await h.handleRequest(req("POST", `/v1/runs/${runId}/cancel`)); + const cancel = await h.handleRequest(req("POST", `/runs/${runId}/cancel`)); assert.equal(cancel.status, 202); assert.equal(childKilled, true, "child terminator ran"); assert.equal(containerStopped, true, "container teardown ran"); await runPromise.catch(() => {}); await flush(); - const record = bodyJson(await h.handleRequest(req("GET", `/v1/runs/${runId}`))); + const record = bodyJson(await h.handleRequest(req("GET", `/runs/${runId}`))); assert.equal(record.status, "cancelled"); }); test("cancel on an unknown run is 404; cancel on a terminal run is 409", async () => { const h = makeHandler({ tools: [NOARG_TOOL], callTool: async () => ({ text: "ok", isError: false, exitStatus: 0 }) }); - const missing = await h.handleRequest(req("POST", "/v1/runs/nope/cancel")); + const missing = await h.handleRequest(req("POST", "/runs/nope/cancel")); assert.equal(missing.status, 404); - const start = bodyJson(await h.handleRequest(req("POST", "/v1/defs/ping/runs?wait=true"))); - const again = await h.handleRequest(req("POST", `/v1/runs/${start.run_id}/cancel`)); + const start = bodyJson(await h.handleRequest(req("POST", "/ping?wait=true"))); + const again = await h.handleRequest(req("POST", `/runs/${start.run_id}/cancel`)); assert.equal(again.status, 409); assert.equal(bodyJson(again).error.code, "E_RUN_TERMINAL"); }); @@ -734,7 +749,7 @@ async function runWithDir(runDir: string): Promise<{ h: ServeHandler; runId: str tools: [NOARG_TOOL], callTool: async () => ({ text: "ok", isError: false, exitStatus: 0, runDir }), }); - const started = bodyJson(await h.handleRequest(req("POST", "/v1/defs/ping/runs?wait=true"))); + const started = bodyJson(await h.handleRequest(req("POST", "/ping?wait=true"))); return { h, runId: started.run_id }; } @@ -750,7 +765,7 @@ function fakeTarget(): StreamTarget & { chunks: string[] } { test("events + artifacts on an unknown run id are 404", async () => { const h = makeHandler(); - for (const path of ["/v1/runs/nope/events", "/v1/runs/nope/artifacts", "/v1/runs/nope/artifacts/x.txt"]) { + for (const path of ["/runs/nope/events", "/runs/nope/artifacts", "/runs/nope/artifacts/x.txt"]) { const res = await h.handleRequest(req("GET", path)); assert.equal(res.status, 404, `${path} → 404`); assert.equal(bodyJson(res).error.code, "E_NOT_FOUND"); @@ -759,7 +774,7 @@ test("events + artifacts on an unknown run id are 404", async () => { test("events + artifacts require the bearer token when one is configured", async () => { const h = makeHandler({ token: "secret" }); - for (const path of ["/v1/runs/x/events", "/v1/runs/x/artifacts", "/v1/runs/x/artifacts/f"]) { + for (const path of ["/runs/x/events", "/runs/x/artifacts", "/runs/x/artifacts/f"]) { assert.equal((await h.handleRequest(req("GET", path))).status, 401, `${path} → 401`); } }); @@ -770,7 +785,7 @@ test("NDJSON events on a terminal run stream the run_summary.jsonl file itself", const journal = '{"type":"RUN_START","run_id":"r"}\n{"type":"RUN_END","run_id":"r"}\n'; writeFileSync(join(runDir, "run_summary.jsonl"), journal); const { h, runId } = await runWithDir(runDir); - const res = await h.handleRequest(req("GET", `/v1/runs/${runId}/events`)); + const res = await h.handleRequest(req("GET", `/runs/${runId}/events`)); assert.equal(res.status, 200); assert.equal(res.headers["content-type"], "application/x-ndjson"); assert.ok(res.bodyFile, "NDJSON is streamed from a file, never buffered whole"); @@ -788,7 +803,7 @@ test("SSE events on a terminal run replay the journal then close with event: end const lines = ['{"type":"RUN_START","run_id":"r"}', '{"type":"RUN_END","run_id":"r"}']; writeFileSync(join(runDir, "run_summary.jsonl"), lines.map((l) => `${l}\n`).join("")); const { h, runId } = await runWithDir(runDir); - const res = await h.handleRequest(req("GET", `/v1/runs/${runId}/events`, { headers: { accept: "text/event-stream" } })); + const res = await h.handleRequest(req("GET", `/runs/${runId}/events`, { headers: { accept: "text/event-stream" } })); assert.equal(res.status, 200); assert.equal(res.headers["content-type"], "text/event-stream"); assert.ok(res.stream, "SSE is served as a stream"); @@ -811,14 +826,14 @@ test("artifacts round-trip through the handler: list then byte-identical downloa writeFileSync(join(runDir, "run_summary.jsonl"), ""); const { h, runId } = await runWithDir(runDir); - const list = await h.handleRequest(req("GET", `/v1/runs/${runId}/artifacts`)); + const list = await h.handleRequest(req("GET", `/runs/${runId}/artifacts`)); assert.equal(list.status, 200); assert.deepEqual( bodyJson(list).artifacts.map((a: any) => a.path), ["out.bin"], ); - const dl = await h.handleRequest(req("GET", `/v1/runs/${runId}/artifacts/out.bin`)); + const dl = await h.handleRequest(req("GET", `/runs/${runId}/artifacts/out.bin`)); assert.equal(dl.status, 200); assert.equal(dl.headers["content-type"], "application/octet-stream"); assert.match(dl.headers["content-disposition"], /filename="out\.bin"/); @@ -828,7 +843,7 @@ test("artifacts round-trip through the handler: list then byte-identical downloa assert.deepEqual(readFileSync(dl.bodyFile!.path), payload, "the streamed file is the published artifact"); // Traversal to a run-dir file (not under artifacts/) is a 404. - const escape = await h.handleRequest(req("GET", `/v1/runs/${runId}/artifacts/${encodeURIComponent("../run_summary.jsonl")}`)); + const escape = await h.handleRequest(req("GET", `/runs/${runId}/artifacts/${encodeURIComponent("../run_summary.jsonl")}`)); assert.equal(escape.status, 404); } finally { rmSync(runDir, { recursive: true, force: true }); @@ -880,8 +895,8 @@ test("a live SSE connection resolves the run dir with at most one scan", async ( }, ssePollMs: 5, }); - const started = bodyJson(await h.handleRequest(req("POST", "/v1/defs/ping/runs"))); - const res = await h.handleRequest(req("GET", `/v1/runs/${started.run_id}/events`, { headers: { accept: "text/event-stream" } })); + const started = bodyJson(await h.handleRequest(req("POST", "/ping"))); + const res = await h.handleRequest(req("GET", `/runs/${started.run_id}/events`, { headers: { accept: "text/event-stream" } })); assert.ok(res.stream); const target = abortableTarget(); const done = res.stream!(target); @@ -910,14 +925,14 @@ test("an artifact past maxArtifactBytes is 413 while a smaller one still streams maxArtifactBytes: 4, callTool: async () => ({ text: "ok", isError: false, exitStatus: 0, runDir }), }); - const started = bodyJson(await h.handleRequest(req("POST", "/v1/defs/ping/runs?wait=true"))); + const started = bodyJson(await h.handleRequest(req("POST", "/ping?wait=true"))); - const big = await h.handleRequest(req("GET", `/v1/runs/${started.run_id}/artifacts/big.bin`)); + const big = await h.handleRequest(req("GET", `/runs/${started.run_id}/artifacts/big.bin`)); assert.equal(big.status, 413); assert.equal(bodyJson(big).error.code, "E_ARTIFACT_TOO_LARGE"); assert.match(bodyJson(big).error.message, /JAIPH_SERVE_MAX_ARTIFACT_BYTES/); - const small = await h.handleRequest(req("GET", `/v1/runs/${started.run_id}/artifacts/small.bin`)); + const small = await h.handleRequest(req("GET", `/runs/${started.run_id}/artifacts/small.bin`)); assert.equal(small.status, 200); assert.equal(small.bodyFile!.size, 4); } finally { @@ -930,7 +945,7 @@ test("artifacts list is empty for a run with no published files", async () => { try { writeFileSync(join(runDir, "run_summary.jsonl"), ""); const { h, runId } = await runWithDir(runDir); - const list = await h.handleRequest(req("GET", `/v1/runs/${runId}/artifacts`)); + const list = await h.handleRequest(req("GET", `/runs/${runId}/artifacts`)); assert.deepEqual(bodyJson(list).artifacts, []); } finally { rmSync(runDir, { recursive: true, force: true }); @@ -987,7 +1002,7 @@ test("POST /mcp tools/call runs the workflow and registers it in the same run re // succeeded, and reused the injected executor exactly once. assert.equal(seen.length, 1); assert.equal(seen[0].def, "build"); - const listed = bodyJson(await h.handleRequest(req("GET", "/v1/runs"))); + const listed = bodyJson(await h.handleRequest(req("GET", "/runs"))); assert.equal(listed.total, 1); assert.equal(listed.runs[0].run_id, seen[0].runId); assert.equal(listed.runs[0].status, "succeeded"); @@ -1003,7 +1018,7 @@ test("POST /mcp tools/call failure comes back as isError, not a protocol error", const body = bodyJson(res); assert.equal(body.result.isError, true); assert.equal(body.result.content[0].text, "boom detail"); - assert.equal(bodyJson(await h.handleRequest(req("GET", "/v1/runs"))).runs[0].status, "failed"); + assert.equal(bodyJson(await h.handleRequest(req("GET", "/runs"))).runs[0].status, "failed"); }); test("POST /mcp tools/call obeys the shared concurrency cap (MCP analogue of 429)", async () => { @@ -1103,7 +1118,7 @@ test("POST /mcp cancel (notifications/cancelled) marks the shared run cancelled, mcpPost({ jsonrpc: "2.0", id: 9, method: "tools/call", params: { name: "ping", arguments: {} } }), ); await flush(); - const listed = bodyJson(await h.handleRequest(req("GET", "/v1/runs?limit=1"))); + const listed = bodyJson(await h.handleRequest(req("GET", "/runs?limit=1"))); assert.equal(listed.runs[0].status, "running"); const runId = listed.runs[0].run_id; @@ -1116,10 +1131,10 @@ test("POST /mcp cancel (notifications/cancelled) marks the shared run cancelled, // Cancelled requests produce no JSON-RPC response. assert.equal(settled.status, 202); assert.equal(settled.body, ""); - assert.equal(bodyJson(await h.handleRequest(req("GET", `/v1/runs/${runId}`))).status, "cancelled"); + assert.equal(bodyJson(await h.handleRequest(req("GET", `/runs/${runId}`))).status, "cancelled"); }); -test("POST /v1/runs/{id}/cancel cancels an MCP-initiated run in the shared registry", async () => { +test("POST /runs/{id}/cancel cancels an MCP-initiated run in the shared registry", async () => { let childKilled = false; let runPromise!: Promise; const h = makeHandler({ @@ -1138,14 +1153,14 @@ test("POST /v1/runs/{id}/cancel cancels an MCP-initiated run in the shared regis mcpPost({ jsonrpc: "2.0", id: 10, method: "tools/call", params: { name: "ping", arguments: {} } }), ); await flush(); - const runId = bodyJson(await h.handleRequest(req("GET", "/v1/runs?limit=1"))).runs[0].run_id; + const runId = bodyJson(await h.handleRequest(req("GET", "/runs?limit=1"))).runs[0].run_id; - const cancel = await h.handleRequest(req("POST", `/v1/runs/${runId}/cancel`)); + const cancel = await h.handleRequest(req("POST", `/runs/${runId}/cancel`)); assert.equal(cancel.status, 202); assert.equal(childKilled, true, "REST cancel tears down the MCP call's child"); await runPromise.catch(() => {}); await call; - assert.equal(bodyJson(await h.handleRequest(req("GET", `/v1/runs/${runId}`))).status, "cancelled"); + assert.equal(bodyJson(await h.handleRequest(req("GET", `/runs/${runId}`))).status, "cancelled"); }); test("POST /mcp SSE hangup cancels the run through the shared cancel path", async () => { @@ -1171,9 +1186,9 @@ test("POST /mcp SSE hangup cancels the run through the shared cancel path", asyn const target = abortableTarget(); const streaming = res.stream!(target); await flush(); - const runId = bodyJson(await h.handleRequest(req("GET", "/v1/runs?limit=1"))).runs[0].run_id; + const runId = bodyJson(await h.handleRequest(req("GET", "/runs?limit=1"))).runs[0].run_id; target.abort(); await runPromise.catch(() => {}); await streaming; - assert.equal(bodyJson(await h.handleRequest(req("GET", `/v1/runs/${runId}`))).status, "cancelled"); + assert.equal(bodyJson(await h.handleRequest(req("GET", `/runs/${runId}`))).status, "cancelled"); }); diff --git a/src/cli/serve/handler.ts b/src/cli/serve/handler.ts index 10ec18c0..a9ad634a 100644 --- a/src/cli/serve/handler.ts +++ b/src/cli/serve/handler.ts @@ -32,11 +32,18 @@ export type { RunStatus, RunRecord, ServeRequest, ServeResponse, ServeHandlerOpt /** 1 MiB cap on request bodies (design doc). */ export const MAX_BODY_BYTES = 1024 * 1024; -/** Default page size for `GET /v1/runs` when the caller gives no `limit`. */ +/** Default page size for `GET /runs` when the caller gives no `limit`. */ export const DEFAULT_RUNS_PAGE = 100; -/** Hard maximum page size for `GET /v1/runs` — a `limit` above this is clamped. */ +/** Hard maximum page size for `GET /runs` — a `limit` above this is clamped. */ export const MAX_RUNS_PAGE = 1000; +/** REST paths behind the same auth as `POST /mcp`. */ +function isRestApi(method: string, path: string): boolean { + if (path === "/defs" || path.startsWith("/defs/")) return true; + if (path === "/runs" || path.startsWith("/runs/")) return true; + return method === "POST" && /^\/[^/]+$/.test(path); +} + function isTerminal(status: RunStatus): boolean { return status === "succeeded" || status === "failed" || status === "cancelled" || status === "interrupted"; } @@ -175,7 +182,7 @@ export class ServeHandler { principal: principal.subject, correlationId: correlationId || undefined, onCancelHandle: (cancelFn) => { - // Wrap so every cancel path (REST /v1/.../cancel, MCP + // Wrap so every cancel path (REST /runs/.../cancel, MCP // notifications/cancelled, SSE hangup, cancelAll) marks the shared // record before killing the child — otherwise an MCP-only cancel // would finalize as `failed` instead of `cancelled`. @@ -254,20 +261,19 @@ export class ServeHandler { }; } - // The MCP Streamable HTTP endpoint and everything under /v1 share one - // authentication boundary. A verified request carries a Principal - // (capabilities + ownership) + correlation id through an AsyncLocalStorage, - // so authorization, ownership, audit, and telemetry read one identity - // consistently across both transports. + // MCP and the REST surface share one authentication boundary. A verified + // request carries a Principal (capabilities + ownership) + correlation id + // through an AsyncLocalStorage, so authorization, ownership, audit, and + // telemetry read one identity consistently across both transports. if (path === "/mcp") { const auth = await this.auth.authenticate(req.headers["authorization"]); if (!auth.ok) return this.error(auth.status, auth.code, auth.message); return this.reqCtx.run({ principal: auth.principal, correlationId: this.correlationOf(req) }, () => this.handleMcp(req)); } - if (path === "/v1" || path.startsWith("/v1/")) { + if (isRestApi(method, path)) { const auth = await this.auth.authenticate(req.headers["authorization"]); if (!auth.ok) return this.error(auth.status, auth.code, auth.message); - return this.reqCtx.run({ principal: auth.principal, correlationId: this.correlationOf(req) }, () => this.handleV1(req)); + return this.reqCtx.run({ principal: auth.principal, correlationId: this.correlationOf(req) }, () => this.handleApi(req)); } return this.error(404, "E_NOT_FOUND", `not found: ${path}`); @@ -311,59 +317,52 @@ export class ServeHandler { return record; } - private handleV1(req: ServeRequest): ServeResponse | Promise { + private handleApi(req: ServeRequest): ServeResponse | Promise { const { method, path } = req; const { principal } = this.currentCtx(); - if (path === "/v1/defs") { + if (path === "/defs") { if (method !== "GET") return this.methodNotAllowed(); if (!principal.capabilities.has("inspect")) return this.forbidden("inspect"); const defs = this.opts.getTools().map((t) => ({ name: t.name, description: t.description, params: t.params })); return this.json(200, { defs }); } - const runPost = /^\/v1\/defs\/([^/]+)\/runs$/.exec(path); - if (runPost) { - if (method !== "POST") return this.methodNotAllowed(); - if (!principal.capabilities.has("invoke")) return this.forbidden("invoke"); - return this.createRun(req, decodeURIComponent(runPost[1])); - } - - if (path === "/v1/runs") { + if (path === "/runs") { if (method !== "GET") return this.methodNotAllowed(); if (!principal.capabilities.has("inspect")) return this.forbidden("inspect"); return this.listRuns(req); } - const events = /^\/v1\/runs\/([^/]+)\/events$/.exec(path); + const events = /^\/runs\/([^/]+)\/events$/.exec(path); if (events) { if (method !== "GET") return this.methodNotAllowed(); if (!principal.capabilities.has("inspect")) return this.forbidden("inspect"); return this.runEvents(req, decodeURIComponent(events[1])); } - const artifactsList = /^\/v1\/runs\/([^/]+)\/artifacts$/.exec(path); + const artifactsList = /^\/runs\/([^/]+)\/artifacts$/.exec(path); if (artifactsList) { if (method !== "GET") return this.methodNotAllowed(); if (!principal.capabilities.has("inspect")) return this.forbidden("inspect"); return this.listRunArtifacts(decodeURIComponent(artifactsList[1])); } - const artifactGet = /^\/v1\/runs\/([^/]+)\/artifacts\/(.+)$/.exec(path); + const artifactGet = /^\/runs\/([^/]+)\/artifacts\/(.+)$/.exec(path); if (artifactGet) { if (method !== "GET") return this.methodNotAllowed(); if (!principal.capabilities.has("inspect")) return this.forbidden("inspect"); return this.downloadArtifact(decodeURIComponent(artifactGet[1]), artifactGet[2]); } - const cancel = /^\/v1\/runs\/([^/]+)\/cancel$/.exec(path); + const cancel = /^\/runs\/([^/]+)\/cancel$/.exec(path); if (cancel) { if (method !== "POST") return this.methodNotAllowed(); if (!principal.capabilities.has("cancel")) return this.forbidden("cancel"); return this.cancelRun(decodeURIComponent(cancel[1])); } - const getRun = /^\/v1\/runs\/([^/]+)$/.exec(path); + const getRun = /^\/runs\/([^/]+)$/.exec(path); if (getRun) { if (method !== "GET") return this.methodNotAllowed(); if (!principal.capabilities.has("inspect")) return this.forbidden("inspect"); @@ -372,6 +371,13 @@ export class ServeHandler { return this.json(200, this.toRunObject(record)); } + // POST /{name} — invoke. + const invoke = /^\/([^/]+)$/.exec(path); + if (invoke && method === "POST") { + if (!principal.capabilities.has("invoke")) return this.forbidden("invoke"); + return this.createRun(req, decodeURIComponent(invoke[1])); + } + return this.error(404, "E_NOT_FOUND", `not found: ${path}`); } @@ -465,7 +471,7 @@ export class ServeHandler { } return { status: 202, - headers: { "content-type": "application/json", location: `/v1/runs/${record.run_id}` }, + headers: { "content-type": "application/json", location: `/runs/${record.run_id}` }, body: JSON.stringify(this.toRunObject(record)), }; } @@ -561,7 +567,7 @@ export class ServeHandler { } /** - * `GET /v1/runs`: newest-first page of runs. `limit` defaults to + * `GET /runs`: newest-first page of runs. `limit` defaults to * {@link DEFAULT_RUNS_PAGE} and is clamped to `[1, MAX_RUNS_PAGE]`; `offset` * defaults to 0 (clamped to `>= 0`). The response can never be unbounded — * at most `MAX_RUNS_PAGE` records regardless of the query. Order is stable: @@ -645,7 +651,7 @@ export class ServeHandler { } /** - * `GET /v1/runs/{id}/events`. Default: the run's `run_summary.jsonl` as + * `GET /runs/{id}/events`. Default: the run's `run_summary.jsonl` as * `application/x-ndjson`, streamed verbatim (never buffered whole), then * close. `Accept: text/event-stream`: SSE replay + live follow until the run * is terminal. The journal's own redaction is the redaction guarantee; raw @@ -700,7 +706,7 @@ export class ServeHandler { }; } - /** `GET /v1/runs/{id}/artifacts`: JSON list of published files (empty when none). */ + /** `GET /runs/{id}/artifacts`: JSON list of published files (empty when none). */ private listRunArtifacts(id: string): ServeResponse { const record = this.lookupRun(id); if (!record) return this.error(404, "E_NOT_FOUND", "unknown run id"); @@ -709,7 +715,7 @@ export class ServeHandler { } /** - * `GET /v1/runs/{id}/artifacts/{path}`: download one published file as + * `GET /runs/{id}/artifacts/{path}`: download one published file as * `application/octet-stream`, streamed with backpressure — the complete * artifact is never buffered, so an arbitrarily large file costs no server * memory. `maxArtifactBytes > 0` refuses larger files with 413. diff --git a/src/cli/serve/openapi.test.ts b/src/cli/serve/openapi.test.ts index 0c1d99ee..933ab8d4 100644 --- a/src/cli/serve/openapi.test.ts +++ b/src/cli/serve/openapi.test.ts @@ -40,15 +40,18 @@ test("buildOpenApi emits one path per exposed workflow, honoring export narrowin assert.deepEqual(tools.map((t) => t.name), ["alpha"]); const doc = buildOpenApi(tools, SERVER_INFO) as any; - const workflowPaths = Object.keys(doc.paths).filter((p) => /^\/v1\/defs\/[^/]+\/runs$/.test(p)); - assert.deepEqual(workflowPaths, ["/v1/defs/alpha/runs"], "exactly one workflow path; beta is not exposed"); - assert.equal(doc.paths["/v1/defs/beta/runs"], undefined); + const reserved = new Set(["/defs", "/runs", "/healthz"]); + const workflowPaths = Object.keys(doc.paths).filter((p) => /^\/[^/]+$/.test(p) && !reserved.has(p)); + assert.deepEqual(workflowPaths, ["/alpha"], "exactly one workflow path; beta is not exposed"); + assert.equal(doc.paths["/beta"], undefined); + assert.equal(doc.paths["/defs/alpha/runs"], undefined); + assert.equal(doc.paths["/v1/alpha"], undefined); }); test("each workflow path carries the exact MCP-derived input schema as its JSON request body", () => { const tools = toolsFrom(EXPORT_NARROWED); const doc = buildOpenApi(tools, SERVER_INFO) as any; - const op = doc.paths["/v1/defs/alpha/runs"].post; + const op = doc.paths["/alpha"].post; assert.equal(op.operationId, "run_alpha"); assert.deepEqual(op.requestBody.content["application/json"].schema, tools[0].inputSchema); // Bearer security is applied to the workflow operation. @@ -63,7 +66,7 @@ test("the document pins info + bearer scheme + run/error component schemas", () assert.ok(doc.components.schemas.Run, "Run schema present"); assert.ok(doc.components.schemas.Error, "Error schema present"); // Static run-resource paths are present. - for (const p of ["/v1/defs", "/v1/runs", "/v1/runs/{id}", "/v1/runs/{id}/cancel", "/healthz"]) { + for (const p of ["/defs", "/runs", "/runs/{id}", "/runs/{id}/cancel", "/healthz"]) { assert.ok(doc.paths[p], `path ${p} present`); } }); diff --git a/src/cli/serve/openapi.ts b/src/cli/serve/openapi.ts index 91658148..d4a0b6f3 100644 --- a/src/cli/serve/openapi.ts +++ b/src/cli/serve/openapi.ts @@ -8,7 +8,7 @@ export interface OpenApiServerInfo { version: string; } -/** A bearer-secured operation on the `/v1/*` surface. */ +/** A bearer-secured operation on the REST surface. */ const BEARER_SECURITY = [{ bearer: [] as string[] }]; /** Standard `{error:{code,message}}` responses, keyed by HTTP status. */ @@ -31,12 +31,11 @@ function runResponse(description: string): Record { * the same `(tools, serverInfo)` always yields the same document, so it can be * regenerated per request and picks up hot-reloaded tool sets for free. * - * One concrete path per workflow (`/v1/defs//runs`) carries that - * workflow's own `operationId`, `#`-comment description, and the exact - * MCP-derived input schema as its JSON request body — which is what makes - * Swagger UI render a usable per-workflow form. The static run-resource paths, - * the run/error component schemas, and the bearer security scheme complete the - * document. + * One concrete path per workflow (`/`) carries that workflow's own + * `operationId`, `#`-comment description, and the exact MCP-derived input + * schema as its JSON request body — which is what makes Swagger UI render a + * usable per-workflow form. The static run-resource paths, the run/error + * component schemas, and the bearer security scheme complete the document. */ export function buildOpenApi(tools: McpToolSpec[], serverInfo: OpenApiServerInfo): Record { const paths: Record = {}; @@ -49,7 +48,7 @@ export function buildOpenApi(tools: McpToolSpec[], serverInfo: OpenApiServerInfo content: { "application/json": { schema: tool.inputSchema } }, } : { required: false, content: { "application/json": { schema: tool.inputSchema } } }; - paths[`/v1/defs/${tool.name}/runs`] = { + paths[`/${tool.name}`] = { post: { operationId: `run_${tool.name}`, summary: `Run the ${tool.name} def`, @@ -115,7 +114,7 @@ export function buildOpenApi(tools: McpToolSpec[], serverInfo: OpenApiServerInfo }, }; - paths["/v1/defs"] = { + paths["/defs"] = { get: { operationId: "listDefs", summary: "List exposed defs", @@ -151,7 +150,7 @@ export function buildOpenApi(tools: McpToolSpec[], serverInfo: OpenApiServerInfo }, }; - paths["/v1/runs"] = { + paths["/runs"] = { get: { operationId: "listRuns", summary: "List runs started by this server (newest first, paginated)", @@ -195,7 +194,7 @@ export function buildOpenApi(tools: McpToolSpec[], serverInfo: OpenApiServerInfo }, }; - paths["/v1/runs/{id}"] = { + paths["/runs/{id}"] = { get: { operationId: "getRun", summary: "Fetch one run", @@ -209,7 +208,7 @@ export function buildOpenApi(tools: McpToolSpec[], serverInfo: OpenApiServerInfo }, }; - paths["/v1/runs/{id}/events"] = { + paths["/runs/{id}/events"] = { get: { operationId: "getRunEvents", summary: "Stream a run's event journal", @@ -234,7 +233,7 @@ export function buildOpenApi(tools: McpToolSpec[], serverInfo: OpenApiServerInfo }, }; - paths["/v1/runs/{id}/artifacts"] = { + paths["/runs/{id}/artifacts"] = { get: { operationId: "listRunArtifacts", summary: "List a run's published artifacts", @@ -272,7 +271,7 @@ export function buildOpenApi(tools: McpToolSpec[], serverInfo: OpenApiServerInfo }, }; - paths["/v1/runs/{id}/artifacts/{path}"] = { + paths["/runs/{id}/artifacts/{path}"] = { get: { operationId: "downloadRunArtifact", summary: "Download one published artifact", @@ -292,7 +291,7 @@ export function buildOpenApi(tools: McpToolSpec[], serverInfo: OpenApiServerInfo }, }; - paths["/v1/runs/{id}/cancel"] = { + paths["/runs/{id}/cancel"] = { post: { operationId: "cancelRun", summary: "Cancel an in-flight run", diff --git a/src/cli/serve/server.test.ts b/src/cli/serve/server.test.ts index 802087f2..fcf7a26e 100644 --- a/src/cli/serve/server.test.ts +++ b/src/cli/serve/server.test.ts @@ -93,7 +93,7 @@ test("destroying a request mid-upload occupies no run slot and the server keeps await new Promise((r) => socket.on("connect", () => r())); // Declare a bigger body than we send, then vanish mid-upload. socket.write( - "POST /v1/defs/ping/runs HTTP/1.1\r\n" + + "POST /ping HTTP/1.1\r\n" + "host: localhost\r\n" + "content-type: application/json\r\n" + "content-length: 64\r\n" + @@ -128,14 +128,14 @@ async function serveArtifact(payloadPath: string, payload: Buffer | null): Promi const handler = makeHandler(async () => ({ text: "ok", isError: false, exitStatus: 0, runDir })); const server = createHttpServer(handler, () => {}); const port = await listen(server, "127.0.0.1", 0); - const res = await fetch(`http://127.0.0.1:${port}/v1/defs/ping/runs?wait=true`, { method: "POST" }); + const res = await fetch(`http://127.0.0.1:${port}/ping?wait=true`, { method: "POST" }); const runId = ((await res.json()) as { run_id: string }).run_id; return { server, port, runId, runDir }; } // Finding H-3: the events endpoint must not stream a run whose keyed journal // chain fails verification. -test("GET /v1/runs/{id}/events hard-fails (409) on a tampered journal, streams a clean one", async () => { +test("GET /runs/{id}/events hard-fails (409) on a tampered journal, streams a clean one", async () => { const runDir = mkdtempSync(join(tmpdir(), "jaiph-srv-events-")); try { // A journal with no valid keyed chain, plus a persisted key → verifiable. @@ -145,10 +145,10 @@ test("GET /v1/runs/{id}/events hard-fails (409) on a tampered journal, streams a const server = createHttpServer(handler, () => {}); const port = await listen(server, "127.0.0.1", 0); try { - const create = await fetch(`http://127.0.0.1:${port}/v1/defs/ping/runs?wait=true`, { method: "POST" }); + const create = await fetch(`http://127.0.0.1:${port}/ping?wait=true`, { method: "POST" }); const runId = ((await create.json()) as { run_id: string }).run_id; - const tampered = await fetch(`http://127.0.0.1:${port}/v1/runs/${runId}/events`); + const tampered = await fetch(`http://127.0.0.1:${port}/runs/${runId}/events`); assert.equal(tampered.status, 409, "a tampered journal is rejected, not served"); assert.equal(((await tampered.json()) as { error: { code: string } }).error.code, "E_TAMPERED"); @@ -159,7 +159,7 @@ test("GET /v1/runs/{id}/events hard-fails (409) on a tampered journal, streams a const l0 = JSON.stringify({ type: "RUN_START", prev_hash: chainHmac("k".repeat(64), CHAIN_GENESIS) }); const l1 = JSON.stringify({ type: "RUN_END", prev_hash: chainHmac("k".repeat(64), l0) }); writeFileSync(join(runDir, "run_summary.jsonl"), `${l0}\n${l1}\n`); - const clean = await fetch(`http://127.0.0.1:${port}/v1/runs/${runId}/events`); + const clean = await fetch(`http://127.0.0.1:${port}/runs/${runId}/events`); assert.equal(clean.status, 200, "a verifying journal streams normally"); assert.match(clean.headers.get("content-type") ?? "", /application\/x-ndjson/); } finally { @@ -174,7 +174,7 @@ test("GET /v1/runs/{id}/events hard-fails (409) on a tampered journal, streams a // original four-suffix rule missed — is served as [REDACTED] on the /events // path. The journal is written through the real RuntimeEventEmitter so this // exercises the production redaction boundary end-to-end, not a hand-built line. -test("GET /v1/runs/{id}/events serves a newly-detected credential as [REDACTED]", async () => { +test("GET /runs/{id}/events serves a newly-detected credential as [REDACTED]", async () => { const runDir = mkdtempSync(join(tmpdir(), "jaiph-srv-redact-")); const secret = "sk_live_super_secret_value_123"; const prevSummaryFile = process.env.JAIPH_RUN_SUMMARY_FILE; @@ -195,9 +195,9 @@ test("GET /v1/runs/{id}/events serves a newly-detected credential as [REDACTED]" const server = createHttpServer(handler, () => {}); const port = await listen(server, "127.0.0.1", 0); try { - const create = await fetch(`http://127.0.0.1:${port}/v1/defs/ping/runs?wait=true`, { method: "POST" }); + const create = await fetch(`http://127.0.0.1:${port}/ping?wait=true`, { method: "POST" }); const runId = ((await create.json()) as { run_id: string }).run_id; - const res = await fetch(`http://127.0.0.1:${port}/v1/runs/${runId}/events`); + const res = await fetch(`http://127.0.0.1:${port}/runs/${runId}/events`); assert.equal(res.status, 200); const body = await res.text(); assert.ok(!body.includes(secret), "the secret must not appear in the served journal"); @@ -218,7 +218,7 @@ test("an artifact download round-trips byte-identically through a real socket wi for (let i = 0; i < payload.length; i += 1) payload[i] = i % 251; const { server, port, runId, runDir } = await serveArtifact("blob.bin", payload); try { - const dl = await fetch(`http://127.0.0.1:${port}/v1/runs/${runId}/artifacts/blob.bin`); + const dl = await fetch(`http://127.0.0.1:${port}/runs/${runId}/artifacts/blob.bin`); assert.equal(dl.status, 200); assert.equal(dl.headers.get("content-length"), String(payload.length)); assert.deepEqual(Buffer.from(await dl.arrayBuffer()), payload, "streamed bytes match the artifact"); @@ -254,7 +254,7 @@ test("disconnecting the client mid-download destroys the artifact file stream", // Never read the response: kernel + stream buffers fill and backpressure // pauses the file stream mid-transfer. socket.pause(); - socket.write(`GET /v1/runs/${runId}/artifacts/big.bin HTTP/1.1\r\nhost: localhost\r\n\r\n`); + socket.write(`GET /runs/${runId}/artifacts/big.bin HTTP/1.1\r\nhost: localhost\r\n\r\n`); await waitFor(() => created.length === 1, "the artifact file stream to open"); assert.equal(created[0].destroyed, false, "the stalled stream stays open while the client is connected"); socket.destroy(); diff --git a/src/cli/serve/types.ts b/src/cli/serve/types.ts index f55780f1..c6639fbe 100644 --- a/src/cli/serve/types.ts +++ b/src/cli/serve/types.ts @@ -101,7 +101,7 @@ export interface ServeHandlerOptions { ) => Promise; /** * Static single-operator bearer token. When set (and no `authenticator` is - * injected) every `/v1/*` and `/mcp` request must present it. Single-operator, + * injected) every REST and `/mcp` request must present it. Single-operator, * not multi-tenant — for per-user identity/authorization pass an `authenticator`. */ token?: string; diff --git a/src/cli/shared/serve-bootstrap.ts b/src/cli/shared/serve-bootstrap.ts index 2f1e554c..5556d250 100644 --- a/src/cli/shared/serve-bootstrap.ts +++ b/src/cli/shared/serve-bootstrap.ts @@ -1,4 +1,4 @@ -import { existsSync, mkdtempSync, rmSync, statSync } from "node:fs"; +import { existsSync, mkdtempSync, rmSync, statSync, writeSync } from "node:fs"; import { tmpdir } from "node:os"; import { dirname, extname, join, resolve } from "node:path"; import { errText } from "../../errors"; @@ -58,7 +58,8 @@ export function parseServerArgs( usage: string, ): { code: number } | { args: ServerArgs } { const log = (line: string): void => { - process.stderr.write(`${line}\n`); + // writeSync so Docker (no TTY) does not block-buffer startup lines. + writeSync(2, `${line}\n`); }; if (hasHelpFlag(rest)) { process.stdout.write(usage); @@ -117,6 +118,8 @@ export function startGeneration( let generations: GenerationTracker; try { + log(`${label}: loading module graph...`); + const graphStarted = Date.now(); const loaded = loadGeneration(inputAbs, workspaceRoot, tempRoot, 0, extraEnv, log, label); if (!loaded.state) { for (const f of loaded.failures) log(f); @@ -124,6 +127,7 @@ export function startGeneration( return { code: 1 }; } generations = createGenerationTracker(loaded.state); + log(`${label}: module graph ready in ${Date.now() - graphStarted}ms`); } catch (err) { log(errText(err)); cleanup(); diff --git a/src/cli/shared/usage.ts b/src/cli/shared/usage.ts index c0ce13e7..b08e5525 100644 --- a/src/cli/shared/usage.ts +++ b/src/cli/shared/usage.ts @@ -69,17 +69,16 @@ export function printUsage(): void { " Serve the file's defs as an HTTP API with a generated OpenAPI 3.1 document", " and a self-contained Swagger UI at /docs (assets embedded — no browser internet", " access needed). Exposure mirrors `jaiph mcp`. Runs are durable resources", - " under .jaiph/runs/: inspect one with GET /v1/runs/{id}, stream its event journal", - " (NDJSON or SSE) via GET /v1/runs/{id}/events, and list/download published files via", - " GET /v1/runs/{id}/artifacts[/{path}]. Set JAIPH_SERVE_TOKEN to", - " require a bearer token on /v1/*; binding a non-loopback --host without it is a", - " startup error. With no JAIPH_SERVE_TOKEN and no OIDC, even a loopback bind is a", - " startup error unless --allow-anonymous is passed (single-user workstation only).", + " under .jaiph/runs/: inspect one with GET /runs/{id}, stream its event journal", + " (NDJSON or SSE) via GET /runs/{id}/events, and list/download published files via", + " GET /runs/{id}/artifacts[/{path}]. Set JAIPH_SERVE_TOKEN to", + " require a bearer token on REST and /mcp. With no JAIPH_SERVE_TOKEN and no OIDC, startup", + " is refused unless --allow-anonymous is passed (permits loopback and non-loopback).", " JAIPH_SERVE_MAX_CONCURRENT (default 4) caps simultaneous runs.", " --host listen address (default: 127.0.0.1)", " --port listen port (default: 5247)", - " --allow-anonymous run open with no auth on loopback (single-user workstation only;", - " every local user gets all capabilities over all runs). Ignored when", + " --allow-anonymous run open with no auth (every caller gets all capabilities over", + " all runs). Permits loopback and non-loopback. Ignored when", " JAIPH_SERVE_TOKEN or OIDC is set.", " --workspace workspace root for import resolution (default: auto-detect).", " --env KEY=VALUE grant KEY to matching `use` clauses on every run (repeatable); --env KEY forwards the host value.", diff --git a/src/cli/shared/workflow-call-exec.ts b/src/cli/shared/workflow-call-exec.ts index 6bab202d..b47210ad 100644 --- a/src/cli/shared/workflow-call-exec.ts +++ b/src/cli/shared/workflow-call-exec.ts @@ -178,7 +178,7 @@ export function attachOutputCollector( * * Diagnostic capture — failed-step detail, raw stderr/stdout, and collected * `log` messages — is credential-redacted here, the single boundary both - * `jaiph serve` (`result_text`, `?wait=true`, `GET /v1/runs/{id}`) and + * `jaiph serve` (`result_text`, `?wait=true`, `GET /runs/{id}`) and * `jaiph mcp` (tool results) return through. Live `__JAIPH_EVENT__` lines are * not redacted at the source, so this must not rely on the event stream. * A successful workflow's return value is intentional API output, not diff --git a/src/runtime/kernel/emit.ts b/src/runtime/kernel/emit.ts index c34fab62..445a14ed 100644 --- a/src/runtime/kernel/emit.ts +++ b/src/runtime/kernel/emit.ts @@ -207,7 +207,7 @@ export function verifyRunSummaryChain( * verified" and let a tampered journal through. * - `{ verified: true, ok }` with the chain result otherwise. * - * Every read/export boundary (run listing, `/v1/runs/{id}/events`, OTLP/Sentry + * Every read/export boundary (run listing, `/runs/{id}/events`, OTLP/Sentry * export) hard-fails when `verified === true && ok === false`. */ export function verifyRunJournal(runDir: string, env: NodeJS.ProcessEnv = process.env): { verified: boolean; ok: boolean; error?: string } { diff --git a/src/runtime/kernel/redact.ts b/src/runtime/kernel/redact.ts index 74e96805..815564d0 100644 --- a/src/runtime/kernel/redact.ts +++ b/src/runtime/kernel/redact.ts @@ -2,7 +2,7 @@ * Credential redaction shared by every surface that persists or returns * workflow output: the durable `run_summary.jsonl` writes in * `RuntimeEventEmitter` — which the OTLP export (`otlp.ts`), the Sentry export - * (`sentry.ts`), and `GET /v1/runs/{id}/events` (`handler.ts`) all read back + * (`sentry.ts`), and `GET /runs/{id}/events` (`handler.ts`) all read back * verbatim — and the call-result text composed for `jaiph serve` and * `jaiph mcp` (`src/cli/shared/workflow-call.ts`). One definition of "credential" keeps the * journal, telemetry, HTTP, and MCP surfaces in agreement. diff --git a/src/runtime/kernel/runtime-event-emitter.ts b/src/runtime/kernel/runtime-event-emitter.ts index 10cb3bd3..a3b384b5 100644 --- a/src/runtime/kernel/runtime-event-emitter.ts +++ b/src/runtime/kernel/runtime-event-emitter.ts @@ -111,7 +111,7 @@ export class RuntimeEventEmitter { } // Redact credential values from the step `params` pairs so a secret passed as // a positional/named argument to a `run`/tool/script step does not land raw in - // the journal (and thus in `GET /v1/runs/{id}/events`). Prompt params are + // the journal (and thus in `GET /runs/{id}/events`). Prompt params are // already redacted at their source; re-running on `[REDACTED]` is a no-op, so // this keeps both step kinds on the same durable boundary as out/err content. if (Array.isArray(durableFull.params)) { diff --git a/start-product-owner.sh b/start-product-owner.sh new file mode 100755 index 00000000..c75c8305 --- /dev/null +++ b/start-product-owner.sh @@ -0,0 +1,60 @@ +#!/bin/sh +# Start the product-owner HTTP + MCP server in Docker. +# Usage: ./start-product-owner.sh +# JAIPH_PO_IMAGE default ghcr.io/jaiphlang/jaiph-runtime +# Rebuild locally with ./docs/build-jaiph-dev-image.sh +# (install-from-local.sh does not). Has jaiph + python3 + git; +# the product-owner def needs `claude` on PATH — point +# this at a derived image, or layer the CLI yourself. +# JAIPH_PO_PORT default 5247 +# JAIPH_SERVE_TOKEN optional. Unset = --allow-anonymous (host publish is +# 127.0.0.1). Set to require Authorization: Bearer. +# State lives under .jaiph/product-owner/ (volume jaiph-po-state, not virtiofs): +# queue-state.md, runs/ +# Host .jaiph/runs is tmpfs-masked so serve and the agent cannot see it. +# Claude creds: ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN on the host. +# Those are backend credentials, not `use` grants — do not --env them. +set -e +root=$(CDPATH= cd -- "$(dirname "$0")" && pwd) +: "${JAIPH_PO_IMAGE:=ghcr.io/jaiphlang/jaiph-runtime}" +: "${JAIPH_PO_PORT:=5247}" +if [ -z "${ANTHROPIC_API_KEY:-}" ] && [ -z "${CLAUDE_CODE_OAUTH_TOKEN:-}" ]; then + printf '%s\n' "start-product-owner: set ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN" >&2 + exit 1 +fi +touch "$root/DONE.md" +# Mount point must exist on the host: /work is :ro, so Docker cannot mkdir it. +mkdir -p "$root/.jaiph/product-owner" +printf '%s\n' \ + "product-owner: http://127.0.0.1:${JAIPH_PO_PORT}/docs" \ + "product-owner: http://127.0.0.1:${JAIPH_PO_PORT}/mcp" +if [ -n "${JAIPH_SERVE_TOKEN:-}" ]; then + printf '%s\n' "product-owner: Authorization: Bearer ${JAIPH_SERVE_TOKEN}" +else + printf '%s\n' "product-owner: open (no auth; published on 127.0.0.1)" +fi +set -- \ + --rm \ + -p "127.0.0.1:${JAIPH_PO_PORT}:5247" \ + -e JAIPH_WORKSPACE=/work \ + -e JAIPH_QUEUE_STATE=/work/.jaiph/product-owner/queue-state.md \ + -e JAIPH_RUNS_DIR=/work/.jaiph/product-owner/runs \ + -v "$root:/work:ro" \ + -v "$root/QUEUE.md:/work/QUEUE.md" \ + -v "$root/DONE.md:/work/DONE.md" \ + -v jaiph-po-state:/work/.jaiph/product-owner \ + --tmpfs /work/.jaiph/runs \ + -w /work +if [ -n "${JAIPH_SERVE_TOKEN:-}" ]; then + set -- "$@" -e "JAIPH_SERVE_TOKEN=${JAIPH_SERVE_TOKEN}" +fi +if [ -n "${ANTHROPIC_API_KEY:-}" ]; then + set -- "$@" -e ANTHROPIC_API_KEY +fi +if [ -n "${CLAUDE_CODE_OAUTH_TOKEN:-}" ]; then + set -- "$@" -e CLAUDE_CODE_OAUTH_TOKEN +fi +exec docker run "$@" \ + "$JAIPH_PO_IMAGE" \ + jaiph serve --host 0.0.0.0 --port 5247 --allow-anonymous \ + .jaiph/product_owner.jh From 00aca702ea1e51666187c37348e5d5283fabfac0 Mon Sep 17 00:00:00 2001 From: Jakub Dzikowski Date: Fri, 11 Sep 2026 11:40:15 +0200 Subject: [PATCH 03/60] Queue: drop the redundant ensure_ci_passes argv workaround and pin format no-op on triple-quoted prompts. The caller-side copy is covered by the recover-path and stdin tasks; the new task locks jaiph format on a shebang const = prompt """ file. Co-authored-by: Cursor --- QUEUE.md | 54 ++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 34 insertions(+), 20 deletions(-) diff --git a/QUEUE.md b/QUEUE.md index 31b53104..053e6911 100644 --- a/QUEUE.md +++ b/QUEUE.md @@ -16,26 +16,6 @@ Process rules: 7. Acceptance criteria are non-negotiable. A task is not done until every acceptance bullet is verified by a test that fails when the contract is violated. -## Point ensure_ci_passes at the existing step capture; do not pass recover bytes through argv #dev-ready - -Context: `.jaiph/ensure_ci_passes.jh` runs `npm_run_test_ci` under `recover (failure)`. On failure the recover body calls `common.save_string_to_file(ci_log_file, failure)`. `save_string_to_file` (`.jaiph/lib_common.jh`) takes content as `$2` / `sys.argv[2]`. Scripts have no stdin channel. The same bytes are already on disk as the failed step's stdout capture under `JAIPH_RUN_DIR` (`NNNNNN-script__npm_run_test_ci.out`, written incrementally by `executeManagedStep` in `src/runtime/kernel/node-workflow-runtime.ts`). - -Problem: a CI log larger than the OS spawn argument limit (macOS `ARG_MAX` is about 1 MB for args + environment; jaiph's sterile env still occupies part of that) never reaches the recover prompt. The write is redundant: recover rematerializes a file that already exists. Playwright `test:ci` output around 1.3 MB is a normal case, not an edge case. - -Remediation — implement exactly this: - -1. In `.jaiph/ensure_ci_passes.jh`, delete the `run common.save_string_to_file(ci_log_file, failure)` call. Do not pass the `failure` binding into any script argument. -2. Resolve `JAIPH_RUN_DIR` with a tiny script that prints only that env value (no content argv). Point the recover prompt at the existing capture: `${run_dir}/*-script__npm_run_test_ci.out`. Tell the agent to `tail` that file. Drop the `.jaiph/tmp/ensure_ci_passes.last.log` copy, or keep a dest path only if a script copies via `cp` from the capture glob (path argv only). -3. Replace `assert_nonempty_file_or_fail(ci_log_file)` with a check that the capture glob exists and is non-empty, still using only path argv. -4. Do not change recover binding semantics, `save_string_to_file`, or the runtime in this task. - -### Acceptance criteria - -- `.jaiph/ensure_ci_passes.jh` has no `save_string_to_file` (or other script) call whose argument is the recover binding. -- The recover prompt names the `JAIPH_RUN_DIR` capture (`*-script__npm_run_test_ci.out`) as the log to read. -- A script in that recover body receives at most short paths / names as argv, never the failed step's stdout. -- `npm run build` and `npm test` pass. - ## Spawn and exec failures are failed steps; always emit STEP_END and RUN_END #dev-ready Context: `spawnAndCapture` (`src/runtime/kernel/node-workflow-runtime.ts`) calls `_scriptSpawn.spawn(command, args, …)` inside a Promise executor with no try/catch. The `'error'` handler settles `status: 1`, but a synchronous throw from `spawn` (including `E2BIG` when argv + env exceed `ARG_MAX`) rejects the Promise. `executeManagedStep` awaits `fn(stepIo)` and only `finally`-stops the idle watchdog; on throw it never writes `STEP_END`. `runRoot` awaits `executeDef` then emits `RUN_END`; on throw it never emits `RUN_END`. `runWorkflowRunner` (`.catch` in `src/runtime/kernel/node-workflow-runner.ts`) prints `jaiph node runner: …` and `process.exit(1)` with no journal close. Observed: `STEP_START` for the oversized script, empty `.out`/`.err`, no `STEP_END`, no `RUN_END`, heartbeat stops. `recover_limit` never applies because recover never sees a failed step. @@ -107,3 +87,37 @@ Remediation — implement exactly this: - `save_string_to_file` / architect_review call sites compile under the new helper contract (path argv + stdin body). - Formatter round-trips `run name(args) stdin expr`. - `npm run build`, `npm test`, `npm run test:e2e`, and editor grammar tests (`plugins/vscode`, `plugins/zed` as already wired) pass. + +## jaiph format is a no-op on shebang + const = prompt triple-quoted def #dev-ready + +Context: `jaiph format` (`src/cli/commands/format.ts`) parses with trivia and re-emits via `emitModule` (`src/format/emit.ts`, `src/format/emit-steps.ts`). A `const name = prompt """ … """` step is a `const` whose RHS is `Expr.prompt`. Trivia on that expr carries `bodyKind: "triple_quoted"` and `rawBody` (author lines, including margin). Docs already state the contract: `docs/cli.md` (`jaiph format`) — shebang preserved, triple-quoted prompt blocks emit verbatim (author margin via trivia), a single blank line between steps is kept. `examples/say_hello.jh` uses this shape and `e2e/tests/128_examples_format_check.sh` checks every example, but no unit test pins the minimal file below. `src/format/emit.test.ts` has `const = prompt "…"` and top-level `const x = """`, not `const = prompt """` with a shebang. + +Problem: this exact source is a normal first-agent file. Format must leave it bit-for-bit unchanged (default `--indent 2`). A rewrite that collapses `"""` to `"…"`, re-indents the prompt body, moves the closing `"""`, drops the shebang, or drops the blank line before `return` is a contract break. Today's suite can stay green while that happens. + +Source that must be a no-op (trailing newline after `}`): + +``` +#!/usr/bin/env jaiph + +export def hello(name) { + const response = prompt """ + Say hello to ${name} and provide a fun fact about a person with the same name. + Respond with a single line. Do not inspect files or run tools. + """ + + return response +} +``` + +Remediation — implement exactly this: + +1. Add a unit test in `src/format/emit.test.ts`: `emitModule` of the def body (same steps, no shebang — shebang is CLI-only) equals the input bit-for-bit, including the 4-space prompt margin, the closing `"""` at 2 spaces, and the blank line before `return`. +2. Add a CLI / e2e check (`e2e/tests/100_format_command.sh` or a sibling): write the full source above (with shebang) and assert `jaiph format --check` exits 0 and `jaiph format` does not change the bytes. A second format pass is also a no-op. +3. If format currently rewrites this source, stop the rewrite. Do not collapse the prompt to a double-quoted string. Do not change `rawBody` / prompt trivia, recover bindings, or `stdin` in this task. + +### Acceptance criteria + +- Unit test: the `export def hello(name) { … }` body above (no shebang) round-trips through `parsejaiphWithTrivia` + `emitModule` with `assert.equal` to the original string. A formatter that emits `prompt "Say hello…"` or re-indents the two body lines must fail this test. +- `jaiph format --check` on a file whose entire contents are the shebang source above exits 0. `jaiph format` on a copy leaves `cmp` equal. +- `${name}` stays as authored interpolation in the formatted file (not substituted, not escaped away). +- `npm run build` and `npm test` pass. If you add the e2e section, `npm run test:e2e` passes that file. From a5a504e3ea70f243b9fc5f24da1bf46db8396414 Mon Sep 17 00:00:00 2001 From: Jakub Dzikowski Date: Mon, 14 Sep 2026 10:08:42 +0200 Subject: [PATCH 04/60] Fix: forward JAIPH_QUEUE_STATE to scripts and make the product-owner queue store writable. Scripts were locking the default path on the read-only workspace mount; the start script now bind-mounts state and lock, and chowns the runs volume. Co-authored-by: Cursor --- .cursor/mcp.json | 7 +++++ docs/env-vars.md | 2 +- docs/jaiph-skill.md | 4 +-- docs/language.md | 2 +- src/runtime/kernel/env-allowlist.test.ts | 28 ++++++++++++++++- src/runtime/kernel/env-allowlist.ts | 1 + .../node-workflow-runtime.script-env.test.ts | 6 ++++ start-product-owner.sh | 30 ++++++++++++++++--- 8 files changed, 71 insertions(+), 9 deletions(-) create mode 100644 .cursor/mcp.json diff --git a/.cursor/mcp.json b/.cursor/mcp.json new file mode 100644 index 00000000..2a6e038e --- /dev/null +++ b/.cursor/mcp.json @@ -0,0 +1,7 @@ +{ + "mcpServers": { + "product-owner": { + "url": "http://127.0.0.1:5247/mcp" + } + } +} diff --git a/docs/env-vars.md b/docs/env-vars.md index 123a4d13..9d927212 100644 --- a/docs/env-vars.md +++ b/docs/env-vars.md @@ -129,7 +129,7 @@ Jaiph rejects some names before it spawns anything. A bare `--env KEY` that is u ## Script subprocess environment {: #script-env} -Scripts are **sterile by default**. A script subprocess (named `script`, `import script`, or inline script) receives only the base environment (`PATH`, `HOME`, locale, TLS/proxy settings), the script runtime contract keys (`JAIPH_WORKSPACE`, `JAIPH_SCRIPTS`, `JAIPH_RUN_DIR`, `JAIPH_ARTIFACTS_DIR`, the config-scoped `JAIPH_AGENT_BACKEND` when set, plus `JAIPH_AGENT_MODEL` kept defined for `set -u`), and the host keys granted through `use` + `--env`. Free-form shell lines in a def body (`sh -c`) run under the same sterile contract, minus the `use` layer — a shell line has no declaration to carry a `use` clause, so granted keys never reach it. The rest of the host environment — including agent credentials — never reaches a script. +Scripts are **sterile by default**. A script subprocess (named `script`, `import script`, or inline script) receives only the base environment (`PATH`, `HOME`, locale, TLS/proxy settings), the script runtime contract keys (`JAIPH_WORKSPACE`, `JAIPH_SCRIPTS`, `JAIPH_RUN_DIR`, `JAIPH_ARTIFACTS_DIR`, `JAIPH_QUEUE_STATE` when set, the config-scoped `JAIPH_AGENT_BACKEND` when set, plus `JAIPH_AGENT_MODEL` kept defined for `set -u`), and the host keys granted through `use` + `--env`. Free-form shell lines in a def body (`sh -c`) run under the same sterile contract, minus the `use` layer — a shell line has no declaration to carry a `use` clause, so granted keys never reach it. The rest of the host environment — including agent credentials — never reaches a script. The `use` clause on a script declaration or a [named prompt](language.md#named-prompts) definition is the request; `--env` is the only grant. A `use GITHUB_TOKEN` receives the key iff the run was started with `--env GITHUB_TOKEN` (host value) or `--env GITHUB_TOKEN=VALUE` (explicit value). Presence of the key on the host environment without the flag is not enough. `jaiph run`, `jaiph serve`, and `jaiph mcp` pre-flight every `use` key in the import graph — script and named-prompt keys alike — and abort with `E_ENV_MISSING` when one was not granted; extra `--env` keys nothing `use`s are allowed. `jaiph test` skips that pre-flight — with `--env` the keys are injected into matching `use` spawns, and without it the key is simply absent. diff --git a/docs/jaiph-skill.md b/docs/jaiph-skill.md index 8ef296c1..5864a24d 100644 --- a/docs/jaiph-skill.md +++ b/docs/jaiph-skill.md @@ -150,7 +150,7 @@ Script semantics: - Bodies are **opaque** to Jaiph orchestration — full shell/Python/whatever, heredocs included. The compiler strips the block's common leading whitespace at parse time (same idea as triple-quoted prompts); `jaiph format` re-adds one indent level for readability. The one check: do not call Jaiph symbols (`run`, def names) from inside a script body or `$(…)`. - **Capture = stdout.** `const v = run parse_json("pkg.json")` binds the script's stdout. Use `echo`/`printf` to return data; use exit codes (`return N` / `exit N`) for pass/fail. - **Arguments arrive as `$1`, `$2`, …** Module `const` values and def bindings are *not* exported into the subprocess environment — pass them explicitly as arguments. -- **Script env is sterile.** A script sees only process basics (`PATH`, `HOME`, locale), the `JAIPH_WORKSPACE` / `JAIPH_SCRIPTS` / `JAIPH_RUN_DIR` / `JAIPH_ARTIFACTS_DIR` contract keys, and host keys it requests with a `use` clause: `script release use GITHUB_TOKEN = …` (also on `import script … as gh use GITHUB_TOKEN`). Each `use` key must be granted at run time with `--env KEY[=VALUE]` or `jaiph run` refuses to start (`E_ENV_MISSING`); host presence alone is not enough. `use` goes on a `script` declaration or a **named `prompt` definition** — never on defs, `run` / `prompt` call sites, or anonymous `prompt` steps. This is the child `env` Jaiph builds, not a sandbox: a tool-using `cursor` / `claude` prompt is the same user as `jaiph`. Operator recipe: [Pass a host key to a script](script-env.md). Why and the limit: [Why Jaiph](why-jaiph.md). +- **Script env is sterile.** A script sees only process basics (`PATH`, `HOME`, locale), the `JAIPH_WORKSPACE` / `JAIPH_SCRIPTS` / `JAIPH_RUN_DIR` / `JAIPH_ARTIFACTS_DIR` / `JAIPH_QUEUE_STATE` (when set) contract keys, and host keys it requests with a `use` clause: `script release use GITHUB_TOKEN = …` (also on `import script … as gh use GITHUB_TOKEN`). Each `use` key must be granted at run time with `--env KEY[=VALUE]` or `jaiph run` refuses to start (`E_ENV_MISSING`); host presence alone is not enough. `use` goes on a `script` declaration or a **named `prompt` definition** — never on defs, `run` / `prompt` call sites, or anonymous `prompt` steps. This is the child `env` Jaiph builds, not a sandbox: a tool-using `cursor` / `claude` prompt is the same user as `jaiph`. Operator recipe: [Pass a host key to a script](script-env.md). Why and the limit: [Why Jaiph](why-jaiph.md). - Alternatively a manual `#!` shebang as the first body line selects the interpreter (mutually exclusive with a fence tag). - A newline inside a single-backtick body is a parse error — use a fenced block. @@ -396,7 +396,7 @@ Precedence: **environment > def-level config > module-level config > defaults**. - **Run directory:** `.jaiph/runs//-/` with numbered `NNNNNN-.out`/`.err` per step (written incrementally — `tail -f` works) and `run_summary.jsonl`, one JSON event per line (`RUN_START/END`, `STEP_START/END`, `LOG`, `INBOX_*`, `PROMPT_*`). When debugging a failed run, read the failure footer the CLI prints, then the referenced `.err`/`.out` files. - **Return value:** if `main` returns a string, the CLI prints it to stdout after the PASS line. - **Capture sources:** def → its explicit `return` value; script → stdout; prompt → the agent's answer. -- Step environment: script env is sterile — process basics plus `JAIPH_WORKSPACE`, `JAIPH_SCRIPTS`, `JAIPH_RUN_DIR`, `JAIPH_ARTIFACTS_DIR` (and `JAIPH_AGENT_MODEL`, kept defined for `set -u`); host keys cross only via a `use` clause granted with `--env`. Def variables are **not** auto-exported — pass them as arguments. +- Step environment: script env is sterile — process basics plus `JAIPH_WORKSPACE`, `JAIPH_SCRIPTS`, `JAIPH_RUN_DIR`, `JAIPH_ARTIFACTS_DIR`, `JAIPH_QUEUE_STATE` when set (and `JAIPH_AGENT_MODEL`, kept defined for `set -u`); host keys cross only via a `use` clause granted with `--env`. Def variables are **not** auto-exported — pass them as arguments. ## Testing your programs diff --git a/docs/language.md b/docs/language.md index c37d9cd6..2bb88547 100644 --- a/docs/language.md +++ b/docs/language.md @@ -465,7 +465,7 @@ Values interpolated into an inline shell step (a free-form body line that runs v Managed script steps (`run` to a named script, `import script`, inline scripts) run in a **sterile** environment. A script subprocess receives only: - process mechanics (`PATH`, `HOME`, locale, TLS/proxy settings — the same base set a prompt agent gets); -- the script runtime contract: `JAIPH_WORKSPACE`, `JAIPH_SCRIPTS`, `JAIPH_RUN_DIR`, `JAIPH_ARTIFACTS_DIR`, the config-scoped `JAIPH_AGENT_BACKEND` (when set), and `JAIPH_AGENT_MODEL` (kept defined, empty when unset, for `set -u` scripts); +- the script runtime contract: `JAIPH_WORKSPACE`, `JAIPH_SCRIPTS`, `JAIPH_RUN_DIR`, `JAIPH_ARTIFACTS_DIR`, `JAIPH_QUEUE_STATE` (when set), the config-scoped `JAIPH_AGENT_BACKEND` (when set), and `JAIPH_AGENT_MODEL` (kept defined, empty when unset, for `set -u` scripts); - the host keys named in the script's own `use` clause, and only when the operator granted each with `--env KEY[=VALUE]`. Nothing else crosses **into that child `env`**: the rest of the host environment, agent credentials (`ANTHROPIC_API_KEY`, `CLAUDE_CODE_OAUTH_TOKEN`, `CURSOR_API_KEY`, `OPENAI_API_KEY`), the audit journal path (`JAIPH_RUN_SUMMARY_FILE`), and the audit-chain key (`JAIPH_CHAIN_KEY`) all stay with the runner. A def in the call tree neither grants nor denies keys — `run bbb()` where `bbb` is a def never injects host keys into `bbb`'s scripts; only each script's own declaration counts. Def inline-shell lines (free-form body lines run via `sh -c`) get the same sterile environment as an inline script; a shell line has no declaration and therefore no `use` clause, so granted keys never reach it — use a named script with `use` when a line needs a host secret. This contract is spawn-env only. A `cursor` or `claude` prompt that can run tools is the same user as `jaiph` and is not confined to that child `env`. See [Pass a host key to a script](script-env.md), [Environment variables](env-vars.md#script-env), and [Why Jaiph](why-jaiph.md). diff --git a/src/runtime/kernel/env-allowlist.test.ts b/src/runtime/kernel/env-allowlist.test.ts index c92a2de6..2f550721 100644 --- a/src/runtime/kernel/env-allowlist.test.ts +++ b/src/runtime/kernel/env-allowlist.test.ts @@ -1,6 +1,6 @@ import test from "node:test"; import assert from "node:assert/strict"; -import { scrubPromptEnv, buildRunnerBaseEnv, isRunnerEnvAllowed } from "./env-allowlist"; +import { scrubPromptEnv, buildRunnerBaseEnv, buildScriptEnv, isRunnerEnvAllowed } from "./env-allowlist"; // scrubPromptEnv builds the env handed to every prompt backend subprocess // (runBackend in prompt.ts). Contract: base environment + JAIPH_* control keys @@ -183,6 +183,32 @@ test("buildRunnerBaseEnv: backend credential keys pass; JAIPH_SERVE_* server key assert.equal(env.JAIPH_SERVE_TOKEN, undefined, "host-only serve token stays off the runner"); }); +test("buildScriptEnv: forwards JAIPH_QUEUE_STATE when the runner has it; omits it when unset", () => { + const grant = new Set(); + const withState = buildScriptEnv( + { + PATH: "/usr/bin", + JAIPH_WORKSPACE: "/work", + JAIPH_QUEUE_STATE: "/work/.jaiph/queue-state.md", + JAIPH_DEBUG: "true", + }, + undefined, + grant, + {}, + ); + assert.equal(withState.JAIPH_QUEUE_STATE, "/work/.jaiph/queue-state.md"); + assert.equal(withState.JAIPH_WORKSPACE, "/work"); + assert.equal(withState.JAIPH_DEBUG, undefined, "non-contract JAIPH_* stays off the script"); + + const withoutState = buildScriptEnv( + { PATH: "/usr/bin", JAIPH_WORKSPACE: "/work" }, + undefined, + grant, + {}, + ); + assert.equal(withoutState.JAIPH_QUEUE_STATE, undefined); +}); + test("isRunnerEnvAllowed: matches base env, JAIPH_* control keys, and credentials only", () => { assert.ok(isRunnerEnvAllowed("PATH")); assert.ok(isRunnerEnvAllowed("JAIPH_DEBUG")); diff --git a/src/runtime/kernel/env-allowlist.ts b/src/runtime/kernel/env-allowlist.ts index f0f2ef7d..0b6781af 100644 --- a/src/runtime/kernel/env-allowlist.ts +++ b/src/runtime/kernel/env-allowlist.ts @@ -186,6 +186,7 @@ export const SCRIPT_CONTRACT_ENV_NAMES = [ "JAIPH_RUN_DIR", "JAIPH_ARTIFACTS_DIR", "JAIPH_AGENT_BACKEND", + "JAIPH_QUEUE_STATE", ] as const; /** diff --git a/src/runtime/kernel/node-workflow-runtime.script-env.test.ts b/src/runtime/kernel/node-workflow-runtime.script-env.test.ts index 66a7d306..07e27286 100644 --- a/src/runtime/kernel/node-workflow-runtime.script-env.test.ts +++ b/src/runtime/kernel/node-workflow-runtime.script-env.test.ts @@ -66,6 +66,7 @@ function makeEnv(root: string, scriptsDir: string): NodeJS.ProcessEnv { JAIPH_RUNS_DIR: join(root, ".jaiph", "runs"), JAIPH_SCRIPTS: scriptsDir, JAIPH_WORKSPACE: root, + JAIPH_QUEUE_STATE: join(root, ".jaiph", "queue-state.md"), }; } @@ -105,6 +106,11 @@ test("sterile: a script with no `use` never sees ambient host keys (incl. agent assert.equal(child.PATH, process.env.PATH, "base env (PATH) is forwarded"); assert.equal(child.JAIPH_WORKSPACE, root, "contract key JAIPH_WORKSPACE is forwarded"); assert.equal(child.JAIPH_SCRIPTS, scriptsDir, "contract key JAIPH_SCRIPTS is forwarded"); + assert.equal( + child.JAIPH_QUEUE_STATE, + join(root, ".jaiph", "queue-state.md"), + "contract key JAIPH_QUEUE_STATE is forwarded", + ); assert.ok(child.JAIPH_RUN_DIR, "contract key JAIPH_RUN_DIR is forwarded"); assert.ok(child.JAIPH_ARTIFACTS_DIR, "contract key JAIPH_ARTIFACTS_DIR is forwarded"); assert.equal(child.JAIPH_AGENT_MODEL, "", "JAIPH_AGENT_MODEL stays defined (set -u scripts)"); diff --git a/start-product-owner.sh b/start-product-owner.sh index c75c8305..bfbf1d86 100755 --- a/start-product-owner.sh +++ b/start-product-owner.sh @@ -9,8 +9,14 @@ # JAIPH_PO_PORT default 5247 # JAIPH_SERVE_TOKEN optional. Unset = --allow-anonymous (host publish is # 127.0.0.1). Set to require Authorization: Bearer. -# State lives under .jaiph/product-owner/ (volume jaiph-po-state, not virtiofs): -# queue-state.md, runs/ +# State: +# queue-state.md — host binds at .jaiph/queue-state.md and +# .jaiph/queue-state.md.lock (gitignored). Scripts see the +# default path $JAIPH_WORKSPACE/.jaiph/queue-state.md (also +# $JAIPH_QUEUE_STATE). /work is :ro; both files must be +# writable mounts or queue.py cannot take its lock. +# runs/ — named volume jaiph-po-state (not virtiofs), at +# .jaiph/product-owner/runs. # Host .jaiph/runs is tmpfs-masked so serve and the agent cannot see it. # Claude creds: ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN on the host. # Those are backend credentials, not `use` grants — do not --env them. @@ -23,8 +29,21 @@ if [ -z "${ANTHROPIC_API_KEY:-}" ] && [ -z "${CLAUDE_CODE_OAUTH_TOKEN:-}" ]; the exit 1 fi touch "$root/DONE.md" -# Mount point must exist on the host: /work is :ro, so Docker cannot mkdir it. +# Mount points must exist on the host: /work is :ro, so Docker cannot mkdir them. mkdir -p "$root/.jaiph/product-owner" +# State + lock must both be writable binds. /work/.jaiph is on the :ro +# workspace mount; bind-mounting only the state file leaves the sibling +# `.lock` on the read-only tree. +touch "$root/.jaiph/queue-state.md" "$root/.jaiph/queue-state.md.lock" +# Named volumes are created root:root; the image runs as uid jaiph (10001). +docker volume create jaiph-po-state >/dev/null +docker run --rm --user root --entrypoint sh \ + -v jaiph-po-state:/data \ + "$JAIPH_PO_IMAGE" \ + -c 'mkdir -p /data/runs && chown -R jaiph:jaiph /data' +if docker inspect jaiph-product-owner >/dev/null 2>&1; then + docker rm -f jaiph-product-owner >/dev/null +fi printf '%s\n' \ "product-owner: http://127.0.0.1:${JAIPH_PO_PORT}/docs" \ "product-owner: http://127.0.0.1:${JAIPH_PO_PORT}/mcp" @@ -35,13 +54,16 @@ else fi set -- \ --rm \ + --name jaiph-product-owner \ -p "127.0.0.1:${JAIPH_PO_PORT}:5247" \ -e JAIPH_WORKSPACE=/work \ - -e JAIPH_QUEUE_STATE=/work/.jaiph/product-owner/queue-state.md \ + -e JAIPH_QUEUE_STATE=/work/.jaiph/queue-state.md \ -e JAIPH_RUNS_DIR=/work/.jaiph/product-owner/runs \ -v "$root:/work:ro" \ -v "$root/QUEUE.md:/work/QUEUE.md" \ -v "$root/DONE.md:/work/DONE.md" \ + -v "$root/.jaiph/queue-state.md:/work/.jaiph/queue-state.md" \ + -v "$root/.jaiph/queue-state.md.lock:/work/.jaiph/queue-state.md.lock" \ -v jaiph-po-state:/work/.jaiph/product-owner \ --tmpfs /work/.jaiph/runs \ -w /work From 0771a375db3087729c4a0b69555eb006ebfc8554 Mon Sep 17 00:00:00 2001 From: Jakub Dzikowski Date: Mon, 14 Sep 2026 19:37:38 +0200 Subject: [PATCH 05/60] Docs: close the 0.14 drift in reference pages and CLI help. The pages still described pre-0.14 env, match, credential, and version details. CLI --env help now matches the grant model. Co-authored-by: Cursor --- README.md | 6 +++--- docs/agent-analyzability.md | 2 +- docs/agent-auth.md | 10 ++++++++-- docs/architecture.md | 4 ++-- docs/artifacts.md | 30 ++++-------------------------- docs/async.md | 11 +++++++++-- docs/cli.md | 10 +++++----- docs/configuration.md | 2 +- docs/configure-backend.md | 12 +++++++----- docs/contributing.md | 9 +++++---- docs/deploy/k8s.yaml | 4 ++-- docs/env-vars.md | 4 ++-- docs/first-agent-run.md | 4 ++-- docs/first-run.md | 2 +- docs/grammar.md | 17 +++++++++++------ docs/hooks.md | 2 +- docs/jaiph-skill.md | 10 +++++----- docs/language.md | 11 ++++++----- docs/libraries.md | 10 ++++++++++ docs/mcp.md | 22 ++++++++++++++-------- docs/observability.md | 17 +++++++++-------- docs/script-env.md | 2 +- docs/serve.md | 2 +- docs/setup.md | 12 ++++++------ docs/spec-async-handles.md | 4 ++-- docs/testing.md | 12 ++++++++++++ docs/why-jaiph.md | 2 +- src/cli/commands/mcp.ts | 2 +- src/cli/commands/run.ts | 3 ++- src/cli/commands/serve.ts | 2 +- src/runtime/embedded-assets.ts | 2 +- 31 files changed, 136 insertions(+), 106 deletions(-) diff --git a/README.md b/README.md index 1df2db98..19c6f601 100644 --- a/README.md +++ b/README.md @@ -77,13 +77,13 @@ npm install -g jaiph In GitHub Actions, install a pinned CLI with the [`setup-jaiph`](actions/setup-jaiph/) composite action (same release binaries, no Node required on the runner): ```yaml -- uses: jaiphlang/jaiph/actions/setup-jaiph@v0.13.0 +- uses: jaiphlang/jaiph/actions/setup-jaiph@v0.14.0 with: - version: 0.13.0 # semver, a release tag, or 'nightly' + version: 0.14.0 # semver, a release tag, or 'nightly' - run: jaiph --version # jaiph is now on PATH for later steps ``` -Verify: `jaiph --version`. Switch versions: `jaiph use nightly` or `jaiph use 0.13.0`. +Verify: `jaiph --version`. Switch versions: `jaiph use nightly` or `jaiph use 0.14.0`. Releases ship a `SHA256SUMS` file plus a detached [minisign](https://jedisct1.github.io/minisign/) signature (`SHA256SUMS.minisig`). The installer verifies the checksum and requires a valid signature. A missing `minisign` aborts the install on every host, including CI, rather than degrading to checksum-only. The `setup-jaiph` action installs `minisign` on the runner so CI installs stay signed. For a deliberate checksum-only install, set `JAIPH_ALLOW_UNSIGNED=1`. See [Verify the release signature](docs/setup.md#verify-the-release-signature). diff --git a/docs/agent-analyzability.md b/docs/agent-analyzability.md index 76b74c1b..27b27216 100644 --- a/docs/agent-analyzability.md +++ b/docs/agent-analyzability.md @@ -75,7 +75,7 @@ Each package is a **deep module**: narrow public surface, large private capabili ### CLI slice isolation -Treat these as vertical slices: `commands`, `run`, `serve`, `mcp`, `exec`, `telemetry`. +Treat these as vertical slices: `commands`, `run`, `serve`, `mcp`, `exec`, `telemetry`. The `mcp` and `exec` slices no longer have their own directories (their code moved into `src/cli/shared`, see below), but the rule still reserves both names so a reintroduced private tree stays guarded. **`commands` is the composition root.** It wires the other slices together (each `jaiph` subcommand launches its feature), so `commands` may import any slice's private tree, which is orchestration and not peer coupling. diff --git a/docs/agent-auth.md b/docs/agent-auth.md index 2b76af07..d3e3995f 100644 --- a/docs/agent-auth.md +++ b/docs/agent-auth.md @@ -8,7 +8,7 @@ diataxis: how-to This guide shows how to set the credentials each agent backend needs, so the CLI's credential pre-flight passes and `prompt` steps can reach the model. -`jaiph run` runs a host-side credential pre-flight before it spawns the runner. The pre-flight checks the backends the entry file declares. A missing `codex` credential is a hard failure with the error `E_AGENT_CREDENTIALS`, and the run stops before any runner is launched. A missing `cursor` credential produces only a `jaiph: warning:` line and the run still proceeds. `claude` is not checked — a stored Claude CLI login is the host path. The behavior is implemented in `src/cli/run/preflight-credentials.ts`. +`jaiph run` runs a host-side credential pre-flight before it spawns the runner. The pre-flight checks the backends the entry file declares. A missing `codex` credential is a hard failure with the error `E_AGENT_CREDENTIALS`, and the run stops before any runner is launched. A missing `cursor` credential produces only a `jaiph: warning:` line and the run still proceeds. `claude` is not checked, because a stored Claude CLI login on the host already works. The behavior is implemented in `src/cli/run/preflight-credentials.ts`. ## Prerequisites @@ -22,7 +22,7 @@ This guide shows how to set the credentials each agent backend needs, so the CLI | `cursor` | `CURSOR_API_KEY` | warn only (a stored `cursor-agent login` may still work) | | `codex` | `OPENAI_API_KEY` | hard error `E_AGENT_CREDENTIALS` (no CLI-login fallback) | -Set credentials on the host. Forward anything else one key at a time with `--env`. +Set the credential on the host, or pass it with `--env KEY=VALUE`. The pre-flight reads the runtime environment, which includes both, because backend credential keys are allowed onto the runner environment. Forward any other host key the same way, one key at a time. ### Which backends get checked @@ -35,6 +35,8 @@ The default is deduplicated against your declarations, so where you set the back To check only the backend you intend to use, set it at module scope or export `JAIPH_AGENT_BACKEND`. Either one becomes the default and absorbs the extra check. See [Configure backend/model](configure-backend.md) for the config scopes. +The pre-flight scans only the entry file. A backend that an imported module sets in its own `config { }` block is not checked, so a `prompt` step in an imported def can still reach an unchecked backend. Set the credential on the host for any backend your imports use. + ## 1. Authenticate Claude Either set the API key directly: @@ -84,6 +86,10 @@ The pre-flight is skipped when the entry file neither declares an explicit backe `jaiph run --raw` also skips the pre-flight. +## Servers report every result as a warning + +`jaiph serve` and `jaiph mcp` run the same pre-flight once at startup, but they print every result as a warning, including the `codex` case that hard-fails under `jaiph run`. A server can start before its credentials are set, because it may outlive a credential fix, so a missing `OPENAI_API_KEY` prints a warning and the server still starts. + ## Verification When every required credential is present, the pre-flight is silent, with no stderr before the banner. A missing `cursor` env var emits a `jaiph: warning:` line and the run still proceeds, because a stored `cursor-agent login` may satisfy the runtime: diff --git a/docs/architecture.md b/docs/architecture.md index ba507d1b..122dc171 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -70,7 +70,7 @@ The `src/` import graph is an acyclic layered DAG: parse/format → transpile - **Visitor table + scope.** Per-step validation has one entry point — `validateStep(step, ctx)` in `validate-step.ts`. It looks the step's `type` up in `VALIDATORS` (the dispatch table), then consults `ctx.scope.allowSteps` (a `Set`) once to decide whether this step is permitted in the current scope. One scope exists: `DEF_SCOPE` (allows every step variant, including `send`; `prompt` is an `Expr.kind` inside an `exec` body, not a step type). The scope also carries `runRefExpect` (`RUN_TARGET_REF_EXPECT`) and `withPromptSchemas` (defs collect prompt-returning bindings). Adding a new step type requires exactly one row in `VALIDATORS` and, if the allowed set needs to differ, an entry in `Scope.allowSteps` — an `AC4` test in `validate-visitor.test.ts` injects a synthetic step type and asserts it produces exactly one diagnostic with the documented `internal: no validator for step type "…"` message until the row is added. - **Single managed-call-shape helper.** Every `call` site runs the same five checks against the typed `Arg[]` directly — shell-redirection rejection (only `literal` args are scanned), nested-unmanaged-call rejection inside `literal` raws, ref resolution (with the scope's `runRefExpect`), arity (`args.length` vs declared params), and `var`-arg resolution against in-scope bindings via `validateArgVarRefs`. The sequence lives once in `validateCallable(expr, ctx)`. There is no longer a separate `validateBareIdentifierArgs` helper, no per-site repetition of the five-step sequence, and no place re-parses an `args: string` payload by splitting on commas or rescanning quotes. - **Diagnostics collector (recoverable errors).** The validator no longer fails fast on the first user-level error. Every recoverable check appends to a `Diagnostics` collector (`src/diagnostics.ts`) via `diag.error(file, line, col, code, msg)`, which records a `JaiphDiagnostic` and short-circuits the current validation unit through a `BailoutError`. Each top-level unit (per-import block, per-def walk, per-def step, per-test-block step, per-channel route) is wrapped in `diag.capture(fn)`, which absorbs the bailout (and any thrown `jaiphError` from leaf helpers like `validate-ref-resolution.ts` / `validate-string.ts` / `validate-prompt-schema.ts` / `shell-jaiph-guard.ts` / `parse/validate-string-content.ts`) so the next sibling unit still runs. `collectDiagnostics(graph)` walks every module and returns the populated collector; the legacy **`validateReferences(graph)`** is now a thin wrapper that throws the first sorted diagnostic via **`jaiphError`** so graph-level callers and existing per-error tests keep working; **`emitScriptsForModuleFromGraph`** still calls **`validateModule(ast, graph)`** per module before emit. `Diagnostics.sorted()` returns errors ordered by `(file, line, col)`; `formatLines()` renders the standard `path:line:col CODE message` shape. A grep test (`src/transpile/diagnostics-collector.test.ts`) pins the migration: `validate.ts` + `validate-step.ts` hold **zero** `throw jaiphError(` sites, and the remaining `throw jaiphError(` call sites under `src/` are confined to a documented allowlist — fatal aborts in the parser (`src/parse/core.ts`), the loader (`src/transpile/module-graph.ts`), and the test-file shape check (`src/cli/commands/test.ts`); the legacy bridge in `src/diagnostics.ts`; and the five leaf validation helpers above, each of which has every caller wrapped in `diag.capture(...)`. - - The validator drives off `StepDef.type` (8 variants) and `Expr.kind` (7 variants). For every value-bearing step (`const` / `return` / `send` / `say`) and for the body of every `exec` step, a single `validateExpr(expr, ...)` dispatcher handles the value: it routes `call` / `inline_script` to call-site validation (`validateCallable`), walks `match` arms, schema-checks `prompt`, and runs the substitution scanner on `literal` raws. There is no dual code path for "managed sidecar vs literal value" — that branch is gone. + - The validator drives off `StepDef.type` (9 variants) and `Expr.kind` (7 variants). For every value-bearing step (`const` / `return` / `send` / `say`) and for the body of every `exec` step, a single `validateExpr(expr, ...)` dispatcher handles the value: it routes `call` / `inline_script` to call-site validation (`validateCallable`), walks `match` arms, schema-checks `prompt`, and runs the substitution scanner on `literal` raws. There is no dual code path for "managed sidecar vs literal value" — that branch is gone. - **No compile-time → runtime imports.** Nothing under `src/transpile/` may `import … from "…/runtime/…"`. Compile-time code must not depend on runtime semantics: when the validator needs the same canonical form the runtime will see (the dedented, escape-decoded view of a triple-quoted match-arm body), both sides import a parser-side helper (`canonicalizeTripleQuotedString` in `src/parse/triple-quote.ts`) rather than reaching across the layer. A grep test (`src/transpile/no-runtime-imports.test.ts`) scans every non-test `*.ts` under `src/transpile/` and fails if any `from "…/runtime/…"` import appears; a separate corpus test (`src/parse/canonicalize-triple-quoted.test.ts`) parses every `.jh` under `test-fixtures/` and `examples/`, collects every triple-quoted match-arm body, and asserts `canonicalizeTripleQuotedString` matches the pre-move `tripleQuotedRawForRuntime` output bit-for-bit. - **Block-scoped def walk.** Each `def` is validated by `validateDef` in `validate-def-scope.ts`, which descends the step tree once through a recursive `descend` helper. Every `if` / `else` / `else if` / `for` / `catch` / `recover` body is its own lexical scope (`LexScope`): a child scope holds only the `const`s, captures, `for_lines` iterators, and nested `script` / `def` / `prompt` decls declared so far in that body, and the set visible at a step is the union up the parent chain, with an inner scope shadowing an ancestor. A name declared inside a branch is therefore out of scope after the branch, so a later `run` / `prompt` / `${name}` / bare arg that names it is `E_VALIDATE` rather than a silent runtime miss when the branch is not taken. Immutable-binding and `script`-collision rules are enforced inline per scope (a nested decl may shadow a module script; a `const` or capture may not), the `if` and `else` bodies are two independent child scopes (so a name in each is two separate locals, not a rebind), and a nested `def` / `prompt` body closes over only the enclosing names visible at its declaration point. A `returns` prompt captured inside a branch types `${r.field}` only within that branch. `descend` is the **only** recursive `StepDef[]` walker, and it now lives in `validate-def-scope.ts`, not `validate.ts`. A pair of grep / AST tests (`src/transpile/validate-single-walk.test.ts`) still pin that the prior pre-pass helpers (`collectKnownVars`, `collectPromptSchemas`, `validateImmutableBindings`) cannot reappear in `validate.ts` and that at most one recursive `StepDef[]` walker lives there. @@ -106,7 +106,7 @@ The `src/` import graph is an acyclic layered DAG: parse/format → transpile - **Formatter (`src/format/index.ts`, `src/format/emit.ts`)** - **Public entry.** Code outside the format package imports the format slice only through `src/format/index.ts`, which re-exports the formatter API (`emitModule` and the `EmitOptions` type). It is not an `export *` barrel of the tree. The `no-deep-imports-into-format` rule in `.dependency-cruiser.cjs` fails any outside import that reaches a `src/format/**` internal directly, such as `emit.ts`. Add a named re-export to `src/format/index.ts` instead of reaching in. Format is layer 1 beside parse, so its sources import only parse and types, never `src/cli`, `src/runtime`, or `src/transpile`. - - `jaiph format` rewrites `.jh` / `.test.jh` files into canonical style. `emitModule(ast, trivia, opts?)` reads the semantic AST together with the parallel **`Trivia`** store ([Trivia (CST layer)](#trivia-cst-layer)) to round-trip leading comments, top-level order, `config` body sequence, `"""..."""` and `bareSource` forms, the original quotedness of top-level `const` values (`EnvDeclDef.wasQuoted` — `true` for `"…"` / `"""…"""` sources, `undefined` for bare tokens — so a quoted value is never silently rewritten as bare based on whether it contains a space), and prompt / script body discriminators. Step emission switches on `StepDef.type` (8 variants) and an `emitExprFirstLine` helper switches on `Expr.kind` (7 kinds) — there are no dual code paths for "managed sidecar vs literal value" because that branch was removed from the AST. Call arguments render straight off the typed `Arg[]` — `var` → bare name, `literal` → raw — so the formatter no longer re-parses any args string or consults a `bareIdentifierArgs` shadow field. Pure data→text emitter; no side-effects beyond file writes. Round-trip is bit-for-bit on every fixture under `examples/` and `test-fixtures/golden-ast/fixtures/` — pinned by `src/format/roundtrip.test.ts`, which asserts `parse → format → parse → format` converges in one step on every fixture. + - `jaiph format` rewrites `.jh` / `.test.jh` files into canonical style. `emitModule(ast, trivia, opts?)` reads the semantic AST together with the parallel **`Trivia`** store ([Trivia (CST layer)](#trivia-cst-layer)) to round-trip leading comments, top-level order, `config` body sequence, `"""..."""` and `bareSource` forms, the original quotedness of top-level `const` values (`EnvDeclDef.wasQuoted` — `true` for `"…"` / `"""…"""` sources, `undefined` for bare tokens — so a quoted value is never silently rewritten as bare based on whether it contains a space), and prompt / script body discriminators. Step emission switches on `StepDef.type` (9 variants) and an `emitExprFirstLine` helper switches on `Expr.kind` (7 kinds) — there are no dual code paths for "managed sidecar vs literal value" because that branch was removed from the AST. Call arguments render straight off the typed `Arg[]` — `var` → bare name, `literal` → raw — so the formatter no longer re-parses any args string or consults a `bareIdentifierArgs` shadow field. Pure data→text emitter; no side-effects beyond file writes. Round-trip is bit-for-bit on every fixture under `examples/` and `test-fixtures/golden-ast/fixtures/` — pinned by `src/format/roundtrip.test.ts`, which asserts `parse → format → parse → format` converges in one step on every fixture. ## Local module graph {: #local-module-graph} diff --git a/docs/artifacts.md b/docs/artifacts.md index 68a875f4..a82af998 100644 --- a/docs/artifacts.md +++ b/docs/artifacts.md @@ -68,7 +68,7 @@ export def main() { } ``` -The runtime also sets `JAIPH_RUN_DIR`, `JAIPH_RUN_SUMMARY_FILE`, and `JAIPH_RUN_ID` for script steps, so you can read the run directory, the summary file, or the run id when you need them. +The runtime also sets `JAIPH_RUN_DIR` for script steps, so you can read the run directory when you need it. Script steps do not receive `JAIPH_RUN_SUMMARY_FILE` or `JAIPH_RUN_ID`. Both are held back from the sterile script environment so a script step cannot read the audit journal path (see [Architecture, keyed hash chain](architecture.md#hash-chain)). See [Environment variables](env-vars.md#script-env) for the full list of variables a script step receives. ## Verification @@ -86,33 +86,11 @@ Replace `` with `.jaiph/runs` when `JAIPH_RUNS_DIR` is unset, or with Every line the runtime appends to `run_summary.jsonl` carries a `prev_hash` field. The field holds a **keyed** HMAC-SHA256 of the previous raw line (keyed genesis for the first line), computed under a per-run secret the audited program never sees. Rewriting a line, or dropping a line and re-linking the survivors, breaks the chain and cannot be re-forged without the key, so you can detect tampering with a run's audit trail. The key is persisted once the run finishes. It is **not** stored in the run directory, which the program can write to. It is stored in an operator-side store instead (`~/.jaiph/audit-keys` by default, or the directory in `JAIPH_AUDIT_KEY_DIR`), keyed by the run directory's identity. See [Architecture, keyed hash chain](architecture.md#hash-chain) for the full contract, including the key-isolation and read/export-boundary guarantees. -To check a run directory, run this self-contained Node script. It resolves the run's key from the operator store, where the `sha256` of the run directory's canonical path names its entry. It then recomputes the keyed chain the same way the runtime does and confirms the journal still ends with its `RUN_END` terminal marker. No jaiph build is required: +To check a run directory, use the exported helpers in `src/runtime/kernel/emit.ts`. `verifyRunJournal(runDir)` resolves the run's key from the operator store, recomputes the keyed chain the same way the runtime does, requires the journal to end with its `RUN_END` terminal marker, and returns `{ verified, ok, error }`. `verifyRunSummaryChain(filePath, key, opts?)` is the lower-level form when you already hold the key. -```bash -node -e ' - const fs = require("fs"), crypto = require("crypto"), path = require("path"), os = require("os"); - const dir = fs.realpathSync(process.argv[1]); - const store = process.env.JAIPH_AUDIT_KEY_DIR || path.join(os.homedir(), ".jaiph", "audit-keys"); - const id = crypto.createHash("sha256").update(dir, "utf8").digest("hex"); - const key = fs.readFileSync(path.join(store, id, "key"), "utf8").trim(); - const hmac = (s) => crypto.createHmac("sha256", key).update(s, "utf8").digest("hex"); - const lines = fs.readFileSync(path.join(dir, "run_summary.jsonl"), "utf8").split("\n").filter(l => l.trim()); - let expected = hmac("0".repeat(64)); - for (let i = 0; i < lines.length; i++) { - if (JSON.parse(lines[i]).prev_hash !== expected) { - console.error(`line ${i + 1}: chain broken`); process.exit(1); - } - expected = hmac(lines[i]); - } - const lastType = lines.length ? JSON.parse(lines[lines.length - 1]).type : null; - if (lastType !== "RUN_END") { - console.error(`journal not terminal: last event is ${lastType} (truncated after run end?)`); process.exit(1); - } - console.log(`chain intact and terminal (${lines.length} lines)`); -' //-/ -``` +A run with no store entry (an unkeyed or legacy run) cannot be verified and is never blocked, so `verifyRunJournal` returns `{ verified: false, ok: true }`. A run that **was** keyed but whose key is missing fails closed and returns `{ verified: true, ok: false }`. Every read and export boundary that Jaiph itself controls (run listing, `GET /runs/{id}/events`, and the OTLP and Sentry exporters) hard-fails a run when `verified && !ok`. -A clean, complete journal prints `chain intact and terminal (N lines)` and exits `0`. A rewritten file prints the first broken line number and exits `1`. A completed journal whose last lines were deleted after the run ended prints that it is not terminal and exits `1`. The chain commits to prefix integrity but not to length, so a shorter journal that still links correctly is caught only by the missing `RUN_END` marker (finding L-3). Inside the repo you can call the exported `verifyRunSummaryChain(filePath, key, opts?)` helper (`src/runtime/kernel/emit.ts`) directly, or `verifyRunJournal(runDir)`, which resolves the key from the store for you, requires the terminal marker, and returns `{ verified, ok, error }`. A run with no store entry (an unkeyed or legacy run) cannot be verified and is never blocked. A run that **was** keyed but whose key is missing fails closed (`verified: true, ok: false`). +The full algorithm lives in one place, [Architecture, keyed hash chain](architecture.md#hash-chain). It covers the genesis value, the keyed HMAC-SHA256 recomputation, the key-store layout, the entry id derived from the run directory's canonical path, and the terminal-marker rule. Read that page before you reimplement the check outside the repo, so an external verifier stays in step with the runtime. ## Related diff --git a/docs/async.md b/docs/async.md index 48d0b927..07772fe8 100644 --- a/docs/async.md +++ b/docs/async.md @@ -17,7 +17,14 @@ This page is a recipe. The value model lives in [Async Handles](spec-async-handl ## 1. Start both sides, read late -Hold each handle in its original binding. Do not interpolate, pass as a `run` argument, or use as an `if` / `match` subject until you need the value. An early read waits there and removes the overlap. +Hold each handle in its original binding. Any read that needs the string resolves the handle, so avoid these until you want the value: + +- interpolating it, such as `log "${lint_h}"`; +- passing it as a `run` argument, since a bare argument is rewritten to `${lint_h}` before the call; +- using it as an `if` or `match` subject; +- copying it with `const copy = lint_h`, since a bare copy is rewritten to `"${lint_h}"` and resolves too. + +An early read makes the def wait at that point, which removes the overlap. ```jaiph def lint() { @@ -53,7 +60,7 @@ export def main() { `recover` retries inside that one branch. `catch` runs once; a successful catch counts the branch as joined-ok. A `catch` `return` becomes the parent def's return when the join adopts it. -## 3. Avoid the `for` footgun +## 3. Resolve a handle before a `for` loop `for line in h` does **not** resolve a handle. The loop iterates the token as one line, so you get one pass over `__JAIPH_HANDLE__…` instead of one pass per result line. Resolve first: diff --git a/docs/cli.md b/docs/cli.md index da8b0f20..a22f4199 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -59,7 +59,7 @@ Every run executes on the host. Isolation is an outer concern: wrap jaiph in a c | `--target` | `` | Keep emitted script files and run metadata under `` instead of a temp directory. | | `--raw` | — | Skip the banner, live progress tree, hooks, and PASS/FAIL footer. The runner child inherits stdio; `__JAIPH_EVENT__` JSON lines go to stderr unchanged. | | `--workspace` | `` | Override the workspace root used for library resolution. A missing value, missing path, or non-directory aborts with a specific message. There is no `JAIPH_WORKSPACE` env equivalent input — that name is reserved for the runner. | -| `--env` | `KEY=VALUE` or `KEY` | Repeatable per-key flag, and the **only grant** for `use` clauses on scripts and [named prompts](language.md#named-prompts): a script runs in a sterile env, a named prompt's agent runs with the prompt env scrub, and each receives a host key iff its declaration `use`s it *and* `--env` names it (see [Environment variables — Script subprocess environment](env-vars.md#script-env)). The granted value is injected only into a subprocess whose declaration `use`s the key; it is not placed on the runner (workflow-leader) process environment, which Jaiph builds from an allowlist (process basics, `JAIPH_*` control keys, backend credentials) rather than a copy of the host environment, so an ungranted host key is absent from it. Pre-flight collects every `use` key in the import graph and aborts with `E_ENV_MISSING` if one was not granted; extra keys nothing `use`s are fine. `--env KEY=VALUE` defines `KEY` with that exact value (first `=` splits; the value may contain `=`; empty is allowed). `--env KEY` forwards the host's current value, aborting with `E_ENV_MISSING` before spawning if `KEY` is unset on the host. `KEY` must match `[A-Za-z_][A-Za-z0-9_]*` (else `E_ENV_INVALID`). Runtime-managed keys (`JAIPH_WORKSPACE`, `JAIPH_RUNS_DIR`, `JAIPH_RUN_ID`, `JAIPH_SCRIPTS`, `JAIPH_MODULE_GRAPH_FILE`, `JAIPH_SOURCE_ABS`, `JAIPH_META_FILE`, `JAIPH_ENV_GRANT`, `JAIPH_ENV_GRANT_FILE`, `JAIPH_AGENT_TRUSTED_WORKSPACE`, `JAIPH_TRUST_PROJECT_HOOKS`, `JAIPH_CHAIN_KEY`, `JAIPH_RUN_SUMMARY_FILE`) are rejected with `E_ENV_RESERVED`. Values are never path-remapped. `jaiph run --raw` applies the same grant but skips the graph-wide pre-flight. | +| `--env` | `KEY=VALUE` or `KEY` | Repeatable per-key flag, and the **only grant** for `use` clauses on scripts and [named prompts](language.md#named-prompts): a script runs in a sterile env, a named prompt's agent runs with the prompt env scrub, and each receives a host key iff its declaration `use`s it *and* `--env` names it (see [Environment variables — Script subprocess environment](env-vars.md#script-env)). The granted value is injected only into a subprocess whose declaration `use`s the key; it is not placed on the runner (workflow-leader) process environment, which Jaiph builds from an allowlist (process basics, `JAIPH_*` control keys, backend credentials) rather than a copy of the host environment, so an ungranted host key is absent from it. Pre-flight collects every `use` key in the import graph and aborts with `E_ENV_MISSING` if one was not granted; extra keys nothing `use`s are fine. `--env KEY=VALUE` defines `KEY` with that exact value (first `=` splits; the value may contain `=`; empty is allowed). `--env KEY` forwards the host's current value, aborting with `E_ENV_MISSING` before spawning if `KEY` is unset on the host. `KEY` must match `[A-Za-z_][A-Za-z0-9_]*` (else `E_ENV_INVALID`). The runtime-managed keys the CLI owns (`JAIPH_WORKSPACE`, `JAIPH_RUNS_DIR`, and the rest of the reserved set listed in [Environment variables — Agent credentials](env-vars.md#agent-credentials)) are rejected with `E_ENV_RESERVED`. Values are never path-remapped. `jaiph run --raw` applies the same grant but skips the graph-wide pre-flight. | | `--` | — | End of Jaiph flags; remaining tokens are forwarded to `export def main`. | ### Pre-flight @@ -81,7 +81,7 @@ After module-graph load, before the runner is spawned, the host CLI runs a crede PASS line: `✓ PASS def main (0.2s)`. TTY runs append a transient `▸ RUNNING def (X.Xs)` line that is replaced by the PASS/FAIL line on exit. `--raw` and non-TTY modes skip both. Disable color globally with `NO_COLOR=1`. -Non-TTY heartbeat cadence is controlled by `JAIPH_NON_TTY_HEARTBEAT_FIRST_SEC` (default `60`) and `JAIPH_NON_TTY_HEARTBEAT_INTERVAL_MS` (default `30000`, floor `250`). Leaf script and prompt steps emit a yellow `⚠` idle warning when they produce no stdout/stderr for `JAIPH_STEP_IDLE_WARN_SEC` (default `180`; `0` disables). +Non-TTY heartbeat cadence is controlled by `JAIPH_NON_TTY_HEARTBEAT_FIRST_SEC` (default `60`) and `JAIPH_NON_TTY_HEARTBEAT_INTERVAL_MS` (default `30000`, minimum `250`). A value below `250`, or a value that is not a number, is not clamped up to `250`. Jaiph uses the `30000` default instead. Leaf script and prompt steps emit a yellow `⚠` idle warning when they produce no stdout/stderr for `JAIPH_STEP_IDLE_WARN_SEC` (default `180`; `0` disables). A leaf script step whose subprocess produces no stdout/stderr for `JAIPH_STEP_IDLE_KILL_SEC` (default `3600`, one hour; `0` disables) is terminated and fails. The runtime emits a red `LOGERR` line naming the step and how long it was silent, then kills the step's subprocess, so a stuck script cannot hold an overnight run open indefinitely. New output resets both the warn clock and the kill clock. The kill applies to script steps only; prompt steps get idle warnings but are not killed. @@ -167,7 +167,7 @@ Paths must end with `.jh`. Formatting is idempotent. Comments and shebangs are p | `--indent` | `` | `2` | Spaces per indent level. | | `--check` | — | — | Verify without writing. Exit `0` when files match canonical form, `1` when any file would change. | -Top-level ordering: the formatter hoists `import`, `config`, and `channel` declarations to the top (in that order, preserving relative source order within each group). Other top-level definitions (`const`, `script`, `def`, `test`) keep their relative source order. Comments before a hoisted construct move with it; comments before non-hoisted definitions stay in place. +Top-level ordering: the formatter hoists `import`, `config`, and `channel` declarations to the top (in that order, preserving relative source order within each group). Other top-level definitions (`const`, `script`, `prompt`, `def`, `test`) keep their relative source order. Comments before a hoisted construct move with it; comments before non-hoisted definitions stay in place. Top-level `const` quoting: the source delimiter is preserved per binding. Bare tokens stay bare, `"""…"""` values emit verbatim, and a double-quoted value stays double-quoted. The one exception is a double-quoted value whose content contains a `"` or a `\`: the formatter emits it as a `"""…"""` block so the text needs no escaping. The formatter never rewrites a quoted value as bare, or a bare token as quoted, based on the value's content (for example, whether it contains a space). @@ -279,7 +279,7 @@ jaiph use | Argument | Effect | |---|---| | `nightly` | Reinstalls from the rolling `nightly` prerelease. | -| `` (e.g. `0.13.0`) | Reinstalls the release binary for tag `v`. | +| `` (e.g. `0.14.0`) | Reinstalls the release binary for tag `v`. | Implementation: with no `JAIPH_INSTALL_COMMAND` override, `jaiph use` downloads the install script from `${JAIPH_SITE}/install` (default `https://jaiph.org`), verifies it against the published `${JAIPH_SITE}/install.sha256`, and only then runs it with `JAIPH_REPO_REF` set to `nightly` or `v`. A mismatched or missing checksum fails closed rather than piping an unverified script to `bash`. Setting `JAIPH_INSTALL_COMMAND` overrides this with a verbatim command (forks, offline bundles, local scripts). The installer then downloads the matching per-platform binary plus `SHA256SUMS` (and its signature), verifies them, and replaces `~/.local/bin/jaiph` (or `JAIPH_BIN_DIR`). @@ -432,7 +432,7 @@ See [Environment variables](env-vars.md) for the complete inventory. The variabl - `JAIPH_RUN_TIMEOUT` — parent-enforced wall-clock cap for a run. - `JAIPH_NON_TTY_HEARTBEAT_FIRST_SEC`, `JAIPH_NON_TTY_HEARTBEAT_INTERVAL_MS` — non-TTY progress cadence. -- `JAIPH_RUNS_DIR`, `JAIPH_WORKSPACE`, `JAIPH_SOURCE_FILE` — run-layout inputs. +- `JAIPH_RUNS_DIR`, `JAIPH_SOURCE_FILE` — run-layout inputs. `JAIPH_WORKSPACE` is runner-managed; set it with `--workspace`, not directly. - `JAIPH_INSTALL_COMMAND`, `JAIPH_REGISTRY`, `JAIPH_SKILL_PATH` — install / init inputs. - `NO_COLOR` — disable ANSI colour output. diff --git a/docs/configuration.md b/docs/configuration.md index d5c6f7ff..3deb0db7 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -210,7 +210,7 @@ Resolution order for a `prompt` step: | 3 | Flags model — `--model ` inside `agent.cursor_flags` / `agent.claude_flags`. | `model_reason: flags`. Codex has no flag channel; this step does not apply. | | 4 | Backend default — Cursor/Claude binaries pick their own. Codex defaults to `gpt-4o` in code. | `model_reason: backend-default`. | -For the Claude backend, when `agent.model` is set and `agent.claude_flags` does not already contain `--model`, Jaiph passes `--model ` to the Claude CLI automatically. If both are set, the value in `agent.claude_flags` wins (appended last). +For the Claude backend, when `agent.model` is set and `agent.claude_flags` does not already contain `--model`, Jaiph passes `--model ` to the Claude CLI automatically. When `agent.claude_flags` already contains its own `--model`, Jaiph does not add a second one, so the model in `agent.claude_flags` is the one the Claude CLI receives. When both are set this way, the `PROMPT_START` / `PROMPT_END` records still carry `model_reason: explicit` with the `agent.model` value, even though the Claude CLI ran with the model from `agent.claude_flags`. `PROMPT_START` / `PROMPT_END` records in `run_summary.jsonl` carry `model` (resolved string, or null when the backend auto-selects) and `model_reason` (`explicit`, `flags`, `backend-default`, or `none` for a [custom agent command](#custom-agent-commands), which has no model concept). diff --git a/docs/configure-backend.md b/docs/configure-backend.md index d62067ac..6a5988a0 100644 --- a/docs/configure-backend.md +++ b/docs/configure-backend.md @@ -22,7 +22,7 @@ Add a module-level `config { … }` block at the top of your `.jh` file: ```jh config { agent.backend = "claude" - agent.model = "sonnet-4" + agent.model = "sonnet" } export def main() { @@ -31,9 +31,9 @@ export def main() { } ``` -The valid backend values are `"cursor"` (the default), `"claude"`, and `"codex"`. The model string is forwarded to the backend, so use a name the backend recognizes (e.g. `gpt-4o` for codex, `sonnet-4` for claude). +The valid backend values are `"cursor"` (the default), `"claude"`, and `"codex"`. The model string is forwarded to the backend, so use a name the backend recognizes (e.g. `gpt-4o` for codex, `sonnet` for claude). -Set `agent.backend` (and `agent.command`) only from the entry file. Jaiph ignores these two keys when an imported module sets them in its own `config { … }`, so a third-party module cannot redirect your `prompt` steps to a different binary. See [Import trust boundary](configuration.md#import-trust-boundary). +Set `agent.backend` (and `agent.command`) only from the entry file. By default, Jaiph ignores these two keys when an imported module sets them in its own `config { … }`, so a third-party module cannot redirect your `prompt` steps to a different binary. See [Import trust boundary](configuration.md#import-trust-boundary). ## 2. Override per-def @@ -55,7 +55,7 @@ A def-level block can set `agent.*` and `run.*` keys. The `module.*` keys are mo ```bash export JAIPH_AGENT_BACKEND="claude" -export JAIPH_AGENT_MODEL="sonnet-4" +export JAIPH_AGENT_MODEL="sonnet" jaiph run ./flow.jh ``` @@ -74,9 +74,11 @@ export JAIPH_CODEX_API_URL="https://api.example.com/v1/chat/completions" Each `prompt` step records the resolved backend and model in `run_summary.jsonl`. After the run, inspect the first `PROMPT_START` line: ```bash -jq -c 'select(.type=="PROMPT_START")' .jaiph/runs//