diff --git a/.claude/skills/porting-to-ventis/SKILL.md b/.claude/skills/porting-to-ventis/SKILL.md new file mode 100644 index 0000000..efe4d0d --- /dev/null +++ b/.claude/skills/porting-to-ventis/SKILL.md @@ -0,0 +1,257 @@ +--- +name: porting-to-ventis +description: Use when porting an existing agent project (LangChain, LangGraph, CrewAI, AutoGen, or a hand-rolled pipeline) onto Ventis +--- + +# Porting an agent project to Ventis + +## A port is four files beside an untouched source tree + +``` +agents/.yaml declares the callable surface +agents/.py the thinnest class that satisfies Ventis +workflow/_workflow.py entry point; calls deploy() +config/global_controller.yaml deployment manifest +config/policy.yaml optional — only to restrict access + NOT EDITED — copied whole into every image +``` + +The two `agents/` files share one basename, as every example does — the stub the +build generates lands where it cannot collide with either. Pick a basename that +is not a module the adapter imports. + +Everything the source already does — prompts, tools, schemas, parsing, retries, +its LLM client — is reached with an `import`. **If your port contains a prompt +string, a tool body, or a model call that already exists in the source, you are +rewriting the project, not porting it.** + +Mechanism and evidence for every claim here: `ventis-contract.md`. +Symptom-to-cause lookup once something breaks: `traps.md`. + +## Step 1 — Survey the source before writing anything + +Ventis loads an agent by doing exactly this: + +```python +module = +agent = getattr(module, )() # no arguments +result = getattr(agent, )(**args) # synchronous +``` + +Everything below is answerable by reading the source, and expensive to answer +after a green build. + +### What the adapter has to fix + +Only the gap between that contract and what the source exposes. Nothing else +belongs in the file. + +| The source exposes | The adapter | +| ----------------------------------------------------- | ---------------------------------------------------------------------- | +| a no-argument class whose methods are synchronous | none — point `entrypoint` at the file it already lives in | +| module-level functions, or `@tool` objects (`StructuredTool` instances, not methods) | a class whose methods call them | +| a compiled graph, a `Crew`, a `GroupChat` | a class, plus the orchestration rewrite of Rule 1 | +| `async def` | a synchronous signature, with `asyncio.run(...)` inside the body | +| framework objects as results (messages, graph state) | the framework's own serializer — `json.dumps` runs on what you return | +| a model client built at import | nothing; `env_file:` carries the key | + +Most LangChain and LangGraph projects are rows 2–5, and none of those rows is a +reason to touch the source. + +### What has to be declared + +The whole tree is copied into the image at its own relative paths, but the +container starts at `/app`, so only what landed flat imports on its own. + +- **The import root.** A `pyproject.toml`, `setup.py` or `setup.cfg` at the root + is what adds `-e .`, and the project's own packaging metadata is what decides + the import root — Ventis never guesses a directory name. No metadata and the + install is skipped, silently. Say so before writing an adapter that imports + across directories; adding metadata to fix it edits the source tree. +- **`requirements:`** on the config entry, a list of strings — anything else is + warned about and dropped whole. It covers what the source imports beyond the + runtime's base list and beyond whatever `-e .` already installed. +- **`env_file:`** in `config/global_controller.yaml`, a path relative to the + project root pointing at a local `.env`, handed to every container as + `docker run --env-file`. The file never enters the image, and `ventis deploy` + fails on a bad path before launching anything. Credentials are not a wall — + declare the keys the source reads and leave the model stack alone. + +**And one thing to report rather than fix.** `-e .` installs +`[project.dependencies]` in the same resolve as `requirements:`, and workshop +projects routinely put their whole toolchain there so that one install sets a +laptop up. Compare each declared name against the source's imports: + +```bash +grep -rl "import \|from " / +``` + +**Report the mismatch and stop there. Do not move entries, and do not delete +them.** A grep finds names, not requirements: a package loaded from a string at +runtime is imported nowhere and still required. Hand over the list, the cost +(Step 3's protobuf wall, a full image build away), and the two places entries can +move to — `[project.optional-dependencies]`, which `-e .` skips, and +`[dependency-groups]`, which never enters package metadata at all. Then let the +owner decide, including deciding not to. + +## Rule 1 — Rewrite orchestration, import everything else + +One kind of source code genuinely cannot be reused: **control flow owned by a +framework runtime.** Ventis has no runtime to execute a LangGraph `StateGraph`, a +CrewAI `Crew` or an AutoGen `GroupChat`, so their wiring is re-expressed as +ordinary Python — in the workflow when it fans out, in the adapter when it does +not. The nodes those edges connected are imported, unchanged. + +| Source code | Treatment | +| ---------------------------------------------------------- | ------------------------------ | +| `StateGraph` / `add_edge` / `Send` / `Command(goto=...)` | rewrite as Python control flow | +| `Crew(...)` / `GroupChat(...)` assembly | rewrite as Python control flow | +| node functions, prompts, tools, schemas, parsers, clients | **import** | +| the source's model provider and SDK | **keep** | + +## Rule 2 — Split only to scale + +**Splitting into multiple agents is a scaling decision, not a format +requirement.** A single agent holding the whole pipeline is a valid Ventis +project. Start there, and hoist a loop into the workflow only when each iteration +fans out to more than one node: + +- a single-agent ReAct loop **stays whole in one agent** — every turn needs the + full message history, and hoisting pushes a growing message list through Redis + each turn. +- a supervisor handing out N tasks, or a `Send` fan-out, is **hoisted** — N + independent runs per request with no shared state is what replicas pay for. + +An agent with `replicas: 1` and no distinct resource profile is a node Ventis +does nothing for. When you do split, say plainly what it buys. + +## Rule 3 — Write the least comments + +A port is glue, not a tutorial. Every line in it is there because Ventis +requires it, and the reason is in this skill — which stays current, while a +comment restating it is the same copy Rule 1 forbids and drifts the same way. +**Never write a module docstring, a method docstring, a banner comment, or any +comment naming Ventis, a graph, a framework, or a rule from this skill.** The +projects under `examples/` are demos written to teach Ventis; their commentary +is not a model for a port. + +Before writing any comment, walk this: + +``` +Is this line's behavior decided by something not visible in this file? +├── No → no comment. The code says it. +└── Yes → Does getting it wrong raise, or fail the build? + ├── Yes → no comment. The traceback says it, and `traps.md` explains it. + └── No — it fails silently → one line, ≤ 80 chars, naming what breaks. +``` + +Almost nothing survives that tree. In a fan-out workflow exactly one comment +does: the dispatch-then-resolve split, which is silently serial when fused. +More than two comments across all four files means the tree was not walked. + +Prose has one home — `description:` on each yaml function. It becomes the +generated stub's docstring, so it is the only text the workflow's author ever +reads; a docstring in the adapter reaches nobody. + +## Step 2 — Write the files + +**yaml** — argument `type` is pasted into an AST unchecked, so use `str` `int` +`float` `bool` `dict` `list` and nothing else. Every declared argument is +required. Argument names must equal the Python parameter names character for +character. `returns` is read by nothing — use `type: dict` to mark the call sites +the workflow must `json.loads`. + +**adapter** — class name equals `agent.name`, and the constructor takes no +arguments: configuration comes from environment variables read in `__init__`. +What each method has to do is Step 1's table. + +**workflow** — a top-level function **named `main`, taking a single +`query: str`**, plus `deploy(main, port=...)` at the end. + +Ventis itself is permissive here: it serves `POST /` and splats the +request body in as kwargs, so any name and any arguments run. The deployment +platform's test endpoint is not. It posts to a hardcoded `/main`, and its body +schema is `{query: string}` under a strict validator, so a differently named +workflow is unreachable through it and any other key is rejected with 400 in the +control plane, before the request ever reaches the host. Pack richer input into +`query`; every other parameter needs a default, because nothing will ever send +it. + +The file is `exec`'d rather than imported, so `__name__ == "__main__"` is true +and `if __name__ == "__main__":` blocks fire in production. `deploy()` blocks. + +Dispatch every call before resolving any of them: + +```python +futures = [agent.work(item=i) for i in items] # returns immediately +results = [json.loads(f.value()) for f in futures] # .value() blocks +``` + +Fused into one comprehension the calls run one after another. It does not error; +it is just silently serial, and the fan-out is gone. + +**config** — each entry's `name` must match a yaml's `agent.name`, or the build +warns, skips that image, and still exits 0. Write `provider: local` in +**lowercase**: the port reservation compares `provider == "local"` with no +normalization, so `Local` leaves the port unreserved and deploy dies. + +**policy** — optional. Absent, or present with no rules, every service is +allowed. Write one only to restrict, and then list every service the workflow +reaches; a name left out is not a startup error but an `Unauthorized` response +after the request was accepted. + +## Step 3 — Build, then probe the image twice + +`ventis build` never imports your agent, so a green build proves almost nothing — +it prints `Build complete.` and tags every image for a project whose container +dies on startup. Ventis compounds this: the controller writes `healthy` to Redis +*before* loading the agent and a heartbeat keeps re-asserting it, so a container +with no agent stays `healthy` and keeps receiving requests. + +So run the image — tagged `ventis-` — and do what the +container does. **Both probes, in this order. Neither covers the other.** + +```bash +# 1. The runtime itself. This is what CMD runs, and it fails before your agent +# is ever reached, so probing the entrypoint alone will miss it. +docker run --rm ventis- python -c "import local_controller" + +# 2. The agent, loaded the way _load_agent loads it. +docker run --rm ventis- python -c " +import importlib.util, sys +spec = importlib.util.spec_from_file_location('m', '.py') +m = importlib.util.module_from_spec(spec); sys.modules['m'] = m +spec.loader.exec_module(m); m.(); print('ok')" +``` + +Probe 1 exists because the gRPC stack is unpinned: `ventis build` runs +`grpc_tools.protoc` on the **host** and copies the generated `_pb2.py` in, where +a resolver that knows nothing about them picks the protobuf runtime. Protobuf +refuses gencode newer than its runtime, so a source whose dependencies hold +protobuf back kills the container on `import local_controller`. An image with few +requirements passes by coincidence. Report it — the fix belongs in +`generate_docker`, not in the port — and if Step 1 flagged declared-but-unimported +dependencies, name the culprit here. + +Probe 2 exists because `_load_agent` catches every exception, logs it and returns +`None`: a missing dependency, a wrong class name, a constructor that wants +arguments, or a broken import inside the source tree are all invisible until the +first request answers `"No agent loaded"`. + +Then `ventis deploy`, which needs Docker and an importable `grpc_stubs/` **on +this host** (it aborts if they were cleaned after the build). It starts its own +Redis container — do not run one. + +## Never do these + +Each turns a port into a rewrite. They are not judgment calls, and the middle +column is the thought that gets you there. + +| Move | The rationalization | Why it is wrong | +| ----------------------------------- | ------------------------------------------------ | --------------------------------------------------------------------------------------- | +| Copy a prompt, tool, or schema into the adapter | "so the adapter stands alone" | It exists in the source. Import it — the whole tree is in the image, and a copy drifts. | +| Explain a Ventis rule in a comment or docstring | "the next reader will need this" | The skill is that explanation and stays current. Restated in the port it drifts, and it buries the one comment that fails silently. | +| Swap the LLM provider | "the image already has one, and the user wants something that runs" | A port that silently changed models does not run *their* project. `requirements:` installs the source's own provider, `env_file:` carries its key. | +| Hardcode a key, or ship it in a file you add | "there is no other way in" | `env_file:` is the way in. Never put a secret in the source tree or the build context. | +| Drop or move a dependency | "this one is obviously dev-only" | Obvious to you, not yours to decide. Declare it under `requirements:`; report the rest and let the owner classify. | +| Edit files in the source tree, or vendor it into `agents/` | "just this one line" | The port must leave `git status` on the source clean, and vendoring is copying. | diff --git a/.claude/skills/porting-to-ventis/traps.md b/.claude/skills/porting-to-ventis/traps.md new file mode 100644 index 0000000..163cc48 --- /dev/null +++ b/.claude/skills/porting-to-ventis/traps.md @@ -0,0 +1,52 @@ +# Traps + +Symptom-to-cause lookup for a port that is already written. The mechanism behind +each row is in `ventis-contract.md`. + +## Before any container starts + + +| Symptom | Cause | +| ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | +| `env_file does not exist` on deploy | the path is resolved against the project root you run from; `cmd_deploy` fails before launching anything | +| `int() argument must be ... not 'NoneType'` on deploy | `provider:` is not lowercase `local`, so no host port was reserved | +| `generated grpc_stubs are missing or not importable` | `ventis build` has not run on this host, or its output was cleaned | +| An agent missing from the deployment | its config `name` matched no yaml; the build logged a warning and exited 0 | + + +## The container dies or serves nothing + + +| Symptom | Cause | +| -------------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| Container exits on `import local_controller` | protobuf gencode newer than the resolved runtime; nothing pins the gRPC stack | +| `"No agent loaded"` on the first request | anything below — the agent container's stdout is the only place the cause exists | +| A replica reports `healthy` but answers nothing | same; `healthy` is written before `_load_agent` runs and is never revised | +| `Missing credentials` loading the agent | no `env_file:`, or the key the source reads is not in it | +| `ModuleNotFoundError` for the source's own modules | the project declares no packaging metadata, so `-e .` was skipped and only flat modules import | +| `ModuleNotFoundError` for a third-party package | an import the source needs is missing from the entry's `requirements:` | +| `NameError` importing a stub | a yaml `type` that is not a builtin | + + +## The request is accepted and then goes wrong + + +| Symptom | Cause | +| --------------------------------------------------- | ------------------------------------------------------------------------------------- | +| `TypeError: unexpected keyword argument` | yaml `arguments[].name` != the Python parameter name | +| `Unauthorized: Policy denied access to service 'X'` | `X` is missing from the `access` list of the policy rule that matched | +| `.value()` returns a `str` of a dict | expected — `json.loads` it | +| `Object of type ... is not JSON serializable` | the adapter returned framework objects; serialize with the framework's own serializer | +| Redis holds `` | the method is `async def`; keep the signature sync and `asyncio.run` inside | +| No faster than the original | calls fused with `.value()`; dispatch all, then resolve all | +| Debug code runs in production | the workflow is `exec`'d, so `__name__ == "__main__"` | + + + +## Through the deployment platform's test endpoint + +| Symptom | Cause | +| --------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| 404 from the test endpoint, container healthy | the workflow function is not named `main`; the platform posts to a hardcoded `/main` | +| 400 before the request reaches the host | the body key is not `query`; the platform's schema is strict and rejects everything else | +| The workflow runs but an argument is missing | only `query` is ever sent; every other parameter needs a default | diff --git a/.claude/skills/porting-to-ventis/ventis-contract.md b/.claude/skills/porting-to-ventis/ventis-contract.md new file mode 100644 index 0000000..56e7132 --- /dev/null +++ b/.claude/skills/porting-to-ventis/ventis-contract.md @@ -0,0 +1,281 @@ +# The Ventis contract + +Mechanism behind every rule in `SKILL.md`. Validate against +[CanyonCodeCoreAI/canyoncodecore](https://github.com/CanyonCodeCoreAI/canyoncodecore). + +## Project layout + +| Path | Where it comes from | +| ------------------------------------------- | ---------------------------------------------------------------------------- | +| `agents/*.yaml` | `cli.py` — `glob(agents_dir/*.yaml)` | +| `stubs/`, `grpc_stubs/` | `cli.py` — generated by `ventis build` | +| `config/global_controller.yaml` | `cli.py` — `DEFAULT_CONFIG_PATH`, overridable with `--config` | +| `config/policy.yaml` | `global_controller.py` `_load_policy_rules` — optional | +| the workflow file | the `workflow_file` key on the `type: workflow` config entry | +| the project root | `cli.py` passes `project_dir=os.getcwd()`; build and deploy run from it | +| `pyproject.toml` / `setup.py` / `setup.cfg` | `_install_step` — its presence is what adds `-e .` | + +## Agent yaml + +```yaml +agent: + name: # required + functions: # optional; absent -> stub class with only __init__ + - name: # required + description: # optional -> becomes the stub method's docstring + arguments: # optional; absent -> no-arg method + - name: # required + type: # optional -> pasted verbatim as an annotation + returns: + type: # read by nothing +``` + +Nothing else is read. Extra keys are ignored silently. + +- **`type` is pasted, never checked.** `_build_stub_method` does + `ast.Name(id=arg["type"])`, and the generated stub imports only `Future` and + `inspect`. Anything that is not a builtin raises `NameError` when the stub is + imported. Use `str` `int` `float` `bool` `dict` `list` — not `List[str]`, not + `Optional[int]`, not a class name. +- **No default values.** `ast.arguments(..., defaults=[])`. Every declared + argument is required at every call site. Optional configuration belongs in the + agent's `__init__`, read from the environment. +- **Parameter names must match exactly.** The controller invokes `method(**args)`. + Order is irrelevant; spelling is not, or the call raises `TypeError` at request + time. +- **`returns` is documentation.** The stub generator never reads it. Its value is + as a marker: `type: dict` tells whoever writes the workflow that this call site + needs `json.loads`. +- **The filename names the stub, not the agent.** `agents/x.yaml` generates + `stubs/x.py`, which the build copies to `/app/x.py` and `/app/agents/x.py`. + Sharing the entrypoint's basename is therefore fine and is the convention: the + entrypoint is copied last, so it wins `/app/x.py` while the stub keeps + `/app/agents/x.py`. What the basename must **not** match is a source module the + adapter imports — `joke_writer.yaml` beside a `joke_writer.py` puts a stub on + top of the source. + +## The three-way name binding + +``` +config entry `name` == agents/x.yaml `agent.name` == the class inside the .py + | + `entrypoint` on that config entry points at the .py +``` + +`cmd_build` looks up each config entry's `name` among the parsed yamls. No match +means a logged warning and **no image built for that agent** — the build still +exits 0. + +## Agent class + +| Requirement | Enforced by | +| ----------------------------------------------- | -------------------------------------------------------------------------------------------- | +| Class name equals `agent.name` | `generate_docker` writes `ENV VENTIS_AGENT_NAME`; `_load_agent` does `getattr(module, name)` | +| Instantiable with no arguments | `_load_agent` calls `agent_class()` | +| Methods are synchronous | the executor calls `method(**args)` — there is no `await` anywhere on this path | +| Return values survive `json.dumps` / `str()` | `_execute_locally` does `json.dumps(result)` for `dict`/`list`, `str(result)` otherwise | + +`self.tools = [...]` appears throughout `examples/` and is read by **nothing** in +`ventis/`. It is decoration. + +**`.value()` always returns a string.** The result is written into Redis as text +and handed back verbatim; there is no deserialization on the way out. + +## Workflow + +The workflow file is **not imported — it is `exec`'d**. +`generate_workflow_docker` writes a `workflow_launcher.py` whose last line is +`exec(open(".py").read())`, and the Dockerfile's CMD runs that launcher. + +- `__name__ == "__main__"` inside your workflow file, so + `if __name__ == "__main__":` blocks **execute in production**. +- `__file__` points at `workflow_launcher.py`. The `sys.path.insert(..., "..", + "stubs")` lines the examples carry resolve to nonexistent paths; imports work + anyway because the stubs and the runtime are placed flat at `/app`, which is + `sys.path[0]`. What makes the *project* tree importable is the editable + install, not `sys.path[0]`. +- `deploy()` ends in `app.run()` and blocks. Nothing after it runs. +- Module-level code runs **once** at container start; the workflow function runs + **per request**, on a Flask worker thread. +- The REST route is `fn.__name__` — rename the function and the endpoint renames + with it. There is no fixed `/main` **in Ventis**. +- The request body is splatted in as kwargs after `_context` is popped off. Any + shape of body works. + +Both of those are why the platform constraint has to be written down rather than +discovered: the control plane's test endpoint posts to a hardcoded `/main` with a +strictly validated `{query: string}` body, so a port must expose `main(query)` to +be reachable through it. Nothing in this repo enforces that or fails without it — +the constraint lives in the control plane (`deploy.routes.ts`, `deploy.agent.ts`, +`deploy.types.ts`), and the transport layer there is generic +(`Record`) while the route schema is not. + +The workflow container also runs its own `LocalController` on 50051 in a +background thread. That is what dispatches the Futures the workflow creates. + +## The build context + +`generate_docker` takes a `project_dir` and `cmd_build` passes it, so the whole +project reaches the image with its relative paths intact — structure is preserved +rather than flattened because packages need it (`src/tools/__init__.py` and +`src/tools/default/__init__.py` flatten to the same name). + +Copy order decides every collision: the swept tree first, then the shared +runtime, then every stub, then the entrypoint. Later writes land on earlier ones. + +| What | Lands where | +| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| the project tree | at its own relative paths (`agents/x.py`, `src/pkg/mod.py`) | +| the shared runtime | flat at the context root, winning over the swept tree — `local_controller.py` is the CMD, so a project file of that name breaks the container | +| every stub | **twice**: flat at the root (the copy imports resolve), and at `agents/.py`, landing on the real implementation so a peer's name gives the caller its stub | +| the entrypoint | flat, last, winning the flat name back — `VENTIS_AGENT_FILE` is a **basename**, loaded from `/app` | +| `requirements.txt` | written before anything is copied, so the sweep skips a project's own `requirements.txt` and root `Dockerfile` | + +The sweep takes every file, not only `.py` — the editable install reads packaging +metadata, and that metadata points at a README or a license. It skips hidden +files and directories (`.env` holds credentials and the context is what ships), +`__pycache__`, and the three directories `ventis build` generates: +`docker_container`, `stubs`, `grpc_stubs`. + +**An agent is no longer one file**, and a yaml sharing the entrypoint's basename +no longer eats its own stub. What an agent loses is the ability to import *its +own* stub by name — the entrypoint shadows it flat. It can still reach it at +`agents/.py`, and nothing in `examples/` wants to. + +### The import root + +`_install_step` writes +`RUN uv pip install --system -r requirements.txt -e .` when the project root has +a `pyproject.toml`, `setup.py` or `setup.cfg`. That editable install is what +makes a `src/` layout importable, and the source's own packaging metadata is what +decides it — `[tool.setuptools.package-dir] "" = "src"` is a typical case. Ventis +never guesses a directory name. + +Without packaging metadata the install is skipped — silently, no warning. The +tree is still copied, but `sys.path[0]` is `/app`, so only modules that landed +flat resolve. `examples/helloworld`, `finance` and `text2sql` are all in this +state; they work because their entrypoints import nothing from the project tree, +only stubs, which land flat. + +**One resolve, not two.** Requirements and `-e .` go to a single `uv pip install` +so the runtime's list and the source's own dependencies resolve against each +other; a genuine conflict fails the build instead of the first request. It also +forces `COPY . .` ahead of the install, so the requirements layer no longer +caches on its own. + +## Dependencies + +`generate_docker` and `generate_workflow_docker` both take a `requirements` +argument, and `cmd_build` passes `_normalize_requirements(agent_cfg)`. The +runtime's own list is unconditional and not declarable: + +``` +agent: grpcio grpcio-tools redis pyyaml psutil ipdb ipython boto3 +workflow: the same, plus flask sqlalchemy psycopg[binary] +``` + +The declared list is appended verbatim. `_normalize_requirements` takes only a +list of strings — a bare string, a mapping, or a list with a non-string in it +each logs one warning and becomes `[]`, so a malformed entry costs the whole list +rather than the one item. Nothing is deduplicated against the base either. + +**The source's own `pyproject.toml` is installed in the same resolve**, so +`requirements:` covers only what the adapter imports and the source does not +declare. The whole dependency list comes along, dev extras included — a workshop +project's can carry jupyter, matplotlib and pandas into a 1GB agent image. + +### The gRPC stack is unpinned + +`cmd_build` runs `grpc_tools.protoc` on the **host** and copies the resulting +`_pb2.py` into the image, where a resolver that knows nothing about them picks +the protobuf runtime. Protobuf refuses to load gencode newer than its runtime, so +a source whose own dependencies drag protobuf down produces a container that dies +on `import local_controller` — before the agent is reached, with a green build +behind it: + +``` +google.protobuf.runtime_version.VersionError: Detected incompatible Protobuf +Gencode/Runtime versions ... gencode 7.35.1 runtime 6.33.6. +``` + +An image with few requirements resolves to the newest wheel, which happens to be +at least as new as the host's, and passes by coincidence. A fix means prepending +`grpcio==`, `grpcio-tools==` and `protobuf>=` at the host's own versions +(`importlib.metadata.version`) to the generated requirements — `>=` on protobuf +because the guarantee runs one way: a runtime at or above the gencode. + +**Check this first on any port that installs a large dependency tree.** Probing +the entrypoint module is not enough — it does not import `local_controller`, +which is what the container's CMD actually runs. + +## Credentials: `env_file` + +`_launch_locally` passes exactly five `-e` flags, all `VENTIS_*` +(`AGENT_PORT`, `AGENT_HOST`, `REDIS_HOST`, `REDIS_PORT`, `POLL_INTERVAL`), plus +`VENTIS_DATABASE_URL` and `VENTIS_PROJECT_ID` on a workflow entry when +configured. User secrets travel a separate road. + +`env_file:` in `config/global_controller.yaml` names a local `.env`. +`resolve_env_file` expands `~`, resolves a relative path against the project +root, and raises if the file is missing, is not a file, or is unreadable — +`cmd_deploy` calls it before `GlobalController` exists, so a bad path is one +error line rather than a fleet of agents with no keys. + +`env_file_args` then hands the file to `docker run` as `--env-file`. A container +on this machine reads the original; a container on a remote host gets a 0600 copy +under `/tmp`, deleted as soon as `docker run` returns. The explicit `VENTIS_*` +flags are appended first and still win, so a stray `VENTIS_*` line in someone's +`.env` cannot break agent wiring. + +Consequences for a port: + +- A source that constructs its model client at module scope **loads fine**. The + key is in the environment before the adapter imports the source. +- The file never enters the image — the sweep skips hidden files, and the + variables reach the container at run time. +- A missing key is no longer `"No agent loaded"`; it is a provider error on + `/status` after the request was accepted. +- `load_dotenv(".env")` in the source still does nothing: the file is not in the + image and `load_dotenv` is silent about a missing one. + +## `config/policy.yaml` is optional + +`_load_policy_rules` logs `No policy file found ..., skipping policy setup` and +returns `[]`, which `_load_and_write_policies` publishes to every host Redis. +`LocalController._check_policy` returns `True` when the rule list is empty, so +**no policy file means everything is allowed.** + +When rules exist they are sorted most-specific-first (by number of `match` keys) +and the first rule whose `match` keys all equal the request context decides: +`access: all`, or membership in the `access` list. A service left out of the +matching rule answers `Unauthorized: Policy denied access to service 'X'` in the +`/status` response — after the request was accepted. If no rule matches at all, +access is denied. + +## `provider` is case-sensitive in one direction only + +`InstanceManager.ensure_instances` tests `provider == "local"` to decide whether +to reserve a host port. `Local` fails that test, `reserved_port` stays `None`, +and `Local/_runtime.py`'s +`int(spec.get("host_port", spec.get("port", next_host_port(host))))` raises +`int() argument must be a string, a bytes-like object or a real number, not +'NoneType'`. The EC2 test on the same value is `.upper() == "EC2"` everywhere, so +it accepts any casing. Every example that works writes lowercase `local`. + +## Failures are silent + +`_load_agent` catches every exception, logs it, and returns `None`. + +| Stage | A missing credential / dependency / wrong class name | +| --------------- | ------------------------------------------------------ | +| `ventis build` | passes — it never imports your agent | +| `ventis deploy` | passes — the container starts, gRPC listens | +| first request | `"No agent loaded"` | + +The real cause exists only in that container's stdout. + +Worse, the node still advertises itself as usable. `LocalController.__init__` +writes `healthy` to `controller:::status` **before** calling +`_load_agent`, and `_metrics_loop` re-writes `healthy` on every tick. Nothing +downgrades the status when the agent fails to load, so a replica that can serve +nothing keeps being routed to. diff --git a/examples/joke_writer_langfuse/.env.example b/examples/joke_writer_langfuse/.env.example new file mode 100644 index 0000000..2366c18 --- /dev/null +++ b/examples/joke_writer_langfuse/.env.example @@ -0,0 +1,37 @@ +# Copy this to `.env` and fill in the token. `config/global_controller.yaml` +# points `env_file:` at that copy, and it reaches every container as +# `docker run --env-file`. +# +# Keep the real token out of THIS file. `.env.example` is the one exception to +# the build context's exclusion of `.env*`, so whatever is written here is baked +# into the image; `.env` itself never enters the build and never leaves the host. + +# A Bedrock API key -- the long-term kind generated in the console, or a +# short-term one. botocore matches this exact name against bedrock-runtime's +# signingName (`bedrock`) and switches the client from SigV4 to bearer auth by +# itself, which is why neither joke_writer.py nor ventis/llm/bedrock.py mentions +# it. An IAM access key works too: drop AWS_ACCESS_KEY_ID and +# AWS_SECRET_ACCESS_KEY in instead and the same client signs with SigV4. +AWS_BEARER_TOKEN_BEDROCK= + +# Neither is a secret, and both have defaults in joke_writer.py -- they are here +# to name what the source reads. +BEDROCK_MODEL_ID=meta.llama3-8b-instruct-v1:0 +AWS_REGION=us-east-1 + +# Langfuse. Optional: with none of these set the SDK disables itself and the +# run is untraced (it does log a warning per traced call site -- see +# joke_writer.py). The secret key is the only secret of the three. +# +# Project settings -> API Keys in Langfuse. Free cloud account at +# https://langfuse.com/cloud. +LANGFUSE_PUBLIC_KEY= +LANGFUSE_SECRET_KEY= +# EU cloud. US is https://us.cloud.langfuse.com, and a self-hosted instance is +# its own URL. Must be set: there is no default that reaches a Langfuse. +LANGFUSE_BASE_URL=https://cloud.langfuse.com + +# Which Langfuse environment these traces belong to, so that runs from a laptop +# do not land in the same dashboards and evaluations as deployed ones. Read by +# the SDK itself; nothing in this project mentions it. +LANGFUSE_TRACING_ENVIRONMENT=development diff --git a/examples/joke_writer_langfuse/LICENSE b/examples/joke_writer_langfuse/LICENSE new file mode 100644 index 0000000..5600729 --- /dev/null +++ b/examples/joke_writer_langfuse/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 LangChain, Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/examples/joke_writer_langfuse/README.md b/examples/joke_writer_langfuse/README.md new file mode 100644 index 0000000..5eabd8e --- /dev/null +++ b/examples/joke_writer_langfuse/README.md @@ -0,0 +1,252 @@ +# Joke Writer + +A LangGraph map-reduce, ported to Ventis. Derived from +[langchain-ai/langchain-academy](https://github.com/langchain-ai/langchain-academy) +at `fa15bec` (`module-4/studio/map_reduce.py`, MIT — see `LICENSE`). + +Unlike the other targets in `examples/`, **the source here is not unmodified**. +`map_reduce.py` built a `ChatOpenAI` at module scope, and at the time of the port +an agent container had no way to carry an `OPENAI_API_KEY` — the port was blocked +at the credential wall until the model call was rewritten onto Bedrock. That wall +is gone now: `env_file` puts any key in the container. The Bedrock rewrite stayed +anyway, and [What the port cost](#what-the-port-cost) is honest about what that +means. + +## Overview + +Given a topic, the graph splits it into sub-topics, writes one joke per +sub-topic in parallel, then picks the best of them. + +1. `generate_topics` — one LLM call, turns the topic into three sub-topics, + validated into `Subjects`. +2. `generate_joke` — one LLM call per sub-topic. `continue_to_jokes` emits a + `Send` per subject, so this node runs N times per request with no shared + state between the runs. `jokes` is an `Annotated[list, operator.add]`, which + is how the N results merge back into one state. +3. `best_joke` — one LLM call over every joke, returns the winner by index. + +``` + START + | + generate_topics 1 call + | + continue_to_jokes Send x N + / | \ + joke joke joke N calls, no shared state + \ | / + best_joke 1 call + | + END +``` + +### Why this one + +It is the smallest project in reach whose control flow does something a single +process cannot: `Send` fans out to N independent calls per request. Everything +else about it is deliberately boring — four packages, no tools, no external +service, one API key. + +## The port + +| File | What it holds | +| --- | --- | +| `joke_writer.py` | The source. Three prompts, two schemas, three nodes, and the graph — still compiled, never executed under Ventis. | +| `agents/joke_agent.py` | `JokeAgent`. Three methods, each calling the source's node with the node's own state dict. Imports `joke_writer`; restates nothing. | +| `agents/joke_agent.yaml` | The three nodes declared as three functions on one agent. | +| `workflow/joke_workflow.py` | Where the graph went — the edges, the `Send` fan-out and the `operator.add` reducer, re-expressed as ordinary Python. | +| `config/global_controller.yaml` | `JokeAgent` at `replicas: 3`, plus the workflow. | +| `config/policy.yaml` | Default-allow for the two services. Not optional — a missing file kills `ventis deploy`. | + +Two decisions worth naming: + +**One agent, not three.** `generate_topics` and `best_joke` run once per request +and have no resource profile of their own. Splitting them out would buy two more +images and two more Redis round trips. What is hoisted is the fan-out, and that +is a workflow concern. + +**The graph is not the port.** `StateGraph`, `Send` and the `Annotated[list, +operator.add]` reducer are control flow owned by the LangGraph runtime, and +Ventis has no runtime to execute them. The workflow dispatches N +`generate_joke` calls across the three replicas and concatenates the results +itself. Every call is dispatched before any is resolved — `.value()` blocks, so +fusing the two lines into one comprehension would silently serialize the fan-out +and remove the reason to be on Ventis at all. + +## What the port cost + +This is no longer upstream's model stack. `ChatOpenAI` and +`with_structured_output` are gone; `ventis.llm.bedrock.call_bedrock` is the raw +converse API, so each node asks for JSON in its prompt and validates the reply +through the same pydantic schema upstream used. `_extract_json` exists only +because `with_structured_output` used to do that work. + +That rewrite is not something the `porting-to-ventis` skill should do on a +user's project — it is the credential wall, and the skill's instruction is to +report it. It was done here deliberately, so that this example is one that +actually deploys. + +**It would not be necessary today.** The rewrite bought one thing: boto3 builds +no client at import, so the agent could be *loaded* with no secret in the +container, back when `_launch_locally` passed five `-e` flags and all five were +`VENTIS_*`. `env_file` removes that constraint — an `OPENAI_API_KEY` now reaches +a container as readily as a Bedrock one, and upstream's `ChatOpenAI` at module +scope would import fine. What the rewrite still buys is narrower: a module-scope +client turns a missing key into `"No agent loaded"`, while a per-call one turns +it into a real error on `/status`. Worth knowing, not worth a rewrite. + +The example stays on Bedrock because it is the model call that has been end-to-end +verified here, and because `ventis/llm/bedrock.py` is where Ventis writes per-call +token telemetry onto the future. + +## Tracing + +Every run is one [Langfuse](https://langfuse.com) trace: + +``` +write-jokes span topic in, final state out + generate-topics span + split-topic generation 1 call + generate-joke span \ + write-joke generation | N of these, in parallel + ... / + select-best-joke span + judge-jokes generation 1 call +``` + +`write_jokes(topic)` is the traced entry point and the only thing added to the +module's surface. `graph.invoke({"topic": ...})` still works and is still what +the graph is for, but it has no observation of its own for the five node spans +to hang under, so each one opens a trace of its own and a single run arrives in +Langfuse as five unrelated traces. `write_jokes` is that missing root, and its +argument and return value are what the tracing table shows for the run. + +Three decisions worth naming: + +**Instrumented by hand, not through the LangChain callback handler.** Langfuse's +integration traces what LangChain runs, and after the port the model call is +boto3's `converse`, not a LangChain model — the handler would draw the graph and +leave every generation empty of the model, the prompt and the token counts that +make a generation worth having. Doing it in the nodes also means the tracing +survives the graph: an agent that imports `generate_joke` and calls it directly, +which is exactly what the Ventis port does, still gets its span and its +generation. The handler would see none of that, because in a Ventis container +the graph is never executed. + +**A span and a generation per node, not one observation.** They carry different +things. The generation's output is the model's raw reply; the span's output is +what came back through `_extract_json` and the pydantic schema. That gap is the +whole failure surface this port added when it dropped `with_structured_output`, +and the two observations side by side are what tells "the model wrote prose" +apart from "the model wrote JSON we then mishandled". Each of the four ways a +call can fail sets `level=ERROR` with its own `status_message`, and the raw +reply stays on the generation's output in every one of them. + +**The generations are named per call site, not per function.** All three go +through the same `_ask`, so naming them after it would leave `write-joke` and +`judge-jokes` indistinguishable in a dashboard filter or as an LLM-as-a-judge +target. Names are an API — evaluators and saved views match on them — so they +are meant to stay put. + +No `session_id` or `user_id`: one topic in, one set of jokes out, with no +conversation to group and no caller to attribute. Add a session if this ever +serves a chat. `LANGFUSE_TRACING_ENVIRONMENT` is set in `.env` instead, so runs +from a laptop stay out of deployed dashboards. + +Credentials go in the same `.env` as the Bedrock key — see `.env.example`. With +none of them set the SDK disables itself and the run is untraced, though not +quietly: it logs a "client will be disabled" warning per traced call site. +`LANGFUSE_TRACING_ENABLED=false` removes about half; the rest are the +`langfuse` logger's. + +A Ventis deployment needs `langfuse` added to the `requirements:` list for +`JokeAgent` in `config/global_controller.yaml`, next to `langgraph`. Note that +the spans a replica emits open their own trace: the fan-out crosses a Redis +dispatch, and nothing carries the W3C trace context across it yet. Making one +request one trace under Ventis means propagating `trace_id` and +`parent_span_id` from the workflow into each agent call. + +## Running it + +Copy `.env.example` to `.env` and put a Bedrock API key in it: + +```shell +cp .env.example .env +$EDITOR .env # AWS_BEARER_TOKEN_BEDROCK=bedrock-api-key-... +``` + +`config/global_controller.yaml` points `env_file:` at that file, and every +container gets it as `docker run --env-file`. Nothing in this project reads the +variable: botocore matches the name against `bedrock-runtime`'s signingName and +switches the client from SigV4 to bearer auth on its own, so +`ventis/llm/bedrock.py` still builds a plain `boto3.client("bedrock-runtime")`. +An IAM access key instead of the bearer token works the same way. + +`.env` is gitignored and excluded from the build context — the key is in the +container's environment and not in the image. Deploy checks the path before it +launches anything, so a missing `.env` is one error line rather than three +replicas that come up and fail every request. + +```shell +ventis build +ventis deploy +``` + +```shell +curl -X POST http://localhost:8080/main \ + -H 'Content-Type: application/json' -d '{"query": "animals"}' +curl http://localhost:8080/status/ +``` + +```json +{"request_id": "cb6cb62d...", "status": "done", "result": { + "topic": "animals", + "subjects": ["Wildlife Conservation", "Animal Behavior", "Endangered Species"], + "jokes": ["...", "...", "..."], + "best_selected_joke": "Why did the chimpanzee go to the doctor? Because it was going bananas!" +}} +``` + +`BEDROCK_MODEL_ID` and `AWS_REGION` (see `.env.example`) have defaults in +`joke_writer.py`; neither is a secret. The region has to match the one the key +was issued for. + +### Running the source outside Ventis + +`joke_writer.py` imports `ventis.llm.bedrock` first and falls back to the flat +`bedrock` copy an agent image gets, so the compiled graph still runs on its own +from a checkout of this repo: + +```shell +pip install -e ../.. # the ventis package +pip install langgraph pydantic typing_extensions boto3 langfuse +``` + +```python +from joke_writer import write_jokes + +write_jokes("animals") +``` + +Or as a script, which also flushes the trace before the interpreter exits -- +the SDK ships spans on a background thread, so a short-lived process that does +not flush can drop them: + +```shell +python joke_writer.py animals +``` + +## Provenance + +Taken from `module-4/studio/`, which holds four unrelated graphs sharing one +directory. Only `map_reduce.py` and its license are here. + +| Left behind | Why | +| --- | --- | +| `parallelization.py`, `research_assistant.py`, `sub_graphs.py` | Other graphs in the same studio directory. The first two also need a Tavily key and Wikipedia. | +| `langgraph.json` | Registers all four graphs and points at `./.env`; a trimmed copy would only be useful for LangGraph Studio. | +| The module-4 notebooks | Teaching material for the same code. | +| `OPENAI_API_KEY`, `TAVILY_API_KEY` in `.env.example` | The first belongs to a model call that is no longer here; the second to the two graphs that are not here. | + +Nothing was added at the project root: there is no `pyproject.toml`, `setup.py` +or `requirements.txt`, exactly as upstream has none for module-4. That is why +`config/global_controller.yaml` has to declare `requirements:` by hand. diff --git a/examples/joke_writer_langfuse/joke_writer.py b/examples/joke_writer_langfuse/joke_writer.py new file mode 100644 index 0000000..4ecafe5 --- /dev/null +++ b/examples/joke_writer_langfuse/joke_writer.py @@ -0,0 +1,301 @@ +"""Map-reduce joke writer. + +Derived from langchain-ai/langchain-academy `module-4/studio/map_reduce.py` +(MIT, see LICENSE). The graph shape, the three prompts and the two schemas are +upstream's. The model call is not: upstream builds a `ChatOpenAI` at module +scope, and when this was ported nothing could carry an OPENAI_API_KEY into an +agent container. Bedrock reaches the model through boto3, which builds no client +at import, so the same code loaded with no secret injected. + +`env_file` has since removed that constraint -- the key now travels to the +container in a .env and botocore reads AWS_BEARER_TOKEN_BEDROCK on its own. The +rewrite stayed regardless; README.md says what that costs. + +`with_structured_output` went with it. `call_bedrock` is the raw converse API, so +each node asks for JSON in the prompt and validates the reply through the same +pydantic schema upstream used. + +Tracing +------- +One run is one Langfuse trace. `write_jokes()` is the traced entry point: it +opens the root span, and the three nodes nest under it as spans, each wrapping +the `generation` that records its Bedrock call -- model, prompt, raw reply and +token usage. + +Instrumented by hand rather than through Langfuse's LangChain callback handler. +The handler traces what LangChain runs, and the model call here is boto3's +converse API, so it would draw the graph and leave every generation empty. Doing +it in the nodes also means the tracing survives the graph: an agent that imports +`generate_joke` and calls it directly -- which is what a Ventis port of this +project does -- still gets its span and its generation. + +Tracing is optional at runtime. With no LANGFUSE_* in the environment the SDK +disables itself, every observation below becomes a no-op and the run returns +what it always returned -- it is not silent about it, though: the SDK logs one +"client will be disabled" warning per traced call site. `LANGFUSE_TRACING_ENABLED +=false` removes about half of them, and `logging.getLogger("langfuse")` is where +the rest live. See README.md. +""" + +import json +import operator +import os +import re +import sys +from typing import Annotated + +from typing_extensions import TypedDict + +from pydantic import BaseModel, ValidationError + +from langgraph.constants import Send +from langgraph.graph import END, StateGraph, START + +# Ventis copies bedrock.py flat into every agent image; the package path is for +# running this module outside a container. +try: + from ventis.llm.bedrock import call_bedrock +except ImportError: + from bedrock import call_bedrock + +# Importing langfuse constructs no client and reads no credential -- `get_client()` +# does, on its first call, which happens inside a node. Anything that loads a +# .env before calling one therefore still gets a configured client, whatever the +# import order here. +# +# With no LANGFUSE_* in the environment the SDK disables itself and every +# observation below becomes a no-op, so the module runs untraced rather than +# refusing to run. +from langfuse import get_client, observe + +_langfuse = None + + +def _client(): + """`get_client()`, resolved once. + + Unconfigured, `get_client()` has no public key to file a client under, so it + builds a fresh disabled one -- and logs a fresh "client will be disabled" + warning -- on every call. Holding the first one turns eighteen of those + lines per run into one. + """ + global _langfuse + if _langfuse is None: + _langfuse = get_client() + return _langfuse + +# Prompts we will use. Upstream's, plus the JSON instruction that +# `with_structured_output` used to add on our behalf. +subjects_prompt = """Generate a list of 3 sub-topics that are all related to this overall topic: {topic}. +Respond with JSON only, no prose: {{"subjects": ["...", "...", "..."]}}""" +joke_prompt = """Generate a joke about {subject}. +Respond with JSON only, no prose: {{"joke": "..."}}""" +best_joke_prompt = """Below are a bunch of jokes about {topic}. Select the best one! Return the ID of the best one, starting 0 as the ID for the first joke. Jokes: \n\n {jokes} +Respond with JSON only, no prose: {{"id": 0}}""" + +# LLM. Both are read once at import; the container gets them from its +# environment, and neither is a secret. +MODEL_ID = os.environ.get("BEDROCK_MODEL_ID", "meta.llama3-8b-instruct-v1:0") +REGION = os.environ.get("AWS_REGION", "us-east-1") + + +def _extract_json(text): + """Pull the first JSON object out of a model reply. + + Even told to answer with JSON only, a model wraps it in a ```json fence or + prefaces it with a sentence. Upstream never needed this because + `with_structured_output` handled it; the converse API does not. + """ + text = re.sub(r"^\s*```(?:json)?|```\s*$", "", text.strip(), flags=re.MULTILINE) + try: + return json.loads(text) + except json.JSONDecodeError: + pass + # Fall back to the outermost braced span. + match = re.search(r"\{.*\}", text, flags=re.DOTALL) + if not match: + raise ValueError(f"joke_writer: no JSON in model output: {text!r}") + return json.loads(match.group(0)) + + +def _usage_details(response): + """Bedrock's `usage` block, keyed the way Langfuse prices generations. + + `total` is sent rather than left to be derived: Langfuse derives it as the + sum of every usage type present, which double-counts the moment a cache + figure is included. Bedrock's own totalTokens is input + output. + """ + usage = response.get("usage") or {} + details = { + "input": usage.get("inputTokens"), + "output": usage.get("outputTokens"), + "total": usage.get("totalTokens"), + "cache_read_input_tokens": usage.get("cacheReadInputTokens"), + "cache_write_input_tokens": usage.get("cacheWriteInputTokens"), + } + return {k: v for k, v in details.items() if v is not None} + + +def _ask(prompt, schema, max_tokens, name): + """One converse() call, recorded as a `generation` and validated into `schema`. + + Raising on a bad reply is deliberate. A node that returned a default would + put a plausible-looking wrong answer into the state, and the reduce step + downstream indexes into the jokes list by an id the model chose -- a silent + default there picks the wrong joke instead of failing. + + `name` is the generation's name in Langfuse and belongs to the call site, + not to this function: all three calls are the same converse() and telling + them apart in the UI, in a dashboard filter or in an LLM-as-a-judge target + is the whole point of naming them separately. + """ + with _client().start_as_current_observation( + as_type="generation", + name=name, + model=MODEL_ID, + # A flat role/content list is what Langfuse renders as a conversation. + # Bedrock's own {"content": [{"text": ...}]} shape renders as raw JSON. + input=[{"role": "user", "content": prompt}], + model_parameters={"maxTokens": max_tokens, "temperature": 0.0}, + metadata={"provider": "bedrock", "api": "converse", "region": REGION}, + ) as generation: + try: + response = call_bedrock( + model_id=MODEL_ID, + messages=[{"role": "user", "content": [{"text": prompt}]}], + inference_config={"maxTokens": max_tokens, "temperature": 0.0}, + region=REGION, + ) + except Exception as exc: + # The span would carry the exception either way; the level is what + # makes it filterable next to the replies that came back malformed. + generation.update( + level="ERROR", status_message=f"{type(exc).__name__}: {exc}" + ) + raise + + text = response["output"]["message"]["content"][0]["text"] + generation.update( + output=text, + usage_details=_usage_details(response), + metadata={ + "stop_reason": response.get("stopReason"), + "latency_ms": (response.get("metrics") or {}).get("latencyMs"), + "request_id": (response.get("ResponseMetadata") or {}).get("RequestId"), + }, + ) + + if not text: + generation.update(level="ERROR", status_message="empty completion") + raise ValueError("joke_writer: LLM returned no output.") + try: + return schema(**_extract_json(text)) + except (ValidationError, TypeError) as exc: + generation.update( + level="ERROR", + status_message=f"{schema.__name__} not satisfied by model output", + ) + raise ValueError( + f"joke_writer: {schema.__name__} not satisfied by model output: {text!r}" + ) from exc + except ValueError: + # _extract_json found nothing to parse. The raw reply is already on + # the generation's output, which is where you go to see why. + generation.update(level="ERROR", status_message="no JSON in model output") + raise + + +# Define the state +class Subjects(BaseModel): + subjects: list[str] + +class BestJoke(BaseModel): + id: int + +class OverallState(TypedDict): + topic: str + subjects: list + jokes: Annotated[list, operator.add] + best_selected_joke: str + +@observe(name="generate-topics") +def generate_topics(state: OverallState): + # Set by hand so the span's input is what the node was actually given. Left + # to `@observe` it is the call shape, {"state": {...}}, carrying whatever + # else the graph has accumulated in the state by the time the node runs. + _client().update_current_span(input={"topic": state["topic"]}) + prompt = subjects_prompt.format(topic=state["topic"]) + response = _ask(prompt, Subjects, max_tokens=300, name="split-topic") + return {"subjects": response.subjects} + +class JokeState(TypedDict): + subject: str + +class Joke(BaseModel): + joke: str + +@observe(name="generate-joke") +def generate_joke(state: JokeState): + _client().update_current_span(input={"subject": state["subject"]}) + prompt = joke_prompt.format(subject=state["subject"]) + response = _ask(prompt, Joke, max_tokens=300, name="write-joke") + return {"jokes": [response.joke]} + +@observe(name="select-best-joke") +def best_joke(state: OverallState): + jokes = "\n\n".join(state["jokes"]) + # `subjects` is in the state by now and is not an input to this decision. + _client().update_current_span( + input={"topic": state["topic"], "jokes": state["jokes"]} + ) + prompt = best_joke_prompt.format(topic=state["topic"], jokes=jokes) + response = _ask(prompt, BestJoke, max_tokens=100, name="judge-jokes") + if not 0 <= response.id < len(state["jokes"]): + raise ValueError( + f"joke_writer: model chose joke {response.id} of {len(state['jokes'])}." + ) + return {"best_selected_joke": state["jokes"][response.id]} + +def continue_to_jokes(state: OverallState): + return [Send("generate_joke", {"subject": s}) for s in state["subjects"]] + +# Construct the graph: here we put everything together to construct our graph +graph_builder = StateGraph(OverallState) +graph_builder.add_node("generate_topics", generate_topics) +graph_builder.add_node("generate_joke", generate_joke) +graph_builder.add_node("best_joke", best_joke) +graph_builder.add_edge(START, "generate_topics") +graph_builder.add_conditional_edges("generate_topics", continue_to_jokes, ["generate_joke"]) +graph_builder.add_edge("generate_joke", "best_joke") +graph_builder.add_edge("best_joke", END) + +# Compile the graph +graph = graph_builder.compile() + + +@observe(name="write-jokes") +def write_jokes(topic: str) -> dict: + """Run the graph once. The traced entry point. + + `graph.invoke({"topic": ...})` still works and is still what the graph is + for; it just has no observation of its own to hang the five node spans + under, so each one opens a trace of its own and a single run arrives in + Langfuse as five unrelated traces. This wrapper is that missing root. + + The trace's input and output are this span's, and they are what the tracing + table shows and what a dataset experiment compares across runs. The output + is the returned state; the input is set by hand, because what `@observe` + captures unaided is the call shape -- {"args": ["animals"], "kwargs": {}} -- + rather than the topic. + """ + _client().update_current_span(input={"topic": topic}) + return graph.invoke({"topic": topic}) + + +if __name__ == "__main__": + topic = sys.argv[1] if len(sys.argv) > 1 else "animals" + print(json.dumps(write_jokes(topic), indent=2)) + # Short-lived process: the SDK batches in the background, so without this + # the interpreter can exit before the spans are shipped. `flush()` blocks + # until the queue is drained. + _client().flush()