[jacobprall]

Fixes #69. Depends on #70; its commit is included here until #70 merges. Review the two commits after it: the adapter, and the telemetry fix.

pydantic({ module, key }) runs an agent on a Pydantic AI agent built in Python:

# python/agents/sonnet.py
from pydantic_ai import Agent
from factory_harness import serve

serve(Agent("anthropic:claude-sonnet-5"))

harness: pydantic({ module: "agents/sonnet.py", key: "ANTHROPIC_API_KEY" }),

The module supplies the model and library options. The factory supplies the prompt (as the run's instructions), its own tools, the submit tool, and the limits, as for every harness.

How it runs

• Startup and key handling:
  • The runner starts python -m factory_harness as the unprivileged runner user and passes the input and key on stdin.
  • The wrapper makes its process non-dumpable before reading the key, so tool subprocesses can't read the key from its memory.
  • It sets the key only in its own environment, for the provider to read.
• Output: submit is an output function whose parameter schema is the agent's output schema. It checks each submission with jsonschema and raises ModelRetry, following the same three-attempt rule and messages as core/submit.ts. The host still revalidates with Zod. max_turns maps to UsageLimits(request_limit), and the time budget to the run timeout.
• Tools: runner.mjs --mcp-tools serves exactly the agent's factory tools over MCP stdio, using the same implementations, scrubbed environment, and read-only policy as pi. It never receives the key.
• Events: the runner revalidates every event the wrapper emits before forwarding it. Usage is priced with Pydantic AI's price data.

Sandbox setup: provision

Instead of app/sandbox/ learning what Python is, a harness now declares its Sandbox setup:

provision?: () => { id; files: { path; source }[]; command?; hash }

• Payload: the host ships the declared files with the runner bundle as one payload, swapped in root-owned when its hash differs.
• Commands: each command runs as root, only when its hash differs from the one the Sandbox recorded, at snapshot build and when a Sandbox opens, before the self-test.
• Pydantic AI's setup: the adapter ships python/factory_harness, the agent modules, and the hash-pinned requirements.txt. Its command builds a root-owned virtual environment with the base image's Python 3.11 and pip, keyed on the requirements hash. Deployments without a pydantic agent get no Python.

Python dependencies are locked with uv in python/. npm run python:lock regenerates uv.lock and requirements.txt.

Also in this PR

• Emitter fix: createEmitter attached its terminated getter with Object.assign, which copied the value (false) once, so emit.terminated was always false. It now reflects the stream. This made a successful pydantic run report exit code 1.
• Telemetry: agent.finished records the model the runner reports in started, since a pydantic harness's model lives in Python.

Tests

• Conformance suite: the pydantic harness runs the whole suite when python/.venv exists. A new CI job syncs the locked environment, checks that requirements.txt matches uv.lock, and runs it.
• Sandbox: provision tests use a fake harness, so app/sandbox/ is checked generically: declared files ship, the command runs after the swap and before verification, and both are skipped when hashes match.
• Adapter: tests check module path validation, the shipped files, the setup hash, and that the version pin matches pyproject.toml.

Verification

• npm run check passes: 162 tests, including the pydantic leg.
• Local live run: Sonnet called the factory's ls and read over MCP, then submitted valid output. No key appeared in the output.
• On the test deployment:
  • Triage ran on Pydantic AI and the spec writer on pi, in one task.
  • The Sandbox swapped in the payload and built the Python environment in about 12 seconds before verification.
  • A snapshot refresh applied the setup at build, and a later Sandbox skipped it.

Deploying

Deploy factory-workflows. To run an agent on Pydantic AI, point its harness at a module in python/agents/, then refresh snapshots so Sandboxes don't each rebuild the environment.

[jacobprall]

Consolidated into #73.