Skip to content

Run agents on imported harness adapters - #70

Closed
jacobprall wants to merge 1 commit into
render-examples:mainfrom
jacobprall:harness-refactor
Closed

jacobprall wants to merge 1 commit into
render-examples:mainfrom
jacobprall:harness-refactor

Conversation

@jacobprall

Copy link
Copy Markdown
Collaborator

Fixes #67. First of three: the Pydantic AI (#69) and Cursor SDK (#68) adapter PRs build on this one.

An agent now names a harness, a value built by an adapter in app/harness/ from its library's own configuration, instead of a model-tier string. The adapter you import is the runtime, and the model is whatever you gave the library:

// app/agents/models.ts
export const medium = pi({ model: getBuiltinModel("anthropic", "claude-sonnet-5") });
export const large = pi({ model: getBuiltinModel("anthropic", "claude-opus-5") });

getBuiltinModel is typed against pi's catalog, so a mistyped model ID fails npm run typecheck. config.models, ModelTier, and modelFor are gone. Default behavior is unchanged: same models, thinking levels, tools, and outputs.

Design

  • The runner is split. app/runner/ now has three parts:
    • core/: the versioned contract, the event emitter with redaction and one final event, the submit checks, input parsing, and keys.
    • tools/: the shared factory tools, plus a FactoryTool view with no library types.
    • pi/: the pi loop.
  • Agent definitions are bundled into the Sandbox runner. The runner runs the agent named in its input on that agent's harness. Prompts are read lazily on the host. The prompt, tools, output schema, and limits still come from the host, whose Zod schema stays authoritative; the bundle supplies only the harness.
  • The contract is versioned, at protocol 2. The input carries protocol_version; model_provider, model, and thinking_level are gone. started reports harness, harness_version, and model, and agent.finished records them.
  • Unsupported tools fail at startup. An adapter declares which factory tools it can provide, and whether its bash applies the read-only policy. An agent whose harness can't give it its tools fails when it's defined.
  • Bundle changes reach existing snapshots. Each Sandbox compares its runner.mjs hash with this deploy's bundle before verify.sh runs, and installs this deploy's bundle, root-owned, when they differ. So harness and model changes need no snapshot refresh.
  • Keys come from the harnesses. runAgent, doctor, and the software-factory-workflows env group derive the key list from the agents' harnesses.

Two simplifications from the issue: the input keeps prompt, tools, schema, and limits (no prompt inlining needed), and the host compares file hashes itself instead of sending a bundle_hash.

Tests

  • A conformance suite over the runner contract runs against the pi harness and a scripted harness with no agent library. That proves a harness can be swapped without host changes. It covers:
    • exactly one final event;
    • the three-attempt submit rule;
    • max_turns;
    • no key in the output or tool environment;
    • the read-only shell policy;
    • edit and write.
  • Bundled runner: it rejects unknown agents, input from another protocol version, and test scripts without --test-stub. --self-test reports every bundled harness.
  • Import boundary: a test fails if the runner bundle imports host-only code. I confirmed it catches an injected app/config import.
  • Sandbox: a stale bundle is replaced, root-owned, before verify.sh; a matching one is left alone.

Verification

  • npm run check passes: 153 tests.
  • A live run of the bundled runner against Claude Sonnet 5 succeeded through the new path.
  • Deployed to a test deployment (live-test on the fork). A Triage run on a real repository replaced the snapshot's stale runner bundle before verification, ran Triage on Sonnet and the spec writer on Opus, and recorded harness, harnessVersion, and model on agent.finished.

Deploying

Deploy factory-workflows after merging. Existing snapshots get the new bundle when each Sandbox opens.

An agent now names a harness value built by an adapter in app/harness/
from its library's own configuration, instead of a model tier string.
The pi adapter takes a typed model from Pi's catalog, so a mistyped
model ID fails typecheck. config.models, ModelTier, and modelFor are
gone; the built-in choices live in app/agents/models.ts.

A harness is the runtime the runner calls in the Sandbox (run and
selfTest, app/runner/core/harness.ts) plus what the host needs before a
run (a label and the credential env var, app/harness/types.ts). What a
harness must do beyond those signatures, such as giving the model only
the factory's tools and submit, is pinned by the conformance suite.

The runner is split into a library-neutral core (versioned contract,
emitter, submit checks, input parsing, keys), shared factory tools with
a library-neutral view, and the pi loop. Agent definitions are bundled
into the runner, which runs the agent named in its input on that
agent's harness.

Each Sandbox replaces a snapshot's runner bundle with this deploy's
when their hashes differ, before verify.sh, so harness and model
changes need no snapshot refresh. Keys come from the harnesses:
runAgent, doctor, and the workflows env group derive them from each
harness's credential. agent.finished records the model the runner
reports.

Tests run a conformance suite over the runner contract against the pi
harness and a scripted harness with no agent library, and check that
the bundle imports no host-only code.

Refs render-examples#67
@jacobprall

Copy link
Copy Markdown
Collaborator Author

Consolidated into #73.

@jacobprall jacobprall closed this Oct 27, 2026
@jacobprall
jacobprall deleted the harness-refactor branch October 27, 2026 23:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Refactor agents around imported harness adapters

1 participant