Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,12 @@ Open PRs against `main` publish a prerelease under the dist-tag `pr-<number>`:
pnpm add -D agent-gwt@pr-123
```

pnpm caches aggressively, so after the pipeline publishes a newer build to the same tag, force a re-resolve:

```bash
pnpm update agent-gwt@pr-123
```

## Releasing

Merging a PR stages the exact prerelease bits as the next semver (not live until approved):
Expand All @@ -19,6 +25,10 @@ pnpm stage approve <stage-id>

Bump size is controlled by PR labels (`major` > `minor` > patch default). See [Publishing](PUBLISHING.md) for trusted-publisher setup.

## Testing

`pnpm test` runs the unit suite with Docker mocked. `pnpm run test:e2e` runs `e2e/` against the real agent images. It needs Docker, and each agent's tests run only when that agent's credential is present on the host: Cursor needs `agent login` (`~/.config/cursor/auth.json`), Claude needs `CLAUDE_CODE_OAUTH_TOKEN` or `ANTHROPIC_API_KEY` in the environment. `globalSetup` builds the images for the agents that have credentials and the rest skip cleanly. On Apple Silicon export `DOCKER_DEFAULT_PLATFORM=linux/amd64` first.

## Architecture

| Layer | Role |
Expand All @@ -28,8 +38,9 @@ Bump size is controlled by PR labels (`major` > `minor` > patch default). See [P
| `agents/` | Shared `createAgent`, Docker invoke, image ensure/build |
| `agents/base/` | Shared Arch base image constants (`agent-gwt/base:local`) |
| `agents/cursor/` | Cursor bindings only (Dockerfile path, image, auth, CLI run) |
| `agents/claude/` | Claude Code bindings only (Dockerfile path, image, credentials, CLI run) |
| `docker/base/` | Shared Arch + yay Dockerfile (all agents `FROM` this tag) |
| `docker/<agent>/` | Per-agent Dockerfile (`FROM agent-gwt/base:local` + that product’s CLI) |
| `package-root` | Relative resolve to this package’s root (`src/` or `lib/` parent) — no directory scans |

Additional agents (Devin, Claude, Copilot, …) add `docker/<name>/Dockerfile` on the shared base, a folder under `agents/`, and a registry entry.
Additional agents (Devin, Copilot, …) add `docker/<name>/Dockerfile` on the shared base, a folder under `agents/`, and a registry entry.
64 changes: 57 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

GWT step functions for repeatable **agent** tests. Works with [vitest-gwt](https://github.com/devzeebo/vitest-gwt) / [gwt-runner](https://github.com/devzeebo/gwt-runner).

v1 ships the **Cursor** agent: create a temp workspace, mount **only** Cursor credentials, run the agent as your host user, and put parsed `--output-format json` on the test context.
Ships the **Cursor** and **Claude Code** agents: create a temp workspace, mount **only** the agent's credentials, run the agent as your host user, and put parsed `--output-format json` on the test context.

## Install

Expand All @@ -13,19 +13,27 @@ pnpm add -D agent-gwt vitest vitest-gwt
## Prerequisites

1. Docker
2. Host Cursor CLI login (`agent login`) so `~/.config/cursor/auth.json` exists
2. Host login for the agent(s) you use:
- **Cursor:** `agent login` so `~/.config/cursor/auth.json` exists
- **Claude Code:** `CLAUDE_CODE_OAUTH_TOKEN` (from `claude setup-token` — Claude subscription) **or** `ANTHROPIC_API_KEY` in the environment, **or** a Linux host's `~/.claude/.credentials.json`. Checked in that order. macOS keeps Claude Code's login in the Keychain, so on a Mac set one of the two variables:

```bash
claude setup-token # prints a long-lived token
export CLAUDE_CODE_OAUTH_TOKEN=<token> # or: export ANTHROPIC_API_KEY=sk-ant-...
```
3. Build the agent Docker image **once per suite** via vitest `globalSetup` (or manually)

## Setup

Build images once in `globalSetup` so parallel test files do not race:
Build images once in `globalSetup` so parallel test files do not race. Build only the agents your suite uses:

```ts
// vitest.global-setup.ts
import { buildAgentImage } from "agent-gwt";

export default async function setup() {
await buildAgentImage("cursor");
await buildAgentImage("claude");
}
```

Expand All @@ -44,6 +52,13 @@ export default defineConfig({

Wire the agent and a disposable workspace with `withAspect`, then write Given/When/Then tests. Agent runs are slow — raise the timeout.

Pick the agent with `agent({ name, model })`; nothing else in the test changes:

```ts
withAspect(agent({ name: "cursor", model: "auto" })); // Cursor CLI
withAspect(agent({ name: "claude", model: "sonnet" })); // Claude Code
```

### Simple prompt

```ts
Expand Down Expand Up @@ -159,14 +174,43 @@ async function question_is_answered(this: Context) {
}
```

### Inspecting the result

`this.agentResult` is the parsed JSON the CLI printed, for either agent. For Claude Code, `ClaudeAgentResult` types the useful fields:

```ts
import type { ClaudeAgentResult } from "agent-gwt";

function used_one_turn(this: Context) {
const result = this.agentResult as ClaudeAgentResult;

expect(result.is_error).toBe(false);
expect(result.num_turns).toBeGreaterThan(0);
expect(result.total_cost_usd).toBeLessThan(0.5);
}
```

A Claude run whose JSON reports `is_error: true` throws from `executing_the_agent` with the agent's message, so a failing run surfaces as the real cause rather than a downstream assertion.

## Docker images

| Image | Role |
| --- | --- |
| `agent-gwt/base:local` | Shared Arch Linux base (`yay` + `aur` user). Used by all agents. |
| `agent-gwt/cursor-cli:local` | Cursor CLI on top of the base |
| `agent-gwt/claude-code:local` | Claude Code CLI (native binary) on top of the base |

`buildAgentImage("cursor")` builds the base first, then the Cursor image; `buildAgentImage("claude")` does the same for Claude Code.

### Apple Silicon

The official `archlinux` image is x86_64-only. On an arm64 Docker host, build and run under amd64 emulation:

```bash
export DOCKER_DEFAULT_PLATFORM=linux/amd64
```

`buildAgentImage("cursor")` builds the base first, then the Cursor image.
Docker Desktop applies this to both `docker build` and `docker run`, so nothing in the library changes.

### Extending with toolchains

Expand All @@ -175,6 +219,7 @@ Install packages in a child image, then point tests at that tag:
```dockerfile
# docker/agent.Dockerfile
FROM agent-gwt/cursor-cli:local
# or: FROM agent-gwt/claude-code:local

# Official Arch packages (as root)
RUN pacman -Sy --noconfirm --needed nodejs npm python rust \
Expand Down Expand Up @@ -218,27 +263,32 @@ Pair workspace lifecycle separately: `withAspect(a_workspace, cleanup_workspace)
## What `executing_the_agent` does

1. Requires `this.workspace`, `this.prompt`, and `this.agent`
2. Calls `this.agent.run(...)` with `this.image` (Cursor: `docker run` with credentials-only mount + `agent -p --force --output-format json`)
2. Calls `this.agent.run(...)` with `this.image`:
- Cursor: `docker run` with credentials-only mount + `agent -p --force --output-format json [--model …] -- <prompt>`
- Claude: `docker run` with the workspace mount and credentials forwarded by env **name** (the value never appears on the host command line) or a read-only `.credentials.json` mount + `claude -p --output-format json --dangerously-skip-permissions [--model …] -- <prompt>`
3. Sets `this.agentResult` to the parsed JSON

## Exports

| Export | Role |
| --- | --- |
| `AgentContext` | Extensible context type (`workspace`, `prompt`, `agent`, `image`, …) |
| `agent(opts)` | `withAspect` before — `{ name, model?, image? }` |
| `agent(opts)` | `withAspect` before — `{ name: "cursor" \| "claude", model?, image? }` |
| `buildAgentImage(name)` | Suite setup — builds base + agent image (use in vitest `globalSetup`) |
| `buildBaseImage()` | Builds `agent-gwt/base:local` only |
| `buildDockerImage(...)` | Builds an arbitrary Dockerfile (e.g. toolchain overlay) |
| `a_workspace` | Creates `/tmp/.agents-gwt/ws-*` (use in `withAspect` before, or in `given`) |
| `cleanup_workspace` | Remove the temp workspace (use in `withAspect` after) |
| `the_prompt(text)` | Curried `given` — sets `this.prompt` |
| `executing_the_agent` | `when` — runs `this.agent.run(...)` |
| `ClaudeAgentResult` | Type for Claude Code's JSON result (`is_error`, `result`, `num_turns`, `total_cost_usd`, …) |
| `ClaudeCredentials` / `resolveClaudeCredentials()` | Credential source for the Claude container (token, API key, or file) |

## Isolation notes

- **Credentials only:** settings, MCP config, projects, and skills from `~/.cursor` are not mounted.
- **Credentials only:** settings, MCP config, projects, and skills from `~/.cursor` are not mounted. Likewise nothing from `~/.claude` (settings, MCP servers, plugins, skills, projects, hooks) reaches the Claude container, and its auto-updater, telemetry, and error reporting are disabled in the image.
- **Non-root:** the container process uses your host uid/gid so workspace files are owned by you.
- **Workspace is the only writable host path.** A `CLAUDE.md` seeded into the workspace is honoured, because Claude Code reads it from the working directory.

## Contributing

Expand Down
13 changes: 11 additions & 2 deletions docker/base/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,9 +1,18 @@
# Shared base for agent-gwt agent images (cursor, and future agents).
# Shared base for agent-gwt agent images (cursor, claude, and future agents).
# Extend with: FROM agent-gwt/base:local
# AUR installs (build-time only): USER aur && yay -S --noconfirm ... && USER root
FROM archlinux:latest

RUN pacman -Sy --noconfirm --needed \
# pacman 7 sandboxes its downloader: drops to the `alpm` user, then applies a
# Landlock filesystem rule and a seccomp syscall denylist. Upstream already
# disables the Landlock half (no Landlock in container kernels). The seccomp
# half cannot load under Rosetta/QEMU user-mode emulation (every seccomp entry
# point returns EINVAL, regardless of --privileged or seccomp=unconfined), which
# breaks builds on Apple Silicon. Disabling only the syscall filter keeps the
# user drop + NO_NEW_PRIVS and is a no-op difference on native x86_64 hosts.
RUN sed -i 's/^#DisableSandboxSyscalls/DisableSandboxSyscalls/' /etc/pacman.conf \
&& grep -q '^DisableSandboxSyscalls' /etc/pacman.conf \
&& pacman -Sy --noconfirm --needed \
base-devel git sudo curl ca-certificates \
&& useradd -m aur \
&& echo 'aur ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/aur \
Expand Down
33 changes: 33 additions & 0 deletions docker/claude/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Claude Code agent image. Shared Arch/yay base is agent-gwt/base:local.
# Extend with toolchains: FROM agent-gwt/claude-code:local
FROM agent-gwt/base:local

# Never self-update or phone home from inside the container — build steps included.
ENV DISABLE_AUTOUPDATER=1
ENV DISABLE_TELEMETRY=1
ENV DISABLE_ERROR_REPORTING=1

# Official native installer drops a single self-contained binary under
# ~/.local/share/claude/versions/<version>. Relocate it to a world-readable
# path so arbitrary host UIDs (docker --user) can run claude.
RUN curl -fsSL https://claude.ai/install.sh | bash \
&& VERSION_BIN="$(find /root/.local/share/claude/versions -mindepth 1 -maxdepth 1 -type f | sort -V | tail -1)" \
&& test -n "$VERSION_BIN" \
&& install -m 0755 "$VERSION_BIN" /usr/local/bin/claude \
&& rm -rf /root/.local/share/claude /root/.local/bin/claude \
&& claude --version

# Empty home for arbitrary host UIDs. ~/.claude is pre-created so a read-only
# .credentials.json bind mount does not leave the directory root-owned.
RUN mkdir -p /home/agent/.claude \
&& chmod -R 0777 /home/agent

ENV HOME=/home/agent
ENV PATH="/usr/local/bin:${PATH}"

WORKDIR /workspace

# Runtime identity is set by agent-gwt via --user <host-uid>:<host-gid>.
# Credentials arrive as env (CLAUDE_CODE_OAUTH_TOKEN / ANTHROPIC_API_KEY) or a
# read-only ~/.claude/.credentials.json mount — see src/agents/claude/run.ts.
# No ENTRYPOINT — the library passes `claude ...` as the container command.
43 changes: 43 additions & 0 deletions e2e/claude.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
import { describe, expect } from "vitest";
import test, { withAspect } from "vitest-gwt";
import {
type AgentContext,
type ClaudeAgentResult,
a_workspace,
agent,
cleanup_workspace,
executing_the_agent,
the_prompt,
} from "../src/index.js";
import { hasClaudeCredential } from "./credentials.js";
import { readme_contains_HELLO_WORLD, readme_exists } from "./steps.js";

describe.skipIf(!hasClaudeCredential())("claude agent (e2e)", () => {
withAspect(agent({ name: "claude", model: "sonnet" }));
withAspect(a_workspace, cleanup_workspace);

test("writes the readme", {
given: {
the_prompt: the_prompt("Write 'Hello World' to README.md"),
},
when: {
executing_the_agent,
},
then: {
readme_exists,
readme_contains_HELLO_WORLD,
result_is_a_successful_claude_run,
},
});
});

function result_is_a_successful_claude_run(this: AgentContext) {
const result = this.agentResult as ClaudeAgentResult;

expect(result.type).toBe("result");
expect(result.is_error).toBe(false);
expect(result.num_turns).toBeGreaterThan(0);
console.log(
`claude: ${result.num_turns} turns, $${result.total_cost_usd.toFixed(4)}, session ${result.session_id}`,
);
}
26 changes: 26 additions & 0 deletions e2e/credentials.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import { existsSync } from "node:fs";
import { homedir } from "node:os";

import {
CLAUDE_API_KEY_ENV,
CLAUDE_OAUTH_TOKEN_ENV,
defaultClaudeHostCredentialsFile,
defaultHostAuthFile,
} from "../src/index.js";

/** Same sources as resolveClaudeCredentials(), as a sync yes/no for skip decisions. */
export function hasClaudeCredential(env: NodeJS.ProcessEnv = process.env): boolean {
const token = env[CLAUDE_OAUTH_TOKEN_ENV];
const apiKey = env[CLAUDE_API_KEY_ENV];

return (
(token !== undefined && token !== "") ||
(apiKey !== undefined && apiKey !== "") ||
existsSync(defaultClaudeHostCredentialsFile(homedir()))
);
}

/** Same check runCursorInDocker() makes before it starts the container. */
export function hasCursorCredential(): boolean {
return existsSync(defaultHostAuthFile(homedir()));
}
35 changes: 35 additions & 0 deletions e2e/cursor.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
import { describe, expect } from "vitest";
import test, { withAspect } from "vitest-gwt";
import {
type AgentContext,
a_workspace,
agent,
cleanup_workspace,
executing_the_agent,
the_prompt,
} from "../src/index.js";
import { hasCursorCredential } from "./credentials.js";
import { readme_contains_HELLO_WORLD, readme_exists } from "./steps.js";

describe.skipIf(!hasCursorCredential())("cursor agent (e2e)", () => {
withAspect(agent({ name: "cursor", model: "auto" }));
withAspect(a_workspace, cleanup_workspace);

test("writes the readme", {
given: {
the_prompt: the_prompt("Write 'Hello World' to README.md"),
},
when: {
executing_the_agent,
},
then: {
readme_exists,
readme_contains_HELLO_WORLD,
result_is_parsed_json,
},
});
});

function result_is_parsed_json(this: AgentContext) {
expect(this.agentResult).toBeTypeOf("object");
}
24 changes: 24 additions & 0 deletions e2e/global-setup.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
import { type AgentName, buildAgentImage } from "../src/index.js";
import { hasClaudeCredential, hasCursorCredential } from "./credentials.js";

const agents: Array<{ name: AgentName; available: boolean; hint: string }> = [
{ name: "cursor", available: hasCursorCredential(), hint: "run `agent login` on the host" },
{
name: "claude",
available: hasClaudeCredential(),
hint: "set CLAUDE_CODE_OAUTH_TOKEN or ANTHROPIC_API_KEY",
},
];

export default async function setup() {
for (const agent of agents) {
if (!agent.available) {
process.stderr.write(
`[e2e] No ${agent.name} credential (${agent.hint}); skipping its image build and tests.\n`,
);
continue;
}

await buildAgentImage(agent.name);
}
}
17 changes: 17 additions & 0 deletions e2e/steps.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import { access, readFile } from "node:fs/promises";
import { join } from "node:path";
import { expect } from "vitest";

import type { AgentContext } from "../src/index.js";

// Shared "then" steps: the same assertions run against whichever agent wrote the file.

export async function readme_exists(this: AgentContext) {
await access(join(this.workspace, "README.md"));
}

export async function readme_contains_HELLO_WORLD(this: AgentContext) {
const contents = await readFile(join(this.workspace, "README.md"), "utf-8");

expect(contents.toLowerCase()).toContain("hello world");
}
16 changes: 16 additions & 0 deletions e2e/vitest.e2e.config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
// End-to-end suite: builds the real agent images and runs each agent in Docker.
// Each agent's tests run only when its credential is present on the host and skip
// cleanly otherwise (Cursor: `agent login`; Claude: CLAUDE_CODE_OAUTH_TOKEN or
// ANTHROPIC_API_KEY). On Apple Silicon export DOCKER_DEFAULT_PLATFORM=linux/amd64.
//
// pnpm run test:e2e
import { defineConfig } from "vitest/config";

export default defineConfig({
test: {
include: ["e2e/**/*.spec.ts"],
globalSetup: ["./e2e/global-setup.ts"],
testTimeout: 180_000,
hookTimeout: 600_000,
},
});
Loading
Loading