Benchmarked on what users feel.
Interaction Bench (ibench) evaluates whether coding agents can investigate and repair interaction defects in real applications — problems a person feels while using the product, not failures a unit test already catches.
This repository ships the CLI, 44 published tasks across 25 open-source apps, solver/scorer configs, and the harness needed to run reproducible evaluations.
v0.1 preview. On the public leaderboard, the best reported pass rate is 34.1% (
GPT-6 Astra). Scores measure repairs judged against reviewed behavioral requirements in fresh environments — not unit-test green.
- 📣 29 Oct 2026 — Introduced Interaction Bench: tasks from real open-source maintenance cases, graded from fresh execution evidence. Read the announcement.
- Evaluate interaction repair, not just code that compiles: agents get the user intention, source, and a runnable offline environment — the diagnosed bug and reference fix are withheld.
- Run real product tasks: each task comes from a merged pull request that fixed a usability problem in an open-source app (map panning, form feedback, publish toasts, and similar).
- Grade from behavior: a separate scoring agent replays the interaction on the patched app and checks expert-defined behavioral requirements; alternate repairs can pass without copying the reference patch.
- Reproduce end to end: locked environments, published configs under
configs/, resume for interrupted runs, and deferred scoring with--solver-only. - Inspect locally: preview task intents, rubrics, and reference recordings; view Harbor trajectories after a run.
In scope are usability problems that show up in use — for example memory demands, unclear feedback, or limited control. Out of scope are appearance-only changes and overall experience ratings. See the about and methodology pages for the full framing.
- Start with a real task — intent, source, and an offline environment from a real maintenance case.
- Investigate and repair — the agent explores the app and code, decides what should change, and submits a patch.
- Judge the repair — a grading agent recreates the scenario, replays the interaction, and checks the outcome against reviewed requirements.
v0.1 preview, full public leaderboard
| Model | Harness | Pass rate (passed / total) | Trajectory |
|---|---|---|---|
| GPT-6 Astra | Codex | 34.1% (15/44) | View in harborhub |
| Claude Opus 5.5 | Claude Code | 27.3% (12/44) | View in harborhub |
| DeepSeek Flash | DSH | 15.9% (7/44) | View in harborhub |
| GLM-5.3 Flash | ZCode | 9.1% (4/44) | View in harborhub |
| Qwen3.8 Max | Codex | 6.8% (3/44) | View in harborhub |
| Kimi K3 | Kimi Code | 4.5% (2/44) | View in harborhub |
git clone https://github.com/eval-sys/ibench.git
cd ibench
uv sync --locked
cp .env.example .env
bash scripts/build-dev-images.sh
uv run ibench check .Add solver and scorer credentials to .env. The example uses OPENAI_API_KEY and DEEPSEEK_API_KEY. Edit files in configs/ to choose other models or endpoints. Requires Python 3.12+ and Docker for task environments.
uv run ibench run \
--task photo-map-panning \
--solver-config configs/gpt-6-astra-high.json \
--score-config configs/deepseek-flash.json \
--output jobs/example \
--n-concurrent 2Repeat --task to select tasks; omit it to run all 44. Task names and IDs are listed in tasks/release.json. Add --profile intent-actions to include interaction steps alongside the default user goal (intent).
Resume an interrupted run:
uv run ibench resume jobs/exampleDefer scoring with --solver-only, then score collected patches:
uv run ibench score jobs/example \
--config configs/deepseek-flash.json --n-concurrent 2Preview tasks and videos at http://127.0.0.1:8090:
uv run ibench preview --no-openView results and trajectories:
uv run harbor view jobs/example/jobsSee uv run ibench --help for debugging, recording, and other commands. Use the audit-cases skill to review task quality.
| Goal | Command |
|---|---|
| Validate the release | uv run ibench check . |
| Run selected / all tasks | uv run ibench run --output jobs/… --solver-config … --score-config … |
| Resume a run | uv run ibench resume jobs/… |
| Score an existing run | uv run ibench score jobs/… --config … |
| Browse tasks locally | uv run ibench preview |
| Start an app for inspection | uv run ibench debug '<task>' --revision base --fixture before --output jobs/… |
| Replay before / reference-after | uv run ibench replay '<task>' --revision … --fixture … --output jobs/… |
Published solver and scorer configs live in configs/ (for example gpt-6-astra-high.json, deepseek-flash.json, claude-opus-5-5-high.json). Instruction profiles in tasks/release.json: intent (default) and intent-actions.
More detail: docs quickstart · task catalog · environments · pipeline.
- Per-run artifacts are written under the
--outputdirectory (Harbor jobs, patches, scores). - Aggregate pass rate, cost, latency, and per-task outcomes are published on the leaderboard.
- Explore individual trajectories on the site when a run is published.
The 0.1 site release is a preview; treat leaderboard numbers as the current published snapshot, not a frozen paper result.
Contributions that improve tasks, configs, or the harness are welcome.
- Install as in Quickstart, then validate with
uv run ibench check .. - Use
preview,debug, andreplaywhile iterating on task materials (contribute guide). - Open a pull request against eval-sys/ibench.
Original Interaction Bench code and benchmark material use Apache-2.0. Third-party application source, assets, and trademarks retained or executed by tasks keep their own terms.
See LICENSE, NOTICE, and THIRD_PARTY_NOTICES.md.