Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
10 changes: 10 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -473,3 +473,13 @@ scripts/run_tasks_45_60.sh
.venv/
/runs/
/agents/configs/*.local.yaml

# Generated trajectory reports and captured API responses
/viz_task_*.html
/task*_viz.html
/trajectory_visualization.html
/agents/error_captures/
# Historical output locations; new task reviews go under runs/.
/evaluate_results/
/evaluate_results_augmented/
/investigate_results/
6 changes: 6 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,9 @@ python benchmarks/score.py --help
Live game/model runs are separate validation. Include the task/harness revision,
settings, and coverage with those results. Keep credentials, virtual environments,
and generated run artifacts out of commits.

Generated trajectories, HTML visualizations, and task-review output belong in
ignored `runs/` directories. Do not commit model API response dumps from
`agents/error_captures/`. Retained historical reports live in `analysis/archive/`;
new utility scripts belong in an appropriate `scripts/` or `analysis/scripts/`
subdirectory rather than the repository root.
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ agent harness, benchmark tasks, and evaluation utilities.
| Run your model on a task, then a benchmark suite | [Benchmark quickstart](docs/benchmark/quickstart.md) |
| Integrate your own agent harness or model adapter | [Custom agents](docs/benchmark/custom-agents.md) |
| Calculate scores and prepare a submission | [Scoring and results](docs/benchmark/scoring.md) |
| Inspect a trajectory visually | [Trajectory visualization](docs/benchmark/visualization.md) |
| Understand where code belongs | [Repository map](docs/repository-map.md) |
| Start or configure the game server | [Server setup](docs/benchmark/server.md) |

Expand All @@ -26,8 +27,9 @@ benchmarks/ Benchmark entry-point documentation and offline SR repo
data_v0.1_multi/ Versioned multi-agent task definitions and verifiers
data_v0.1_solo/ Solo task definitions
examples/agents/ Custom-agent adapter example
analysis/ Research analysis and CCE judge scripts
analysis/ Research analysis, CCE scripts, and historical report archive
packages/ TypeScript game server, client, shared code, and tools
scripts/ Development utilities and manual game diagnostics
docs/ Onboarding, protocol notes, and game documentation
tests/benchmark/ Offline regression tests for benchmark entry points
```
Expand Down
310 changes: 0 additions & 310 deletions agents/error_captures/520_error_20260116_035549.json

This file was deleted.

241 changes: 0 additions & 241 deletions agents/error_captures/520_error_20260116_080852.json

This file was deleted.

310 changes: 0 additions & 310 deletions agents/error_captures/520_error_20260116_082100.json

This file was deleted.

310 changes: 0 additions & 310 deletions agents/error_captures/520_error_20260116_082353.json

This file was deleted.

310 changes: 0 additions & 310 deletions agents/error_captures/520_error_20260116_082456.json

This file was deleted.

340 changes: 0 additions & 340 deletions agents/error_captures/520_error_20260116_082941.json

This file was deleted.

340 changes: 0 additions & 340 deletions agents/error_captures/520_error_20260116_083401.json

This file was deleted.

739 changes: 0 additions & 739 deletions agents/error_captures/520_error_20260116_100159.json

This file was deleted.

504 changes: 0 additions & 504 deletions agents/error_captures/520_error_20260116_123319.json

This file was deleted.

504 changes: 0 additions & 504 deletions agents/error_captures/520_error_20260116_123522.json

This file was deleted.

879 changes: 0 additions & 879 deletions agents/error_captures/520_error_20260116_130626.json

This file was deleted.

356 changes: 0 additions & 356 deletions agents/error_captures/520_error_20260116_172323.json

This file was deleted.

839 changes: 0 additions & 839 deletions agents/error_captures/520_error_20260116_201332.json

This file was deleted.

877 changes: 0 additions & 877 deletions agents/error_captures/520_error_20260116_202026.json

This file was deleted.

1,140 changes: 0 additions & 1,140 deletions agents/error_captures/520_error_20260116_211528.json

This file was deleted.

899 changes: 0 additions & 899 deletions agents/error_captures/520_error_20260117_091230.json

This file was deleted.

1,272 changes: 0 additions & 1,272 deletions agents/error_captures/520_error_20260117_104519.json

This file was deleted.

977 changes: 0 additions & 977 deletions agents/error_captures/520_error_20260117_135601.json

This file was deleted.

850 changes: 0 additions & 850 deletions agents/error_captures/520_error_20260117_182500.json

This file was deleted.

850 changes: 0 additions & 850 deletions agents/error_captures/520_error_20260117_183302.json

This file was deleted.

1,276 changes: 0 additions & 1,276 deletions agents/error_captures/520_error_20260117_184538.json

This file was deleted.

1,295 changes: 0 additions & 1,295 deletions agents/error_captures/520_error_20260117_185108.json

This file was deleted.

903 changes: 0 additions & 903 deletions agents/error_captures/520_error_20260117_185435.json

This file was deleted.

903 changes: 0 additions & 903 deletions agents/error_captures/520_error_20260117_185545.json

This file was deleted.

1,314 changes: 0 additions & 1,314 deletions agents/error_captures/520_error_20260117_185850.json

This file was deleted.

941 changes: 0 additions & 941 deletions agents/error_captures/520_error_20260117_191130.json

This file was deleted.

941 changes: 0 additions & 941 deletions agents/error_captures/520_error_20260117_191249.json

This file was deleted.

945 changes: 0 additions & 945 deletions agents/error_captures/520_error_20260117_191712.json

This file was deleted.

945 changes: 0 additions & 945 deletions agents/error_captures/520_error_20260117_191824.json

This file was deleted.

960 changes: 0 additions & 960 deletions agents/error_captures/520_error_20260117_192532.json

This file was deleted.

1,021 changes: 0 additions & 1,021 deletions agents/error_captures/520_error_20260117_194543.json

This file was deleted.

1,017 changes: 0 additions & 1,017 deletions agents/error_captures/520_error_20260117_195141.json

This file was deleted.

1,017 changes: 0 additions & 1,017 deletions agents/error_captures/520_error_20260117_195304.json

This file was deleted.

1,017 changes: 0 additions & 1,017 deletions agents/error_captures/520_error_20260117_195415.json

This file was deleted.

1,040 changes: 0 additions & 1,040 deletions agents/error_captures/520_error_20260117_200454.json

This file was deleted.

1,055 changes: 0 additions & 1,055 deletions agents/error_captures/520_error_20260117_201316.json

This file was deleted.

1,055 changes: 0 additions & 1,055 deletions agents/error_captures/520_error_20260117_201514.json

This file was deleted.

1,215 changes: 0 additions & 1,215 deletions agents/error_captures/520_error_20260117_211958.json

This file was deleted.

1,211 changes: 0 additions & 1,211 deletions agents/error_captures/520_error_20260117_212449.json

This file was deleted.

1,257 changes: 0 additions & 1,257 deletions agents/error_captures/520_error_20260117_214151.json

This file was deleted.

1,276 changes: 0 additions & 1,276 deletions agents/error_captures/520_error_20260117_214907.json

This file was deleted.

467 changes: 0 additions & 467 deletions agents/error_captures/520_error_20260126_143304.json

This file was deleted.

490 changes: 0 additions & 490 deletions agents/error_captures/520_error_20260126_143447.json

This file was deleted.

772 changes: 0 additions & 772 deletions agents/error_captures/520_error_20260128_000916.json

This file was deleted.

612 changes: 0 additions & 612 deletions agents/error_captures/520_error_20260128_004242.json

This file was deleted.

725 changes: 0 additions & 725 deletions agents/error_captures/520_error_20260128_052223.json

This file was deleted.

633 changes: 0 additions & 633 deletions agents/error_captures/520_error_20260128_065514.json

This file was deleted.

816 changes: 0 additions & 816 deletions agents/error_captures/520_error_20260130_175253.json

This file was deleted.

1,142 changes: 0 additions & 1,142 deletions agents/error_captures/520_error_20260130_180655.json

This file was deleted.

417 changes: 0 additions & 417 deletions agents/error_captures/520_error_20260201_055650.json

This file was deleted.

387 changes: 0 additions & 387 deletions agents/error_captures/520_error_20260201_094305.json

This file was deleted.

641 changes: 0 additions & 641 deletions agents/error_captures/520_error_20260201_103402.json

This file was deleted.

683 changes: 0 additions & 683 deletions agents/error_captures/520_error_20260201_221754.json

This file was deleted.

933 changes: 0 additions & 933 deletions agents/error_captures/520_error_20260202_000336.json

This file was deleted.

851 changes: 0 additions & 851 deletions agents/error_captures/520_error_20260202_031242.json

This file was deleted.

833 changes: 0 additions & 833 deletions agents/error_captures/520_error_20260202_090304.json

This file was deleted.

593 changes: 0 additions & 593 deletions agents/error_captures/520_error_20260202_113613.json

This file was deleted.

795 changes: 0 additions & 795 deletions agents/error_captures/520_error_20260202_140227.json

This file was deleted.

903 changes: 0 additions & 903 deletions agents/error_captures/520_error_20260202_205929.json

This file was deleted.

961 changes: 0 additions & 961 deletions agents/error_captures/520_error_20260202_210614.json

This file was deleted.

373 changes: 0 additions & 373 deletions agents/error_captures/520_error_20260202_213757.json

This file was deleted.

461 changes: 0 additions & 461 deletions agents/error_captures/520_error_20260202_214928.json

This file was deleted.

395 changes: 0 additions & 395 deletions agents/error_captures/520_error_20260202_214929.json

This file was deleted.

439 changes: 0 additions & 439 deletions agents/error_captures/520_error_20260202_215003.json

This file was deleted.

505 changes: 0 additions & 505 deletions agents/error_captures/520_error_20260202_215405.json

This file was deleted.

483 changes: 0 additions & 483 deletions agents/error_captures/520_error_20260202_215540.json

This file was deleted.

527 changes: 0 additions & 527 deletions agents/error_captures/520_error_20260202_215542.json

This file was deleted.

909 changes: 0 additions & 909 deletions agents/error_captures/520_error_20260202_230059.json

This file was deleted.

827 changes: 0 additions & 827 deletions agents/error_captures/520_error_20260202_230310.json

This file was deleted.

963 changes: 0 additions & 963 deletions agents/error_captures/520_error_20260202_231123.json

This file was deleted.

885 changes: 0 additions & 885 deletions agents/error_captures/520_error_20260202_231137.json

This file was deleted.

1,245 changes: 0 additions & 1,245 deletions agents/error_captures/520_error_20260203_001511.json

This file was deleted.

281 changes: 0 additions & 281 deletions agents/error_captures/520_error_20260203_002407.json

This file was deleted.

335 changes: 0 additions & 335 deletions agents/error_captures/520_error_20260203_003324.json

This file was deleted.

497 changes: 0 additions & 497 deletions agents/error_captures/520_error_20260203_010108.json

This file was deleted.

555 changes: 0 additions & 555 deletions agents/error_captures/520_error_20260203_010829.json

This file was deleted.

483 changes: 0 additions & 483 deletions agents/error_captures/520_error_20260203_010834.json

This file was deleted.

555 changes: 0 additions & 555 deletions agents/error_captures/520_error_20260203_010930.json

This file was deleted.

555 changes: 0 additions & 555 deletions agents/error_captures/520_error_20260203_011040.json

This file was deleted.

241 changes: 0 additions & 241 deletions agents/error_captures/520_error_20260203_011843.json

This file was deleted.

263 changes: 0 additions & 263 deletions agents/error_captures/520_error_20260203_012103.json

This file was deleted.

317 changes: 0 additions & 317 deletions agents/error_captures/520_error_20260203_013434.json

This file was deleted.

329 changes: 0 additions & 329 deletions agents/error_captures/520_error_20260203_013801.json

This file was deleted.

335 changes: 0 additions & 335 deletions agents/error_captures/520_error_20260203_013820.json

This file was deleted.

479 changes: 0 additions & 479 deletions agents/error_captures/520_error_20260203_020044.json

This file was deleted.

569 changes: 0 additions & 569 deletions agents/error_captures/520_error_20260203_021857.json

This file was deleted.

749 changes: 0 additions & 749 deletions agents/error_captures/520_error_20260203_024932.json

This file was deleted.

749 changes: 0 additions & 749 deletions agents/error_captures/520_error_20260203_025107.json

This file was deleted.

1,073 changes: 0 additions & 1,073 deletions agents/error_captures/520_error_20260203_033528.json

This file was deleted.

715 changes: 0 additions & 715 deletions agents/error_captures/520_error_20260203_042423.json

This file was deleted.

263 changes: 0 additions & 263 deletions agents/error_captures/520_error_20260203_155652.json

This file was deleted.

417 changes: 0 additions & 417 deletions agents/error_captures/520_error_20260203_180741.json

This file was deleted.

1,081 changes: 0 additions & 1,081 deletions agents/error_captures/520_error_20260203_201016.json

This file was deleted.

19 changes: 19 additions & 0 deletions analysis/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Research analysis

- `scripts/`: CCE analysis and task-quality review tooling.
- `results/`: Existing research analysis results.
- `archive/`: Historical task reviews, investigations, and category notes moved
out of the repository root.

For current model scoring, start with [Scoring and results](../docs/benchmark/scoring.md).
`analysis/scripts/evaluate_tasks.py` reviews task design using the Claude CLI;
it does **not** run an agent benchmark or calculate model scores.

```bash
python analysis/scripts/evaluate_tasks.py --help
```

Its default repository is inferred from the script path. New reports go under
ignored `runs/task-reviews/benchmark/` or `runs/task-reviews/augmented/`.
`--base-dir` and `--results-dir` override those locations. Running an evaluation
requires a configured Claude CLI and may incur model usage.
27 changes: 27 additions & 0 deletions analysis/archive/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Historical analysis archive

These are retained research notes and task-quality evaluations, not current
leaderboard results. Existing report contents are preserved, including descriptions
of older runs and their original filesystem paths.

| Location | Previous root location | Contents |
| --- | --- | --- |
| `task-reviews/main/` | `evaluate_results/` | Main-task evaluation reports and summaries |
| `task-reviews/augmented/` | `evaluate_results_augmented/` | Augmented-task evaluation reports and summaries |
| `investigations/` | `investigate_results/` | Task 57–59 investigation notes |
| `task_categories.txt` | `task_categories.txt` | Historical 106-task categorization; not the current suite manifest |

New task-quality evaluations should write to `runs/task-reviews/` using
`analysis/scripts/evaluate_tasks.py`. Current model scoring instructions are in
[Scoring and results](../../docs/benchmark/scoring.md).

Generated root HTML visualizations and `agents/error_captures/` response dumps
were removed from the tracked tree. Their old versions remain in Git history.
To recover a specific visualization without putting it back in the root:

```bash
mkdir -p runs/visualizations
git show e1935203:viz_task_05.html > runs/visualizations/viz_task_05.html
```

For new reports, use the [trajectory visualization guide](../../docs/benchmark/visualization.md).
Loading
Loading