Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .github/workflows/benchmark-onboarding.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: Benchmark onboarding
on:
pull_request:
paths:
- 'agents/**'
- 'benchmarks/**'
- 'examples/agents/**'
- 'tests/benchmark/**'
- 'data_v0.1_multi/v1.3*/**'
- 'analysis/scripts/compute_cce_v2.py'
- 'analysis/requirements-cce.txt'
- '.github/workflows/benchmark-onboarding.yml'
workflow_dispatch:
permissions:
contents: read
jobs:
offline-checks:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: python -m pip install -r agents/requirements.txt -r analysis/requirements-cce.txt
- run: python -m unittest discover -s tests/benchmark -v
- run: python agents/run.py --help
- run: python benchmarks/score.py --help
- run: python analysis/scripts/compute_cce_v2.py --help
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -466,3 +466,10 @@ benchmark_results
**/results_*/
scripts/run_parallel_45_60.sh
scripts/run_tasks_45_60.sh

# Local benchmark configuration and generated artifacts
.env
.env.local
.venv/
/runs/
/agents/configs/*.local.yaml
27 changes: 27 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Contributing to AgentWorld

Start with the [repository map](docs/repository-map.md) and
[benchmark quickstart](docs/benchmark/quickstart.md).

- **Model or harness integrations:** supply a minimal adapter/configuration,
a smoke-task trajectory, and the exact reproduction command.
- **Task/verifier fixes:** include a failing example and a regression check.
Describe which scores or historical runs may be affected; do not silently
change benchmark criteria as part of a formatting cleanup.
- **Documentation:** use root-relative commands and environment-based examples.
Separate portable workflows from machine-specific experiment launchers.
- **Result reviews:** use the [GitHub issue submission workflow](docs/benchmark/scoring.md).
Check configuration, coverage, protocol, and raw evidence before publication.

Run the offline onboarding checks without game credentials or model calls:

```bash
python -m pip install -r agents/requirements.txt
python -m unittest discover -s tests/benchmark -v
python agents/run.py --help
python benchmarks/score.py --help
```

Live game/model runs are separate validation. Include the task/harness revision,
settings, and coverage with those results. Keep credentials, virtual environments,
and generated run artifacts out of commits.
Loading
Loading