Standardize ExploitBench task configuration and harnesses - #1

Draft

pabloRom2004 wants to merge 105 commits into

Generality-Labs:mainfrom

pabloRom2004:main

Commits on Sep 7, 2026

Streamline ExploitBench around the original image tools
pabloRom2004
committed
c952ce8
Add minimal ReAct default with token-budget reminders
pabloRom2004
committed
437dc72
Format the supplied README using the HLE section structure
pabloRom2004
committed
ed753bb
Add native ReAct shell tools
pabloRom2004
committed
90d6561
Standardize the eval entrypoint as task.py
pabloRom2004
committed
c440673
Align default and original configuration keys
pabloRom2004
committed
1baa60b
Rename agents to default and original
pabloRom2004
committed
71711dc
Update task.py
pabloRom2004
committed
8757f43
Preserve capability flags across epochs and enable native ReAct submission
pabloRom2004
committed
4ac0749
Configure harnesses through HLE-style native solver sections
pabloRom2004
committed
a0f7538
Separate YAML configuration arguments with blank lines
pabloRom2004
committed
08e3f5d
Keep YAML entries compact within top-level blocks
pabloRom2004
committed
2432221
Simplify run configs and report average and maximum flags
pabloRom2004
committed
4352f59
Use xhigh reasoning in both run configurations
pabloRom2004
committed
d78d604
Rename agent modules to agent_default and agent_original
pabloRom2004
committed
f0350bf
Document selecting individual challenge images in run configs
pabloRom2004
committed
a2ae4ed
Remove unused message limit from run configurations
pabloRom2004
committed
6fa117c
Updated wording
pabloRom2004
committed
603783a
Use native tool output limits and provider-default temperature
pabloRom2004
committed
7e9732b
Leave sandbox concurrency to the run environment
pabloRom2004
committed
9efeca2
Use only the token budget for default attempt limits
pabloRom2004
committed
c2f1af5
Keep the default time limit visible as null
pabloRom2004
committed
2917169
Score recorded progress after agent errors and continue evaluations
pabloRom2004
committed
b966689
Verify ReAct stopping and ungraded-attempt accounting
pabloRom2004
committed
c1d1bbe

Commits on Sep 8, 2026

Remind ReAct agents to grade alongside token usage
pabloRom2004
committed
9dfaae6
Document running ExploitBench with Docker on Modal VMs
pabloRom2004
committed
fe45ead
Make ReAct token and grading reminders configurable
pabloRom2004
committed
d010910
Rename ReAct grade reminder option
pabloRom2004
committed
57373bd
Align ExploitBench with HLE structure
pabloRom2004
committed
76e50b7
Rename ExploitBench tasks and add original harness nudging
pabloRom2004
committed
09023c3
Add CLI harnesses and unify ExploitBench task configuration
pabloRom2004
committed
1d00ce4
Consolidate YAML configuration loading
pabloRom2004
committed
0b809a3
Raise default token budgets to 100 million
pabloRom2004
committed
8dac84c
Continue attempts with configurable nudges and stop on ACE
pabloRom2004
committed
a6a8d3f
Remove nudge count cap absent from original implementation
pabloRom2004
committed
5babdb3
Disable internet access across all CLI harnesses
pabloRom2004
committed
7b9d34d
Automatically clean up evaluation logs created by pytest
pabloRom2004
committed
aa7913b
Align README with HLE and consolidate release version
pabloRom2004
committed
447453c
Update README.md
pabloRom2004
committed
3a248d1
Update README.md
pabloRom2004
committed
c991316
Update README.md
pabloRom2004
committed
86e1af5
Update README.md
pabloRom2004
committed
40050dc
Update README.md
pabloRom2004
committed
6c573ed
Small formatting changes
pabloRom2004
committed
e843193

Commits on Sep 9, 2026

Commits on Sep 10, 2026

Use grayscale scoring diagram in README
pabloRom2004
committed
bb5d149
Simplify scoring diagram borders
pabloRom2004
committed
8ab394d
Store intermediate grading evidence in TaskState
pabloRom2004
committed
0509298
Updated docs
pabloRom2004
committed
5d6d1b1
Refactored default agent and CLI agent
pabloRom2004
committed
bc5b865
Fix OpenCode context output config
pabloRom2004
committed
eec2794

Commits on Sep 11, 2026

Commits on Sep 12, 2026

Expose bundled npm for Gemini CLI
pabloRom2004
committed
3bd5d44
Use nodejs-wheel console npm for Gemini
pabloRom2004
committed
095c951
Handle OpenCode stream exit
pabloRom2004
committed
06002f0
Add native tool timeout for ReAct
pabloRom2004
committed
1eff69e
Increase grader MCP timeout
pabloRom2004
committed
9247a3d
Make OpenCode compact before provider overflow
pabloRom2004
committed
0aeaad5
Document explicit OpenCode context for long runs
pabloRom2004
committed
f99405e
Request streamed usage for OpenCode compaction
pabloRom2004
committed
333ca06

Commits on Sep 14, 2026

Commits on Sep 15, 2026

Commits on Sep 16, 2026

Reserve sandbox storage and separate ExploitBench workers
pabloRom2004
committed
0d31905
Keep evaluation failures out of capability scores
pabloRom2004
committed
3165e32
Keep score-on-error enabled by default
pabloRom2004
committed
82554ec
Allow two hours for OpenCode bridge RPCs
pabloRom2004
committed
b93253f
Use smaller Hawk sandboxes and regional ECR image caching
pabloRom2004
committed
6467d9e
Allow explicit Kubernetes node selection for sandbox placement
pabloRom2004
committed
f5862fd
Align main with tested live-run settings and timeout fixes
pabloRom2004
committed
6da7c1d
Configure native CLI sandbox polling timeouts
pabloRom2004
committed
fb86bac

Commits on Sep 17, 2026

Align OpenCode provider timeouts with Inspect
pabloRom2004
committed
096bff3
Handle MCP broken-stream teardown race
pabloRom2004
committed
5aae52b
Add recoverable CLI evaluation state
pabloRom2004
committed
f618141
Accept compact recovery bundles
pabloRom2004
committed
4903d39
Fix recovery transcript messages without tools
pabloRom2004
committed
d7280d8
Restore recovered home ownership
pabloRom2004
committed
01fbc06
Support encrypted remote recovery bundles
pabloRom2004
committed
4245991
Remove cancelled recovery run bundle
pabloRom2004
committed
583bded
Add configurable task failure handling and ReAct checkpoint recovery probe
pabloRom2004
committed
0bd92c9
Use scripted task default without requesting a Middleman model
pabloRom2004
committed
7f0c591
Merge branch 'main' into codex/react-checkpoint-proof
pabloRom2004
committed
e26c72a
Preserve checkpoints across all harnesses and scorer retries
pabloRom2004
committed
fcf0226
Prefetch verified CLI archives to avoid release API rate limits
pabloRom2004
committed
586765f
Keep early setup errors visible in scorer recovery probes
pabloRom2004
committed
b411929
Document twenty-minute checkpoints and retention limits
pabloRom2004
committed
30bfd3a
Recommend Hawk default ten-minute checkpoints
pabloRom2004
committed
081916f

Commits on Sep 18, 2026

Test upstream Inspect log compression fix on Hawk
pabloRom2004
committed
bb73307
Document verified upstream recovery fix and independent retries
pabloRom2004
committed
6774eda
Accept the verified canonical DeepSeek Flash response alias
pabloRom2004
committed
2304a7e
Keep native CLI checkpoints open across voluntary continuations
pabloRom2004
committed
3253c98
Fail checkpointed attempts when the sandbox grader connection dies
pabloRom2004
committed
7612d69
Verify native MCP recovery after the saved tool request
pabloRom2004
committed
131d810