Skip to content

Standardize ExploitBench task configuration and harnesses - #1

Draft
pabloRom2004 wants to merge 105 commits into
Generality-Labs:mainfrom
pabloRom2004:main
Draft

Standardize ExploitBench task configuration and harnesses#1
pabloRom2004 wants to merge 105 commits into
Generality-Labs:mainfrom
pabloRom2004:main

Conversation

@pabloRom2004

@pabloRom2004 pabloRom2004 commented Sep 8, 2026 •
edited
Loading

Copy link
Copy Markdown

Changes Made:

  • Cleaned up the code to be super minimal following the structure from the hle task
  • Added the ReAct harness with a default of 100M tokens and compaction on, so we can see the real cyber capabilities of models now without them being held back by the old config
  • Added a proper recreation of the original papers setup with the same features and prompts (300 turns max, 5h clock time, 65k max output tokens, nudge prompt)
  • Added support for Claude Code, Codex, Gemini CLI, Kimi Code and OpenCode through Inspect SWE (I have not tested this super in depth for every single harness and there are likely issues with some of the harnesses on super long runs, but if people want to try them they are supported, they may just need to debug any issues themselves)

@pabloRom2004
Streamline ExploitBench around the original image tools
c952ce8 
Keep the task, original loop, and capability scorer thin; reduce run configuration to default and original YAML files. Add function summaries and focused mock tests, update the README, and remove unused reporting and harness variants.
@pabloRom2004
Add minimal ReAct default with token-budget reminders
437dc72 
Use native Inspect ReAct with a 5M-token budget and 75% summary compaction. Report cumulative token usage, retain original-agent behavior, and raise only the maintained response cap to 131072 tokens without selecting a model. Validate with 25 local tests and two mockllm runs through all six tools in the cached Modal image.
@pabloRom2004
Format the supplied README using the HLE section structure
ed753bb 
Retain the supplied introduction and usage prose, organize it under the HLE README outline, and update commands, configuration defaults, image details, and scoring for the current implementation. Validate Markdown structure, example syntax, local links, and documented parameters.
@pabloRom2004
Add native ReAct shell tools
90d6561
@pabloRom2004
Standardize the eval entrypoint as task.py
c440673
@pabloRom2004
Align default and original configuration keys
1baa60b
@pabloRom2004
Rename agents to default and original
71711dc
@pabloRom2004
Update task.py
8757f43
@pabloRom2004
Preserve capability flags across epochs and enable native ReAct submi…
4ac0749 
…ssion
@pabloRom2004
Configure harnesses through HLE-style native solver sections
a0f7538
@pabloRom2004
Separate YAML configuration arguments with blank lines
08e3f5d
@pabloRom2004
Keep YAML entries compact within top-level blocks
2432221
@pabloRom2004
Simplify run configs and report average and maximum flags
4352f59
@pabloRom2004
Use xhigh reasoning in both run configurations
d78d604
@pabloRom2004
Rename agent modules to agent_default and agent_original
f0350bf
@pabloRom2004
Document selecting individual challenge images in run configs
a2ae4ed
@pabloRom2004
Remove unused message limit from run configurations
6fa117c
@pabloRom2004
Updated wording
603783a
@pabloRom2004
Use native tool output limits and provider-default temperature
7e9732b
@pabloRom2004
Leave sandbox concurrency to the run environment
9efeca2
@pabloRom2004
Use only the token budget for default attempt limits
c2f1af5
@pabloRom2004
Keep the default time limit visible as null
2917169
@pabloRom2004
Score recorded progress after agent errors and continue evaluations
b966689
@pabloRom2004
Verify ReAct stopping and ungraded-attempt accounting
c1d1bbe
@pabloRom2004
Remind ReAct agents to grade alongside token usage
9dfaae6
@pabloRom2004
Document running ExploitBench with Docker on Modal VMs
fe45ead
@pabloRom2004
Make ReAct token and grading reminders configurable
d010910
@pabloRom2004
Rename ReAct grade reminder option
57373bd
@pabloRom2004
Align ExploitBench with HLE structure
76e50b7
@pabloRom2004
Rename ExploitBench tasks and add original harness nudging
09023c3
@pabloRom2004
Use smaller Hawk sandboxes and regional ECR image caching
6467d9e
@pabloRom2004
Allow explicit Kubernetes node selection for sandbox placement
f5862fd
@pabloRom2004
Align main with tested live-run settings and timeout fixes
6da7c1d
@pabloRom2004
Configure native CLI sandbox polling timeouts
fb86bac
@pabloRom2004
Align OpenCode provider timeouts with Inspect
096bff3
@pabloRom2004
Handle MCP broken-stream teardown race
5aae52b
@pabloRom2004
Add recoverable CLI evaluation state
f618141
@pabloRom2004
Accept compact recovery bundles
4903d39
@pabloRom2004
Fix recovery transcript messages without tools
d7280d8
@pabloRom2004
Restore recovered home ownership
01fbc06
@pabloRom2004
Support encrypted remote recovery bundles
4245991
@pabloRom2004
Remove cancelled recovery run bundle
583bded
@pabloRom2004
Add configurable task failure handling and ReAct checkpoint recovery …
0bd92c9 
…probe
@pabloRom2004
Use scripted task default without requesting a Middleman model
7f0c591
@pabloRom2004
Merge branch 'main' into codex/react-checkpoint-proof
e26c72a
@pabloRom2004
Preserve checkpoints across all harnesses and scorer retries
fcf0226
@pabloRom2004
Prefetch verified CLI archives to avoid release API rate limits
586765f
@pabloRom2004
Keep early setup errors visible in scorer recovery probes
b411929
@pabloRom2004
Document twenty-minute checkpoints and retention limits
30bfd3a
@pabloRom2004
Recommend Hawk default ten-minute checkpoints
081916f
@pabloRom2004
Test upstream Inspect log compression fix on Hawk
bb73307
@pabloRom2004
Document verified upstream recovery fix and independent retries
6774eda
@pabloRom2004
Accept the verified canonical DeepSeek Flash response alias
2304a7e
@pabloRom2004
Keep native CLI checkpoints open across voluntary continuations
3253c98
@MattFisher

MattFisher commented Sep 18, 2026

Copy link
Copy Markdown

FYI this is the relevant PR if you haven't seen it: pabloRom2004#1

@pabloRom2004
pabloRom2004 marked this pull request as draft September 18, 2026 23:47
@pabloRom2004

pabloRom2004 commented Sep 18, 2026

Copy link
Copy Markdown
Author

I'm just trying to figure out the harness recovery and such, I will open this PR again once this is all done

@pabloRom2004

pabloRom2004 commented Sep 18, 2026

Copy link
Copy Markdown
Author

pabloRom2004#1

I saw this PR but it's in draft? Is it good to merge in I am happy to merge it I took a look at it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@pabloRom2004 @JustinOlive @MattFisher @generality-labs-bot