Skip to content

fix(agent): bound exact trajectory materialization #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rushilbhat merged 1 commit into from
Aug 28, 2026
Merged

fix(agent): bound exact trajectory materialization #31

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 16 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,10 +167,12 @@ subtracts these values when generating content while retaining the manifest's
absolute prompt targets for reporting.

`--max-active-agents` bounds simultaneous trajectories while retaining every plan
in the manifest. BatchBench initially admits up to that limit and immediately
admits the next queued trajectory whenever any active trajectory terminates. The
replacement inherits the freed scheduler/routing slot, keeping active load balanced
across data-parallel ranks when perfect routing is enabled.
in the manifest. BatchBench initially admits up to that limit. Whenever an active
trajectory terminates, the next queued trajectory prepares its initial prompt and
then inherits the freed scheduler/routing slot. Multiple replacements prepare
concurrently through the same bounded worker pool while preserving FIFO admission
order, keeping active load balanced across data-parallel ranks when perfect routing
is enabled.

```bash
batchbench-agent \
Expand All @@ -187,7 +189,16 @@ batchbench-agent \

Do not combine `--agent-plans-jsonl` with `--agents` or the synthetic workload-shape
flags. `--seed` remains useful because it makes generated placeholder text
reproducible.
reproducible. Generated content is accepted only when it re-encodes to the requested
token count; an unconstructable field fails with its trajectory context instead of
silently changing the workload.

Replay manifests remain compact in memory: BatchBench materializes the initially
active prompts before the benchmark clock starts, then generates only the current
environment response and the next reset prompt while the current HTTP request is in
flight. Token generation uses a bounded worker pool, and its per-agent/per-turn random
streams are independent of asynchronous scheduling. Request JSON is serialized from
the live conversation without first cloning the complete message tree.

`--agent-events-jsonl` writes each trajectory's queue/admission time, reusable
routing slot, optional DP rank, finish time, runtime, and completion status. The
Expand Down
1 change: 1 addition & 0 deletions rust/Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion rust/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,9 @@ anyhow = "1"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
sha2 = "0.10"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "time", "fs", "signal"] }
tokio = { version = "1", features = ["macros", "rt-multi-thread", "time", "fs", "signal", "sync"] }
reqwest = { version = "0.11", default-features = false, features = ["json", "rustls-tls"] }
bytes = "1"
thiserror = "1"
clap = { version = "4", features = ["derive"] }
rand = "0.8"
Expand Down
Loading
Loading