When Do Multi-Agent Systems Help? An Information Bottleneck Perspective

Paper (arXiv)  |  Project Page

Wendi Yu*, Lianhao Zhou*, Xiangjue Dong, Sai Sudarshan Barath, Declan Staunton, Byung-Jun Yoon, Xiaoning Qian, James Caverlee, Shuiwang Ji (* equal contribution), Texas A&M University

We study when LLM-based multi-agent systems (MAS) outperform single-agent systems (SAS) through an information-bottleneck lens: a SAS accumulates its full reasoning trace in one shared context, while a MAS connects isolated worker contexts through bounded relay messages. MAS gains arise from a trade-off between upstream context reduction and relay information loss, and this trade-off shifts with the downstream model's capability. We validate this with 18 controlled experiments across five agentic benchmarks (ALFWorld, WebShop, WorkBench, WideSearch, TravelPlanner) and three model scales (Qwen2.5-7B-Instruct, GPT-4o-mini, Qwen3.5-27B), using four controlled prototypes: SAS, SAS-contextflow, MAS, and SAS-Plan.

Installation

Requires Python >= 3.10.

git clone https://github.com/divelab/MAS-SAS.git
cd MAS-SAS
pip install -r requirements.txt

Set OPENAI_API_KEY for GPT-4o-mini experiments. Qwen2.5-7B-Instruct and Qwen3.5-27B-AWQ-4bit are run against a local OpenAI-compatible endpoint (e.g. vLLM); point llm.base_url at your server (see configs/README.md for a worked example). WideSearch's web search uses DuckDuckGo (ddgs) by default, so no API key is needed.

Data Preparation

All configs reference benchmark data under a single environment variable:

export MAS_SAS_DATA_DIR=/path/to/your/data

Each benchmark's raw data must be obtained from its official source (see the paper's references) and placed under $MAS_SAS_DATA_DIR in the layout below. WideSearch's task/decomposition jsonl is already checked into this repo (data/widesearch/), since it is the exact 66-task subset and GPT-5-decomposed subqueries used in the paper.

ALFWorld

$MAS_SAS_DATA_DIR/alfworld/
  json_2.1.1/{train,valid_seen,valid_unseen}/...
  logic/alfred.pddl
  logic/alfred.twl2

Install the alfworld PyPI package (already in requirements.txt) and fetch the data with the package's own alfworld-download command, then point MAS_SAS_DATA_DIR at the resulting alfworld/ directory. We evaluate on the valid_unseen split (134 tasks); the task list used is data/alfworld/alfworld_tasks_suffix.json (already in this repo).

WebShop

WebShop is not a static file directory. tasks/webshop/env.py talks to a running instance of the official WebShop Flask simulator over HTTP. Stand up the simulator yourself (product data + server, per the original WebShop release) and point each config's task.webshop_url at it (default expects http://localhost:3000). We evaluate on test_100 (100 sampled tasks).

WorkBench

$MAS_SAS_DATA_DIR/workbench/data/processed/
  calendar_events.csv
  emails.csv
  analytics_data.csv
  project_tasks.csv
  customer_relationship_manager_data.csv

We evaluate on the 210-task multi-domain split.

WideSearch

$MAS_SAS_DATA_DIR/widesearch/widesearch_gold/{instance_id}.csv   # gold answer tables

The task/query jsonl is already checked into this repo; see data/widesearch/. You only need the gold-answer CSVs from the official WideSearch release.

TravelPlanner

$MAS_SAS_DATA_DIR/travelplanner/database/
  flights/clean_Flights_2022.csv
  accommodations/clean_accommodations_2022.csv
  restaurants/clean_restaurant_2022.csv
  attractions/attractions.csv
  background/citySet.txt
  background/citySet_with_states.txt
  background/stateSet.txt
  googleDistanceMatrix/distance.csv

Task queries are loaded automatically from the osunlp/TravelPlanner HuggingFace dataset (validation split, 180 tasks), so no manual download is needed for the queries themselves, only the tool database above.

Quick Start

# Run a single experiment. Writes to <output_dir>/<experiment_id>/,
# both set inside the YAML (here: outputs/qwen2.5-7b/alfworld/sas_react_alfworld_qwen/)
python main.py run --config configs/qwen2.5-7b/alfworld/sas.yaml

# Override the task count / experiment id without editing the YAML
python main.py run --config configs/qwen2.5-7b/alfworld/sas.yaml \
    --max-tasks 5 --experiment-id smoke_test

Configs are organized as configs/<model>/<benchmark>/<prototype>.yaml (qwen2.5-7b, 4omini, qwen27b × sas, sas_contextflow, mas, plus ablation variants). See configs/README.md for details.

Each run writes one JSONL trace file per episode plus a summary.yaml with aggregate metrics, and resumes automatically if interrupted.

Citation

@article{yu2026masvssas,
  title   = {When Do Multi-Agent Systems Help? An Information Bottleneck Perspective},
  author  = {Yu, Wendi and Zhou, Lianhao and Dong, Xiangjue and Barath, Sai Sudarshan
             and Staunton, Declan and Yoon, Byung-Jun and Qian, Xiaoning
             and Caverlee, James and Ji, Shuiwang},
  year    = {2026},
}

This work builds on five existing agentic benchmarks. Please also cite ALFWorld, WebShop, WorkBench, WideSearch, and TravelPlanner if you use the corresponding parts of this repository (see the reference list in our paper).

License

MIT. See LICENSE.