DARTS

Official implementation of DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning.

DARTS accelerates LLM reinforcement learning by actively shaping the rollout trajectory distribution. Instead of only scheduling around long-tail generations, DARTS samples from an intra-prompt redundant rollout pool, favors concise trajectories when verbosity is ineffective, preserves necessary long reasoning paths, and overlaps rollout with training through token-level streaming.

Overview

DARTS contains four main components:

• Distribution-aware trajectory sampling, which selects training trajectories from a redundant candidate pool via dual-end length sampling.
• Adaptive redundancy allocation, which assigns larger exploration budgets to prompts with higher historical length variance.
• Tail pruning with early stopping, which avoids wasting rollout compute on extreme long-tail candidates.
• Token-level streaming, which overlaps rollout generation and training forward computation.

In the paper experiments, DARTS achieves up to 1.77x end-to-end speedup over VeRL while maintaining model quality.

Code Structure

.
+-- README.md
`-- darts/
    |-- env.yml
    |-- requirements.txt
    |-- recipe/darts/
    |   |-- README.md
    |   |-- main_ppo.py
    |   |-- run_darts_qwen2_5_math_7b_deepmath.sh
    |   |-- run_darts_qwen2_5_14b_dapo_math.sh
    |   |-- run_rollout_qwen3_30b_a3b_dapo_math.sh
    |   |-- run_verl_qwen2_5_math_7b_deepscaler.sh
    |   `-- config/
    |       |-- darts_trainer.yaml
    |       `-- darts_megatron_trainer.yaml
    `-- verl/
        |-- workers/actor/
        `-- workers/rollout/vllm_rollout/

Core DARTS logic is implemented under darts/recipe/darts/, with VeRL worker and rollout extensions under darts/verl/.

Installation

The paper experiments use VeRL v0.4, vLLM 0.8.5.post1, PyTorch 2.6.0, Ray, FSDP, and NVIDIA H20 GPUs.

cd darts
conda env create -f env.yml
conda activate csw-verl
pip install -e .[vllm,math,gpu]

If you prefer a lighter setup, install from requirements.txt and ensure compatible versions of Python >=3.10, PyTorch 2.6.0, vLLM <=0.8.5, Ray >=2.41.0, flash-attn, and CUDA runtime libraries.

Data and Models

Training scripts expect local Hugging Face model directories and parquet datasets. The math recipes follow VeRL/DAPO data conventions and use prompt as the input field:

MODEL_PATH=/model/Qwen2.5-Math-7B
TRAIN_FILE=/data/dapo-math-17k.parquet
TEST_FILE=/data/aime-2024.parquet
RAY_DATA_HOME=/darts

For long-context Qwen math runs, the scripts set:

+actor_rollout_ref.model.override_config.max_position_embeddings=32768

Make sure the downloaded model config supports this context length, or reduce data.max_prompt_length + data.max_response_length.

Reproduction Recipes

Run commands from darts/. Public recipe scripts follow:

run_<method>_<model>_<dataset>.sh

Script	Method	Model	Dataset / eval
recipe/darts/run_darts_qwen2_5_math_7b_deepmath.sh	DARTS	Qwen2.5-Math-7B	DeepMath / AIME2024
recipe/darts/run_darts_qwen2_5_14b_dapo_math.sh	DARTS	Qwen2.5-14B	DAPO-MATH / AIME2024
recipe/darts/run_rollout_qwen3_30b_a3b_dapo_math.sh	rollout variant	Qwen3-30B-A3B	DAPO-MATH / AIME2024
recipe/darts/run_verl_qwen2_5_math_7b_deepscaler.sh	VeRL baseline	Qwen2.5-Math-7B	DeepScaler / AIME2024
recipe/darts/run_verl_qwen2_5_7b_gsm8k.sh	VeRL baseline	Qwen2.5-7B-Instruct	GSM8K

Example:

cd darts

MODEL_PATH=/model/Qwen2.5-Math-7B \
TRAIN_FILE=/data/deepmath/train.parquet \
TEST_FILE=/data/aime-2024.parquet \
RAY_DATA_HOME=/darts \
NNODES=4 \
NGPUS_PER_NODE=8 \
bash recipe/darts/run_darts_qwen2_5_math_7b_deepmath.sh

The DARTS scripts use disaggregated resources by default. For example, on each 8-GPU node, 4 GPUs are used for rollout and 4 GPUs for training.

Key Hyperparameters

The shell recipes override YAML defaults and should be treated as the source of truth for reproduction.

Parameter	Paper-style value	Description
algorithm.adv_estimator	grpo	RL objective used in the main experiments.
actor_rollout_ref.rollout.n	8	GRPO group size M.
algorithm.redundancy	8	Redundant rollout budget control.
algorithm.trainer_name	stream	DARTS trainer. Use basic for VeRL-style baseline and roll for rollout variant.
algorithm.yita	1.0 or 1.25	Allocation / sampling strength used by trainer variants.
data.train_batch_size	128 or 256	Prompt batch size N.
data.max_prompt_length	2048	Maximum prompt length.
data.max_response_length	8192 or 16384	Maximum rollout length.
actor_rollout_ref.rollout.temperature	1.0	Rollout sampling temperature.
actor_rollout_ref.rollout.top_p	1.0	Rollout nucleus sampling.
actor_rollout_ref.rollout.top_k	-1	vLLM rollout top-k setting.
actor_rollout_ref.rollout.tensor_model_parallel_size	4	vLLM tensor parallel size.
actor_rollout_ref.rollout.gpu_memory_utilization	0.80	vLLM memory utilization.
actor_rollout_ref.actor.optim.lr	1e-6	Actor learning rate.
actor_rollout_ref.actor.clip_ratio_low/high	0.2 / 0.28	DAPO-style clip-higher setting.
reward_model.reward_manager	dapo	Rule-based math reward manager.
reward_model.overlong_buffer.enable	True	DAPO-style overlong reward shaping.

In the paper, dual-end sampling selects K shortest and L longest trajectories with K + L = M; main experiments use L = 1. Adaptive allocation bounds are Mlow = M and Mup = 2M.

Hardware

The paper uses an 8-node H20 cluster. Each node has 8 NVIDIA H20 96GB GPUs with NVLink, connected by 1.6Tbps InfiniBand.

Model	GPUs
Qwen2.5-3B	8 H20
Qwen2.5-Math-7B	32 H20
Qwen2.5-14B	32 H20
Qwen2.5-32B	64 H20
Qwen3-30B-A3B	64 H20

For smaller clusters, reduce data.train_batch_size, actor_rollout_ref.actor.ppo_mini_batch_size, actor_rollout_ref.rollout.tensor_model_parallel_size, or data.max_response_length.

Citation

@inproceedings{darts2026,
  title = {DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning},
  author = {Wang, Yujie and Chen, Siwei and Luo, Longzan and Liu, Xinyi and Miao, Xupeng and Fu, Fangcheng and Cui, Bin},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
  year = {2026}
}