Skip to content
BranchesTags
 
 

Latest commit

 

History

2,135 Commits

Folders and files

NameName
Last commit message
Last commit date
.gemini
.gemini
 
 
.github
.github
 
 
.vscode
.vscode
 
 
deployment
deployment
 
 
docker
docker
 
 
docs
docs
 
 
examples
examples
 
 
recipe @ 3490a22
recipe @ 3490a22
 
 
scripts
scripts
 
 
tests
tests
 
 
verl
verl
 
 
.git-blame-ignore-revs
.git-blame-ignore-revs
 
 
.gitignore
.gitignore
 
 
.gitmodules
.gitmodules
 
 
.pre-commit-config.yaml
.pre-commit-config.yaml
 
 
.readthedocs.yaml
.readthedocs.yaml
 
 
CONTRIBUTING.md
CONTRIBUTING.md
 
 
LICENSE
LICENSE
 
 
Notice.txt
Notice.txt
 
 
README.md
README.md
 
 
launch.sh
launch.sh
 
 
pyproject.toml
pyproject.toml
 
 
requirements-cuda.txt
requirements-cuda.txt
 
 
requirements-npu.txt
requirements-npu.txt
 
 
requirements-test.txt
requirements-test.txt
 
 
requirements.txt
requirements.txt
 
 
requirements_sglang.txt
requirements_sglang.txt
 
 
run_grpo.py
run_grpo.py
 
 
run_peft.sh
run_peft.sh
 
 
run_sft.py
run_sft.py
 
 
setup.py
setup.py
 
 

Repository files navigation

Verl for Tomlab

This fork of verl is tested and documented for easy use on Nexus. main is intended to be kept up to date with the official Verl, and work on specific projects can be in branches or just forks of this repo.

Project branches:

Quickstart

(This is a Nexus-confirmed version of the official quickstart.)

  1. Setup environment. This sets up Verl to use FSDP instead of Megatron and Vllm instead of SGLang. This installs from a pre-built wheel of FlashAttention (the normal way), but it has a conflict with the version of glibc on Nexus so there is a second step below here to fix this.

    module load cuda/12.8.1
module load cudnn/v9.10.2
conda create -n verl python=3.12
conda activate verl
export USE_MEGATRON=0
export USE_SGLANG=0
bash scripts/install_vllm_sglang_mcore.sh
pip install --no-deps -e .

  2. Fix the glibc Nexus conflict in FlashAttention. If you run it currently you will get something like "ImportError: /lib64/libc.so.6: version GLIBC_2.32 not found". We use a tool called polyfill-glibc to update the FlashAttention .so binary file so it is compatible with Nexus's version of glibc, even though it was built for a different version of glibc. These instructions are from here.

    # Confirm the problem. This should give you an error message. If no error, you can skip this whole part.
python -c "import flash_attn"
# Show the mismatched versions:
FLASH_PATH=$(find $CONDA_PREFIX/lib/python*/site-packages/flash_attn* -name "flash_attn_2_cuda*.so" 2>/dev/null | head -1)
FLASH_GLIBC_VERSION=$(objdump -T "$FLASH_PATH" 2>/dev/null | grep -oP 'GLIBC_\K\d+\.\d+' | sort -V | uniq | tail -1)
NEXUS_GLIBC_VERSION=$(ldd --version | head -n1 | awk '{print $NF}')
echo "$FLASH_GLIBC_VERSION must be equal to or less than $NEXUS_GLIBC_VERSION"
# Now fix with polyfill
cd ..
git clone --single-branch --branch feat/single-threaded https://github.com/tuxxi/polyfill-glibc.git
cd polyfill-glibc
ninja polyfill-glibc
./polyfill-glibc --target-glibc $NEXUS_GLIBC_VERSION $FLASH_PATH
# Confirm it works now. Should import silently.
python -c "import flash_attn"

  3. Confirm installation by running a few GRPO training steps. One RTXA5000 has enough memory for this:

    python run_grpo.py --model Qwen/Qwen3-0.6B --dataset openai/gsm8k --num_examples 10

Training

SFT

Quick test. The script automatically detects the number of available GPUs and uses FSDP to shard the model and parallelize data across all of them.

python run_sft.py --model Qwen/Qwen3-0.6B --dataset openai/gsm8k --num_examples 10

Full-parameter SFT with Qwen3-8B.

  • 4x RTX A6000 (48GB): Possible with only ~60 GB of CPU memory without CPU offloading (--no-offload_weights_and_states).
  • 2-3x RTX A6000: Possible, but requires ~150 GB of CPU memory with offloading (--offload_weights_and_states).
python run_sft.py --model Qwen/Qwen3-8B --dataset openai/gsm8k --val_split test --epochs 4 --lr 1e-5 --lr_schedule cosine --batch_size 256 --micro_batch_size_per_gpu 1 --max_length 1024 --gradient_checkpointing --save_freq 50 --val_freq 20

GRPO

This reproduces the results from https://github.com/volcengine/verl/blob/main/examples/grpo_trainer/run_qwen3-8b.sh. It runs successfully on four RTX A6000s with 120 GB of CPU memory. These details are captured in launch.sh.

python run_grpo.py --model Qwen/Qwen3-8B --dataset openai/gsm8k --val_split test --epochs 15 --lr 1e-6 --batch_size 256 --rollout_batch_size 1024 --num_generations 5 --max_response_length 1024 --kl_coef 0.001 --lr_schedule constant --save_freq 20 --val_freq 5 --vllm_cache_utilization 0.4 --batch_size_per_gpu 1 --max_prompt_length 512 --vllm_model_shards 2

DrGRPO

This reproduces DrGRPO as described in https://verl.readthedocs.io/en/latest/algo/grpo.html#drgrpo

python run_grpo.py --algorithm drgrpo

LoRA

SFT with LoRA:

python run_sft.py --lora_rank 8 --lora_alpha 16 --lora_target_modules all-linear

GRPO with LoRA:

python run_grpo.py --lora_rank 8 --lora_alpha 16 --lora_target_modules all-linear

Both scripts accept --lora_target_modules with choices: all-linear (default), all-linear-and-embedding, all-attention, qv-only.

all-linear-and-embedding applies LoRA to the embedding layer by including embed_tokens in target_modules. This works but is non-standard — the standard peft approach is to use modules_to_save=["embed_tokens"], which fully fine-tunes the embedding instead of applying a low-rank approximation. To enable this, add "modules_to_save": ["embed_tokens"] to the lora_config dict in the upstream verl SFT trainer: fsdp_sft_trainer.py#L277-L283.

Memory Management

VLLM Cache Utilization and CPU Offloading

During GRPO runs, the context length for generations is limited by VLLM Cache Utilization, which Verl defaults to 0.6. The context length for the backward pass is limited by the activations, which grow roughly linearly because of flash attention. Memory taken by the optimizer states remains on the GPU during VLLM generation unless we do offloading. If we don't do offloading, we need to lower VLLM Cache Utilization to the largest possible value it can be and still fit the optimizer states on the GPU. Finding this vllm_cache_utilization value depends only on model size and GPU memory, not context length. If we have enough CPU memory to do offloading, the optimizer states are taken off the GPU during the rollout generations and we can safely set vllm_cache_utilization to it's highest possible value (0.8 or a little above).

Max Context Length

This table lists the max context length possible for GRPO for a given model on a given GPU, and the vllm_cache_utilization setting that enables the max context length. See the notes below for a discussion of when context length is limited by vllm rollouts and when it is limited by training activations. All values assume CPU offloading enabled (--offload_weights_and_states), which requires a surprising amount of CPU memory. Without CPU offloading, FSDP keeps optimizer states on GPU (~24 GB/GPU for 8B), so you must use a much lower cache (0.3-0.4) and context lengths are shorter.

Numbers measured on RTX A5000 (24 GB) and RTX A6000 (48 GB) GPUs using GRPO, FSDP2, vLLM, with CPU offloading enabled (--offload_weights_and_states). 80 GB values are extrapolated. Because we did CPU offloading and set the vllm_cache_utilization to max, the limiting factor was always the backward pass and not the rollouts.

Model 24GB GPU 48GB GPU 80GB GPU
0.6B, 1 Shard 16K, cache 0.8 32K, cache 0.8 32K, cache 0.8
1.7B, 1 Shard 8K, cache 0.8 32K, cache 0.8 32K, cache 0.8
1.7B, 2 Shards 8K, cache 0.8 32K, cache 0.8 32K, cache 0.8
4B, 1 Shard 8K, cache 0.8 16K, cache 0.8 32K, cache 0.8
4B, 2 Shards 8K, cache 0.8 16K, cache 0.8 32K, cache 0.8
8B, 1 Shard OOM 16K, cache 0.8 32K, cache 0.8
8B, 2 Shards 8K, cache 0.8 16K, cache 0.8 32K, cache 0.8
16B, 1 Shard OOM OOM 4K, cache 0.8
16B, 2 Shards OOM 4K, cache 0.8 8K, cache 0.8

Notes:

  • Num Shards is the vLLM tensor parallel size. Splits model weights across GPUs for generations, freeing per-GPU VRAM for KV cache. Does not reduce training activation memory. Requires 1 GPU per shard.
  • --vllm_cache_utilization controls how much GPU memory vLLM reserves for model weights + KV cache during rollout. With vLLM ≥ 0.8.5 and sleep_level=2 (verl's default), vLLM releases all GPU memory (weights + KV cache) during training. This means higher cache utilization helps rollout (more KV cache for longer sequences) without hurting training. Cache of 0.3 caused rollout failure because KV cache was too small. Default 0.6 is a good starting point; increase to 0.7-0.8 if you need longer rollout sequences.
  • Two independent bottlenecks for long context: With sleep_level=2, vLLM releases all GPU memory during training, so the rollout and training phases never compete for memory. Each phase has its own limiting factor:
    1. Rollout generation (KV cache): During generation, vLLM needs enough KV cache to hold the full sequence. If vllm_cache_utilization is too low, the KV cache is too small and rollout fails at initialization. Example: 0.6B model, 16K prompt, cache=0.3 on 24GB GPU — vLLM reserved only 7.2 GB total (0.3×24), leaving ~6 GB for KV cache after model weights, which wasn't enough for 16K tokens.
    2. Training backward pass (activations): During training, PyTorch stores activation tensors that scale with context length × model size. Even with gradient checkpointing, very long contexts exceed GPU memory during the backward pass. Example: 0.6B model, 32K prompt on 24GB GPU — rollout succeeded fine, but the backward pass OOM'd because 32K activations required ~40 GB. Example: 8B model (2 shards), 32K prompt on 48GB GPU — rollout succeeded (model split across GPUs with plenty of KV cache), but training OOM'd because activations required ~48 GB.
  • Num Generations only affects wall clock time, and not linearly.
    num_generations Wall Time
    2 199s
    5 243s
    10 344s
    16 413s

CPU Memory

Verl's GRPO trainer uses Ray, which is pretty CPU-memory intensive. In addition, FSDP2's model offloading feature uses an extreme amount of CPU memory (something like 10x the the model weights.) The baseline memory requirement from Ray and Vllm initialization with a single GPU and no model-offloading is 26 GB. Every addtional GPU adds another 6GB of CPU memory from Ray worker overhead.

These values look pretty extreme, but here is what came out of some tests:

Model GPUs Shards CPU RAM
Qwen3-0.6B 1 1 32 GB
Qwen3-0.6B 7 1 70 GB
Qwen3-1.7B 1 1 58 GB
Qwen3-4B 1 1 114 GB
Qwen3-8B 2 2 209 GB

Other Notes

Weights Bucket Note

The weight transfer between FSDP2 and vLLM calls .full_tensor() on each model layer, and there is a "weights bucket" setting that reserves space for this. The Verl default is 2048, and that was fine for FSDP1, but the Verl default fails for 8B models and FSDP2. For 8B Qwen models, the embedding layer (151936 x 4096 x float32 = ~2.4 GB) exceeds the default 2048 MB weight update bucket, causing an assertion error. We override the Verl default with --update_weights_bucket_mb 4096 to fix this, and it can be set higher for larger models. For reference, see the TODO at verl/workers/rollout/vllm_rollout/vllm_rollout.py:209.

Bfloat16 Warning:

You will notice in the run log a warning about bf16 vs fp32, but you can ignore this: Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3Model is torch.float32. What is happening is that Verl intentionally loads the model in fp32 because it uses this to initialize the optimizer in fp32, and then immediately converts the model to bf16 by default:

verl/verl/workers/fsdp_workers.py

Line 313 in f93129d

# note that we have to create model in fp32. Otherwise, the optimizer is in bf16, which is incorrect

verl/verl/workers/fsdp_workers.py

Line 495 in f93129d

param_dtype = PrecisionType.to_dtype(mixed_precision_config.get("param_dtype", "bf16"))

If we want to override the default, we can do so with this:

actor_rollout_ref.actor.fsdp_config.dtype=float32
actor_rollout_ref.ref.fsdp_config.dtype=float32

Viewing Tomlab-specific Git History

Show only Tomlab commits:

git log --author="Monte Hoover\|Andrew Zheng\|Abhimanyu Hans" --oneline

Show commits with dates:

git log --author="Monte Hoover\|Andrew Zheng\|Abhimanyu Hans" --format="%h %an %ad %s" --date=short

If you want to make it a git alias:

git config alias.team-log 'log --author="Monte Hoover\|Andrew Zheng\|Abhimanyu Hans" --format="%h %an %ad %s" --date=short'
git team-log

Original README:

👋 Hi, everyone! verl is a RL training library initiated by ByteDance Seed team and maintained by the verl community.

Ask DeepWiki.com GitHub Repo stars Twitter Documentation

seed logo

verl: Volcano Engine Reinforcement Learning for LLMs

verl is a flexible, efficient and production-ready RL training library for large language models (LLMs).

verl is the open-source version of HybridFlow: A Flexible and Efficient RLHF Framework paper.

verl is flexible and easy to use with:

  • Easy extension of diverse RL algorithms: The hybrid-controller programming model enables flexible representation and efficient execution of complex post-training dataflows. Build RL dataflows such as GRPO, PPO in a few lines of code.

  • Seamless integration of existing LLM infra with modular APIs: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as FSDP, Megatron-LM, vLLM, SGLang, etc

  • Flexible device mapping: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes.

  • Ready integration with popular HuggingFace models

verl is fast with:

  • State-of-the-art throughput: SOTA LLM training and inference engine integrations and SOTA RL throughput.

  • Efficient actor model resharding with 3D-HybridEngine: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases.

verl-arch.png

News

  • [2026/01] verl has been migrated to the verl-project
  • [2026/01] verl first meetup was successfully held in Shanghai on 01/10, hosted by Volcengine and NVIDIA, the slides has been uploaded to verl-data.
  • [2026/01] The recipe directory has been migrated to a dedicated repository: verl-recipe and added as a submodule. See verl-project#4795. It can be used as it was after git submodule update --init --recursive recipe. Note that transfer_queue, fully_async_policy, one_step_off_policy and vla are kept under verl/experimental since they are planned to be merged into the main library. Use them through verl.experimental.{module}.
  • [2025/12] Mind Lab successfully used verl and Megatron-bridge to train GRPO Lora for Trillion-parameter model on 64 H800 - See their techblog.
  • [2025/10] verl is presented in the PyTorch Conference 2025.
  • [2025/08] verl is presented in the PyTorch Expert Exchange Webinar. Slides available.
  • [2025/07] The ReTool recipe is fully open sourced. Blog
  • [2025/07] The first verl meetup will be held at ICML Vancouver on July 16th! Please join us if you are at ICML! (onsite only)
  • [2025/06] verl with Megatron backend enables large MoE models such as DeepSeek-671B and Qwen3-235B.
  • [2025/03] DAPO is the open-sourced SOTA RL algorithm that achieves 50 points on AIME 2024 based on the Qwen2.5-32B pre-trained model, surpassing the previous SOTA achieved by DeepSeek's GRPO (DeepSeek-R1-Zero-Qwen-32B). DAPO's training is fully powered by verl and the reproduction code is available in recipe/dapo now.
more...

Key Features

Upcoming Features and Changes

Getting Started

Documentation

Quickstart:

Running a PPO example step-by-step:

Reproducible algorithm baselines:

For code explanation and advance usage (extension):

Blogs from the community

Performance Tuning Guide

The performance is essential for on-policy RL algorithm. We have written a detailed performance tuning guide to help you optimize performance.

Upgrade to vLLM >= v0.8.2

verl now supports vLLM>=0.8.2 when using FSDP as the training backend. Please refer to this document for the installation guide and more information. Please avoid vllm 0.7.x, which contains bugs that may lead to OOMs and unexpected errors.

Use Latest SGLang

SGLang is fully supported with verl, and SGLang RL Group is working extensively on building unique features, including multi-turn agentic RL, VLM RLHF, server-based RL, and partial rollout. Please refer to this document for the installation guide and more information.

Upgrade to FSDP2

verl is fully embracing FSDP2! FSDP2 is recommended by torch distributed team, providing better throughput and memory usage, and is composible with other features (e.g. torch.compile). To enable FSDP2, simply use verl main and set the following options:

actor_rollout_ref.ref.strategy=fsdp2
actor_rollout_ref.actor.strategy=fsdp2
critic.strategy=fsdp2

Furthermore, FSDP2 cpu offloading is compatible with gradient accumulation. You can turn it on to save memory with actor_rollout_ref.actor.fsdp_config.offload_policy=True. For more details, see verl-project#1026

AMD Support (ROCm Kernel)

verl now supports FSDP as the training engine (Megatron support coming soon) and both integrates with vLLM and SGLang as inference engines. Please refer to this document for the installation guide and more information, and this document for the vLLM performance tuning for ROCm.

Citation and acknowledgement

If you find the project helpful, please cite:

@article{sheng2024hybridflow,
  title   = {HybridFlow: A Flexible and Efficient RLHF Framework},
  author  = {Guangming Sheng and Chi Zhang and Zilingfeng Ye and Xibin Wu and Wang Zhang and Ru Zhang and Yanghua Peng and Haibin Lin and Chuan Wu},
  year    = {2024},
  journal = {arXiv preprint arXiv: 2409.19256}
}

verl is inspired by the design of Nemo-Aligner, Deepspeed-chat and OpenRLHF. The project is adopted and contributed by Bytedance, Anyscale, LMSys.org, Alibaba Qwen team, Shanghai AI Lab, Tsinghua University, UC Berkeley, UCLA, UIUC, University of Hong Kong, ke.com, All Hands AI, ModelBest, JD AI Lab, Microsoft Research, StepFun, Amazon, LinkedIn, Meituan, Camel-AI, OpenManus, Xiaomi, NVIDIA research, Baichuan, RedNote, SwissAI, Moonshot AI (Kimi), Baidu, Snowflake, Skywork.ai, JetBrains, IceSword Lab, and many more.

Awesome Projects Built with verl

Welcome to register your awesome project build with verl for other developers' reference!

  • TinyZero: a reproduction of DeepSeek R1 Zero recipe for reasoning tasks GitHub Repo stars
  • SkyThought: RL training for Sky-T1-7B by NovaSky AI team. GitHub Repo stars
  • simpleRL-reason: SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild GitHub Repo stars
  • Easy-R1: Multi-modal RL training framework GitHub Repo stars
  • OpenManus-RL: LLM Agents RL tuning framework for multiple agent environments. GitHub Repo stars
  • rllm: async RL training with verl-pipeline GitHub Repo stars
  • RAGEN: a general-purpose reasoning agent training framework GitHub Repo stars
  • Search-R1: RL with reasoning and searching (tool-call) interleaved LLMs GitHub Repo stars
  • ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning GitHub Repo stars
  • Skywork-OR1: Skywork open reaonser series GitHub Repo stars
  • ToRL: Scaling tool-integrated RL GitHub Repo stars
  • Absolute Zero Reasoner: A no human curated data self-play framework for reasoning GitHub Repo stars
  • verl-agent: A scalable training framework for long-horizon LLM/VLM agents, along with a new algorithm GiGPO GitHub Repo stars
  • RL-Factory: An easy and efficient RL post-training framework for Agentic Learning GitHub Repo stars
  • ReTool: ReTool: reinforcement learning for strategic tool use in LLMs. Code release is in progress...
  • verl-tool: An unified and easy-to-extend tool-agent training framework based on verlGitHub Repo stars
  • PRIME: Process reinforcement through implicit rewards GitHub Repo stars
  • MemAgent: MemAgent: Reshaping Long-Context LLM with Multi-Conv RL based Memory Agent GitHub Repo stars
  • POLARIS: A Post-training recipe for scaling RL on Advanced Reasoning models GitHub Repo stars
  • GUI-R1: GUI-R1: A Generalist R1-style Vision-Language Action Model For GUI Agents GitHub Repo stars
  • DeepRetrieval: RL Training of Search Agent with Search/Retrieval Outcome GitHub Repo stars
  • Code-R1: Reproducing R1 for Code with Reliable Rewards GitHub Repo stars
  • DeepResearcher: Scaling deep research via reinforcement learning in real-world environments GitHub Repo stars
  • VAGEN: Training VLM agents with multi-turn reinforcement learning GitHub Repo stars
  • RM-R1: RL training of reasoning reward models GitHub Repo stars
  • LUFFY: Learning to Reason under Off-Policy GuidanceGitHub Repo stars
  • DeepMath: DeepMath-103K data and series models for math reasoningGitHub Repo stars
  • PACS: Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR GitHub Repo stars
  • Entropy Mechanism of RL: The Entropy Mechanism of Reinforcement Learning for Large Language Model ReasoningGitHub Repo stars
  • LLaSA-TTS-GRPO: TTS fine-tuning with GRPO optimization based on LLASA models GitHub Repo stars
  • PF-PPO: Policy Filtration for PPO based on the reliability of reward signals for more efficient and robust RLHF.
  • RACRO: Build multi-modal reasoning models via decoupling it into query-conditioned captioning and text-only reasoning GitHub Repo stars
  • Agent Lightning: A flexible and extensible framework that enables seamless agent optimization for any existing agent framework. GitHub Repo stars
  • VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use. GitHub Repo stars
  • Kimina-Prover-RL: Training pipeline for formal theorem proving, based on a paradigm inspired by DeepSeek-R1.
  • RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization.
  • rStar2-Agent: Using reinforcement learning with multi-step tool-calling for math tasks, rStar2-Agent-14B reaches frontier-level math reasoning in just 510 RL training steps GitHub Repo stars
  • Vision-SR1: Self-Rewarding Vision-Language Model via Reasoning Decomposition GitHub Repo stars
  • SimpleVLA-RL: SimpleVLA-RL: A Simple yet Effective Vision-Language Action Model for Reinforcement Learning GitHub Repo stars
  • Table-R1: Table-R1: Inference-Time Scaling for Table Reasoning GitHub Repo stars
  • Revisual-R1: Revisual-R1: Advancing Multimodal Reasoning From Optimized Cold Start to Staged Reinforcement Learning GitHub Repo stars
  • ARES: ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping GitHub Repo stars
  • Meta-Bandit-LLM: Meta-Bandit-LLM: Long-horizon multiturn interactive training for meta-bandit agents GitHub Repo stars
  • PokeeResearch: PokeeResearch: State-of-the-art 7B DeepResearch Agent that leverages web search and content reading capabilities to answer complex questions using the most up-to-date information available online. Github Repo Stars
  • Search Self-play: Pushing the Frontier of Agent Capability without Supervision GitHub Repo stars
  • OneThinker: All-in-one Reasoning Model for Image and Video GitHub Repo stars
  • OpenTinker: Democratizing Agentic Reinforcement Learning as a Service GitHub Repo stars
  • FlowRL: Matching reward distributions via flow balance for diverse exploration and generalizable reasoning GitHub Repo stars
  • Logic-RL: a reproduction of DeepSeek R1 Zero on 2K Tiny Logic Puzzle Dataset. GitHub Repo stars
  • Seed-Coder: RL training of Seed-Coder boosts performance on competitive programming GitHub Repo stars
  • all-hands/openhands-lm-32b-v0.1: A strong, open coding agent model, trained with multi-turn fine-tuning
  • s3 Efficient Yet Effective Search Agent Training via RL GitHub Repo stars
  • Rec-R1: Bridging Generative Large Language Models and Recommendation Systems via Reinforcement Learning
  • Explore RL Data Scaling: Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback
  • FIRE: Flaming-hot initiation with regular execution sampling for large language models
  • DQO: Enhancing multi-Step reasoning abilities of language models through direct Q-function optimization
  • ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
  • cognition-engineering: Test time scaling drives cognition engineering. GitHub Repo stars
  • Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoning. GitHub Repo stars
  • AdaRFT: Efficient Reinforcement Finetuning via Adaptive Curriculum Learning GitHub Repo stars
  • critic-rl: LLM critics for code generation GitHub Repo stars
  • self-rewarding-reasoning-LLM: self-rewarding and correction with generative reward models GitHub Repo stars
  • DeepEnlighten: Reproduce R1 with social reasoning tasks and analyze key findings GitHub Repo stars
  • MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse GitHub Repo stars
  • PURE: Credit assignment is the key to successful reinforcement fine-tuning using process reward model GitHub Repo stars
  • cognitive-behaviors: Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs GitHub Repo stars
  • deepscaler: iterative context scaling with GRPO GitHub Repo stars
  • DAPO: the fully open source SOTA RL algorithm that beats DeepSeek-R1-zero-32B GitHub Repo stars
  • NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation GitHub Repo stars
  • SPEAR: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning (ICLR 2026) GitHub Repo stars
  • RuleReasoner: RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling (ICLR 2026) GitHub Repo stars
  • MetaphorStar: Image Metaphor Understanding and Reasoning with End-to-End Visual Reinforcement Learning GitHub Repo stars

Contribution Guide

See contributions guide

About ByteDance Seed Team

Founded in 2023, ByteDance Seed Team is dedicated to crafting the industry's most advanced AI foundation models. The team aspires to become a world-class research team and make significant contributions to the advancement of science and society. You can get to know Bytedance Seed better through the following channels👇

We are HIRING! Send us an email if you are interested in internship/FTE opportunities in RL for agents.

About

verl: Volcano Engine Reinforcement Learning for LLMs

verl.readthedocs.io/en/latest/index.html

Resources

Readme
License

Contributing

Contributing
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages