V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control

This is the official implementation of V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control (Reinforcement Learning Journal, 2026).

Installation

Dependencies are managed with uv. To install it, run:

curl -LsSf https://astral.sh/uv/install.sh | sh

Pin the Python version for your GPU (this selects the JAX stack):

GPU	Pin	JAX stack
RTX 30x0 / 40x0	uv python pin 3.10	jax 0.4.25
RTX 50x0 / Bx00 (Blackwell)	uv python pin 3.11	jax 0.6.2

To build the environment (CUDA wheels included), run:

uv sync

To install the MuJoCo/OpenGL system libraries, run:

sudo apt-get install -y libglew-dev libglib2.0-0 libgl1-mesa-dev libosmesa6-dev

To install the MuJoCo 2.1.0 binaries (required by mujoco-py for Adroit and Meta-World), run:

wget https://github.com/deepmind/mujoco/releases/download/2.1.0/mujoco210-linux-x86_64.tar.gz
mkdir -p ~/.mujoco && tar -xzf mujoco210-linux-x86_64.tar.gz -C ~/.mujoco
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.mujoco/mujoco210/bin:/usr/lib/nvidia

For headless rendering, set:

export MUJOCO_GL="egl"
export MUJOCO_EGL_DEVICE_ID="0"
export MKL_SERVICE_FORCE_INTEL="0"

Docker

If you'd rather not install the system libraries on the host, use the dev image:

docker build -f deps/Dockerfile.dev \
    --build-arg UID=$(id -u) --build-arg GID=$(id -g) -t vsimba-dev .

docker run --gpus all -it \
    -v "$PWD":/workspace \
    -v vsimba-venv-py311:/home/user/venv \
    -v vsimba-uvcache:/home/user/.cache/uv \
    vsimba-dev

# then, inside the container (once; the named volume keeps it across runs):
uv sync

Usage

To run a single experiment (defaults: V-Simba on visual DMC), run:

uv run python run_online.py \
    --overrides env.env_name=cheetah-run

To benchmark on a task suite, run:

uv run python run_parallel.py \
    --env_type vsimba_1M \
    --device_ids <list of gpu devices to use> \
    --num_seeds <num_seeds> \
    --num_exp_per_device <number>

To reproduce the paper experiments, run:

bash scripts/vsimba_dmc.sh
bash scripts/vsimba_adroit.sh
bash scripts/vsimba_metaworld.sh

License

This project is released under the Apache 2.0 license.

Citation

@article{kim2026vsimba,
  title={V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control},
  author={Donghu Kim and Youngdo Lee and Hojoon Lee and Johan Obando-Ceron and Byungkun Lee and Aaron Courville and Pablo Samuel Castro and Jaegul Choo and Clare Lyle},
  journal={Reinforcement Learning Journal},
  year={2026}
}