FetchMan

Learning Visual Humanoid Loco-Manipulation Policies from Simulated Experiences

Website · Paper · Data · Checkpoints

Setup

conda create -n g1 python=3.10 -y && conda activate g1
pip install -r requirements.txt -r train/requirements.txt

Example

1. Collect 1000 bowl pick episodes in one house, with 8 workers.

MAX_EPISODES=1000 bash molmospaces/scripts/collect_parallel.sh 8 0 bowl_pick

2. Fold the per-worker shards into one dataset.

python molmospaces/scripts/merge_datasets.py \
    --input_glob='data/data_*/*' \
    --output_root=data/local/bowl_pick --output_repo_id=local/bowl_pick

3. Train on it, with a held-out val split and rollouts in the simulator.

python -u train/main.py \
    --data_root=data/local/bowl_pick --repo_id=local/bowl_pick \
    --env=molmospaces/configs/bowl_pick.py \
    --devices=0 --agent.batch_size=128 \
    --val_episodes=50 --val_interval=1 \
    --eval_interval=1000 --eval_episodes=25 --eval_num_envs=8 --eval_render_gpu=1

4. Fine-tune the behavior-cloned policy with RL, scored on a fixed benchmark.

# export 200 successful resets to replay as a fixed eval set
python main.py --env=molmospaces/configs/bowl_pick.py --episodes=450 \
    --save_json=molmospaces/assets/eval_sets/bowl_pick.jsonl

python -u rl/main.py \
    --checkpoint=exp/g1/Debug/<run>/step=00016000.ckpt \
    --env=molmospaces/configs/bowl_pick.py \
    --num_groups=32 --group_size=8 --parallel_envs=64 \
    --noise_level=0.7 --num_sampling_steps=10 --actor_lr=5e-6 \
    --time_limit=40 --max_steps_per_episode=15000 \
    --gpu=0 --render_gpus=1,2,3,4,5,6,7 \
    --eval_every_n_updates=10 --save_every_n_updates=10 \
    --benchmark=bowl_pick \
    --run_name=bowl_pick_grpo

Benchmark

Each benchmark is a set of saved reset states that it restores exactly.

1. Download the BC and RL checkpoints, and the houses the benchmarks use.

hf download omarrayyann/fetchman --local-dir checkpoints
python molmospaces/scripts/download_scenes.py --benchmark bowls all

2. Score a checkpoint (bowls_bc, bowls_rl, all_bc or all_rl) on its benchmark.

python benchmark/main.py --checkpoint=checkpoints/bowls_rl/model.ckpt --benchmark=bowls \
    --gpu=0 --render_gpus=1,2,3,4,5,6,7

bowls (100 episodes)	Behavior Cloning	+ Reinforcement Learning
success rate	62	79

all (450 episodes)	Behavior Cloning	+ Reinforcement Learning
success rate	29	42

Success rates are noisy: one pass moves by several points. The all benchmark here is larger than the 50 episodes in the paper, and larger benchmarks will be added.

Citation

@article{rayyan2026fetchman,
  title   = {FetchMan: Learning Visual Humanoid Loco-Manipulation Policies from Simulated Experiences},
  author  = {Rayyan, Omar and Li, Zhi and Argus, Max and Jiang, Yuxin and Yu, Chang and Jiang, Chenfanfu and Cui, Yuchen},
  journal = {arXiv preprint arXiv:2608.17027},
  year    = {2026}
}