Hi EventVLA authors,
Thank you very much for releasing EventVLA, RoboTwin-MeM benchmark, checkpoints, and training/evaluation code.
We are currently reproducing EventVLA, and we encounter the following problems:
1. Missing eval step limits
- The released code does not include
_eval_step_limit.yml. In envs/_base_task.py any task not listed falls back to step_lim = 1000.
can you publish the exact per-task eval step limits used to produce the Table 2 results (or ship _eval_step_limit.yml).
we wonder if this mismatch cause Pick Objects in Order cannot reproduce Table 2 result(50% vs 90%)
2. Undocumented baselines training config
The paper reports the pi0.5 baseline at ~20% success on rearrange_blocks_hard, but the training recipe for this baseline is not documented anywhere in the release. The following are unspecified:
- number of training steps
- global batch size
- LoRA vs full fine-tuning
- action horizon and the number of chunk steps executed open-loop per inference at eval
Our reproduction attempt — pi0.5 (PaliGemma-2B + 300M action-expert, LoRA), global batch 32, 30k steps, 50 demos, action horizon 50, executing the full 50-step chunk per inference — gives on rearrange_blocks_hard:
- 0/30 success, average max-reward ≈ 0.28.
Could you publish the pi0.5 baseline training config (steps, batch size, LoRA/full) and the eval protocol (step limit, chunk/replan length, number of trials) used for Table 2.
Hi EventVLA authors,
Thank you very much for releasing EventVLA, RoboTwin-MeM benchmark, checkpoints, and training/evaluation code.
We are currently reproducing EventVLA, and we encounter the following problems:
1. Missing eval step limits
_eval_step_limit.yml. Inenvs/_base_task.pyany task not listed falls back tostep_lim = 1000.can you publish the exact per-task eval step limits used to produce the Table 2 results (or ship
_eval_step_limit.yml).we wonder if this mismatch cause Pick Objects in Order cannot reproduce Table 2 result(50% vs 90%)
2. Undocumented baselines training config
The paper reports the pi0.5 baseline at ~20% success on
rearrange_blocks_hard, but the training recipe for this baseline is not documented anywhere in the release. The following are unspecified:Our reproduction attempt — pi0.5 (PaliGemma-2B + 300M action-expert, LoRA), global batch 32, 30k steps, 50 demos, action horizon 50, executing the full 50-step chunk per inference — gives on
rearrange_blocks_hard:Could you publish the pi0.5 baseline training config (steps, batch size, LoRA/full) and the eval protocol (step limit, chunk/replan length, number of trials) used for Table 2.