Skip to content

Missing eval step limits & undocumented pi0.5 baseline training config block reproduction of the Table 2 baseline #9

Description

@hhh0108

Hi EventVLA authors,

Thank you very much for releasing EventVLA, RoboTwin-MeM benchmark, checkpoints, and training/evaluation code.

We are currently reproducing EventVLA, and we encounter the following problems:

1. Missing eval step limits

  • The released code does not include _eval_step_limit.yml. In envs/_base_task.py any task not listed falls back to step_lim = 1000.
    can you publish the exact per-task eval step limits used to produce the Table 2 results (or ship _eval_step_limit.yml).

we wonder if this mismatch cause Pick Objects in Order cannot reproduce Table 2 result(50% vs 90%)

2. Undocumented baselines training config

The paper reports the pi0.5 baseline at ~20% success on rearrange_blocks_hard, but the training recipe for this baseline is not documented anywhere in the release. The following are unspecified:

  • number of training steps
  • global batch size
  • LoRA vs full fine-tuning
  • action horizon and the number of chunk steps executed open-loop per inference at eval

Our reproduction attempt — pi0.5 (PaliGemma-2B + 300M action-expert, LoRA), global batch 32, 30k steps, 50 demos, action horizon 50, executing the full 50-step chunk per inference — gives on rearrange_blocks_hard:

  • 0/30 success, average max-reward ≈ 0.28.

Could you publish the pi0.5 baseline training config (steps, batch size, LoRA/full) and the eval protocol (step limit, chunk/replan length, number of trials) used for Table 2.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions