Skip to content

Pick Objects in Order cannot reproduce Table 2 result (56% vs 90%) #4

Description

@yunhang8658

Thanks for releasing the code and checkpoint. We reproduced most RoboTwin-MeM tasks close to the paper, but
pick_objects_in_order remains substantially lower.

Using the released checkpoint and the reported evaluation setup:

  • checkpoint: RoboTwin-MeM/final_model/pytorch_model.pt
  • task_config=demo_clean
  • instruction_type=unseen
  • ACTION_MODE=abs
  • unnorm_key=new_embodiment
  • camera=LargeView (FOV 50)
  • seed starts at 100000
  • step limit: 1500
  • one policy server for this task

we obtain 28/50 = 56.0% on seeds 100000–100049, while Table 2 reports 90% for Pick Objects in Order.

The rollout logs show normal execution and no crashes. Other tasks are broadly consistent with the paper, so this
appears task-specific.

We also noticed that a previous reproduction issue reported the authors' local result as 5/9 for this task, which
is close to our 56%, but it was not further addressed.

Could you please clarify:

  1. Does Table 2's 90% use exactly pick_objects_in_order with the released steps_100000 / final_model
    checkpoint?
  2. Can you share the exact evaluation command/config used to obtain the reported 90%?

We can provide rollout videos and full logs if useful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions