Unable to reproduce RoboTwin-MeM evaluation with the released checkpoint

[Ryuuranwlb]

Thanks for open-sourcing EventVLA and the RoboTwin-MeM benchmark. While reproducing the RoboTwin-MeM evaluation with the released checkpoint (commit <4b5b260>), we encountered several independent issues before eval.sh could complete a rollout. Reporting them together since they all block the same reproduction path; happy to split if you prefer.

1. RoboTwin-Mem/task_config/ is missing. script/eval_policy.py opens ./task_config/<task_config>.yml and task_config/_camera_config.yml; envs/_base_task.py reads task_config/_eval_step_limit.yml; envs/_GLOBAL_CONFIGS.py points CONFIGS_PATH there. Only script/_task_config_template.json ships. We copied task_config/ from upstream RoboTwin 2.0, which works, but it is unclear whether those settings match the paper's numbers. Could you add the directory used for the paper?

2. Released checkpoint declares framework.name: QwenOFT, which is not in FRAMEWORK_REGISTRY. baseframework.from_pretrained() raises NotImplementedError. Editing the checkpoint's config.yaml to EventVLA appears correct (its memory_ablation_mode: pure_image_keyframe_memory and use_keyframe_predict_head: auto match the EventVLA class) — can you confirm, or add a QwenOFT alias?

3. The configured unnorm_key does not match the released statistics.
examples/RoboTwin-Mem/eval_files/deploy_policy.yml sets:unnorm_key: robotwin_mem，but the released dataset_statistics.json contains only the key new_embodiment. The evaluation currently works only because _check_unnorm_key() silently falls back to the first available key. For a checkpoint containing multiple statistics entries, this behavior could silently select unintended normalization statistics. Possible fixes would be to set unnorm_key: new_embodiment in the released deployment config, provide a matching robotwin_mem entry in dataset_statistics.json, and/or make the fallback emit a warning or error.

4. eval.sh dies silently under set -e. Both wait_for_*_server loops advance timers with ((elapsed++)); when elapsed is 0 the expression returns status 1, so set -euo pipefail kills the script on the first iteration — before any server can finish loading — and the spawned server is orphaned. Fix: elapsed=$((elapsed + 1)) or ((++elapsed)).

5. script/_install.sh clones curobo unpinned. The MeM install script does a bare git clone https://github.com/NVlabs/curobo.git, but current curobo main uses the new curobo/_src layout, which is incompatible with envs/robot/planner.py's legacy imports (curobo.types.math.Pose, curobo.wrap.reacher.motion_gen), so the expert-check crashes on every seed. Upstream RoboTwin 2.0 pins --branch v0.7.8 and pip install warp-lang==1.12.0; The MeM fork currently does not retain either of these pins. (The warp pin matters too: curobo's dependency resolution upgrades warp to ≥1.15, which removed the wp.torch lazy attribute that curobo 0.7.8 relies on.) Workaround: reinstall curobo at v0.7.8 and force warp-lang==1.12.0.

6. MeM-specific assets are not in the RoboTwin 2.0 asset bundle and this is undocumented. 18 object directories (e.g. 004_numbercard, 005_button, 003_cover) required by the MeM tasks exist only in the HF dataset ganlinyang/RoboTwin-MeM under assets/. Following the RoboTwin 2.0 asset download instructions yields xxx is not exist model file!. A line in the README (and/or an updated _download_assets.sh) would save a lot of head-scratching.

7. Eval video logging crashes with the default camera config. _base_task.py (around line 1535) hardcodes now_obs["third_view_rgb"] for the eval video stream, but demo_clean.yml sets data_type.third_view: false, so enabling eval_video_log raises KeyError. Suggest falling back to head_camera or gating on the config flag.

Environment: repo commit <4b5b260>; released RoboTwin-MeM multi-task checkpoint (final_model/pytorch_model.pt); conda envs per README (eventvla: torch 2.6.0+cu124, transformers 4.57.0; RoboTwin-MeM: torch 2.4.1+cu121, sapien 3.0.0b1). After applying the workarounds above, the evaluation runs end-to-end: keyframe-memory commits are triggered and rollouts complete successfully. These blockers therefore appear to be release, configuration, asset-packaging, and dependency issues rather than failures in the released model itself.

Happy to provide additional logs, test proposed fixes, or help split these into separate issues. Thanks again!