[Reproduction Issue] Possible batch-slot mismatch in runtime memory and configuration clarifications

[JLUbaijh2323]

---

Possible runtime memory batching issue and configuration clarification for RoboTwin-MeM reproduction

Hi EventVLA authors,

Thank you very much for releasing EventVLA, RoboTwin-MeM benchmark, checkpoints, and training/evaluation code.

We are currently reproducing EventVLA on RoboTwin-MeM and have been carefully checking the training pipeline. During reproduction, we noticed several behaviors that may affect runtime keyframe memory propagation and the final performance.

We are not sure whether these are expected behaviors or differences between the public release and the internal training setup, so we would like to ask for clarification.

Our reproduction is based on:

Repository:
https://github.com/InternRobotics/EventVLA
Commit:

4b5b26030abddf83bc60e1a6b067de8f521fd0ec

---

1. Possible mismatch between runtime memory and batch slot ordering

From our understanding of EventVLA's runtime memory mechanism, predicted keyframes are maintained according to batch slots.

The runtime memory contains states such as:

_runtime_keyframe_image_bank[slot_idx]

_runtime_pending_keyframe_writes[slot_idx]

_runtime_slot_episode_ids[slot_idx]

Therefore, we assume that each batch slot should maintain a causal trajectory:

Batch 0:

slot 0 -> episode A, t=0

slot 1 -> episode B, t=0

slot 2 -> episode C, t=0

...

Batch 1:

slot 0 -> episode A, t=50

slot 1 -> episode B, t=50

slot 2 -> episode C, t=50

...

so that the keyframe predicted at an earlier timestep can be consumed by later timesteps of the same episode.

However, when using the current public SequentialEpisodeBatchSampler, we observed that samples seem to be flattened first and then split into batches.

For example:

Batch 0:

slot 0 -> episode A, t=0

slot 1 -> episode A, t=50

slot 2 -> episode A, t=100

slot 3 -> episode A, t=150

Batch 1:

slot 0 -> episode A, t=200

slot 1 -> episode A, t=250

slot 2 -> episode A, t=300

slot 3 -> episode A, t=350

In this case, the temporal sequence of one memory slot becomes:

slot 0:

t=0 -> t=200 -> t=400 ...

instead of:

t=0 -> t=50 -> t=100 ...

This may affect:

1. predicted keyframe propagation;

2. exact keyframe fetching;

3. pending memory write consumption;

4. runtime memory bank population after teacher forcing decreases.

Especially with:

keyframe_train_memory_source: teacher_to_predict

keyframe_eval_memory_source: predict

a keyframe predicted at timestep t may not be consumed by the next causal timestep because the next timestep may already be located in another batch slot.

---

2. Experiment: persistent episode-slot sampler

To verify this hypothesis, we implemented an optional sampler modification.

Instead of assigning arbitrary samples to batch slots, we preserve episode-slot correspondence:

Batch 0:

slot 0 -> episode A, t=0

slot 1 -> episode B, t=0

slot 2 -> episode C, t=0
Batch 1:
slot 0 -> episode A, t=50

slot 1 -> episode B, t=50

slot 2 -> episode C, t=50

This allows runtime memory to evolve causally.

We compared three settings:

1. Original public reproduction.

2. Oracle GT memory (always using ground-truth keyframes).

3. Modified persistent episode-slot sampler.

---

Figure 1: Original reproduction vs GT memory oracle

Green:
Original reproduction with predicted runtime memory.

Yellow:
Always using GT keyframe memory.

Observation:

The original reproduction shows a significant reduction of runtime memory keyframe count after switching from teacher memory to predicted memory, while the GT oracle remains stable.

---

Figure 2: Modified persistent episode-slot sampler

Blue:
Persistent episode-slot sampler.

Observation:

After preserving episode-slot causality, runtime memory remains much more stable and behaves closer to the GT setting.

---

Based on these observations, we would like to ask:

1. Was the original EventVLA training performed with a sampler that preserves episode-to-batch-slot correspondence?

2. Does the released SequentialEpisodeBatchSampler represent the exact training sampler used for the paper?

3. Should runtime memory slots always correspond to the same episode across consecutive batches when using teacher_to_predict?

4. Is there any additional mechanism in the internal code that handles this memory-slot propagation?

---

3. Configuration differences between paper, YAML, and released scripts

During reproduction, we also noticed some differences between the paper description, default YAML configuration, and launcher scripts.

For example:

Item	Paper / checkpoint description	Public YAML
Training steps	80k	80k
Released checkpoint	steps_100000	100k
Memory size	5	4
Batch size	4	8
Keyframe NMS	unclear	20
Keyframe cooldown	unclear	20

We are unsure whether these correspond to different experimental versions.

Could you please clarify the exact configuration used for the reported RoboTwin-MeM results?

Especially:

• global batch size;

• number of GPUs;

• training steps;

• maximum memory keyframes;

• keyframe loss weight;

• teacher-to-predict schedule;

• sampling interval.

---

4. Dataset mixture clarification

We noticed that the default:

data_mix: robotwin_mem

does not exactly match the eight tasks described in the README.

The README lists:

cover_blocks_hard
find_seal_and_seal_stamp
pick_objects_in_order
pick_the_unhidden_block
press_button_keyframe
put_back_block_hard
rearrange_blocks_hard
reproduce_route

However, the current public robotwin_mem mixture contains:

cover_blocks_hard
put_back_block_hard
rearrange_blocks_hard
observe_and_pickup_hard
find_seal_and_seal_stamp
observe_and_pickup_object
reproduct_route
press_button_keyframe

We noticed:

• pick_objects_in_order is replaced by observe_and_pickup_hard;

• pick_the_unhidden_block is replaced by observe_and_pickup_object;

• reproduct_route may be a typo of reproduce_route.

The launcher seems to use robotwin_mem8, but this mixture contains author-specific absolute paths.

Could you please clarify:

1. Which dataset mixture was used for the released checkpoint?

2. Is the README eight-task list the correct benchmark training set?

3. Should robotwin_mem8 be considered the official training mixture?

---

5. QwenOFT naming clarification

We also noticed a possible naming inconsistency.

The paper describes EventVLA as based on QwenOFT, while the public framework entry uses:

framework:
  name: EventVLA

At the same time, some evaluation scripts still contain names such as:

weights_8tasks_pure_image_keyframe_memory_teacher_qwenoft.sh

Could you please clarify:

1. Is QwenOFT the original VLA backbone internally used by EventVLA?

2. Is framework.name=EventVLA the correct public entry point?

3. Is the qwenoft naming only historical/legacy naming?

---

Thank you again for releasing this work.

We appreciate any clarification about the intended training pipeline.
If needed, we can also provide:

• modified sampler implementation;

• detailed batch-slot / episode-slot logs;

• runtime memory statistics;

• additional reproduction curves.

Thank you very much!

[zhangzhengtu]

Hi, thank you very much for the detailed reproduction report and analysis.

May I ask whether you have already trained EventVLA on the RoboTwin-MeM dataset with the modified persistent episode-slot sampler? If so, did you obtain reasonably good success rates compared with the released checkpoint or the reported paper results?

It would be very helpful if you could share the per-task success rates, or at least the overall average success rate, under this setting. This can help us better understand whether preserving episode-to-batch-slot causality is indeed critical for reproducing the reported RoboTwin-MeM performance.

Thanks again for your careful investigation and for sharing these findings.