A question about inference speed

[wsBest]

Hello, author! I'm very glad to see such an excellent piece of work! I have a question about the inference speed. I would like to ask if it is supported to perform inference on a batch of images simultaneously. According to the demo code you provided, it seems that the reasoning speed for a pair of images is limited. Does it support batch inference ?

[RussRobin]

Thank you for your interest in our work.
Please see: #5

[wsBest]

ok！Thank you for your reply！

[Butter111561]

Hello, author! I'm very glad to see such an excellent piece of work! I have a question about the inference speed. I would like to ask if it is supported to perform inference on a batch of images simultaneously. According to the demo code you provided, it seems that the reasoning speed for a pair of images is limited. Does it support batch inference ?

I've solved the problem; most of the errors were due to legacy issues with bunny, especially padding. Below is a demo of batch inference. ` def to_generate(self, prompts, images1, ids=None, idx=None):
"""
:param prompts: list of str
:param images1: list of PIL.Image.Image or torch.Tensor
:param ids: list of str
:param idx: int
"""
batch_size = len(prompts)

    processed_images1 = []
    for img in images1:
        if isinstance(img, torch.Tensor):
            # 确保 tensor 维度正确并转为 PIL
            if img.ndim == 4:
                img = img.squeeze(0)
            img = tensor_to_pil(img)
        processed_images1.append(img)

    # 批量获取深度图
    images2 = depth_process(self.depth_model, processed_images1)

    # 统一进行 processor 处理，返回 [Batch, C, H, W] 的 Tensor
    image1_tensor = self.spatial_processor.preprocess(processed_images1, return_tensors='pt')['pixel_values'].to(
        self.device, dtype=torch.bfloat16)
    depth_tensor = self.spatial_processor.preprocess(images2, return_tensors='pt')['pixel_values'].to(self.device,
                                                                                                      dtype=torch.bfloat16)
    image_tensor = [image1_tensor, depth_tensor]
    self.spatial_model.get_vision_tower().to(self.device)

    batch_input_ids = []

    for prompt in prompts:
        text = f"A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image 1>\n<image 2>\n{prompt} ASSISTANT:"
        text_chunks = [tokenizer(chunk).input_ids for chunk in text.split('<image 1>\n<image 2>\n')]

        # 拼接特殊占位符
        seq = text_chunks[0] + [-201, -202] + text_chunks[1][offset_bos:]
        batch_input_ids.append(seq)

    max_len = max(len(seq) for seq in batch_input_ids)
    pad_token_id = tokenizer.pad_token_id if tokenizer.pad_token_id is not None else tokenizer.eos_token_id

    self.spatial_model.config.tokenizer_padding_side = "left"
    padded_input_ids = []
    attention_masks = []

    # 必须使用左侧填充，否则 `.generate()` 会产生乱码
    for seq in batch_input_ids:
        pad_len = max_len - len(seq)
        padded_input_ids.append([pad_token_id] * pad_len + seq)
        attention_masks.append([0] * pad_len + [1] * len(seq))

    input_ids = torch.tensor(padded_input_ids, dtype=torch.long).to(self.device)
    attention_mask = torch.tensor(attention_masks, dtype=torch.long).to(self.device)


    output_ids = self.spatial_model.generate(
        input_ids=input_ids,
        attention_mask=attention_mask,
        images=image_tensor,
        max_new_tokens=200,
        use_cache=True,  # 批量推理强烈建议开启 KV Cache 加速
        pad_token_id=pad_token_id
    )
    gen_ids = output_ids[:, input_ids.shape[1]:]


    output_texts = tokenizer.batch_decode(gen_ids, skip_special_tokens=True)

    output_texts = [t.strip() for t in output_texts]`