llava-hf/llava-interleave-qwen-7b-hf · pass multi-images to the processor

Jul 19

Your README only provides examples for interleaved text-image prompts, but how to pass multi-images to the processor.

image_file = "http://images.cocodataset.org/val2017/000000039769.jpg"
raw_image = Image.open(requests.get(image_file, stream=True).raw)
inputs = processor(prompt, raw_image, return_tensors='pt').to(0, torch.float16)

RaushanTurganbay

Llava Hugging Face org Jul 20

@Benjamin02 hey, if you want to use multiple images in one prompt, you can do as follows:


# chat template in interleaved format work same as in sampling videos. Just pass in as many images you want for a prompt
conversation = [
    {

      "role": "user",
      "content": [
          {"type": "image"},
          {"type": "image"},
          {"type": "text", "text": "What is the difference between these two images?"},
        ],
    },
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(prompt, [image_1,  image_2], return_tensors='pt')

Same goes for batched inputs, just pass in images as a list in the same order as they go in prompts. For ex:

# prompt1 has 1 image and prompt2 has 2 images
# image_1 will be used in prompt1, while images 2 and 3 are used for prompt2
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(text=[prompt1, prompt2], images=[image_1,  image_2, image_1], return_tensors='pt')

prasb

Sep 5

Hi, is it possible to pass 2 images for 1 single prompt using transformers.pipeline?

RaushanTurganbay

Llava Hugging Face org Sep 6

@prasb unfortunately no, we are working on adding a proper pipeline for VLMs. Current pipelines were designed mostly for image captioning so they expect one image per prompt