Batch inference
Does MiniCPM support batch inference using the transformers library? Inference using only 1 image with question is very slow, I want to run batch inference, where each batch it's N images with N corresponding questions.
Batch inference is supported. The msgs parameter could be a list of msgs, like:
res = model.chat(
image=None,
msgs=[msgs, msgs],
tokenizer=tokenizer,
sampling=True,
stream=False
)
where msgs is as the same as that in the example.
Inputing a list of None is too weird, we update the code and disable image parameter in batch inference. Please check the update.
@epishchik
Now I'm really confused. I read your documentation more carefully, and as I understand it, you always use image=None and include images in the msgs parameter. But I have already experimented (with the version downloaded before this whole discussion) with the code below and it works (I don't get errors and my VRAM usage increases when I increase the number of images in the list).
from PIL import Image
images = [
Image.open("img1.jpg"),
Image.open("img2.jpg")
]
questions = [
[{"role": "user", "content": "Question for img1.jpg"}],
[{"role": "user", "content": "Question for img2.jpg"}]
]
res = model.chat(
image=images,
msgs=msgs,
tokenizer=tokenizer
)
print(res)
Will I get incorrect output using this code and is it really batch inference, it seems to work in batch mode but I'm confused because your documentation and code in modelling_minicpmv.py says that I should pass image=None and include images in the msgs parameter to use batch inference.
Actually we want you to construct messages more carefully when using batch inference. When you send image to the model, it will add the image to the begin of the messages.
So you can run with the follows instead.
questions = [
[{"role": "user", "content": [Image.open("img1.jpg"), "Question for img1.jpg"]}],
[{"role": "user", "content": [Image.open("img1.jpg"), "Question for img2.jpg"]}]
]
Based on the possibility user may send a list of images and we don't know whether it is batched input or multi-images input. So we disabled this choice and recommend to use msgs. Sry for the inconvenience.
Got it, tnx for the explanation!