Update README.md

README.md

@@ -1,3 +1,111 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+base_model:
+- Qwen/Qwen2.5-3B-Instruct
+- google/siglip-so400m-patch14-384
+tags:
+- Transformers
+---
+
+
+# Huggingface Model Card
+
+IvyVL is a lightweight multimodal model with only 3B parameters. It accepts both image and text inputs to generate text outputs. Thanks to its lightweight design, it can be deployed on edge devices such as AI glasses and smartphones, offering low memory usage and high speed while maintaining strong performance on multimodal tasks. The model is built upon the [Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) language model, with [google/siglip-so400m-patch14-384](https://huggingface.co/google/siglip-so400m-patch14-384) serving as the vision encoder. 
+
+# Model Summary:
+
+*   Developed: Standford, CMU, AI Safeguard
+    
+*   Model type: Multi-modal model (image+text)
+    
+*   Language: Engligh and Chinese
+    
+*   License: Apache 2.0
+    
+*   Architecture: Based on LLaVA-One-Vision
+    
+
+# Evaluation：
+
+![image.jpeg](evaluation.jpeg)
+
+Most of the performance data comes from the VLMEvalKit leaderboard or the original papers. We conducted evaluations using VLMEvalKit. Due to differences in environments and the LLMs used for evaluation, there may be slight variations in performance.
+
+# How to use:
+
+```python
+# pip install git+https://github.com/LLaVA-VL/LLaVA-NeXT.git
+from llava.model.builder import load_pretrained_model
+from llava.mm_utils import process_images, tokenizer_image_token
+from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN
+from llava.conversation import conv_templates
+from PIL import Image
+import requests
+import copy
+import torch
+import warnings
+
+warnings.filterwarnings("ignore")
+
+pretrained = "AI-Safeguard/Ivy-VL"
+
+model_name = "llava_qwen"
+device = "cuda"
+device_map = "auto"
+tokenizer, model, image_processor, max_length = load_pretrained_model(pretrained, None, model_name, device_map=device_map)  # Add any other thing you want to pass in llava_model_args
+
+model.eval()
+
+# load image from url
+url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
+image = Image.open(requests.get(url, stream=True).raw)
+
+# load image from local environment
+# url = "./local_image.jpg"
+# image = Image.open(url)
+
+image_tensor = process_images([image], image_processor, model.config)
+image_tensor = [_image.to(dtype=torch.float16, device=device) for _image in image_tensor]
+
+conv_template = "qwen_1_5"  # Make sure you use correct chat template for different models
+question = DEFAULT_IMAGE_TOKEN + "\nWhat is shown in this image?"
+conv = copy.deepcopy(conv_templates[conv_template])
+conv.append_message(conv.roles[0], question)
+conv.append_message(conv.roles[1], None)
+prompt_question = conv.get_prompt()
+
+input_ids = tokenizer_image_token(prompt_question, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to(device)
+image_sizes = [image.size]
+
+cont = model.generate(
+    input_ids,
+    images=image_tensor,
+    image_sizes=image_sizes,
+    do_sample=False,
+    temperature=0,
+    max_new_tokens=4096,
+)
+
+text_outputs = tokenizer.batch_decode(cont, skip_special_tokens=True)
+
+print(text_outputs)
+```
+
+# Future Plan:
+
+*   We plan to release more versions of LLMs in different sizes.
+    
+*   We will focus on improving the performance of the video modality.
+    
+
+# Citation:
+
+```plaintext
+@misc{ivy2024ivy-vl,
+    title={LLaVA-NeXT: Improved reasoning, OCR, and world knowledge},
+    url={https://llava-vl.github.io/blog/2024-01-30-llava-next/},
+    author={Ivy Zhang,Jenny,Theresa and David Qiu},
+    month={December},
+    year={2024}
+}
+```