Update README.md

README.md

@@ -1,3 +1,132 @@
----
-license: mit
----
+---
+license: apache-2.0
+---
+<div align="center">
+<img src="assets/logo.png" alt="Lit-LLaMA" width="200"/>
+
+# HawkLlama
+[🤗**Huggingface**](https://huggingface.co/AIM-ZJU/HawkLlama_8b) | [🗂️**Github**](https://github.com/aim-uofa/VLModel)  | [📖**Technical Report**](assets/technical_report.pdf)
+
+Zhejiang University, China
+
+</div>
+
+
+This is the official implementation of HawkLlama, an open-source multimodal large language model designed for real-world vision and language understanding applications. Our model features the following highlights.
+
+1. HawkLlama-8B is constructed utilizing:
+    - [Llama3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B), the latest open-source large language model, trained on over 15 trillion tokens.
+    - [SigLIP](https://huggingface.co/google/siglip-so400m-patch14-384), an enhancement over CLIP employing sigmoid loss, which achieves superior performance in image recognition.
+    - An efficient vision-language connector, designed to capture high-resolution details without increasing the number of visual tokens, helps reduce the training overhead associated with high-resolution images.
+
+2. For model training, we utilize [Llava-Pretrain](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain) dataset for pretraining and a mixed dataset specifically curated for instruction tuning, which contains both multimodal and language-only data for supervised fine-tuning. 
+
+3. HawkLlama-8B is developed on [NeMo](https://github.com/NVIDIA/NeMo.git) framework, which facilitates 3D parallelism and offers scalability potential for future extension.
+
+Our model is open-source and reproducable. Please check our [technical report](assets/technical_report.pdf) for more details. 
+
+
+<!-- ## News
+
+[04/30] Llama3-LaMMly-8B is released, trained on a larger dataset, supporting higher resolution images, and also supporting Llama3 as the backbone. For LaMMly, we constructed a multimodal dataset containing 2.6M SFT sample, ensuring that LaMMly can achieve better generalization and improved image understanding. For more details, please refer to our [blog] and [technical report]. -->
+
+## Contents
+- [Setup](#setup)
+- [Model Weights](#model-weights)
+- [Inference](#inference)
+- [Evaluation](#evaluation)
+- [Demo](#demo)
+
+
+## Setup
+
+1. Create envoirment and activate it.
+```Shell
+conda create -n hawkllama python=3.10 -y
+conda activate hawkllama
+```
+
+2. Clone and install this repo.
+```
+git clone https://github.com/aim-uofa/VLModel.git
+cd VLModel
+pip install -e .
+pip install -e third_party/VLMEvalKit
+```
+
+## Model Weights
+
+Please refer to our [HuggingFace repository](https://huggingface.co/AIM-ZJU/HawkLlama_8b) to download the pretrained model weights.
+
+## Inference
+
+We provide an example code for inference.
+
+```Python
+import torch
+from PIL import Image
+from HawkLlama.model import LlavaNextProcessor, LlavaNextForConditionalGeneration
+from HawkLlama.utils.conversation import conv_llava_llama_3, DEFAULT_IMAGE_TOKEN
+
+processor = LlavaNextProcessor.from_pretrained("AIM-ZJU/HawkLlama_8b")
+
+model = LlavaNextForConditionalGeneration.from_pretrained("AIM-ZJU/HawkLlama_8b", torch_dtype=torch.bfloat16, low_cpu_mem_usage=True) 
+model.to("cuda:0")
+
+image_file = "assets/coin.png"
+image = Image.open(image_file).convert('RGB')
+
+prompt = "what coin is that?"
+prompt = DEFAULT_IMAGE_TOKEN + "\n" + prompt
+
+conversation = conv_llava_llama_3.copy()
+user_role_ind = 0
+bot_role_ind = 1
+conversation.append_message(conversation.roles[user_role_ind], prompt)
+conversation.append_message(conversation.roles[bot_role_ind], "")
+prompt = conversation.get_prompt()
+inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")
+inputs['pixel_values'] = inputs['pixel_values'].to(torch.bfloat16)
+output = model.generate(**inputs, eos_token_id=processor.tokenizer.eos_token_id, max_new_tokens=2048, do_sample=False, use_cache=True)
+
+print(processor.decode(output[0], skip_special_tokens=True))
+```
+
+## Evaluation
+
+Evaluate is modified based on the VLMEval codebase.
+
+``` bash
+# single gpu
+python third_party/VLMEvalKit/run.py --data MMBench_DEV_EN MMMU_DEV_VAL SEEDBench_IMG --model hawkllama_llama3_vlm --verbose
+# multi-gpus
+torchrun --nproc-per-node=8 third_party/VLMEvalKit/run.py --data MMBench_DEV_EN MMMU_DEV_VAL SEEDBench_IMG --model hawkllama_llama3_vlm --verbose
+```
+
+The results are shown below:
+
+| Benchmark       | Our MethodName | LLaVA-Llama3-v1.1 | LLaVA-Next |
+|-----------------|----------------|-------------------|------------|
+| MMMU val        | **37.8**       | 36.8              | 36.9       |
+| SEEDBench img   | **71.0**       | 70.1              | 70.0       |
+| MMBench-EN dev  | **70.6**       | 70.4              | 68.0       |
+| MMBench-CN dev  | **64.4**       | 64.2              | 60.6       |
+| CCBench         | **33.9**       | 31.6              | 24.7       |
+| AI2D test       | 65.6           | **70.0**          | 67.1       |
+| ScienceQA test  | **76.1**       | 72.9              | 70.4       |
+| HallusionBench  | 41.0           | **47.7**          | 35.2       |
+| MMStar          | 43.0           | **45.1**          | 38.1       |
+
+## Demo
+
+Welcome to try our [demo](http://115.236.57.99:30020/)!
+
+
+
+## Acknowledgements
+
+We express our appreciation to the following projects for their outstanding contributions in academia and code development: [LLaVA](https://github.com/haotian-liu/LLaVA), [NeMo](https://github.com/NVIDIA/NeMo), [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) and [xtuner](https://github.com/InternLM/xtuner).
+
+## License
+
+HawkLlama is released under the [Apache 2.0](https://github.com/Lightning-AI/lightning-llama/blob/main/LICENSE) license.