LanguageBind commited on
Commit
7208cfe
โ€ข
1 Parent(s): 532e5ac

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +213 -0
README.md CHANGED
@@ -1,3 +1,216 @@
1
  ---
2
  license: apache-2.0
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
  ---
4
+
5
+ <p align="center">
6
+ <img src="https://s11.ax1x.com/2023/12/28/piqvDMV.png" width="250" style="margin-bottom: 0.2;"/>
7
+ <p>
8
+ <h2 align="center"> <a href="https://arxiv.org/abs/2401.15947">MoE-LLaVA: Mixture of Experts for Large Vision-Language Models</a></h2>
9
+ <h5 align="center"> If you like our project, please give us a star โญ on GitHub for latest update. </h2>
10
+
11
+ <h5 align="center">
12
+
13
+
14
+
15
+ </h5>
16
+
17
+
18
+ ## ๐Ÿ“ฐ News
19
+ * **[2024.01.30]** The [paper](https://arxiv.org/abs/2401.15947) is released.
20
+ * **[2024.01.27]** ๐Ÿค—[Hugging Face demo](https://huggingface.co/spaces/LanguageBind/MoE-LLaVA) and **all codes & datasets** are available now! Welcome to **watch** ๐Ÿ‘€ this repository for the latest updates.
21
+
22
+ ## ๐Ÿ˜ฎ Highlights
23
+
24
+ MoE-LLaVA shows excellent performance in multi-modal learning.
25
+
26
+ ### ๐Ÿ”ฅ High performance, but with fewer parameters
27
+ - with just **3B sparsely activated parameters**, MoE-LLaVA demonstrates performance comparable to the LLaVA-1.5-7B on various visual understanding datasets and even surpasses the LLaVA-1.5-13B in object hallucination benchmarks.
28
+
29
+
30
+ ### ๐Ÿš€ Simple baseline, learning multi-modal interactions with sparse pathways.
31
+ - With the addition of **a simple MoE tuning stage**, we can complete the training of MoE-LLaVA on **8 V100 GPUs** within 2 days.
32
+
33
+
34
+
35
+ ## ๐Ÿค— Demo
36
+
37
+ ### Gradio Web UI
38
+
39
+ Highly recommend trying out our web demo by the following command, which incorporates all features currently supported by MoE-LLaVA. We also provide [online demo](https://huggingface.co/spaces/LanguageBind/MoE-LLaVA) in Huggingface Spaces.
40
+ ```bash
41
+ # use phi2
42
+ deepspeed --include localhost:0 moellava/serve/gradio_web_server.py --model-path "LanguageBind/MoE-LLaVA-Phi2-2.7B-4e"
43
+ # use qwen
44
+ deepspeed --include localhost:0 moellava/serve/gradio_web_server.py --model-path "LanguageBind/MoE-LLaVA-Qwen-1.8B-4e"
45
+ # use stablelm
46
+ deepspeed --include localhost:0 moellava/serve/gradio_web_server.py --model-path "LanguageBind/MoE-LLaVA-StableLM-1.6B-4e"
47
+ ```
48
+
49
+
50
+
51
+ ### CLI Inference
52
+
53
+ ```bash
54
+ # use phi2
55
+ deepspeed --include localhost:0 moellava/serve/cli.py --model-path "LanguageBind/MoE-LLaVA-Phi2-2.7B-4e" --image-file "image.jpg"
56
+ # use qwen
57
+ deepspeed --include localhost:0 moellava/serve/cli.py --model-path "LanguageBind/MoE-LLaVA-Qwen-1.8B-4e" --image-file "image.jpg"
58
+ # use stablelm
59
+ deepspeed --include localhost:0 moellava/serve/cli.py --model-path "LanguageBind/MoE-LLaVA-StableLM-1.6B-4e" --image-file "image.jpg"
60
+ ```
61
+
62
+
63
+ ## ๐Ÿณ Model Zoo
64
+
65
+ | Model | LLM | Checkpoint | Avg | VQAv2 | GQA | VizWiz | SQA | T-VQA | POPE | MM-Bench| LLaVA-Bench-Wild | MM-Vet |
66
+ |----------|-----------|-----------|---|---|---|---|---|---|---|---|---|---|
67
+ | MoE-LLaVA-1.6Bร—4-Top2 | 1.6B | [LanguageBind/MoE-LLaVA-StableLM-1.6B-4e](https://huggingface.co/LanguageBind/MoE-LLaVA-StableLM-1.6B-4e) | 60.0 | 76.0 | 60.4 | 37.2 | 62.6 | 47.8 | 84.3 | 59.4 | 85.9 | 26.1 |
68
+ | MoE-LLaVA-1.8Bร—4-Top2 | 1.8B | [LanguageBind/MoE-LLaVA-Qwen-1.8B-4e](https://huggingface.co/LanguageBind/MoE-LLaVA-Qwen-1.8B-4e) | 60.2 | 76.2 | 61.5 | 32.6 | 63.1 | 48.0 | 87.0 | 59.6 | 88.7 | 25.3 |
69
+ | MoE-LLaVA-2.7Bร—4-Top2 | 2.7B | [LanguageBind/MoE-LLaVA-Phi2-2.7B-4e](https://huggingface.co/LanguageBind/MoE-LLaVA-Phi2-2.7B-4e) | 63.9 | 77.1 | 61.1 | 43.4 | 68.7 | 50.2 | 85.0 | 65.5 | 93.2 | 31.1 |
70
+
71
+ <!--
72
+ | LLaVA-1.5 | 7B | [liuhaotian/llava-v1.5-7b](https://huggingface.co/liuhaotian/llava-v1.5-7b) | 62.0 | 78.5 | 62.0 | 50.0 | 66.8 | 58.2 | 85.9 | 64.3 | 31.1 |
73
+ | LLaVA-1.5 | 13B | [liuhaotian/llava-v1.5-13b](https://huggingface.co/liuhaotian/llava-v1.5-13b) | 64.9 | 80.0 | 63.3 | 53.6 | 71.6 | 61.3 | 85.9 | 67.7 | 36.1 |
74
+ -->
75
+
76
+ ## โš™๏ธ Requirements and Installation
77
+ * Python >= 3.10
78
+ * Pytorch == 2.0.1
79
+ * CUDA Version >= 11.7
80
+ * **Transformers == 4.36.2**
81
+ * **Tokenizers==0.15.1**
82
+ * Install required packages:
83
+ ```bash
84
+ git clone https://github.com/PKU-YuanGroup/MoE-LLaVA
85
+ cd MoE-LLaVA
86
+ conda create -n moellava python=3.10 -y
87
+ conda activate moellava
88
+ pip install --upgrade pip # enable PEP 660 support
89
+ pip install -e .
90
+ pip install -e ".[train]"
91
+ pip install flash-attn --no-build-isolation
92
+
93
+ # Below are optional. For Qwen model.
94
+ git clone https://github.com/Dao-AILab/flash-attention
95
+ cd flash-attention && pip install .
96
+ # Below are optional. Installing them might be slow.
97
+ # pip install csrc/layer_norm
98
+ # If the version of flash-attn is higher than 2.1.1, the following is not needed.
99
+ # pip install csrc/rotary
100
+ ```
101
+
102
+ ## ๐Ÿ—๏ธ Training & Validating
103
+ The training & validating instruction is in [TRAIN.md](docs/TRAIN.md) & [EVAL.md](docs/EVAL.md).
104
+
105
+ ## ๐Ÿ’ก Customizing your MoE-LLaVA
106
+ The instruction is in [CUSTOM.md](docs/CUSTOM.md).
107
+
108
+ ## ๐Ÿ˜ Visualization
109
+ The instruction is in [VISUALIZATION.md](docs/VISUALIZATION.md).
110
+
111
+ ## ๐Ÿค– API
112
+ **We open source all codes.** If you want to load the model (e.g. ```LanguageBind/MoE-LLaVA```) on local, you can use the following code snippets.
113
+
114
+ **Using the following command to run the code.**
115
+
116
+ ```bash
117
+ deepspeed predict.py
118
+ ```
119
+
120
+ ```python
121
+ import torch
122
+ from moellava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN
123
+ from moellava.conversation import conv_templates, SeparatorStyle
124
+ from moellava.model.builder import load_pretrained_model
125
+ from moellava.utils import disable_torch_init
126
+ from moellava.mm_utils import tokenizer_image_token, get_model_name_from_path, KeywordsStoppingCriteria
127
+
128
+ def main():
129
+ disable_torch_init()
130
+ image = 'moellava/serve/examples/extreme_ironing.jpg'
131
+ inp = 'What is unusual about this image?'
132
+ model_path = 'LanguageBind/MoE-LLaVA-Phi2-2.7B-4e' # LanguageBind/MoE-LLaVA-Qwen-1.8B-4e or LanguageBind/MoE-LLaVA-StableLM-1.6B-4e
133
+ device = 'cuda'
134
+ load_4bit, load_8bit = False, False # FIXME: Deepspeed support 4bit or 8bit?
135
+ model_name = get_model_name_from_path(model_path)
136
+ tokenizer, model, processor, context_len = load_pretrained_model(model_path, None, model_name, load_8bit, load_4bit, device=device)
137
+ image_processor = processor['image']
138
+ conv_mode = "phi" # qwen or stablelm
139
+ conv = conv_templates[conv_mode].copy()
140
+ roles = conv.roles
141
+ image_tensor = image_processor.preprocess(image, return_tensors='pt')['pixel_values'].to(model.device, dtype=torch.float16)
142
+
143
+ print(f"{roles[1]}: {inp}")
144
+ inp = DEFAULT_IMAGE_TOKEN + '\n' + inp
145
+ conv.append_message(conv.roles[0], inp)
146
+ conv.append_message(conv.roles[1], None)
147
+ prompt = conv.get_prompt()
148
+ input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).cuda()
149
+ stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2
150
+ keywords = [stop_str]
151
+ stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids)
152
+
153
+ with torch.inference_mode():
154
+ output_ids = model.generate(
155
+ input_ids,
156
+ images=image_tensor,
157
+ do_sample=True,
158
+ temperature=0.2,
159
+ max_new_tokens=1024,
160
+ use_cache=True,
161
+ stopping_criteria=[stopping_criteria])
162
+
163
+ outputs = tokenizer.decode(output_ids[0, input_ids.shape[1]:], skip_special_tokens=True).strip()
164
+ print(outputs)
165
+
166
+ if __name__ == '__main__':
167
+ main()
168
+ ```
169
+
170
+ ## ๐Ÿ™Œ Related Projects
171
+ * [Video-LLaVA](https://github.com/PKU-YuanGroup/Video-LLaVA) This framework empowers the model to efficiently utilize the united visual tokens.
172
+ * [LanguageBind](https://github.com/PKU-YuanGroup/LanguageBind) An open source five modalities language-based retrieval framework.
173
+
174
+ ## ๐Ÿ‘ Acknowledgement
175
+ * [LLaVA](https://github.com/haotian-liu/LLaVA) The codebase we built upon and it is an efficient large language and vision assistant.
176
+
177
+ ## ๐Ÿ”’ License
178
+ * The majority of this project is released under the Apache 2.0 license as found in the [LICENSE](https://github.com/PKU-YuanGroup/MoE-LLaVA/blob/main/LICENSE) file.
179
+ * The service is a research preview intended for non-commercial use only, subject to the model [License](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md) of LLaMA, [Terms of Use](https://openai.com/policies/terms-of-use) of the data generated by OpenAI, and [Privacy Practices](https://chrome.google.com/webstore/detail/sharegpt-share-your-chatg/daiacboceoaocpibfodeljbdfacokfjb) of ShareGPT. Please contact us if you find any potential violation.
180
+
181
+
182
+
183
+ ## โœ๏ธ Citation
184
+ If you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil:.
185
+
186
+ ```BibTeX
187
+ @misc{lin2024moellava,
188
+ title={MoE-LLaVA: Mixture of Experts for Large Vision-Language Models},
189
+ author={Bin Lin and Zhenyu Tang and Yang Ye and Jiaxi Cui and Bin Zhu and Peng Jin and Junwu Zhang and Munan Ning and Li Yuan},
190
+ year={2024},
191
+ eprint={2401.15947},
192
+ archivePrefix={arXiv},
193
+ primaryClass={cs.CV}
194
+ }
195
+ ```
196
+
197
+ ```BibTeX
198
+ @article{lin2023video,
199
+ title={Video-LLaVA: Learning United Visual Representation by Alignment Before Projection},
200
+ author={Lin, Bin and Zhu, Bin and Ye, Yang and Ning, Munan and Jin, Peng and Yuan, Li},
201
+ journal={arXiv preprint arXiv:2311.10122},
202
+ year={2023}
203
+ }
204
+ ```
205
+
206
+
207
+
208
+ ## โœจ Star History
209
+ [![Star History](https://api.star-history.com/svg?repos=PKU-YuanGroup/MoE-LLaVA&type=Date)](https://star-history.com/#PKU-YuanGroup/MoE-LLaVA&Date)
210
+
211
+
212
+ ## ๐Ÿค Contributors
213
+
214
+ <a href="https://github.com/PKU-YuanGroup/MoE-LLaVA/graphs/contributors">
215
+ <img src="https://contrib.rocks/image?repo=PKU-YuanGroup/MoE-LLaVA" />
216
+ </a>