Duplicate from deepseek-ai/deepseek-vl2-tiny

Co-authored-by: Wen Liu <doubility123@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,142 @@
+---
+license: other
+license_name: deepseek
+license_link: https://github.com/deepseek-ai/DeepSeek-LLM/blob/HEAD/LICENSE-MODEL
+pipeline_tag: image-text-to-text
+library_name: transformers
+---
+
+## 1. Introduction
+
+Introducing DeepSeek-VL2, an advanced series of large Mixture-of-Experts (MoE) Vision-Language Models that significantly improves upon its predecessor, DeepSeek-VL. DeepSeek-VL2 demonstrates superior capabilities across various tasks, including but not limited to visual question answering, optical character recognition,  document/table/chart understanding, and visual grounding. Our model series is composed of three variants: DeepSeek-VL2-Tiny, DeepSeek-VL2-Small and DeepSeek-VL2, with 1.0B, 2.8B and 4.5B activated parameters respectively.
+DeepSeek-VL2 achieves competitive or state-of-the-art performance with similar or fewer activated parameters compared to existing open-source dense and MoE-based models.
+
+
+[DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://arxiv.org/abs/2412.10302)
+
+[**Github Repository**](https://github.com/deepseek-ai/DeepSeek-VL2)
+
+
+Zhiyu Wu*, Xiaokang Chen*, Zizheng Pan*, Xingchao Liu*, Wen Liu**, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, Chong Ruan*** (* Equal Contribution, ** Project Lead, *** Corresponding author)
+
+![](https://github.com/deepseek-ai/DeepSeek-VL2/tree/main/images/vl2_teaser.jpeg)
+
+
+### 2. Model Summary
+
+DeepSeek-VL2-tiny is built on DeepSeekMoE-3B (total activated parameters are 1.0B).
+
+
+## 3. Quick Start
+
+### Installation
+
+On the basis of `Python >= 3.8` environment, install the necessary dependencies by running the following command:
+
+```shell
+pip install -e .
+```
+
+### Notifications
+1. We suggest to use a temperature T <= 0.7 when sampling. We observe a larger temperature decreases the generation quality.
+2. To keep the number of tokens managable in the context window, we apply dynamic tiling strategy to <=2 images. When there are >=3 images, we directly pad the images to 384*384 as inputs without tiling.
+3. The main difference between DeepSeek-VL2-Tiny, DeepSeek-VL2-Small and DeepSeek-VL2 is the base LLM.
+
+### Simple Inference Example
+
+```python
+import torch
+from transformers import AutoModelForCausalLM
+
+from deepseek_vl.models import DeepseekVLV2Processor, DeepseekVLV2ForCausalLM
+from deepseek_vl.utils.io import load_pil_images
+
+
+# specify the path to the model
+model_path = "deepseek-ai/deepseek-vl2-small"
+vl_chat_processor: DeepseekVLV2Processor = DeepseekVLV2Processor.from_pretrained(model_path)
+tokenizer = vl_chat_processor.tokenizer
+
+vl_gpt: DeepseekVLV2ForCausalLM = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
+vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()
+
+## single image conversation example
+conversation = [
+    {
+        "role": "<|User|>",
+        "content": "<image>\n<|ref|>The giraffe at the back.<|/ref|>.",
+        "images": ["./images/visual_grounding.jpeg"],
+    },
+    {"role": "<|Assistant|>", "content": ""},
+]
+
+## multiple images (or in-context learning) conversation example
+# conversation = [
+#     {
+#         "role": "User",
+#         "content": "<image_placeholder>A dog wearing nothing in the foreground, "
+#                    "<image_placeholder>a dog wearing a santa hat, "
+#                    "<image_placeholder>a dog wearing a wizard outfit, and "
+#                    "<image_placeholder>what's the dog wearing?",
+#         "images": [
+#             "images/dog_a.png",
+#             "images/dog_b.png",
+#             "images/dog_c.png",
+#             "images/dog_d.png",
+#         ],
+#     },
+#     {"role": "Assistant", "content": ""}
+# ]
+
+# load images and prepare for inputs
+pil_images = load_pil_images(conversation)
+prepare_inputs = vl_chat_processor(
+    conversations=conversation,
+    images=pil_images,
+    force_batchify=True,
+    system_prompt=""
+).to(vl_gpt.device)
+
+# run image encoder to get the image embeddings
+inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)
+
+# run the model to get the response
+outputs = vl_gpt.language_model.generate(
+    inputs_embeds=inputs_embeds,
+    attention_mask=prepare_inputs.attention_mask,
+    pad_token_id=tokenizer.eos_token_id,
+    bos_token_id=tokenizer.bos_token_id,
+    eos_token_id=tokenizer.eos_token_id,
+    max_new_tokens=512,
+    do_sample=False,
+    use_cache=True
+)
+
+answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)
+print(f"{prepare_inputs['sft_format'][0]}", answer)
+```
+
+### Gradio Demo (TODO)
+
+
+## 4. License
+
+This code repository is licensed under [MIT License](./LICENSE-CODE). The use of DeepSeek-VL2 models is subject to [DeepSeek Model License](./LICENSE-MODEL). DeepSeek-VL2 series supports commercial use.
+
+## 5. Citation
+
+```
+@misc{wu2024deepseekvl2mixtureofexpertsvisionlanguagemodels,
+      title={DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding}, 
+      author={Zhiyu Wu and Xiaokang Chen and Zizheng Pan and Xingchao Liu and Wen Liu and Damai Dai and Huazuo Gao and Yiyang Ma and Chengyue Wu and Bingxuan Wang and Zhenda Xie and Yu Wu and Kai Hu and Jiawei Wang and Yaofeng Sun and Yukun Li and Yishi Piao and Kang Guan and Aixin Liu and Xin Xie and Yuxiang You and Kai Dong and Xingkai Yu and Haowei Zhang and Liang Zhao and Yisong Wang and Chong Ruan},
+      year={2024},
+      eprint={2412.10302},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV},
+      url={https://arxiv.org/abs/2412.10302}, 
+}
+```
+
+## 6. Contact
+
+If you have any questions, please raise an issue or contact us at [service@deepseek.com](mailto:service@deepseek.com).

config.json

@@ -0,0 +1,150 @@
+{
+  "candidate_resolutions": [
+    [
+      384,
+      384
+    ],
+    [
+      384,
+      768
+    ],
+    [
+      768,
+      384
+    ],
+    [
+      384,
+      1152
+    ],
+    [
+      1152,
+      384
+    ],
+    [
+      384,
+      1536
+    ],
+    [
+      1536,
+      384
+    ],
+    [
+      768,
+      768
+    ],
+    [
+      384,
+      1920
+    ],
+    [
+      1920,
+      384
+    ],
+    [
+      384,
+      2304
+    ],
+    [
+      2304,
+      384
+    ],
+    [
+      768,
+      1152
+    ],
+    [
+      1152,
+      768
+    ],
+    [
+      384,
+      2688
+    ],
+    [
+      2688,
+      384
+    ],
+    [
+      384,
+      3072
+    ],
+    [
+      3072,
+      384
+    ],
+    [
+      768,
+      1536
+    ],
+    [
+      1536,
+      768
+    ],
+    [
+      384,
+      3456
+    ],
+    [
+      3456,
+      384
+    ],
+    [
+      1152,
+      1152
+    ]
+  ],
+  "global_view_pos": "head",
+  "language_config": {
+    "architectures": [
+      "DeepseekV2ForCausalLM"
+    ],
+    "auto_map": {
+      "AutoConfig": "configuration_deepseek.DeepseekV2Config",
+      "AutoModel": "modeling_deepseek.DeepseekV2Model",
+      "AutoModelForCausalLM": "modeling_deepseek.DeepseekV2ForCausalLM"
+    },
+    "bos_token_id": 0,
+    "eos_token_id": 1,
+    "first_k_dense_replace": 1,
+    "hidden_size": 1280,
+    "intermediate_size": 6848,
+    "kv_lora_rank": null,
+    "lm_head": true,
+    "max_position_embeddings": 4096,
+    "model_type": "deepseek_v2",
+    "moe_intermediate_size": 896,
+    "n_group": 1,
+    "n_routed_experts": 64,
+    "n_shared_experts": 2,
+    "num_attention_heads": 10,
+    "num_experts_per_tok": 6,
+    "num_hidden_layers": 12,
+    "num_key_value_heads": 10,
+    "q_lora_rank": null,
+    "qk_nope_head_dim": 0,
+    "qk_rope_head_dim": 0,
+    "rm_head": false,
+    "topk_group": 1,
+    "topk_method": "greedy",
+    "torch_dtype": "bfloat16",
+    "use_mla": false,
+    "v_head_dim": 0,
+    "vocab_size": 129280
+  },
+  "model_type": "deepseek_vl_v2",
+  "projector_config": {
+    "model_type": "mlp_projector",
+    "n_embed": 1280
+  },
+  "tile_tag": "2D",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.38.2",
+  "vision_config": {
+    "layers": 27,
+    "mlp_ratio": 3.7362,
+    "model_name": "siglip_so400m_patch14_384",
+    "model_type": "vision",
+    "patch_size": 14,
+    "width": 1152
+  }
+}

model-00001-of-000001.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc1e5047280253e224b299677bea16b7960c2436ec676d28911ebd9de3bb0074
+size 6741334208

processor_config.json

@@ -0,0 +1,116 @@
+{
+  "add_special_token": false,
+  "candidate_resolutions": [
+    [
+      384,
+      384
+    ],
+    [
+      384,
+      768
+    ],
+    [
+      768,
+      384
+    ],
+    [
+      384,
+      1152
+    ],
+    [
+      1152,
+      384
+    ],
+    [
+      384,
+      1536
+    ],
+    [
+      1536,
+      384
+    ],
+    [
+      768,
+      768
+    ],
+    [
+      384,
+      1920
+    ],
+    [
+      1920,
+      384
+    ],
+    [
+      384,
+      2304
+    ],
+    [
+      2304,
+      384
+    ],
+    [
+      768,
+      1152
+    ],
+    [
+      1152,
+      768
+    ],
+    [
+      384,
+      2688
+    ],
+    [
+      2688,
+      384
+    ],
+    [
+      384,
+      3072
+    ],
+    [
+      3072,
+      384
+    ],
+    [
+      768,
+      1536
+    ],
+    [
+      1536,
+      768
+    ],
+    [
+      384,
+      3456
+    ],
+    [
+      3456,
+      384
+    ],
+    [
+      1152,
+      1152
+    ]
+  ],
+  "downsample_ratio": 2,
+  "ignore_id": -100,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_token": "<image>",
+  "mask_prompt": false,
+  "normalize": true,
+  "pad_token": "<\uff5c\u2581pad\u2581\uff5c>",
+  "patch_size": 14,
+  "processor_class": "DeepseekVLV2Processor",
+  "sft_format": "deepseek"
+}

special_tokens_map.json

@@ -0,0 +1,39 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<|User|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<|Assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": {
+    "content": "<｜begin▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<｜end▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<｜▁pad▁｜>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}