Update README.md

README.md

@@ -7,7 +7,7 @@ tags:
 - multimodal
 - aria
 ---
-<p align="center">
+<!-- <p align="center">
   <br>Aria</br>
 </p> 
 
@@ -16,35 +16,113 @@ tags:
  ·🖤 <a href="https://huggingface.co" target="_blank">GitHub</a>  💜 <a href="https://huggingface.co" target="_blank">Discord</a>
 · 💙 <a href="https://huggingface.co" target="_blank">Twitter</a>
 </p> 
-
-# Highlights
-
+ -->
+# Aria Model Card
+<!-- 
 - Aria is the **first open multimodal native MoE** model, capable of seamlessly handling various input modalities within a MoE architecture.
 - Aria performs **on par with GPT-4o mini and Gemini 1.5 Flash** across a range of multimodal tasks while maintaining strong performance on **text**-only tasks.
 - Compared to similar or even larger models, Aria boasts **faster speeds** and **lower costs**. This high efficiency stems from its ability to activate only 3.9B parameters during inference – the **fewest** among models with comparable performance.
+ -->
+## Key features
 
-# Key features
-
-- **Robust multimodal understanding**: Aria processes various input modalities, including video, images, code, and text. It demonstrates strong performance across diverse downstream tasks such as long-context video and image understanding and OCR. Moreover, it excels in instruction following.
-- **Flexible image handling**: Aria supports variable image sizes and aspect ratios while maintaining high quality.
-- **Extended context capacity**: Aria can manage multiple images within a long context window of 64k tokens.
-- **Advanced text understanding**: Aria demonstrates competitive performance across language and coding tasks.
+- **SoTA Multimodal Native Performance**: Aria achieves strong performance on a wide range of multimodal, language, and coding tasks. It is superior in video and document understanding.   
+- **Lightweight and Fast**: Aria is a mixture-of-expert model with 3.9B activated parameters per token. It efficently encodes visual input of variable sizes and aspect ratios.  
+- **Long Multimodal Context Window**: Aria supports multimodal input of up to 64K tokens. It can caption a 256-frame video in 10 seconds.
 
-# Model Info
+<!-- # Model Info
 
 | Model  | Download  | Parameter | Context Length |
 | :---- | :------- | :------------ | :------ |
-| Aria | < HF link - TBD> | • Activation: 3.9B (3.5B MoE + 0.4B Visual Encoder) <br> • Total: 25.3B | 64K           |
-
-# Benchmark
-
-
-
-# Quick Start
-
-
-
-
-# License
-
-This repo is released under the Apache 2.0 License.
+| Aria | < HF link - TBD> | • Activation: 3.9B (3.5B MoE + 0.4B Visual Encoder) <br> • Total: 25.3B | 64K           | -->
+
+## Benchmark
+| Category                            | Benchmark         | Aria  | Pixtral 12B | Llama3.2 11B | GPT-4o mini | GPT-4o | Gemini-1.5 Flash | Gemini-1.5 Pro |
+|-------------------------------------|-------------------|-------|-------------|--------------|-------------|--------|------------------|----------------|
+| **Knowledge (Multimodal)**          | MMMU              | 54.9  | 52.5        | 49.6         | 59.4        | 69.1   | 56.1             | 62.2           |
+| **Math (Multimodal)**               | MathVista         | 66.1  | 58.0        | 51.5         | -           | 54.7   | 63.8             | 58.4           |
+| **Document**                        | DocQA             | 92.6  | 90.7        | 84.4         | -           | 92.8   | 89.9             | 93.1           |
+| **Chart**                           | ChartQA           | 86.4  | 81.8        | 78.7         | -           | 85.7   | 85.4             | 87.2           |
+| **Scene Text**                      | TextVQA           | 81.1  | -           | 78.2         | -           | -      | 78.7             | 78.7           |
+| **General Visual QA**               | MMBench-1.1       | 80.3  | -           | -            | 76.0        | 82.2   | -                | 73.9           |
+| **Video Understanding**             | LongVideoBench    | 66.6  | 47.4        | 45.7         | 58.8        | 66.7   | 62.4             | 64.4           |
+| **Knowledge (Language)**            | MMLU (5-shot)     | 73.3  | 69.2        | 69.4         | -           | 89.1   | 78.9             | 85.9           |
+| **Math (Language)**                 | MATH              | 50.8  | 48.1        | 51.9         | 70.2        | 76.6   | -                | -              |
+| **Reasoning (Language)**            | ARC Challenge     | 91.0  | -           | 83.4         | 96.4        | 96.7   | -                | -              |
+| **Coding**                          | HumanEval         | 73.2  | 72.0        | 72.6         | 87.2        | 90.2   | 74.3             | 84.1           |
+
+
+## Quick Start
+### Installation
+```
+pip install git+github.com/rhymes-ai/Aria.git
+pip install flash-attn --no-build-isolation
+```
+
+### Inference
+
+Aria has 25.3B total parameters, it can be loaded in one A100 (80GB) GPU with bfloat16 precision.
+
+Here is a code snippet to show you how to use Aria.
+
+```python
+import requests
+import torch
+from PIL import Image
+from transformers import AutoModelForCausalLM, AutoProcessor
+
+model_id_or_path = "rhymes-ai/Aria"
+
+model = AutoModelForCausalLM.from_pretrained(model_id_or_path, device_map="auto", torch_dtype=torch.bfloat16, trust_remote_code=True)
+
+processor = AutoProcessor.from_pretrained(model_id_or_path, trust_remote_code=True)
+
+image_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png"
+
+image = Image.open(requests.get(image_path, stream=True).raw)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"text": None, "type": "image"},
+            {"text": "what is the image?", "type": "text"},
+        ],
+    }
+]
+
+text = processor.apply_chat_template(messages, add_generation_prompt=True)
+inputs = processor(text=text, images=image, return_tensors="pt")
+inputs["pixel_values"] = inputs["pixel_values"].to(model.dtype)
+inputs = {k: v.to(model.device) for k, v in inputs.items()}
+
+with torch.inference_mode(), torch.cuda.amp.autocast(dtype=torch.bfloat16):
+    output = model.generate(
+        **inputs,
+        max_new_tokens=500,
+        stop_strings=["<|im_end|>"],
+        tokenizer=processor.tokenizer,
+        do_sample=True,
+        temperature=0.9,
+    )
+    output_ids = output[0][inputs["input_ids"].shape[1]:]
+    result = processor.decode(output_ids, skip_special_tokens=True)
+
+print(result)
+```
+
+### Advanced Inference and Fine-tuning
+We provide a [codebase](https://github.com/rhymes-ai/Aria) for more advanced usage of Aria,
+including vllm inference, cookbooks, and fine-tuning on custom datasets.
+
+
+
+## Citation
+If you find our work helpful, please consider citing.
+```
+@article{aria,
+  title={},
+  author={},
+  year={2024},
+  journal={}
+}
+```