Update README.md

README.md

@@ -7,9 +7,9 @@ pipeline_tag: image-text-to-text
 
 # llama-3.1-8B-vision-378
 
-THIS IS A SLAPPED-TOGETHER RELEASE; IF IT WORKS, IT IS A MIRACLE OF LATENT SPACE
+Projection module trained to add vision capabilties to Llama 3 using SigLIP, then applied to LLama-3.1-8B-Instruct. Built by [@yeswondwerr](https://x.com/yeswondwerr) and [@qtnx_](https://x.com/qtnx_).
 
-## usage
+## Usage
 
 ```python
 import torch
@@ -38,6 +38,46 @@ print(
 )
 ```
 
+## 4-bit quantization
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from transformers import BitsAndBytesConfig
+import requests
+from io import BytesIO
+
+
+url = "https://huggingface.co/qresearch/llama-3-vision-alpha-hf/resolve/main/assets/demo-2.jpg"
+response = requests.get(url)
+image = Image.open(BytesIO(response.content))
+
+bnb_cfg = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_compute_dtype=torch.float16,
+    llm_int8_skip_modules=["mm_projector", "vision_model"],
+)
+
+model = AutoModelForCausalLM.from_pretrained(
+    "qresearch/llama-3.1-8B-vision-378",
+    trust_remote_code=True,
+    torch_dtype=torch.float16,
+    quantization_config=bnb_cfg,
+)
+
+tokenizer = AutoTokenizer.from_pretrained(
+    "qresearch/llama-3.1-8B-vision-378",
+    use_fast=True,
+)
+
+print(
+    model.answer_question(
+        image, "Briefly describe the image", tokenizer, max_new_tokens=128, do_sample=True, temperature=0.3
+    ),
+)
+```
+
 ```
                                        .x+=:.                                                             
                                       z`    ^%                                                  .uef^"