Created Readme

README.md

@@ -0,0 +1,129 @@
+---
+library_name: transformers
+license: apache-2.0
+pipeline_tag: text-generation
+tags:
+- 4bit
+- AWQ
+- AutoAWQ
+- 7b
+- quantized
+- Mistral
+- Mistral-7B
+---
+
+
+# Model Card for alokabhishek/Mistral-7B-Instruct-v0.2-4bit-AWQ
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+This repo contains 4-bit quantized (using AutoAWQ) model of Mistral AI_'s Mistral-7B-Instruct-v0.2
+
+AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration is developed by MIT-HAN-Lab
+
+
+## Model Details
+
+- Model creator: [Mistral AI_](https://huggingface.co/mistralai)
+- Original model: [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)
+
+
+### About 4 bit quantization using AutoAWQ
+
+- AutoAWQ github repo: [AutoAWQ github repo](https://github.com/casper-hansen/AutoAWQ/tree/main)
+- MIT-han-lab llm-aws github repo:  [MIT-han-lab llm-aws github repo](https://github.com/mit-han-lab/llm-awq/tree/main)
+
+@inproceedings{lin2023awq,
+  title={AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration},
+  author={Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Chen, Wei-Ming and Wang, Wei-Chen and Xiao, Guangxuan and Dang, Xingyu and Gan, Chuang and Han, Song},
+  booktitle={MLSys},
+  year={2024}
+}
+
+
+# How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+## How to run from Python code
+
+#### First install the package
+```shell
+!pip install autoawq
+!pip install accelerate
+```
+
+#### Import 
+
+```python
+import torch
+import os
+from torch import bfloat16
+from huggingface_hub import login, HfApi, create_repo
+from transformers import AutoTokenizer, pipeline
+from awq import AutoAWQForCausalLM
+```
+
+#### Use a pipeline as a high-level helper
+
+```python
+# define the model ID
+model_id_llama = "alokabhishek/Mistral-7B-Instruct-v0.2-4bit-AWQ"
+
+# Load model
+tokenizer_llama = AutoTokenizer.from_pretrained(model_id_llama, use_fast=True)
+model_llama = AutoAWQForCausalLM.from_quantized(model_id_llama, fuse_layer=True, trust_remote_code = False, safetensors = True)
+
+# Set up the prompt and prompt template. Change instruction as per requirements.
+prompt_llama = "Tell me a funny joke about Large Language Models meeting a Blackhole in an intergalactic Bar."
+fromatted_prompt = f'''[INST] <<SYS>> You are a helpful, and fun loving assistant. Always answer as jestfully as possible. <</SYS>> {prompt_llama} [/INST] '''
+
+tokens = tokenizer_llama(fromatted_prompt, return_tensors="pt").input_ids.cuda()
+
+# Generate output, adjust parameters as per requirements
+generation_output = model_llama.generate(tokens, do_sample=True, temperature=1.7, top_p=0.95, top_k=40, max_new_tokens=512)
+
+# Print the output
+print(tokenizer_llama.decode(generation_output[0], skip_special_tokens=True))
+
+
+```
+
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]