Update README.md

README.md

@@ -1,3 +1,88 @@
 ---
-license: apache-2.0
+base_model: yanolja/EEVE-Korean-Instruct-10.8B-v1.0
+inference: false
+language:
+- ko
+library_name: transformers
+license: cc-by-nc-4.0
+pipeline_tag: text-generation
 ---
+
+# Synatra-kiqu-10.7b-awq
+- Model creator: [Yanolja](https://huggingface.co/yanolja)
+- Original model: [yanolja/EEVE-Korean-Instruct-10.8B-v1.0](https://huggingface.co/yanolja/EEVE-Korean-Instruct-10.8B-v1.0)
+
+<!-- description start -->
+## Description
+
+This repo contains AWQ model files for [yanolja/EEVE-Korean-Instruct-10.8B-v1.0](https://huggingface.co/yanolja/EEVE-Korean-Instruct-10.8B-v1.0).
+
+
+### About AWQ
+
+AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference with equivalent or better quality compared to the most commonly used GPTQ settings.
+
+It is supported by:
+
+- [Text Generation Webui](https://github.com/oobabooga/text-generation-webui) - using Loader: AutoAWQ
+- [vLLM](https://github.com/vllm-project/vllm) - Llama and Mistral models only
+- [Hugging Face Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference)
+- [Transformers](https://huggingface.co/docs/transformers) version 4.35.0 and later, from any code or client that supports Transformers
+- [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) - for use from Python code
+
+<!-- description end -->
+
+<!-- README_AWQ.md-use-from-vllm start -->
+## Using OpenAI Chat API with vLLM
+
+Documentation on installing and using vLLM [can be found here](https://vllm.readthedocs.io/en/latest/).
+
+- Please ensure you are using vLLM version 0.2 or later.
+- When using vLLM as a server, pass the `--quantization awq` parameter.
+
+#### Start the OpenAI-Compatible Server:
+- vLLM can be deployed as a server that implements the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API
+
+```shell
+python3 -m vllm.entrypoints.openai.api_server --model Copycats/EEVE-Korean-Instruct-10.8B-v1.0-AWQ --quantization awq --dtype auto
+```
+
+#### Querying the model using OpenAI Chat API:
+- You can use the create chat completion endpoint to communicate with the model in a chat-like interface:
+
+```shell
+curl http://localhost:8000/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "Copycats/EEVE-Korean-Instruct-10.8B-v1.0-AWQ",
+        "messages": [
+            {"role": "system", "content": "당신은 사용자의 질문에 친절하게 답변하는 어시스턴트입니다."},
+            {"role": "user", "content": "괜스레 슬퍼서 눈물이 나면 어떻게 하나요?"}
+        ]
+    }'
+```
+
+#### Python Client Example:
+- Using the openai python package, you can also communicate with the model in a chat-like manner:
+
+```python
+from openai import OpenAI
+# Set OpenAI's API key and API base to use vLLM's API server.
+openai_api_key = "EMPTY"
+openai_api_base = "http://localhost:8000/v1"
+
+client = OpenAI(
+    api_key=openai_api_key,
+    base_url=openai_api_base,
+)
+
+chat_response = client.chat.completions.create(
+    model="Copycats/EEVE-Korean-Instruct-10.8B-v1.0-AWQ",
+    messages=[
+        {"role": "system", "content": "당신은 사용자의 질문에 친절하게 답변하는 어시스턴트입니다."},
+        {"role": "user", "content": "괜스레 슬퍼서 눈물이 나면 어떻게 하나요?"},
+    ]
+)
+print("Chat response:", chat_response)
+```
+<!-- README_AWQ.md-use-from-vllm start -->