Suparious commited on
Commit
af49f9c
1 Parent(s): 8c716f1

update README details

Browse files
Files changed (1) hide show
  1. README.md +30 -0
README.md CHANGED
@@ -26,3 +26,33 @@ quantized_by: Suparious
26
  - Original model: [WestLake 7B v2](https://huggingface.co/senseable/WestLake-7B-v2)
27
 
28
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/6585ffb10eeafbd678d4b3fe/jnqnl8a_zYYMqJoBpX8yS.png)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
  - Original model: [WestLake 7B v2](https://huggingface.co/senseable/WestLake-7B-v2)
27
 
28
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/6585ffb10eeafbd678d4b3fe/jnqnl8a_zYYMqJoBpX8yS.png)
29
+
30
+ ## Model description
31
+
32
+ This repo contains AWQ model files for [Common Sense's WestLake 7B v2](https://huggingface.co/senseable/WestLake-7B-v2).
33
+
34
+ These files were quantised using hardware kindly provided by [SolidRusT Networks](https://solidrust.net/).
35
+
36
+ ### About AWQ
37
+
38
+ AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference with equivalent or better quality compared to the most commonly used GPTQ settings.
39
+
40
+ AWQ models are currently supported on Linux and Windows, with NVidia GPUs only. macOS users: please use GGUF models instead.
41
+
42
+ It is supported by:
43
+
44
+ - [Text Generation Webui](https://github.com/oobabooga/text-generation-webui) - using Loader: AutoAWQ
45
+ - [vLLM](https://github.com/vllm-project/vllm) - version 0.2.2 or later for support for all model types.
46
+ - [Hugging Face Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference)
47
+ - [Transformers](https://huggingface.co/docs/transformers) version 4.35.0 and later, from any code or client that supports Transformers
48
+ - [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) - for use from Python code
49
+
50
+ ## Prompt template: ChatML
51
+
52
+ ```plaintext
53
+ <|im_start|>system
54
+ {system_message}<|im_end|>
55
+ <|im_start|>user
56
+ {prompt}<|im_end|>
57
+ <|im_start|>assistant
58
+ ```