Ubuntu commited on
Commit
8e56875
1 Parent(s): d7f45b0

adding initial model card

Browse files
Files changed (1) hide show
  1. README.md +58 -2
README.md CHANGED
@@ -1,13 +1,69 @@
1
  ---
 
 
 
 
 
 
 
 
2
  inference: false
 
3
  ---
4
  # meta-llama/Meta-Llama-3-8B AWQ
5
 
6
- ** PROCESSING .... ETA 30mins **
7
-
8
  - Model creator: [meta-llama](https://huggingface.co/meta-llama)
9
  - Original model: [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B)
10
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
  ### About AWQ
12
 
13
  AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference with equivalent or better quality compared to the most commonly used GPTQ settings.
 
1
  ---
2
+ library_name: transformers
3
+ tags:
4
+ - 4-bit
5
+ - AWQ
6
+ - text-generation
7
+ - autotrain_compatible
8
+ - endpoints_compatible
9
+ pipeline_tag: text-generation
10
  inference: false
11
+ quantized_by: Suparious
12
  ---
13
  # meta-llama/Meta-Llama-3-8B AWQ
14
 
 
 
15
  - Model creator: [meta-llama](https://huggingface.co/meta-llama)
16
  - Original model: [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B)
17
 
18
+
19
+
20
+ ## How to use
21
+
22
+ ### Install the necessary packages
23
+
24
+ ```bash
25
+ pip install --upgrade autoawq autoawq-kernels
26
+ ```
27
+
28
+ ### Example Python code
29
+
30
+ ```python
31
+ from awq import AutoAWQForCausalLM
32
+ from transformers import AutoTokenizer, TextStreamer
33
+
34
+ model_path = "solidrust/Meta-Llama-3-8B-AWQ"
35
+ system_message = "You are Meta-Llama-3-8B, incarnated as a powerful AI. You were created by meta-llama."
36
+
37
+ # Load model
38
+ model = AutoAWQForCausalLM.from_quantized(model_path,
39
+ fuse_layers=True)
40
+ tokenizer = AutoTokenizer.from_pretrained(model_path,
41
+ trust_remote_code=True)
42
+ streamer = TextStreamer(tokenizer,
43
+ skip_prompt=True,
44
+ skip_special_tokens=True)
45
+
46
+ # Convert prompt to tokens
47
+ prompt_template = """\
48
+ <|im_start|>system
49
+ {system_message}<|im_end|>
50
+ <|im_start|>user
51
+ {prompt}<|im_end|>
52
+ <|im_start|>assistant"""
53
+
54
+ prompt = "You're standing on the surface of the Earth. "\
55
+ "You walk one mile south, one mile west and one mile north. "\
56
+ "You end up exactly where you started. Where are you?"
57
+
58
+ tokens = tokenizer(prompt_template.format(system_message=system_message,prompt=prompt),
59
+ return_tensors='pt').input_ids.cuda()
60
+
61
+ # Generate output
62
+ generation_output = model.generate(tokens,
63
+ streamer=streamer,
64
+ max_new_tokens=512)
65
+ ```
66
+
67
  ### About AWQ
68
 
69
  AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference with equivalent or better quality compared to the most commonly used GPTQ settings.