wenhuach commited on
Commit
88ef27a
1 Parent(s): 702dc57

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -26
README.md CHANGED
@@ -5,36 +5,11 @@ datasets:
5
  ---
6
  ## Model Details
7
 
8
- This model is an int4 model with group_size128 and sym quantization of [microsoft/phi-2](https://huggingface.co/microsoft/phi-2) generated by [intel/auto-round](https://github.com/intel/auto-round). We found there is a large accuracy drop of asym kernel for this model. If you need AutoGPTQ format, please load the model with revision 5973e3a
9
 
10
 
11
 
12
  ### How To Use
13
- ### INT4 Inference with ITREX on CPU
14
- Install the latest [intel-extension-for-transformers](
15
- https://github.com/intel/intel-extension-for-transformers)
16
- ```python
17
- from intel_extension_for_transformers.transformers import AutoModelForCausalLM
18
- from transformers import AutoTokenizer
19
- quantized_model_dir = "Intel/phi-2-int4-inc"
20
- model = AutoModelForCausalLM.from_pretrained(quantized_model_dir,
21
- device_map="auto",
22
- trust_remote_code=False,
23
- use_neural_speed=False,
24
- ## revision="5973e3a" ##AutoGPTQ format
25
- )
26
- tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, use_fast=True)
27
- print(tokenizer.decode(model.generate(**tokenizer("There is a girl who likes adventure,", return_tensors="pt").to(model.device),max_new_tokens=50)[0]))
28
- """
29
- There is a girl who likes adventure,
30
- She loves to explore and to venture.
31
- She travels to faraway lands,
32
- And meets people from different lands.
33
- She learns new languages and cultures,
34
- And makes friends with all kinds of people.
35
- She is curious and brave and
36
- """
37
- ```
38
 
39
 
40
  ### INT4 Inference
 
5
  ---
6
  ## Model Details
7
 
8
+ This model is an int4 model with group_size128 and sym quantization of [microsoft/phi-2](https://huggingface.co/microsoft/phi-2) generated by [intel/auto-round](https://github.com/intel/auto-round). If you need AutoGPTQ format, please load the model with revision 5973e3a
9
 
10
 
11
 
12
  ### How To Use
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
13
 
14
 
15
  ### INT4 Inference