cicdatopea
commited on
Update README.md
Browse files
README.md
CHANGED
@@ -5,7 +5,7 @@ datasets:
|
|
5 |
|
6 |
## Model Details
|
7 |
|
8 |
-
This model is an int4 model with group_size 128 and symmetric quantization of [
|
9 |
|
10 |
## How To Use
|
11 |
### INT4 Inference(CPU/HPU/CUDA)
|
@@ -13,7 +13,7 @@ This model is an int4 model with group_size 128 and symmetric quantization of [f
|
|
13 |
from auto_round import AutoRoundConfig ##must import for auto_round format
|
14 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
15 |
|
16 |
-
quantized_model_dir = "OPEA/
|
17 |
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
|
18 |
model = AutoModelForCausalLM.from_pretrained(
|
19 |
quantized_model_dir,
|
@@ -72,7 +72,7 @@ text = "There is a girl who likes adventure,"
|
|
72 |
pip3 install lm-eval==0.4.5
|
73 |
|
74 |
```bash
|
75 |
-
auto-round --model "OPEA/
|
76 |
```
|
77 |
|
78 |
| Metric | BF16 | INT4 |
|
@@ -94,7 +94,7 @@ auto-round --model "OPEA/falcon-three-7b-int4-sym-inc" --eval --eval_bs 16 --ta
|
|
94 |
Here is the sample command to generate the model.
|
95 |
```bash
|
96 |
auto-round \
|
97 |
-
--model
|
98 |
--device 0 \
|
99 |
--group_size 128 \
|
100 |
--bits 4 \
|
|
|
5 |
|
6 |
## Model Details
|
7 |
|
8 |
+
This model is an int4 model with group_size 128 and symmetric quantization of [Falcon3-7B-Base](https://huggingface.co/tiiuae/Falcon3-7B-Base) generated by [intel/auto-round](https://github.com/intel/auto-round). Load the model with revision `a10e358` to use AutoGPTQ format, with revision `e9aa317` to use AutoAWQ format
|
9 |
|
10 |
## How To Use
|
11 |
### INT4 Inference(CPU/HPU/CUDA)
|
|
|
13 |
from auto_round import AutoRoundConfig ##must import for auto_round format
|
14 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
15 |
|
16 |
+
quantized_model_dir = "OPEA/Falcon3-7B-Base-int4-sym-inc"
|
17 |
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
|
18 |
model = AutoModelForCausalLM.from_pretrained(
|
19 |
quantized_model_dir,
|
|
|
72 |
pip3 install lm-eval==0.4.5
|
73 |
|
74 |
```bash
|
75 |
+
auto-round --model "OPEA/Falcon3-7B-Base-int4-sym-inc" --eval --eval_bs 16 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,mmlu
|
76 |
```
|
77 |
|
78 |
| Metric | BF16 | INT4 |
|
|
|
94 |
Here is the sample command to generate the model.
|
95 |
```bash
|
96 |
auto-round \
|
97 |
+
--model tiiuae/Falcon3-7B-Base \
|
98 |
--device 0 \
|
99 |
--group_size 128 \
|
100 |
--bits 4 \
|