|
--- |
|
datasets: |
|
- NeelNanda/pile-10k |
|
--- |
|
## Model Card Details |
|
|
|
This is a recipe of int4 model with group_size 128 for [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) generated by [intel/auto-round](https://github.com/intel/auto-round). |
|
|
|
## Reproduce the model |
|
|
|
This is an outdated recipe. We recommend using symmetric quantization by removing '--asym' |
|
|
|
```bash |
|
|
|
auto-round \ |
|
--model_name meta-llama/Meta-Llama-3.1-8B-Instruct \ |
|
--device 0 \ |
|
--group_size 128 \ |
|
--bits 4 \ |
|
--nsamples 512 \ |
|
--iters 1000 \ |
|
--asym \ |
|
--model_dtype "fp16" \ |
|
--deployment_device 'auto_round' \ |
|
--eval_bs 16 \ |
|
--output_dir "./tmp_autoround" \ |
|
|
|
``` |
|
|
|
|
|
|
|
## Inference on CPU/HPU//CUDA |
|
|
|
1 git clone https://github.com/intel/auto-round && cd auto-round && pip install -vvv --no-build-isolation -e . |
|
|
|
2 CPU only machine: pip install intel-extension-for-transformers |
|
|
|
HPU: docker image with Gaudi Software Stack is recommended, please refer to following script for environment setup. More details can be found in [Gaudi Guide](https://docs.habana.ai/en/latest/Installation_Guide/Bare_Metal_Fresh_OS.html#launch-docker-image-that-was-built). |
|
|
|
CUDA: no extra operations |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
from auto_round.auto_quantizer import AutoHfQuantizer |
|
|
|
quantized_model_dir = "Intel/Meta-Llama-3.1-8B-Instruct-AutoRound" |
|
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir) |
|
model = AutoModelForCausalLM.from_pretrained(quantized_model_dir, |
|
device_map="auto" |
|
) |
|
|
|
##import habana_frameworks.torch.core as htcore ## uncommnet it for HPU |
|
##import habana_frameworks.torch.hpu as hthpu ## uncommnet it for HPU |
|
##model = model.to(torch.bfloat16).to("hpu") ## uncommnet it for HPU |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir) |
|
text = "There is a girl who likes adventure," |
|
inputs = tokenizer(text, return_tensors="pt").to(model.device) |
|
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0])) |
|
|
|
text = "There is a girl who likes adventure," |
|
##There is a girl who likes adventure, and she is always ready to go on a journey. She is a traveler, a wanderer, and a seeker of new experiences. She is always looking for something new and exciting, and she is not afraid to take risks. She is a free |
|
|
|
text = "Once upon a time, " |
|
##Once upon a time, in the land of the free and the home of the brave, there was a young girl named Lily. She was a bright and curious child, with a heart full of wonder and a mind full of questions. She lived in a small town surrounded by |
|
|
|
text = "Which one is bigger, 9.11 or 9.8 " |
|
##Which one is bigger, 9.11 or 9.8 9.11 is bigger. 9.11 is bigger than 9.8. 9.11 is bigger than 9.8. 9.11 is bigger than 9.8. 9.11 is bigger than |
|
|
|
text = "Tell me something about Intel. " |
|
##Tell me something about Intel. I've heard they're a big company, but I don't know much about them. What do they do?" \n |
|
##Intel is a multinational technology company that designs and manufactures semiconductor chips, primarily for the computing and communications industries. They are one of |
|
|
|
``` |
|
|
|
|
|
|
|
## Evaluate the model |
|
|
|
lm-eval 0.4.2 eval_bs 16, 0-shot in default |
|
|
|
| Metric | BF16 | INT4 | INT4 lmhead | |
|
|:------------------------- |:-------|:------ |:----------- | |
|
| Avg. | 0.6539 | 0.6483 | 0.6454 | |
|
| Avg w/o gsm8k | 0.6425 | 0.6394 | 0.6357 | |
|
| mmlu | 0.6778 | 0.6676 | 0.6602 | |
|
| lambada_openai | 0.7306 | 0.7262 | 0.7180 | |
|
| hellaswag | 0.5911 | 0.5848 | 0.5840 | |
|
| winogrande | 0.7388 | 0.7261 | 0.7198 | |
|
| piqa | 0.7998 | 0.7976 | 0.7992 | |
|
| truthfulqa_mc1 | 0.3709 | 0.3635 | 0.3709 | |
|
| openbookqa | 0.3380 | 0.3440 | 0.3380 | |
|
| boolq | 0.8410 | 0.8361 | 0.8339 | |
|
| arc_easy | 0.8203 | 0.8220 | 0.8182 | |
|
| arc_challenge | 0.5162 | 0.5256 | 0.5145 | |
|
| gsm8k(5shot) strict match | 0.7688 | 0.7377 | 0.7422 | |
|
|
|
## Ethical Considerations and Limitations |
|
|
|
The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs. |
|
|
|
Therefore, before deploying any applications of the model, developers should perform safety testing. |
|
|
|
## Caveats and Recommendations |
|
|
|
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. |
|
|
|
Here are a couple of useful links to learn more about Intel's AI software: |
|
|
|
* Intel Neural Compressor [link](https://github.com/intel/neural-compressor) |
|
* Intel Extension for Transformers [link](https://github.com/intel/intel-extension-for-transformers) |
|
|
|
## Disclaimer |
|
|
|
The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes. |
