---
datasets:
- NeelNanda/pile-10k
---
## Model Card Details

This is a recipe of int4 model with group_size 128 for [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) generated by [intel/auto-round](https://github.com/intel/auto-round). 

## Reproduce the model

This is an outdated recipe. We recommend using symmetric quantization by removing '--asym'

```bash

auto-round \
--model_name  meta-llama/Meta-Llama-3.1-8B-Instruct \
--device 0 \
--group_size 128 \
--bits 4 \
--nsamples 512 \
--iters 1000 \
--asym \
--model_dtype "fp16" \
--deployment_device 'auto_round' \
--eval_bs 16 \
--output_dir "./tmp_autoround" \

```



## Inference on CPU/HPU//CUDA

1 git clone https://github.com/intel/auto-round && cd auto-round && pip install -vvv --no-build-isolation -e .

2 CPU only machine: pip install intel-extension-for-transformers

  HPU: docker image with Gaudi Software Stack is recommended, please refer to following script for environment setup. More details can be found in [Gaudi Guide](https://docs.habana.ai/en/latest/Installation_Guide/Bare_Metal_Fresh_OS.html#launch-docker-image-that-was-built).

  CUDA: no extra operations

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_round.auto_quantizer import AutoHfQuantizer

quantized_model_dir = "Intel/Meta-Llama-3.1-8B-Instruct-AutoRound"
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
model = AutoModelForCausalLM.from_pretrained(quantized_model_dir,
                                             device_map="auto"
                                             )

##import habana_frameworks.torch.core as htcore ## uncommnet it for HPU
##import habana_frameworks.torch.hpu as hthpu ## uncommnet it for HPU
##model = model.to(torch.bfloat16).to("hpu") ## uncommnet it for HPU

tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))

text = "There is a girl who likes adventure,"
##There is a girl who likes adventure, and she is always ready to go on a journey. She is a traveler, a wanderer, and a seeker of new experiences. She is always looking for something new and exciting, and she is not afraid to take risks. She is a free

text = "Once upon a time, "
##Once upon a time,  in the land of the free and the home of the brave, there was a young girl named Lily. She was a bright and curious child, with a heart full of wonder and a mind full of questions. She lived in a small town surrounded by

text = "Which one is bigger, 9.11 or 9.8 "
##Which one is bigger, 9.11 or 9.8 9.11 is bigger. 9.11 is bigger than 9.8. 9.11 is bigger than 9.8. 9.11 is bigger than 9.8. 9.11 is bigger than 

text = "Tell me something about Intel. "
##Tell me something about Intel.  I've heard they're a big company, but I don't know much about them.  What do they do?" \n
##Intel is a multinational technology company that designs and manufactures semiconductor chips, primarily for the computing and communications industries.  They are one of

```



## Evaluate the model 

lm-eval 0.4.2 eval_bs 16, 0-shot in default

| Metric                    | BF16   | INT4   | INT4 lmhead |
|:------------------------- |:-------|:------ |:----------- |
| Avg.                      | 0.6539 | 0.6483 | 0.6454      |
| Avg w/o gsm8k             | 0.6425 | 0.6394 | 0.6357      |
| mmlu                      | 0.6778 | 0.6676 | 0.6602      |
| lambada_openai            | 0.7306 | 0.7262 | 0.7180      |
| hellaswag                 | 0.5911 | 0.5848 | 0.5840      |
| winogrande                | 0.7388 | 0.7261 | 0.7198      |
| piqa                      | 0.7998 | 0.7976 | 0.7992      |
| truthfulqa_mc1            | 0.3709 | 0.3635 | 0.3709      |
| openbookqa                | 0.3380 | 0.3440 | 0.3380      |
| boolq                     | 0.8410 | 0.8361 | 0.8339      |
| arc_easy                  | 0.8203 | 0.8220 | 0.8182      |
| arc_challenge             | 0.5162 | 0.5256 | 0.5145      |
| gsm8k(5shot) strict match | 0.7688 | 0.7377 | 0.7422      |

## Ethical Considerations and Limitations

The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.

Therefore, before deploying any applications of the model, developers should perform safety testing.

## Caveats and Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

Here are a couple of useful links to learn more about Intel's AI software:

* Intel Neural Compressor [link](https://github.com/intel/neural-compressor)
* Intel Extension for Transformers [link](https://github.com/intel/intel-extension-for-transformers)

## Disclaimer

The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.