---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
tags:
- AutoGPTQ
- 4bit
- GPTQ
---

Model created using [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) on a [GPT-2](https://huggingface.co/gpt2) model with 4-bit quantization.

You can load this model with the AutoGPTQ library, installed with the following command:

```
pip install auto-gptq
```

You can then download the model from the hub using the following code:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

model_name = "Sanrove/gpt2-GPTQ-4b"
tokenizer = AutoTokenizer.from_pretrained(model_name)
quantize_config = BaseQuantizeConfig.from_pretrained(model_name)
model = AutoGPTQForCausalLM.from_quantized(model_name,
                                           model_basename="gptq_model-4bit-128g",
                                           device="cuda:0",
                                           use_triton=True,
                                           use_safetensors=True,
                                           quantize_config=quantize_config)
```

This model works with the traditional [Text Generation pipeline](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.TextGenerationPipeline).

Example of generation with the input text "I have a dream":

```
I have a dream." – William Shakespeare

With this opening line, one can see how Shakespeare was very influenced by the story of the great poet. The great poet was the first true English poet, as well as the son of a English noble
```