|
--- |
|
language: |
|
- ar |
|
- en |
|
pipeline_tag: text-generation |
|
--- |
|
<!-- README_GPTQ.md-use-from-python start --> |
|
## How to use this GPTQ model from Python code |
|
|
|
### Install the necessary packages |
|
|
|
Requires: Transformers 4.32.0 or later, Optimum 1.12.0 or later, and AutoGPTQ 0.4.2 or later. |
|
|
|
```shell |
|
pip3 install transformers>=4.32.0 optimum>=1.12.0 |
|
pip3 install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/ # Use cu117 if on CUDA 11.7 |
|
``` |
|
|
|
### You can then generate a simple gradioweb with_quant.py |
|
'''python |
|
python web_quant.py --model-name ${model-path} |
|
''' |
|
|
|
|
