|
--- |
|
language: |
|
- ar |
|
- en |
|
model_name: AceGPT 7B Chat |
|
model_type: AceGPT |
|
quantized_by: FreedomIntelligence |
|
--- |
|
|
|
<!-- description start --> |
|
## Description |
|
|
|
This repo contains int4 model(GPTQ) for [AceGPT-7B-Chat](https://huggingface.co/FreedomIntelligence/AceGPT-7B-chat). |
|
|
|
The performance of the int4 version has experienced some degradation. For a better user experience, please use the fp16 version. |
|
For details, see [AceGPT-7B-Chat](https://huggingface.co/FreedomIntelligence/AceGPT-7B-chat) and [AceGPT-13B-Chat](https://huggingface.co/FreedomIntelligence/AceGPT-13B-chat). |
|
|
|
|
|
|
|
<!-- description end --> |
|
|
|
<!-- README_GPTQ.md-use-from-python start --> |
|
## How to use this GPTQ model from Python code |
|
|
|
### Install the necessary packages |
|
|
|
Requires: Transformers 4.32.0 or later, Optimum 1.12.0 or later, and AutoGPTQ 0.4.2 or later. |
|
|
|
|
|
```shell |
|
pip3 install transformers>=4.32.0 optimum>=1.12.0 #See requirements.py for verified versions. |
|
pip3 install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/ # Use cu117 if on CUDA 11.7 |
|
``` |
|
|
|
### You can then generate a simple gradioweb with_quant.py |
|
|
|
```bash |
|
python web_quant.py --model-name ${model-path} |
|
|
|
``` |
|
<!-- README_GPTQ.md-use-from-python end --> |
