|
--- |
|
license: apache-2.0 |
|
inference: false |
|
--- |
|
|
|
# MistralLite-AWQ Model |
|
|
|
MistralLite-AWQ is a version of the [MistralLite](https://huggingface.co/amazon/MistralLite) model that was |
|
quantized using the AWQ method developed by [Lin et al. (2023)](https://arxiv.org/abs/2306.00978). |
|
The MistralLite-AWQ models are approximately **70% smaller** than those of MistralLite whilst maintaining comparable performance. |
|
|
|
Please refer to the [original MistralLite model card](https://huggingface.co/amazon/MistralLite) for details about the model |
|
preparation and training processes. |
|
|
|
## MistralLite-AWQ Variants |
|
|
|
| Branch | Approx. Model Size | `q_group_size` | `w_bit` | `version` | |
|
|--------|---:|---------------:|--------:|-----------| |
|
| [main](https://huggingface.co/amazon/MistralLite-AWQ/tree/main) | 3.9 GB | 128 | 4 | GEMM | |
|
| [MistralLite-AWQ-64g-4b-GEMM](https://huggingface.co/amazon/MistralLite-AWQ/tree/MistralLite-AWQ-64g-4b-GEMM) | 4.0 GB | 64 | 4 | GEMM | |
|
| [MistralLite-AWQ-32g-4b-GEMM](https://huggingface.co/amazon/MistralLite-AWQ/tree/MistralLite-AWQ-32g-4b-GEMM) | 4.3 GB | 32 | 4 | GEMM | |
|
|
|
## Dependencies |
|
- [`autoawq==0.2.5`](https://pypi.org/project/autoawq/0.2.5/) – [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) was used to quantize the MistralLite model. |
|
- [`vllm==0.4.2`](https://pypi.org/project/vllm/0.4.2/) – [vLLM](https://github.com/vllm-project/vllm) was used to host models for benchmarking. |
|
|
|
## Evaluations |
|
|
|
### Long Context |
|
|
|
The following benchmark results are shown as _accuracy_ (%) values, unless stated otherwise. |
|
|
|
#### Topic Retrieval |
|
|
|
See https://lmsys.org/blog/2023-06-29-longchat/ |
|
|
|
| Model Name | n_topics=05 | n_topics=10 | n_topics=15 | n_topics=20 | n_topics=25 | |
|
|:---------------------------------------------------|--------------:|--------------:|--------------:|--------------:|--------------:| |
|
| _n_tokens_ (approx.) = | _3048_ | _5966_ | _8903_ | _11832_ | _14757_ | |
|
| MistralLite | 100 | 100 | 100 | 100 | 98 | |
|
| **MistralLite-AWQ** | **100** | **100** | **100**| **100** | **98** | |
|
| **MistralLite-AWQ-64g-4b-GEMM** | **100** | **100** | **100**| **100** | **98** | |
|
| **MistralLite-AWQ-32g-4b-GEMM** | **100** | **100** | **100**| **100** | **98** | |
|
| Mistral-7B-Instruct-v0.1 | 96 | 52 | 2 | 0 | 0 | |
|
| Mistral-7B-Instruct-v0.2 | 100 | 100 | 100 | 100 | 100 | |
|
| Mixtral-8x7B-v0.1 | 0 | 0 | 0 | 0 | 0 | |
|
| Mixtral-8x7B-Instruct-v0.1 | 100 | 100 | 100 | 100 | 100 | |
|
|
|
#### [Line Retrieval](https://lmsys.org/blog/2023-06-29-longchat/#longeval-results) |
|
|
|
See https://lmsys.org/blog/2023-06-29-longchat/#longeval-results |
|
|
|
| Model Name | n_lines=200 | n_lines=300 | n_lines=400 | n_lines=500 | n_lines=600 | n_lines=680 | |
|
|:----------|-------------:|-------------:|------------:|-----------:|-----------:|-----------:| |
|
| _n_tokens_ (approx.) = | _4317_ | _6415_ | _8510_ | _10610_ | _12698_ | _14373_ | |
|
| MistralLite | 100 | 94 | 86 | 82 | 76 | 66 | |
|
| **MistralLite-AWQ** | **96**| **94**| **88** | **80** | **70**| **62** | |
|
| **MistralLite-AWQ-64g-4b-GEMM** | **96**| **96**| **90** | **70** | **72**| **60** | |
|
| **MistralLite-AWQ-32g-4b-GEMM** | **98**| **96**| **84** | **76** | **70**| **62** | |
|
| Mistral-7B-Instruct-v0.1 | 96 | 56 | 38 | 36 | 30 | 30 | |
|
| Mistral-7B-Instruct-v0.2 | 100 | 100 | 96 | 98 | 96 | 84 | |
|
| Mixtral-8x7B-v0.1 | 54 | 38 | 56 | 66 | 62 | 38 | |
|
| Mixtral-8x7B-Instruct-v0.1 | 100 | 100 | 100 | 100 | 100 | 100 | |
|
|
|
#### Pass Key Retrieval |
|
|
|
See https://github.com/epfml/landmark-attention/blob/main/llama/run_test.py#L101 |
|
|
|
| Model Name | n_garbage=12000 | n_garbage=20000 | n_garbage=31000 | n_garbage=38000 | n_garbage=45000 | n_garbage=60000 | |
|
|:----------|-------------:|-------------:|------------:|-----------:|-----------:|-----------:| |
|
| _n_tokens_ (approx.) = | _3272_ | _5405_ | _8338_ | _10205_ | _12071_ | _16072_ | |
|
| MistralLite | 100 | 100 | 100 | 100 | 100 | 100| |
|
| **MistralLite-AWQ** | **100** | **100**| **100**| **100** | **100**| **100**| |
|
| **MistralLite-AWQ-64g-4b-GEMM** | **100** | **100**| **100**| **100** | **100**| **100**| |
|
| **MistralLite-AWQ-32g-4b-GEMM** | **100** | **100**| **100**| **100** | **100**| **100**| |
|
| Mistral-7B-Instruct-v0.1 | 100 | 50 | 30 | 20 | 10 | 10 | |
|
| Mistral-7B-Instruct-v0.2 | 100 | 100 | 100 | 100 | 100 | 100 | |
|
| Mixtral-8x7B-v0.1 | 100 | 100 | 100 | 100 | 100 | 100 | |
|
| Mixtral-8x7B-Instruct-v0.1 | 100 | 100 | 100 | 90 | 100 | 100 | |
|
|
|
|
|
#### QuALITY (Question Answering with Long Input Texts, Yes! |
|
|
|
See https://nyu-mll.github.io/quality/ |
|
|
|
|Model Name| Test set Accuracy | Hard subset Accuracy| |
|
|:----------|-------------:|-------------:| |
|
| MistralLite | 56.8 | 74.5 | |
|
| **MistralLite-AWQ** | **55.3** | **71.8** | |
|
| **MistralLite-AWQ-64g-4b-GEMM** | **55.2** | **72.9** | |
|
| **MistralLite-AWQ-32g-4b-GEMM** | **56.6** | **72.8** | |
|
| Mistral-7B-Instruct-v0.1 | 45.2 | 58.9 | |
|
| Mistral-7B-Instruct-v0.2 | 55.5 | 74 | |
|
| Mixtral-8x7B-v0.1 | 75 | 74.1 | |
|
| Mixtral-8x7B-Instruct-v0.1 | 68.7 | 83.3 | |
|
|
|
## Usage |
|
|
|
## Inference via vLLM HTTP Host |
|
|
|
### Launch Host |
|
```bash |
|
python -m vllm.entrypoints.openai.api_server \ |
|
--model amazon/MistralLite-AWQ \ |
|
--quantization awq |
|
``` |
|
|
|
### Query Host |
|
```bash |
|
curl -X POST http://localhost:8000/v1/completions \ |
|
-H "Content-Type: application/json" \ |
|
-d '{ "model": "amazon/MistralLite-AWQ", |
|
"prompt": "<|prompter|>What are the main challenges to support a long context for LLM?</s><|assistant|>", |
|
"temperature": 0, |
|
"echo": false |
|
}' |
|
``` |
|
|
|
## Inference via [vLLM Offline Inference](https://docs.vllm.ai/en/latest/getting_started/examples/offline_inference.html) |
|
```python |
|
from vllm import LLM, SamplingParams |
|
|
|
prompts = [ |
|
"<|prompter|>What are the main challenges to support a long context for LLM?</s><|assistant|>", |
|
] |
|
sampling_params = SamplingParams(temperature=0, max_tokens=100) |
|
|
|
llm = LLM(model="amazon/MistralLite-AWQ") |
|
|
|
outputs = llm.generate(prompts, sampling_params) |
|
|
|
# Print the outputs. |
|
for output in outputs: |
|
prompt = output.prompt |
|
generated_text = output.outputs[0].text |
|
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") |
|
|
|
``` |
|
|
|
## License |
|
|
|
Apache 2.0 |
|
|
|
## Limitations |
|
|
|
Before using the MistralLite-AWQ model, it is important to perform your own |
|
independent assessment, and take measures to ensure that your use would comply |
|
with your own specific quality control practices and standards, and that your |
|
use would comply with the local rules, laws, regulations, licenses and terms |
|
that apply to you, and your content. |
|
|
