README.md

---
base_model:
- TinyLlama/TinyLlama-1.1B-Chat-v1.0
---
# Model Card for TinyLlama-1.1B-Chat-v1.0 (Quantized)

This is a quantized version of **TinyLlama-1.1B-Chat-v1.0**.

### Performance Evaluation

The quantized model was tested on the `hellaswag` dataset with the following results:

| Metric                  | Base Model | Quantized Model | Change           |
|-------------------------|------------|-----------------|------------------|
| hellaswag accuracy      | 0.456      | 0.462           | unchanged        |
| hellaswag normalized accuracy | 0.64       | 0.64            | unchanged        |
| eval time (GPU) - seconds | 219.67     | 209.34          | 4.70% decrease   |

The quantized version of TinyLlama-1.1B-Chat-v1.0 maintains similar accuracy while achieving a 4.7% reduction in evaluation time. This evaluation was conducted using GPU resources on a subset of 100 `hellaswag` samples for expediency. For production purposes, it is recommended to perform a full evaluation.

**Quantization Approach**  
The model was quantized to 4-bits using the Q4_K_M method with `llama.cpp`, specifically designed for optimized GPU performance. The following steps were used:

1. Convert the original model to GGUF format:

   ```bash
   python ./llama.cpp/convert_hf_to_gguf.py ./llama.cpp/models/TinyLlama-1.1B-Chat-v1.0/

2. Quantize the GGUF model to 4-bit Q4_K_M:

./llama.cpp/build/bin/llama-quantize ./llama.cpp/models/TinyLlama-1.1B-Chat-v1.0/ggml-model-Q4_K_M.gguf q4_k_m