FantasiaFoundry
commited on
Commit
•
32d703a
1
Parent(s):
cc01c1e
Attributions.
Browse files
README.md
CHANGED
@@ -7,6 +7,11 @@ tags:
|
|
7 |
- quantized
|
8 |
- text-generation-inference
|
9 |
---
|
|
|
|
|
|
|
|
|
|
|
10 |
Simple python script (`gguf-imat.py`) to generate various GGUF-IQ-Imatrix quantizations from a Hugging Face `author/model` input, for Windows and NVIDIA hardware.
|
11 |
|
12 |
This is setup for a Windows machine with 8GB of VRAM, assuming use with an NVIDIA GPU. If you want to change the the `-ngl` (number of GPU layers) amount, you can do so at **line 120**. This is only relevant during the `--imatrix` data generation. If you don't have enough VRAM you can decrease the `-ngl` amount or set it to 0 to only use your System RAM instead for all layers.
|
@@ -25,4 +30,9 @@ python .\gguf-imat.py
|
|
25 |
```
|
26 |
Quantizations will be output into the created `models\{model-name}-GGUF` folder.
|
27 |
<br><br>
|
28 |
-
|
|
|
|
|
|
|
|
|
|
|
|
7 |
- quantized
|
8 |
- text-generation-inference
|
9 |
---
|
10 |
+
|
11 |
+
**GGUF-IQ-Imatrix-Quantization-Script:**
|
12 |
+
|
13 |
+
![image/png](https://cdn-uploads.huggingface.co/production/uploads/65ddabb9bbffb280f4b45d8e/zQNp-muoarsSglA8cyHal.png)
|
14 |
+
|
15 |
Simple python script (`gguf-imat.py`) to generate various GGUF-IQ-Imatrix quantizations from a Hugging Face `author/model` input, for Windows and NVIDIA hardware.
|
16 |
|
17 |
This is setup for a Windows machine with 8GB of VRAM, assuming use with an NVIDIA GPU. If you want to change the the `-ngl` (number of GPU layers) amount, you can do so at **line 120**. This is only relevant during the `--imatrix` data generation. If you don't have enough VRAM you can decrease the `-ngl` amount or set it to 0 to only use your System RAM instead for all layers.
|
|
|
30 |
```
|
31 |
Quantizations will be output into the created `models\{model-name}-GGUF` folder.
|
32 |
<br><br>
|
33 |
+
|
34 |
+
### **Credits:**
|
35 |
+
|
36 |
+
**If this proves useful for you, feel free to credit and share the repository.**
|
37 |
+
|
38 |
+
**Made in conjunction with [@Lewdiculous](https://huggingface.co/Lewdiculous).**
|