FantasiaFoundry
commited on
Update README.md
Browse files
README.md
CHANGED
@@ -14,9 +14,9 @@ tags:
|
|
14 |
|
15 |
Simple python script (`gguf-imat.py`) to generate various GGUF-IQ-Imatrix quantizations from a Hugging Face `author/model` input, for Windows and NVIDIA hardware.
|
16 |
|
17 |
-
This is setup for a Windows machine with 8GB of VRAM, assuming use with an NVIDIA GPU. If you want to change the
|
18 |
|
19 |
-
Your `imatrix.txt` is expected to be located inside the `imatrix` folder.
|
20 |
|
21 |
Adjust `quantization_options` in [**line 133**](https://huggingface.co/FantasiaFoundry/GGUF-Quantization-Script/blob/main/gguf-imat.py#L133).
|
22 |
|
@@ -39,6 +39,8 @@ Quantizations will be output into the created `models\{model-name}-GGUF` folder.
|
|
39 |
|
40 |
### **Credits:**
|
41 |
|
|
|
|
|
42 |
**If this proves useful for you, feel free to credit and share the repository.**
|
43 |
|
44 |
**Made in conjunction with [@Lewdiculous](https://huggingface.co/Lewdiculous).**
|
|
|
14 |
|
15 |
Simple python script (`gguf-imat.py`) to generate various GGUF-IQ-Imatrix quantizations from a Hugging Face `author/model` input, for Windows and NVIDIA hardware.
|
16 |
|
17 |
+
This is setup for a Windows machine with 8GB of VRAM, assuming use with an NVIDIA GPU. If you want to change the `-ngl` (number of GPU layers) amount, you can do so at [**line 120**](https://huggingface.co/FantasiaFoundry/GGUF-Quantization-Script/blob/main/gguf-imat.py#L120). This is only relevant during the `--imatrix` data generation. If you don't have enough VRAM you can decrease the `-ngl` amount or set it to 0 to only use your System RAM instead for all layers, this will make the imatrix data generation take longer, so kta good idea to find the number that gives your own machine the best results.
|
18 |
|
19 |
+
Your `imatrix.txt` is expected to be located inside the `imatrix` folder. I have already included a file that is considered a good starting option, [this discussion](https://github.com/ggerganov/llama.cpp/discussions/5263#discussioncomment-8395384) is where it came from. If you have suggestions or other imatrix data to recommend please do so.
|
20 |
|
21 |
Adjust `quantization_options` in [**line 133**](https://huggingface.co/FantasiaFoundry/GGUF-Quantization-Script/blob/main/gguf-imat.py#L133).
|
22 |
|
|
|
39 |
|
40 |
### **Credits:**
|
41 |
|
42 |
+
Feel free to Pull Request with your own features and improvements to this script.
|
43 |
+
|
44 |
**If this proves useful for you, feel free to credit and share the repository.**
|
45 |
|
46 |
**Made in conjunction with [@Lewdiculous](https://huggingface.co/Lewdiculous).**
|