dranger003
commited on
Commit
•
f80614d
1
Parent(s):
4f67f3f
Update README.md
Browse files
README.md
CHANGED
@@ -21,7 +21,7 @@ The quants here are meant to test imatrix quantized weights.
|
|
21 |
**2024-04-13**: Support for this model has just being merged - [`PR #6515`](https://github.com/ggerganov/llama.cpp/pull/6515).
|
22 |
**<u>You will need this llama.cpp commit [`4bd0f93e`](https://github.com/ggerganov/llama.cpp/commit/4bd0f93e4ab4fe6682e7d0241c1bdec1397e954a) to run this model</u>**
|
23 |
|
24 |
-
Quants in this repo are tested running the following command (quants under
|
25 |
```
|
26 |
./build/bin/main -ngl 41 -s 0 -e -p "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nWrite an essay about AI.<|im_end|>\n<|im_start|>assistant\n" -m ggml-dbrx-instruct-16x12b-<<quant-to-test>>.gguf
|
27 |
```
|
|
|
21 |
**2024-04-13**: Support for this model has just being merged - [`PR #6515`](https://github.com/ggerganov/llama.cpp/pull/6515).
|
22 |
**<u>You will need this llama.cpp commit [`4bd0f93e`](https://github.com/ggerganov/llama.cpp/commit/4bd0f93e4ab4fe6682e7d0241c1bdec1397e954a) to run this model</u>**
|
23 |
|
24 |
+
Quants in this repo are tested running the following command (quants under IQ3 are very sensitive and unreliable so far - the imatrix may require to be trained on FP16 weights rather than Q8_0 and for longer than 200 chunks):
|
25 |
```
|
26 |
./build/bin/main -ngl 41 -s 0 -e -p "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nWrite an essay about AI.<|im_end|>\n<|im_start|>assistant\n" -m ggml-dbrx-instruct-16x12b-<<quant-to-test>>.gguf
|
27 |
```
|