Update README.md
Browse files
README.md
CHANGED
@@ -26,7 +26,8 @@ Coherent at 32K Context. Not as good as a natively trained 32K model, but much b
|
|
26 |
|
27 |
Relevant Axolotl Configurations:
|
28 |
<br>-> Taken from [winglian/Llama-3-8b-64k-PoSE](https://huggingface.co/winglian/Llama-3-8b-64k-PoSE)
|
29 |
-
<br>\- I tried to find my own configs,
|
|
|
30 |
|
31 |
```
|
32 |
sequence_len: 8192
|
|
|
26 |
|
27 |
Relevant Axolotl Configurations:
|
28 |
<br>-> Taken from [winglian/Llama-3-8b-64k-PoSE](https://huggingface.co/winglian/Llama-3-8b-64k-PoSE)
|
29 |
+
<br>\- I tried to find my own configs, hours of tinkering but the one he used worked best, so I stuck to it.
|
30 |
+
<br>\- 2M Rope Theta had the best loss results during training compared to other values.
|
31 |
|
32 |
```
|
33 |
sequence_len: 8192
|