Sao10K commited on
Commit
bd0b38d
1 Parent(s): bd34d85

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -1
README.md CHANGED
@@ -26,7 +26,8 @@ Coherent at 32K Context. Not as good as a natively trained 32K model, but much b
26
 
27
  Relevant Axolotl Configurations:
28
  <br>-> Taken from [winglian/Llama-3-8b-64k-PoSE](https://huggingface.co/winglian/Llama-3-8b-64k-PoSE)
29
- <br>\- I tried to find my own configs, but his worked best. 2M Theta had the best loss results during training compared to other values.
 
30
 
31
  ```
32
  sequence_len: 8192
 
26
 
27
  Relevant Axolotl Configurations:
28
  <br>-> Taken from [winglian/Llama-3-8b-64k-PoSE](https://huggingface.co/winglian/Llama-3-8b-64k-PoSE)
29
+ <br>\- I tried to find my own configs, hours of tinkering but the one he used worked best, so I stuck to it.
30
+ <br>\- 2M Rope Theta had the best loss results during training compared to other values.
31
 
32
  ```
33
  sequence_len: 8192