SicariusSicariiStuff
commited on
Commit
•
38dd9d6
1
Parent(s):
6aa0083
Update README.md
Browse files
README.md
CHANGED
@@ -34,17 +34,17 @@ language:
|
|
34 |
|
35 |
This model was trained on ~25M tokens, in **3 phases**, the first and longest phase was an FFT to teach the model new stuff, and to confuse the shit out of it too, so it would be **a little bit less inclined to use GPTisms**.
|
36 |
|
37 |
-
It worked pretty well. In fact, the model was so damn confused, that the little
|
38 |
|
39 |
In the next phase, a DEEP QLORA of **R = 512** was used on a new dataset, to... unconfuse it. A completely different dataset was used to avoid overfitting.
|
40 |
|
41 |
Finally, another somewhat deep QLORA of **R = 128** was used to tie it all together in a coherent way, and connect all the dots, and this was also with a different dataset as well.
|
42 |
|
43 |
-
The results are **sometimes** surprisingly good, it even managed to fool some people into thinking it's a MUCH larger model, and sometimes... sometimes it behaves just like you would expect a 3B model
|
44 |
|
45 |
-
Fun fact: the model was uploaded while there were 200 ICBMs headed my way,
|
46 |
|
47 |
-
I lived, so expect more models in the future
|
48 |
|
49 |
## Impish_LLAMA_3B is available at the following quantizations:
|
50 |
|
|
|
34 |
|
35 |
This model was trained on ~25M tokens, in **3 phases**, the first and longest phase was an FFT to teach the model new stuff, and to confuse the shit out of it too, so it would be **a little bit less inclined to use GPTisms**.
|
36 |
|
37 |
+
It worked pretty well. In fact, the model was so damn thoroughly confused, that the little devil didn't even make any sense, but the knowledge was there.
|
38 |
|
39 |
In the next phase, a DEEP QLORA of **R = 512** was used on a new dataset, to... unconfuse it. A completely different dataset was used to avoid overfitting.
|
40 |
|
41 |
Finally, another somewhat deep QLORA of **R = 128** was used to tie it all together in a coherent way, and connect all the dots, and this was also with a different dataset as well.
|
42 |
|
43 |
+
The results are **sometimes** surprisingly good, it even managed to fool some people into thinking it's a MUCH larger model, and sometimes... sometimes it behaves just like you would expect a 3B model to...
|
44 |
|
45 |
+
Fun fact: the model was uploaded while there were 200 ICBMs headed my way, just flying there in the sky.
|
46 |
|
47 |
+
I lived, so expect more models in the future!
|
48 |
|
49 |
## Impish_LLAMA_3B is available at the following quantizations:
|
50 |
|