Update README.md
Browse files
README.md
CHANGED
@@ -28,7 +28,7 @@ Even though [Falcon-11B](https://huggingface.co/tiiuae/falcon-11B) is trained on
|
|
28 |
![image/png](https://cdn-uploads.huggingface.co/production/uploads/660c0a02cf274b3ab77dd6b7/QeaL9bOrPskustzFpjMUP.png)
|
29 |
This is why the choice is made to prune 50% of the layers.
|
30 |
Note that \~1B of continued pre-training (\~1M rows of 1k tokens) is still required to restore the perplexity of this model in the desired language.
|
31 |
-
I'm planning on doing that for certain languages, depending on how much compute will be available.
|
32 |
|
33 |
# sliced
|
34 |
|
|
|
28 |
![image/png](https://cdn-uploads.huggingface.co/production/uploads/660c0a02cf274b3ab77dd6b7/QeaL9bOrPskustzFpjMUP.png)
|
29 |
This is why the choice is made to prune 50% of the layers.
|
30 |
Note that \~1B of continued pre-training (\~1M rows of 1k tokens) is still required to restore the perplexity of this model in the desired language.
|
31 |
+
I'm planning on doing that for certain languages when fineweb-edu-{specific_language} will be available, depending on how much compute will be available.
|
32 |
|
33 |
# sliced
|
34 |
|