ssmits commited on
Commit
e9d1ee2
1 Parent(s): ab408ab

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -28,7 +28,7 @@ Even though [Falcon-11B](https://huggingface.co/tiiuae/falcon-11B) is trained on
28
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/660c0a02cf274b3ab77dd6b7/QeaL9bOrPskustzFpjMUP.png)
29
  This is why the choice is made to prune 50% of the layers.
30
  Note that \~1B of continued pre-training (\~1M rows of 1k tokens) is still required to restore the perplexity of this model in the desired language.
31
- I'm planning on doing that for certain languages, depending on how much compute will be available.
32
 
33
  # sliced
34
 
 
28
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/660c0a02cf274b3ab77dd6b7/QeaL9bOrPskustzFpjMUP.png)
29
  This is why the choice is made to prune 50% of the layers.
30
  Note that \~1B of continued pre-training (\~1M rows of 1k tokens) is still required to restore the perplexity of this model in the desired language.
31
+ I'm planning on doing that for certain languages when fineweb-edu-{specific_language} will be available, depending on how much compute will be available.
32
 
33
  # sliced
34