Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,45 @@
|
|
| 1 |
---
|
| 2 |
license: bigscience-bloom-rail-1.0
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: bigscience-bloom-rail-1.0
|
| 3 |
---
|
| 4 |
+
Multilingual Generative Pretrained Transformer with 176B parameters with capacity for Finnish.
|
| 5 |
+
This model is built upon pretrained [BLOOM](https://huggingface.co/bigscience/bloom) which is then further pretrained with ROOTS + Finnish for 40B tokens.
|
| 6 |
+
|
| 7 |
+
|
| 8 |
+
**Datasets**
|
| 9 |
+
|
| 10 |
+
We used a combination of multiple Finnish resources.
|
| 11 |
+
|
| 12 |
+
* Finnish Internet Parsebank https://turkunlp.org/finnish_nlp.html
|
| 13 |
+
mC4 multilingual colossal, cleaned Common Crawl https://huggingface.co/datasets/mc4
|
| 14 |
+
* Common Crawl Finnish https://TODO
|
| 15 |
+
* Finnish Wikipedia https://fi.wikipedia.org/wiki
|
| 16 |
+
* Lönnrot Projekti Lönnrot http://www.lonnrot.net/
|
| 17 |
+
* ePub National library ”epub” collection
|
| 18 |
+
* National library ”lehdet” collection
|
| 19 |
+
* Suomi24 The Suomi 24 Corpus 2001-2020 http://urn.fi/urn:nbn:fi:lb-2021101527
|
| 20 |
+
* Reddit r/Suomi submissions and comments https://www.reddit.com/r/Suomi
|
| 21 |
+
* STT Finnish News Agency Archive 1992-2018 http://urn.fi/urn:nbn:fi:lb-2019041501
|
| 22 |
+
* Yle Finnish News Archive 2011-2018 http://urn.fi/urn:nbn:fi:lb-2017070501
|
| 23 |
+
* Yle Finnish News Archive 2019-2020 http://urn.fi/urn:nbn:fi:lb-2021050401
|
| 24 |
+
* Yle News Archive Easy-to-read Finnish 2011-2018 http://urn.fi/urn:nbn:fi:lb-2019050901
|
| 25 |
+
* Yle News Archive Easy-to-read Finnish 2019-2020 http://urn.fi/urn:nbn:fi:lb-2021050701
|
| 26 |
+
* [ROOTS](https://arxiv.org/abs/2303.03915) - original BLOOM training corpus
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
**Sampling ratios**
|
| 31 |
+
|
| 32 |
+
|Dataset | Chars | Ratio | Weight | W.Ratio |
|
| 33 |
+
|----------|--------|---------|--------|---------|
|
| 34 |
+
|Parsebank | 35.0B | 16.9\% | 1.5 | 22.7\%|
|
| 35 |
+
|mC4-Fi | 46.3B | 22.4\% | 1.0 | 20.0\%|
|
| 36 |
+
|CC-Fi | 79.6B | 38.5\% | 1.0 | 34.4\%|
|
| 37 |
+
|Fiwiki | 0.8B | 0.4\% | 3.0 | 1.0\%|
|
| 38 |
+
|Lönnrot | 0.8B | 0.4\% | 3.0 | 1.0\%|
|
| 39 |
+
|Yle | 1.6B | 0.8\% | 2.0 | 1.4\%|
|
| 40 |
+
|STT | 2.2B | 1.1\% | 2.0 | 1.9\%|
|
| 41 |
+
|ePub | 13.5B | 6.5\% | 1.0 | 5.8\%|
|
| 42 |
+
|Lehdet | 5.8B | 2.8\% | 1.0 | 2.5\%|
|
| 43 |
+
|Suomi24 | 20.6B | 9.9\% | 1.0 | 8.9\%|
|
| 44 |
+
|Reddit-Fi | 0.7B | 0.4\% | 1.0 | 0.3\%|
|
| 45 |
+
|**TOTAL** | **207.0B** | **100.0\%** | **N/A** | **100.0\%** |
|