Update README.md
Browse files
README.md
CHANGED
|
@@ -138,7 +138,8 @@ Similar to the others [Falcon](https://huggingface.co/tiiuae/falcon-11B) suite m
|
|
| 138 |
Note that at inference the context-length is not relevant as the Mamba architecture has no limit on long range dependency.
|
| 139 |
At the last training stage, small portion of high-quality curated data was used to further enhance performance.
|
| 140 |
|
| 141 |
-
Overall, the data sources included RefinedWeb-English,
|
|
|
|
| 142 |
|
| 143 |
The data was tokenized with the Falcon-[7B](https://huggingface.co/tiiuae/falcon-7B)/[11B](https://huggingface.co/tiiuae/falcon-11B) tokenizer.
|
| 144 |
|
|
|
|
| 138 |
Note that at inference the context-length is not relevant as the Mamba architecture has no limit on long range dependency.
|
| 139 |
At the last training stage, small portion of high-quality curated data was used to further enhance performance.
|
| 140 |
|
| 141 |
+
Overall, the data sources included RefinedWeb-English, high quality technical data, code data and conversational data extracted from public sources.
|
| 142 |
+
In particular, we used samples coming from [Fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu).
|
| 143 |
|
| 144 |
The data was tokenized with the Falcon-[7B](https://huggingface.co/tiiuae/falcon-7B)/[11B](https://huggingface.co/tiiuae/falcon-11B) tokenizer.
|
| 145 |
|