Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model:
|
| 4 |
+
- bertin-project/filiberto-124M
|
| 5 |
+
library_name: transformers
|
| 6 |
+
language:
|
| 7 |
+
- es
|
| 8 |
+
pipeline_tag: text-generation
|
| 9 |
+
---
|
| 10 |
+
|
| 11 |
+
**Filiberto 124M** is a small specialized foundation model trained on Spanish Golden Age Dramas.
|
| 12 |
+
|
| 13 |
+
Filiberto 124M OCR is only 124 million parameters. It can run easily on CPU or provide correction at scale on GPUs (>10k tokens/seconds).
|
| 14 |
+
|
| 15 |
+
## Training
|
| 16 |
+
The pre-training material included a collection of works taken from the [TEXORO](https://etso.es/texoro) corpus, via a collaboration with [ETSO](https://etso.es/), totalling ~5 million tokens.
|
| 17 |
+
|
| 18 |
+
Pre-training ran on 5 epochs with levanter (500 steps total, each processing 1024 sequences of 512 tokens) on a TPUv4-32 for 15 minutes.
|
| 19 |
+
|
| 20 |
+
Tokenization is currently done with the GPT-2 tokenizer.
|