| # ALBERT Tiny Spanish | |
| This is an [ALBERT](https://github.com/google-research/albert) model trained on a [big spanish corpora](https://github.com/josecannete/spanish-corpora). | |
| The model was trained on a single TPU v3-8 with the following hyperparameters and steps/time: | |
| - LR: 0.00125 | |
| - Batch Size: 2048 | |
| - Warmup ratio: 0.0125 | |
| - Warmup steps: 125000 | |
| - Goal steps: 10000000 | |
| - Total steps: 8300000 | |
| - Total training time (aprox): 58.2 days | |
| ## Training loss | |
|  |