| language: | |
| - it | |
| pipeline_tag: fill-mask | |
| This model (based on [bert-tiny](https://huggingface.co/google/bert_uncased_L-2_H-128_A-2)) has been trained from scratch on 5k italian Wikipedia articles and on a wide collection of italian words and dictionary definitions. | |
| The project is still a work in progress, new versions will come soon. | |
| # Training | |
| - epochs: 200 | |
| - lr: 1e-5 | |
| - optim: AdamW | |
| - weight_decay: 1e-3 | |
| # Eval | |
| - perplexity: 50 | |