iwan-rg/XLM-Roberta-base-Finetuned-Arabic-ADAPTER

xlm-roberta-base-finetuned-Adapter-ar-mlm-0.15-large-29OCT

This model is a fine-tuned version of xlm-roberta-base on the None dataset. It achieves the following results on the evaluation set:

• Loss: 2.0107
• Model Preparation Time: 0.0044

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 32
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	Model Preparation Time
3.5857	0.4703	1000	2.9678	0.0044
2.8254	0.9407	2000	2.5072	0.0044
2.5882	1.4110	3000	2.3254	0.0044
2.4612	1.8814	4000	2.2290	0.0044
2.3731	2.3517	5000	2.1540	0.0044
2.316	2.8221	6000	2.1089	0.0044
2.2806	3.2924	7000	2.0712	0.0044
2.2416	3.7628	8000	2.0418	0.0044
2.21	4.2331	9000	2.0209	0.0044
2.1977	4.7035	10000	2.0107	0.0044

Framework versions

• Transformers 4.43.4
• Pytorch 2.1.1+cu121
• Datasets 3.0.2
• Tokenizers 0.19.1