w2v-bert-2.0-odia_v1

This model is a fine-tuned version of facebook/w2v-bert-2.0 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.0767
• Wer: 0.1256

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3.5356e-05
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 10
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
0.4216	0.3733	300	0.2149	0.3309
0.2996	0.7465	600	0.1719	0.2572
0.2271	1.1198	900	0.1366	0.2390
0.1917	1.4930	1200	0.1137	0.2054
0.167	1.8663	1500	0.1208	0.2046
0.1371	2.2395	1800	0.0995	0.1995
0.133	2.6128	2100	0.1006	0.1944
0.1214	2.9860	2400	0.0958	0.1715
0.101	3.3593	2700	0.0853	0.1602
0.1007	3.7325	3000	0.0851	0.1667
0.0898	4.1058	3300	0.0820	0.1532
0.089	4.4790	3600	0.0814	0.1539
0.0776	4.8523	3900	0.0792	0.1479
0.0655	5.2255	4200	0.0782	0.1438
0.0708	5.5988	4500	0.0770	0.1391
0.0662	5.9720	4800	0.0727	0.1372
0.0556	6.3453	5100	0.0757	0.1372
0.0629	6.7185	5400	0.0729	0.1319
0.0472	7.0918	5700	0.0771	0.1369
0.0546	7.4650	6000	0.0760	0.1378
0.041	7.8383	6300	0.0750	0.1402
0.0405	8.2115	6600	0.0776	0.1340
0.0395	8.5848	6900	0.0741	0.1306
0.0366	8.9580	7200	0.0742	0.1255
0.0288	9.3313	7500	0.0767	0.1296
0.0329	9.7045	7800	0.0767	0.1256

Framework versions

• Transformers 4.41.1
• Pytorch 2.1.2+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1