w2v-bert-2.0-CV_Fleurs-lg-50hrs-v5

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.3755
• Wer: 0.3108
• Cer: 0.0651

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 2
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 80
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
0.8769	1.0	6320	0.3126	0.3631	0.0763
0.2352	2.0	12640	0.2544	0.3542	0.0705
0.2161	3.0	18960	0.2639	0.3415	0.0695
0.2088	4.0	25280	0.2666	0.3524	0.0745
0.2063	5.0	31600	0.2863	0.3655	0.0789
0.2043	6.0	37920	0.2792	0.3409	0.0700
0.2036	7.0	44240	0.2787	0.3519	0.0736
0.2051	8.0	50560	0.2774	0.3550	0.0746
0.1967	9.0	56880	0.2710	0.3457	0.0728
0.1754	10.0	63200	0.2714	0.3425	0.0721
0.157	11.0	69520	0.2800	0.3490	0.0727
0.1411	12.0	75840	0.2571	0.3165	0.0671
0.1305	13.0	82160	0.2768	0.3486	0.0726
0.1164	14.0	88480	0.2963	0.3330	0.0718
0.1067	15.0	94800	0.2663	0.3131	0.0670
0.0954	16.0	101120	0.2660	0.3254	0.0667
0.0849	17.0	107440	0.2751	0.3103	0.0659
0.0769	18.0	113760	0.2721	0.3290	0.0695
0.0675	19.0	120080	0.2986	0.3148	0.0670
0.0606	20.0	126400	0.2850	0.3122	0.0653
0.0536	21.0	132720	0.2987	0.3260	0.0687
0.0478	22.0	139040	0.3226	0.3191	0.0654
0.0429	23.0	145360	0.2981	0.3373	0.0678
0.038	24.0	151680	0.3210	0.3172	0.0656
0.0343	25.0	158000	0.3454	0.3056	0.0635
0.0311	26.0	164320	0.3092	0.3153	0.0655
0.0283	27.0	170640	0.3285	0.3165	0.0647
0.0265	28.0	176960	0.3413	0.3125	0.0650
0.024	29.0	183280	0.3894	0.3062	0.0636
0.0223	30.0	189600	0.3681	0.3084	0.0645
0.0205	31.0	195920	0.3552	0.3134	0.0655
0.0188	32.0	202240	0.3656	0.3105	0.0661
0.018	33.0	208560	0.3640	0.3148	0.0659
0.0163	34.0	214880	0.3805	0.3099	0.0649
0.0153	35.0	221200	0.3755	0.3108	0.0651

Framework versions

• Transformers 4.46.1
• Pytorch 2.1.0+cu118
• Datasets 3.1.0
• Tokenizers 0.20.1