w2v-bert-2.0-CV_Fleurs-lg-10hrs-v4

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.8790
• Wer: 0.3341
• Cer: 0.0706

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 4
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
1.4106	1.0	513	0.5293	0.5666	0.1264
0.4775	2.0	1026	0.4882	0.5251	0.1118
0.3696	3.0	1539	0.3973	0.4565	0.0998
0.3007	4.0	2052	0.3896	0.4311	0.0932
0.2564	5.0	2565	0.3743	0.4372	0.0966
0.2273	6.0	3078	0.3549	0.4184	0.0880
0.203	7.0	3591	0.3836	0.4001	0.0847
0.1775	8.0	4104	0.3694	0.4132	0.0855
0.159	9.0	4617	0.3487	0.4137	0.0890
0.1444	10.0	5130	0.3705	0.4277	0.0892
0.1262	11.0	5643	0.3620	0.3982	0.0828
0.1139	12.0	6156	0.3666	0.4055	0.0842
0.1015	13.0	6669	0.3656	0.4011	0.0836
0.0914	14.0	7182	0.3573	0.3928	0.0834
0.0836	15.0	7695	0.3798	0.3795	0.0790
0.0729	16.0	8208	0.4289	0.3896	0.0822
0.0652	17.0	8721	0.4778	0.4157	0.0874
0.0599	18.0	9234	0.4570	0.3796	0.0792
0.0504	19.0	9747	0.4444	0.4125	0.0856
0.0498	20.0	10260	0.4612	0.3945	0.0842
0.0434	21.0	10773	0.4881	0.4002	0.0843
0.0386	22.0	11286	0.5099	0.3777	0.0804
0.0381	23.0	11799	0.4904	0.3866	0.0824
0.0344	24.0	12312	0.4622	0.4028	0.0834
0.0302	25.0	12825	0.4986	0.3918	0.0820
0.0268	26.0	13338	0.5162	0.3954	0.0812
0.0272	27.0	13851	0.4748	0.3774	0.0791
0.0235	28.0	14364	0.4718	0.3823	0.0785
0.0219	29.0	14877	0.5318	0.3738	0.0797
0.0205	30.0	15390	0.5196	0.3769	0.0783
0.0203	31.0	15903	0.5203	0.3741	0.0788
0.019	32.0	16416	0.5031	0.3858	0.0809
0.0179	33.0	16929	0.5772	0.3745	0.0810
0.0175	34.0	17442	0.4906	0.3676	0.0763
0.0166	35.0	17955	0.5371	0.3694	0.0786
0.0138	36.0	18468	0.5748	0.3744	0.0788
0.0134	37.0	18981	0.5343	0.3697	0.0778
0.0135	38.0	19494	0.5407	0.3839	0.0804
0.0123	39.0	20007	0.5343	0.3661	0.0767
0.0124	40.0	20520	0.5633	0.3801	0.0817
0.0131	41.0	21033	0.5581	0.3633	0.0774
0.0094	42.0	21546	0.5862	0.3684	0.0789
0.0101	43.0	22059	0.5479	0.3646	0.0761
0.0094	44.0	22572	0.5738	0.3621	0.0761
0.0078	45.0	23085	0.5284	0.3782	0.0777
0.0074	46.0	23598	0.6277	0.3725	0.0790
0.01	47.0	24111	0.5826	0.3686	0.0765
0.0088	48.0	24624	0.5601	0.3660	0.0761
0.0083	49.0	25137	0.5410	0.3606	0.0769
0.0074	50.0	25650	0.5592	0.3613	0.0780
0.007	51.0	26163	0.5891	0.3690	0.0779
0.0067	52.0	26676	0.5807	0.3662	0.0779
0.0067	53.0	27189	0.5851	0.3640	0.0773
0.0065	54.0	27702	0.5989	0.3667	0.0767
0.005	55.0	28215	0.5746	0.3757	0.0785
0.0071	56.0	28728	0.5823	0.3610	0.0757
0.005	57.0	29241	0.6048	0.3562	0.0758
0.0046	58.0	29754	0.6254	0.3561	0.0753
0.0055	59.0	30267	0.6036	0.3533	0.0755
0.004	60.0	30780	0.5876	0.3605	0.0758
0.0042	61.0	31293	0.5782	0.3643	0.0776
0.0034	62.0	31806	0.6118	0.3656	0.0748
0.004	63.0	32319	0.5830	0.3650	0.0756
0.0049	64.0	32832	0.5946	0.3579	0.0755
0.0034	65.0	33345	0.5856	0.3482	0.0725
0.0023	66.0	33858	0.6186	0.3513	0.0739
0.0019	67.0	34371	0.5910	0.3664	0.0760
0.002	68.0	34884	0.6762	0.3597	0.0764
0.0027	69.0	35397	0.6270	0.3503	0.0723
0.0026	70.0	35910	0.6596	0.3551	0.0728
0.0024	71.0	36423	0.6216	0.3563	0.0744
0.0021	72.0	36936	0.6250	0.3483	0.0728
0.0018	73.0	37449	0.5963	0.3524	0.0735
0.0028	74.0	37962	0.6323	0.3541	0.0745
0.0019	75.0	38475	0.6252	0.3459	0.0735
0.0021	76.0	38988	0.6399	0.3500	0.0734
0.0013	77.0	39501	0.6548	0.3499	0.0734
0.001	78.0	40014	0.6746	0.3503	0.0743
0.0011	79.0	40527	0.6395	0.3533	0.0739
0.0007	80.0	41040	0.6779	0.3478	0.0732
0.0007	81.0	41553	0.6806	0.3463	0.0724
0.0009	82.0	42066	0.7214	0.3453	0.0726
0.0017	83.0	42579	0.6250	0.3445	0.0721
0.0009	84.0	43092	0.6632	0.3443	0.0717
0.0005	85.0	43605	0.6850	0.3397	0.0709
0.0002	86.0	44118	0.7168	0.3417	0.0717
0.0001	87.0	44631	0.7629	0.3432	0.0720
0.0002	88.0	45144	0.7349	0.3385	0.0718
0.0003	89.0	45657	0.7347	0.3377	0.0715
0.0002	90.0	46170	0.7449	0.3433	0.0720
0.0001	91.0	46683	0.7630	0.3353	0.0708
0.0	92.0	47196	0.7952	0.3339	0.0705
0.0	93.0	47709	0.8144	0.3341	0.0705
0.0	94.0	48222	0.8309	0.3346	0.0707
0.0	95.0	48735	0.8456	0.3350	0.0708
0.0	96.0	49248	0.8585	0.3346	0.0706
0.0	97.0	49761	0.8673	0.3346	0.0706
0.0	98.0	50274	0.8743	0.3343	0.0707
0.0	99.0	50787	0.8773	0.3341	0.0706
0.0	100.0	51300	0.8790	0.3341	0.0706

Framework versions

• Transformers 4.46.0
• Pytorch 2.1.0+cu118
• Datasets 3.0.2
• Tokenizers 0.20.1