mms-1b-swagen-combined-20hrs-model

This model is a fine-tuned version of facebook/mms-1b-all on the SWAGEN - SWA dataset. It achieves the following results on the evaluation set:

• Loss: 0.2249
• Wer: 0.1913

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 2500.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
13.3661	0.0596	100	0.4504	0.2830
0.6536	0.1193	200	0.2726	0.2036
0.4925	0.1789	300	0.2521	0.1979
0.5177	0.2385	400	0.2536	0.2035
0.53	0.2982	500	0.2374	0.1964
0.4791	0.3578	600	0.2359	0.1938
0.4699	0.4174	700	0.2374	0.1982
0.4791	0.4770	800	0.2356	0.1954
0.4269	0.5367	900	0.2317	0.1951
0.4646	0.5963	1000	0.2311	0.1958
0.4492	0.6559	1100	0.2326	0.1954
0.4438	0.7156	1200	0.2309	0.1924
0.4551	0.7752	1300	0.2329	0.1951
0.4828	0.8348	1400	0.2290	0.1895
0.4502	0.8945	1500	0.2273	0.1915
0.4818	0.9541	1600	0.2249	0.1913
0.4286	1.0137	1700	0.2280	0.1918
0.42	1.0733	1800	0.2303	0.1939
0.4584	1.1330	1900	0.2288	0.1925
0.4255	1.1926	2000	0.2249	0.1924

Framework versions

• Transformers 4.47.1
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0