Edit model card

W2V2-BERT-withLM-Studio-Malayalam

This model is a fine-tuned version of facebook/w2v-bert-2.0 on IMASC, OpenSLR Malayalam Train split, Festvox Malayalamdataset. It achieves the following results on the evaluation set:

  • Loss: 0.1587
  • Wer: 0.1157

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 16
  • eval_batch_size: 8
  • seed: 42
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 32
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 500
  • num_epochs: 10
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Wer
1.0335 0.4932 600 0.3654 0.4387
0.1531 0.9864 1200 0.2373 0.3332
0.1074 1.4797 1800 0.2069 0.2953
0.0928 1.9729 2400 0.2146 0.2814
0.0734 2.4661 3000 0.1947 0.2433
0.0678 2.9593 3600 0.1938 0.2406
0.0522 3.4525 4200 0.1566 0.2053
0.0493 3.9457 4800 0.1649 0.1988
0.0366 4.4390 5400 0.1417 0.1834
0.0372 4.9322 6000 0.1542 0.1749
0.028 5.4254 6600 0.1476 0.1620
0.0263 5.9186 7200 0.1388 0.1622
0.0195 6.4118 7800 0.1384 0.1495
0.0185 6.9051 8400 0.1351 0.1383
0.0136 7.3983 9000 0.1404 0.1344
0.0119 7.8915 9600 0.1253 0.1276
0.0087 8.3847 10200 0.1443 0.1284
0.0066 8.8779 10800 0.1475 0.1252
0.0049 9.3711 11400 0.1577 0.1227
0.0038 9.8644 12000 0.1587 0.1157

Framework versions

  • Transformers 4.42.2
  • Pytorch 2.1.1+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1
Downloads last month
20
Safetensors
Model size
606M params
Tensor type
F32
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for vrclc/W2V2-BERT-withLM-Malayalam-Studio

Finetuned
(173)
this model

Datasets used to train vrclc/W2V2-BERT-withLM-Malayalam-Studio

Collection including vrclc/W2V2-BERT-withLM-Malayalam-Studio

Evaluation results