metadata

library_name: transformers
license: apache-2.0
base_model: facebook/wav2vec2-large-xlsr-53
tags:
  - generated_from_trainer
metrics:
  - accuracy
  - precision
  - recall
  - f1
model-index:
  - name: speech-emotion-recognition-with-facebook-wav2vec2-large-xlsr-53
    results: []

🎧 Speech Emotion Recognition with Wav2Vec2

This project leverages the Wav2Vec2 model to recognize emotions in speech. The goal is to classify audio recordings into different emotional categories, such as Happy, Sad, Surprised, and etc.

🗂 Dataset

The dataset used for training and evaluation is sourced from multiple datasets, including:

The dataset contains recordings labeled with various emotions. Below is the distribution of the emotions in the dataset:

Emotion	Count
sad	752
happy	752
angry	752
neutral	716
disgust	652
fearful	652
surprised	652
calm	192

This distribution reflects the balance of emotions in the dataset, with some emotions having more samples than others. Excluded the "calm" emotion during training due to its underrepresentation.

🎤 Preprocessing

Audio Loading: Using Librosa to load the audio files and convert them to numpy arrays.
Feature Extraction: The audio data is processed using the Wav2Vec2 Feature Extractor, which standardizes and normalizes the audio features for input to the model.

🔧 Model

The model used is the Wav2Vec2 Large XLR-53 model, fine-tuned for audio classification tasks:

Model: facebook/wav2vec2-large-xlsr-53
Output: Emotion labels (Angry', 'Disgust', 'Fearful', 'Happy', 'Neutral', 'Sad', 'Surprised') I map the emotion labels to numeric IDs and use them for model training and evaluation.

⚙️ Training

The model is trained with the following parameters:

Learning Rate: 5e-05
Train Batch Size: 2
Eval Batch Size: 2
Random Seed: 42
Gradient Accumulation Steps: 5
Total Train Batch Size: 10 (effective batch size after gradient accumulation)
Optimizer: Adam with parameters: betas=(0.9, 0.999) and epsilon=1e-08
Learning Rate Scheduler: linear
Warmup Ratio for LR Scheduler: 0.1
Number of Epochs: 25
Mixed Precision Training: Native AMP (Automatic Mixed Precision)

These parameters ensure efficient model training and stability, especially when dealing with large datasets and deep models like Wav2Vec2. The training utilizes Wandb for experiment tracking and monitoring.

📊 Metrics

The following evaluation metrics were obtained after training the model:

Loss: 0.4989
Accuracy: 0.9168
Precision: 0.9209
Recall: 0.9168
F1 Score: 0.9166

These metrics demonstrate the model's performance on the speech emotion recognition task. The high values for accuracy, precision, recall, and F1 score indicate that the model is effectively identifying emotional states from speech data.

🧪 Results

After training, the model is evaluated on the test dataset, and the results are monitored using Wandb in this Link.

Training Loss	Epoch	Step	Validation Loss	Accuracy	Precision	Recall	F1
1.9343	0.9995	394	1.9277	0.2505	0.1425	0.2505	0.1691
1.7944	1.9990	788	1.6446	0.4574	0.5759	0.4574	0.4213
1.4601	2.9985	1182	1.3242	0.5953	0.6183	0.5953	0.5709
1.0551	3.9980	1576	1.0764	0.6623	0.6659	0.6623	0.6447
0.8934	5.0	1971	0.9209	0.7059	0.7172	0.7059	0.6825
1.1156	5.9995	2365	0.8292	0.7465	0.7635	0.7465	0.7442
0.6307	6.9990	2759	0.6439	0.8043	0.8090	0.8043	0.8020
0.774	7.9985	3153	0.6666	0.7921	0.8117	0.7921	0.7916
0.5537	8.9980	3547	0.5111	0.8245	0.8268	0.8245	0.8205
0.3762	10.0	3942	0.5506	0.8306	0.8390	0.8306	0.8296
0.716	10.9995	4336	0.5499	0.8276	0.8465	0.8276	0.8268
0.5372	11.9990	4730	0.5463	0.8377	0.8606	0.8377	0.8404
0.3746	12.9985	5124	0.4758	0.8611	0.8714	0.8611	0.8597
0.4317	13.9980	5518	0.4438	0.8742	0.8843	0.8742	0.8756
0.2104	15.0	5913	0.4426	0.8803	0.8864	0.8803	0.8806
0.3193	15.9995	6307	0.4741	0.8671	0.8751	0.8671	0.8683
0.3445	16.9990	6701	0.3850	0.9037	0.9047	0.9037	0.9038
0.2777	17.9985	7095	0.4802	0.8834	0.8923	0.8834	0.8836
0.4406	18.9980	7489	0.4053	0.9047	0.9096	0.9047	0.9043
0.1707	20.0	7884	0.4434	0.9067	0.9129	0.9067	0.9069
0.2138	20.9995	8278	0.5051	0.9037	0.9155	0.9037	0.9053
0.1812	21.9990	8672	0.4238	0.8955	0.9007	0.8955	0.8953
0.3639	22.9985	9066	0.4021	0.9138	0.9182	0.9138	0.9143
0.3193	23.9980	9460	0.4989	0.9168	0.9209	0.9168	0.9166
0.2067	24.9873	9850	0.4959	0.8976	0.9032	0.8976	0.8975

🚀 How to Use

from transformers import AutoModelForAudioClassification, AutoFeatureExtractor
import librosa
import torch
import numpy as np

model_id = "firdhokk/speech-emotion-recognition-with-facebook-wav2vec2-large-xlsr-53"
model = AutoModelForAudioClassification.from_pretrained(model_id)

feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, do_normalize=True, return_attention_mask=True)
id2label = model.config.id2label

def preprocess_audio(audio_path, feature_extractor, max_duration=30.0):
    audio_array, sampling_rate = librosa.load(audio_path, sr=feature_extractor.sampling_rate)
    
    max_length = int(feature_extractor.sampling_rate * max_duration)
    if len(audio_array) > max_length:
        audio_array = audio_array[:max_length]
    else:
        audio_array = np.pad(audio_array, (0, max_length - len(audio_array)))

    inputs = feature_extractor(
        audio_array,
        sampling_rate=feature_extractor.sampling_rate,
        max_length=max_length,
        truncation=True,
        return_attention_mask=True,
        return_tensors="pt",
    )
    return inputs

def predict_emotion(audio_path, model, feature_extractor, id2label, max_duration=30.0):
    inputs = preprocess_audio(audio_path, feature_extractor, max_duration)
    
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model = model.to(device)
    inputs = {key: value.to(device) for key, value in inputs.items()}

    with torch.no_grad():
        outputs = model(**inputs)

    logits = outputs.logits
    predicted_id = torch.argmax(logits, dim=-1).item()
    predicted_label = id2label[predicted_id]
    
    return predicted_label

audio_path = "/content/drive/MyDrive/Audio/Speech_URDU/Happy/SM5_F4_H058.wav"

predicted_emotion = predict_emotion(audio_path, model, feature_extractor, id2label)
print(f"Predicted Emotion: {predicted_emotion}")

🎯 Framework versions

Transformers 4.44.2
Pytorch 2.4.1+cu121
Datasets 3.0.0
Tokenizers 0.19.1