ExHuBERT: Enhancing HuBERT Through Block Extension and Fine-Tuning on 37 Emotion Datasets

Authors: Shahin Amiriparian, Filip Packań, Maurice Gerczuk, Björn W. Schuller

Fine-tuned and backbone extended HuBERT Large on EmoSet++, comprising 37 datasets, totaling 150,907 samples and spanning a cumulative duration of 119.5 hours. The model is expecting a 3 second long raw waveform resampled to 16 kHz. The original 6 Ouput classes are combinations of low/high arousal and negative/neutral/positive valence. Further details are available in the corresponding paper.

EmoSet++ subsets used for fine-tuning the model:

ABC [1]	AD [2]	BES [3]	CASIA [4]	CVE [5]
Crema-D [6]	DES [7]	DEMoS [8]	EA-ACT [9]	EA-BMW [9]
EA-WSJ [9]	EMO-DB [10]	EmoFilm [11]	EmotiW-2014 [12]	EMOVO [13]
eNTERFACE [14]	ESD [15]	EU-EmoSS [16]	EU-EV [17]	FAU Aibo [18]
GEMEP [19]	GVESS [20]	IEMOCAP [21]	MES [3]	MESD [22]
MELD [23]	PPMMK [2]	RAVDESS [24]	SAVEE [25]	ShEMO [26]
SmartKom [27]	SIMIS [28]	SUSAS [29]	SUBSECO [30]	TESS [31]
TurkishEmo [2]	Urdu [32]

Usage

import torch
import torch.nn as nn
from transformers import AutoModelForAudioClassification, Wav2Vec2FeatureExtractor


# CONFIG and MODEL SETUP
model_name = 'amiriparian/ExHuBERT'
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("facebook/hubert-base-ls960")
model = AutoModelForAudioClassification.from_pretrained(model_name, trust_remote_code=True,
                                                        revision="b158d45ed8578432468f3ab8d46cbe5974380812")

# Freezing half of the encoder for further transfer learning
model.freeze_og_encoder()

sampling_rate = 16000
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)



# Example application from a local audiofile
import numpy as np
import librosa
import torch.nn.functional as F
# Sample taken from the Toronto emotional speech set (TESS) https://tspace.library.utoronto.ca/handle/1807/24487
waveform, sr_wav = librosa.load("YAF_date_angry.wav")
# Max Padding to 3 Seconds at 16k sampling rate for the best results
waveform = feature_extractor(waveform, sampling_rate=sampling_rate,padding = 'max_length',max_length = 48000)
waveform = waveform['input_values'][0]
waveform = waveform.reshape(1, -1)
waveform = torch.from_numpy(waveform).to(device)
with torch.no_grad():
    output = model(waveform)
    output = F.softmax(output.logits, dim = 1)
    output = output.detach().cpu().numpy().round(2)
    print(output)

    # [[0.      0.      0.      1.      0.      0.]]
    #          Low          |          High                 Arousal
    # Neg.     Neut.   Pos. |  Neg.    Neut.   Pos          Valence
    # Disgust, Neutral, Kind| Anger, Surprise, Joy          Example emotions

Example of How to Train the Model for Transfer Learning

The datasets used for showcasing are EmoDB and IEMOCAP from the HuggingFace Hub. As noted above, the model has seen both datasets before.

import pandas as pd
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import librosa
import io
from transformers import AutoModelForAudioClassification, Wav2Vec2FeatureExtractor

# CONFIG and MODEL SETUP
model_name = 'amiriparian/ExHuBERT'
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("facebook/hubert-base-ls960")
model = AutoModelForAudioClassification.from_pretrained(model_name, trust_remote_code=True,
                                                        revision="b158d45ed8578432468f3ab8d46cbe5974380812")

# Replacing Classifier layer
model.classifier = nn.Linear(in_features=256, out_features=7)
# Freezing the original encoder layers and feature encoder (as in the paper) for further transfer learning
model.freeze_og_encoder()
model.freeze_feature_encoder()
model.train()

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)

# Define a custom dataset class
class EmotionDataset(Dataset):
    def __init__(self, dataframe, feature_extractor, max_length):
        self.dataframe = dataframe
        self.feature_extractor = feature_extractor
        self.max_length = max_length

    def __len__(self):
        return len(self.dataframe)

    def __getitem__(self, idx):
        row = self.dataframe.iloc[idx]
        # emotion = torch.tensor(row['label'], dtype=torch.int64)  # For the IEMOCAP example
        emotion = torch.tensor(row['emotion'], dtype=torch.int64)  # EmoDB specific

        # Decode audio bytes from the Huggingface dataset with librosa
        audio_bytes = row['audio']['bytes']
        audio_buffer = io.BytesIO(audio_bytes)
        audio_data, samplerate = librosa.load(audio_buffer, sr=16000)

        # Use the feature extractor to preprocess the audio. Padding/Truncating to 3 seconds gives better results
        audio_features = self.feature_extractor(audio_data, sampling_rate=16000, return_tensors="pt", padding="max_length",
                                                truncation=True, max_length=self.max_length)

        audio = audio_features['input_values'].squeeze(0)
        return audio, emotion

# Load your DataFrame. Samples are shown for EmoDB and IEMOCAP from the Huggingface Hub
df = pd.read_parquet("hf://datasets/renumics/emodb/data/train-00000-of-00001-cf0d4b1ae18136ff.parquet")
# splits = {'session1': 'data/session1-00000-of-00001-04e11ca668d90573.parquet', 'session2': 'data/session2-00000-of-00001-f6132100b374cb18.parquet', 'session3': 'data/session3-00000-of-00001-6e102fcb5c1126b4.parquet', 'session4': 'data/session4-00000-of-00001-e39531a7c694b50d.parquet', 'session5': 'data/session5-00000-of-00001-03769060403172ce.parquet'}
# df = pd.read_parquet("hf://datasets/Zahra99/IEMOCAP_Audio/" + splits["session1"])

# Dataset and DataLoader
dataset = EmotionDataset(df, feature_extractor, max_length=3 * 16000)
dataloader = DataLoader(dataset, batch_size=4, shuffle=True)

# Training setup
criterion = nn.CrossEntropyLoss()
lr = 1e-5
non_frozen_parameters = [p for p in model.parameters() if p.requires_grad]
optim = torch.optim.AdamW(non_frozen_parameters, lr=lr, betas=(0.9, 0.999), eps=1e-08)

# Function to calculate accuracy
def calculate_accuracy(outputs, targets):
    _, predicted = torch.max(outputs, 1)
    correct = (predicted == targets).sum().item()
    return correct / targets.size(0)

# Training loop
num_epochs = 3
for epoch in range(num_epochs):
    model.train()
    total_loss = 0.0
    total_correct = 0
    total_samples = 0
    for batch_idx, (inputs, targets) in enumerate(dataloader):
        inputs, targets = inputs.to(device), targets.to(device)

        optim.zero_grad()
        outputs = model(inputs).logits
        loss = criterion(outputs, targets)
        loss.backward()
        optim.step()

        total_loss += loss.item()
        total_correct += (outputs.argmax(1) == targets).sum().item()
        total_samples += targets.size(0)

    epoch_loss = total_loss / len(dataloader)
    epoch_accuracy = total_correct / total_samples
    print(f'Epoch [{epoch + 1}/{num_epochs}], Average Loss: {epoch_loss:.4f}, Average Accuracy: {epoch_accuracy:.4f}')

# Example outputs:
# Epoch [3/3], Average Loss: 0.4572, Average Accuracy: 0.8249 for IEMOCAP
# Epoch [3/3], Average Loss: 0.1511, Average Accuracy: 0.9850 for EmoDB

Citation Info

ExHuBERT has been accepted for presentation at INTERSPEECH 2024.

@inproceedings{amiriparian24_interspeech,
  title     = {ExHuBERT: Enhancing HuBERT Through Block Extension and Fine-Tuning on 37 Emotion Datasets},
  author    = {Shahin Amiriparian and Filip Packań and Maurice Gerczuk and Björn W. Schuller},
  year      = {2024},
  booktitle = {Interspeech 2024},
  pages     = {2635--2639},
  doi       = {10.21437/Interspeech.2024-280},
  issn      = {2958-1796},
}

References

[1] B. Schuller, D. Arsic, G. Rigoll, M. Wimmer, and B. Radig. Audiovisual Behavior Modeling by Combined Feature Spaces. Proc. ICASSP 2007, Apr. 2007.

[2] M. Gerczuk, S. Amiriparian, S. Ottl, and B. W. Schuller. EmoNet: A Transfer Learning Framework for Multi-Corpus Speech Emotion Recognition. IEEE Transactions on Affective Computing, 14(2):1472–1487, Apr. 2023.

[3] T. L. Nwe, S. W. Foo, and L. C. De Silva. Speech emotion recognition using hidden Markov models. Speech Communication, 41(4):603–623, Nov. 2003.

[4] The selected speech emotion database of institute of automation chineseacademy of sciences (casia). http://www.chineseldc.org/resource_info.php?rid=76. accessed March 2024.

[5] P. Liu and M. D. Pell. Recognizing vocal emotions in Mandarin Chinese: A validated database of Chinese vocal emotional stimuli. Behavior Research Methods, 44(4):1042–1051, Dec. 2012.

[6] H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma. CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset. IEEE transactions on affective computing, 5(4):377–390, 2014.

[7] I. S. Engberg, A. V. Hansen, O. K. Andersen, and P. Dalsgaard. Design Recording and Verification of a Danish Emotional Speech Database: Design Recording and Verification of a Danish Emotional Speech Database. EUROSPEECH’97 : 5th European Conference on Speech Communication and Technology, Patras, Rhodes, Greece, 22-25 September 1997, pages Vol. 4, pp. 1695–1698, 1997.

[8] E. Parada-Cabaleiro, G. Costantini, A. Batliner, M. Schmitt, and B. W. Schuller. DEMoS: An Italian emotional speech corpus. Language Resources and Evaluation, 54(2):341–383, June 2020.

[9] B. Schuller. Automatische Emotionserkennung Aus Sprachlicher Und Manueller Interaktion. PhD thesis, Technische Universität München, 2006.

[10] F. Burkhardt, A. Paeschke, M. Rolfes, W. F. Sendlmeier, and B. Weiss. A database of German emotional speech. In Interspeech 2005, pages 1517–1520. ISCA, Sept. 2005.

[11] E. Parada-Cabaleiro, G. Costantini, A. Batliner, A. Baird, and B. Schuller. Categorical vs Dimensional Perception of Italian Emotional Speech. In Interspeech 2018, pages 3638–3642. ISCA, Sept. 2018.

[12] A. Dhall, R. Goecke, J. Joshi, K. Sikka, and T. Gedeon. Emotion Recognition In The Wild Challenge 2014: Baseline, Data and Protocol. In Proceedings of the 16th International Conference on Multimodal Interaction, ICMI ’14, pages 461–466, New York, NY, USA, Nov. 2014. Association for Computing Machinery.

[13] G. Costantini, I. Iaderola, A. Paoloni, and M. Todisco. EMOVO Corpus: An Italian Emotional Speech Database. In N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, and S. Piperidis, editors, Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 3501–3504, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA).

[14] O. Martin, I. Kotsia, B. Macq, and I. Pitas. The eNTERFACE’ 05 Audio-Visual Emotion Database. In 22nd International Conference on Data Engineering Workshops (ICDEW’06), pages 8–8, Apr. 2006.

[15] K. Zhou, B. Sisman, R. Liu, and H. Li. Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset, Feb. 2021.

[16] H. O’Reilly, D. Pigat, S. Fridenson, S. Berggren, S. Tal, O. Golan, S. Bölte, S. Baron-Cohen, and D. Lundqvist. The EU-Emotion Stimulus Set: A validation study. Behavior Research Methods, 48(2):567–576, June 2016.

[17] A. Lassalle, D. Pigat, H. O’Reilly, S. Berggen, S. Fridenson-Hayo, S. Tal, S. Elfström, A. Rade, O. Golan, S. Bölte, S. Baron-Cohen, and D. Lundqvist. The EU-Emotion Voice Database. Behavior Research Methods, 51(2):493–506, Apr. 2019.

[18] A. Batliner, S. Steidl, and E. Noth. Releasing a thoroughly annotated and processed spontaneous emotional database: The FAU Aibo Emotion Corpus. 2008.

[19] K. R. Scherer, T. B¨anziger, and E. Roesch. A Blueprint for Affective Computing: A Sourcebook and Manual. OUP Oxford, Sept. 2010.

[20] R. Banse and K. R. Scherer. Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology, 70(3):614–636, 1996.

[21] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan. IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4):335–359, Dec. 2008.

[22] M. M. Duville, L. M. Alonso-Valerdi, and D. I. Ibarra-Zarate. The Mexican Emotional Speech Database (MESD): Elaboration and assessment based on machine learning. Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference, 2021:1644–1647, Nov. 2021.

[23] S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea. MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations, June 2019.

[24] S. R. Livingstone and F. A. Russo. The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLOS ONE, 13(5):e0196391, May 2018.

[25] S. Haq and P. J. B. Jackson. Speaker-dependent audio-visual emotion recognition. In Proc. AVSP 2009, pages 53–58, 2009.

[26] O. Mohamad Nezami, P. Jamshid Lou, and M. Karami. ShEMO: A large-scale validated database for Persian speech emotion detection. Language Resources and Evaluation, 53(1):1–16, Mar. 2019.

[27] F. Schiel, S. Steininger, and U. T¨urk. The SmartKom Multimodal Corpus at BAS. In M. González Rodríguez and C. P. Suarez Araujo, editors, Proceedings of the Third International Conference on Language Resources and Evaluation (LREC’02), Las Palmas, Canary Islands - Spain, May 2002. European Language Resources Association (ELRA).

[28] B. Schuller, F. Eyben, S. Can, and H. Feussner. Speech in Minimal Invasive Surgery - Towards an Affective Language Resource of Real-life Medical Operations. 2010.

[29] J. H. L. Hansen and S. E. Bou-Ghazale. Getting started with SUSAS: A speech under simulated and actual stress database. In Proc. Eurospeech 1997, pages 1743–1746, 1997.

[30] S. Sultana, M. S. Rahman, M. R. Selim, and M. Z. Iqbal. SUST Bangla Emotional Speech Corpus (SUBESCO): An audio-only emotional speech corpus for Bangla. PLOS ONE, 16(4):e0250173, Apr. 2021.

[31] M. K. Pichora-Fuller and K. Dupuis. Toronto emotional speech set (TESS), Feb. 2020.

[32] S. Latif, A. Qayyum, M. Usman, and J. Qadir. Cross Lingual Speech Emotion Recognition: Urdu vs. Western Languages. In 2018 International Conference on Frontiers of Information Technology (FIT), pages 88–93, Dec. 2018.