|
--- |
|
language: multilingual |
|
thumbnail: |
|
tags: |
|
- audio-classification |
|
- speechbrain |
|
- embeddings |
|
- Language |
|
- Identification |
|
- pytorch |
|
- ECAPA-TDNN |
|
- TDNN |
|
- VoxLingua107 |
|
license: "apache-2.0" |
|
datasets: |
|
- VoxLingua107 |
|
metrics: |
|
- Accuracy |
|
widget: |
|
- label: English Sample |
|
src: https://cdn-media.huggingface.co/speech_samples/LibriSpeech_61-70968-0000.flac |
|
--- |
|
|
|
# VoxLingua107 ECAPA-TDNN Spoken Language Identification Model |
|
|
|
## Model description |
|
|
|
This is a spoken language recognition model trained on the VoxLingua107 dataset using SpeechBrain. |
|
The model uses the ECAPA-TDNN architecture that has previously been used for speaker recognition. |
|
|
|
The model can classify a speech utterance according to the language spoken. |
|
It covers 107 different languages ( |
|
Abkhazian, |
|
Afrikaans, |
|
Amharic, |
|
Arabic, |
|
Assamese, |
|
Azerbaijani, |
|
Bashkir, |
|
Belarusian, |
|
Bulgarian, |
|
Bengali, |
|
Tibetan, |
|
Breton, |
|
Bosnian, |
|
Catalan, |
|
Cebuano, |
|
Czech, |
|
Welsh, |
|
Danish, |
|
German, |
|
Greek, |
|
English, |
|
Esperanto, |
|
Spanish, |
|
Estonian, |
|
Basque, |
|
Persian, |
|
Finnish, |
|
Faroese, |
|
French, |
|
Galician, |
|
Guarani, |
|
Gujarati, |
|
Manx, |
|
Hausa, |
|
Hawaiian, |
|
Hindi, |
|
Croatian, |
|
Haitian, |
|
Hungarian, |
|
Armenian, |
|
Interlingua, |
|
Indonesian, |
|
Icelandic, |
|
Italian, |
|
Hebrew, |
|
Japanese, |
|
Javanese, |
|
Georgian, |
|
Kazakh, |
|
Central Khmer, |
|
Kannada, |
|
Korean, |
|
Latin, |
|
Luxembourgish, |
|
Lingala, |
|
Lao, |
|
Lithuanian, |
|
Latvian, |
|
Malagasy, |
|
Maori, |
|
Macedonian, |
|
Malayalam, |
|
Mongolian, |
|
Marathi, |
|
Malay, |
|
Maltese, |
|
Burmese, |
|
Nepali, |
|
Dutch, |
|
Norwegian Nynorsk, |
|
Norwegian, |
|
Occitan, |
|
Panjabi, |
|
Polish, |
|
Pushto, |
|
Portuguese, |
|
Romanian, |
|
Russian, |
|
Sanskrit, |
|
Scots, |
|
Sindhi, |
|
Sinhala, |
|
Slovak, |
|
Slovenian, |
|
Shona, |
|
Somali, |
|
Albanian, |
|
Serbian, |
|
Sundanese, |
|
Swedish, |
|
Swahili, |
|
Tamil, |
|
Telugu, |
|
Tajik, |
|
Thai, |
|
Turkmen, |
|
Tagalog, |
|
Turkish, |
|
Tatar, |
|
Ukrainian, |
|
Urdu, |
|
Uzbek, |
|
Vietnamese, |
|
Waray, |
|
Yiddish, |
|
Yoruba, |
|
Mandarin Chinese). |
|
|
|
## Intended uses & limitations |
|
|
|
The model has two uses: |
|
|
|
- use 'as is' for spoken language recognition |
|
- use as an utterance-level feature (embedding) extractor, for creating a dedicated language ID model on your own data |
|
|
|
The model is trained on automatically collected YouTube data. For more |
|
information about the dataset, see [here](http://bark.phon.ioc.ee/voxlingua107/). |
|
|
|
|
|
#### How to use |
|
|
|
```python |
|
import torchaudio |
|
from speechbrain.pretrained import EncoderClassifier |
|
language_id = EncoderClassifier.from_hparams(source="TalTechNLP/voxlingua107-epaca-tdnn", savedir="tmp") |
|
# Download Thai language sample from Omniglot and cvert to suitable form |
|
signal = language_id.load_audio("https://omniglot.com/soundfiles/udhr/udhr_th.mp3") |
|
prediction = language_id.classify_batch(signal) |
|
print(prediction) |
|
(tensor([[0.3210, 0.3751, 0.3680, 0.3939, 0.4026, 0.3644, 0.3689, 0.3597, 0.3508, |
|
0.3666, 0.3895, 0.3978, 0.3848, 0.3957, 0.3949, 0.3586, 0.4360, 0.3997, |
|
0.4106, 0.3886, 0.4177, 0.3870, 0.3764, 0.3763, 0.3672, 0.4000, 0.4256, |
|
0.4091, 0.3563, 0.3695, 0.3320, 0.3838, 0.3850, 0.3867, 0.3878, 0.3944, |
|
0.3924, 0.4063, 0.3803, 0.3830, 0.2996, 0.4187, 0.3976, 0.3651, 0.3950, |
|
0.3744, 0.4295, 0.3807, 0.3613, 0.4710, 0.3530, 0.4156, 0.3651, 0.3777, |
|
0.3813, 0.6063, 0.3708, 0.3886, 0.3766, 0.4023, 0.3785, 0.3612, 0.4193, |
|
0.3720, 0.4406, 0.3243, 0.3866, 0.3866, 0.4104, 0.4294, 0.4175, 0.3364, |
|
0.3595, 0.3443, 0.3565, 0.3776, 0.3985, 0.3778, 0.2382, 0.4115, 0.4017, |
|
0.4070, 0.3266, 0.3648, 0.3888, 0.3907, 0.3755, 0.3631, 0.4460, 0.3464, |
|
0.3898, 0.3661, 0.3883, 0.3772, 0.9289, 0.3687, 0.4298, 0.4211, 0.3838, |
|
0.3521, 0.3515, 0.3465, 0.4772, 0.4043, 0.3844, 0.3973, 0.4343]]), tensor([0.9289]), tensor([94]), ['th']) |
|
# The scores in the prediction[0] tensor can be interpreted as cosine scores between |
|
# the languages and the given utterance (i.e., the larger the better) |
|
# The identified language ISO code is given in prediction[3] |
|
print(prediction[3]) |
|
['th'] |
|
|
|
# Alternatively, use the utterance embedding extractor: |
|
emb = language_id.encode_batch(signal) |
|
print(emb.shape) |
|
torch.Size([1, 1, 256]) |
|
``` |
|
|
|
#### Limitations and bias |
|
|
|
Since the model is trained on VoxLingua107, it has many limitations and biases, some of which are: |
|
|
|
- Probably it's accuracy on smaller languages is quite limited |
|
- Probably it works worse on female speech than male speech (because YouTube data includes much more male speech) |
|
- Based on subjective experiments, it doesn't work well on speech with a foreign accent |
|
- Probably it doesn't work well on children's speech and on persons with speech disorders |
|
|
|
|
|
## Training data |
|
|
|
The model is trained on [VoxLingua107](http://bark.phon.ioc.ee/voxlingua107/). |
|
|
|
VoxLingua107 is a speech dataset for training spoken language identification models. |
|
The dataset consists of short speech segments automatically extracted from YouTube videos and labeled according the language of the video title and description, with some post-processing steps to filter out false positives. |
|
|
|
VoxLingua107 contains data for 107 languages. The total amount of speech in the training set is 6628 hours. |
|
The average amount of data per language is 62 hours. However, the real amount per language varies a lot. There is also a seperate development set containing 1609 speech segments from 33 languages, validated by at least two volunteers to really contain the given language. |
|
|
|
## Training procedure |
|
|
|
We used [SpeechBrain](https://github.com/speechbrain/speechbrain) to train the model. |
|
Training recipe will be published soon. |
|
|
|
## Evaluation results |
|
|
|
Error rate: 7% on the development dataset |
|
|
|
|
|
### BibTeX entry and citation info |
|
|
|
```bibtex |
|
@inproceedings{valk2021slt, |
|
title={{VoxLingua107}: a Dataset for Spoken Language Recognition}, |
|
author={J{\"o}rgen Valk and Tanel Alum{\"a}e}, |
|
booktitle={Proc. IEEE SLT Workshop}, |
|
year={2021}, |
|
} |
|
``` |
|
|