--- library_name: transformers language: - da license: openrail base_model: facebook/wav2vec2-xls-r-300m datasets: - alexandrainst/coral metrics: - wer - cer model-index: - name: roest-315m results: - task: name: Automatic Speech Recognition type: automatic-speech-recognition dataset: name: CoRal read-aloud type: alexandrainst/coral split: test args: read_aloud metrics: - name: CER type: cer value: 6.6% ± 0.2% - name: WER type: wer value: 17.0% ± 0.4% - task: name: Automatic Speech Recognition type: automatic-speech-recognition dataset: name: Danish Common Voice 17 type: mozilla-foundation/common_voice_17_0 split: test args: da metrics: - name: CER type: cer value: 6.6% ± 0.6% - name: WER type: wer value: 16.7% ± 0.8% pipeline_tag: automatic-speech-recognition --- # Røst-315m This is a Danish state-of-the-art speech recognition model, trained by [the Alexandra Institute](https://alexandra.dk/). Try it out in [our interactive demo](https://huggingface.co/spaces/alexandrainst/roest-demo)! ## Quick Start Start by installing the required libraries: ```shell $ pip install transformers kenlm pyctcdecode ``` Next you can use the model using the `transformers` Python package as follows: ```python >>> from transformers import pipeline >>> audio = get_audio() # 16kHz raw audio array >>> transcriber = pipeline(model="alexandrainst/roest-315m") >>> transcriber(audio) {'text': 'your transcription'} ``` ## Evaluation Results We have evaluated both our and existing models on the CoRal test set as well as the Danish Common Voice 17 test set. To ensure as robust an evaluation as possible, we have bootstrapped the results 1000 times and report here the mean scores along with a 95% confidence interval (lower is better; best scores in **bold**, second-best in *italics*): | Model | Number of parameters | [CoRal](https://huggingface.co/datasets/alexandrainst/coral/viewer/read_aloud/test) CER | [CoRal](https://huggingface.co/datasets/alexandrainst/coral/viewer/read_aloud/test) WER | [Danish Common Voice 17](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0/viewer/da/test) CER | [Danish Common Voice 17](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0/viewer/da/test) WER | |:---|---:|---:|---:|---:|---:| | Røst-315m (this model) | 315M | **6.6% ± 0.2%** | **17.0% ± 0.4%** | 6.6% ± 0.6% | 16.7% ± 0.8% | | [chcaa/xls-r-300m-danish-nst-cv9](https://hf.co/chcaa/xls-r-300m-danish-nst-cv9) | 315M | 14.4% ± 0.3% | 36.5% ± 0.6% | **4.1% ± 0.5%** | **12.0% ± 0.8%** | | [mhenrichsen/hviske](https://hf.co/mhenrichsen/hviske) | 1540M | 14.2% ± 0.5% | 33.2% ± 0.7% | *5.2% ± 0.4%* | *14.2% ± 0.8%* | | [openai/whisper-large-v3](https://hf.co/openai/whisper-large-v3) | 1540M | *11.4% ± 0.3%* | *28.3% ± 0.6%* | *5.5% ± 0.4%* | *14.8% ± 0.8%* | | [openai/whisper-large-v2](https://hf.co/openai/whisper-large-v2) | 1540M | 13.9% ± 0.9% | 32.6% ± 1.2% | 7.2% ± 0.5% | 18.5% ± 0.9% | | [openai/whisper-large](https://hf.co/openai/whisper-large) | 1540M | 14.5% ± 0.3% | 35.4% ± 0.6% | 9.2% ± 0.5% | 22.9% ± 1.0% | | [openai/whisper-medium](https://hf.co/openai/whisper-medium) | 764M | 17.2% ± 1.3% | 40.5% ± 2.1% | 9.4% ± 0.5% | 24.0% ± 1.0% | | [openai/whisper-small](https://hf.co/openai/whisper-small) | 242M | 23.4% ± 1.2% | 55.2% ± 2.3% | 15.9% ± 1.0% | 38.9% ± 1.2% | | [openai/whisper-base](https://hf.co/openai/whisper-base) | 73M | 43.5% ± 3.1% | 89.3% ± 4.6% | 33.4% ± 4.7% | 71.4% ± 7.0% | | [openai/whisper-tiny](https://hf.co/openai/whisper-tiny) | 38M | 52.0% ± 2.5% | 103.7% ± 3.5% | 42.2% ± 3.9% | 83.6% ± 2.7% | ### Detailed Evaluation Across Demographics on the CoRal Test Set ![CER comparison plot](https://filedn.com/lRBwPhPxgV74tO0rDoe8SpH/coral/roest-xlsr-comparison-cer-plot.png) ![WER comparison plot](https://filedn.com/lRBwPhPxgV74tO0rDoe8SpH/coral/roest-xlsr-comparison-wer-plot.png) ## Training Data This model is the result of four different stages of training: 1. "Pretraining" on 436,000 hours of unlabelled multilingual publicly available data, 13,628 hours of which is Danish. Pretraining here means that the model learnt to "fill in" gaps of raw audio - no transcriptions were used (or available) during this process. The pretraining data is distributed as follows: - 372,000 hours from [VoxPopuli](https://aclanthology.org/2021.acl-long.80/), being speeches from the European Parliament in 23 European languages. This includes 13,600 hours of Danish speech. - 51,000 hours from [Multilingual LibriSpeech](https://doi.org/10.21437/Interspeech.2020-2826), being audiobooks in 8 European languages. This does not include any Danish speech. - 7,000 hours from [Common Voice 6](https://doi.org/10.48550/arXiv.1912.06670), being read-aloud speech in 60 diverse languages. This does not include any Danish speech. - 6,600 hours from [VoxLingua107](https://doi.org/10.1109/SLT48900.2021.9383459), being audio from YouTube videos in 107 languages. This includes 28 hours of Danish speech. - 1,000 hours from [BABEL](https://eprints.whiterose.ac.uk/152840/), being conversational telephone speech in 17 African and Asian languages. This does not include any Danish speech. 2. "Finetuning" on 373 hours of labelled Danish publicly available data. "Finetuning" indicates that this stage of training was supervised, i.e. the model was trained on both audio and transcriptions to perform the speech-to-text task (also known as automatic speech recognition). The finetuning data is as follows: - The read-aloud training split of the [CoRal dataset](https://huggingface.co/datasets/alexandrainst/coral) (revision fb20199b3966d3373e0d3a5ded2c5920c70de99c), consisting of 361 hours of Danish read-aloud speech, diverse across dialects, accents, ages and genders. 3. An n-gram language model has been trained separately, and is used to guide the transcription generation of the finetuned speech recognition model. This n-gram language model has been trained on the following datasets: - [Danish Wikipedia](https://huggingface.co/datasets/alexandrainst/scandi-wiki/viewer/da) (approximately 287,000 articles). - [Danish Common Voice 17 training split](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0/viewer/da) (approximately 3,500 comments). - [Danish Reddit](https://huggingface.co/datasets/alexandrainst/scandi-reddit/viewer/da) (approximately 5 million comments). Note that all samples from the CoRal test dataset have been removed from all of these datasets, to ensure that the n-gram model has not seen the test data. The first step was trained by [Babu et al. (2021)](https://doi.org/10.48550/arXiv.2111.09296) and the second and third step by [Nielsen et al. (2024)](https://huggingface.co/alexandrainst/roest-315m). The final product is then the combination of the finetuned model along with the n-gram model, and this is what is used when you use the model as mentioned in the Quick Start section above. ## Intended use cases This model is intended to be used for Danish automatic speech recognition. Note that Biometric Identification is not allowed using the CoRal dataset and/or derived models. For more information, see addition 4 in our [license](https://huggingface.co/alexandrainst/roest-315m/blob/main/LICENSE). ## Why the name Røst? Røst is both the [Danish word for the human voice](https://ordnet.dk/ddo/ordbog?query=r%C3%B8st) as well as being the name of [one of the cold-water coral reefs in Scandinavia](https://da.wikipedia.org/wiki/Koralrev#Koldtvandskoralrev). ## License The dataset is licensed under a custom license, adapted from OpenRAIL-M, which allows commercial use with a few restrictions (speech synthesis and biometric identification). See [license](https://huggingface.co/alexandrainst/roest-315m/blob/main/LICENSE). ## Creators and Funders The CoRal project is funded by the [Danish Innovation Fund](https://innovationsfonden.dk/) and consists of the following partners: - [Alexandra Institute](https://alexandra.dk/) - [University of Copenhagen](https://www.ku.dk/) - [Agency for Digital Government](https://digst.dk/) - [Alvenir](https://www.alvenir.ai/) - [Corti](https://www.corti.ai/) ## Citation We will submit a research paper soon, but until then, if you use this model in your research or development, please cite it as follows: ```bibtex @dataset{coral2024, author = {Dan Saattrup Nielsen, Sif Bernstorff Lehmann, Simon Leminen Madsen, Anders Jess Pedersen, Anna Katrine van Zee, Anders Søgaard and Torben Blach}, title = {CoRal: A Diverse Danish ASR Dataset Covering Dialects, Accents, Genders, and Age Groups}, year = {2024}, url = {https://hf.co/datasets/alexandrainst/coral}, } ```