Text-to-Speech
Transformers
Safetensors
parler_tts
text2text-generation
annotation
ylacombe commited on
Commit
4135e6c
·
verified ·
1 Parent(s): 990f309

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +304 -122
README.md CHANGED
@@ -1,199 +1,381 @@
1
  ---
2
  library_name: transformers
3
- tags: []
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4
  ---
5
 
6
- # Model Card for Model ID
7
 
8
- <!-- Provide a quick summary of what the model is/does. -->
9
 
 
 
 
10
 
 
11
 
12
- ## Model Details
13
 
14
- ### Model Description
15
 
16
- <!-- Provide a longer summary of what this model is. -->
17
 
18
- This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
19
 
20
- - **Developed by:** [More Information Needed]
21
- - **Funded by [optional]:** [More Information Needed]
22
- - **Shared by [optional]:** [More Information Needed]
23
- - **Model type:** [More Information Needed]
24
- - **Language(s) (NLP):** [More Information Needed]
25
- - **License:** [More Information Needed]
26
- - **Finetuned from model [optional]:** [More Information Needed]
27
 
28
- ### Model Sources [optional]
 
 
 
 
 
 
 
 
 
29
 
30
- <!-- Provide the basic links for the model. -->
31
 
32
- - **Repository:** [More Information Needed]
33
- - **Paper [optional]:** [More Information Needed]
34
- - **Demo [optional]:** [More Information Needed]
35
 
36
- ## Uses
 
 
37
 
38
- <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
39
 
40
- ### Direct Use
 
 
41
 
42
- <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
43
 
44
- [More Information Needed]
 
 
45
 
46
- ### Downstream Use [optional]
 
 
 
 
 
 
 
47
 
48
- <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
 
49
 
50
- [More Information Needed]
 
51
 
52
- ### Out-of-Scope Use
 
 
 
 
 
53
 
54
- <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
55
 
56
- [More Information Needed]
57
 
58
- ## Bias, Risks, and Limitations
59
 
60
- <!-- This section is meant to convey both technical and sociotechnical limitations. -->
61
 
62
- [More Information Needed]
 
 
 
 
63
 
64
- ### Recommendations
65
 
66
- <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
 
67
 
68
- Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
69
 
70
- ## How to Get Started with the Model
 
71
 
72
- Use the code below to get started with the model.
 
 
 
73
 
74
- [More Information Needed]
75
 
76
- ## Training Details
 
 
 
 
 
 
 
77
 
78
- ### Training Data
79
 
80
- <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
81
 
82
- [More Information Needed]
83
 
84
- ### Training Procedure
 
 
 
 
85
 
86
- <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
87
 
88
- #### Preprocessing [optional]
 
 
89
 
90
- [More Information Needed]
 
91
 
 
 
92
 
93
- #### Training Hyperparameters
 
 
 
94
 
95
- - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
96
 
97
- #### Speeds, Sizes, Times [optional]
 
98
 
99
- <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
 
 
 
 
100
 
101
- [More Information Needed]
102
 
103
- ## Evaluation
 
104
 
105
- <!-- This section describes the evaluation protocols and provides the results. -->
 
106
 
107
- ### Testing Data, Factors & Metrics
 
108
 
109
- #### Testing Data
 
 
 
110
 
111
- <!-- This should link to a Dataset Card if possible. -->
112
 
113
- [More Information Needed]
114
 
115
- #### Factors
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
116
 
117
- <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
118
 
119
- [More Information Needed]
 
 
 
 
120
 
121
- #### Metrics
122
 
123
- <!-- These are the evaluation metrics being used, ideally with a description of why. -->
124
 
125
- [More Information Needed]
 
126
 
127
- ### Results
 
128
 
129
- [More Information Needed]
 
130
 
131
- #### Summary
 
132
 
 
 
133
 
 
 
134
 
135
- ## Model Examination [optional]
 
136
 
137
- <!-- Relevant interpretability work for the model goes here -->
 
138
 
139
- [More Information Needed]
 
140
 
141
- ## Environmental Impact
 
142
 
143
- <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
144
 
145
- Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
146
 
147
- - **Hardware Type:** [More Information Needed]
148
- - **Hours used:** [More Information Needed]
149
- - **Cloud Provider:** [More Information Needed]
150
- - **Compute Region:** [More Information Needed]
151
- - **Carbon Emitted:** [More Information Needed]
152
 
153
- ## Technical Specifications [optional]
154
 
155
- ### Model Architecture and Objective
156
 
157
- [More Information Needed]
158
 
159
- ### Compute Infrastructure
160
-
161
- [More Information Needed]
162
-
163
- #### Hardware
164
-
165
- [More Information Needed]
166
-
167
- #### Software
168
-
169
- [More Information Needed]
170
-
171
- ## Citation [optional]
172
-
173
- <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
174
-
175
- **BibTeX:**
176
-
177
- [More Information Needed]
178
-
179
- **APA:**
180
-
181
- [More Information Needed]
182
-
183
- ## Glossary [optional]
184
-
185
- <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
186
-
187
- [More Information Needed]
188
-
189
- ## More Information [optional]
190
-
191
- [More Information Needed]
192
-
193
- ## Model Card Authors [optional]
194
-
195
- [More Information Needed]
196
-
197
- ## Model Card Contact
198
-
199
- [More Information Needed]
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  library_name: transformers
3
+ tags:
4
+ - text-to-speech
5
+ - annotation
6
+ license: apache-2.0
7
+ language:
8
+ - en
9
+ - as
10
+ - bn
11
+ - gu
12
+ - hi
13
+ - kn
14
+ - ks
15
+ - or
16
+ - ml
17
+ - mr
18
+ - ne
19
+ - pa
20
+ - sa
21
+ - sd
22
+ - ta
23
+ - te
24
+ - ur
25
+ - om
26
+ pipeline_tag: text-to-speech
27
+ inference: false
28
+ datasets:
29
+ - ai4b-hf/GLOBE-annotated
30
  ---
31
 
32
+ <img src="https://huggingface.co/datasets/parler-tts/images/resolve/main/thumbnail.png" alt="Parler Logo" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
33
 
34
+ # Indic Parler-TTS
35
 
36
+ <a target="_blank" href="https://huggingface.co/spaces/PHBJT/multi_parler_tts">
37
+ <img src="https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm.svg" alt="Open in HuggingFace"/>
38
+ </a>
39
 
40
+ **Indic Parler-TTS** is a multilingual Indic extension of [Parler-TTS Mini](https://huggingface.co/parler-tts/parler-tts-mini-v1.1).
41
 
42
+ It is a fine-tuned version of [Indic Parler-TTS Pretrained](https://huggingface.co/ai4b-hf/indic-parler-tts-pretrained-v3), trained on a **1,806 hours** multilingual Indic and English dataset.
43
 
44
+ **Indic Parler-TTS Mini** can officially speak in 20 Indic languages, making it comprehensive for regional language technologies, and in English. The **21 languages** supported are: Assamese, Bengali, Bodo, Dogri, English, Gujarati, Hindi, Kannada, Konkani, Maithili, Malayalam, Manipuri, Marathi, Nepali, Odia, Sanskrit, Santali, Sindhi, Tamil, Telugu, and Urdu.
45
 
46
+ Thanks to its **better prompt tokenizer**, it can easily be extended to other languages. This tokenizer has a larger vocabulary and handles byte fallback, which simplifies multilingual training.
47
 
48
+ 🚨 This work is the result of a collaboration between the **HuggingFace audio team** and the **[AI4Bharat](https://ai4bharat.iitm.ac.in/) team**. 🚨
49
 
 
 
 
 
 
 
 
50
 
51
+ ## 📖 Quick Index
52
+ * [👨‍💻 Installation](#👨‍💻-installation)
53
+ * [🛠️ Key capabilities](#🛠️-key-capabilities)
54
+ * [🎲 Using a random voice](#🎲-random-voice)
55
+ * [🌍 Switching languages](#[🌍-switching-languages)
56
+ * [🎯 Using a specific speaker](#🎯-using-a-specific-speaker)
57
+ * [Some Description Examples](#some-description-examples)
58
+ * [📐Evaluation](#📐-evaluation)
59
+ * [Motivation](#motivation)
60
+ * [Optimizing inference](https://github.com/huggingface/parler-tts/blob/main/INFERENCE.md)
61
 
62
+ ### 👨‍💻 Installation
63
 
64
+ Using Parler-TTS is as simple as "bonjour". Simply install the library once:
 
 
65
 
66
+ ```sh
67
+ pip install git+https://github.com/huggingface/parler-tts.git
68
+ ```
69
 
70
+ ## 🛠️ Key capabilities
71
 
72
+ The model accepts two primary inputs:
73
+ 1. **Transcript** - The text to be converted to speech.
74
+ 2. **Caption** - A detailed description of how the speech should sound, e.g., "Leela speaks in a high-pitched, fast-paced, and cheerful tone, full of energy and happiness. The recording is very high quality with no background noise."
75
 
76
+ ### Key Features
77
 
78
+ 1. **Language Support**
79
+ - **Officially supported languages**: Assamese, Bengali, Bodo, Dogri, Kannada, Malayalam, Marathi, Sanskrit, Nepali, English, Telugu, Hindi, Gujarati, Konkani, Maithili, Manipuri, Odia, Santali, Sindhi, Tamil, and Urdu.
80
+ - **Unofficial support**: Chhattisgarhi, Kashmiri, Punjabi.
81
 
82
+ 2. **Speaker Diversity**
83
+ - **69 unique voices** across the supported languages.
84
+ - Supported languages have a set of **recommended voices** optimized for naturalness and intelligibility.
85
+
86
+ 3. **Emotion Rendering**
87
+ - **10 languages** officially support emotion-specific prompts: Assamese, Bengali, Bodo, Dogri, Kannada, Malayalam, Marathi, Sanskrit, Nepali, and Tamil.
88
+ - Emotion support for other languages exists but has not been extensively tested.
89
+ - **Available emotions** include: Command, Anger, Narration, Conversation, Disgust, Fear, Happy, Neutral, Proper Noun, News, Sad, and Surprise.
90
 
91
+ 4. **Accent Flexibility**
92
+ - The model **officially supports Indian English accents** through its English voices, providing clear and natural speech.
93
+ - For other accents, the model allows customization by specifying accent details, such as "A male British speaker" or "A female American speaker," using style transfer for more dynamic and personalized outputs.
94
 
95
+ 5. **Customizable Output**
96
+ Indic Parler-TTS offers precise control over various speech characteristics using the **caption** input:
97
 
98
+ - **Background Noise**: Adjust the noise level in the audio, from clear to slightly noisy environments.
99
+ - **Reverberation**: Control the perceived distance of the voice, from close-sounding to distant-sounding speech.
100
+ - **Expressivity**: Specify how dynamic or monotone the speech should be, ranging from expressive to slightly expressive or monotone.
101
+ - **Pitch**: Modify the pitch of the speech, including high, low, or balanced tones.
102
+ - **Speaking Rate**: Change the speaking rate, from slow to fast.
103
+ - **Voice Quality**: Control the overall clarity and naturalness of the speech, adjusting from basic to refined voice quality.
104
 
 
105
 
106
+ ## 🎲 Random voice
107
 
108
+ 🚨 Unlike previous versions of Parler-TTS, here we use two tokenizers - one for the prompt and one for the description. 🚨
109
 
110
+ **Indic Parler-TTS** has been trained to generate speech with features that can be controlled with a simple text prompt, for example:
111
 
112
+ ```py
113
+ import torch
114
+ from parler_tts import ParlerTTSForConditionalGeneration
115
+ from transformers import AutoTokenizer
116
+ import soundfile as sf
117
 
118
+ device = "cuda:0" if torch.cuda.is_available() else "cpu"
119
 
120
+ model = ParlerTTSForConditionalGeneration.from_pretrained("ai4b-hf/indic-parler-tts-pretrained-then-finetuned").to(device)
121
+ tokenizer = AutoTokenizer.from_pretrained("ai4b-hf/indic-parler-tts-pretrained-then-finetuned")
122
+ description_tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder._name_or_path)
123
 
124
+ prompt = "Hey, how are you doing today?"
125
+ description = "A female speaker with a British accent delivers a slightly expressive and animated speech with a moderate speed and pitch. The recording is of very high quality, with the speaker's voice sounding clear and very close up."
126
 
127
+ input_ids = description_tokenizer(description, return_tensors="pt").input_ids.to(device)
128
+ prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
129
 
130
+ generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)
131
+ audio_arr = generation.cpu().numpy().squeeze()
132
+ sf.write("indic_tts_out.wav", audio_arr, model.config.sampling_rate)
133
+ ```
134
 
135
+ Indic Parler-TTS provides highly effective control over key aspects of speech synthesis using descriptive captions. Below is a summary of what each control parameter can achieve:
136
 
137
+ | **Control Type** | **Capabilities** |
138
+ |--------------------------|----------------------------------------------------------------------------------|
139
+ | **Background Noise** | Adjusts the level of background noise, supporting clear and slightly noisy environments. |
140
+ | **Reverberation** | Controls the perceived distance of the speaker’s voice, allowing close or distant sounds. |
141
+ | **Expressivity** | Modulates the emotional intensity of speech, from monotone to highly expressive. |
142
+ | **Pitch** | Varies the pitch to achieve high, low, or moderate tonal output. |
143
+ | **Speaking Rate** | Changes the speed of speech delivery, ranging from slow to fast-paced. |
144
+ | **Speech Quality** | Improves or degrades the overall audio clarity, supporting basic to refined outputs. |
145
 
 
146
 
147
+ ## 🌍 Switching languages
148
 
149
+ The model automatically adapts to the language it detects in the prompt. You don't need to specify the language you want to use. For example, to switch to Hindi, simply use an Hindi prompt:
150
 
151
+ ```py
152
+ import torch
153
+ from parler_tts import ParlerTTSForConditionalGeneration
154
+ from transformers import AutoTokenizer
155
+ import soundfile as sf
156
 
157
+ device = "cuda:0" if torch.cuda.is_available() else "cpu"
158
 
159
+ model = ParlerTTSForConditionalGeneration.from_pretrained("ai4b-hf/indic-parler-tts-pretrained-then-finetuned").to(device)
160
+ tokenizer = AutoTokenizer.from_pretrained("ai4b-hf/indic-parler-tts-pretrained-then-finetuned")
161
+ description_tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder._name_or_path)
162
 
163
+ prompt = "अरे, तुम आज कैसे हो?"
164
+ description = "A female speaker delivers a slightly expressive and animated speech with a moderate speed and pitch. The recording is of very high quality, with the speaker's voice sounding clear and very close up."
165
 
166
+ input_ids = description_tokenizer(description, return_tensors="pt").input_ids.to(device)
167
+ prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
168
 
169
+ generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)
170
+ audio_arr = generation.cpu().numpy().squeeze()
171
+ sf.write("indic_tts_out.wav", audio_arr, model.config.sampling_rate)
172
+ ```
173
 
174
+ ## 🎯 Using a specific speaker
175
 
176
+ To ensure speaker consistency across generations, this checkpoint was also trained on pre-determined speakers, characterized by name (e.g. Rohit, Karan, Leela, Maya, Sita, ...).
177
+ To take advantage of this, simply adapt your text description to specify which speaker to use: `Divya's voice is monotone yet slightly fast in delivery, with a very close recording that almost has no background noise.`
178
 
179
+ ```py
180
+ import torch
181
+ from parler_tts import ParlerTTSForConditionalGeneration
182
+ from transformers import AutoTokenizer
183
+ import soundfile as sf
184
 
185
+ device = "cuda:0" if torch.cuda.is_available() else "cpu"
186
 
187
+ model = ParlerTTSForConditionalGeneration.from_pretrained("parler-tts/parler-tts-mini-v1").to(device)
188
+ tokenizer = AutoTokenizer.from_pretrained("parler-tts/parler-tts-mini-v1")
189
 
190
+ prompt = "अरे, तुम आज कैसे हो?"
191
+ description = "Divya's voice is monotone yet slightly fast in delivery, with a very close recording that almost has no background noise."
192
 
193
+ input_ids = tokenizer(description, return_tensors="pt").input_ids.to(device)
194
+ prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
195
 
196
+ generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)
197
+ audio_arr = generation.cpu().numpy().squeeze()
198
+ sf.write("parler_tts_out.wav", audio_arr, model.config.sampling_rate)
199
+ ```
200
 
201
+ The model includes **69 speakers** across 18 officially supported languages, with each language having a set of recommended voices for optimal performance. Below is a table summarizing the available speakers for each language, along with the recommended ones.
202
 
203
+ Here is the table based on the provided data:
204
 
205
+ | **Language** | **Available Speakers** | **Recommended Speakers** |
206
+ |--------------------|-------------------------------------------------------------|---------------------------------|
207
+ | Assamese | Amit, Sita, Poonam, Rakesh | Amit, Sita |
208
+ | Bengali | Arjun, Aditi, Tapan, Rashmi, Arnav, Riya | Arjun, Aditi |
209
+ | Bodo | Bikram, Maya, Kalpana | Bikram, Maya |
210
+ | Chhattisgarhi | Bhanu, Champa | Bhanu, Champa |
211
+ | Dogri | Karan | Karan |
212
+ | English | Thoma, Mary, Swapna, Dinesh, Meera, Jatin, Aakash, Sneha, Kabir, Tisha, Chingkhei, Thoiba, Priya, Tarun, Gauri, Nisha, Raghav, Kavya, Ravi, Vikas, Riya | Thoma, Mary |
213
+ | Gujarati | Yash, Neha | Yash, Neha |
214
+ | Hindi | Rohit, Divya, Aman, Rani | Rohit, Divya |
215
+ | Kannada | Suresh, Anu, Chetan, Vidya | Suresh, Anu |
216
+ | Malayalam | Anjali, Anju, Harish | Anjali, Harish |
217
+ | Manipuri | Laishram, Ranjit | Laishram, Ranjit |
218
+ | Marathi | Sanjay, Sunita, Nikhil, Radha, Varun, Isha | Sanjay, Sunita |
219
+ | Nepali | Amrita | Amrita |
220
+ | Odia | Manas, Debjani | Manas, Debjani |
221
+ | Punjabi | Divjot, Gurpreet | Divjot, Gurpreet |
222
+ | Sanskrit | Aryan | Aryan |
223
+ | Tamil | Kavitha, Jaya | Jaya |
224
+ | Telugu | Prakash, Lalitha, Kiran | Prakash, Lalitha |
225
 
 
226
 
227
+ **Tips**:
228
+ * We've set up an [inference guide](https://github.com/huggingface/parler-tts/blob/main/INFERENCE.md) to make generation faster. Think SDPA, torch.compile, batching and streaming!
229
+ * Include the term "very clear audio" to generate the highest quality audio, and "very noisy audio" for high levels of background noise
230
+ * Punctuation can be used to control the prosody of the generations, e.g. use commas to add small breaks in speech
231
+ * The remaining speech features (gender, speaking rate, pitch and reverberation) can be controlled directly through the prompt
232
 
 
233
 
234
+ ## Some Description Examples
235
 
236
+ 1. **Aditi - Slightly High-Pitched, Expressive Tone**:
237
+ *"Aditi speaks with a slightly higher pitch in a close-sounding environment. Her voice is clear, with subtle emotional depth and a normal pace, all captured in high-quality recording."*
238
 
239
+ 2. **Sita - Rapid, Slightly Monotone**:
240
+ *"Sita speaks at a fast pace with a slightly low-pitched voice, captured clearly in a close-sounding environment with excellent recording quality."*
241
 
242
+ 3. **Tapan - Male, Moderate Pace, Slightly Monotone**:
243
+ *"Tapan speaks at a moderate pace with a slightly monotone tone. The recording is clear, with a close sound and only minimal ambient noise."*
244
 
245
+ 4. **Sunita - High-Pitched, Happy Tone**:
246
+ *"Sunita speaks with a high pitch in a close environment. Her voice is clear, with slight dynamic changes, and the recording is of excellent quality."*
247
 
248
+ 5. **Karan - High-Pitched, Positive Tone**:
249
+ *"Karan’s high-pitched, engaging voice is captured in a clear, close-sounding recording. His slightly slower delivery conveys a positive tone."*
250
 
251
+ 6. **Amrita - High-Pitched, Flat Tone**:
252
+ *"Amrita speaks with a high pitch at a slow pace. Her voice is clear, with excellent recording quality and only moderate background noise."*
253
 
254
+ 7. **Aditi - Slow, Slightly Expressive**:
255
+ *"Aditi speaks slowly with a high pitch and expressive tone. The recording is clear, showcasing her energetic and emotive voice."*
256
 
257
+ 8. **Young Male Speaker, American Accent**:
258
+ *"A young male speaker with a high-pitched American accent delivers speech at a slightly fast pace in a clear, close-sounding recording."*
259
 
260
+ 9. **Bikram - High-Pitched, Urgent Tone**:
261
+ *"Bikram speaks with a higher pitch and fast pace, conveying urgency. The recording is clear and intimate, with great emotional depth."*
262
 
263
+ 10. **Anjali - High-Pitched, Neutral Tone**:
264
+ *"Anjali speaks with a high pitch at a normal pace in a clear, close-sounding environment. Her neutral tone is captured with excellent audio quality."*
265
 
 
266
 
267
+ ## 📐 Evaluation
268
+
269
+ Indic Parler-TTS has been evaluated using a MOS-like framework by native and non-native speakers. The results highlight its exceptional performance in generating natural and intelligible speech, especially for native speakers of Indian languages.
270
+
271
+ | **Language** | **Native Speaker Score (%)** | **Highlights** |
272
+ |--------------|-------------------------------|--------------------------------------------------------------------------------------------------|
273
+ | Assamese | 87.36 ± 1.81 | Clear, natural synthesis with excellent expressiveness. |
274
+ | Bengali | 86.16 ± 1.85 | High-quality outputs with smooth intonation. |
275
+ | Bodo | 94.47 ± 4.12 | Near-perfect accuracy for a lesser-resourced language. |
276
+ | Dogri | 88.80 ± 3.57 | Robust and consistent synthesis for Dogri. |
277
+ | Gujarati | 75.36 ± 1.78 | Strong clarity and naturalness even for smaller languages. |
278
+ | Hindi | 84.79 ± 2.09 | Reliable and expressive outputs for India's most widely spoken language. |
279
+ | Kannada | 88.17 ± 2.81 | Highly natural and accurate voices for Kannada. |
280
+ | Konkani | 76.60 ± 4.14 | Produces clear and natural outputs for diverse speakers. |
281
+ | Maithili | 95.36 ± 2.52 | Exceptionally accurate, showcasing fine-tuning success. |
282
+ | Malayalam | 86.54 ± 1.67 | Smooth, high-quality synthesis with expressive outputs. |
283
+ | Manipuri | 85.63 ± 2.60 | Natural intonation with minimal errors. |
284
+ | Marathi | 76.96 ± 1.45 | Maintains clarity and naturalness across speakers. |
285
+ | Nepali | 80.02 ± 5.75 | Strong synthesis for native and proximal Nepali speakers. |
286
+ | Odia | 88.94 ± 3.26 | High expressiveness and quality for Odia speakers. |
287
+ | Sanskrit | 99.79 ± 0.34 | Near-perfect synthesis, ideal for classical use cases. |
288
+ | Sindhi | 76.46 ± 1.29 | Clear and natural voices for underrepresented languages. |
289
+ | Tamil | 75.48 ± 2.18 | Delivers intelligible and expressive speech. |
290
+ | Telugu | 88.54 ± 1.86 | Smooth and natural tonal quality for Telugu. |
291
+ | Urdu | 77.75 ± 3.82 | Produces high-quality speech despite resource constraints. |
292
+
293
+ **Key Strengths**:
294
+ - Exceptional performance for native speakers, with top scores for **Maithili (95.36)**, **Sanskrit (99.79)**, and **Bodo (94.47)**.
295
+ - Competitive results for lesser-resourced and unofficially supported languages like **Kashmiri (55.30)** and **Sindhi (76.46)**.
296
+ - Adaptability to non-native and anonymous speaker scenarios with consistently high clarity.
297
+
298
+
299
+ ## Motivation
300
+
301
+ Parler-TTS is a reproduction of work from the paper [Natural language guidance of high-fidelity text-to-speech with synthetic annotations](https://www.text-description-to-speech.com) by Dan Lyth and Simon King, from Stability AI and Edinburgh University respectively.
302
+
303
+ Parler-TTS was released alongside:
304
+ * [The Parler-TTS repository](https://github.com/huggingface/parler-tts) - you can train and fine-tuned your own version of the model.
305
+ * [The Data-Speech repository](https://github.com/huggingface/dataspeech) - a suite of utility scripts designed to annotate speech datasets.
306
+ * [The Parler-TTS organization](https://huggingface.co/parler-tts) - where you can find the annotated datasets as well as the future checkpoints.
307
+
308
+
309
+ ## Training dataset
310
+
311
+ - **Description**:
312
+ The model was fine-tuned on a subset of the dataset used to train the pre-trained version: **Indic-Parler Dataset**, a large-scale multilingual speech corpus designed to train the **Indic Parler-TTS** model.
313
+
314
+ - **Key Statistics**:
315
+
316
+ | Dataset | Duration (hrs) | Languages Covered | No. of Utterances | License |
317
+ |:---------------:|:--------------:|:-----------------:|:-----------------:|:------------:|
318
+ | GLOBE | 535.0 | 1 | 581,725 | CC V1 |
319
+ | IndicTTS | 382.0 | 12 | 220,606 | CC BY 4.0 |
320
+ | LIMMITS | 568.0 | 7 | 246,008 | CC BY 4.0 |
321
+ | Rasa | 288.0 | 9 | 155,734 | CC BY 4.0 |
322
 
323
+ - **Languages Covered**:
324
+ The dataset supports **16 official languages** of India, along with English and Chhattisgarhi, making it comprehensive for regional language technologies. These languages include Assamese, Bengali, Bodo, Chhattisgarhi, Dogri, English, Gujarati, Hindi, Kannada, Malayalam, Manipuri, Marathi, Nepali, Odia, Punjabi, Sanskrit, Tamil and Telugu.
 
 
 
325
 
 
326
 
327
+ - **Language-Wise Data Breakdown**:
328
 
329
+ Here’s the table combining the duration (hours) and the number of utterances from the provided stats:
330
 
331
+ | Language | Duration (hrs) | No. of Utterances |
332
+ |:---------------:|:--------------:|:-----------------:|
333
+ | Assamese | 69.78 | 41,210 |
334
+ | Bengali | 140.04 | 70,305 |
335
+ | Bodo | 49.14 | 27,012 |
336
+ | Chhattisgarhi | 80.11 | 38,148 |
337
+ | Dogri | 16.14 | 7,823 |
338
+ | English | 802.81 | 735,482 |
339
+ | Gujarati | 21.24 | 5,679 |
340
+ | Hindi | 107.00 | 46,135 |
341
+ | Kannada | 125.01 | 54,575 |
342
+ | Malayalam | 25.21 | 14,988 |
343
+ | Manipuri | 20.77 | 19,232 |
344
+ | Marathi | 122.47 | 54,894 |
345
+ | Nepali | 28.65 | 16,016 |
346
+ | Odia | 19.18 | 11,558 |
347
+ | Punjabi | 11.07 | 6,892 |
348
+ | Sanskrit | 19.91 | 8,720 |
349
+ | Tamil | 52.25 | 29,204 |
350
+ | Telugu | 95.91 | 37,405 |
351
+
352
+
353
+ ## Citation
354
+
355
+ If you found this repository useful, please consider citing this work and also the original Stability AI paper:
356
+
357
+ ```
358
+ @misc{lacombe-etal-2024-parler-tts,
359
+ author = {Yoach Lacombe and Vaibhav Srivastav and Sanchit Gandhi},
360
+ title = {Parler-TTS},
361
+ year = {2024},
362
+ publisher = {GitHub},
363
+ journal = {GitHub repository},
364
+ howpublished = {\url{https://github.com/huggingface/parler-tts}}
365
+ }
366
+ ```
367
+
368
+ ```
369
+ @misc{lyth2024natural,
370
+ title={Natural language guidance of high-fidelity text-to-speech with synthetic annotations},
371
+ author={Dan Lyth and Simon King},
372
+ year={2024},
373
+ eprint={2402.01912},
374
+ archivePrefix={arXiv},
375
+ primaryClass={cs.SD}
376
+ }
377
+ ```
378
+
379
+ ## License
380
+
381
+ This model is permissively licensed under the Apache 2.0 license.