HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation by Hierarchical Variational Inference for Zero-shot Speech Synthesis

The official implementation of HierSpeech2 | Paper | Demo page | Checkpoint |

Sang-Hoon Lee, Ha-Yeong Choi, Seung-Bin Kim, Seong-Whan Lee

Department of Artificial Intelligence, Korea University, Seoul, Korea

Abstract

[Abs.] Sorry for too long abstract😅

Recently, large-scale language models (LLM)-based speech synthesis has shown a significant performance in zero-shot speech synthesis. However, they require a large-scale data and even suffer from the same limitation of previous autoregressive speech models such as slow inference speed and lack of robustness. Following the previous powerful end-to-end text-to-speech framework of VITS (but now that's what we call classical), this paper proposes HierSpeech++, a fast and strong zero-shot speech synthesizer for text-to-speech (TTS) and voice conversion (VC). In the previous our works (HierSpeech and HierVST), we verified that hierarchical speech synthesis frameworks could significantly improve the robustness and expressiveness of the synthetic speech by adopting hierarchical variational autoencoder and leveraging self-supervised speech represenation as an additional linguistic information to bridge an information gap between text and speech. In this work, we once again significantly improve the naturalness and speaker similarity of the synthetic speech even in the zero-shot speech synthesis scenarios. We first introduce multi-audio acoustic encoder for the enhanced acoustics posterior, and adopt a hierarchical adaptive waveform generator with conditional/unconditional generation. Second, we additionally utilize a F0 information and introduce source-filter theory-based multi-path semantic encoder for speaker-agnostic and speaker-related semantic representation. We also leverage hierarchical variational autoencoder to connect multiple representations, and present a BiT-Flow which is a bidirectional normalizing flow Transformer networks with AdaLN-Zero for better speaker adaptation and train-inference mismatch reduction. Without any text transcripts, we only utilize the speech dataset to train the speech synthesizer for data flexibility. For text-to-speech, we introduce text-to-vec (TTV) frameworks to generate a self-supervised speech representation and F0 representation from text representation and prosody prompt. Then, the speech synthesizer of HierSpeech++ generates the speech from generated vector, F0, and voice prompt. In addition, we propose the high-efficient speech super-resolution framework which can upsample the waveform audio from 16 kHz to 48 kHz, and this facilitate training the speech synthesizer in that we can use easily available low-resolution (16 kHz) speech data for scaling-up. The experimental results demonstrated that hierarchical variational autoencoder could be a strong zero-shot speech synthesizer by beating LLM-based models and diffusion-based models for TTS and VC tasks. Furthermore, we also verify the data efficiency in that our model trained with a small dataset still shows a better performance in both naturalness and similarity than other models trained with large-scale dataset. Moreover, we achieve the first human-level quality in zero-shot speech synthesis.

This repository contains:

🪐 A PyTorch implementation of HierSpeech++ (TTV, Hierarchical Speech Synthesizer, SpeechSR)
⚡️ Pre-trained HierSpeech++ models trained on LibriTTS (Train-460, Train-960, and more dataset)

Previous Our Works

[1] HierSpeech: Bridging the Gap between Text and Speech by Hierarchical Variational Inference using Self-supervised Representations for Speech Synthesis
[2] HierVST: Hierarchical Adaptive Zero-shot Voice Style Transfer

This paper is an extenstion version of above papers.

Todo

Hierarchical Speech Synthesizer

HierSpeechpp-Backbone

TTV

Model	Language	Params	Dataset	Hour	Speaker	Checkpoint
TTV	Eng	107M	LibriTTS (train-960)	555	2,311	[Download]

SpeechSR

Model	Sampling Rate	Params	Dataset	Checkpoint
SpeechSR-24k	16kHz --> 24 kHz	0.13M	LibriTTS (train-960), MMS (Kor)	speechsr24k
SpeechSR-48k	16kHz --> 48 kHz	0.13M	MMS (Kor), Expresso (Eng), VCTK (Eng)	speechsr48k

Text-to-Speech

sh inference.sh

# --ckpt "logs/hierspeechpp_libritts460/hierspeechpp_lt460_ckpt.pth" \ LibriTTS-460
# --ckpt "logs/hierspeechpp_libritts960/hierspeechpp_lt960_ckpt.pth" \ LibriTTS-960
# --ckpt "logs/hierspeechpp_eng_kor/hierspeechpp_v1_ckpt.pth" \ Large_v1 epoch 60 (paper version)
# --ckpt "logs/hierspeechpp_eng_kor/hierspeechpp_v2_ckpt.pth" \ Large_v2 epoch 110 (08. Nov. 2023)

CUDA_VISIBLE_DEVICES=0 python3 inference.py \
                --ckpt "logs/hierspeechpp_eng_kor/hierspeechpp_v2_ckpt.pth" \
                --ckpt_text2w2v "logs/ttv_libritts_v1/ttv_lt960_ckpt.pth" \
                --output_dir "tts_results_eng_kor_v2" \
                --noise_scale_vc "0.333" \
                --noise_scale_ttv "0.333" \
                --denoise_ratio "0"

For better robustness, we recommend a noise_scale of 0.333
For better expressiveness, we recommend a noise_scale of 0.667
Find your best parameters for your style prompt 😵

Noise Control

# without denoiser
--denoise_ratio "0"
# with denoiser
--denoise_ratio "1"
# Mixup (Recommended 0.6~0.8)
--denoise_rate "0.8"

Voice Conversion

This method only utilize a hierarchical speech synthesizer for voice conversion.

sh inference_vc.sh

# --ckpt "logs/hierspeechpp_libritts460/hierspeechpp_lt460_ckpt.pth" \ LibriTTS-460
# --ckpt "logs/hierspeechpp_libritts960/hierspeechpp_lt960_ckpt.pth" \ LibriTTS-960
# --ckpt "logs/hierspeechpp_eng_kor/hierspeechpp_v1_ckpt.pth" \ Large_v1 epoch 60 (paper version)
# --ckpt "logs/hierspeechpp_eng_kor/hierspeechpp_v2_ckpt.pth" \ Large_v2 epoch 110 (08. Nov. 2023)

CUDA_VISIBLE_DEVICES=0 python3 inference_vc.py \
                --ckpt "logs/hierspeechpp_eng_kor/hierspeechpp_v2_ckpt.pth" \
                --output_dir "vc_results_eng_kor_v2" \
                --noise_scale_vc "0.333" \
                --noise_scale_ttv "0.333" \
                --denoise_ratio "0"

For better robustness, we recommend a noise_scale of 0.333
For better expressiveness, we recommend a noise_scale of 0.667
Find your best parameters for your style prompt 😵
Voice Conversion is vulnerable to noisy target prompt so we recommend to utilize a denoiser with noisy prompt
For noisy source speech, a wrong F0 may be extracted by YAPPT resulting in a quality degradation.

Speech Super-resolution

SpeechSR-24k and SpeechSR-48 are provided in TTS pipeline. If you want to use SpeechSR only, please refer inference_speechsr.py.
If you change the output resolution, add this

--output_sr "48000" # Default
--output_sr "24000" # 
--output_sr "16000" # without super-resolution.

Speech Denoising for Noise-free Speech Synthesis (Only used in Speaker Encoder during Inference)

For denoised style prompt, we utilize a denoiser (MP-SENet).
When using a long reference audio, there is an out-of-memory issue with this model so we have a plan to learn a memory efficient speech denoiser in the future.
If you have a problem, we recommend to use a clean reference audio or denoised audio before TTS pipeline or denoise the audio with cpu (but this will be slow😥).

TTV-v2

TTV-v1 is a simple model which is very slightly modified from VITS. Although this simple TTV could synthesize a speech with high-quality and high speaker similarity, we thought that there is room for improvement in terms of expressiveness such as prosody modeling.
For TTV-v2, we modify some components and training process (Model size: 107M --> 278M)
1. Intermediate hidden size: 256 --> 384
2. Loss masking for wav2vec reconstruction loss (I left out masking the loss for zero-padding sequences😥)
3. For long sentence generation, we finetune the model with full LibriTTS-train dataset without data filtering (Decrease the learning rate to 2e-5 with batch size of 8 per gpus)

GAN VS Diffusion

[Read More]

We think that we could not confirm which is better yet. There are many advatanges for each model so you can utilize each model for your own purposes and each study must be actively conducted simultaneously.

GAN (Specifically, GAN-based End-to-End Speech Synthesis Models)

(pros) Fast Inference Speed
(pros) High-quality Audio
(cons) Slow Training Speed (Over 7~20 Days)
(cons) Lower Voice Style Transfer Performance than Diffusion Models
(cons) Perceptually High-quality but Over-smoothed Audio because of Information Bottleneck by the sampling from the low-dimensional Latent Variable

Diffusion (Diffusion-based Mel-spectrogram Generation Models)

(pros) Fast Training Speed (within 3 Days)
(pros) High-quality Voice Style Transfer
(cons) Slow Inference Speed
(cons) Lower Audio quality than End-to-End Speech Synthesis Models

(In this wors) Our Approaches for GAN-based End-to-End Speech Synthesis Models

Improving Voice Style Transfer Performance in End-to-End Speech Synthesis Models for OOD (Zero-shot Voice Style Transfer for Novel Speaker)
Improving the Audio Quality beyond Perceptal Quality for Much more High-fidelity Audio Generation

(Our other works) Diffusion-based Mel-spectrogram Generation Models

DDDM-VC: Disentangled Denoising Diffusion Models for High-quality and High-diversity Speech Synthesis Models
Diff-hierVC: Hierarhical Diffusion-based Speech Synthesis Model with Diffusion-based Pitch Modeling

Our Goals

Integrating each model for High-quality, High-diversity and High-fidelity Speech Synthesis Models

LLM-based Models

We hope to compare LLM-based models for zero-shot TTS baselines. However, there is no public-available official implementation of LLM-based TTS models. Unfortunately, unofficial models have a poor performance in zero-shot TTS so we hope they will release their model for a fair comparison and reproducibility and for our speech community. THB I could not stand the inference speed almost 1,000 times slower than e2e models It takes 5 days to synthesize the full sentences of LibriTTS-test subsets. Even, the audio quality is so bad. I hope they will release their official source code soon.

In my very personal opinion, VITS is still the best TTS model I have ever seen. But, I acknowledge that LLM-based models have much powerful potential for their creative generative performance from the large-scale dataset but not now.

Limitation of our work

Slow training speed and Relatively large model size (Compared with VITS) --> Future work: Light-weight and Fast training pipeline and much larger model...
Could not generate realistic background sound --> Future work: adding audio generation part by disentangling speech and sound.
Could not generate a speech from a too long sentence becauase of our training setting. We see increasing max length could improve the model performance. However, we do not have GPUs with 80 GB 😢
```
 # Data Filtering for limited computation resource. 
  wav_min = 32
  wav_max = 600 # 12s 
  text_min = 1
  text_max = 200
```

TTV v2 may reduce this issue significantly...!

Results [Download]

We have attached all samples from LibriTTS test-clean and test-other.

Reference