|
--- |
|
license: mit |
|
pipeline_tag: feature-extraction |
|
tags: |
|
- bark |
|
- tts |
|
- hubert |
|
- text-to-speech |
|
datasets: |
|
- GitMylo/bark-semantic-training |
|
--- |
|
# Bark-voice-cloning |
|
Bark-voice-cloning is a model which processes the outputs from a HuBERT model, and turns them into semantic tokens compatible with bark text to speech. |
|
|
|
This can be used for many things, including speech transfer and voice cloning. |
|
|
|
[code repo](https://github.com/gitmylo/bark-voice-cloning-HuBERT-quantizer) |
|
[audio webui](https://github.com/gitmylo/audio-webui) |
|
|
|
# The models in this repo |
|
* [quantifier_hubert_base_ls960.pth](https://huggingface.co/GitMylo/bark-voice-cloning/blob/main/quantifier_hubert_base_ls960.pth) (the model trained on literature for 4 epochs) |
|
* [quantifier_hubert_base_ls960_14.pth](https://huggingface.co/GitMylo/bark-voice-cloning/blob/main/quantifier_hubert_base_ls960_14.pth) (the model trained on literature for 10 more epochs, based on the previous) |
|
* [quantifier_V1_hubert_base_ls960_23.pth](https://huggingface.co/GitMylo/bark-voice-cloning/blob/main/quantifier_V1_hubert_base_ls960_23.pth) (a larger model, trained for more epochs on the same dataset) |
|
|
|
(Please use the model manager from the [code repo](https://github.com/gitmylo/bark-voice-cloning-HuBERT-quantizer) for easy downloading of models) |
|
|
|
# Voice cloning |
|
Voice cloning is creating a new voice for text-to-speech. |
|
|
|
Process: |
|
1. Load your wav audio file into your pytorch application |
|
2. For the fine prompt extract [discrete representations](https://github.com/facebookresearch/encodec#extracting-discrete-representations). (These are used by bark to know about the voice), **make sure to `.squeeze()` the resulting codes.** |
|
3. For the coarse prompt do `fine_prompt[:2, :]`, to get the coarse prompt from a fine prompt. |
|
4. For the semantics, load a HuBERT model without Kmeans (I personally use the [audiolm-pytorch](https://github.com/lucidrains/audiolm-pytorch) implementation's hubertwithkmeans, but i edited it to skip kmeans.) |
|
5. Next, to get the actual semantic tokens, run the tokens through this model. Your output will be compatible with bark. |
|
6. Save these files in an npz with `numpy.savez(semantic_prompt=semantics, fine_prompt=fine, coarse_prompt=coarse)`. This is your speaker file containing your cloned voice. |
|
|
|
# Voice masking |
|
Voice masking is replacing a voice in an audio clip for speech-to-speech. |
|
|
|
## Random |
|
Replacing a voice in an audio clip with a voice generated by bark. |
|
|
|
process: |
|
1. Extract semantics from the audio clip using HuBERT and this model |
|
2. Run `semantic_to_waveform` from `bark.api` with the extracted semantics |
|
3. The previous step returns the generated audio. |
|
|
|
## Transfer |
|
Replacing a voice with a voice from another audio clip. |
|
|
|
process: |
|
1. Create a speaker file using the steps under the voice cloning section |
|
2. Extract the semantics from the clip with the text you want spoken |
|
3. Run `semantics_to_waveform` from `bark.api` with the extracted semantics, and the speaker prompt that you created on step 1. |
|
4. The previous step returns the generated audio. |
|
|
|
# Disclaimer |
|
I am not responsible for any misuse of this model. I do not agree with cloning people's voices without permission. Please make sure it is appropriate to clone someone's voice before doing so. |
