This repo implements modified version of [VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Network](https://arxiv.org/abs/2007.15256) using Pytorch, for actual VocGAN checkout to `baseline` branch. I bit modify the VocGAN's generator and used Full-Band MelGAN's discriminator instead of VocGAN's discriminator, as in my research I found MelGAN's discriminator is very fast while training and enough powerful to train Generator to produce high fidelity voice whereas VocGAN Hierarchically-nested JCU discriminator is quite huge and extremely slows the training process.
Tested on Python 3.6
pip install -r requirements.txt- Download dataset for training. This can be any wav files with sample rate 22050Hz. (e.g. LJSpeech was used in paper)
- preprocess:
python preprocess.py -c config/default.yaml -d [data's root path] - Edit configuration
yamlfile
-
python trainer.py -c [config yaml file] -n [name of the run]cp config/default.yaml config/config.yamland then editconfig.yaml- Write down the root path of train/validation files to 2nd/3rd line.
-
tensorboard --logdir logs/
- This repo implements modified VocGAN for faster training although for true VocGAN implementation please checkout
baselinebranch, In my testing I am available to generate High-Fidelity audio in real time from Modified VocGAN. - Training cost for
baselineVocGAN'sDiscriminatoris too high (2.8 sec/it on P100 with batch size 16) as compared toGenerator(7.2 it/sec on P100 with batch size 16), so it's unfeasible for me to train this model for long time. - May be we can optimizer
baselineVocGAN's Discriminator by downsampling the audio on pre-processing stage instead of Training stage (currently I usedtorchaudio.transform.Resampleas layer for downsampling the audio), this step might be speed-up overallDiscriminatortraining. - I trained
baselinemodel for 300 epochs (with batch size 16) on LJSpeech, and quality of generated audio is similar to the MelGAN at same epoch on same dataset. Author recommend to train model till 3000 epochs which is not feasible at current training speed(2.80 sec/it). - I am open for any suggestion and modification on this repo.
- For more complete and end to end Voice cloning or Text to Speech (TTS) toolbox 🤖 please visit Deepsync Technologies.
python inference.py -p [checkpoint path] -i [input mel path]
Two pretrained model are provided. Both pretrained models are trained using modified-VocGAN structure.
- LJSpeech-1.1 (English, single-female speaker, trained for 4000 epochs) [ download ]
- KSS dataset (Korean, single-female speaker, trained for 4500 epochs) [ download ]
- VCTK (English, multispeaker, trained for 3180 epochs) [download]
Using pretrained models, we can reconstruct audio samples. Visit here to listen.
[WIP]
