train_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7975e2e5",
   "metadata": {},
   "source": [
    "# 利用Modelscope的文件处理和NeMo训练fs2\n",
    "Use modelscope for data processing & use nemo for tts training.\n",
    "\n",
    "流程:\n",
    "1. 数据处理 \n",
    "3. 获取辅助特征 \n",
    "4. 转为NeMo所需格式,以及其他杂项 \n",
    "\n",
    "\n",
    "本次采用了峰哥素材库 2022年所有直播,最后保留约40次直播, 约50小时.\n",
    "\n",
    "**实际上如果为了不计划学习一个复杂的情感(峰哥的情感大多数时候不算复杂hh), 10个小时左右基本可以满足要求.下述时间都是在10h数据上完成的.**\n",
    "\n",
    "**所有实验在3090和同等算力的显卡上完成,显存要求均小于24g. 如果你有3090和一个24核的CPU,下述的时间都是准确的.**\n",
    "\n",
    "最后,制作视频和notebook都花费了不少时间和精力,希望大家多多关注, 多多三连.\n",
    "<!-- \n",
    "Steps:\n",
    "1. data processing\n",
    "2. get auxiliary features\n",
    "3. trans metadata to nemo format, run fastpitch and hifigan -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f368f5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "!apt install ffmpeg -y\n",
    "!pip install pqdm\n",
    "pip install \"modelscope[audio]\" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22c2628c",
   "metadata": {},
   "source": [
    "----------\n",
    "## 数据处理pipeline\n",
    "\n",
    "更改了modelscope的几个问题(感兴趣可以帮忙merge一下):\n",
    "\n",
    "1. 重要! funasr/bin/vad_inference.py 报错 (更新到modelscope1.4.3后报错,之前的版本好像不会)\n",
    "```\n",
    "funasr/bin/vad_inference.py line 275:\n",
    "    删除 results[i] = json.loads(results[i])\n",
    "```\n",
    "\n",
    "1. funasr/modules/nets_utils.py pad_list在len(xs) == 1时不再pad *(减小CPU占用)*\n",
    "```\n",
    "funasr/modules/nets_utils.py line 54:\n",
    "    if len(xs) == 1:\n",
    "        return xs\n",
    "```\n",
    "\n",
    "2. funasr/models/frontend/wav_frontend.py在 apply_cmvn 时使用to(device)改为torch.tensor(, device=device) *(减小CPU占用)*\n",
    "```\n",
    "funasr/models/frontend/wav_frontend.py line 53:\n",
    "    inputs += torch.tensor(means, dtype=dtype, device=device)\n",
    "    inputs *= torch.tensor(vars, dtype=dtype, device=device)\n",
    "```\n",
    "\n",
    "funasr/utils/postprocess_utils.py @的问题,在batch infer时出现的错误. 感谢modelscope钉钉群的解答!\n",
    "```\n",
    "funasr/utils/postprocess_utils.py line 9:\n",
    "    删除分支 ch == '@'\n",
    "```\n",
    "\n",
    "funasr/bin/asr_inference_paraformer.py self.frontend.forward() *(未使用GPU加速)*\n",
    "```\n",
    "funasr/bin/asr_inference_paraformer.py line 213:\n",
    "    feats, feats_len = self.frontend.forward(speech.to(self.device), speech_lengths.to(self.device))\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb1ce161",
   "metadata": {},
   "outputs": [],
   "source": [
    "from glob import glob\n",
    "from tqdm import tqdm\n",
    "import os\n",
    "import json\n",
    "import soundfile as sf\n",
    "import subprocess\n",
    "import numpy as np\n",
    "import re\n",
    "\n",
    "import random\n",
    "from IPython.display import Audio\n",
    "import IPython.display as iply\n",
    "\n",
    "from pqdm.processes import pqdm\n",
    "from pqdm.threads import pqdm as pqdmT\n",
    "import pandas as pd\n",
    "\n",
    "import tgt\n",
    "from pypinyin import lazy_pinyin, Style\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.pyplot import imshow\n",
    "\n",
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81472b2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义一下路径, 在该路径下,把音频放入/raw/\n",
    "VERSION = ''\n",
    "start_path = f\"./{VERSION}\"\n",
    "if start_path[-1] != '/':\n",
    "    start_path = start_path + '/'\n",
    "    \n",
    "os.makedirs(os.path.join(start_path, 'raw'), exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6304a784",
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_multiprocess(x):\n",
    "    DEVICE, IJOB = x\n",
    "    subprocess.run(f\"CUDA_VISIBLE_DEVICES={DEVICE} \"+\n",
    "                   f\"nohup python {os.path.join(start_path, 'tmp', 'temp.py')} {IJOB} \"+\n",
    "                   f\"> {os.path.join(start_path, 'tmp','tmp_'+str(IJOB))} 2>&1\",\n",
    "                   shell=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10737f76",
   "metadata": {},
   "outputs": [],
   "source": [
    "SR = 22050\n",
    "sg_len = 300 # 秒, 5分钟*60\n",
    "CPU_kernels = 64 # 建议有几核就写几核"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0befe26e",
   "metadata": {},
   "source": [
    "### 数据定义和切片 (CPU友好任务,速度应该很快)\n",
    "\n",
    "1. 预筛选了一些没有连麦,音乐较少的视频,保留并删除其他的奇怪视频(比如户外).\n",
    "2. 数据切片, 将数据转换成16000采样率(建议22050,因为有train好的hifigan), wav格式, 且每段5分钟\n",
    "\n",
    "随后继续使用VAD模型,将音频切成10-20秒左右的短片段\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4850cabf",
   "metadata": {},
   "outputs": [],
   "source": [
    "def run(x):\n",
    "    name = \"/\".join(x.split(\"/\")[:-1]).replace(\"/raw\",\"/wavs/\") + x.split(\"/\")[-1].split(\".\")[0]\n",
    "    return subprocess.run(f\"ffmpeg -hide_banner -loglevel panic -i '{x}' -ac 1 -ar 16000 -f s16le - \"+\n",
    "                   f\"| ffmpeg -hide_banner -loglevel panic -f s16le -ar 16000 -i - -f segment -segment_time {sg_len} {name}_%03d.wav\",\n",
    "                   shell=True)\n",
    "\n",
    "os.makedirs(os.path.join(start_path, \"wavs\"), exist_ok=True)\n",
    "wavs_name = glob(os.path.join(start_path, \"raw/*.*\"))\n",
    "\n",
    "# run(wavs_name[0]) # 理论上你可以看到,在wavs文件夹会出现很多的音频."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63a2333f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 多进程运行 ffmpeg.\n",
    "res = pqdm(wavs_name, run, n_jobs=CPU_kernels)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1af3a723",
   "metadata": {},
   "source": [
    "-----------"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "445cff92",
   "metadata": {},
   "source": [
    "### VAD数据切片(预计30min)\n",
    "继续使用VAD模型,将音频切成10-20秒左右的短片段\n",
    "\n",
    "`ps -ef | grep tmp/temp.py | grep -v grep | cut -c 9-16| xargs kill -9` 来kill掉已经跑起来的进程.\n",
    "\n",
    "`watch -n1 \"cat vad/vad_*/1best_recog/text |wc -l\"` 来查看条数.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70d7b64c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cuda_devices = [0]*2\n",
    "cuda_devices = [0,1,2,3]*2 # 显卡个数,多卡代表0,1,2,3. 如果是单卡,设置为 [0]*2\n",
    "NJOB = len(cuda_devices) # 使用多少个进程同时进行VAD计算. 通常一个GPU可以放下2-4个进程\n",
    "\n",
    "min_length = 2.0 # 音频最短长度, 如果小于该长度,会和后面的结果合并\n",
    "max_length = 15.0 # 音频最长长度, 如果超过该长度, 会直接从中间砍开"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1a936d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf {os.path.join(start_path, \"tmp\")}\n",
    "os.makedirs(os.path.join(start_path, \"tmp\"), exist_ok=True)\n",
    "\n",
    "_script = f'''\n",
    "import os\n",
    "import sys\n",
    "from modelscope.pipelines import pipeline\n",
    "from modelscope.utils.constant import Tasks\n",
    "\n",
    "start_path = '{start_path}'\n",
    "inference_pipeline = pipeline(\n",
    "    task=Tasks.voice_activity_detection,\n",
    "    model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch',\n",
    "    model_revision='v1.1.8',\n",
    "    output_dir=os.path.join(start_path, \"vad\", \"vad_\"+sys.argv[1]),\n",
    ")\n",
    "\n",
    "print(os.path.join(start_path, 'tmp', \"temp_\"+sys.argv[1]+\".scp\"))\n",
    "inference_pipeline(audio_in=os.path.join(start_path, 'tmp', \"temp_\"+sys.argv[1]+\".scp\"))\n",
    "'''\n",
    "with open(os.path.join(start_path, \"tmp\", \"temp.py\"), \"w\") as f:\n",
    "    f.write(_script)\n",
    "    \n",
    "data = glob(os.path.join(start_path,\"wavs\", \"*.wav\"))\n",
    "slice_len = (len(data) + NJOB - 1) // NJOB\n",
    "for IJOB in range(NJOB):\n",
    "    data_dir = data[IJOB*slice_len : (IJOB+1)*slice_len]\n",
    "    with open(os.path.join(start_path, \"tmp\", f\"temp_{IJOB}.scp\"), \"w\") as f:\n",
    "        for i in data_dir:\n",
    "            f.writelines(i.split(\"/\")[-1].split(\".\")[0] + \" \"+ i + \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f234e287",
   "metadata": {},
   "outputs": [],
   "source": [
    "res = pqdm(list(zip(cuda_devices, list(range(NJOB)))), run_multiprocess, n_jobs=NJOB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b41b560",
   "metadata": {},
   "outputs": [],
   "source": [
    "datas = glob(os.path.join(start_path,\"vad/*/1best_recog/*\"))\n",
    "os.makedirs(start_path + \"slices\", exist_ok=True)\n",
    "os.makedirs(start_path + \"metas\", exist_ok=True)\n",
    "\n",
    "with open(start_path+\"metas/meta.csv\", \"w\") as meta:\n",
    "    for data in tqdm(datas):\n",
    "        with open(data) as f:\n",
    "            f={i[:i.find(\" \")]:json.loads(i[i.find(\" \"):]) for i in f.readlines()}\n",
    "        for name in f:\n",
    "            wav, fs = sf.read(start_path + f\"wavs/{name}.wav\")\n",
    "            \n",
    "            if f[name] == []:\n",
    "                continue\n",
    "            \n",
    "            temp = []\n",
    "            count = 0\n",
    "            for idxs in f[name]:\n",
    "                temp.append([int(idxs[0] / 1000 * fs), int(idxs[1] / 1000 * fs)])\n",
    "                if (idxs[1] - idxs[0]) / 1000 < min_length:\n",
    "                    continue\n",
    "                \n",
    "                _wavs = [np.concatenate([wav[i[0]:i[1]] for i in temp])]\n",
    "                if (_wavs[0].shape[0] / fs) > max_length:\n",
    "                    _k = 2 ** np.ceil(np.log2((_wavs[0].shape[0]) / fs / max_length))\n",
    "                    for i in range(int(_k)):\n",
    "                        _wavs.append(_wavs[0][int(i/_k*_wavs[0].shape[0]) : int((i+1)/_k*_wavs[0].shape[0])])\n",
    "                    _wavs = _wavs[1:]\n",
    "                    \n",
    "                for _wav in _wavs:\n",
    "                    sf.write(start_path + f\"slices/{name}_{str(count).zfill(4)}.wav\", _wav, fs)\n",
    "                    _meta = {'audio_filepath': start_path + f\"slices/{name}_{str(count).zfill(4)}.wav\", \"duration\": round(_wav.shape[0]/fs, 3)}\n",
    "                    meta.writelines(json.dumps(_meta) + '\\n')\n",
    "                \n",
    "                    count += 1\n",
    "                temp = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc215d1a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "with open(start_path+\"metas/meta.csv\") as meta:\n",
    "    meta = [json.loads(i) for i in meta.readlines()]\n",
    "    \n",
    "meta = pd.DataFrame(meta)\n",
    "print(\n",
    "    '15分位数时长', meta['duration'].quantile(0.15), \n",
    "    '\\n85分位数时长', meta['duration'].quantile(0.85), \n",
    "    '\\n总时长', round(meta['duration'].sum()/60/60, 5)\n",
    "     )\n",
    "\n",
    "print(meta.head(5))\n",
    "Audio(random.choice(meta.values)[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daa665e0",
   "metadata": {},
   "source": [
    "-----\n",
    "### modelscope ASR识别(预计30min)\n",
    "\n",
    "直接调用modelscope的ASR模型.\n",
    "\n",
    "`watch -n1 \"cat asr/asr_*/1best_recog/text |wc -l\"` 来看当前进度."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b505f79",
   "metadata": {},
   "outputs": [],
   "source": [
    "cuda_devices = [0,1,2,3] * 2\n",
    "NJOB = len(cuda_devices) \n",
    "\n",
    "# batch取决于你的显卡. 显存多可以适当拉大\n",
    "# 注意一下, batch变大似乎会使结果有轻微下降.\n",
    "# 同时,batch大于1的时候,会出现之前提到的 `@` 的问题.\n",
    "# 但是很随机,可能就几条音频会出现. 如果不想改代码,就1.\n",
    "BATCH = 1 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02af98eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf {os.path.join(start_path, \"tmp\")}\n",
    "os.makedirs(os.path.join(start_path, \"tmp\"), exist_ok=True)\n",
    "\n",
    "_script = f'''\n",
    "import os\n",
    "import sys\n",
    "from modelscope.pipelines import pipeline\n",
    "from modelscope.utils.constant import Tasks\n",
    "\n",
    "start_path = '{start_path}'\n",
    "inference_pipeline = pipeline(\n",
    "    task=Tasks.auto_speech_recognition,\n",
    "    model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch',\n",
    "    output_dir=os.path.join(start_path, \"asr\", \"asr_\"+sys.argv[1]),\n",
    "    batch={BATCH},\n",
    ")\n",
    "\n",
    "print(os.path.join(start_path, 'tmp', \"temp_\"+sys.argv[1]+\".scp\"))\n",
    "inference_pipeline(audio_in=os.path.join(start_path, 'tmp', \"temp_\"+sys.argv[1]+\".scp\"))\n",
    "'''\n",
    "\n",
    "with open(os.path.join(start_path, \"tmp\", \"temp.py\"), \"w\") as f:\n",
    "    f.write(_script)\n",
    "    \n",
    "data = glob(os.path.join(start_path, \"slices\", \"*.wav\"))\n",
    "slice_len = (len(data) + NJOB - 1) // NJOB\n",
    "for IJOB in range(NJOB):\n",
    "    data_dir = data[IJOB*slice_len : (IJOB+1)*slice_len]\n",
    "    with open(os.path.join(start_path, \"tmp\", f\"temp_{IJOB}.scp\"), \"w\") as f:\n",
    "        for i in data_dir:\n",
    "            f.writelines(i.split(\"/\")[-1].split(\".\")[0] + \" \"+ i + \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea5350a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "res = pqdm(list(zip(cuda_devices, list(range(NJOB)))), run_multiprocess, n_jobs=NJOB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2171ba7",
   "metadata": {},
   "outputs": [],
   "source": [
    "datas = glob(os.path.join(start_path,\"asr/*/1best_recog/text\"))\n",
    "\n",
    "try:\n",
    "    with open(start_path+\"metas/meta.csv\") as meta:\n",
    "        meta = [json.loads(i) for i in meta.readlines()]\n",
    "    meta = pd.DataFrame(meta).set_index('audio_filepath')\n",
    "except:\n",
    "    meta = pd.read_csv(start_path + \"metas/meta.csv\").set_index(\"audio_filepath\")\n",
    "meta['_text'] = None\n",
    "\n",
    "for data in tqdm(datas):\n",
    "    with open(data) as f:\n",
    "        _f = {}\n",
    "        for i in f.readlines():\n",
    "            if len(i.strip().split()) > 1:\n",
    "                _f[start_path +'slices/'+ i.split()[0] + '.wav'] = \" \".join(i.split()[1:]).strip()\n",
    "    for name in _f:\n",
    "        meta.at[name, '_text'] = _f[name]\n",
    "        \n",
    "data_dir = meta[meta['_text'].apply(lambda x: x is None)].reset_index()['audio_filepath'].values.tolist()\n",
    "if len(data_dir) > 0:        \n",
    "    print(\"有句子没搞出来? batch为1的时候应该都能搞出来才对.\")\n",
    "    \n",
    "meta = meta[meta['_text'].apply(lambda x: x is not None)].reset_index()\n",
    "meta.to_csv(start_path + \"metas/meta.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1dc32b6",
   "metadata": {},
   "source": [
    "\n",
    "### modelscope 标点符号 (预计10min)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c0b6a86",
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf {os.path.join(start_path, \"tmp\")}\n",
    "os.makedirs(os.path.join(start_path, \"tmp\"), exist_ok=True)\n",
    "\n",
    "_script = f'''\n",
    "import os\n",
    "import sys\n",
    "from modelscope.pipelines import pipeline\n",
    "from modelscope.utils.constant import Tasks\n",
    "\n",
    "start_path = '{start_path}'\n",
    "inference_pipeline = pipeline(\n",
    "    task=Tasks.punctuation,\n",
    "    model='damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch',\n",
    "    output_dir=os.path.join(start_path, \"punc\", \"punc_\"+sys.argv[1])\n",
    ")\n",
    "\n",
    "print(os.path.join(start_path, 'tmp', \"temp_\"+sys.argv[1]+\".txt\"))\n",
    "rec_result = inference_pipeline(text_in=os.path.join(start_path, 'tmp', \"temp_\"+sys.argv[1]+\".txt\"))    \n",
    "'''\n",
    "with open(os.path.join(start_path, \"tmp\", \"temp.py\"), \"w\") as f:\n",
    "    f.write(_script)\n",
    "    \n",
    "meta = pd.read_csv(start_path + \"metas/meta.csv\")\n",
    "slice_len = (len(meta) + NJOB - 1) // NJOB\n",
    "for IJOB in range(NJOB):\n",
    "    data_dir = meta.values[IJOB*slice_len : (IJOB+1)*slice_len]\n",
    "    with open(os.path.join(start_path, \"tmp\", f\"temp_{IJOB}.txt\"), \"w\") as f:\n",
    "        for i in data_dir:\n",
    "            f.writelines(i[0].split(\"/\")[-1].split(\".\")[0] + \"\\t\"+ i[2] + \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bb1dca3",
   "metadata": {},
   "outputs": [],
   "source": [
    "res = pqdm(list(zip(cuda_devices, list(range(NJOB)))), run_multiprocess, n_jobs=NJOB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3110b6e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pypinyin import lazy_pinyin, Style\n",
    "def get_pinyin(text):\n",
    "    text = text.lower()\n",
    "    initials = lazy_pinyin(text, neutral_tone_with_five=False, style=Style.INITIALS, strict=False)\n",
    "    finals = lazy_pinyin(text, neutral_tone_with_five=False, style=Style.FINALS_TONE3)\n",
    "\n",
    "    text_phone = []\n",
    "    for _o in zip(initials, finals):\n",
    "        if _o[0] != _o[1] and _o[0] != '':\n",
    "            _o = ['@'+i for i in _o]\n",
    "            text_phone.extend(_o)\n",
    "        elif _o[0] != _o[1] and _o[0] == '':\n",
    "            text_phone.append('@'+_o[1])\n",
    "        else:\n",
    "            text_phone.extend(list(_o[0]))\n",
    "\n",
    "    text_phone = \" \".join(text_phone)\n",
    "    return text_phone"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5675b52e",
   "metadata": {},
   "outputs": [],
   "source": [
    "datas = glob(os.path.join(start_path,\"punc/*/infer.out\"))\n",
    "meta = pd.read_csv(start_path + \"metas/meta.csv\").set_index(\"audio_filepath\")\n",
    "\n",
    "meta['text'] = None\n",
    "meta['_text_with_punc'] = None\n",
    "\n",
    "for data in tqdm(datas):\n",
    "    with open(data) as f:\n",
    "        f = {start_path +'slices/'+ i.split(\"\\t\")[0] + '.wav': i.split(\"\\t\")[-1].strip() for i in f.readlines()}\n",
    "        \n",
    "    for name in f:\n",
    "        text = \"\".join(f[name].split())\n",
    "        meta.at[name, 'text'] = text\n",
    "        meta.at[name, '_text_with_punc'] = get_pinyin(text)\n",
    "            \n",
    "meta = meta.reset_index()\n",
    "meta.to_csv(start_path + \"metas/meta.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e71cafbd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 检查是否有连续punc和None, 出现过奇奇怪怪的bug\n",
    "# 出现了请联系我😂.\n",
    "\n",
    "def _if_continue_punc(x):\n",
    "    punc = False\n",
    "    for i in list(x):\n",
    "        if i in '、，。？！～…—':\n",
    "            if punc == True:\n",
    "                return True\n",
    "            else:\n",
    "                punc = True\n",
    "        else:\n",
    "            punc = False\n",
    "    return False\n",
    "        \n",
    "meta[meta['text'].apply(lambda x:_if_continue_punc(x) or x is None or x == '')] "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88ee5d07",
   "metadata": {},
   "source": [
    "### 转到目标SR\n",
    "由于在modelscope的运算时,会调用降采样算法到16000,速度很慢,所以一开始先统一到了16000.\n",
    "\n",
    "为了在目标采样率上训练,需要重新进行第一步的操作."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb953f8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def run(x):\n",
    "    name = \"/\".join(x.split(\"/\")[:-1]).replace(\"/raw\",\"/wavs/\") + x.split(\"/\")[-1].split(\".\")[0]\n",
    "    return subprocess.run(f\"ffmpeg -hide_banner -loglevel panic -i '{x}' -ac 1 -ar {SR} -f s16le - \"+\n",
    "                   f\"| ffmpeg -hide_banner -loglevel panic -f s16le -ar {SR} -i - -f segment -segment_time {sg_len} {name}_%03d.wav\",\n",
    "                   shell=True)\n",
    "\n",
    "os.makedirs(os.path.join(start_path, \"wavs\"), exist_ok=True)\n",
    "wavs_name = glob(os.path.join(start_path, \"raw/*.*\"))\n",
    "\n",
    "res = pqdm(wavs_name, run, n_jobs=CPU_kernels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "447aaaef",
   "metadata": {},
   "outputs": [],
   "source": [
    "datas = glob(os.path.join(start_path,\"vad/*/1best_recog/*\"))\n",
    "for data in tqdm(datas):\n",
    "    with open(data) as f:\n",
    "        f={i[:i.find(\" \")]:json.loads(i[i.find(\" \"):]) for i in f.readlines()}\n",
    "    for name in f:\n",
    "        wav, fs = sf.read(start_path + f\"wavs/{name}.wav\")\n",
    "        assert fs == SR\n",
    "        \n",
    "        if f[name] == []:\n",
    "            continue\n",
    "\n",
    "        temp = []\n",
    "        count = 0\n",
    "        for idxs in f[name]:\n",
    "            temp.append([int(idxs[0] / 1000 * fs), int(idxs[1] / 1000 * fs)])\n",
    "            if (idxs[1] - idxs[0]) / 1000 < min_length:\n",
    "                continue\n",
    "\n",
    "            _wavs = [np.concatenate([wav[i[0]:i[1]] for i in temp])]\n",
    "            if (_wavs[0].shape[0] / fs) > max_length:\n",
    "                _k = 2 ** np.ceil(np.log2((_wavs[0].shape[0]) / fs / max_length))\n",
    "                for i in range(int(_k)):\n",
    "                    _wavs.append(_wavs[0][int(i/_k*_wavs[0].shape[0]) : int((i+1)/_k*_wavs[0].shape[0])])\n",
    "                _wavs = _wavs[1:]\n",
    "\n",
    "            for _wav in _wavs:\n",
    "                sf.write(start_path + f\"slices/{name}_{str(count).zfill(4)}.wav\", _wav, fs)\n",
    "                count += 1\n",
    "            temp = []"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69495083",
   "metadata": {},
   "source": [
    "-----\n",
    "## 获取辅助特征\n",
    "\n",
    "### 获取bert特征 (预计30min)\n",
    "将bert得到的feat,按照音素的结构进行简单的复制.\n",
    "\n",
    "比如 我的A(1 2 3): w o3 d e4 a -> 我 我 的 的 a -> 1 1 2 2 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "940981bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "use_gpt = False # 使用GPT2特征或Bert-based特征,好像大差不差."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7527a6c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf {os.path.join(start_path, \"tmp\")}\n",
    "os.makedirs(os.path.join(start_path, \"tmp\"), exist_ok=True)\n",
    "\n",
    "_script = f'''\n",
    "import os\n",
    "import sys\n",
    "import torch\n",
    "from tqdm import tqdm\n",
    "from glob import glob\n",
    "import json\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from pypinyin import lazy_pinyin, Style\n",
    "\n",
    "if {not use_gpt}:\n",
    "    from transformers import AutoTokenizer, AutoModelForMaskedLM\n",
    "    tokenizer = AutoTokenizer.from_pretrained(\"hfl/chinese-roberta-wwm-ext-large\")\n",
    "    model = AutoModelForMaskedLM.from_pretrained(\"hfl/chinese-roberta-wwm-ext-large\")\n",
    "\n",
    "if {use_gpt}:\n",
    "    from transformers import AutoTokenizer, AutoModelForCausalLM\n",
    "    tokenizer = AutoTokenizer.from_pretrained(\"uer/gpt2-chinese-cluecorpussmall\")\n",
    "    model = AutoModelForCausalLM.from_pretrained(\"uer/gpt2-chinese-cluecorpussmall\")\n",
    "\n",
    "device = 'cuda:0'\n",
    "model = model.to(device)\n",
    "\n",
    "start_path = '{start_path}'\n",
    "\n",
    "NJOB={NJOB}\n",
    "meta = pd.read_csv(start_path + \"metas/meta.csv\")\n",
    "slice_len = (len(meta) + NJOB - 1) // NJOB\n",
    "meta = meta.loc[int(sys.argv[1])*slice_len : (int(sys.argv[1])+1)*slice_len]\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce95d675",
   "metadata": {},
   "outputs": [],
   "source": [
    "_script += f'''\n",
    "os.makedirs(start_path + 'bert_feats/', exist_ok=True)\n",
    "for series in tqdm(meta.iloc):\n",
    "    name = start_path + \"bert_feats/\" + series['audio_filepath'].split(\"/\")[-1].replace(\".wav\",\".npy\")\n",
    "    text = series['text']\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        inputs = tokenizer(text, return_tensors='pt')\n",
    "        for i in inputs:\n",
    "            inputs[i] = inputs[i].to(device)\n",
    "        res = model(**inputs, output_hidden_states=True)\n",
    "        res = torch.cat(res['hidden_states'][-3:-2], -1)[0].cpu().numpy() # 有sos和eos token\n",
    "    \n",
    "    initials = lazy_pinyin(text, neutral_tone_with_five=False, style=Style.INITIALS, strict=False)\n",
    "    finals = lazy_pinyin(text, neutral_tone_with_five=False, style=Style.FINALS_TONE3)\n",
    "    \n",
    "    _vecs = []\n",
    "    _text = []\n",
    "    _chars = []\n",
    "    for _o in zip(zip(initials, finals), text, res[1:-1]):\n",
    "        _o, _c, _vec = _o\n",
    "        if _o[0] != _o[1] and _o[0] != '':\n",
    "            _text.extend(['@'+i for i in _o])\n",
    "            _chars.extend([_c]*2)\n",
    "            _vecs.extend([_vec]*2)\n",
    "        elif _o[0] != _o[1] and _o[0] == '':\n",
    "            _text.append('@'+_o[1])\n",
    "            _chars.append(_c)\n",
    "            _vecs.append(_vec)\n",
    "        else:\n",
    "            _text.extend(list(_o[0]))  \n",
    "            _chars.extend([_c]*len(_o[0]))\n",
    "            _vecs.extend([_vec]*len(_o[0]))\n",
    "    try:\n",
    "        assert len(_text) == len(_chars)\n",
    "        assert len(_vecs) == len(_text)\n",
    "    except:\n",
    "        print(name)\n",
    "        continue\n",
    "        \n",
    "    _vecs = np.stack([res[0]] + _vecs + [res[-1]])\n",
    "    np.save(name, _vecs)\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b74aa5b",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "with open(os.path.join(start_path, \"tmp\", \"temp.py\"), \"w\") as f:\n",
    "    f.write(_script)\n",
    "res = pqdm(list(zip(cuda_devices, list(range(NJOB)))), run_multiprocess, n_jobs=NJOB)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b5ed886",
   "metadata": {},
   "source": [
    "------\n",
    "## To NEMO format & Ready to go.\n",
    "将meta转为nemo需要的格式. 可以开始运行啦!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e354188a",
   "metadata": {},
   "source": [
    "### 打包成需要的数据格式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f0ef5f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "for_valid = 128 # valid数据长度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "919d41d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def meta_to_nemo(meta, g):\n",
    "    for series in tqdm(meta):\n",
    "        lines = {}\n",
    "        lines['audio_filepath'] = start_path + 'slices/' + series[0].split(\"/\")[-1]\n",
    "        lines['duration'] = series[1]\n",
    "        lines['text'] = series[-2]\n",
    "        \n",
    "        temp = []\n",
    "        for j in series[-1].split():\n",
    "            if True:\n",
    "                temp.append(j)\n",
    "        if len(temp) == 0:\n",
    "            print(i)\n",
    "            continue\n",
    "        lines['normalized_text'] = \" \".join(temp)\n",
    "        \n",
    "        g.writelines(json.dumps(lines, ensure_ascii=False)+'\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a94bf72",
   "metadata": {},
   "outputs": [],
   "source": [
    "meta = pd.read_csv(start_path + 'metas/meta.csv')\n",
    "\n",
    "portion = (meta['_text_with_punc'].apply(lambda x:len(x.split())) / meta['duration'])\n",
    "meta_processed = meta[(portion >= portion.quantile(0.05)) &(portion <= portion.quantile(0.95))]\n",
    "meta_processed = meta_processed[(3 < meta_processed['duration']) & (meta_processed['duration'] < 13)]\n",
    "meta_processed = meta_processed.values\n",
    "\n",
    "print(\"总计:\", len(meta_processed))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "172d473a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "np.random.shuffle(meta_processed)\n",
    "os.makedirs(start_path + \"metas/nemo\", exist_ok=True)\n",
    "\n",
    "with open(start_path + \"metas/nemo/train_manifest.json\", \"w\") as g:\n",
    "    meta_to_nemo(meta_processed[:-for_valid], g)\n",
    "        \n",
    "with open(start_path + \"metas/nemo/val_manifest.json\", \"w\") as g:\n",
    "    meta_to_nemo(meta_processed[-for_valid:], g)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69767e66",
   "metadata": {},
   "source": [
    "### 训练Fastpitch(预计7h)\n",
    "\n",
    "主要是2步,首先是训练fastpitch,注意要安装一下NeMo的requirement.\n",
    "\n",
    "10小时数据每epoch约50秒,预计训练500epoch,7小时后可以训练完毕."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b05ada8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.makedirs(os.path.join(start_path, \"codes\"), exist_ok=True)\n",
    "!cd {os.path.join(start_path, \"codes\")} && git clone https://github.com/NVIDIA/NeMo.git    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f14b23a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 修改一些文件\n",
    "##################################\n",
    "# 1. 数据类的主要是增加了一个tokenizer. 方便于测试MoeGoe的国际音标(尽管多语会有帮助, 中文并没有什么实际的作用, 所以实际还是用的pinyin)\n",
    "# 2. 增加了bert特征, 需要改动模型代码以及load data的代码.\n",
    "# 3. 由于使用了pyin作为pitch的提取器,首个epoch速度会很慢,改成了pyworld速度能快不少\n",
    "# 4. 一些基本的调参参数改了一下, 如学习率, batchsize等等\n",
    "# 5. nemo版的MAS使用的是numba, 默认占满所有的CPU资源, 实际上限制thread到4速度不会受到影响.\n",
    "# 6. 最后,使用fp16来训. 对于一些很糟糕的数据有可能把他train爆, 爆了换32跑(但其实概率非常低). \n",
    "##################################\n",
    "\n",
    "!cp ./replace_files/tts_tokenizers.py {start_path}/codes/NeMo/nemo/collections/common/tokenizers/text_to_speech/tts_tokenizers.py\n",
    "!cp ./replace_files/dataset.py {start_path}/codes/NeMo/nemo/collections/tts/data/dataset.py\n",
    "!cp ./replace_files/tts_data_types.py {start_path}/codes/NeMo/nemo/collections/tts/torch/tts_data_types.py\n",
    "\n",
    "!cp ./replace_files/fs_model.py {start_path}/codes/NeMo/nemo/collections/tts/models/fastpitch.py\n",
    "!cp ./replace_files/fs_modules.py {start_path}/codes/NeMo/nemo/collections/tts/modules/fastpitch.py\n",
    "!cp ./replace_files/bert_tsfm.py {start_path}/codes/NeMo/nemo/collections/tts/modules/transformer.py\n",
    "\n",
    "!cp ./replace_files/fastpitch_align_v1.05.yaml {start_path}/codes/NeMo/examples/tts/conf/fastpitch_align_v1.05.yaml\n",
    "!cp ./replace_files/fs_run.py {start_path}/codes/NeMo/examples/tts/fastpitch.py\n",
    "\n",
    "# !cd {start_path}/codes/NeMo && ./reinstall.sh\n",
    "!cd {start_path}/codes/NeMo && cat ./requirements/requirements_tts.txt ./requirements/requirements_asr.txt \\\n",
    "./requirements/requirements_common.txt ./requirements/requirements_lightning.txt ./requirements/requirements.txt > ./requirements.txt \\\n",
    "&& pip install -r requirements.txt\n",
    "\n",
    "!conda install -c conda-forge pyworld -y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2be3987d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 提取一下所需要的pitch,随后用于计算pitch的值\n",
    "# echo得到的值运行一下, 如果在colab上的话可以删除echo直接运行.\n",
    "\n",
    "!echo PYTHONPATH={start_path}/codes/NeMo CUDA_VISIBLE_DEVICES=0 python {start_path}/codes/NeMo/examples/tts/fastpitch.py \\\n",
    "    train_dataset={start_path}/metas/nemo/train_manifest.json \\\n",
    "    validation_datasets={start_path}/metas/nemo/val_manifest.json \\\n",
    "    sup_data_path={start_path}/sup_data \\\n",
    "    exp_manager.exp_dir={start_path}/tmp \\\n",
    "    bert_path={start_path}/bert_feats \\\n",
    "    trainer.strategy=null name=testing pitch_mean=130.01991 pitch_std=50.18665 trainer.max_epochs=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8461f7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 提取pitch的均值方差. 首先通过nemo自己生成sup文件,stop后重新计算.\n",
    "\n",
    "temp = []\n",
    "for i in tqdm(glob(start_path + \"sup_data/pitch/*\")):\n",
    "    _pitch = torch.load(i)\n",
    "    _pitch = _pitch[_pitch!=0].numpy()\n",
    "    temp.append(_pitch)\n",
    "    \n",
    "_pitch = np.concatenate(temp)\n",
    "pmax, pmin = _pitch.max(), _pitch.min()\n",
    "pstd, pmean = _pitch.std(), _pitch.mean()\n",
    "\n",
    "print(pmax, pmin)\n",
    "print(pstd, pmean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "148df702",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 开始训练!\n",
    "\n",
    "!echo PYTHONPATH={start_path}/codes/NeMo CUDA_VISIBLE_DEVICES=0 python {start_path}/codes/NeMo/examples/tts/fastpitch.py \\\n",
    "    train_dataset={start_path}/metas/nemo/train_manifest.json \\\n",
    "    validation_datasets={start_path}/metas/nemo/val_manifest.json \\\n",
    "    sup_data_path={start_path}/sup_data \\\n",
    "    exp_manager.exp_dir={start_path}/results \\\n",
    "    bert_path={start_path}/bert_feats \\\n",
    "    trainer.strategy=null name=fs2 pitch_mean={pmean} pitch_std={pstd} pitch_fmin={pmin} pitch_fmax={pmax} \n",
    "#     +init_from_pretrained_model='tts_zh_fastpitch_sfspeech'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6074e50",
   "metadata": {},
   "source": [
    "### 训练HIFIGAN(预计?)\n",
    "\n",
    "该步时间较长,一般train的时间越长越不fuzzy.\n",
    "\n",
    "等到出了ckpt就可以定期在下面的infer那里check一下.\n",
    "\n",
    "hifigan的训练是比较慢的, 因为没有FP16(FP16会有一定的下降)\n",
    "<br>一个epoch约3分钟,通常要100个epoch左右即5个小时."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24c25b90",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 提取训练fs2的mel的结果!\n",
    "\n",
    "!echo PYTHONPATH={start_path}/codes/NeMo CUDA_VISIBLE_DEVICES=0 python {start_path}/codes/NeMo/examples/tts/fastpitch.py \\\n",
    "    train_dataset={start_path}/metas/nemo/train_manifest.json \\\n",
    "    validation_datasets={start_path}/metas/nemo/val_manifest.json \\\n",
    "    sup_data_path={start_path}/sup_data \\\n",
    "    exp_manager.exp_dir={start_path}/results \\\n",
    "    bert_path={start_path}/bert_feats \\\n",
    "    name=fs2 pitch_mean={pmean} pitch_std={pstd} pitch_fmin={pmin} pitch_fmax={pmax} \\\n",
    "    model.train_ds.dataloader_params.batch_size=1 trainer.precision=32 \\\n",
    "    model.get_mel_result={start_path}/sup_data/pred_mels \\\n",
    "    trainer.strategy=null trainer.max_epochs=1000000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a0b2956",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 生成meta文件\n",
    "\n",
    "with open(f'{start_path}/metas/nemo/train_manifest_mel.json', \"w\") as g:\n",
    "    for s in tqdm(glob(f'{start_path}/sup_data/pred_mels/*.wav')):\n",
    "        _wav, _fs = sf.read(s)\n",
    "        _dict = {\n",
    "            'audio_filepath': s,\n",
    "            'duration': round(_wav.shape[0] / _fs, 3),\n",
    "            'mel_filepath': s.replace(\".wav\", \".npy\"),\n",
    "        }\n",
    "        g.writelines(json.dumps(_dict, ensure_ascii=False)+'\\n')\n",
    "        \n",
    "!head -n32 {start_path}/metas/nemo/train_manifest_mel.json > {start_path}/metas/nemo/val_manifest_mel.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "669cc3c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# finetune HIFIGAN.\n",
    "# 这一步会下载nemo的ckpt, 可能会很慢>_<\n",
    "\n",
    "!echo PYTHONPATH={start_path}/codes/NeMo CUDA_VISIBLE_DEVICES=0 python {start_path}/codes/NeMo/examples/tts/hifigan_finetune.py \\\n",
    "    train_dataset={start_path}/metas/nemo/train_manifest_mel.json \\\n",
    "    validation_datasets={start_path}/metas/nemo/val_manifest_mel.json \\\n",
    "    exp_manager.exp_dir={start_path}/results \\\n",
    "    model/train_ds=train_ds_finetune model/validation_ds=val_ds_finetune \\\n",
    "    trainer.strategy=null name=hifigan \\\n",
    "    trainer.check_val_every_n_epoch=1 \\\n",
    "    +init_from_pretrained_model='tts_zh_hifigan_sfspeech' \\\n",
    "    --config-name hifigan.yaml "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53bf5f2a",
   "metadata": {},
   "source": [
    "----------\n",
    "## Infer with NeMo\n",
    "\n",
    "如果上述任务均训练完毕, 可以开始进行infer.\n",
    "\n",
    "按顺序执行代码即可,修改raw_text为你需要的文本."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50f4ce52",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION']='python'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78ede162",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import sys\n",
    "logging.disable(logging.ERROR)\n",
    "sys.path.append(start_path + \"codes/NeMo\")\n",
    "from nemo.collections.tts.models import HifiGanModel, FastPitchModel\n",
    "from transformers import AutoTokenizer, AutoModelForMaskedLM\n",
    "\n",
    "device='cuda:0'\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"hfl/chinese-roberta-wwm-ext-large\")\n",
    "bert_model = AutoModelForMaskedLM.from_pretrained(\"hfl/chinese-roberta-wwm-ext-large\")\n",
    "bert_model = bert_model.to(device).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa869775",
   "metadata": {},
   "outputs": [],
   "source": [
    "hfg_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3673a737",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load hifigan\n",
    "hfg_path = glob(start_path + \"results/hifigan/*/checkpoints/*last*\")\n",
    "if len(hfg_path) > 0:\n",
    "    vocoder_model_pt = HifiGanModel.load_from_checkpoint(checkpoint_path=hfg_path[0]).eval().to(device)\n",
    "else:\n",
    "    # 如果没train完可以听个动静\n",
    "    print(\"使用nemo给的voocder,想达到更好效果需要tune!\")\n",
    "    vocoder_model_pt = HifiGanModel.from_pretrained(model_name=\"tts_zh_hifigan_sfspeech\").to(device) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6bc530c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load fastpitch\n",
    "fastpitch_model_path =glob(start_path + \"results/fs2/checkpoints/*last*\")[0]\n",
    "spec_gen_model = FastPitchModel.load_from_checkpoint(checkpoint_path=fastpitch_model_path).eval().to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bae3b6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在这里修改.\n",
    "raw_text = \"大家好我是二次元峰哥\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c12eb10",
   "metadata": {},
   "outputs": [],
   "source": [
    "punc_e2c = {',':'，', '.':'。','?':'？',\"、\":\"，\"}\n",
    "raw_text=\"\".join([punc_e2c[i] if i in punc_e2c else i for i in raw_text])\n",
    "text = raw_text.replace(\" \",\"\").lower()\n",
    "\n",
    "with torch.no_grad():\n",
    "    inputs = tokenizer(\"\".join(text), return_tensors='pt')\n",
    "    for i in inputs:\n",
    "        inputs[i] = inputs[i].to(device)\n",
    "    res = bert_model(**inputs, output_hidden_states=True)\n",
    "    res = torch.cat(res['hidden_states'][-3:-2], -1)[0].cpu().numpy()\n",
    "\n",
    "initials = lazy_pinyin(raw_text, neutral_tone_with_five=False, style=Style.INITIALS, strict=False)\n",
    "finals = lazy_pinyin(raw_text, neutral_tone_with_five=False, style=Style.FINALS_TONE3)\n",
    "\n",
    "_vecs = []\n",
    "_text = []\n",
    "for _o in zip(zip(initials, finals), list(raw_text), res[1:-1]):\n",
    "    _o, _c, _vec = _o\n",
    "    if _o[0] != _o[1] and _o[0] != '':\n",
    "        _text.extend(['@'+i for i in _o])\n",
    "        _vecs.extend([_vec]*2)\n",
    "    elif _o[0] != _o[1] and _o[0] == '':\n",
    "        _text.append('@'+_o[1])\n",
    "        _vecs.append(_vec)\n",
    "    else:\n",
    "        _text.extend(list(_o[0]))\n",
    "        _vecs.extend([_vec]*len(_o[0]))\n",
    "\n",
    "_vecs = np.stack([res[0]] + _vecs + [res[-1]])\n",
    "bert = torch.tensor(_vecs.T[None]).to(device)\n",
    "phoneme = \" \".join(_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08f99e16",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "with torch.no_grad():\n",
    "    parsed = spec_gen_model.parse(str_input=phoneme, normalize=False)\n",
    "    res = spec_gen_model.generate_spectrogram(\n",
    "        tokens=parsed, pace=1,\n",
    "        bert_feats=bert,\n",
    "    )\n",
    "    spectrogram = res\n",
    "    audio = vocoder_model_pt.convert_spectrogram_to_audio(spec=spectrogram)\n",
    "\n",
    "spectrogram = spectrogram.to('cpu').numpy()[0]\n",
    "audio = audio.to('cpu').numpy()\n",
    "# audio = audio / np.abs(audio).max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31708c19",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "iply.display(iply.Audio(audio[0], rate=22050))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f28dca6",
   "metadata": {},
   "source": [
    "## 总结\n",
    "\n",
    "1. 对于TTS任务而言, 通常10个小时就足以得到好的效果. 如果有一个好的底模,训练速度能有更大的提升.\n",
    "2. 对于大多数情况, 不需要训练VITS, 对于视频制作,直播而言, fs2就够了. \n",
    "3. 更多的数据意味更好得效果, 不论是chatgpt还是stable diff, 都告诉我们数据层面的军备竞赛远没有结束. 在作者自己的实验上来看,数据从10h到50h,效果是一直在提高的.\n",
    "3. 更好的模型意味更大的风险, 比如用chatgpt诈骗, unstable diff生涩图都是技术的反面. 避免技术的反面将一直是重要的挑战. 在制作视频期间,网信办也起草了新的管理办法,相信在法律的规范下,技术也能逐渐更好的服务每一个人.\n",
    "\n",
    "最后,制作视频和notebook都花费了不少时间和精力,希望大家多多关注, 多多三连.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.6 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "216.4px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}