How to finetune a target voice（about 20 hours） without decay of robustness

[Liujingxiu23]

I train tortoise with 10w hours of chinese audio, and then finetune it with 20 hours dataset of a female voice.
I tried:

1.lr = 0.00001, finetune all the parameters
2.lr = 0.00001, finetune using lora
3.lr = 0.00001, freeze part of the model and only finetune few layers

Thought the synthesized waves sounds good, but the robustness declained.
Repeat sometimes happens especially for long sentences.
For example, the input text is "I think .. blabla ......, I am happy",
the syntheszied wave is "I think .. blabla ......, I am happy, I am happy"

How to finetune to keep robustness?