Wer increases but the training and validation loss decreases

[mm0177]

I have been trying to fine tune the whisper base on an english dataset only to improve the transciption of the model as the wer seemed to be high without fine tuning. I have a total of 560 audio files to fine tune it .

The above picture shows the base model performnace , there was actually very high difference in the train and validation loss os i used lora optimizer. but after using that you can see the abouve issue in the image which i am facing

When i used large-v2 model the above image as you can see , i am facing that. In this also i used the lora optimizer.

I have been stuck in this what is the issue and how can i solve this

Thank you in advance!
@Vaibhavs10 @sanchit-gandhi