How To Resume Training

[Renshaw26]

Accessing a remote computer to run nnUNet, the job time is limited to 2 days so the model training was cut short, how can i resume it at the current fold/epcoh?

[mehnaz1985]

@Renshaw26
nnU-Net stores a checkpoint every 50 epochs. If you need to continue a previous training, just add a -c to the training command.

nnUNet_train CONFIGURATION TRAINER_CLASS_NAME TASK_NAME_OR_ID FOLD --npz-c
like this.

[lisatwyw]

In version 2:

nnUNetv2_train  101 2d 4 --npz --c  

# Explanation:  dataset ID 101 | configuration "2d" | fold 4 | --c for resume

[pwrightkcl]

I had a follow-up question. On my system, if the job is interrupted by the system, it will fire up a new instance of the docker container and rerun the entrypoint. Do I need to add some code to detect whether a checkpoint file exists and then select the train command with or without the --c flag accordingly or can I just use a single command with --c and expect nnUNet to ignore it if there is no saved checkpoint?

[shahzadir]

I am experiencing almost similar issue. When I resume training using the --continue_training flag, the training starts as a new instance with the learning rate reset to the base learning rate. There is no indication in the log files that training has resumed from the last checkpoint. While I can start a new instance using the learning rate from the last epoch of the previous model, how to ensure that the weights from the previous model are being used for training? @lisatwyw