Running out of computation time after the 1000 epochs are done and before the training finishes

[paul-reiners]

After the 1000 epochs of training are finished for a given fold, there is still a lot of work that nnU-Net does with the fold. The problem I'm running into is that for the size of data I'm using, this 'clean-up' work takes more than 24 hours, which is the maximum time our SLURM system allows for a job.

Is there a way around this? I'm trying to run this last part without a GPU, which means I can use 36 hours of computation time. It's too early to tell yet whether that will work.