How to train all 5 folds sequentially by one single command?

[MrMarkusJ]

Hi,

how can the training for all 5 folds can be initialized by one single command?

• currently for each fold I need to call the training

  e.g.:
  nnUNet_train 3d_fullres nnUNetTrainerV2 501 0 (fold 0)
  nnUNet_train 3d_fullres nnUNetTrainerV2 501 1 (fold 1)
  nnUNet_train 3d_fullres nnUNetTrainerV2 501 2 (fold 2)
  nnUNet_train 3d_fullres nnUNetTrainerV2 501 3 (fold 3)
  nnUNet_train 3d_fullres nnUNetTrainerV2 501 4 (fold 4)

I tried the following command:

• nnUNet_train 3d_fullres nnUNetTrainerV2 501 0;1;2;3;4
  but this command initialized a training which used more GPU RAM as compared to a training of a single fold

I am sure there is a simple answere but I couldn´t figure out / haven´t found it

Thanks for your help!

Best,
Markus

[FabianIsensee]

nnUNet_train 3d_fullres nnUNetTrainerV2 501 0 && nnUNet_train 3d_fullres nnUNetTrainerV2 501 1 && nnUNet_train 3d_fullres nnUNetTrainerV2 501 2 && nnUNet_train 3d_fullres nnUNetTrainerV2 501 3 && nnUNet_train 3d_fullres nnUNetTrainerV2 501 4

That should do it

[MrMarkusJ]

Thanks! That´s exactly what I am looking for

[FabianIsensee]

I am assuming you plan to train one fold after the other, this is what my answer does. If you want to run them in parallel, you need to assign them different GPUs

[MrMarkusJ]

Since too often I could read "DDP is not recommended" I plan to approach the multiGPU training as suggested in an older discussion topic (e.g. #962).

--> sequentially start multi trains by assigning the fold to specific GPU by

CUDA_VISIBLE_DEVICES=0 nnUNet_train 3d_fullres nnUNetTrainerV2 501 0
CUDA_VISIBLE_DEVICES=1 nnUNet_train 3d_fullres nnUNetTrainerV2 501 1

since 2 GPUs are available.

Thanks for your feedback & the great support & nice code!

Best,
Markus

[FabianIsensee]

Yeah that's the way to do it :-) Good luck with your training!