nnUNet does not complete validation postprocessing - Error

[i-moes]

I havent had any problems in using nnUNet before. Amazing job!

However, at present, when training, I get an error, occurring after the final fold. I am training nnUNet v1 on a GPU.
Is this a GPU/torch problem? Or related to nnUNet using PyTorch?

Traceback (most recent call last):
File "/data/scratch/imoes/nnUNet/venv_nnunet/bin/nnUNet_train", line 11, in
load_entry_point('nnunet', 'console_scripts', 'nnUNet_train')()
File "/data/scratch/imoes/nnUNet/nnUNet/nnunet/run/run_training.py", line 201, in main
predict_next_stage(trainer, join(dataset_directory, trainer.plans['data_identifier'] + "_stage%d" % 1))
File "/data/scratch/imoes/nnUNet/nnUNet/nnunet/training/cascade_stuff/predict_next_stage.py", line 73, in predict_next_stage
data_nextstage = np.load(data_file_nextstage)['data']
File "/data/scratch/imoes/nnUNet/venv_nnunet/lib/python3.7/site-packages/numpy/lib/npyio.py", line 445, in load
raise ValueError("Cannot load file containing pickled data "
ValueError: Cannot load file containing pickled data when allow_pickle=False
Exception in thread Thread-5:
Traceback (most recent call last):
File "/opt/ohpc/pub/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/opt/ohpc/pub/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/data/scratch/imoes/nnUNet/venv_nnunet/lib/python3.7/site-packages/batchgenerators/dataloading/multi_threaded_augmenter.py", line 92, in results_loop
raise RuntimeError("Abort event was set. So someone died and we should end this madness. \nIMPORTANT: "
RuntimeError: Abort event was set. So someone died and we should end this madness.
IMPORTANT: This is not the actual error message! Look further up to see what caused the error. Please also check whether your RAM was full

Exception in thread Thread-4:
Traceback (most recent call last):
File "/opt/ohpc/pub/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/opt/ohpc/pub/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/data/scratch/imoes/nnUNet/venv_nnunet/lib/python3.7/site-packages/batchgenerators/dataloading/multi_threaded_augmenter.py", line 92, in results_loop
raise RuntimeError("Abort event was set. So someone died and we should end this madness. \nIMPORTANT: "
RuntimeError: Abort event was set. So someone died and we should end this madness.
IMPORTANT: This is not the actual error message! Look further up to see what caused the error. Please also check whether your RAM was full

[i-moes]

#555
#509
#447