No condition to check a single node saves the model in multiGPU training: might lead to saving corrupt model

[SwetaMahajan]

open_clip/src/open_clip_train/main.py

Lines 512 to 517 in 7260a46

	if args.save_most_recent:
	# try not to corrupt the latest checkpoint if save fails
	tmp_save_path = os.path.join(args.checkpoint_path, "tmp.pt")
	latest_save_path = os.path.join(args.checkpoint_path, LATEST_CHECKPOINT_NAME)
	torch.save(checkpoint_dict, tmp_save_path)
	os.replace(tmp_save_path, latest_save_path)

Seems like there is no condition to check that a single node should save the model checkpoint in case of multi GPU training. This might lead to corrupt model checkpoint if all of the nodes try to save the same model at the same path. Can someone please explain.

[rwightman]

It's only saved on master

https://github.com/mlfoundations/open_clip/blob/7260a46e7b4bcf518f5200fea06da5bc85aae025/src/open_clip_train/main.py#L427C5-L427C83

open_clip/src/open_clip_train/main.py

Line 490 in 7260a46

if args.save_logs: