Default process group is not initialized

[rbunn80110]

I'm trying to run training_clip with a single GPU, however I am hitting an error involving distributed.

Single gpu training mode
vitb32 True <class 'bool'> 2

Traceback (most recent call last):
File "/home/bob/PycharmProjects/self_supervised/examples/multimodal/training_clip.py", line 67, in
def main(
File "/home/bob/anaconda3/envs/fastai/lib/python3.8/site-packages/fastcore/script.py", line 110, in call_parse
return _f()
File "/home/bob/anaconda3/envs/fastai/lib/python3.8/site-packages/fastcore/script.py", line 105, in _f
tfunc(**merge(args, args_from_prog(func, xtra)))
File "/home/bob/PycharmProjects/self_supervised/examples/multimodal/training_clip.py", line 150, in main
else: learner.fit_flat_cos(epochs, lr, pct_start=0.25)
File "/home/bob/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/callback/schedule.py", line 131, in fit_flat_cos
if self.opt is None: self.create_opt()
File "/home/bob/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py", line 149, in create_opt
self.opt = self.opt_func(self.splitter(self.model), lr=self.lr)
File "/home/bob/PycharmProjects/self_supervised/examples/multimodal/training_clip.py", line 112, in zero
return OptimWrapper(ZeroRedundancyOptimizer(params, optimizer_class=torch.optim.Adam, lr=lr))
File "/home/bob/PycharmProjects/self_supervised/examples/multimodal/zero_optimizer.py", line 170, in init
self.world_size = dist.get_world_size(self.group)
File "/home/bob/anaconda3/envs/fastai/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 638, in get_world_size
return _get_group_size(group)
File "/home/bob/anaconda3/envs/fastai/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 220, in _get_group_size
_check_default_pg()
File "/home/bob/anaconda3/envs/fastai/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 210, in _check_default_pg
assert _default_pg is not None,
AssertionError: Default process group is not initialized

[rbunn80110]

This is what needed to be changed:

:opt: Param("Optimizer", str)='ranger',

[KeremTurgutlu]

That script uses zero optimizer by default which works on multiple GPUs. Yes, you can change and use a different optimizer. Also, read about deepspeed zero optimizer its pretty cool stuff 😃