How to fix the stages of the backbone when I finetune a model?

[ZJU-lishuang]

[congee524]

You can reference

mmaction2/mmaction/models/backbones/x3d.py

Line 458 in 0149a0e

def _freeze_stages(self):

[ZJU-lishuang]

I try to modify the config.
python model = dict(backbone=dict(frozen_stages=2))

single gpu is ok.
But multiple gpus meet error.
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argumentfind_unused_parameters=Truetotorch.nn.parallel.DistributedDataParallel; (2) making sure all forwardfunction outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module'sforwardfunction. Please include the loss function and the structure of the return value offorwardof your module when reporting this issue(e.g. list, dict, iterable).

[innerlee]

@ZJU-lishuang Which backbone are you using? The backbone codes may need to be modified too. @congee524 could offer some help if more details are provided.

[congee524]

I try to modify the config.
python model = dict(backbone=dict(frozen_stages=2))

single gpu is ok.
But multiple gpus meet error.
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argumentfind_unused_parameters=Truetotorch.nn.parallel.DistributedDataParallel; (2) making sure all forwardfunction outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module'sforwardfunction. Please include the loss function and the structure of the return value offorwardof your module when reporting this issue(e.g. list, dict, iterable).

Try to set find_unused_parameters = True in your config file.

[ZJU-lishuang]

shell
bash tools/dist_train.sh configs/recognition/tsm/tsm_r50_1x1x8_50e_realviolent_rgb.py 2 --work-dir work_dirs/tsm_r50_1x1x8_50e_realviolent_lr0025_k400_freeze --validate --seed 0 --deterministic
config

tsm_r50_1x1x8_50e_realviolent_rgb.txt

And using find_unused_parameters = True solves the problem.
Thanks.

[irvingzhang0512]

@innerlee @congee524 How about adding some docs for all frozen_stages, i.e. here, remind users to set find_unused_parameters = True for distributed training/testing

[innerlee]

Add it to FAQ