RuntimeError: GET was unable to find an engine to execute this computation

[YusukeImadegawa]

Thanks for your help.
I changed the number of labels from 3 to 2. But, I got RuntimeError: GET was unable to find an engine to execute this computationwhen i fine-tuned. I tried to do as a privious setting (3 labels), and there was no error. I guess loss=dict(type='CrossEntropyLoss', loss_weight=1.0), is the cause of error. Maybe like torch.nn.BCEWithLogitsLoss()

config

_base_ = [
    '../_base_/models/vgg16.py',
    '../_base_/datasets/imagenet_bs256_davit_224.py',
    '../_base_/schedules/imagenet_bs1024_adamw_conformer.py',
    '../_base_/default_runtime.py'
]

model = dict(
    type='ImageClassifier',
    backbone=dict(
        type='VGG',
        depth=16,
        num_classes=2, ###
        init_cfg=dict( 
            type='Pretrained',
            checkpoint='https://download.openmmlab.com/mmclassification/v0/vgg/vgg16_batch256_imagenet_20210208-db26f1a5.pth',
            prefix='backbone',
        ),
        frozen_stages=2, 
    ),
    neck=None,
    head=dict(
        type='ClsHead',
        loss=dict(type='CrossEntropyLoss', loss_weight=1.0),
        topk=(1, ),
        cal_acc=True, 
    ))

[Ezra-Yu]

Can the error message be displayed more completely?

[YusukeImadegawa]

Thank you for your reply.
The error is like below sentence.
Maybe mydataset's label is [0,2], so I have to set as one-hot [0, 1]...

05/14 10:08:59 - mmengine - WARNING - The model and loaded state dict do not match exactly

size mismatch for classifier.6.weight: copying a param with shape torch.Size([1000, 4096]) from checkpoint, the shape in current model is torch.Size([2, 4096]).
size mismatch for classifier.6.bias: copying a param with shape torch.Size([1000]) from checkpoint, the shape in current model is torch.Size([2]).
05/14 10:08:59 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io
05/14 10:08:59 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future.
05/14 10:08:59 - mmengine - INFO - Checkpoints will be saved to /content/drive/MyDrive/MMPreTrain/output.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [1,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [2,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [4,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [5,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [6,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [8,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [11,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [12,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [15,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [24,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [26,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [28,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [29,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [30,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
../aten/src/ATen/native/cuda/Loss.cu:176: nll_loss_forward_no_reduce_cuda_kernel: block: [0,0,0], thread: [31,0,0] Assertion `cur_target >= 0 && cur_target < n_classes` failed.
Traceback (most recent call last):
  File "/content/mmpretrain/tools/train.py", line 159, in <module>
    main()
  File "/content/mmpretrain/tools/train.py", line 155, in main
    runner.train()
  File "/usr/local/lib/python3.10/dist-packages/mmengine/runner/runner.py", line 1721, in train
    model = self.train_loop.run()  # type: ignore
  File "/usr/local/lib/python3.10/dist-packages/mmengine/runner/loops.py", line 96, in run
    self.run_epoch()
  File "/usr/local/lib/python3.10/dist-packages/mmengine/runner/loops.py", line 112, in run_epoch
    self.run_iter(idx, data_batch)
  File "/usr/local/lib/python3.10/dist-packages/mmengine/runner/loops.py", line 128, in run_iter
    outputs = self.runner.model.train_step(
  File "/usr/local/lib/python3.10/dist-packages/mmengine/model/base_model/base_model.py", line 116, in train_step
    optim_wrapper.update_params(parsed_losses)
  File "/usr/local/lib/python3.10/dist-packages/mmengine/optim/optimizer/optimizer_wrapper.py", line 184, in update_params
    self.backward(loss)
  File "/usr/local/lib/python3.10/dist-packages/mmengine/optim/optimizer/optimizer_wrapper.py", line 208, in backward
    loss.backward(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: GET was unable to find an engine to execute this computation

[Ezra-Yu]

hello, please check your cuda version. refer to https://stackoverflow.com/questions/75776895/runtimeerror-get-was-unable-to-find-an-engine-to-execute-this-computation-when

To verify that your environment is correct, you can try running the Cifra dataset

[YusukeImadegawa]

I could avoid the Error by setting sigmoid, but I have to change the ground truth to one-hot as imput.
But I'm not sure how to do that... Could you tell me if you know that ?

model = dict(
    type='ImageClassifier',
    backbone=dict(
        type='VGG',
        depth=16,
        num_classes=2, 
        init_cfg=dict( 
            type='Pretrained',
            checkpoint='https://download.openmmlab.com/mmclassification/v0/vgg/vgg16_batch256_imagenet_20210208-db26f1a5.pth',
            prefix='backbone',
        ),
        frozen_stages=2, 
    ),
    neck=None,
    head=dict(
        type='ClsHead',
        loss=dict(type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0), ###
        topk=(1, ),
        cal_acc=True, 
    ))