unexpected keys

[3d-illusions]

with the model path set to use the dpt_beit_large_512.pt model I downloaded, I get the following error :

model_loader.py", line 54, in load_model
    model = DPTDepthModel(
            ^^^^^^^^^^^^^^
dpt_depth.py", line 163, in __init__
    self.load(path)
RuntimeError: Error(s) in loading state_dict for DPTDepthModel:
        Unexpected key(s) in state_dict: "pretrained.model.blocks.0.attn.relative_position_index", "pretrained.model.blocks.1.attn.relative_position_index", "pretrained.model.blocks.2.att
n.relative_position_index", "pretrained.model.blocks.3.attn.relative_position_index", "pretrained.model.blocks.4.attn.relative_position_index", "pretrained.model.blocks.5.attn.rela
tive_position_index", "pretrained.model.blocks.6.attn.relative_position_index", "pretrained.model.blocks.7.attn.relative_position_index", "pretrained.model.blocks.8.attn.relative_p
osition_index", "pretrained.model.blocks.9.attn.relative_position_index", "pretrained.model.blocks.10.attn.relative_position_index", "pretrained.model.blocks.11.attn.relative_posit
ion_index", "pretrained.model.blocks.12.attn.relative_position_index", "pretrained.model.blocks.13.attn.relative_position_index", "pretrained.model.blocks.14.attn.relative_position
_index", "pretrained.model.blocks.15.attn.relative_position_index", "pretrained.model.blocks.16.attn.relative_position_index", "pretrained.model.blocks.17.attn.relative_position_in
dex", "pretrained.model.blocks.18.attn.relative_position_index", "pretrained.model.blocks.19.attn.relative_position_index", "pretrained.model.blocks.20.attn.relative_position_index
", "pretrained.model.blocks.21.attn.relative_position_index", "pretrained.model.blocks.22.attn.relative_position_index", "pretrained.model.blocks.23.attn.relative_position_index".

[3d-illusions]

I'm attempting to use it with python 3.11.7. It's not an option to use a lower python version unfortunately.

pip -m install torch torchvision opencv-python==4.6.0.66 timm einops==0.6.0 imutils==0.5.4 --user

If I try and pip install timm==0.6.12 I get

RuntimeError: Error: mutable default <class 'timm.models.maxxvit.MaxxVitConvCfg'> for field conv_cfg is not allowed: use default_factory

[3d-illusions]

Using with python 3.10 has no issues. How can I get this working with python 3.11.7?

[3d-illusions]

It seems that timm needed to be modified to work with python 3.11, huggingface/pytorch-image-models@9ecab46

Now that timm is updated, MiDaS no longer works. Where would I need to modify the Midas code, or is it the model itself that would need updating?

[heyoeyo]

Based on the first error you posted, it looks like maybe the newer versions of the timm library don't include weights for the relative_position_index (since every listed key seems to include that). A simple thing to try is to let the model load the weights without strict mode enabled. You'd have to modify the loading line to something like:

self.load_state_dict(parameters, strict=False)

If that doesn't work, you could try explicitly removing those keys from the loaded parameters variable in that same code block (i.e. in a for loop, delete all the keys of the parameters dictionary that end with attn.relative_position_index before trying to load the model).

If that doesn't work, then (shameless plug) I have a script that can load the beit models and doesn't depend on timm. It also has support for the depth-anything models which tend to give better results if you don't specifically need beit. Also, for really good (but slow) depth results, it's worth checking out the marigold model if you haven't already.

[3d-illusions]

Thanks very much, I'll give those things a try 👍

[WeRideForCoffee]

I had this same error today, inside a bare-bones python venv (no conda) on Python 3.10.6, installing dependancies manually and disregarding versions (so by default all were installed with the latest version as of 2024-09-24)

I was able to get this repository working and producing sensible output by pip uninstall timm and pip install timm==0.6.13; one revision older than the version 3d-illusions was having issues with.

I tried several more recent versions but jumping back a few versions at a time, 0.6.13 was the latest non-indev version to work for me. This might not work for everybody but since I was able to get around this error I wanted to share what worked for me.