Input scaling and datatype on version 1 RT_DETR pytorch model

[jeonghyunjo]

Hello,

I wanted to use scaling and different input data type on rt-detrv2 but due to fixed TensorRT version, I can't use rt-detr v2. So, I put some tricks on the version 1 for dataloader to provide non-scaled int8(0-255) HWC format sample and cast the input of RT_DETR forward function to the float(0-255) CHW format to use the original implementation without extra changes. However, when I convert it into TensorRT engine, I got a lot of errors for the weights quantization.

These are the list of errors I got:
1 weights are affected by this issue: Detected FP32 infinity values and converted them to corresponding FP16 infinity.
271 weights are affected by this issue: Detected subnormal FP16 values.
67 weights are affected by this issue: Detected values less than smallest positive FP16 subnormal value and converted them to the FP16 minimum subnormalized value.
6 weights are affected by this issue: Detected finite FP32 values which would overflow in FP16 and converted them to the closest finite FP16 value.

Is it because of the structure of rt-detr version 1 or TensorRT problem? I'm wondering about if there is specific reason that scaling and dtype implementation is missing in the version 1 for pytorch.

Thank you

[lyuwenyu]

I don't think it's a model problem. The input is inconsistent with the training after you change it like this.