INT8 Quantization of dinov2 TensorRT Model is Not Faster than FP16 Quantization

[mr-lz]

Hello,

I used PyTorch-Quantization for post-training INT8 quantization on the dinov2-base model and then converted it to a TensorRT model. However, I found that the INT8 model is slightly slower than the FP16 model (the same conclusion was observed on A100, V100, and A10). Is this behavior normal?

Thank you.

[lix19937]

maybe many ops not run with int8 and add more reformat layers.

[asfiyab-nvidia]

cc @akhilg-nv for additional comment