Keeping aspect ratio with CropAndResize Plugin

[MaximeDebarbat]

Hi,

I'm looking for a way to keep the aspect ration with the plugin CropAndResize (yolo style):
https://github.com/NVIDIA/TensorRT/blob/release/10.6/plugin/cropAndResizePlugin/cropAndResizePlugin.cpp

I tried to create a custom torch model which would do this padding but it's unsuccessful for now.
Any help would be greatly appreciated!

Cheers

[lix19937]

You can ref tf.image.crop_and_resize.

[asfiyab-nvidia]

cc @venkywonka for additional comment

[venkywonka]

Hi @MaximeDebarbat , thank you for taking the time to create the github issue. Could you provide more details on which specific part of the plugin you're facing problem with?
Is the plugin now working as expected in your setup?

I tried to create a custom torch model which would do this padding but it's unsuccessful for now.

I'm afraid I don't understand how this might be related to the plugin. You mean the above was the alternative you tried?
Thank you @lix19937 for help. @MaximeDebarbat have you tried @lix19937 's suggestion?

Thanks!

[MaximeDebarbat]

Hi @venkywonka, thank you for answering. Here are some details:
As the plugin does exactly crop and resize, I would like to know if you had any solution to keep the aspect ratio of each bounding box.

Let's say, you have 2 bounding boxes like these (x1, y1, x2, y2):
0, 0, 100, 200
0, 0, 200, 100
And a resize shape of 256x256.

The plugin would distort these 2 rectangular crops to resize them into 256x256. What I would love to have, would be a crop and "padded" resize which resizes the crops while adding padding:
0, 0, 100, 200 -> padding mostly vertical
0, 0, 200, 100 -> padding mostly horizontal

As a picture is worth a thousand words :

Also I mentioned the torch model as I tried to add some features to the TensorRT crop-and-resize plugin in torch->onnx->tensorRT but without success unfortunately. I mostly tried to resize the boxes to fit the aspect ratio of the expected output while keeping for each bboxes the amount of stretching applied, then turning the stretching into padding afterwards.

[venkywonka]

Thank you for the above details! Unfortunately I don't see a trivial way of doing that through the existing cropAndResizePlugin without modifying underlying code.

But what I do believe is that you can encode that logic to custom-pad it, into the plugin's source, and recompile the plugin library and try.

The full custom compute definition of how the cropAndResize Plugin works is open to local modifications, so until we have a new version/variant of the same plugin officially shipped by TensorRT to do the task of preserving the aspect ratio, unfortunately you'd probably have to get dirty with the plugin's C++ code.

I shall take this as useful user feedback and internally push towards adding this feature.
Thank you!

[venkywonka]

for an easier pythonic way to do the same, do checkout Writing custom operators with TensorRT Python plugins, which shows an example of defining the Circular padding plugin.
You can directly leverage pytorch here so you don't reinvent the wheel.

PS: make sure you have an updated TensorRT to leverage the above feature.

Hopefully that helps.

[MaximeDebarbat]

That could be amazing to integrate it directly yes as many models require to keep a good aspect ratio, that could speed up the process a lot!
I guess this is a recent feature of tensorRT, that's absolutely amazing! Thanks for sharing and for your answer! Definitely helps a lot!

Cheers!