Question: Will TensorRT 9 be available in the 23.11 NGC container ?

[MatthieuToulemont]

Hello, thanks for all the great work !

Some of my models require bfloat16 at inference time, I saw it was added in TensorRT 9 with TensorRT-LLM, and I was wondering when TensorRT 9 will be available as the default version in the NGC containers.

Thank you,

[zerollzeng]

@nvpohanh ^ ^

[nvpohanh]

23.11 NGC container will still use TRT 8.6 since all the TRT 9.x releases are only recommended for LLM workloads. We will update the TRT version in the NGC container when there is a general TRT release for all use cases. Thanks

[MatthieuToulemont]

Thank you, do you know when bfloat16 will be added to TensorRT then ?

bloat16 has become so widespread in training that it feels unfortunate not to have it available on the fastest inference framework. Even more so when a model trained in bfloat16 does not work in float16. Happened to me a couple of times :/

[MatthieuToulemont]

We will update the TRT version in the NGC container when there is a general TRT release for all use cases

Do you have an estimated released date ?

[david-PHR]

Having any guideline on how adapting a BF16 trained model for a TensorRT (version < 9) would be super useful in the meantime

[copasseron]

Any update on this ?

can we hope for the 24.02 NGC container ?

it's mainly related to this issue for my side:
#3124

[nvpohanh]

There won't be NGC container with TRT 9+ until Q2. Until then, please build the container using: https://github.com/NVIDIA/TensorRT/blob/v9.3.0/docker/ubuntu-22.04.Dockerfile

Thanks

[ttyio]

closing since no activity for more than 3 weeks, thanks all