incompatible types Int64 and Int32

[antithing]

I am trying to build an engine using TensorRT 10 and the superpoint_v1_sim_int32.onnx model.

I get:

[12/04/2024-13:22:39] [E] [TRT] ModelImporter.cpp:948: --- End node ---
[12/04/2024-13:22:39] [E] [TRT] ModelImporter.cpp:951: ERROR: ModelImporter.cpp:197 In function parseNode:
[6] Invalid Node - Concat_43
ITensor::getDimensions: Error Code 4: API Usage Error (Concat_43: concat input tensors 0 and 3 have incompatible types Int64 and Int32)

This was working well with TensorRT 8.6, what do I need to do to build the engine in TRT 10?

Thanks!

[asfiyab-nvidia]

Can you please share the ONNX model and the command/script you're using to build the engine?

[lix19937]

incompatible types， onnx-trt has check the two inputs's type.

[antithing]

@asfiyab-nvidia absolutely, the model is:

https://github.com/sair-lab/AirSLAM/blob/master/output/superpoint_v1_sim_int32.onnx

And my build code is:

bool SuperPoint::build() {
    
    if(deserialize_engine()){
        return true;
    }
    auto builder = TensorRTUniquePtr<nvinfer1::IBuilder>(nvinfer1::createInferBuilder(gLogger.getTRTLogger()));
    if (!builder) {
        std::cout << "1" << std::endl;
        return false;
    }
    const auto explicit_batch = 1U << static_cast<uint32_t>(nvinfer1::NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);
    auto network = TensorRTUniquePtr<nvinfer1::INetworkDefinition>(builder->createNetworkV2(explicit_batch));
    if (!network) {
        std::cout << "2" << std::endl;
        return false;
    }
    auto config = TensorRTUniquePtr<nvinfer1::IBuilderConfig>(builder->createBuilderConfig());
    if (!config) {
        std::cout << "3" << std::endl;
        return false;
    }
    auto parser = TensorRTUniquePtr<nvonnxparser::IParser>(
            nvonnxparser::createParser(*network, gLogger.getTRTLogger()));
    if (!parser) {
        std::cout << "4" << std::endl;
        return false;
    }
    
    auto profile = builder->createOptimizationProfile();
    if (!profile) {
        std::cout << "5" << std::endl;
        return false;
    }
    profile->setDimensions(super_point_config_.input_tensor_names[0].c_str(),
                           nvinfer1::OptProfileSelector::kMIN, nvinfer1::Dims4(1, 1, 100, 100));
    profile->setDimensions(super_point_config_.input_tensor_names[0].c_str(),
                           nvinfer1::OptProfileSelector::kOPT, nvinfer1::Dims4(1, 1, 500, 500));
    profile->setDimensions(super_point_config_.input_tensor_names[0].c_str(),
                           nvinfer1::OptProfileSelector::kMAX, nvinfer1::Dims4(1, 1, 1500, 1500));
    config->addOptimizationProfile(profile);
    
    auto constructed = construct_network(builder, network, config, parser);
    if (!constructed) {
        std::cout << "6" << std::endl; //FAILS HERE
        return false;
    }
    auto profile_stream = makeCudaStream();
    if (!profile_stream) {
        std::cout << "7" << std::endl;
        return false;
    }
    config->setProfileStream(*profile_stream);
    TensorRTUniquePtr<nvinfer1::IHostMemory> plan{builder->buildSerializedNetwork(*network, *config)};
    if (!plan) {
        std::cout << "8" << std::endl;
        return false;
    }
    TensorRTUniquePtr<nvinfer1::IRuntime> runtime{nvinfer1::createInferRuntime(gLogger.getTRTLogger())};
    if (!runtime) {
        std::cout << "9" << std::endl;
        return false;
    }
    engine_ = std::shared_ptr<nvinfer1::ICudaEngine>(runtime->deserializeCudaEngine(plan->data(), plan->size()));
    if (!engine_) {
        std::cout << "10" << std::endl;
        return false;
    }
    save_engine();
    ASSERT(network->getNbInputs() == 1);
    input_dims_ = network->getInput(0)->getDimensions();
    ASSERT(input_dims_.nbDims == 4);
    ASSERT(network->getNbOutputs() == 2);
    semi_dims_ = network->getOutput(0)->getDimensions();
    ASSERT(semi_dims_.nbDims == 3);
    desc_dims_ = network->getOutput(1)->getDimensions();
    ASSERT(desc_dims_.nbDims == 4);
    return true;
}




bool SuperPoint::construct_network(TensorRTUniquePtr<nvinfer1::IBuilder> &builder,
                                   TensorRTUniquePtr<nvinfer1::INetworkDefinition> &network,
                                   TensorRTUniquePtr<nvinfer1::IBuilderConfig> &config,
                                   TensorRTUniquePtr<nvonnxparser::IParser> &parser) const {
    auto parsed = parser->parseFromFile(super_point_config_.onnx_file.c_str(),
                                        static_cast<int>(gLogger.getReportableSeverity()));
    if (!parsed) {
        return false;
    }
    // config->setMaxWorkspaceSize(512_MiB);
    config->setFlag(nvinfer1::BuilderFlag::kFP16);
    enableDLA(builder.get(), config.get(), super_point_config_.dla_core);
    return true;
}

[LeoZDong]

@antithing Upon inspecting your provided model, it seems to be an issue with the model itself.

The error is the following:

Error (Concat_43: concat input tensors 0 and 3 have incompatible types Int64 and Int32)

The violating op Concat_43 takes in 5 inputs, where input tensor 0 is name: 67 and input tensor 3 is name: 142. name: 67 is a variable input and inferred to be type int64 (it comes from the output of the Shape_30 op), while name: 142 is an initializer with a fixed value of 8 and type int32.

ONNX requires all inputs to Concat to be of the same type (see here, where the input type is a type variable T instead of a type union), so it's not actually a valid operation.

In fact, onnxruntime also rejects the model. Likely due to a different graph traversal order than TensorRT, it complains about a different violating op Mult_49, but the error is the same.

Script:

import onnxruntime as ort
import numpy as np

session = ort.InferenceSession("superpoint_v1_sim_int32.onnx")

# Get input details
input_name = session.get_inputs()[0].name
input_shape = session.get_inputs()[0].shape
input_type = session.get_inputs()[0].type

print(f"Input Name: {input_name}")
print(f"Input Shape: {input_shape}")
print(f"Input Type: {input_type}")

# Generate dummy input
# Note: If input shape contains None or dynamic dimensions, adjust appropriately
dummy_input = np.random.rand(*[dim if dim is not None else 1 for dim in input_shape]).astype(np.float32)

# Run inference
outputs = session.run(None, {input_name: dummy_input})

Error:

onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from superpoint_v1_sim_int32.onnx failed:Type Error: Type parameter (T) of Optype (Mul) bound to different types (tensor(int64) and tensor(int32) in node (Mul_49).

To resolve this, you could either change all the initializers to int64 or add explicit cast operators in your model.

[antithing]

Okay thank you for looking into this for me! Much appreciated.