Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed in exporting to onnx format from pb format #1

Closed
mengbingrock opened this issue Apr 19, 2021 · 4 comments
Closed

Failed in exporting to onnx format from pb format #1

mengbingrock opened this issue Apr 19, 2021 · 4 comments
Assignees
@mengbingrock @markxue-0107
Labels
onnx

Comments

@mengbingrock
Copy link
Member

mengbingrock commented Apr 19, 2021 •
edited
Loading

reproduce step:
Used the code to generate pb file

import tensorflow as tf
from tensorflow.python.framework.graph_util import convert_variables_to_constants

sess = tf.Session()
Saver = tf.train.import_meta_graph("convbert_medium-small/model.ckpt.meta")
Saver.restore(sess, tf.train.latest_checkpoint("convbert_medium-small/"))
graph = tf.get_default_graph()

output_graph_def = convert_variables_to_constants(sess, sess.graph_def, output_node_names=['name'])

with tf.gfile.FastGFile('convbert.pb', mode='wb') as f:
    f.write(output_graph_def.SerializeToString())
@mengbingrock mengbingrock self-assigned this Apr 19, 2021
@mengbingrock
Copy link
Member Author 

(tensorflow_1.x) E:\Hackathon>python -m tf2onnx.convert --saved-model ./saved_model --output convbert.onnx
2021-04-19 20:13:16.272284: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found
2021-04-19 20:13:16.272505: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
D:\Anaconda\envs\tensorflow_1.x\lib\runpy.py:125: RuntimeWarning: 'tf2onnx.convert' found in sys.modules after import of package 'tf2onnx', but prior to execution of 'tf2onnx.convert'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
WARNING:tensorflow:From D:\Anaconda\envs\tensorflow_1.x\lib\site-packages\tf2onnx\verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2021-04-19 20:13:18,503 - WARNING - From D:\Anaconda\envs\tensorflow_1.x\lib\site-packages\tf2onnx\verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2021-04-19 20:13:18.509095: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2021-04-19 20:13:19.186341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce MX250 major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:2d:00.0
2021-04-19 20:13:19.188157: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found
2021-04-19 20:13:19.189991: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cublas64_100.dll'; dlerror: cublas64_100.dll not found
2021-04-19 20:13:19.192503: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cufft64_100.dll'; dlerror: cufft64_100.dll not found
2021-04-19 20:13:19.194268: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'curand64_100.dll'; dlerror: curand64_100.dll not found
2021-04-19 20:13:19.196068: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cusolver64_100.dll'; dlerror: cusolver64_100.dll not found
2021-04-19 20:13:19.197859: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cusparse64_100.dll'; dlerror: cusparse64_100.dll not found
2021-04-19 20:13:19.199704: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudnn64_7.dll'; dlerror: cudnn64_7.dll not found
2021-04-19 20:13:19.199885: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-04-19 20:13:19.200467: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2021-04-19 20:13:19.203032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-19 20:13:19.203191: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]
2021-04-19 20:13:19,204 - WARNING - '--tag' not specified for saved_model. Using --tag serve
[libprotobuf ERROR T:\src\github\protobuf\protobuf\src\google\protobuf\wire_format_lite.cc:577] String field 'tensorflow.MetaGraphDef.MetaInfoDef.meta_graph_version' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
Traceback (most recent call last):
  File "D:\Anaconda\envs\tensorflow_1.x\lib\site-packages\tensorflow_core\python\saved_model\loader_impl.py", line 68, in parse_saved_model
    saved_model.ParseFromString(file_content)
google.protobuf.message.DecodeError: Error parsing message

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Anaconda\envs\tensorflow_1.x\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "D:\Anaconda\envs\tensorflow_1.x\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\Anaconda\envs\tensorflow_1.x\lib\site-packages\tf2onnx\convert.py", line 467, in <module>
    main()
  File "D:\Anaconda\envs\tensorflow_1.x\lib\site-packages\tf2onnx\convert.py", line 213, in main
    args.large_model, return_initialized_tables=True, return_tensors_to_rename=True)
  File "D:\Anaconda\envs\tensorflow_1.x\lib\site-packages\tf2onnx\tf_loader.py", line 477, in from_saved_model
    _from_saved_model_v1(sess, model_path, input_names, output_names, tag, signatures)
  File "D:\Anaconda\envs\tensorflow_1.x\lib\site-packages\tf2onnx\tf_loader.py", line 251, in _from_saved_model_v1
    imported = tf.saved_model.loader.load(sess, tag, model_path)
  File "D:\Anaconda\envs\tensorflow_1.x\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "D:\Anaconda\envs\tensorflow_1.x\lib\site-packages\tensorflow_core\python\saved_model\loader_impl.py", line 268, in load
    loader = SavedModelLoader(export_dir)
  File "D:\Anaconda\envs\tensorflow_1.x\lib\site-packages\tensorflow_core\python\saved_model\loader_impl.py", line 284, in __init__
    self._saved_model = parse_saved_model(export_dir)
  File "D:\Anaconda\envs\tensorflow_1.x\lib\site-packages\tensorflow_core\python\saved_model\loader_impl.py", line 71, in parse_saved_model
    raise IOError("Cannot parse file %s: %s." % (path_to_pb, str(e)))
OSError: Cannot parse file b'./saved_model\\saved_model.pb': Error parsing message.

@mengbingrock mengbingrock pinned this issue Apr 19, 2021
@mengbingrock
Copy link
Member Author

Same error in docker container:

2021-04-19 12:54:25.946803: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/lib/python3.8/runpy.py:127: RuntimeWarning: 'tf2onnx.convert' found in sys.modules after import of package 'tf2onnx', but prior to execution of 'tf2onnx.convert'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2021-04-19 12:54:27,134 - WARNING - From /usr/local/lib/python3.8/dist-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2021-04-19 12:54:27.151814: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-04-19 12:54:27.184084: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-19 12:54:27.184765: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1665] Found device 0 with properties: 
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:07.0
2021-04-19 12:54:27.184796: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-04-19 12:54:27.188915: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-04-19 12:54:27.190260: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-04-19 12:54:27.190583: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-04-19 12:54:27.191727: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2021-04-19 12:54:27.192737: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-04-19 12:54:27.192921: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-04-19 12:54:27.193055: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-19 12:54:27.193761: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-19 12:54:27.194356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1793] Adding visible gpu devices: 0
2021-04-19 12:54:27.202916: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2500000000 Hz
2021-04-19 12:54:27.203413: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5846be0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-04-19 12:54:27.203448: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-04-19 12:54:27.318189: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-19 12:54:27.318942: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5848be0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-04-19 12:54:27.318980: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla T4, Compute Capability 7.5
2021-04-19 12:54:27.319265: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-19 12:54:27.319894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1665] Found device 0 with properties: 
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:07.0
2021-04-19 12:54:27.319932: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-04-19 12:54:27.319961: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-04-19 12:54:27.319973: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-04-19 12:54:27.319983: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-04-19 12:54:27.319994: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2021-04-19 12:54:27.320004: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-04-19 12:54:27.320015: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-04-19 12:54:27.320111: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-19 12:54:27.320759: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-19 12:54:27.321352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1793] Adding visible gpu devices: 0
2021-04-19 12:54:27.321394: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-04-19 12:54:27.743957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-19 12:54:27.744020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212]      0 
2021-04-19 12:54:27.744030: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1225] 0:   N 
2021-04-19 12:54:27.744325: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-19 12:54:27.745063: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-19 12:54:27.745688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1351] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 13839 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:07.0, compute capability: 7.5)
2021-04-19 12:54:27,746 - WARNING - '--tag' not specified for saved_model. Using --tag serve
2021-04-19 12:54:27,747 - WARNING - '--signature_def' not provided. Using all signatures.
[libprotobuf ERROR google/protobuf/wire_format_lite.cc:577] String field 'tensorflow.MetaGraphDef.MetaInfoDef.meta_graph_version' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes. 
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/saved_model/loader_impl.py", line 68, in parse_saved_model
    saved_model.ParseFromString(file_content)
google.protobuf.message.DecodeError: Error parsing message

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/tf2onnx/convert.py", line 467, in <module>
    main()
  File "/usr/local/lib/python3.8/dist-packages/tf2onnx/convert.py", line 211, in main
    graph_def, inputs, outputs, initialized_tables, tensors_to_rename = tf_loader.from_saved_model(
  File "/usr/local/lib/python3.8/dist-packages/tf2onnx/tf_loader.py", line 491, in from_saved_model
    _from_saved_model_v1(sess, model_path, input_names, output_names, tag, signatures)
  File "/usr/local/lib/python3.8/dist-packages/tf2onnx/tf_loader.py", line 255, in _from_saved_model_v1
    imported = tf.saved_model.loader.load(sess, tag, model_path)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/util/deprecation.py", line 330, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/saved_model/loader_impl.py", line 268, in load
    loader = SavedModelLoader(export_dir)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/saved_model/loader_impl.py", line 284, in __init__
    self._saved_model = parse_saved_model(export_dir)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/saved_model/loader_impl.py", line 71, in parse_saved_model
    raise IOError("Cannot parse file %s: %s." % (path_to_pb, str(e)))
OSError: Cannot parse file b'./saved_model.pb': Error parsing message.

@mengbingrock
Copy link
Member Author

mengbingrock commented Apr 21, 2021 •
edited
Loading

Will try to load the model, run some training, and save as pb file. Then re-export the model to onnx, to see if the pb file generated previously lost something.
save_and_load

@mengbingrock mengbingrock changed the title Failed in exporting to onnx format from pb format from tensorflow Failed in exporting to onnx format from pb format Apr 21, 2021
@mengbingrock
Copy link
Member Author

fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mengbingrock mengbingrock

@markxue-0107 markxue-0107

Labels
onnx
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mengbingrock @markxue-0107