Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TODO] Implementation of Proto Splitter (breakthrough of 2GB contract for Protocol Buffers) #455

Open
PINTO0309 opened this issue Aug 16, 2023 · 1 comment
Labels
ProtocolBuffers > 2.0GB ProtocolBuffers > 2.0GB third party Third-party tool issues TODO TODO

Comments

@PINTO0309
Copy link
Owner

PINTO0309 commented Aug 16, 2023

Issue Type

Others

OS

Linux

onnx2tf version number

1.15.12

onnx version number

1.13.1

onnxruntime version number

1.15.1

onnxsim (onnx_simplifier) version number

0.4.33

tensorflow version number

1.13.0

Download URL for ONNX

N/A

Parameter Replacement JSON

N/A

Description

  1. Personal
  2. Implementation of Proto Splitter (breakthrough of 2GB contract for Protocol Buffers)
    https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/proto_splitter
    https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/proto_splitter/g3doc/in-depth-guide.md
    https://github.com/tensorflow/community/blob/master/rfcs/20230720-unbound-saved-model.md

Implementing this in the backend should make it possible to break through the 2GB limit of Protocol Buffers.

TensorFlow v2.15.0+

image

@PINTO0309 PINTO0309 added third party Third-party tool issues TODO TODO ProtocolBuffers > 2.0GB ProtocolBuffers > 2.0GB labels Aug 16, 2023
@PINTO0309
Copy link
Owner Author

PINTO0309 commented Dec 9, 2023

  • TensorFlow v2.15.0, v2.16.1, v2.17.0
    ERROR: The option `experimental_image_format` is disabled in OSS.
    Traceback (most recent call last):
      File "/home/b920405/git/onnx2tf/onnx2tf/onnx2tf.py", line 1173, in convert
        tf.saved_model.save(model, output_folder_path, options=tf.saved_model.SaveOptions(experimental_image_format=True))
      File "/home/b920405/.local/lib/python3.10/site-packages/tensorflow/python/saved_model/save_options.py", line 189, in __init__
        raise ValueError(
    ValueError: The option `experimental_image_format` is disabled in OSS.
    
  • test code
    https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/saved_model/proto_splitter_save_test.py
    image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ProtocolBuffers > 2.0GB ProtocolBuffers > 2.0GB third party Third-party tool issues TODO TODO
Projects
None yet
Development

No branches or pull requests

1 participant