Migrate to Python 3

[DKandrew]

TL;DR: Pytorch 1.6 version has abandoned the python 2 completely. We should also consider migrate the entire codebase from python 2 to python 3.

It appears to me that pytorch 1.6 has completely deprecated the Python 2. When trying to load a model trained in pytorch 1.6, I encounter this error. The key message is that RuntimeError: version_ <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /pytorch/caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 2. Your PyTorch installation may be too old. (init at /pytorch/caffe2/serialize/inline_container.cc:132) .

qinru@cc-01:~/codebase/ros_workspace/src$ roslaunch semantic_mapping test.launch 
... logging to /home/users/qinru/.ros/log/7054498e-09e5-11eb-847c-1c1b0dec0de6/roslaunch-cc-01-3259.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://cc-01:41007/

SUMMARY
========

PARAMETERS
 * /rosdistro: kinetic
 * /rosversion: 1.12.14

NODES
  /
    map_reduction (map_reduction/map_reduction)
    semantic_segmentation_node (semantic_mapping/semantic_segmentation_node.py)

ROS_MASTER_URI=http://localhost:11311

process[semantic_segmentation_node-1]: started with pid [3276]
process[map_reduction-2]: started with pid [3277]
[ INFO] [1602218684.971374707, 1581540853.630179194]: Received map.
Traceback (most recent call last):
  File "/home/users/qinru/codebase/ros_workspace/src/semantic_mapping_v2/src/semantic_segmentation_node.py", line 210, in <module>
    main()
  File "/home/users/qinru/codebase/ros_workspace/src/semantic_mapping_v2/src/semantic_segmentation_node.py", line 202, in main
    ros_node = SemanticSegmentationNode(cfg)
  File "/home/users/qinru/codebase/ros_workspace/src/semantic_mapping_v2/src/semantic_segmentation_node.py", line 91, in __init__
    self.network = SemanticSegmentation(network_cfg)
  File "/home/users/qinru/codebase/ros_workspace/src/semantic_mapping_v2/src/semantic_segmentation_node.py", line 45, in __init__
    checkpoint = torch.load(cfg.MODEL.WEIGHT, map_location=torch.device('cpu'))
  File "/home/users/qinru/.local/lib/python2.7/site-packages/torch/serialization.py", line 527, in load
    with _open_zipfile_reader(f) as opened_zipfile:
  File "/home/users/qinru/.local/lib/python2.7/site-packages/torch/serialization.py", line 224, in __init__
    super(_open_zipfile_reader, self).__init__(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: version_ <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /pytorch/caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 2. Your PyTorch installation may be too old. (init at /pytorch/caffe2/serialize/inline_container.cc:132)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7fcbe9b6d193 in /home/users/qinru/.local/lib/python2.7/site-packages/torch/lib/libc10.so)
frame #1: caffe2::serialize::PyTorchStreamReader::init() + 0x1f5b (0x7fcb7c2969eb in /home/users/qinru/.local/lib/python2.7/site-packages/torch/lib/libtorch.so)
frame #2: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::string const&) + 0x64 (0x7fcb7c297c04 in /home/users/qinru/.local/lib/python2.7/site-packages/torch/lib/libtorch.so)
frame #3: <unknown function> + 0x6c8b56 (0x7fcbea688b56 in /home/users/qinru/.local/lib/python2.7/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x29660b (0x7fcbea25660b in /home/users/qinru/.local/lib/python2.7/site-packages/torch/lib/libtorch_python.so)
frame #5: python() [0x4eef5e]
frame #6: python() [0x4eeb66]
frame #7: python() [0x4aaafb]
<omitting python frames>
frame #10: python() [0x4d5669]
frame #11: python() [0x4eef5e]
frame #12: python() [0x4eeb66]
frame #13: python() [0x4aaafb]
frame #18: python() [0x4d5669]
frame #19: python() [0x4eef5e]
frame #24: python() [0x4d5669]
frame #25: python() [0x4eef5e]
frame #32: python() [0x4eb69f]
frame #36: __libc_start_main + 0xf0 (0x7fcc0c019830 in /lib/x86_64-linux-gnu/libc.so.6)

[semantic_segmentation_node-1] process has died [pid 3276, exit code 1, cmd /home/users/qinru/codebase/ros_workspace/src/semantic_mapping_v2/src/semantic_segmentation_node.py __name:=semantic_segmentation_node __log:=/home/users/qinru/.ros/log/7054498e-09e5-11eb-847c-1c1b0dec0de6/semantic_segmentation_node-1.log].
log file: /home/users/qinru/.ros/log/7054498e-09e5-11eb-847c-1c1b0dec0de6/semantic_segmentation_node-1*.log

It is impossible to install pytorch 1.6 with pip in Python 2 version. I tried:

qinru@cc-01:~/codebase/ros_workspace/src$ pip install --upgrade torch torchvision
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Defaulting to user installation because normal site-packages is not writeable
Requirement already up-to-date: torch in /home/users/qinru/.local/lib/python2.7/site-packages (1.4.0)
Requirement already up-to-date: torchvision in /home/users/qinru/.local/lib/python2.7/site-packages (0.5.0)
Requirement already satisfied, skipping upgrade: numpy in /home/users/qinru/.local/lib/python2.7/site-packages (from torchvision) (1.16.6)
Requirement already satisfied, skipping upgrade: six in /usr/lib/python2.7/dist-packages (from torchvision) (1.10.0)
Requirement already satisfied, skipping upgrade: pillow>=4.1.1 in /home/users/qinru/.local/lib/python2.7/site-packages (from torchvision) (6.2.2)

The latest version of pytorch that supports python 2 is 1.4.0. I would prefer NOT to downgrade the pytorch to 1.4 version as distributed training of the network is more efficient and stable in 1.6.0 version (I observe deadlock during the network training in pytorch 1.3, but did not observe it in pytorch 1.6).

[AftermathK]

I'll look into this. I remember that your original deeplab implementation worked with Python3; this was then adapted to work with rospy. Does your current version work well with Python3 (independently of the ROS env)?

[DKandrew]

Some code in the network training part does use Python 3 specific features, but the inference part fully supports Python 2, so inferencing should not be a problem. Whose "current version" are you referring to? Do you mean this semantic mapping repo?