Handle big-endian platform with visp::cnpy::npz_load/npz_save

[s-trinh]

• Github CI:

apt-get update && apt-get install -y lsb-release git build-essential cmake valgrind gdb
cmake .. -DCMAKE_BUILD_TYPE=RelWithDebInfo [...]
gdb -ex run -ex quit ./modules/core/testNPZ
valgrind --leak-check=full ./modules/core/testNPZ

• code:

    std::string directory_filename = createTmpDir();
    std::string npz_filename = directory_filename + "/test_npz_read_write.npz";

    {
      const std::string save_string = "Open Source Visual Servoing Platform";
      std::vector<char> vec_save_string(save_string.begin(), save_string.end());
      // Manually add a null-terminated character
      // https://stackoverflow.com/a/8247804
      vec_save_string.push_back('\0');
      const std::string identifier = "String";
      visp::cnpy::npz_save(npz_filename, identifier, &vec_save_string[0], { vec_save_string.size() }, "w");

      visp::cnpy::npz_t npz_data = visp::cnpy::npz_load(npz_filename);
      visp::cnpy::NpyArray arr_string_data = npz_data[identifier];
      const std::string read_string = std::string(arr_string_data.data<char>()); // line 160
      std::cout << "(save_string == read_string)=" << (save_string == read_string) << std::endl;
    }

• Build on ubuntu22.04 s390x (Big Endian)

log

2024-05-07T18:50:17.5140547Z Copyright (C) 2022 Free Software Foundation, Inc.
2024-05-07T18:50:17.5141565Z License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
2024-05-07T18:50:17.5142705Z This is free software: you are free to change and redistribute it.
2024-05-07T18:50:17.5143653Z There is NO WARRANTY, to the extent permitted by law.
2024-05-07T18:50:17.5144467Z Type "show copying" and "show warranty" for details.
2024-05-07T18:50:17.5145355Z This GDB was configured as "s390x-linux-gnu".
2024-05-07T18:50:17.5146091Z Type "show configuration" for configuration details.
2024-05-07T18:50:17.5146818Z For bug reporting instructions, please see:
2024-05-07T18:50:17.5147500Z <https://www.gnu.org/software/gdb/bugs/>.
2024-05-07T18:50:17.5148294Z Find the GDB manual and other documentation resources online at:
2024-05-07T18:50:17.5149177Z     <http://www.gnu.org/software/gdb/documentation/>.
2024-05-07T18:50:17.5149678Z 
2024-05-07T18:50:17.5149850Z For help, type "help".
2024-05-07T18:50:17.5150481Z Type "apropos word" to search for commands related to "word"...
2024-05-07T18:50:17.5284713Z Reading symbols from �[32m./modules/core/testNPZ�[m...
2024-05-07T18:50:17.6031972Z Starting program: �[32m/home/runner/work/visp/visp/build/modules/core/testNPZ�[m 
2024-05-07T18:50:17.6039634Z warning: Error disabling address space randomization: Operation not permitted
2024-05-07T18:50:17.6074405Z warning: Could not trace the inferior process.
2024-05-07T18:50:17.6076273Z warning: ptrace: Function not implemented
2024-05-07T18:50:17.6158108Z During startup program exited with code 127.
2024-05-07T18:50:17.7402515Z ==21314== Memcheck, a memory error detector
2024-05-07T18:50:17.7403689Z ==21314== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
2024-05-07T18:50:17.7404509Z ==21314== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
2024-05-07T18:50:17.7405054Z ==21314== Command: ./modules/core/testNPZ
2024-05-07T18:50:17.7405379Z ==21314== 
2024-05-07T18:50:17.9585095Z ==21314== error calling PR_SET_PTRACER, vgdb might block
2024-05-07T18:50:24.2180607Z ==21314== Invalid read of size 8
2024-05-07T18:50:24.2186269Z ==21314==    at 0x10A608: main (testNPZ.cpp:160)
2024-05-07T18:50:24.2188851Z ==21314==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
2024-05-07T18:50:24.2191230Z ==21314== 
2024-05-07T18:50:24.2195407Z ==21314== 
2024-05-07T18:50:24.2196538Z ==21314== Process terminating with default action of signal 11 (SIGSEGV): dumping core
2024-05-07T18:50:24.2197641Z ==21314==  Access not within mapped region at address 0x0
2024-05-07T18:50:24.2201668Z ==21314==    at 0x10A608: main (testNPZ.cpp:160)
2024-05-07T18:50:24.2202486Z ==21314==  If you believe this happened as a result of a stack
2024-05-07T18:50:24.2203525Z ==21314==  overflow in your program's main thread (unlikely but
2024-05-07T18:50:24.2204390Z ==21314==  possible), you can try to increase the size of the
2024-05-07T18:50:24.2205348Z ==21314==  main thread stack using the --main-stacksize= flag.
2024-05-07T18:50:24.2206243Z ==21314==  The main thread stack size used in this run was 16777216.
2024-05-07T18:50:24.2357213Z ==21314== 
2024-05-07T18:50:24.2359925Z ==21314== HEAP SUMMARY:
2024-05-07T18:50:24.2360542Z ==21314==     in use at exit: 78,226 bytes in 12 blocks
2024-05-07T18:50:24.2361441Z ==21314==   total heap usage: 65 allocs, 53 frees, 130,525 bytes allocated
2024-05-07T18:50:24.2362194Z ==21314== 
2024-05-07T18:50:24.2431539Z ==21314== LEAK SUMMARY:
2024-05-07T18:50:24.2432500Z ==21314==    definitely lost: 0 bytes in 0 blocks
2024-05-07T18:50:24.2433512Z ==21314==    indirectly lost: 0 bytes in 0 blocks
2024-05-07T18:50:24.2434389Z ==21314==      possibly lost: 0 bytes in 0 blocks
2024-05-07T18:50:24.2435131Z ==21314==    still reachable: 78,226 bytes in 12 blocks
2024-05-07T18:50:24.2435895Z ==21314==         suppressed: 0 bytes in 0 blocks
2024-05-07T18:50:24.2436512Z ==21314== Reachable blocks (those to which a pointer was found) are not shown.
2024-05-07T18:50:24.2437271Z ==21314== To see them, rerun with: --leak-check=full --show-leak-kinds=all
2024-05-07T18:50:24.2437705Z ==21314== 
2024-05-07T18:50:24.2438085Z ==21314== For lists of detected and suppressed errors, rerun with: -s
2024-05-07T18:50:24.2438671Z ==21314== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
2024-05-07T18:50:24.2510597Z qemu: uncaught target signal 11 (Segmentation fault) - core dumped
2024-05-07T18:50:24.3450513Z /home/runner/work/_actions/uraimo/run-on-arch-action/v2.2.1/src/run-on-arch-commands.sh: line 19: 21314 Segmentation fault      (core dumped) valgrind --leak-check=full ./modules/core/testNPZ
2024-05-07T18:50:24.9499409Z ##[error]The process '/home/runner/work/_actions/uraimo/run-on-arch-action/v2.2.1/src/run-on-arch.sh' failed with exit code 139
2024-05-07T18:50:24.9585940Z Post job cleanup.
2024-05-07T18:50:25.0389233Z [command]/usr/bin/git version
2024-05-07T18:50:25.0439484Z git version 2.43.2
2024-05-07T18:50:25.0508693Z Temporarily overriding HOME='/home/runner/work/_temp/d0403bb5-b057-4ea4-bdca-c0d29991a48a' before making global git config changes
2024-05-07T18:50:25.0510599Z Adding repository directory to the temporary git global config as a safe directory
2024-05-07T18:50:25.0514519Z [command]/usr/bin/git config --global --add safe.directory /home/runner/work/visp/visp
2024-05-07T18:50:25.0555260Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-05-07T18:50:25.0592768Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-05-07T18:50:25.0872505Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-05-07T18:50:25.0895121Z http.https://github.com/.extraheader
2024-05-07T18:50:25.0909366Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2024-05-07T18:50:25.0944773Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-05-07T18:50:25.1526481Z Cleaning up orphan processes

• Build on ubuntu20.04 s390x (Big Endian)

log

Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "s390x-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./modules/core/testNPZ...
Starting program: /home/runner/work/visp/visp/build/modules/core/testNPZ 
warning: Error disabling address space randomization: Operation not permitted
warning: Could not trace the inferior process.
warning: ptrace: Function not implemented
During startup program exited with code 127.
==32760== Memcheck, a memory error detector
==32760== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==32760== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==32760== Command: ./modules/core/testNPZ
==32760== 
==32760== error calling PR_SET_PTRACER, vgdb might block
==32760== Invalid read of size 8
==32760==    at 0x10A3F6: main (testNPZ.cpp:160)
==32760==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==32760== 
==32760== 
==32760== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==32760==  Access not within mapped region at address 0x0
==32760==    at 0x10A3F6: main (testNPZ.cpp:160)
==32760==  If you believe this happened as a result of a stack
==32760==  overflow in your program's main thread (unlikely but
==32760==  possible), you can try to increase the size of the
==32760==  main thread stack using the --main-stacksize= flag.
==32760==  The main thread stack size used in this run was 16777216.

[s-trinh]

Possible code improvements:

• pkestene/cnpy-cmake@37d8ad2
• pkestene/cnpy-cmake@e79f495
• look at ENH: np.save() to align data at 64 bytes numpy/numpy#9025 ? see how to deal with format versioning? https://github.com/numpy/numpy/pull/9654/files
• https://numpy.org/doc/stable/reference/generated/numpy.memmap.html / https://stackoverflow.com/questions/16149803/working-with-big-data-in-python-and-numpy-not-enough-ram-how-to-save-partial-r