"One line installation" fails on a vanilla MACOS 14 install

[c-joly]

Hello @costashatz, this is an issue linked to the JOSS review openjournals/joss-reviews#6771

I tried to install robot_dart on a fresh install of MacOS 14.5 (Virtual Machine, M2 series Processor, MACOS 14.5), but it did not succeed. I attached the terminal output at https://gist.github.com/c-joly/e26956198d2438ef62f1291c0b6c2809. Here are the main problems that I see (but it might have others):

• pip3 install dartpy does not pass
• cmake is not found, so no Makefile is generated and the make command provided no output
• waf not found
• After installing cmake via brew, there are other problems with pip3 that appears obviously ("This environment is externally managed; To install Python packages system-wide, try brew install...").
• I also saw an error python not found... By default, the system uses python3 command only, there is no symlink python on a my vanilla MACOS-14.

It looks like the one line installation script makes several hypotheses concerning the machine configuration (homebrew, cmake, some installation of python). Can you list the requirements?

Finally, I realized that in case of error, the script continues to execute, so it is difficult to know if the installation succeeded or not (it might be possible to find some errors in the middle of the log). I suggest to configure the script to stop at the first encountered error (see bash strict mode). When it is configured like this, it stops with the error python not found.

[costashatz]

Hello @c-joly,

Thanks for the comment and the effort.

I attached the terminal output at https://gist.github.com/c-joly/e26956198d2438ef62f1291c0b6c2809

Thanks for this. Really helpful.

pip3 install dartpy does not pass

Hmmm. I used pip to avoid having to deal with compiling from source. I am suspecting that the dart developers do not have pre-built binaries for M2 processors. I will update the script to build DART from source.

cmake is not found, so no Makefile is generated and the make command provided no output

This is very weird. brew install dartsim --only-dependencies and brew install binutils pybind11 should install cmake! In my MacOS CI (MacOS 12), I am not installing cmake explicitly and it works. In any case, this is an easy fix: I will update the script to install all the base compilation packages (clang, cmake, etc).

waf not found

Where did you run the script from? You should be in the root of the robot_dart repo. I guess this might be related to the python issue. Ahh no, I see why this happened. I will fix it.

there are other problems with pip3 that appears obviously ("This environment is externally managed; To install Python packages system-wide, try brew install...").

These should be only warnings. That's what I see in the output as well. We can safely ignore those. Or these were errors? I do not see any such error in the log.

I also saw an error python not found...

Yes. I will fix that.

It looks like the one line installation script makes several hypotheses concerning the machine configuration (homebrew, cmake, some installation of python). Can you list the requirements?

python should be there by default, no?

I'll handle cmake in the updated version. I will add homebrew as a requirement (I assumed it's always there as I know no Mac user without using brew).

I suggest to configure the script to stop at the first encountered error (see bash strict mode).

Thanks for the suggestion. I will update the script.

@c-joly thanks for the comments! I will update the script and let you know so that you can try on a clean installation again! I am not using Mac and I trusted the GitHub CI Mac machine; turns out I needed to be more careful!

[c-joly]

Hello @costashatz ,

Thanks for your fast reply!

This is very weird. brew install dartsim --only-dependencies and brew install binutils pybind11 should install cmake! In my MacOS CI (MacOS 12), I am not installing cmake explicitly and it works. In any case, this is an easy fix: I will update the script to install all the base compilation packages (clang, cmake, etc).

I was surprised too, since homebrew provides cmake

These should be only warnings. That's what I see in the output as well. We can safely ignore those. Or these were errors? I do not see any such error in the log.

These were errors. Eventually, I decided to not paste them since it might be due to the problems of python. I saw these errors after I launched the script a second time (after performing brew install cmake)

python should be there by default, no?

Yes, but on the install I used, the command is python3 and the OS does not create a symbolic link python by default. To be honest, I never use the python that comes with the OS, since I cannot really control it. Instead, I use conda environments and I have a specific version of python for each project. I may recommend to do the same for robot_dart, but at a first step, I wanted to test the standard install procedure.

I'll handle cmake in the updated version. I will add homebrew as a requirement (I assumed it's always there as I know no Mac user without using brew).

Thanks. I am not a big fan of homebrew since it changes the ownership of /opt/ (so I don't install it on my main system). For 99% of my usecases, the libraries I need can be found in conda or pypi repositories, so I don't need other package managers like homebrew or macport anymore.

Thanks very much again for your fast answer. Will try again the install when the new script is available.

[c-joly]

Hi @costashatz , I think I succeed to update the install script for MacOS Silicon. I will summarize bellow the parts that are not working well with the original script. If you want, I can eventually propose a pull request of my script as an extra script install_silicon.sh

Prerequisites

The script does not work with an out of the box install of MacOS. In my previous message, I mentioned that cmake is missing and homebrew is needed. I figured out that we need more:

• XQuartz: this is a software used to start an X server and launch X11 applications. After the installation, it add some includes files, especially "GLx.h" which is not there by default and is needed for one of the graphic libraries compilation (magnum or corrade, I don’t remember). XQuartz used to be very popular to run some software that have been port from Linux, e.g. Inkscape, so many Mac users used to install it. However, many of these software (including Inkscape) have been updated and do not need XQuartz anymore. So, there is a high probability that a recent Mac does not have this software already installed.

• python: by default, MacOS comes with the builtin command python3, there is not any symbolic link with python command. Although it is possible to create one directly, I do not recommend this option. I chose to install an other python with brew and then create a virtual environment with the command:

   python3 -m venv ~/env_dart
  

  Note: before creating the virtual environment, make sure that python3 command is the one installed by homebrew by using:

  which python3
  

  If it is not the case, restart your terminal after python installation via homebrew.

  Then, once the environment is installed, it can be activated with

  source ~/env_dart/bin/activate
  

  This will provide directly the command python and so, the user does not have anything to do more. Using venv has two advantages:

  1. It avoids any side effect with the system installation of python, which is something I like
  2. It avoids an error with pip, that detects that some libraries are installed with brew and does not like anymore mixing the "system"pip and other libraries. (Note, it is still possible to cancel this behavior by passing the option —break-system-packages to pip, but as the name suggests, it might have some horrible side effects, including a sudden loss of cabine pressure 😀)
     An other possible way might ber the use of a conda environment. However, dart documentation mentions some issues with that, so I do not recommend for an automatic installation.

Missing include in a jpeglib.h installed by homebrew

The compilation of MagnumPlugins fails because FILE is not recognized in the file jpeglib.h. I figured out this is because we need to include <stdio.h>. Maybe some linux compilers make this include by default, making the explicit include unnecessary, but it was not the case with the compiler called by cmake on my computer (Clang). Thus, I add in the script some lines to add automatically this include if necessary:

if ! grep -q '#include <stdio.h>' '/opt/homebrew/Cellar/jpeg-turbo/3.0.3/include/jpeglib.h'; then
   sed -i '' '1s/^/ #include <stdio.h>\n/' /
fi

Note: this solution is not perfect since I am using a hardcoded path with the specific version of the library. It might be possible to use some special variables of homebrew to make it more generic, but I did not have time to look how to do this. By the way, this is an ugly patch that makes a change in the source code of a library installed by homebrew! I think an issue should be posted in order to have an official update of this header.

Dart and dartpy compilation

I add a block to compile and install dart and dartpy, based on the official documentation. I figured out that I had to checkout a tagged version. I picked the last one.; if I don’t do the checkout, the waf script dart.py ends with an error (it looks to have troubles to detect the version number, so it fails).

dart is finally installed in /opt/dart

Environment variables

I figured out that the variable LD_LIBRARY is not enough to find the libraries at runtime. Depending on the prefix install of the libraries, I got some errors with @rpath at runtime, showing that the shared libraries cannot be found. On MacOS, we can add values to rpath with the environment variable DYLD_LIBRARY_PATH. Moreover, I had to manually add dart since it has been compiled. Finally, I update the PYTHONPATH variable to find dartpy which is not installed in the standard path for python packages. So, I add the following in the script:

export LD_LIBRARY_PATH=/opt/dart/lib:/opt/magnum/lib:$LD_LIBRARY_PATH
export DYLD_LIBRARY_PATH=$LD_LIBRARY_PATH
export PYTHONPATH=/opt/dart:$PYTHONPATH 

(I don't think LD_LIBRARY_PATH is necessary, but I am not sure, so I kept it).

waf configuration

When installing the compiled libraries in /opt/dart and /opt/magnum, the waf configuration script fails to find them. I updated the command line with the options providing the path of these librairies. Then, I also prefixed the waf command by python3 as recommended in the manual installation page

python3 ./waf configure --prefix /opt/robot_dart --python --dart /opt/dart/ --corrade_install_dir /opt/magnum --magnum_install_dir /opt/magnum --magnum_plugins_install_dir /opt/magnum --magnum_integration_install_dir /opt/magnum
python3 ./waf -j8
python3 ./waf examples -j8
sudo python3 ./waf install

Miscellaneous

• I changed the first line with set -e to make sure the script stops at the first error
• When compiling the libraries, I add lines to check if the directory build exists or not; otherwise, mkdir will produce an error if build exists which is blocking with set -e:

  if [ -d "build" ] # In case of a previous attempt that has not been clean
  then
    rm -rf build
  fi
  

• Finally, if we want to run the prebuilt C++ examples or import RobotDART in python, we just need to update the environment variables PYTHONPATH and DYLD_LIBRARY_PATH. In my case:

    export DYLD_LIBRARY_PATH=/opt/dart/lib:/opt/magnum/lib:$DYLD_LIBRARY_PATH
    export PYTHONPATH=/opt/dart:/opt/robot_dart/lib/python3.12/site-packages
  

  These two lines should be add at the end of one of the startup script launched with the terminal (.zshrc,.zshprofile, .bashrc,... depending on the shell used).

[costashatz]

@c-joly this is great! Yes please do provide a pull request (please allow for changes), and I will try to adapt it for all MacOSes.

Thanks again for your hard work! Appreciated!

[costashatz]

@c-joly I have updated the installation script for OSX in the master branch. I tried on an M1 apple machine and everything worked fine! Your script (from the PR) was very valuable! Thanks again!

One note is that I never encountered the Missing include in a jpeglib.h installed by homebrew issue. So I did not include this in my script as it appears that this has something to do with your local installation (and it is hacky to put hard-coded paths)?

Can you please verify that new script works also for you (apart maybe from this jpeglib issue)? Thanks! If the jpeglib issue persists, I will put a comment in the installation instructions linking to this issue in case someone encounters something similar.

[c-joly]

Hi @costashatz ,

I am currently trying the new MacOS script on a fresh VM (MacOS 14, last version). I first installed ffmpeg, Xquartz and CMake (Xquartz brings some headers linked to OpenGL that I don't have by default).

But the installation does not succeed, I have some weird errors linked to waf that should be found in the dart directory but cannot be found during the installation of magnum_bindings. Bellow the last lines of the console output (complete log of the magnum_bindings installation available on https://gist.github.com/c-joly/ba003b67c5db31eb904d45736fb0bf82):

running install_egg_info
running egg_info
creating magnum.egg-info
writing magnum.egg-info/PKG-INFO
writing dependency_links to magnum.egg-info/dependency_links.txt
writing top-level names to magnum.egg-info/top_level.txt
writing manifest file 'magnum.egg-info/SOURCES.txt'
reading manifest file 'magnum.egg-info/SOURCES.txt'
writing manifest file 'magnum.egg-info/SOURCES.txt'
Copying magnum.egg-info to /opt/magnum/lib/python3/site-packages/magnum-0.0.0-py3.12.egg-info
running install_scripts
/opt/homebrew/Cellar/[email protected]/3.12.5/Frameworks/Python.framework/Versions/3.12/Resources/Python.app/Contents/MacOS/Python: can't open file '/Users/cyril/robot_dart/temp_robot_dart/dart/waf': [Errno 2] No such file or directory

It looks like a file is missing in dart directory... This is weird because I did not have any similar error previously.

Then, I have an other problem when I launch again the script: the automatic cleaning does not work (access denied). Removing temp_dart manually does not work: it looks like some files/folders belong to root. I have to remove them by using sudo (which I think is not the expected behavior). FYI, the result of ls in one of the subsubdirectory of temp_dart:

cyril@Machine-virtuelle-de-cyril lib.macosx-14.0-arm64-cpython-312 % pwd
/Users/cyril/robot_dart/temp_robot_dart/dart/build/magnum-bindings/build/src/python/build/lib.macosx-14.0-arm64-cpython-312
cyril@Machine-virtuelle-de-cyril lib.macosx-14.0-arm64-cpython-312 % ls -la
total 8088
drwxr-xr-x   6 root  staff      192  6 sep 05:38 .
drwxr-xr-x   3 root  staff       96  6 sep 05:38 ..
-rw-r--r--   1 root  staff   130128  6 sep 05:38 _corrade.cpython-312-darwin.so
-rw-r--r--   1 root  staff  4007984  6 sep 05:38 _magnum.cpython-312-darwin.so
drwxr-xr-x   6 root  staff      192  6 sep 05:38 corrade
drwxr-xr-x  13 root  staff      416  6 sep 05:38 magnum

Can you check on your side? And tell me with which version of MacOS it is working fine?

Finally, it looks like the GitHub action that makes the MacOS installation test fails too.

[costashatz]

And tell me with which version of MacOS it is working fine?

14.5

Can you check on your side?

As far as the errors concerns, I had forgotten a folder depth. The osx script should be fixed now.

As for the privilege issue, I guess I should pre-prend sudo in every rm command since sudo make install creates files with root priviliges.

The script in the master branch should be OK (I hope -- I lost access to the Mac PC and I am working on finding a working VM/PC at the moment).

Let me know! Thanks!

Finally, it looks like the GitHub action that makes the MacOS installation test fails too.

This is because of dartsim/dart#1837 from DART. Should be fixed in the latest release. I will update the scripts to use the latest release (v6.14.5).

[c-joly]

Hi @costashatz

There is still the problem with waf --> Need to go up one more folder.

After I corrected the problem with the cd command, it does not work, because of dart (6.14.5 is still not working). I changed for the version I used in my version of the script (6.13.2), and then it works.

[costashatz]

@c-joly this is great!

I pushed a commit fixing this (see: 03011e5). Closing this as completed!

Thanks a ton for your help!