Undeterministic installation order for rosdep install

[Bjoernolav]

We are struggeling with versioning of dependencies in our ROS2 system. It seems like rosdep install does not install dependencies in a deterministic order.

Use case:
Given two dependencies, apt_pkg (installed by apt) and pip_pkg (installed by pip), that both have a python dependency on e.g. mypy.

• apt_pkg has a python3-mypy dependency tag, and installs mypy using apt.
• pip_pkg has the mypy dependency listed in the setup.py, and mypy is installed using pip.

The resulting version of mypy will depend on which of dependencies are handled first since installing it with apt and pip yields two different versions.

Running rosdep in simulated mode multiple times gives the follow output:

root@6d3184b2af94:/ros2# rosdep install -is --from-paths src -y --reinstall
#[pip] Installation commands:
  pip3 install -U -I pip_pkg
#[apt] Installation commands:
  apt-get install -y apt_pkg

root@6d3184b2af94:/ros2# rosdep install -is --from-paths src -y --reinstall
#[apt] Installation commands:
  apt-get install -y apt_pkg
#[pip] Installation commands:
  pip3 install -U -I pip_pkg

Is this the intended behavior? I would expect the order to be deterministic.

[cottsay]

Hmm, the nondeterminism certainly isn't intentional. The list of uninstalled rules gets dumped into a Python dict at some point, so there wasn't any effort to retain ordering there. That said, insertion order is retained in a dict with Python 3.7 and newer.

What platform are you experiencing this on?

[Bjoernolav]

We ran this in a docker container based on ros:humble. There is no parallell processing in rosdep potentially introducing this?

Some output:

root@6d3184b2af94:/# printenv
HOSTNAME=6d3184b2af94
PWD=/
HOME=/root
LANG=C.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
TERM=xterm
SHLVL=1
LC_ALL=C.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ROS_DISTRO=humble
_=/usr/bin/printenv

root@6d3184b2af94:/# python3 --version
Python 3.10.4

[lights0123]

As a workaround, I run this script to get all the packages rosdep wantsat once, then passes all of them to apt-get, where order doesn't matter. This provides a significant speed improvement as well:

packages=$(rosdep install -s | grep -oP 'apt-get install\s+\K\S+')
# echo $packages # if you want to see what will be installed
echo $packages | xargs apt-get install -y

My environment doesn't have any pip dependencies, but a similar script could be used for that as well.