Future3/update dependencies (MiczFlor#2103)

Update python venv
Update Bookworm
Update node to v20 (lts) and npm minor dependencies
Update docker to all of the above

.githooks/post-merge

@@ -43,7 +43,7 @@ warn_python_requirements() {
   echo "ATTENTION: Python requirements have changed since last pull!"
   echo ""
   echo "To update python requirements on the RPi run"
-  echo "$ sudo python3 -m pip install --upgrade -r requirements.txt"
+  echo "$ python -m pip install --upgrade -r requirements.txt"
   echo "************************************************************"
   echo -e "\n"
 }

docker/armv7/jukebox.Dockerfile

@@ -25,7 +25,7 @@ RUN apt-get update && apt-get install -qq -y \
     --allow-downgrades --allow-remove-essential --allow-change-held-packages \
     at wget gcc \
     mpc mpg123 git ffmpeg spi-tools netcat alsa-tools \
-    python3 python3-dev python3-pip python3-setuptools python3-mutagen python3-gpiozero
+    python3 python3-venv python3-dev python3-pip python3-setuptools python3-mutagen python3-gpiozero
 #samba samba-common-bin
 #raspberrypi-kernel-headers
 #resolvconf
@@ -57,17 +57,21 @@ RUN mkdir -p ${ZMQ_TMP_DIR} && cd ${ZMQ_TMP_DIR}; \
 #     zeromq-${ZMQ_VERSION}/configure --prefix=${ZMQ_PREFIX} --enable-drafts; \
 #     make && make install;
 
+ENV VIRTUAL_ENV=/opt/venv
+RUN python3 -m venv $VIRTUAL_ENV
+ENV PATH="$VIRTUAL_ENV/bin:$PATH"
+
 USER ${USER}
 WORKDIR ${HOME}
 COPY --chown=${USER}:${USER} . ${INSTALLATION_PATH}/
 
-RUN pip3 install --no-cache-dir -r ${INSTALLATION_PATH}/requirements.txt
-RUN pip3 install --no-cache-dir --pre --no-binary pyzmq pyzmq
+RUN pip install --no-cache-dir -r ${INSTALLATION_PATH}/requirements.txt
+RUN pip install --no-cache-dir --pre --no-binary pyzmq pyzmq
 
 EXPOSE 5555 5556
 
 WORKDIR ${INSTALLATION_PATH}/src/jukebox
 
 # Run Jukebox
 # CMD bash
-CMD python3 ${INSTALLATION_PATH}/src/jukebox/run_jukebox.py
+CMD python ${INSTALLATION_PATH}/src/jukebox/run_jukebox.py

docker/docker-compose.yml

@@ -42,7 +42,7 @@ services:
     volumes:
       - ../shared:/root/RPi-Jukebox-RFID/shared
       - ./config/docker.pulse.mpd.conf:/root/.config/mpd/mpd.conf
-    command: python3 run_jukebox.py
+    command: python run_jukebox.py
 
   webapp:
     build:

docker/jukebox.Dockerfile

@@ -25,14 +25,18 @@ RUN apt-get update && apt-get install -qq -y \
     --allow-downgrades --allow-remove-essential --allow-change-held-packages \
     gcc at wget \
     espeak mpc mpg123 git ffmpeg spi-tools netcat \
-    python3 python3-dev python3-pip python3-mutagen python3-gpiozero
+    python3 python3-venv python3-dev python3-pip python3-mutagen python3-gpiozero
+
+ENV VIRTUAL_ENV=/opt/venv
+RUN python3 -m venv $VIRTUAL_ENV
+ENV PATH="$VIRTUAL_ENV/bin:$PATH"
 
 USER ${USER}
 WORKDIR ${HOME}
 COPY --chown=${USER}:${USER} . ${INSTALLATION_PATH}/
 
-RUN pip3 install --no-cache-dir -r ${INSTALLATION_PATH}/requirements.txt
-RUN pip3 install pyzmq
+RUN pip install --no-cache-dir -r ${INSTALLATION_PATH}/requirements.txt
+RUN pip install pyzmq
 
 EXPOSE 5555 5556
 

docker/webapp.Dockerfile

@@ -1,4 +1,4 @@
-FROM node:18-bullseye-slim
+FROM node:20-alpine
 
 USER node
 RUN mkdir -p /home/node/webapp

documentation/content/developers/development-environment.md

@@ -62,7 +62,7 @@ Github, which is taken care of by the installation script. For regular
 machines, the normal package can be installed:
 
 ``` bash
-pip3 install pyzmq
+pip install pyzmq
 ```
 
 You will have to start Jukebox core application and the WebUI

documentation/content/userguide/installation.md

@@ -8,15 +8,29 @@
 Before you can install the Phoniebox software, you need to prepare your Raspberry Pi.
 
 1. Connect a Micro SD card to your computer (preferable an SD card with high read throughput)
-2. [Download](https://www.raspberrypi.org/software/)
-   the [Raspberry Pi Imager](https://www.raspberrypi.org/blog/raspberry-pi-imager-imaging-utility/) and open it
-3. Select **Raspberry Pi OS Lite (32-bit)** (without desktop environment) as the operating system. future3 does not support 64bit kernels (`aarch64`). You can use the settings menu of the Raspberry Pi Imager to configure SSH and WiFi in a more userfriendly way, or do it manually as described in the next step. In case you already have a 64bit system installed, [you can fix the issue like this](https://github.com/MiczFlor/RPi-Jukebox-RFID/issues/2041>).
-4. Select your Micro SD card (your card will be formatted)
-5. Click *Write*
-6. Wait for the imaging process to be finished (it'll take a few minutes)
-
-
-## Pre-boot preparation
+2. Download the [Raspberry Pi Imager](https://www.raspberrypi.com/software/) and run it
+3. Click on "Raspberry Pi Device" and select "No filtering"
+4. Select **Raspberry Pi OS Lite (32-bit)** (without desktop environment) as the operating system. `future3` does not support 64bit kernels (`aarch64`).
+5. Select your Micro SD card (your card will be formatted)
+6. After you click `Next`, a prompt will ask you if you like to customize the OS settings
+    * Click `Edit Settings`
+    * Switch to the `General` tab
+        * Provide a hostname. (When on Mac, you will be able to use it to connect via SSH)
+        * Username currently MUST be `pi`. Other usernames are currently not supported.
+        * Password
+        * Wifi
+        * Set locale settings
+    * Switch to the `Services` tab. Enable SSH with "Use password authentication"
+    * Click `Save`
+6. In the same dialog, click `Yes`
+7. Confirm the next warning about erasing the SD card with `Yes`
+8. Wait for the imaging process to be finished (it'll take a few minutes)
+
+<details>
+
+<summary>In case you forgot to customize the OS settings, follow these instructions after RPi OS has been written to the SD card.</summary>
+
+### Pre-boot preparation
 
 You will need a terminal, like PuTTY for Windows or the Terminal app for Mac to proceed with the next steps.
 
@@ -62,6 +76,8 @@ You will need a terminal, like PuTTY for Windows or the Terminal app for Mac to
 9. Login into your Raspberry Pi, username is `pi` and password is `raspberry`.
    If `raspberrypi.local` does not work, find out your Raspberry Pi's IP address from your router.
 
+</details>
+
 ## Install Phoniebox software
 
 Run the following command in your SSH terminal and follow the instructions
@@ -90,4 +106,3 @@ If you suspect an error you can monitor the installation-process with
 ```bash
 cd; tail -f INSTALL-<fullname>.log
 ```
-

installation/includes/03_welcome.sh

@@ -16,14 +16,15 @@ You are turning your Raspberry Pi into a Phoniebox.
 Good choice!
 
 Depending on your hardware, this installation might last
-around 60 minutes. It updates OS packages, installs
-Phoniebox dependencies and registers settings. Be patient
-and don't let your computer go to sleep. It might
-disconnect your SSH connection causing the interruption of
-the installation process.
+around 60 minutes (usually it's faster). It updates OS
+packages, installs Phoniebox dependencies and registers
+settings. Be patient and don't let your computer go to
+sleep. It might disconnect your SSH connection causing the
+interruption of the installation process.
 
-By the way, we write a log file to:
-${INSTALLATION_LOGFILE}
+By the way, you can follow the installation details here
+in a separate SSH session:
+cd; tail -f ${INSTALLATION_LOGFILE}
 
 Let's set up your Phoniebox now?! [Y/n]" 1>&3
 

installation/install-jukebox.sh

@@ -32,15 +32,15 @@ checkPrerequisite() {
   #currently the user 'pi' is mandatory
   #https://github.com/MiczFlor/RPi-Jukebox-RFID/issues/1785
   if [ "${CURRENT_USER}" != "pi" ]; then
-    echo 
+    echo
     echo "ERROR: User must be 'pi'!"
     echo "       Other usernames are currently not supported."
     echo "       Please check the wiki for further information"
     exit 2
   fi
 
   if [ "${HOME_PATH}" != "/home/pi" ]; then
-    echo 
+    echo
     echo "ERROR: HomeDir must be '/home/pi'!"
     echo "       Other usernames are currently not supported."
     echo "       Please check the wiki for further information"

installation/routines/customize_options.sh

@@ -35,7 +35,7 @@ _option_autohotspot() {
   # ENABLE_AUTOHOTSPOT
   echo "Do you want to enable a WiFi hotspot on demand?
 When enabled, this service spins up a WiFi hotspot
-when the Phonbox is unable to connect to a known
+when the Phoniebox is unable to connect to a known
 WiFi. This way you can still access it.
 [y/N] " 1>&3
   read -r response
@@ -196,25 +196,29 @@ _option_webapp_devel_build() {
     if [[ "$ENABLE_WEBAPP_PROD_DOWNLOAD" = "release-only" ]]; then
       ENABLE_WEBAPP_PROD_DOWNLOAD=false
     fi
-    echo -e "Your are installing from a non-release branch.
+
+    if [[ "$ENABLE_WEBAPP_PROD_DOWNLOAD" = false ]]; then
+      echo -e "Your are installing from a non-release branch.
 This means, you will need to build the web app locally.
 For that you'll need Node.
-Do you want to install Node now?  [Y/n] " 1>&3
-    read -r response
-    case "$response" in
-      [nN])
-        ENABLE_INSTALL_NODE=false
-        ;;
-      *)
-        ;;
-    esac
-    # This message will be displayed at the end of the installation process
-    FIN_MESSAGE="$FIN_MESSAGE\n\nATTENTION: You need to build the web app locally with
-    $ cd ~/RPi-Jukebox-RFID/src/webapp && ./run_rebuild.sh -u
-    This must be done after reboot, due to memory restrictions.
-    Read the documentation regarding local Web App builds!"
-  fi
+Do you want to install Node?  [Y/n] " 1>&3
+      read -r response
+      case "$response" in
+        [nN])
+          ENABLE_INSTALL_NODE=false
+          ;;
+        *)
+          ;;
+      esac
 
+      # This message will be displayed at the end of the installation process
+      FIN_MESSAGE="$FIN_MESSAGE\n\nATTENTION: You need to build the web app locally with
+      $ cd ~/RPi-Jukebox-RFID/src/webapp && ./run_rebuild.sh -u
+      This must be done after reboot, due to memory restrictions.
+      Read the documentation regarding local Web App builds!"
+      ENABLE_WEBAPP_PROD_DOWNLOAD=false
+    fi
+  fi
 }
 
 customize_options() {

installation/routines/set_raspi_config.sh

@@ -6,7 +6,7 @@ set_raspi_config() {
   # Source: https://raspberrypi.stackexchange.com/a/66939
 
   # Autologin
-  echo "  * Enable Autologin for 'pi' user"
+  echo "  * Enable Autologin for user"
   sudo raspi-config nonint do_boot_behaviour B2
 
   # Wait for network at boot

installation/routines/setup_jukebox_core.sh

@@ -23,7 +23,7 @@ _jukebox_core_install_os_dependencies() {
   sudo apt-get -y update; sudo apt-get -y install \
     at \
     alsa-utils \
-    python3 python3-dev python3-pip python3-setuptools \
+    python3 python3-venv python3-dev python3-pip python3-setuptools \
     python3-rpi.gpio python3-gpiozero \
     espeak ffmpeg mpg123 \
     pulseaudio pulseaudio-module-bluetooth pulseaudio-utils caps \
@@ -33,14 +33,12 @@ _jukebox_core_install_os_dependencies() {
     --allow-remove-essential \
     --allow-change-held-packages
 
-  # add configuration that we break the global python system packages
-  # (required for bookworm, see 
-  # https://github.com/MiczFlor/RPi-Jukebox-RFID/issues/2050). This
-  # should be removed once the jukebox has been isolated to a dedicated
-  # venv
-  sudo python3 -m pip config set global.break-system-packages true
+  VIRTUAL_ENV="${HOME_PATH}/.venv"
+  python3 -m venv $VIRTUAL_ENV
+  PATH="$VIRTUAL_ENV/bin:$PATH"
+  source "$VIRTUAL_ENV/bin/activate"
 
-  sudo pip3 install --upgrade pip
+  pip install --upgrade pip
 }
 
 _jukebox_core_configure_pulseaudio() {
@@ -86,7 +84,7 @@ _jukebox_core_build_and_install_pyzmq() {
   # https://github.com/MonsieurV/ZeroMQ-RPi/blob/master/README.md
   echo "  Build and install pyzmq with WebSockets Support"
 
-  if ! sudo pip3 list | grep -F pyzmq >> /dev/null; then
+  if ! pip list | grep -F pyzmq >> /dev/null; then
     # Download pre-compiled libzmq from Google Drive because RPi has trouble compiling it
     echo "    Download pre-compiled libzmq from Google Drive because RPi has trouble compiling it"
 
@@ -106,8 +104,8 @@ _jukebox_core_build_and_install_pyzmq() {
       _jukebox_core_download_prebuilt_libzmq_with_drafts
     fi
 
-    sudo ZMQ_PREFIX="${ZMQ_PREFIX}" ZMQ_DRAFT_API=1 \
-         pip3 install --no-cache-dir --no-binary "pyzmq" --pre pyzmq
+    ZMQ_PREFIX="${ZMQ_PREFIX}" ZMQ_DRAFT_API=1 \
+      pip install --no-cache-dir --no-binary "pyzmq" --pre pyzmq
   else
     echo "    Skipping. pyzmq already installed"
   fi
@@ -116,7 +114,7 @@ _jukebox_core_build_and_install_pyzmq() {
 _jukebox_core_install_python_requirements() {
   echo "  Install requirements"
   cd "${INSTALLATION_PATH}"  || exit_on_error
-  sudo pip3 install --no-cache-dir -r "${INSTALLATION_PATH}/requirements.txt"
+  pip install --no-cache-dir -r "${INSTALLATION_PATH}/requirements.txt"
 }
 
 _jukebox_core_install_settings() {

installation/routines/setup_jukebox_webapp.sh

@@ -4,7 +4,7 @@
 GD_ID_COMPILED_WEBAPP="1EE_1MdneGtKL5V7GyYZC0nb6ODQWTsPb" # https://drive.google.com/file/d/1EE_1MdneGtKL5V7GyYZC0nb6ODQWTsPb/view?usp=sharing
 
 # For ARMv7+
-NODE_MAJOR=16
+NODE_MAJOR=20
 # For ARMv6
 # To update version, follow these links
 # https://github.com/sdesalas/node-pi-zero

installation/routines/setup_rfid_reader.sh

@@ -3,7 +3,7 @@
 setup_rfid_reader() {
   echo "Install RFID Reader" | tee /dev/fd/3
 
-  python3 "${INSTALLATION_PATH}/src/jukebox/run_register_rfid_reader.py" | tee /dev/fd/3
+  python "${INSTALLATION_PATH}/src/jukebox/run_register_rfid_reader.py" | tee /dev/fd/3
 
   echo "DONE: setup_rfid_reader"
 }

resources/default-services/jukebox-daemon.service

@@ -10,8 +10,8 @@ After=network.target sound.target mpd.service pulseaudio.service
 Requires=mpd.service pulseaudio.service
 
 [Service]
-ExecStart=/usr/bin/python3 run_jukebox.py
 WorkingDirectory=/home/pi/RPi-Jukebox-RFID/src/jukebox
+ExecStart=/bin/bash -c 'source /home/pi/.venv/bin/activate && /home/pi/.venv/bin/python run_jukebox.py'
 StandardOutput=inherit
 StandardError=inherit
 Restart=always

src/jukebox/components/rfid/configure/__init__.py

@@ -27,17 +27,17 @@ def reader_install_dependencies(reader_path: str, dependency_install: str) -> No
             # The python dependencies (if any)
             print("\nInstalling/Checking Python dependencies  ...\n")
             print("IMPORTANT for developers: Python dependencies will be installed using "
-                  " $ sudo pip3 install --upgrade -r requirements.txt'\n"
+                  " $ pip install --upgrade -r requirements.txt'\n"
                   " i.e. on system level. This is target for the default RPI setup. "
                   "If you do not want that, but rather have them in a local or virtual environment, "
                   "hit No here and manually install the dependencies from your virtual environment\n"
-                  " $ pip3 install --upgrade -r requirements.txt'\n"
+                  " $ pip install --upgrade -r requirements.txt'\n"
                   "It is no problem to install them after running this script.\n\n")
             if dependency_install == 'auto' or pyil.input_yesno("Install Python dependencies?", blank=True,
                                                                 prompt_color=Colors.lightgreen, prompt_hint=True):
                 print(f"{'=' * 80}")
                 quiet_level = '-q' if logger.isEnabledFor(logging.DEBUG) else ''
-                subprocess.run(f"sudo pip3 install --upgrade {quiet_level} -r requirements.txt", cwd=reader_path,
+                subprocess.run(f"pip install --upgrade {quiet_level} -r requirements.txt", cwd=reader_path,
                                shell=True, check=False)
                 print(f"\n{'=' * 80}\nInstalling dependencies ... done!")
         if os.path.exists(reader_path + '/setup.inc.sh'):
@@ -78,7 +78,7 @@ def reader_load_module(reader_name):
                             "If this script is called with -d a, an attempt will be made to install the dependencies "
                             "automatically\n"
                             "You may install the dependencies manually before re-executing this script by:\n"
-                            "'$ pip3 install -r requirements.txt' in the reader's submodule directory and \n"
+                            "'$ pip install -r requirements.txt' in the reader's submodule directory and \n"
                             "'$ ./setup.inc.sh'\n"
                             "In case of doubt reboot!\n\n"
                             f"{'=' * 80}\n")

src/jukebox/components/rfid/hardware/pn532_i2c_py532/requirements.txt

@@ -1,3 +1,3 @@
 # PN532 related requirements
-# You need to install these with `sudo python3 -m pip install --upgrade --force-reinstall -q -r requirements.txt`
+# You need to install these with `python -m pip install --upgrade --force-reinstall -q -r requirements.txt`
 py532lib

src/jukebox/components/rfid/hardware/rc522_spi/requirements.txt

@@ -1,5 +1,5 @@
 # RC522 related requirements
-# You need to install these with `sudo python3 -m pip install --upgrade --force-reinstall -q -r requirements.txt`
+# You need to install these with `python -m pip install --upgrade --force-reinstall -q -r requirements.txt`
 
 pi-rc522==2.3.0
 

src/jukebox/jukebox/NvManager.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 import json
 import os

src/jukebox/jukebox/playlistgenerator.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python
 """
 Playlists are build from directory content in the following way:
 a directory is parsed and files are added to the playlist in the following way

src/jukebox/run_configure_audio.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python
 """
 Setup tool to register the PulseAudio sinks as primary and secondary audio outputs.
 

src/jukebox/run_jukebox.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python
 """
 This is the main app and starts the Jukebox Core.