Merge pull request #416 from indigo-dc/devel3

Devel3

.github/workflows/codespell.yml

@@ -0,0 +1,23 @@
+# Codespell configuration is within setup.cfg
+---
+name: Codespell
+
+on:
+  push:
+    branches: [master]
+  pull_request:
+    branches: [master]
+
+permissions:
+  contents: read
+
+jobs:
+  codespell:
+    name: Check for spelling errors
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Codespell
+        uses: codespell-project/actions-codespell@v2

CHANGELOG.md

@@ -1,5 +1,27 @@
 # Changelog
 
+## udocker (1.3.13)
+
+* udocker improve binary executables identification
+* udocker fix fakechroot parsing of so, exec_path and add cmd subst
+* udocker implement minor pylint compliance improvements
+* udocker mode Pn make links2symlinks feature disabled by default in config: solves #412
+* New udockertools 1.2.11 tarball
+* udockertools mode Fn glibc fix dladdr(), dlopen(), dlmopen(), dl_iterate_phdr()
+* udockertools mode Fn glibc add dladdr1()
+* udockertools mode Fn glibc add execvpe(), execveat()
+* udockertools mode Fn glibc add getauxval()
+* udockertools mode Fn glibc add scandirat(), scandirat64()
+* udockertools mode Fn glibc change stat64(), lstat64(), stat()
+* udockertools mode Fn glibc add narrowing of program_invocation_name
+* udockertools mode Fn glibc improve command substitution
+* udockertools mode Fn musl fix dladdr(), dlopen(), dlmopen(), dl_iterate_phdr()
+* udockertools mode Fn musl execvpe()
+* udockertools mode Fn musl improve command substitution
+* udockertools mode Fn added support for Alpine 3.19 (x86_64)
+* udockertools mode Fn added support for Fedora 39 (x86_64, aarch64, ppc64le)
+* udockertools mode Rn include runc 1.1.12
+
 ## udocker (1.3.12)
 
 * fix units tests, no modifications w.r.t. 1.3.11

README.md

@@ -116,7 +116,8 @@ but is still available
           load                          :Load image from stdin (saved by docker)
           save -o <imagefile> <repo/image:tag>  :Save image with layers to file
 
-          inspect -p <repo/image:tag>   :Return low level information on image
+          inspect <repo/image:tag>      :Return low level information on image
+          inspect -p <container>        :Return path to container location
           verify <repo/image:tag>       :Verify a pulled or loaded image
           manifest inspect <repo/image:tag> :Print manifest metadata
 
@@ -146,18 +147,20 @@ but is still available
 
 Some examples of usage:
 
-Search container images in dockerhub.
+Search container images in dockerhub and listing tags.
 
 ```bash
 udocker search  fedora
 udocker search  ubuntu
-udocker search  indigodatacloud
+udocker search  debian
+
+udocker search --list-tags ubuntu
 ```
 
 Pull from dockerhub and list the pulled images.
 
 ```bash
-udocker pull   fedora:29
+udocker pull   fedora:39
 udocker pull   busybox
 udocker pull   iscampos/openqcd
 udocker images
@@ -182,7 +185,8 @@ udocker tag centos/centos8  mycentos/centos8:arm64
 
 Create a container from a pulled image, assign a name to the created
 container and run it. A created container can be run multiple times
-until it is explicitely removed.
+until it is explicitly removed. Files modified or added to the container
+remain available across executions until the container is removed.
 
 ```bash
 udocker create --name=myfed  fedora:29
@@ -221,7 +225,7 @@ udocker run  -v /tmp  --entrypoint=/bin/bash  myfed  -c 'cd /tmp; ./myscript.sh'
 
 Execute mounting the host /var, /proc, /sys and /tmp in the same container
 directories. Notice that the content of these container directories will
-be obfuscated.
+be obfuscated by the host files.
 
 ```bash
 udocker run -v /var -v /proc -v /sys -v /tmp  myfed  /bin/bash
@@ -283,6 +287,14 @@ udocker setup  --execmode=S1  myfed
 udocker run  --user=root myfed  yum install -y firefox pulseaudio gnash-plugin
 ```
 
+Change execution to enable nvidia ready applications. Requires that
+the nvidia drivers are installed in the host system.
+
+```bash
+udocker setup  --nvidia  mytensorflow
+```
+
+
 ## Security
 
 By default udocker via PRoot offers the emulation of the root user. This
@@ -341,8 +353,8 @@ udocker is mainly oriented at providing a run-time environment for
 containers execution in user space. udocker is particularly suited to
 run user applications encapsulated in docker containers.
 
-Debugging inside of udocker with the PRoot engine will not work due to
-the way PRoot implements the chroot environment
+Debugging or using strace with the PRoot engine will not work as both
+the debuggers and PRoot use the same tracing mechanism.  
 
 ## Execution mode specific limitations
 

codemeta.json

@@ -6,7 +6,7 @@
     "@type": "SoftwareSourceCode",
     "identifier": "udocker",
     "name": "udocker",
-    "version": "1.3.12",
+    "version": "1.3.13",
     "description": "A basic user tool to execute simple docker containers in batch or interactive systems without root privileges",
     "license": "Apache Software License 2.0, OSI Approved :: Apache Software License",
     "author": [

docs/installation_manual.md

@@ -32,18 +32,18 @@ udocker requires:
 Download a release tarball from <https://github.com/indigo-dc/udocker/releases>:
 
 ```bash
-wget https://github.com/indigo-dc/udocker/releases/download/1.3.12/udocker-1.3.12.tar.gz
-tar zxvf udocker-1.3.12.tar.gz
-export PATH=`pwd`/udocker-1.3.12/udocker:$PATH
+wget https://github.com/indigo-dc/udocker/releases/download/1.3.13/udocker-1.3.13.tar.gz
+tar zxvf udocker-1.3.13.tar.gz
+export PATH=`pwd`/udocker-1.3.13/udocker:$PATH
 ```
 
 Alternatively use `curl` instead of `wget` as follows:
 
 ```bash
-curl -L https://github.com/indigo-dc/udocker/releases/download/1.3.12/udocker-1.3.12.tar.gz \
-  > udocker-1.3.12.tar.gz
-tar zxvf udocker-1.3.12.tar.gz
-export PATH=`pwd`/udocker-1.3.12/udocker:$PATH
+curl -L https://github.com/indigo-dc/udocker/releases/download/1.3.13/udocker-1.3.13.tar.gz \
+  > udocker-1.3.13.tar.gz
+tar zxvf udocker-1.3.13.tar.gz
+export PATH=`pwd`/udocker-1.3.13/udocker:$PATH
 ```
 
 udocker executes containers using external tools and libraries that
@@ -353,8 +353,8 @@ The udocker tool should be installed as shown in section 2.1:
 
 ```bash
 cd /sw
-wget https://github.com/indigo-dc/udocker/releases/download/1.3.12/udocker-1.3.12.tar.gz
-tar zxvf udocker-1.3.12.tar.gz
+wget https://github.com/indigo-dc/udocker/releases/download/1.3.13/udocker-1.3.13.tar.gz
+tar zxvf udocker-1.3.13.tar.gz
 ```
 
 Directing users to the central udocker installation can be done using the

docs/reference_card.md

@@ -18,7 +18,8 @@ The configuration files allow overriding of the udocker `Config` class
 `conf` dictionary. Example of the `udocker.conf` syntax:
 
 ```ini
-dockerio_registry_url = "https://myregistry.mydomain:5000"
+[DEFAULT]
+dockerio_registry_url = https://myregistry.mydomain:5000
 http_insecure = True
 verbose_level = 5
 ```

docs/udocker.1

@@ -1,7 +1,7 @@
 .\" Manpage for udocker
 .\" Contact [email protected] to correct errors or typos.
 .\" To read this man page use:   man -l udocker.1
-.TH udocker 1 "2 Nov 2023" "version 1.3.12" "udocker man page"
+.TH udocker 1 "5 Feb 2024" "version 1.3.13" "udocker man page"
 .SH NAME
 udocker \- execute Docker containers in user space without privileges
 .SH SYNOPSIS

paper/paper.md

@@ -258,7 +258,7 @@ executables and `$HOME/.udocker/lib` for libraries.
 Currently the external tools and libraries compiled and provided with
 udocker support *x86_64*, *aarch64*, *arm* 32bit and *i386* for use with the
 **P** modes. The binaries for the remaining execution modes are currently
-only provided for *x86_64* systems, this may change in the future as thes
+only provided for *x86_64* systems, this may change in the future as these
 and other architectures become more widely used.
 
 The **F** mode is particularly unique to udocker. It relies on the interception

setup.cfg

@@ -11,3 +11,11 @@ test = pytest
 [tool:pytest]
 collect_ignore = ['setup.py']
 
+
+[codespell]
+# Ref: https://github.com/codespell-project/codespell#using-a-config-file
+skip = .git,paper.bib
+check-hidden = true
+ignore-regex = \bFinisFinish Terrae II\b
+# some commands and unfortunate variable names
+ignore-words-list = buildd,struc,regist

tests/unit/test_elfpatcher.py

@@ -34,7 +34,7 @@ def test_01__init(self, mock_path, mock_hinfo):
         mock_path.return_value = "/some_contdir"
         mock_hinfo.uid = "1000"
         elfp = ElfPatcher(self.local, self.contid)
-        self.assertTrue(mock_path.callled)
+        self.assertTrue(mock_path.called)
         self.assertEqual(elfp._uid, "1000")
 
     @patch('udocker.helper.elfpatcher.FileUtil.find_exec')

udocker/__init__.py

@@ -22,7 +22,7 @@
 """
 
 __author__ = "[email protected]"
-__copyright__ = "Copyright 2016 - 2023, LIP"
+__copyright__ = "Copyright 2016 - 2024, LIP"
 __credits__ = [
     "PRoot http://proot.me",
     "runC https://runc.io",
@@ -32,5 +32,5 @@
     "Singularity http://singularity.lbl.gov"
     ]
 __license__ = "Licensed under the Apache License, Version 2.0"
-__version__ = "1.3.12"
-__date__ = "2023"
+__version__ = "1.3.13"
+__date__ = "2024"

udocker/cli.py

@@ -104,7 +104,7 @@ def _check_imagerepo(self, imagerepo, def_imagerepo=None):
 
     def _set_repository(self, registry_url, index_url=None,
                         imagerepo=None, http_proxy=None):
-        """Select docker respository"""
+        """Select docker repository"""
         transport = "https:"
         if http_proxy:
             self.dockerioapi.set_proxy(http_proxy)
@@ -1215,7 +1215,6 @@ def do_setup(self, cmdp):
         --purge                 :clean mountpoints and files created by udocker
         --fixperm               :attempt to fix file permissions
         --nvidia                :add NVIDIA libraries and binaries
-                                 (nvidia support is EXPERIMENTAL)
 
         <mode> is one of the following execution modes:
         P1: proot accelerated mode using seccomp filtering (default)

udocker/cmdparser.py

@@ -65,7 +65,7 @@ def missing_options(self):
 
     def get(self, opt_name, opt_where="CMD_OPT", opt_multiple=False):
         """Get the value of a command line option --xyz=
-        multiple=true  multiple occurences of option can be present
+        multiple=true  multiple occurrences of option can be present
         """
         if opt_where == "CMD":
             return self._argv_split["CMD"]

udocker/config.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-"""Configurations options and treatment/overrinding"""
+"""Configurations options and treatment/overriding"""
 import os
 import sys
 from udocker.msg import Msg
@@ -29,13 +29,13 @@ class Config(object):
 
     # udocker installation tarball the release is the minimum requirement
     # the actual tarball used in the installation can have a higher version
-    conf['tarball_release'] = "1.2.10"
+    conf['tarball_release'] = "1.2.11"
     conf['tarball'] = (
         "https://download.ncg.ingrid.pt/"
-        "webdav/udocker/udocker-englib-1.2.10.tar.gz"
+        "webdav/udocker/udocker-englib-1.2.11.tar.gz"
         " "
         "https://raw.githubusercontent.com"
-        "/jorge-lip/udocker-builds/master/tarballs/udocker-englib-1.2.10.tar.gz"
+        "/jorge-lip/udocker-builds/master/tarballs/udocker-englib-1.2.11.tar.gz"
     )
     conf['installinfo'] = [
         "https://raw.githubusercontent.com/indigo-dc/udocker/master/messages", ]
@@ -46,7 +46,7 @@ class Config(object):
     conf['tmpdir'] = os.getenv("TMPDIR", "/tmp")    # for tmp files only
 
     # defaults for container execution
-    conf['cmd'] = ["bash", "-i"]  # Comand to execute
+    conf['cmd'] = ["bash", "-i"]  # Command to execute
 
     # default path for executables
     conf['root_path'] = "/usr/sbin:/sbin:/usr/bin:/bin"
@@ -80,7 +80,7 @@ class Config(object):
     # conf['proot_noseccomp'] = True
     conf['proot_noseccomp'] = None
     conf['proot_killonexit'] = True   # PRoot --kill-on-exit
-    conf['proot_link2symlink'] = True   # PRoot --link2symlink
+    conf['proot_link2symlink'] = False   # PRoot --link2symlink
 
     # fakechroot engine get ld_library_paths from ld.so.cache
     conf['ld_so_cache'] = "/etc/ld.so.cache"
@@ -92,12 +92,21 @@ class Config(object):
     # translate symbolic links into pathnames None means automatic
     conf['fakechroot_expand_symlinks'] = None
 
-    # patterns to search for libc.so for bypass in fakechroot
-    conf['libc_search'] = ("/lib64/libc.so.[0-9]", "/usr/lib64/libc.so.[0-9]",
-                           "/usr/lib/libc.so.[0-9]", "/lib/libc.so.[0-9]",
-                           "/usr/libc.so.[0-9]", "/libc.so.[0-9]", "/libc.so",)
-
-    # override the above search for libc with a specified relative pathname
+    conf['fakechroot_cmd_subst'] = \
+        "/sbin/ldconfig=#RETURN(TRUE)#:/usr/sbin/ldconfig=#RETURN(TRUE)#"
+
+    # patterns to search for libc.so in fakechroot
+    conf['libc_search'] = ("/usr/lib64/libc.so.[0-9]",
+                           "/usr/lib/x86_64-linux-gnu/libc.so.[0-9]",
+                           "/lib64/libc.so.[0-9]",
+                           "/usr/lib/libc.so.[0-9]",
+                           "/lib/libc.so.[0-9]",
+                           "/usr/libc.so.[0-9]",
+                           "/libc.so.[0-9]",
+                           "/libc.so",)
+
+    # override the above search for libc with a specific pathname
+    # relative to the container root directory (excluding host prefix)
     conf['fakechroot_libc'] = None
 
     # sharable library directories
@@ -110,7 +119,7 @@ class Config(object):
                                        "/usr/lib", )
     conf['lib_dirs_list_append'] = (".", )
 
-    # fakechroot access files, used to circunvent openmpi init issues
+    # fakechroot access files, used to circumvent openmpi init issues
     conf['access_files'] = ("/sys/class/infiniband", "/dev/open-mx",
                             "/dev/myri0", "/dev/myri1", "/dev/myri2",
                             "/dev/myri3", "/dev/myri4", "/dev/myri5",
@@ -196,7 +205,7 @@ def _conf_file_read(self, cfpath, ignore_keys=None):
         """
         Read config file
         """
-        Msg().out('Info: using config file: ', cfpath)
+        Msg().out('Info: using config file: ', cfpath, l=Msg.VER)
         cfnparser = ConfigParser()
         cfnparser.read(cfpath)
         for (key, val) in cfnparser.items('DEFAULT'):
@@ -269,6 +278,9 @@ def _env_override(self):
         Config.conf['use_patchelf_executable'] = \
             os.getenv("UDOCKER_USE_PATCHELF_EXECUTABLE",
                       Config.conf['use_patchelf_executable'])
+        Config.conf['fakechroot_cmd_subst'] = \
+            os.getenv("UDOCKER_FAKECHROOT_CMD_SUBST",
+                      Config.conf['fakechroot_cmd_subst'])
 
         Config.conf['fakechroot_expand_symlinks'] = \
             os.getenv("UDOCKER_FAKECHROOT_EXPAND_SYMLINKS",

udocker/container/localrepo.py

@@ -294,11 +294,10 @@ def setup_container(self, imagerepo, tag, container_id):
             out_imagerepo = open(container_dir + "/imagerepo.name", 'w')
         except (IOError, OSError):
             return None
-        else:
-            out_imagerepo.write(imagerepo + ":" + tag)
-            out_imagerepo.close()
-            self.cur_containerdir = container_dir
-            return container_dir
+        out_imagerepo.write(imagerepo + ":" + tag)
+        out_imagerepo.close()
+        self.cur_containerdir = container_dir
+        return container_dir
 
     def _is_tag(self, tag_dir):
         """Does this directory contain an image tag ?
@@ -486,9 +485,8 @@ def setup_tag(self, tag):
             out_tag = open(directory + "/TAG", 'w')
         except (IOError, OSError):
             return False
-        else:
-            out_tag.write(self.cur_repodir + ":" + tag)
-            out_tag.close()
+        out_tag.write(self.cur_repodir + ":" + tag)
+        out_tag.close()
         return True
 
     def set_version(self, version):