Merge pull request #421 from indigo-dc/devel3

Devel3

README.md

@@ -294,7 +294,6 @@ the nvidia drivers are installed in the host system.
 udocker setup  --nvidia  mytensorflow
 ```
 
-
 ## Security
 
 By default udocker via PRoot offers the emulation of the root user. This
@@ -354,7 +353,7 @@ containers execution in user space. udocker is particularly suited to
 run user applications encapsulated in docker containers.
 
 Debugging or using strace with the PRoot engine will not work as both
-the debuggers and PRoot use the same tracing mechanism.  
+the debuggers and PRoot use the same tracing mechanism.
 
 ## Execution mode specific limitations
 

codemeta.json

@@ -6,7 +6,7 @@
     "@type": "SoftwareSourceCode",
     "identifier": "udocker",
     "name": "udocker",
-    "version": "1.3.13",
+    "version": "1.3.14",
     "description": "A basic user tool to execute simple docker containers in batch or interactive systems without root privileges",
     "license": "Apache Software License 2.0, OSI Approved :: Apache Software License",
     "author": [

conftest.py

@@ -0,0 +1,3 @@
+"""
+This file is required for pytest to find and load udocker as module.
+"""

docs/installation_manual.md

@@ -32,18 +32,18 @@ udocker requires:
 Download a release tarball from <https://github.com/indigo-dc/udocker/releases>:
 
 ```bash
-wget https://github.com/indigo-dc/udocker/releases/download/1.3.13/udocker-1.3.13.tar.gz
-tar zxvf udocker-1.3.13.tar.gz
-export PATH=`pwd`/udocker-1.3.13/udocker:$PATH
+wget https://github.com/indigo-dc/udocker/releases/download/1.3.14/udocker-1.3.14.tar.gz
+tar zxvf udocker-1.3.14.tar.gz
+export PATH=`pwd`/udocker-1.3.14/udocker:$PATH
 ```
 
 Alternatively use `curl` instead of `wget` as follows:
 
 ```bash
-curl -L https://github.com/indigo-dc/udocker/releases/download/1.3.13/udocker-1.3.13.tar.gz \
-  > udocker-1.3.13.tar.gz
-tar zxvf udocker-1.3.13.tar.gz
-export PATH=`pwd`/udocker-1.3.13/udocker:$PATH
+curl -L https://github.com/indigo-dc/udocker/releases/download/1.3.14/udocker-1.3.14.tar.gz \
+  > udocker-1.3.14.tar.gz
+tar zxvf udocker-1.3.14.tar.gz
+export PATH=`pwd`/udocker-1.3.14/udocker:$PATH
 ```
 
 udocker executes containers using external tools and libraries that
@@ -169,7 +169,7 @@ The configuration of udocker has the following hierarchy:
    it will override 1.
 3. If environment variables are set ([section 5. Environment variables](#5-environment-variables)),
    they will override 2.
-4. The presence of general udocker command line options, will override 3. .
+4. The presence of general udocker command line options, will override 3.
 
 ### 3.1. Directories
 
@@ -257,19 +257,32 @@ the environment variables described below together with the default behavior.
 A value of `UDOCKER` will force the usage of the executables provided by the
 udocker installation.
 
-A full pathname can be used to select a specific executable (or library) from the
-host or from the udocker installation.
+A full pathname can be used to force selection of a specific executable (or library)
+from the host or from the udocker installation.
 
 * `UDOCKER_USE_PROOT_EXECUTABLE`: path to proot, default is proot from udocker.
 * `UDOCKER_USE_RUNC_EXECUTABLE`: path to runc, default is search the host and
   if not found use runc from udocker.
 * `UDOCKER_USE_SINGULARITY_EXECUTABLE`: path to singularity, default is search
   the host.
 * `UDOCKER_FAKECHROOT_SO`: path to a fakechroot library, default is search
-  in udocker.
+  in udocker under `$HOME/.udocker/lib`.
 * `UDOCKER_DEFAULT_EXECUTION_MODE`: default execution mode can be P1, P2, F1,
   S1, R1, R2 or R3.
 
+Several executables and libraries are shipped with udocker. For instance
+the executable for the Rn modes can be selected to be either `runc` or
+`crun`. This can be accomplished by setting `UDOCKER_USE_RUNC_EXECUTABLE`
+to the path of the desired executable. If `runsc` is available in the
+host it can also be selected in this manner.
+
+```
+# Forcing the use of crun instead of runc
+export UDOCKER_USE_RUNC_EXECUTABLE=$HOME/.udocker/bin/crun-x86_64
+export UDOCKER_DEFAULT_EXECUTION_MODE=R1
+udocker run <mycontainerid>
+```
+
 ## 6. External tools and libraries
 
 ### 6.1. Source code repositories
@@ -323,10 +336,33 @@ versions. The tools are also delivered for several architectures.
 | Mode  | Supported architecture                       |
 |-------|:---------------------------------------------|
 | **P** | x86_64, i386, aarch64 and arm                |
-| **F** | x86_64                                       |
-| **R** | x86_64                                       |
+| **F** | x86_64, aarch64, ppc64le                     |
+| **R** | x86_64  aarch64                              |
 | **S** | uses the binaries present in the host system |
 
+### 6.3. Compiling
+
+udocker already provides executables and libraries for the engines. These
+are statically compiled to be used across different Linux distributions.
+In some cases these executables may not work and may require recompilation.
+Use the repositories in section 6.2 if you which to compile the executables
+or libraries. Notice that the git repositories that are specific of udocker
+have branches or tags like `UDOCKER-x` where `x` is a number. Use the branch
+or tag with the highest number.
+
+A notable case are the fakechroot libraries used in the Fn modes that need
+to match the libc in the container. This means that a libfakechroot.so must
+be produced for each different distribution release and intended architecture.
+Two implementations of the `libc` are supported `glibc` and `musl`, choose
+the one that matches the distribution inside the container. Once compiled the
+selection of the library can be forced by setting the environment variable
+`UDOCKER_FAKECHROOT_SO`.
+
+```
+udocker setup --execmode=F3 <mycontainerid>
+UDOCKER_FAKECHROOT_SO=$HOME/mylibfakechroot.so  udocker run <mycontainerid>
+```
+
 The latest binary tarball can be produced from the source code using:
 
 ```bash
@@ -353,8 +389,8 @@ The udocker tool should be installed as shown in section 2.1:
 
 ```bash
 cd /sw
-wget https://github.com/indigo-dc/udocker/releases/download/1.3.13/udocker-1.3.13.tar.gz
-tar zxvf udocker-1.3.13.tar.gz
+wget https://github.com/indigo-dc/udocker/releases/download/1.3.14/udocker-1.3.14.tar.gz
+tar zxvf udocker-1.3.14.tar.gz
 ```
 
 Directing users to the central udocker installation can be done using the

docs/udocker.1

@@ -1,7 +1,7 @@
 .\" Manpage for udocker
 .\" Contact [email protected] to correct errors or typos.
 .\" To read this man page use:   man -l udocker.1
-.TH udocker 1 "5 Feb 2024" "version 1.3.13" "udocker man page"
+.TH udocker 1 "4 Apr 2024" "version 1.3.14" "udocker man page"
 .SH NAME
 udocker \- execute Docker containers in user space without privileges
 .SH SYNOPSIS
@@ -112,7 +112,7 @@ Obtain and print information about an IMAGE manifest from a remote registry.
 .BR mkrepo " " DIRECTORY
 Setup a local repository in the host DIRECTORY. The required directory structure is created.
 .TP
-.BR login " " \--username=USERNAME " " | " " \--password=PASSWORD " " | " " \--registry=REGISTRY
+.BR login " " \--username=USERNAME " " | " " \--password=PASSWORD " " |  " " \--password-stdin " " | " " \--registry=REGISTRY
 Setup of authentication information for access to remote Docker registries. Enables "pull" of IMAGES from private registries. The option --registry can be used to access registries other than the default dockerhub. If USERNAME or PASSWORD are not provided in the command line, the user will be prompted to provide them.
 .TP
 .BR logout " " [ " " \-a " " ] " " | " " \--registry=REGISTRY
@@ -196,7 +196,10 @@ Override the container metadata entrypoint.
 Specify the operating system and/or architecture of the image to be pulled and executed.
 .TP
 --pull=WHEN
-Specify when to pull the image. The argument WHEN can take the values of "missing", "never" or "always".
+Specify when to pull the image. The argument WHEN can take the values of "missing", "never", "always" or "reuse".
+.TP
+--httpproxy=PROXY
+Specify an http, socks4 or socks5 proxy, see the "pull" command for the syntax.
 .RE
 
 .SH ENVIRONMENT

docs/user_manual.md

@@ -275,6 +275,7 @@ udocker pull --httpproxy=socks5h://host:port busybox
 udocker pull --httpproxy=socks4a://user:pass@host:port busybox
 udocker pull --httpproxy=socks5h://user:pass@host:port busybox
 udocker pull --platform=linux/arm64 fedora:latest
+udocker pull --platform=linux/ppc64le centos:7
 ```
 
 ### 3.6. images
@@ -638,7 +639,8 @@ Options:
 * `--kernel=KERNELID` use a specific kernel id to emulate useful when the host kernel is too old
 * `--location=DIR` execute a container in a given directory
 * `--platform=os/architecture` specify a different platform to be pulled
-* `--pull=missing|never|always` specify when to pull the image
+* `--pull=missing|never|always|reuse` specify when to pull the image
+* `--httpproxy=PROXY` uses an http or socks proxy, see `pull`
 
 Options valid only in Pn execution modes:
 
@@ -661,11 +663,20 @@ udocker create --name=myfed  fedora:29
 # execute a cat inside of the container
 udocker run  myfed  cat /etc/redhat-release
 
-# The above three operations could have done with a single command
-# However each time udocker is invoked this way a new container
-# directory tree is created consuming additional space and time
+# The above three operations can be done with a single command
+# However each time udocker is invoked in this way a new container
+# directory tree is created. This will consume additional space
+# and may considerably increase the time for the container to start.
 udocker run fedora:29 cat /etc/redhat-release
 
+# For repeated invocations of the same container image the issue
+# described above can be prevented by using --pull=reuse with --name.
+# With the option --pull=reuse udocker will first try to execute
+# a container with the same name specified by --name and only if
+# it doesn't exist will it pull and create. In this way repeated
+# calls to run only create a single container that is then reused.
+udocker run --name=F29 --pull=reuse fedora:29 cat /etc/redhat-release
+
 # In this example the host /tmp is mapped to the container /tmp
 udocker run --volume=/tmp  myfed  /bin/bash
 
@@ -736,7 +747,7 @@ UDOCKER_LOGLEVEL=2 udocker run busybox:latest /bin/ls
 ### 3.23. login
 
 ```bash
-udocker login [--username=USERNAME] [--password=PASSWORD] [--registry=REGISTRY]
+udocker login [--username=USERNAME] [--password=PASSWORD | --password-stdin ] [--registry=REGISTRY]
 ```
 
 Login into a Docker registry using v2 API. Only basic authentication
@@ -748,15 +759,22 @@ Options:
 
 * `--username=USERNAME` provide the username in the command line
 * `--password=PASSWORD` provide the password in the command line
+* `--password-stdin`    provide the password via stdin
 * `--registry=REGISTRY` credentials are for this registry
 
 Examples:
 
 ```bash
+# To use dockerhub private repositories
 udocker login --username=xxxx --password=yyyy
-udocker login --registry="https://hostname:5000"
+
+# To use a different container registry (the https:// is optional)
+udocker login --registry=https://hostname
 username: xxxx
 password: ****
+
+# To use a private repository at AWS ECR
+aws ecr get-login-password --region eu-north-1 | udocker login --username=AWS --password-stdin --registry=000000000000.dkr.ecr.eu-north-1.amazonaws.com
 ```
 
 ### 3.24. logout
@@ -929,7 +947,7 @@ Newer versions of Singularity may run without requiring privileges but
 need a recent kernel in the host system with support for rootless user
 mode namespaces similar to runc in mode R1.
 Singularity cannot be compiled statically due to dependencies on
-dynamic libraries and therefore is not provided with udocker.
+dynamic libraries and therefore is not shipped with udocker.
 In CentOS 6 and CentOS 7 Singularity must be installed with privileges
 by a system administrator as it requires suid or capabilities.
 The S1 mode also offers root emulation to facilitate software installation
@@ -966,7 +984,7 @@ changed through the configuration files by changing the attribute
 **UDOCKER_DEFAULT_EXECUTION_MODE**. Only the following modes can be used as
 default modes:
 **P1**, **P2**, **F1**, **S1**, and **R1**. Changing the default execution
-mode can be useful in case the default does not work as expected.
+mode can be useful if the default does not work as expected.
 
 Example:
 
@@ -1000,11 +1018,20 @@ udocker manifest inspect REPO/IMAGE:TAG
 ```
 
 Obtain and print information about an IMAGE manifest from a remote registry.
+Can be used to obtain the platform architectures supported by the IMAGE.
+
+Options:
+
+* `--index=url` specify an index other than index.docker.io
+* `--registry=url` specify a registry other than registry-1.docker.io
+* `--httpproxy=proxy` specify a socks proxy for downloading, see `pull`
+* `--platform=os/architecture` specify a platform to be inspected
 
 Example:
 
 ```bash
 udocker manifest inspect centos:centos7
+udocker manifest --platform=linux/ppc64le inspect centos:7
 ```
 
 ## 4. Running MPI jobs
@@ -1252,7 +1279,7 @@ as root. In other modes execution as root is achieved by invoking
 run with the `--user=root` option:
 
 ```bash
-udocker run --user=root <container-id>`
+udocker run --user=root <container-id>
 ```
 
 ### 7.1. Running as root in Pn modes
@@ -1381,18 +1408,48 @@ considerations may hold:
   mode may exhibit a large performance penalty. This also
   applies to P1 in older kernels without **SECCOMP filtering**
 * Fn modes are generally faster than Pn modes and do not have
-  multi threading or I/O limitations.
+  the multi threading or I/O limitations.
 * Singularity and runc should provide similar performances.
 * Depending on application the Fn modes are often faster than
   all other modes.
 
 ## 10. Hardware architectures
 
-The udocker Python code was the built-in logic to support several hardware
+The udocker Python code has the built-in logic to support several hardware
 architectures namely i386, x86_64, arm (32 bit) and aarch64 (arm 64 bit).
 However the required engine binaries and/or libraries must also be provided
-for each of the architectures. Currently only the Pn modes are provided with
-compiled executables to support execution on x86, x86_64, ARM and ARM64.
+for each of the architectures. Currently only some modes have compiled
+binaries to support execution on x86, x86_64, ARM, ARM64 and
+ppc64le. The executables and libraries for the execution engines shipped
+with udocker have a suffix that identifies the architecture, check the
+relevant udocker installation directories usually `$HOME/.udocker/bin`
+and `$HOME/.udocker/lib`.
+
+Users may compile the same executables shipped in the udockertools in
+their linux hosts to support different or newer distributions, and/or
+architectures. See the [installation manual](installation_manual.md)
+for further information.
+
+Checking which architectures are supported by a given container can
+be verified using `udocker manifest inspect IMAGE`. If the intended architecture
+is available it can be pulled using `udocker pull --platform=OS/ARCH`.
+
+```bash
+udocker manifest inspect centos:7
+udocker pull --platform=linux/arm64 centos:7
+udocker create --name=C7 centos:7
+udocker run C7
+```
+
+In general, if the binaries in the container have been compiled for
+an architecture that is different from the host then the execution
+will not be possible. However, execution may still be possible provided
+that `qemu-user` is locally installed. In many distributions `qemu-user`
+is provided by the package `qemu-user-static`. In such case the default
+engine of udocker Pn will automatically use the qemu emulation to support
+the execution. Since the architecture is emulated the execution will be
+much slower. Emulation for the Fn modes may also work if the `qemu-user`
+binaries are both installed and also appear in `/proc/sys/fs/binfmt_misc/`.
 
 ## 11. Host environment specific notes
 
@@ -1412,9 +1469,11 @@ udocker run arm64v8/fedora:35
 udocker can run on Google Colab using the **P** or **F** modes.
 
 ```bash
-!PATH=`pwd`/udocker:$PATH udocker --allow-root pull centos:centos7
-!PATH=`pwd`/udocker:$PATH udocker --allow-root create --name=c7 centos:centos7
-!PATH=`pwd`/udocker:$PATH udocker --allow-root run c7
+! pip install udocker
+! udocker install
+! udocker --allow-root pull centos:centos7
+! udocker --allow-root create --name=c7 centos:centos7
+! udocker --allow-root run c7
 ```
 
 ### 11.3. Docker
@@ -1431,15 +1490,13 @@ udocker --allow-root run ub18
 
 ## 12. Issues
 
-To avoid corruption the execution of data backups and container copies should
-only be performed when the container is not being executed (not locally nor
-in any other host if the filesystem is shared).
-
 Containers should only be copied for transfer when they are in the execution
 modes Pn or Rn. The modes Fn perform changes to the containers that will make
 them fail if they are execute in a different host where the absolute pathname
 to the container location is different. In this later case convert back to P1
-(using:  `udocker setup --execmode=P1`) before performing the backup.
+(using:  `udocker setup --execmode=P1`) before performing the backup. Sharing
+of containers can be done across hosts in an homogeneous cluster or between
+hosts with the very same directory structure.
 
 When experiencing issues in the default execution mode (P1) you may try
 to setup the container to execute using mode P2 or one of the Fn or
@@ -1448,7 +1505,8 @@ Rn modes. See section 3.27 for information on changing execution modes.
 Some execution modes require the creation of auxiliary files, directories
 and mount points. These can be purged from a given container using
 `setup --purge`, however this operation must be performed when the
-container is not being executed.
+container is not being executed (nor locally nor in another host of the
+cluster).
 
 ## Acknowledgments