Couldn't always determine user account information in slurm

[laurapoggio-sptools]

Version of Singularity:

What version of Singularity are you using?

$ singularity --version 

singularity version 3.8.3 

Expected behavior

I am launching singularity from a SLURM file in an HPC setting

module load singularity/3.8.3 
F=${SLURM_ARRAY_TASK_ID} 
FILE_LIST="path/to/list/of/filesToprocess.txt 

singularity exec -B /storage/:/storage/ /path/to/singularity_image \ 
    path_to_my_bash_script.sh \ 
    voi="something" \ 
    infile=`cat $FILE_LIST | sort | sed -n "${F}p"` 

Expected behaviour is that the SLURM job starts and each array starts a process opening the singularity image (accessible to all nodes). This is happening 80% of the time

Actual behaviour

Often I get the following error

Failed to get singularity version:  
WARNING: Could not lookup the current user's information: user: lookup userid numeric_UID: no such file or directory  
FATAL: Couldn't determine user account information: user: lookup userid numeric_UID: no such file or directory 

The numeric_UID is my actual numeric UID on the HPC but it is missing the step to get the UID (not numeric).

Example: my UID is user002, my numeric_UID is 1234567.

As far as I understand, the join between the two happens via active directory as the UID is the ID used for all services (including windows)

Steps to reproduce this behavior

I have not been able to fully reproduce this behaviour. it seems a combination of limited network connection to the active directory component, number of users connected and something else not yet identified

1. Is there a way to catch the behaviour of singularity in the SLURM file? As I can not reproduce it everytime, it is difficult to catch it
2. Is there a way to set the UID manually? As it would be always the same in this case, it could help

Any other suggestions would be welcomed

What OS/distro are you running

$ cat /etc/os-release 
NAME="Ubuntu" 
VERSION="20.04 LTS (Focal Fossa)" 
ID=ubuntu 
ID_LIKE=debian 
PRETTY_NAME="Ubuntu 20.04 LTS" 
VERSION_ID="20.04" 
HOME_URL="https://www.ubuntu.com/" 
SUPPORT_URL="https://help.ubuntu.com/" 
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 
VERSION_CODENAME=focal 
UBUNTU_CODENAME=focal 

How did you install Singularity

It is installed as a module on the HPC, not sure how the administrators installed it.

[github-actions]

New issues are no longer accepted in this repository. If singularity --version says singularity-ce, submit instead to https://github.com/sylabs/singularity, otherwise submit to https://github.com/apptainer/apptainer.