supercomputers deployment with unidist MPI backend

[luweizheng]

hi all!

I read modin's doc and do not find any example on how to start unidist backend on supercomputers.
Supercomputers have job schedulers like SLURM or PBS. I can start modin on ray following ray's SLURM deployment document. What about unidist? I see there is MPI backend in unidist. And supercomputers usually have InfiniBand and MPI support.

Now I am using Intel MPI which can support spawning MPI processes without using mpiexec command.

What I do:

### SLURM script
export MODIN_ENGINE=unidist
export UNIDIST_BACKEND=mpi

# export MPI hosts here 
# as docs say that we need to set hosts: https://unidist.readthedocs.io/en/latest/using_unidist/unidist_on_mpi.html#id1
export UNIDIST_MPI_HOSTS=$joined


python modin_query.py

Now I get the following error message:

[mpiexec@cpu64c-5] check_exit_codes (../../../../../src/pm/i_hydra/libhydra/demux/hydra_demux_poll.c:117): unable to run bstrap_proxy on cpu64c-5 (pid 28288, exit code 256)
[mpiexec@cpu64c-5] poll_for_event (../../../../../src/pm/i_hydra/libhydra/demux/hydra_demux_poll.c:159): check exit codes error
[mpiexec@cpu64c-5] HYD_dmx_poll_wait_for_proxy_event (../../../../../src/pm/i_hydra/libhydra/demux/hydra_demux_poll.c:212): poll for event error
[mpiexec@cpu64c-5] HYD_bstrap_setup (../../../../../src/pm/i_hydra/libhydra/bstrap/src/intel/i_hydra_bstrap.c:1065): error waiting for event
[mpiexec@cpu64c-5] HYD_print_bstrap_setup_error_message (../../../../../src/pm/i_hydra/mpiexec/intel/i_mpiexec.c:1027): error setting up the bootstrap proxies
[mpiexec@cpu64c-5] Possible reasons:
[mpiexec@cpu64c-5] 1. Host is unavailable. Please check that all hosts are available.
[mpiexec@cpu64c-5] 2. Cannot launch hydra_bstrap_proxy or it crashed on one of the hosts. Make sure hydra_bstrap_proxy is available on all hosts and it has right permissions.
[mpiexec@cpu64c-5] 3. Firewall refused connection. Check that enough ports are allowed in the firewall and specify them with the I_MPI_PORT_RANGE variable.
[mpiexec@cpu64c-5] 4. slurm bootstrap cannot launch processes on remote host. You may try using -bootstrap option to select alternative launcher.

It will be helpful if there are official unidist docs so I can know if this is the problem with my configuration or if the way I use unidist is wrong.

[mvashishtha]

@luweizheng thank you for creating this issue. Unfortunately I know very little about MPI and unidist.

Is the documentation here helpful at all? The error messages seem to say that unidist was unable to set up the remote workers. Would it help to customize the worker hosts here? Have you checked whether your controller node is able to communicate with the workers?

cc'ing @YarShev @modin-project/unidist-core for help. I think they can be more helpful.

[luweizheng]

Thanks for your reply. Both of the settings from unidist docs are already configured in my case.
Hope developers from unidist could help.

[YarShev]

Hi @luweizheng, thank you for this post and sorry for the delayed response. I am currently on a trip and did not have a chance to reply here earlier. So far, we do not have any page in unidist docs yet on how to launch unidist on MPI using SLURM, but what I see looking at your steps with unidist configuration settings seems correct to me. In addition, I recommend you to set UNIDIST_CPUS environment variable in order to spawn the needed number of MPI workers in your cluster. UNIDIST_CPUS should be equal to the number of processes + 1 you want to allocate with SLURM because one of MPI workers is going to be used for some monitoring tasks in unidist. That said, I think the issue you are experiencing comes down to the configuration settings of a concrete MPI implementation, namely, Intel MPI in your case. Could you take a look into this post, pick the suggestions/solutions up from it and try those out? Please let us know if that helps.

[luweizheng]

So currently modin can only be hosted on ray? I must connect a ray cluster and submit my job to this ray cluster?

[YarShev]

To run Modin with unidist on MPI you don't need Ray at all. We haven't tested Modin with unidist on MPI thoroughly in a cluster environment yet, but, theoretically, it should work. In the example posted here you use unidist on MPI with the dynamic spawning of workers. Though it works well in a single node, I am not sure the dynamic spawning would work with a SLURM environment. I have to ask developers of the Intel MPI implementation on this matter. Meanwhile, I think the most appropriate way to run Modin with unidist on MPI using SLURM is to run the MPI backend in SPMD mode as shown here. You should initialize unidist at the beginning of your script so after that you would get Controller/Worker mode (main process you interact with and worker processes performing tasks).

### SLURM script
export MODIN_ENGINE=unidist
export UNIDIST_BACKEND=mpi
export UNIDIST_IS_MPI_SPAWN_WORKERS=False
mpiexec -host host1:n1,...,hostM:nM python script.py

### Python script
# initialize unidist at the beginning
import unidist
unidist.init()

For more details about Intel MPI usage with SLURM please look into here.

[no-ponomarev]

@luweizheng could you, please, provide more details about you use-case. You are starting one process application and then spawn the processes with Intel MPI? What slurm features are you using exactly?

[luweizheng]

Now I can run on unidist MPI backend successfully!

My SLURM script:

export MODIN_ENGINE=unidist
export UNIDIST_BACKEND=mpi

python xxx.py

remove all ray code and add import unidist;unidist.init()l into xxx.py:

import unidist
unidist.init()

I use Intel MPI and Intel MPI can automatically detect hosts by reading the $SLURM_JOB_NODELIST from SLURM.

Maybe we should update the docs? I can help to contribute. Where should I put these things into the docs?

[YarShev]

Happy to hear that you were able to run unidist on MPI. It would be great if you could help us update the docs on this matter. Could you please update this page (link) by adding an additional section named like Deploying on Slurm.

Meanwhile, I would like to ask you a few questions to make sure you got the expected behavior.

1. Did you set UNIDIST_MPI_HOSTS? If not, are you sure that unidist on MPI spawned workers on all nodes listed in SLURM_JOB_NODELIST?
2. Did you set UNIDIST_CPUS to a specific value? If not, did you have the expected number of worker processes on all nodes listed in SLURM_JOB_NODELIST?
3. Which SLURM commands did you use to allocate the necessary amount of workers per node? Did you see that each node was able to run the requested amount of workers?

[luweizheng]

1. Did you set UNIDIST_MPI_HOSTS? If not, are you sure that unidist on MPI spawned workers on all nodes listed in SLURM_JOB_NODELIST?

No. If I set UNIDIST_MPI_HOSTS=cpu24c-15,cpu24c-16,cpu24c-17,cpu24c-18, where cpu24c-15 is the hostname on my site, I will get the following error:

[proxy:1:2@cpu24c-17] main (../../../../../src/pm/i_hydra/proxy/proxy.c:938): assert (proxy_params.immediate.process.num_children * sizeof(int)) failed
[proxy:1:3@cpu24c-18] main (../../../../../src/pm/i_hydra/proxy/proxy.c:938): assert (proxy_params.immediate.process.num_children * sizeof(int)) failed
[proxy:1:1@cpu24c-16] main (../../../../../src/pm/i_hydra/proxy/proxy.c:938): assert (proxy_params.immediate.process.num_children * sizeof(int)) failed
srun: error: cpu24c-17: task 2: Exited with exit code 5
srun: error: cpu24c-16: task 1: Exited with exit code 5
srun: error: cpu24c-18: task 3: Exited with exit code 5

2. Did you set UNIDIST_CPUS to a specific value? If not, did you have the expected number of worker processes on all nodes listed in SLURM_JOB_NODELIST?

UNIDIST_CPUS does not have any effect on my environment. This workload takes about 40 to 50 seconds. I am not sure if UNIDIST_CPUS can affect performance.

3. Which SLURM commands did you use to allocate the necessary amount of workers per node? Did you see that each node was able to run the requested amount of workers?

Here is my SLURM options to allocate resources.

#SBATCH --nodes=4
#SBATCH --cpus-per-task=24
#SBATCH --ntasks-per-node=1

Each node will start multiple workers, I use top and htop to watch the CPU workloads.

Case 1:

#SBATCH --cpus-per-task=24
#SBATCH --ntasks-per-node=1

It seems that the workload cannot run on all CPU cores even I set UNIDIST_CPUS to a very large value like 48. In my case, 6 python processes on a 24 cores machine. Some cores' utilization is 0.0%. Some cores' utilization is 100%.

Case 2:

#SBATCH --cpus-per-task=4
#SBATCH --ntasks-per-node=6

More cores are used.

Case 3:

#SBATCH --cpus-per-task=1
#SBATCH --ntasks-per-node=24

Every core's utilization is 100%. But the overall speed is very slow.

[YarShev]

No. If I set UNIDIST_MPI_HOSTS=cpu24c-15,cpu24c-16,cpu24c-17,cpu24c-18, where cpu24c-15 is the hostname on my site, I will get the following error:

If you don't set UNIDIST_MPI_HOSTS, I am not sure that MPI processes are spawned across all nodes you listed in SLURM_JOB_NODELIST. Do you really see MPI processes spawned on all nodes without setting UNIDIST_MPI_HOSTS?

UNIDIST_CPUS does not have any effect on my environment. This workload takes about 40 to 50 seconds. I am not sure if UNIDIST_CPUS can affect performance.

UNIDIST_CPUS should affect performance because it specifies how many MPI workers should be spawned.

Here is my SLURM options to allocate resources.

Regarding your 3 cases, the third one is the best. NTasks per node can vary but cpus per task should be equal to 1. This is how Modin on Ray/Dask works.

Anyway, I really encourage you to try to run unidist on MPI in SPMD mode as I said above because there is a direct mapping between SLURM commands and mpiexec. When you run a program with mpiexec, it respects/copies settings from the SLURM batch job.

Look forward to your response.

[luweizheng]

If you don't set UNIDIST_MPI_HOSTS, I am not sure that MPI processes are spawned across all nodes you listed in SLURM_JOB_NODELIST. Do you really see MPI processes spawned on all nodes without setting UNIDIST_MPI_HOSTS?

Here are two screen snapshots on 2 nodes with slurm setting 1. MPI processes spawn on multiple nodes.