Use of parallel meep on distributed computing nodes

[kalosu]

Hello again friends from the MEEP community,

I have been using parallel meep for a couple of months on my local computer. Now, I have access to more computational resources but these resources are clustered in different nodes.

I have set some basic configuration for sharing the same simulation file between my two nodes and I have the same mpi based meep implementation installed in both of my nodes. (I have also configured the ssh connection between both nodes in such a way that I can connect between both machines in a password-less manner)

The issue is that I have installed the parallel meep python implementation through the conda package manager. As you know, in order to use then this implementation it is needed to activate the conda environment containing the meep parallel python implementation in connection with mpich (with access to mpirun).

Is it possible to run a parallel simulation in this manner using my two nodes? How do I guarantee that the proper conda environment will be started at my "remote"node once I call mpirun from my "localhost" computer?

I have tried running some test under this configuration by using:

mpirun -np 32 -hosts worker,localhost python3.11 dip_flat_GaAs_pos_sweep_ext_eff_NA.py

But I get the following error:
[proxy:0:0@comp-power-2] HYDU_sock_connect (utils/sock/sock.c:141): unable to connect from "comp-power-2" to "localhost" (Connection refused)

comp-power-2 is my "remote"(worker) node and localhost is the computer from where I am running the mpirun command.

If you have any reference on how to properly implement such a simulation in this case (using the conda version of parallel meep), I would appreciate your help and feedback.

[stevengj]

Often on a large parallel machine they have a queuing system where you provide a shell script that is run on every node to launch the job, in which case the shell script can activate the environment you want.

Alternatively, you might be able to set up your shell .profile so that it automatically activates the conda environment when logging into a node.

You should ask your sysadmin how to do this on your cluster system.

[kalosu]

Thanks a lot. I have managed to execute the parallel simulation at the end.
I was wondering if there is any special network configuration that might be needed in order to optimize the simulation runtime.

At the moment I have 64 cores available, distributed in 4 nodes.

I have tried running a simulation on these 64 nodes by using the mpi interface but I have noticed that in general the simulation is significantly slower than in the case of running the simulation by using only the vCPUs that I have at one single node, i.e, 16 CPUs.

I can imagine that there is some communication bottleneck there that does not allow me to "gain" something from utilizing all of these 64 cores.

Is there any feasible way to optimize this?

[smartalecH]

So keep in mind that FDTD is a memory-bound algorithm. So network latency, as you suggest, will certainly kill the performance. There are a lot of factors that determine to what extent this will actually happen.

For example, if your nodes are connected via infiniband controllers, the latency is on par with a standard single host model. If your nodes are separated across the country and tied together over Ethernet.... you're going to have some major issues. There's a continuous interpolation of scenarios between these two extremes, one of which your system probably relies on.

But assuming you're using a cluster with a decent hardware networking infrastructure, the mpi configuration will make a difference too. Make sure you use a version of MPI that your system maintainers install, as they usually configure in a variety of features that maximize the available hardware. Furthermore, there are varios mpirun flags you can play with that will "pin" processes to different cores etc. This is important, as some native configurations will pin all the procs to a single core. So make sure you are using your resources as expected. I once spent 80 hours configuring a system mpi build to properly use the infrastructure... it requires some effort.

Also keep in mind that even if your system was completely connected by the same bus, you quickly hit a point where more procs actually degrades the performance of the simulation.

So at the end of the day, there are plenty of knobs you can turn to maximize the performance. So many in fact, that you'll have to talk to your system maintainer, as we won't be able to help you much. Although there are quite a few tips here: https://meep.readthedocs.io/en/latest/Parallel_Meep/

[kalosu]

Hi @smartalecH ,

Thanks a lot for your feedback on this. I have checked with the administrators for the cluster that I wanted to use and apparently the nodes they provide are not connected via infiniband controllers.
Nevertheless, I got access to another computing cluster at where they do have such a high speed interface between nodes.

For this other HPC system, I have checked and they already have an MPI implementation that is used for running applications from all different users.

In this case, given that MPI is already there and supported, would it be better if I compile meep on this cluster? Or could I also use the conda version of parallel MEEP but by using the MPI implementation maintained by the system and not the one installed via conda while installing parallel MEEP?

Thanks for your comments and suggestions!

[smartalecH]

In this case, given that MPI is already there and supported, would it be better if I compile meep on this cluster?

Yes, to properly use their build of MPI you have to build meep from source using that particular MPI. in other words, you cannot use the conda binaries and simply try to "launch" your script with the local build of MPI.

[MahmutRuzi]

@kalosu I had some experience working with meep on HPC. I attached a slurm job schedule script that might be helpful. I Installed meep using conda on an environment called parallel-meep. The input file (Au_nanobar_v1.py) has meep & python related codes. The simpetus project page is very helpful. There are several examples and I closely followed the one on thermal radiation. http://www.simpetus.com/projects.html#meep_thermal_radiation.

#!/bin/bash
#SBATCH -A user
#SBATCH -J Aunano
#SBATCH --nodes=1
#SBATCH -o some1.out
#SBATCH --partition=debug
#SBATCH --ntasks=20
#SBATCH --mem=80000
#SBATCH --error=slurm-%j.err
#SBATCH --time=0-00:15:00
#SBATCH --mail-type=All
#SBATCH --mail-user=[email protected]

#Setup environment

module purge

eval "$ (/truba/home/user/miniconda3/bin/conda shell.bash hook)"

conda activate parallel-meep

input=Au_nanobar_v1.py

mpirun -n $ SLURM_NTASKS python -u $ {input} > data1.out

exit