Replies: 1 comment 1 reply
-
Yes, this is how MPI works -- each process gets its own allocation, regardless of where it is "located".
I've never heard of an "MPI shared memory interface". That seems to defeat the point of MPI (large-scale distributed computing). However, meep does support shared-memory parallelism using openMP (which is also compatible with multiple processes launched using MPI). In this case, you can launch a single process on your machine (thereby creating a single array) but set the Note that the current thread-level parallelism does not scale as well as the process-level parallelism, as described here.
is there an actual memory leak? Or are you just running out of memory? Can you trace the leak with |
Beta Was this translation helpful? Give feedback.
-
|
Hi smartalecH, Thanks for the quick reply. I agree with you about the fact that with MPI and mpi4py
That does not, however, necessarily imply that each process must load the data into a different memory slots. Each process in the present implementation makes a Instead of repeatedly copying the array by each process, it could be copied into shared memory once, and each process could access it using a pointer to the shared memory. This would allow each process to access its chunk of data to process. def allocate_shared_memory(complete_shape_np, dtype=np.float32):
nelements = np.prod(np.array(complete_shape_np)) if comm_rank == 0 else 0
if MPI:
itemsize = np.dtype(dtype).itemsize
print_mpi(f"CREATING SHARED MEMMORY of {nelements} elements in {nelements*itemsize} bits")
win = MPI.Win.Allocate_shared(nelements*itemsize, itemsize, comm=MPI.COMM_WORLD)
print_mpi(f"SHARED MEMMORY CREATED of {nelements*itemsize} elements")
# create a numpy array whose data points to the shared mem
buf, itemsize = win.Shared_query(0)
print(buf, itemsize)
assert itemsize == np.dtype(dtype).itemsize
data_np = np.ndarray(buffer=buf, dtype=dtype, shape=complete_shape_np)
else:
data_np = np.array(dtype=dtype, shape=complete_shape_np)
print_mpi("NUMPY ARRAY OF SHAPE", data_np.shape)
return data_npI will have a look at the suggested shared-memory parallelism using openMP. |
Beta Was this translation helpful? Give feedback.
-
Dear authors,
I'm attempting to use several MPI processes with meep to simulate a huge volume. After checking the code, it seems that when passing a NumPy array to meep, the array is copied multiple times for every processes even if the core is inside the same machine. code
This error looks similar to the issue #1082, where a segmentation fault occurred in susceptibility object for large parallel simulations.
A possible workaround for this memory leak is instead of coping the data at each process in a different memory address, allocate the material information in the shared memory only once using C++ and MPI shared memory interface.
I attempted to load the NumPy array in the shared memory before passing it to meep. However, the data is copied multiple times.
An
example.pyfile is :When using this script with
mpiexcec -n 16 python example.pythe following memory consumption is presented in the cluster:After some time the process ends up with a segmentation fault due to insufficient memory:
Beta Was this translation helpful? Give feedback.
All reactions