multiprocessing connection errors on NERSC

[realmarcin]

Traceback (most recent call last):
File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.10/nersc-python/lib/python3.10/multiprocessing/pool.py", line 131, in worker
put((job, i, result))
File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.10/nersc-python/lib/python3.10/multiprocessing/queues.py", line 377, in put
self._writer.send_bytes(obj)
File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.10/nersc-python/lib/python3.10/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.10/nersc-python/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.10/nersc-python/lib/python3.10/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.10/nersc-python/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.10/nersc-python/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.10/nersc-python/lib/python3.10/multiprocessing/pool.py", line 136, in worker
put((job, i, (False, wrapped)))
File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.10/nersc-python/lib/python3.10/multiprocessing/queues.py", line 377, in put
self._writer.send_bytes(obj)
File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.10/nersc-python/lib/python3.10/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.10/nersc-python/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.10/nersc-python/lib/python3.10/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)