Farfield scattering of nanoparticle in 3D

[rolly-ng]

Hi, I would like to calculate the farfield scattering pattern of a dielectric nano particle in 3D. I am using the Scheme interface since I submit my job to a cluster. It seems to work until it starts calculating the far field data.

Here is my .out file,
...
on time step 49967 (time=49.967), 0.615 s/step
on time step 49974 (time=49.974), 0.617154 s/step
on time step 49981 (time=49.981), 0.613482 s/step
on time step 49988 (time=49.988), 0.618693 s/step
on time step 49995 (time=49.995), 0.615587 s/step
on time step 50002 (time=50.002), 0.617393 s/step
field decay(t = 50.005): 8.921616453034594e-11 / 0.12570445246165293 = 7.097295504116052e-10
run 0 finished at t = 50.005 (50005 timesteps)
get_farfields_array working on point 77061 of 250000 (30% done), 5.19099e-05 s/point
get_farfields_array working on point 154332 of 250000 (61% done), 5.17659e-05 s/point
get_farfields_array working on point 232220 of 250000 (92% done), 5.13558e-05 s/point

Here is my slurm.out,
srun: error: node01: tasks 2,55,61: Killed
srun: error: node02: tasks 128,149,175: Killed
[node01:33258:0:33258] ib_mlx5_log.c:113 FATAL: Error on QP 0x41bd wqe[30895]: Transport retry count exceeded (synd 0x15 vend 0x81) opcode SEND
==== backtrace ====
0 0x0000000000029469 uct_ib_mlx5_completion_with_err() ???:0
1 0x000000000004436a uct_dc_mlx5_ep_flush() ???:0
2 0x00000000000181aa ucp_worker_progress() ???:0
3 0x00000000000031e7 mca_pml_ucx_progress() ???:0
4 0x000000000003188c opal_progress() ???:0
5 0x0000000000038085 ompi_sync_wait_mt() ???:0
6 0x000000000004ad17 ompi_request_default_wait() ???:0
7 0x00000000000a1fba ompi_coll_base_allreduce_intra_ring_segmented() ???:0
8 0x00000000000046dc ompi_coll_tuned_allreduce_intra_dec_fixed() ???:0
9 0x0000000000060e9f MPI_Allreduce() ???:0
10 0x0000000000094395 meep::dft_near2far::get_farfields_array() ???:0
11 0x0000000000094707 meep::dft_near2far::save_farfields() ???:0
12 0x0000000000467681 _wrap_meep_dft_near2far_save_farfields() meep_wrap.cxx:0
13 0x00000000000f111e scm_vm_engine() ???:0
14 0x0000000000066773 scm_primitive_eval() ???:0
15 0x0000000000088c9b scm_primitive_load() ???:0
16 0x00000000000f109f scm_vm_engine() ???:0
17 0x0000000000065fc7 scm_call_1() ???:0
18 0x0000000000496878 main_entry() ???:0
19 0x000000000008378d scm_make_locale() ???:0
20 0x000000000005c83a scm_at_abort() ???:0
21 0x00000000000f1175 scm_vm_engine() ???:0
22 0x0000000000066143 scm_call_4() ???:0
23 0x000000000005cfbf scm_at_abort() ???:0
24 0x000000000005d0a5 scm_c_with_continuation_barrier() ???:0
25 0x00000000000d882c scm_current_processor_count() ???:0
26 0x0000000000015de2 GC_call_with_stack_base() ???:0
27 0x00000000000d8c28 scm_with_guile() ???:0
28 0x0000000000083955 scm_boot_guile() ???:0
29 0x0000000000418a4d main() ???:0
30 0x0000000000022555 __libc_start_main() ???:0
31 0x0000000000418a8c _start() ???:0

srun: error: node01: task 57: Killed
srun: error: node02: task 150: Killed
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: got SIGCONT

It stops at get_farfields_array working on point 232220 of 250000 (92% done), and failed many times. May I know if this is due to a problem of my script? Here is my farfield.ctl:

`(set-param! eps-averaging? true)

(define-param w 0.045) ;; width of nanocube

(define wvl-min 0.592)
(define wvl-max 0.692)

(define frq-min (/ wvl-max))
(define frq-max (/ wvl-min))
(define frq-cen (* 0.5 (+ frq-min frq-max)))
(define dfrq (- frq-max frq-min))
(define nfrq 128)

(define ff-distance 3e5) ; far-field distance from near-field monitor
(define ff-angle 0.95) ; far-field cone angle
(define ff-npts 500) ; number of far-field points
(define ff-length (* ff-distance (tan (deg->rad ff-angle))))
(define ff-res (/ ff-npts ff-length))

(set-param! resolution 500)

(define dpml (* 0.5 wvl-min))
(define dair (* 0.5 wvl-min))

(define boundary-layers (list (make pml (thickness dpml))))
(set! pml-layers boundary-layers)

(define s (* 2 (+ dpml dair w)))
(define cell (make lattice (size s s s)))
(set! geometry-lattice cell)

(define PMMA (make medium (index 1.4887)))
(define H2O (make medium (index 1.3315)))

(set! geometry (list (make block (material PMMA) (size s s (* 0.25 s)) (center 0 0 (- (* 0.375 s))))
(make block (material H2O) (size s s (* 0.75 s)) (center 0 0 (* 0.125 s)))))

(define pw-src (make source
(src (make gaussian-src (frequency frq-cen) (fwidth dfrq) (is-integrated? true)))
(center 0 0 (+ (* -0.5 s) dpml))
(size s s 0)
(component Ey)))

(set! sources (list pw-src))

(set! k-point (vector3 0))

(define dpad (* 0.5 dair))

(define n2f-pt (vector3 0 0 (+ w dpad)))
(define n2f-obj (add-near2far frq-cen dfrq nfrq
(make near2far-region (center n2f-pt) (size (* 2 (+ w dpad)) (* 2 (+ w dpad)) 0) (weight +1))))

(run-sources+ (stop-when-fields-decayed 10 Ey n2f-pt 1e-9))

(output-farfields n2f-obj "far-field-source" (volume (center 0 0 ff-distance) (size ff-length ff-length)) ff-res)

(reset-meep)

(define-param dpe 0.010) ; depth of embedment

(set! geometry (list (make block (material PMMA) (size s s (* 0.25 s)) (center 0 0 (- (* 0.375 s))))
(make block (material H2O) (size s s (* 0.75 s)) (center 0 0 (* 0.125 s)))
(make block (material PMMA) (size w w w) (center 0 0 (- (* -0.65 w) dpe))
(e1 0 0.707106781 -0.707106781) (e2 -0.707106781 0.5 0.5) (e3 0.707106781 0.5 0.5)))) ; rotate YZ axes by 90 deg

(set! geometry-lattice cell)

(set! pml-layers boundary-layers)

(set! sources (list pw-src))

(set! k-point (vector3 0))

(define n2f-pt (vector3 0 0 (+ w dpad)))
(define n2f-obj (add-near2far frq-cen dfrq nfrq
(make near2far-region (center n2f-pt) (size (* 2 (+ w dpad)) (* 2 (+ w dpad)) 0) (weight +1))))

(run-sources+ (stop-when-fields-decayed 10 Ey n2f-pt 1e-9))

(output-farfields n2f-obj "far-field-PMMA-45nm" (volume (center 0 0 ff-distance) (size ff-length ff-length)) ff-res)`