gpu performance with EB

[indra098124]

Hi PeleC community,
what is your experience with gpu + EB? I am finding that with gpu EB code is very slow.

[marchdf]

Hi, thanks for reaching out! We have used EB pretty well on Frontier GPUs without issues. What type of GPUs are you using and do you have. a recent version of PeleC? We did many performance improvements in recent months. Also, can you post the profiling results (need to build with TINY_PROFILE=TRUE?

[drummerdoc]

Also check your version of PelePhysics if you are not using the submodule version control. Lots and lots of changes in PelePhysics.

[indra098124]

Thank you for looking at it. I am using the latest code (downloaded a week ago). Here is the profile with EB-Converging nozzle with base grid 144 x 96 x 96 and max_level 3. This is with A100

CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 63704481792
Total GPU global memory (MB) spread across MPI: [81004 ... 81004]
Free GPU global memory (MB) spread across MPI: [43854 ... 45810]
[The Arena] space (MB) allocated spread across MPI: [60753 ... 60753]
[The Arena] space (MB) used spread across MPI: [0 ... 0]
[The Device Arena] space (MB) allocated spread across MPI: [8 ... 8]
[The Device Arena] space (MB) used spread across MPI: [0 ... 0]
[The Pinned Arena] space (MB) allocated spread across MPI: [5882 ... 11598]
[The Pinned Arena] space (MB) used spread across MPI: [0 ... 0]
AMReX (23.12-24-g75571e2dcbf2-dirty) finalized

this is for 288 x 192 x 192, max_level 3 (mesh count approx. 210M)
CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 150576711552
Total GPU global memory (MB) spread across MPI: [81004 ... 81004]
Free GPU global memory (MB) spread across MPI: [0 ... 0]
[The Arena] space (MB) allocated spread across MPI: [136362 ... 145888]
[The Arena] space (MB) used spread across MPI: [0 ... 0]
[The Device Arena] space (MB) allocated spread across MPI: [8 ... 8]
[The Device Arena] space (MB) used spread across MPI: [0 ... 0]
[The Pinned Arena] space (MB) allocated spread across MPI: [18573 ... 33607]
[The Pinned Arena] space (MB) used spread across MPI: [0 ... 0]
AMReX (23.12-24-g75571e2dcbf2-dirty) finalized

[baperry2]

@indra098124 that shows the memory utilization, but when you run after compiling with TINY_PROFILE=TRUE, you should get some compute time profiling at the end of the terminal output for your simulation similar to this:

TinyProfiler total time across processes [min...avg...max]: 27.35 ... 27.35 ... 27.35

--------------------------------------------------------------------------------------------
Name                                         NCalls  Excl. Min  Excl. Avg  Excl. Max   Max %
--------------------------------------------------------------------------------------------
ApplyMLRedistribution()                        2240      11.73      11.73      11.73  42.89%
PeleC::pc_hyp_mol_flux()                       2240      5.665      5.665      5.665  20.71%
PeleC::diffusion_flux()                        2240      1.996      1.996      1.996   7.30%
PeleC::initialize_signed_distance()               1      1.798      1.798      1.798   6.57%
PeleC::getMOLSrcTerm()                           40      1.764      1.764      1.764   6.45%
PeleC::pc_apply_face_stencil()                 2240     0.9781     0.9781     0.9781   3.58%

That's the profile for the default EB-ConvergingNozzle case running on CPU. It looks like state redistribution algorithm might be what's slowing things down, but please also run on your system to verify. For this simple case using GammaLaw EOS, you may be able to use flux redistribution (pelec.redistribution_type = FluxRedist) and achieve better performance without losing robustness.

@marchdf - thoughts on why redistribution is so slow here? Is it just that most cases we run have chemistry and that dwarfs redistribution cost?

[jrood-nrel]

On the GPU make sure to use tiny_profiler.device_synchronize_around_region=1.

[indra098124]

Thanks @baperry2 and @jrood-nrel for the suggestions and looking into this. This is my observation too that the redistribution takes very long. Our nodes are down for maintenance, but hopefully, I will get the new numbers by tomorrow. @baperry2 on CPU, I get a similar output, but with gpu with TINY_PROFILE TRUE, I only got the output that I posted. I will try with what @jrood-nrel suggested.

@baperry2 may I ask if you have any guidelines/document that may give some suggestions on where flux_redistribution might be ok to use?

[baperry2]

The available documentation on redistribution is here: https://amrex-combustion.github.io/PeleC/geometry/EB.html#redistribution

Flux redistribution should be ok whenever using the GammaLaw equation of state. The issue can come in for multi-species flows (Eos_model = Fuego or Soave_Redlich-Kwong) when the heats of formation are included in the species internal energies. In those cases, the flux redistribution algorithm sometimes causes negative temperatures in small cells at the EB. That depends on specifics of the EB geometry in an unpredictable manner through; sometimes flux redistribution is fine in combustion flows, sometimes it isn't.

[indra098124]

Here is the output of the profiler. disabled checkpoint and plotfile writing. Copied here the top rows to show the most expensive calls. Here the parallelCopy takes 50% of the time. This is EB-ConvergingNozzle case with max_level 3 and the base grid 288 x 192 x 192

TinyProfiler total time across processes [min...avg...max]: 405.9 ... 405.9 ... 405.9

Name NCalls Excl. Min Excl. Avg Excl. Max Max %

FabArray::ParallelCopy_finish() 636 94.63 153.8 200.6 49.43%
AmrLevel::AmrLevel(dm) 13 30.33 42.53 55.49 13.67%
PeleC::pc_apply_face_stencil() 46484 26.86 38.41 52.5 12.93%
PeleC::initialize_eb2_structs() 13 16.61 25.61 36.51 8.99%
FabArray::ParallelCopy_nowait() 636 20.43 28.04 32.3 7.96%
PeleC::getMOLSrcTerm() 30 14.96 18.49 21.96 5.41%
FillPatchInterp(MF) 48 14.59 17.91 21.75 5.36%
Amr::regrid() 3 10.45 16.4 20.7 5.10%
FPinfo::FPinfo() 15 16.63 18.92 19.94 4.91%
FillPatchTwoLevels 158 8.416 9.797 11.08 2.73%
FillPatcher::fillCFB 28 6.216 7.166 8.454 2.08%
StateData::define() 39 2.584 4.121 6.031 1.49%
ApplyMLRedistribution() 46484 3.161 4.074 5.053 1.24%
FabArray::setVal() 400 2.9 3.068 3.295 0.81%
PeleC::InitialRedistribution() 13 1.594 1.899 2.141 0.53%

---

Name NCalls Incl. Min Incl. Avg Incl. Max Max %

main() 1 405.9 405.9 405.9 100.00%
Amr::init() 1 248.9 249.2 249.3 61.41%
Amr::initialInit() 1 248.9 249.2 249.3 61.41%
Amr::FinalizeInit() 1 246.7 246.9 247 60.85%
Amr::bldFineLevels() 1 245.7 245.7 245.8 60.54%
FabArray::ParallelCopy() 636 127.3 183.4 232.6 57.29%
Amr::regrid() 3 229.7 229.8 229.8 56.62%
FabArray::ParallelCopy_finish() 636 94.63 153.8 200.6 49.43%
Amr::coarseTimeStep() 1 154.3 154.6 154.7 38.11%
Amr::timeStep() 15 154.3 154.6 154.7 38.11%
PeleC::advance() 15 142.5 146.9 151.9 37.43%
PeleC::do_mol_advance() 15 142.2 146.6 151.7 37.37%
AmrLevel::AmrLevel(dm) 13 115.2 117.2 119.7 29.50%
AmrLevel::FillPatch() 211 65.13 82.14 98.68 24.31%
FillPatchIterator::Initialize 211 64.89 81.92 98.52 24.27%
FillPatchIterator::FillFromTwoLevels() 158 63.02 80.37 97.2 23.95%
FillPatchTwoLevels 158 63.02 80.37 97.2 23.95%
AmrLevel::FillPatcherFill() 30 67.24 81.68 94.34 23.24%
FillPatcher::fill() 28 67.18 81.61 94.26 23.22%
PeleC::initData() 13 61.48 78.17 94.24 23.22%
PeleC::InitialRedistribution() 13 61.44 78.13 94.22 23.21%
FillPatcher::fillCFB 28 63.77 78.28 89.35 22.01%
PeleC::getMOLSrcTerm() 30 47.63 64.54 83.99 20.69%
PeleC::pc_apply_face_stencil() 46484 26.86 38.41 52.5 12.93%
FillPatchSingleLevel 259 6.863 23.79 41.49 10.22%
PeleC::initialize_eb2_structs() 13 16.64 25.62 36.52 9.00%
FabArray::ParallelCopy_nowait() 636 22.13 29.6 33.83 8.33%
FabArrayBase::TheFPinfo() 48 22.1 24.52 26.94 6.64%
FPinfo::FPinfo() 15 22.1 24.52 26.94 6.64%
FillPatchInterp(MF) 48 14.59 17.91 21.75 5.36%
PeleC::post_timestep() 15 2.746 7.674 11.84 2.92%

[marchdf]

FabArray::ParallelCopy_finish() typically indicates a load balancing problem... There are flags in amrex you can use to tweak this: https://amrex-codes.github.io/amrex/docs_html/ManagingGridHierarchy_Chapter.html. In your input files: what is max_grid_size and what is blocking_factor? Those can have impacts on your performance. On GPU you probably want the former to be something like 128.

[baperry2]

How does the profile compare if you use FluxRedist?

Also, I'm not sure what your intentions with this simulation are, but a 288x192x192 base grid with 3 additional refinement levels seems like overkill for the converging nozzle case.

[indra098124]

@baperry2 I agree with you. This simulation with this configuration is an absolute overkill. I just used this simulation for testing the gpu performance and it was easy to communicate the details for this simulation. I went this way because with my other simulation I noticed that the simulations were noticeably slow with EB on gpu compared to a similar nonEB case.

I tried with FluxRedistribution and the cost of ApplyMLRedistribution() went up from 1.24% for StateRedistribution to 5.36% for FluxRedistibution. Here are the numbers.
TinyProfiler total time across processes [min...avg...max]: 299.7 ... 299.7 ... 299.7

---

Name NCalls Excl. Min Excl. Avg Excl. Max Max %

FabArray::ParallelCopy_finish() 575 62.29 109.5 144.7 48.27%
PeleC::pc_apply_face_stencil() 40812 26.13 39.42 56.69 18.92%
AmrLevel::AmrLevel(dm) 13 17.52 24.44 32.07 10.70%
PeleC::getMOLSrcTerm() 30 13.23 17.83 22.21 7.41%
PeleC::initialize_eb2_structs() 13 12.53 15.06 18.75 6.26%
FabArray::ParallelCopy_nowait() 575 14.88 16.02 18.42 6.15%
FillPatchInterp(MF) 36 11.6 13.91 17.31 5.77%
ApplyMLRedistribution() 40812 8.139 11.82 16.05 5.36%
Amr::regrid() 3 5.158 9.392 13.8 4.60%
FillPatcher::fillCFB 28 8.532 10.14 12.64 4.22%
PeleC::extend_signed_distance() 1 0.6956 4.944 10.34 3.45%
FillBoundary_finish() 68 0.2837 5.556 9.879 3.30%
FPinfo::FPinfo() 9 4.137 4.94 5.964 1.99%
StateData::define() 39 1.704 2.403 3.38 1.13%
FabArray::setVal() 388 2.111 2.137 2.175 0.73%

@marchdf I am using blocking_factor 8 and used max_grid 64. Tried with max_grid 128, the numbers did not change much. This is with stateredistribution, blocking_factor 8 and max_grid 128.

TinyProfiler total time across processes [min...avg...max]: 382.9 ... 382.9 ... 382.9

---

Name NCalls Excl. Min Excl. Avg Excl. Max Max %

FabArray::ParallelCopy_finish() 636 101.4 151.7 198.4 51.80%
PeleC::pc_apply_face_stencil() 40942 21.61 32.67 47.03 12.28%
AmrLevel::AmrLevel(dm) 13 25.98 34.52 43.92 11.47%
FabArray::ParallelCopy_nowait() 636 17.4 27.51 31.55 8.24%
PeleC::initialize_eb2_structs() 13 14 20.67 29.13 7.61%
PeleC::getMOLSrcTerm() 30 13.38 17.86 22.13 5.78%
FillPatchInterp(MF) 48 14.39 16.73 19.52 5.10%
FPinfo::FPinfo() 15 15.18 17.56 19.37 5.06%
Amr::regrid() 3 8.073 12.04 15.36 4.01%
FillPatchTwoLevels 158 7.806 9.206 10.33 2.70%
PeleC::extend_signed_distance() 1 0.555 4.722 9.746 2.55%
FillBoundary_finish() 68 0.2209 5.165 9.478 2.48%
FillPatcher::fillCFB 28 6.261 7.218 8.631 2.25%
StateData::define() 39 2.144 3.397 4.879 1.27%
ApplyMLRedistribution() 40942 3.174 4.04 4.862 1.27%

[baperry2]

Okay, that makes sense about just using this is a test case. Because of the load balance issues mentioned by @marchdf and all the time spent at ParallelCopy_finish(), the other timings in the profile aren't that interoretable - the notable thing is that changing the redistribution type seems to have reduced the total runtime by ~25% (400s to 300s). But that might not actually be useful if the case you care about is a multi-species case with non-GammaLaw EoS and needs StateRedist.

You may also want to try some higher values for blocking_factor.