Velocity perturbations using random number generator in bcnormal() function for GPU

[eskimmel]

Hello,

I am trying to impose a 2% streamwise velocity perturbation to cells at the inlet of my domain using GPUs. I am trying to do this by modifying the value of M_in (which affects u_in) in the bcnormal() function of the prob.H file. Previously, when I was running on CPUs, I used the following line to perturb the Mach number:

amrex::Real M_in_perturbed = (1.0 - perturbation)*prob_parm.M_inlet + static_cast <float> (rand()) / ( static_cast <float> (RAND_MAX/(2.0*perturbation*prob_parm.M_inlet + 1e-16)));

Unfortunately, rand() is not compatible with GPUs. A similar question was asked in the AMReX GitHub and tried I implementing a similar method by making a loop over one value.

amrex::Real n = 1;
amrex::ParallelForRNG(n,
[=] AMREX_GPU_DEVICE (amrex::RandomEngine const& rd) noexcept
{
M_in_perturbed = (1.0 - perturbation)*prob_parm.M_inlet + 2.0*pert*amrex::Random(rd)*prob_parm.M_inlet;
});

However, using the amrex::ParallelForRNG loop does not allow me to access the new value of M_in_perturbed later in the function when it is used to define u_in. The error I get is "identifier M_in_perturbed is undefined" when I try to call it later.

All I want to do is create one random number in the bcnormal() function in a GPU-compatible way that I can use to perturb the velocities in a manner similar to what I did when using CPUs. Any assistance would be greatly appreciated. Thank you!

[baperry2]

Thanks for reaching out with your question!

Unfortunately, I don’t think that what you’re trying here will work for a couple reasons. Algorithmically, the bcnormal function is called whenever the State MultiFab gets FillPatched. This can happen several times per timestep, e.g. on each SDC or MOL iteration, and you’d want the fluctuation to be consistent every time and across different AMR levels, which would be difficult to enforce. From an implementation perspective, if you follow the call stack back, bcnormal is already a GPU kernel called from within a ParallelFor within some AMReX code, so I’m not sure your strategy of having a nested ParallelForRNG can work. I haven’t done much with AMReX random number generation so I'm not sure if there is a way to generate random numbers on GPUs from code launched by a ParallelFor rather than ParallelForRNG.

You can probably accomplish what you want using the (still mostly undocumented) TurbInflow capability. It’s set up to add velocity fluctuations from a synthetic turbulent spectrum, which depending on your intended application may provide more physically realistic spatial and temporal correlations between the fluctuations. It would be a bit of overkill, but you could also create a data file of the appropriate format with purely random fluctuations, and that would ensure consistency when they are applied. The best example of the workflow for this capability is this PeleLMeX regtest, although it works a slightly differently in PeleC. In PeleC, the 3 components of velocity are interpolated from your provided file and passed to bcnormal as turb_fluc and you can do whatever you want with them within bcnormal.

[eskimmel]

Hi, thanks for the response. My domain uses a development region, so I really just need velocity perturbations to trigger some instabilities to allow turbulence to form naturally within the development region. I was able to solve my problem, so I thought I would write my solution method on here for anyone who runs into a similar issue.

I used cuRAND to generate the random numbers at my inlet. In bcnormal, I used the following lines with curand.h and curand_kernel.h in the header:

curandState s;
int seed = floor(abs(10000000.0 * a_time * x[1] * x[2]) + 25) ; (to create a different seed at every inlet cell)
curand_init(seed, 0, 0, &s);
amrex::Real random_number = curand_uniform(&s);
amrex::Real M_in_perturbed = (1.0 - perturbation)*prob_parm.M_inlet + 2.0*perturbation*random_number*prob_parm.M_inlet;