P8 multi-threading?

[benjamin-cash]

What steps do I need to take to run with multi-threading in P8?

@JessicaMeixner-NOAA

[JessicaMeixner-NOAA]

@benjamin-cash are you referring to using ESMF managed threading? If so, please see the regression test where this is enabled: cpld_bmark_esmfthreads_p8
Note this feature was added after the P8 tag, so you will need to use a later version of the code to enable this feature.

[benjamin-cash]

Thanks! Is there such a thing as a 'dry run' version of the regression tests, that will just generate the batch script and the various configuration files? So far as I know the regression tests do not actually run on Frontera, but all I really need is a sample set of input.nml, model_configure, job script etc. Or will I need to run this on Orion and then export?

[JessicaMeixner-NOAA]

Running on orion, saving the directory is a great way to go. Otherwise, you could back-out all the information you need form the scripts, but it might be simpler to just make the directory. It seems like @jiandewang might have a directory he can point you to in the meantime.

[jiandewang]

@benjamin-cash I may have what you need. I had some testing of UFS yesterday, normally the run dir will be deleted automatically but since one of my job failed to match baseline, the run dir must be still on disk. HERA is down now, do you have access to GAEA ? If not I can move them to ORION, just let me know

[benjamin-cash]

@jiandewang All I have access to currently is Orion. The process is in motion to get me access to Hera etc, but my understanding is that isn't likely to come through for months. So if you could move it to Orion that would be great. :)

[jiandewang]

@benjamin-cash OK I will move it to ORION shortly

[benjamin-cash]

Thank you!

[jiandewang]

Thank you!

Just realized HERA is down, there is no way to scp from GAEA to ORION directly, will have what you need once HERA is back

[benjamin-cash]

No worries, it's not a rush.

[jiandewang]

@benjamin-cash see /work/noaa/marine/Jiande.Wang/For-others/Ben/cpld_esmfthreads_p8

[benjamin-cash]

Thanks @jiandewang - Could you also point me to your job script? Are there specific environment variables you need to set to enable multithreading, for example?

[benjamin-cash]

And are there any different flags you need to set at compile time?

[jiandewang]

@benjamin-cash I assume "job_card" is the one you are looking for, it is included in the directory. But I am not sure on your 2nd question, I assume it doesn't need special flag

[benjamin-cash]

Whoops! Not sure how I missed that. I see that OMP_NUM_THREADS=1, is threading being set by PSM_RANKS_PER_CONTEXT? That's not a variable I am familiar with.

[jiandewang]

@benjamin-cash I am not familiar on that part, I think either @junwang-noaa or @DeniseWorthen will be able to give you a clear answer

[benjamin-cash]

@jiandewang - Did you see a significant speed up using this test? Just swapping in the new executable and the PSM_* variables in job_card seems to have had essentially no effect, which makes me think there must be another setting I have missed.

[jiandewang]

@jiandewang - Did you see a significant speed up using this test? Just swapping in the new executable and the PSM_* variables in job_card seems to have had essentially no effect, which makes me think there must be another setting I have missed.

my understanding is that EIB people added cpld_esmfthreads_p8 case to make sure it works under multi threading, there is no counterpart to compare the speed.

[benjamin-cash]

Ah, ok. Thanks!

[ShawnCebulaNOAA]

@benjamin-cash - Were you able to get this figured out? I can check with the EPIC team with NOAA for more answers if you need them!

[mark-a-potts]

I have found that a good way to get all the inputs you need for a case is to run the rt.sh script for the case you want with a command like this --
./rt.sh -r -k -l rt.p8

where the file rt.p8 has just the one test you want to run in it (cut it out of the rt.conf file which has all the tests). Then, after the compile step of the test is completed, chmod the resulting fv3_001.exe file in ufs-weather-model/tests to -w, so that it will get copied, but won't run. Your executable job will fail, but all the input data will be in your case directory with the possible exception of aerosol data in ExtDat, which seems to be soft linked, currently. You should be able to run a tar with the "-h" flag to dereference symlinks and get everything, though.

In terms of the threading, I believe that it is set in the nems.configure file. Each component has an *_omp_num_threads: entry and ESMF will use that to handle the threading internally. I think it overrides any environment variable you set. For the run, there will be a PET for every thread, so if you ask for 156 (0 155) PETs and 2 omp threads in ATM , for example, it will essentially run with 78 MPI processes that each have 2 openMP threads.

[benjamin-cash]

Thanks @mark-a-potts and @ShawnCebulaNOAA , this was very helpful. :)

[benjamin-cash]

@mark-a-potts - Follow up question to this. Where does rt.sh put the information on all of the settings for a given test? In this case specifically I'm trying to figure out what is being changed between the initial and restart runs of p8, what files get moved where, etc. My runs on Frontera aren't reproducing and I am beating my head against the wall trying to figure out why.

[mark-a-potts]

I am not sure, tbh. I know for the restart runs, some files are moved/copied from the initial run and used for the restart, but I don't know where that is handled in the guts of rt.sh. I can do a run of both on hera and do the chmod -x trick on the restart case, if you like. Which cases, in particular do you want to use?

[benjamin-cash]

Hi @mark-a-potts - Thanks for the offer! I think I have tracked everything down at this point and the model is restarting bit-for-bit. It would still be good to know where this is happening and logged in rt.sh for future reference, but hopefully I won't need to go digging into it like this again for the foreseeable future.