Adjust C768 resources for Hera

[DavidHuber-NOAA]

Description

This modifies the resources for gdasfcst (everywhere) and enkfgdaseupd (Hera only). For the fcst job, the number of write tasks is increased to prevent out of memory errors from the inline post. For the eupd, the number of tasks is decreased to prevent out of memory errors. The runtime for the eupd job was just over 10 minutes.

Resolves #2506
Resolves #2498
Resolves #2916

Type of change

• Bug fix (fixes something broken)

Change characteristics

• Is this a breaking change (a change in existing functionality)? YES/NO
• Does this change require a documentation update? YES/NO
• Does this change require an update to any of the following submodules? YES (If YES, please add a link to any PRs that are pending.)
  • GSI-utils Send/receive layers to reduce buffer transfer time GSI-utils#49 and Bugfix for where some fields are not in the input increment file GSI-utils#53

How has this been tested?

Successfully ran through the mentioned jobs at least once each. More testing to come.

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing tests pass with my changes

[spanNOAA]

It seems that this pull request does not resolve the issue with eupd. For further details, you may review the log files located at:
/scratch2/BMC/wrfruc/Sijie.Pan/ufs-ar/comroot/stmp/RUNDIRS/C768_6hourly_0210/eupd.1499050/stderr
/scratch2/BMC/wrfruc/Sijie.Pan/ufs-ar/comroot/stmp/RUNDIRS/C768_6hourly_0210/eupd.2575750/stderr
These logs should provide more insight into the problem.

[DavidHuber-NOAA]

@spanNOAA You are correct. Somewhere along the way, I dropped the working configuration by accident. It should be running two tasks/node (i.e. threads_per_task=20).

I am working presently to see if I can spread the tasks out differently so more tasks can run on each node. The issue is that the first 4 tasks (the I/O tasks) must store an enormous amount of data in memory (about 40GB each). The remaining tasks are not as memory intensive. One way to solve this is to develop an arbitrary distribution scheme rather than the default block distribution. Something like

srun -N <n nodes> ... --distribution=arbitrary -w tux[0,1,2,3,4,4,4,4,4,5,5,5,5,5,...,n-1, n-1, n-1, n-1, n-1]

But this is not trivial to program. I will likely go with threads_per_task=20 for the time being, but want to run one more test first and then I will update the PR.

[spanNOAA]

@spanNOAA You are correct. Somewhere along the way, I dropped the working configuration by accident. It should be running two tasks/node (i.e. threads_per_task=20).

I am working presently to see if I can spread the tasks out differently so more tasks can run on each node. The issue is that the first 4 tasks (the I/O tasks) must store an enormous amount of data in memory (about 40GB each). The remaining tasks are not as memory intensive. One way to solve this is to develop an arbitrary distribution scheme rather than the default block distribution. Something like

srun -N <n nodes> ... --distribution=arbitrary -w tux[0,1,2,3,4,4,4,4,4,5,5,5,5,5,...,n-1, n-1, n-1, n-1, n-1]

But this is not trivial to program. I will likely go with threads_per_task=20 for the time being, but want to run one more test first and then I will update the PR.

@DavidHuber-NOAA Thanks for the update.

[guoqing-noaa]

Thanks, @DavidHuber-NOAA!

@spanNOAA Does threads_per_task=20 work for your case?

[spanNOAA]

@DavidHuber-NOAA ntasks=80 works now.

[guoqing-noaa]

This PR solved our C768 issues on Hera.

Thanks, @DavidHuber-NOAA

[DavidHuber-NOAA]

@WalterKolczynski-NOAA @aerorahul @RussTreadon-NOAA Note that this PR (and the dependent gsi-utils PR NOAA-EMC/GSI-utils#49) has changes that will affect the C768 forecast on all machines and the gdasanalcalc job at all resolutions. I have not performed any testing to verify that values did not change. I should be able to do that on WCOSS2 next week. Opening for review in the meantime, but please do not merge until I have done that testing.

[RussTreadon-NOAA]

Thank you @DavidHuber-NOAA for running tests on WCOSS2. Hopefully the changes to gdasanalcalc via GSI-utils PR #49 do not alter results.

[RussTreadon-NOAA]

@CatherineThomas-NOAA , tagging you for awareness. Not sure if / how these changes might impact plans for using Hera to run GFS v17 parallels.

[emcbot]

CI Failed on Orion in Build# 3
Built and ran in directory /work2/noaa/stmp/CI/ORION/2819

Experiment C48_S2SWA_gefs_1921288e Terminated with  tasks failed and  dead at Thu Sep 19 11:12:47 AM CDT 2024
Experiment C48_S2SWA_gefs_1921288e Terminated: **
Experiment C96_atm3DVar_1921288e Terminated with  tasks failed and  dead at Thu Sep 19 11:12:47 AM CDT 2024
Experiment C96_atm3DVar_1921288e Terminated: **
Experiment C48_S2SW_1921288e Terminated with  tasks failed and  dead at Thu Sep 19 11:12:47 AM CDT 2024
Experiment C48_S2SW_1921288e Terminated: **
Experiment C96C48_hybatmDA_1921288e Terminated with  tasks failed and  dead at Thu Sep 19 11:12:47 AM CDT 2024
Experiment C96C48_hybatmDA_1921288e Terminated: **

[WalterKolczynski-NOAA]

The tracker is explicitly disabled on Hercules in the J-Jobs:

global-workflow/jobs/JGFS_ATMOS_CYCLONE_GENESIS

Line 9 in 7588d2b

if [[ "${machine}" == 'HERCULES' ]]; then exit 0; fi

I'll remove this and place it in config.base instead for both Hercules and Orion.

We really should be setting his in the hosts file.

[DavidHuber-NOAA]

@TerrenceMcGuinness-NOAA Can you tell what happened with the Orion CI tests? It almost looks like the EXPDIR directories were deleted mid-run.

[TerrenceMcGuinness-NOAA]

@DavidHuber-NOAA
! Can you tell what happened with the Orion CI tests? It almost looks like the EXPDIR directories were deleted mid-run.

Yes it did ...

The Jenkins custom Workspace is designated per the PR name. When a PR CI is re-ran from a previous fail it uses the same path and removes COMROOT and EXPDIR under its RUNDIRS directory. I couldn't tell what happened specificly with the miss labeling from the logs. My hunch is that the jobs was actually hung in the controller.

[DavidHuber-NOAA]

The tracker is explicitly disabled on Hercules in the J-Jobs:

global-workflow/jobs/JGFS_ATMOS_CYCLONE_GENESIS

Line 9 in 7588d2b

if [[ "${machine}" == 'HERCULES' ]]; then exit 0; fi

I'll remove this and place it in config.base instead for both Hercules and Orion.

We really should be setting his in the hosts file.

This will require a change to all of the host files and the CI defaults (which override whatever is in the host files) and will need to be tested on each platform. I would prefer to open a new issue to tackle this as this PR is already tackling a few different issues and going beyond its scope.

[DavidHuber-NOAA]

Opened #2942

[emcbot]

CI Passed on Hercules in Build# 4
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2819

Experiment C48_ATM_62ea3892 Completed 1 Cycles: *SUCCESS* at Thu Sep 19 13:24:53 CDT 2024
Experiment C48_S2SW_62ea3892 Completed 1 Cycles: *SUCCESS* at Thu Sep 19 14:31:32 CDT 2024
Experiment C96_atm3DVar_62ea3892 Completed 3 Cycles: *SUCCESS* at Thu Sep 19 14:43:42 CDT 2024
Experiment C96C48_hybatmDA_62ea3892 Completed 3 Cycles: *SUCCESS* at Thu Sep 19 14:56:08 CDT 2024
Experiment C48_S2SWA_gefs_62ea3892 Completed 1 Cycles: *SUCCESS* at Thu Sep 19 14:56:59 CDT 2024

[DavidHuber-NOAA]

Looking through Orion's logs, I see that all experiments completed successfully. However, for some reason the C96_atm3DVar and C48_ATM tests, once completed, did not trigger SUCCESS notifications. Manually changing label to CI-Orion-Passed.

[emcbot]

Experiment C48_S2SWA_gefs FAILED on Orion in Build# 5 in
/work2/noaa/stmp/CI/ORION/2819/RUNTESTS/EXPDIR/C48_S2SWA_gefs_62ea3892

[emcbot]

CI Failed on Orion in Build# 5
Built and ran in directory /work2/noaa/stmp/CI/ORION/2819

**CI Failed** on Orion in Build# 3<br>Built and ran in directory `/work2/noaa/stmp/CI/ORION/2819`

Experiment C48_S2SWA_gefs_1921288e Terminated with tasks failed and dead at Thu Sep 19 11:12:47 AM CDT 2024
Experiment C48_S2SWA_gefs_1921288e Terminated: **
Experiment C96_atm3DVar_1921288e Terminated with tasks failed and dead at Thu Sep 19 11:12:47 AM CDT 2024
Experiment C96_atm3DVar_1921288e Terminated: **
Experiment C48_S2SW_1921288e Terminated with tasks failed and dead at Thu Sep 19 11:12:47 AM CDT 2024
Experiment C48_S2SW_1921288e Terminated: **
Experiment C96C48_hybatmDA_1921288e Terminated with tasks failed and dead at Thu Sep 19 11:12:47 AM CDT 2024
Experiment C96C48_hybatmDA_1921288e Terminated: **
Experiment C48_ATM_62ea3892 Completed 1 Cycles: SUCCESS at Thu Sep 19 07:33:08 PM CDT 2024
Experiment C96_atm3DVar_62ea3892 Completed 3 Cycles: SUCCESS at Fri Sep 20 05:59:17 AM CDT 2024
Experiment C96C48_hybatmDA_62ea3892 Completed 3 Cycles: SUCCESS at Fri Sep 20 06:48:06 AM CDT 2024
Experiment C48_S2SW_62ea3892 Completed 1 Cycles: SUCCESS at Fri Sep 20 07:06:25 AM CDT 2024
Experiment C48_S2SWA_gefs_62ea3892 Terminated with tasks failed and dead at Fri Sep 20 09:09:42 AM CDT 2024
Experiment C48_S2SWA_gefs_62ea3892 Terminated: **