Allow APP to differ between RUNs #2943

DavidHuber-NOAA · 2024-09-19T18:25:52Z

Description

This enables APP to be specified for each RUN. This also removes the need for a _no_run configuration dictionary and somewhat simplifies the _init_finalize method.

Resolves #2908

Type of change

New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any submodules? NO

How has this been tested?

Generated .xml files on Hera for all available CI tests and verified they were identical to develop.
Created an arbitrary APP configuration ({"gdas": "S2S", "gfs": "ATM"})

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes

DavidHuber-NOAA · 2024-09-19T19:21:42Z

This is going to require some more leg work as some APP-specific flags are currently read in without specifying a RUN:

global-workflow/workflow/applications/applications.py

Lines 34 to 50 in c6e3262

    
           base = conf.parse_config('config.base') 
        
           self.mode = base['MODE'] 
        
           if self.mode not in self.VALID_MODES: 
        
               raise NotImplementedError(f'{self.mode} is not a valid application mode.\n' 
        
                                         f'Valid application modes are:\n' 
        
                                         f'{", ".join(self.VALID_MODES)}\n') 
        
           self.net = base['NET'] 
        
           self.model_app = base.get('APP', 'ATM') 
        
           self.do_atm = base.get('DO_ATM', True) 
        
           self.do_wave = base.get('DO_WAVE', False) 
        
           self.do_wave_bnd = base.get('DOBNDPNT_WAVE', False) 
        
           self.do_ocean = base.get('DO_OCN', False) 
        
           self.do_ice = base.get('DO_ICE', False) 
        
           self.do_aero = base.get('DO_AERO', False)

This will require reworking the get_task_names methods to key off of RUN-specific flags.

DavidHuber-NOAA · 2024-09-24T15:11:09Z

workflow/applications/applications.py

@WalterKolczynski-NOAA I have made some significant changes to the Application classes to allow the APP to vary by RUN. Next, I will need to update the Rocoto class to work with this modified class. Before I start that, could you take a look at these and see if you're OK with what I have done. Happy to meet to go over details if needed.

DavidHuber-NOAA · 2024-10-01T19:29:04Z

This PR now correctly generates identical CI XMLs as compared to develop (with the exception of the stage_ic and cleanup jobs which are now run on service partitions). An additional case was tested based on the C96C48_hybatmaerosnowDA case, but the APP was set to ATMA for RUN==gdas and ATM for RUN==gfs. In this case, the gfs tasks in the resultant XML did not include any aero-component jobs.

Marking ready for review.

DavidHuber-NOAA · 2024-10-02T18:56:37Z

workflow/applications/applications.py

+            if run_options[run]['do_aero']:
+                aero_anl_run = run_base.get('AERO_ANL_RUN', 'BOTH').lower()
+                if aero_anl_run in ['both']:
+                    aero_anl_runs = ['gfs', 'gdas']
+                elif aero_anl_run in ['gfs', 'gdas']:
+                    aero_anl_runs = [aero_anl_run]
+
+                aero_fcst_run = run_base.get('AERO_FCST_RUN', None).lower()
+                if aero_fcst_run in ['both']:
+                    aero_fcst_runs = ['gfs', 'gdas']
+                elif aero_fcst_run in ['gfs', 'gdas']:
+                    aero_fcst_runs = [aero_fcst_run]
+
+                run_options[run]['do_aero_anl'] = True if run in aero_anl_runs else False
+                run_options[run]['do_aero_fcst'] = True if run in aero_fcst_runs else False
+


I think this logic can be redesigned in config.base so that, e.g. do_aero_anl would be set based on RUN, making the Python logic simpler and making it easier for developers to modify. There are some other places I will tag that should be moveable as well. Should this be done as part of this PR or could it wait for a future issue?

DavidHuber-NOAA · 2024-10-02T18:59:24Z

workflow/applications/gfs_cycled.py

+                if run == 'gdas':
+                    if not options['do_jediatmvar']:
+                        task_names[run] += ['analdiag']
+
+                    if options['do_wave'] and run in self.wave_runs:
+                        task_names[run] += wave_prep_tasks
+
+                    if options['do_aero'] and run in options['aero_anl_runs']:
+                        task_names[run] += ['aeroanlgenb']
+
+                else:
+                    if options['do_wave'] and run in self.wave_runs:
+                        task_names[run] += wave_prep_tasks


These could be made into RUN-specific options in config.base.

DavidHuber-NOAA · 2024-10-02T19:03:07Z

workflow/applications/applications.py

+            wave_runs = []
+            if run_options[run]['do_wave']:
+                wave_run = run_base.get('WAVE_RUN', 'BOTH').lower()
+                if wave_run in ['both']:
+                    wave_runs = ['gfs', 'gdas']
+                elif wave_run in ['gfs', 'gdas']:
+                    wave_runs = [wave_run]


This could be handled in config.base so do_wave is set based on RUN.

DavidHuber-NOAA · 2024-10-02T19:04:54Z

workflow/applications/gfs_cycled.py

+                    if options['do_wave'] and run in self.wave_runs:
+                        task_names[run] += wave_prep_tasks
+
+                if options['do_aero'] and run in options['aero_anl_runs']:


do_aero could be set in config.base based on RUN.

DavidHuber-NOAA · 2024-10-02T19:06:24Z

workflow/applications/gfs_cycled.py

+                    if options['do_metp']:
+                        task_names[run] += ['metp']
+
+                if options['do_wave'] and run in options['wave_runs']:


do_wave could be set in config.base based on RUN.

DavidHuber-NOAA · 2024-10-02T19:08:39Z

workflow/applications/gfs_forecast_only.py

+        if options['do_aero']:
+            aero_fcst_run = options['aero_fcst_run']
            if self.run in aero_fcst_run or aero_fcst_run == "both":
-                if not self.exp_warm_start:
+                if not options['exp_warm_start']:
                    tasks += ['aerosol_init']


do_aero_fcst could be set in config.base based on RUN.

DavidHuber-NOAA · 2024-10-02T19:10:34Z

workflow/rocoto/gfs_tasks.py

-        if self.app_config.do_aero and \
-           self.run in self.app_config.aero_fcst_runs and \
+        if self.options['do_aero'] and \
+           self.run in self.options['aero_fcst_runs'] and \


do_aero_fcst could be set in config.base based on RUN.

DavidHuber-NOAA · 2024-10-02T19:10:59Z

workflow/rocoto/gfs_tasks.py

            dep_dict = {'type': 'task', 'name': f'{self.run}marineanlfinal'}
            dependencies.append(rocoto.add_dependency(dep_dict))

-        if self.app_config.do_aero and self.run in self.app_config.aero_anl_runs:
+        if self.options['do_aero'] and self.run in self.options['aero_anl_runs']:


do_aero_anl could be set in config.base based on RUN.

DavidHuber-NOAA added 4 commits September 19, 2024 16:34

Additions to allow APP to differ by RUN

c0102bf

Make cleanup and stage jobs run on service partitions

d987c38

Combine get_valid_runs and get_task_names

599beeb

Simplify get_task_names

eaa1049

DavidHuber-NOAA added 4 commits September 20, 2024 13:13

Make APP-specific options based on RUN

3787be7

Move _get_app_configs invokation to _source_configs

a4ab75c

Make the Application classes run-configurable

4ae0f36

Fix f-string formating

4374bad

DavidHuber-NOAA commented Sep 24, 2024

View reviewed changes

DavidHuber-NOAA added 9 commits September 24, 2024 15:02

Application bug fixes

e6b1920

Update app_config options in the Tasks classes

d94b685

Fix hybvar ocn analysis task tabbing

10763cc

Merge develop branch

2f07ddb

do_extractvars bug fix

7dbf11a

Fix do_extractvars bug, remove unused modules

de57402

Fix additional do_* bugs

7253eef

Rearrange gfs_cycled task order

9e96844

Rearrange XML RUN order to match develop

8275caf

DavidHuber-NOAA force-pushed the feature/multi-app branch from 217ec77 to 8275caf Compare October 1, 2024 15:31

DavidHuber-NOAA added 2 commits October 1, 2024 10:44

Merge develop, address conflicts in gfs_cycled and gfs_tasks

60f6384

Fix merge bugs

4477e60

DavidHuber-NOAA requested review from WalterKolczynski-NOAA, aerorahul and EricSinsky-NOAA October 1, 2024 19:23

DavidHuber-NOAA marked this pull request as ready for review October 1, 2024 19:24

DavidHuber-NOAA commented Oct 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow APP to differ between RUNs #2943

Allow APP to differ between RUNs #2943

DavidHuber-NOAA commented Sep 19, 2024 •

edited

Loading

DavidHuber-NOAA commented Sep 19, 2024

DavidHuber-NOAA Sep 24, 2024

DavidHuber-NOAA commented Oct 1, 2024

DavidHuber-NOAA Oct 2, 2024

DavidHuber-NOAA Oct 2, 2024

DavidHuber-NOAA Oct 2, 2024

DavidHuber-NOAA Oct 2, 2024

DavidHuber-NOAA Oct 2, 2024

DavidHuber-NOAA Oct 2, 2024

DavidHuber-NOAA Oct 2, 2024

DavidHuber-NOAA Oct 2, 2024

Allow APP to differ between RUNs #2943

Are you sure you want to change the base?

Allow APP to differ between RUNs #2943

Conversation

DavidHuber-NOAA commented Sep 19, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

DavidHuber-NOAA commented Sep 19, 2024

Choose a reason for hiding this comment

DavidHuber-NOAA commented Oct 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DavidHuber-NOAA commented Sep 19, 2024 •

edited

Loading