Adding a gefs_arch task to GEFS workflow #3211

AntonMFernando-NOAA · 2025-01-07T22:27:21Z

Description

The gefs_arch task is not generated during the execution of the workflow setup scripts. It is only created when do_extractvars is set to true, with do_extractvars being false by default. This pull request ensures that the gefs_arch task will be created in the workflow regardless of the value of do_extractvars.

Resolves The GEFS workflow does not create a gefs_arch task #3151

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)

How has this been tested?

CI test on HERCULES and HERA

Checklist

WalterKolczynski-NOAA · 2025-01-08T04:33:45Z

It's coming back to me why we only had the archive task when extract_vars is on: the archive job doesn't do much for GEFS right now when extract_vars is off (well, now it may archive the expdir). If you look at parm/archive/master_gefs.yaml.j2, all of the non-expdir included lists are extracted ones. There is a parm/archive/gefs_arcdir.yaml.j2, but it doesn't seem to be included anywhere.

I think to complete this, we need the following:

The inclusion of the extracted files need to be hidden behind the DO_EXTRACTVARS flag (it should be failing right now if it is off)
The gefs_arcdir file needs to be included when extract vars is off

@DavidHuber-NOAA @EricSinsky-NOAA please let me know if I'm incorrect here.

DavidHuber-NOAA · 2025-01-08T13:08:30Z

@WalterKolczynski-NOAA Yes, I think you are correct. There would need to be an if-block around

global-workflow/parm/archive/master_gefs.yaml.j2

Lines 7 to 12 in f1afac9

    
           {% filter indent(width=4) %} 
        
               {% include "gefs_extracted_atmos.yaml.j2" %} 
        
               {% include "gefs_extracted_ocean.yaml.j2" %} 
        
               {% include "gefs_extracted_ice.yaml.j2" %} 
        
               {% include "gefs_extracted_wave.yaml.j2" %} 
        
           {% endfilter %}

EricSinsky-NOAA · 2025-01-08T13:58:44Z

To make things more simple, I believe we can remove the gefs_extracted* yaml files and only use gefs_arcdir.yaml.j2. The only place these gefs_extracted* yaml files are needed are in the feature/gefs_reforecast branch. I do not believe they are needed in develop if we use gefs_arcdir.yaml.j2.

…ando-NOAA/global-workflow into feature/add-gefs-arch

parm/archive/master_gefs.yaml.j2

AntonMFernando-NOAA · 2025-01-14T21:18:03Z

@WalterKolczynski-NOAA A CI test passed in HERCULES. @EricSinsky-NOAA Are you satisfied with the changes? If so we can go ahead and run CIs in the PR.

EricSinsky-NOAA

These changes look good. Thank you, @AntonMFernando-NOAA.

emcbot · 2025-01-16T12:05:05Z

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3211/RUNTESTS/COMROOT/C48_S2SWA_gefs_59fb602b/logs/2021032312/gefs_arch.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-01-16T12:05:13Z

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3211/RUNTESTS/EXPDIR/C48_S2SWA_gefs_59fb602b

emcbot · 2025-01-16T12:41:09Z

CI Failed on Hera in Build# 1
Built and ran in directory /scratch1/NCEPDEV/global/CI/3211


Experiment C96_S2SWA_gefs_replay_ics_59fb602b Completed 1 Cycles: *SUCCESS* at Thu Jan 16 10:38:46 UTC 2025
Experiment C48mx500_3DVarAOWCDA_59fb602b Completed 2 Cycles: *SUCCESS* at Thu Jan 16 10:44:50 UTC 2025
Experiment C48mx500_hybAOWCDA_59fb602b Completed 2 Cycles: *SUCCESS* at Thu Jan 16 10:50:55 UTC 2025
Experiment C48_ATM_59fb602b Completed 2 Cycles: *SUCCESS* at Thu Jan 16 10:50:56 UTC 2025
Experiment C96C48_hybatmaerosnowDA_59fb602b Completed 3 Cycles: *SUCCESS* at Thu Jan 16 12:04:27 UTC 2025
Experiment C48_S2SWA_gefs_59fb602b Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Jan 16 12:04:54 UTC 2025
Experiment C48_S2SWA_gefs_59fb602b Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3211/RUNTESTS/COMROOT/C48_S2SWA_gefs_59fb602b/logs/2021032312/gefs_arch.log
Experiment C96_atm3DVar_59fb602b Completed 3 Cycles: *SUCCESS* at Thu Jan 16 12:10:20 UTC 2025
Experiment C96C48_hybatmDA_59fb602b Completed 3 Cycles: *SUCCESS* at Thu Jan 16 12:10:57 UTC 2025
Experiment C48_S2SW_59fb602b Completed 2 Cycles: *SUCCESS* at Thu Jan 16 12:28:42 UTC 2025
Experiment C96C48_ufs_hybatmDA_59fb602b Completed 3 Cycles: *SUCCESS* at Thu Jan 16 12:40:55 UTC 2025

parm/archive/master_gefs.yaml.j2

…ando-NOAA/global-workflow into feature/add-gefs-arch

scripts/exglobal_archive.py

DavidHuber-NOAA · 2025-01-16T15:21:41Z

@AntonMFernando-NOAA the parm/archive/gefs_extractvars* YAMLs should also be deleted. These will only be retained in the gefs_reforecast branch (not develop).

DavidHuber-NOAA

Changes look good to me. Will wait for @EricSinsky-NOAA's approval before starting CI testing.

parm/archive/master_gefs.yaml.j2

emcbot · 2025-01-16T21:11:30Z

CI Passed on Hera in Build# 2
Built and ran in directory /scratch1/NCEPDEV/global/CI/3211


Experiment C48mx500_3DVarAOWCDA_c1b23a60 Completed 2 Cycles: *SUCCESS* at Thu Jan 16 18:58:29 UTC 2025
Experiment C48mx500_hybAOWCDA_c1b23a60 Completed 2 Cycles: *SUCCESS* at Thu Jan 16 19:04:47 UTC 2025
Experiment C48_ATM_c1b23a60 Completed 2 Cycles: *SUCCESS* at Thu Jan 16 19:10:48 UTC 2025
Experiment C96_S2SWA_gefs_replay_ics_c1b23a60 Completed 1 Cycles: *SUCCESS* at Thu Jan 16 19:10:49 UTC 2025
Experiment C96C48_hybatmDA_c1b23a60 Completed 3 Cycles: *SUCCESS* at Thu Jan 16 20:19:12 UTC 2025
Experiment C96_atm3DVar_c1b23a60 Completed 3 Cycles: *SUCCESS* at Thu Jan 16 20:25:34 UTC 2025
Experiment C48_S2SWA_gefs_c1b23a60 Completed 1 Cycles: *SUCCESS* at Thu Jan 16 20:26:22 UTC 2025
Experiment C96C48_hybatmaerosnowDA_c1b23a60 Completed 3 Cycles: *SUCCESS* at Thu Jan 16 20:31:48 UTC 2025
Experiment C48_S2SW_c1b23a60 Completed 2 Cycles: *SUCCESS* at Thu Jan 16 20:57:59 UTC 2025
Experiment C96C48_ufs_hybatmDA_c1b23a60 Completed 3 Cycles: *SUCCESS* at Thu Jan 16 21:11:18 UTC 2025

modified gefs.py

26e42ea

AntonMFernando-NOAA force-pushed the feature/add-gefs-arch branch from 995f3a3 to 26e42ea Compare January 7, 2025 22:50

AntonMFernando-NOAA and others added 6 commits January 7, 2025 17:56

modified workflow\applications\gefs.py

72af87f

updated workflow\rocoto\gefs_tasks.py

8a880e2

updated workflow\rocoto\gefs_tasks.py

6f47e5c

modified workflow\applications\gefs.py

c5d5afd

updated workflow\rocoto\gefs_tasks.py

37fd408

Merge branch 'develop' into feature/add-gefs-arch

5f6938c

AntonMFernando-NOAA marked this pull request as ready for review January 8, 2025 03:48

AntonMFernando-NOAA requested review from WalterKolczynski-NOAA, aerorahul and DavidHuber-NOAA as code owners January 8, 2025 03:48

AntonMFernando-NOAA self-assigned this Jan 8, 2025

AntonMFernando-NOAA requested a review from EricSinsky-NOAA January 8, 2025 05:09

AntonMFernando-NOAA and others added 6 commits January 8, 2025 14:12

Merge branch 'develop' into feature/add-gefs-arch

fb0ba8b

Merge branch 'NOAA-EMC:develop' into feature/add-gefs-arch

8ea285d

Merge branch 'develop' into feature/add-gefs-arch

ad57862

modify parm\archive\master_gefs.yaml.j2

e5813e3

Merge branch 'feature/add-gefs-arch' of https://github.com/AntonMFern…

613a53e

…ando-NOAA/global-workflow into feature/add-gefs-arch

modified DO_EXTRACTVARS

b2b1823

WalterKolczynski-NOAA requested changes Jan 14, 2025

View reviewed changes

parm/archive/master_gefs.yaml.j2 Outdated Show resolved Hide resolved

AntonMFernando-NOAA and others added 2 commits January 14, 2025 12:37

edited parm\archive\master_gefs.yaml.j2

24813e7

Merge branch 'develop' into feature/add-gefs-arch

5eed515

WalterKolczynski-NOAA previously approved these changes Jan 14, 2025

View reviewed changes

Merge branch 'develop' into feature/add-gefs-arch

638962d

EricSinsky-NOAA previously approved these changes Jan 15, 2025

View reviewed changes

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Jan 16, 2025

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Jan 16, 2025

Merge branch 'develop' into feature/add-gefs-arch

3e52033

DavidHuber-NOAA reviewed Jan 16, 2025

View reviewed changes

parm/archive/master_gefs.yaml.j2 Outdated Show resolved Hide resolved

AntonMFernando-NOAA added 2 commits January 16, 2025 10:13

modified parm\archive\master_gefs.yaml.j2

5bef72a

Merge branch 'feature/add-gefs-arch' of https://github.com/AntonMFern…

bb64c92

…ando-NOAA/global-workflow into feature/add-gefs-arch

AntonMFernando-NOAA dismissed WalterKolczynski-NOAA’s stale review via bb64c92 January 16, 2025 15:13

DavidHuber-NOAA reviewed Jan 16, 2025

View reviewed changes

scripts/exglobal_archive.py Outdated Show resolved Hide resolved

modified scripts\exglobal_archive.py

c73dda0

deleted gefs_extracted_*vars*.yaml.j2

c1b23a6

DavidHuber-NOAA requested a review from EricSinsky-NOAA January 16, 2025 15:33

DavidHuber-NOAA approved these changes Jan 16, 2025

View reviewed changes

EricSinsky-NOAA requested changes Jan 16, 2025

View reviewed changes

parm/archive/master_gefs.yaml.j2 Show resolved Hide resolved

EricSinsky-NOAA approved these changes Jan 16, 2025

View reviewed changes

DavidHuber-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Jan 16, 2025

WalterKolczynski-NOAA merged commit 3a5af9a into NOAA-EMC:develop Jan 16, 2025
10 of 11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding a gefs_arch task to GEFS workflow #3211

Adding a gefs_arch task to GEFS workflow #3211

AntonMFernando-NOAA commented Jan 7, 2025 •

edited

Loading

WalterKolczynski-NOAA commented Jan 8, 2025

DavidHuber-NOAA commented Jan 8, 2025

EricSinsky-NOAA commented Jan 8, 2025

AntonMFernando-NOAA commented Jan 14, 2025

EricSinsky-NOAA left a comment

emcbot commented Jan 16, 2025

emcbot commented Jan 16, 2025

emcbot commented Jan 16, 2025

DavidHuber-NOAA commented Jan 16, 2025

DavidHuber-NOAA left a comment

emcbot commented Jan 16, 2025

Adding a gefs_arch task to GEFS workflow #3211

Adding a gefs_arch task to GEFS workflow #3211

Conversation

AntonMFernando-NOAA commented Jan 7, 2025 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

WalterKolczynski-NOAA commented Jan 8, 2025

DavidHuber-NOAA commented Jan 8, 2025

EricSinsky-NOAA commented Jan 8, 2025

AntonMFernando-NOAA commented Jan 14, 2025

EricSinsky-NOAA left a comment

Choose a reason for hiding this comment

emcbot commented Jan 16, 2025

emcbot commented Jan 16, 2025

emcbot commented Jan 16, 2025

DavidHuber-NOAA commented Jan 16, 2025

DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

emcbot commented Jan 16, 2025

AntonMFernando-NOAA commented Jan 7, 2025 •

edited

Loading