Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Additional fixes for downstream jobs #3187

Merged
merged 37 commits into from
Jan 14, 2025

Conversation

DavidHuber-NOAA
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA commented Dec 20, 2024

Description

This fixes a number of issues with the postsnd and arch jobs discovered during testing of HR4.

Resolves #3019

Type of change

  • Bug fix (fixes something broken)

Change characteristics

How has this been tested?

  • Extended test on Hera
  • Extended test on WCOSS2

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
    • Portions of this change are covered by existing CI, but others are not (e.g. archiving, which is disabled on WCOSS2)
    • Updates will need to be made to run the AWIPS jobs in the existing C96_atm3DVar_extended test

* emc/develop:
  Change orog gravity wave drag scheme for grid sizes less than 10km (NOAA-EMC#3175)
  Switch snow DA to use 2DVar for deterministic and ensemble mean (NOAA-EMC#3163)
  Update compression options for GEFS history files (NOAA-EMC#3184)
@DavidHuber-NOAA
Copy link
Contributor Author

All post jobs now run for the C96_atm3dvar_extended test when DO_AWIPS=YES on WCOSS2. I have thus enabled AWIPS for this test to ensure the AWIPS jobs and linked downstream jobs are always run to avoid lapses in support.

The AWIPS jobs still cannot be enabled on other machines due to the fbwinds executable which currently only runs on WCOSS2. That will be an issue for another time.

Also, during this testing, I found that the MOS jobs, when enabled, do not run on WCOSS2. I opened issue #3196 to look into this, but I believe the MOS package will need an overhaul to work with the new COM structure.

Marking this PR ready for review.

Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Just one observation. Approve.

workflow/applications/gfs_cycled.py Show resolved Hide resolved
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.

aerorahul
aerorahul previously approved these changes Jan 3, 2025
@aerorahul
Copy link
Contributor

@WalterKolczynski-NOAA
Can you please run the ci on wcoss2?

@DavidHuber-NOAA DavidHuber-NOAA added CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Jan 6, 2025
@DavidHuber-NOAA
Copy link
Contributor Author

Ah, right, I have a gfs-utils PR to open... Opened NOAA-EMC/gfs-utils#90.

@DavidHuber-NOAA DavidHuber-NOAA added CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS and removed CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed labels Jan 13, 2025
@emcbot emcbot added the CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS label Jan 13, 2025
@KateFriedman-NOAA KateFriedman-NOAA removed the CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS label Jan 13, 2025
@emcbot emcbot added CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Jan 13, 2025
@emcbot
Copy link

emcbot commented Jan 13, 2025

CI Tests set up to run in /lfs/h2/emc/ptmp/emc.global/PR/PR_3187/RUNTESTS on WCOSS

@DavidHuber-NOAA
Copy link
Contributor Author

It looks like the gdas_gempakmetancdc job failed on WCOSS2. I'm investigating.

@DavidHuber-NOAA DavidHuber-NOAA added CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed and removed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels Jan 14, 2025
@KateFriedman-NOAA KateFriedman-NOAA added CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed labels Jan 14, 2025
@KateFriedman-NOAA
Copy link
Member

KateFriedman-NOAA commented Jan 14, 2025

Manually pulled in fixes to CI clone on WCOSS2 and reran failed jobs. Failing CI test completed successfully on WCOSS2, as did others.

Tue Jan 14 16:46:41 UTC 2025
******** C48_ATM_3187 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Jan 13 2025 19:55:31    Jan 13 2025 21:05:25
202103231800        Done    Jan 13 2025 19:55:31    Jan 13 2025 21:15:34

******** C48_S2SW_3187 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Jan 13 2025 19:55:34    Jan 13 2025 21:40:34
202103231800        Done    Jan 13 2025 19:55:34    Jan 13 2025 21:50:29

******** C48_S2SWA_gefs_3187 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Jan 13 2025 19:55:39    Jan 13 2025 22:50:36

******** C96_atm3DVar_extended_3187 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Jan 13 2025 19:55:43    Jan 13 2025 20:15:47
202112210000        Done    Jan 13 2025 19:55:43    Jan 14 2025 16:30:40
202112210600        Done    Jan 13 2025 19:55:43    Jan 14 2025 01:10:39
202112211200        Done    Jan 13 2025 20:20:54    Jan 14 2025 16:35:34
202112211800        Done    Jan 14 2025 01:15:37    Jan 14 2025 06:05:49

******** C96C48_hybatmDA_3187 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Jan 13 2025 19:55:46    Jan 13 2025 20:15:51
202112210000        Done    Jan 13 2025 19:55:46    Jan 13 2025 22:25:49
202112210600        Done    Jan 13 2025 19:55:46    Jan 13 2025 22:15:52

******** C96_S2SWA_gefs_replay_ics_3187 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202011010000        Done    Jan 13 2025 19:55:49    Jan 13 2025 20:35:54

@DavidHuber-NOAA
Copy link
Contributor Author

I added some fixes for the gfs_waveawipsbulls job, but I was unable to test the job completely due to missing buoy data that should be generated by the gfs_wavepostpnt job. This is documented in #3226. Once this issue is resolved in a future PR, the new C48_S2SW_extended test can be enabled on WCOSS2.

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit aea82a8 into NOAA-EMC:develop Jan 14, 2025
5 checks passed
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this pull request Jan 15, 2025
…kf_sfc_update_com_in_out

* upstream/develop:
  Resolve bug with LMOD_TMOD_FIND_FIRST setting affecting build on WCOSS2 (NOAA-EMC#3229)
  Reinstate product groups (NOAA-EMC#3208)
  Additional fixes for downstream jobs (NOAA-EMC#3187)
  Turn IAU off during staging job for cold start experiments (NOAA-EMC#3215)
  Update the gdas.cd hash and enable GDASApp to run on WCOSS2 (NOAA-EMC#3220)
  Update upload-artifact to v4 (NOAA-EMC#3216)
  Prevent duplicate case generation in generate_workflows.sh (NOAA-EMC#3217)
  Update g-w to cycle with C1152 ATM (NOAA-EMC#3206)
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this pull request Jan 15, 2025
…kf_sfc_update_com_in_out

* upstream/develop:
  Resolve bug with LMOD_TMOD_FIND_FIRST setting affecting build on WCOSS2 (NOAA-EMC#3229)
  Reinstate product groups (NOAA-EMC#3208)
  Additional fixes for downstream jobs (NOAA-EMC#3187)
  Turn IAU off during staging job for cold start experiments (NOAA-EMC#3215)
  Update the gdas.cd hash and enable GDASApp to run on WCOSS2 (NOAA-EMC#3220)
  Update upload-artifact to v4 (NOAA-EMC#3216)
  Prevent duplicate case generation in generate_workflows.sh (NOAA-EMC#3217)
  Update g-w to cycle with C1152 ATM (NOAA-EMC#3206)
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this pull request Jan 15, 2025
…kf_sfc_update_com_in_out

* upstream/develop:
  Resolve bug with LMOD_TMOD_FIND_FIRST setting affecting build on WCOSS2 (NOAA-EMC#3229)
  Reinstate product groups (NOAA-EMC#3208)
  Additional fixes for downstream jobs (NOAA-EMC#3187)
  Turn IAU off during staging job for cold start experiments (NOAA-EMC#3215)
  Update the gdas.cd hash and enable GDASApp to run on WCOSS2 (NOAA-EMC#3220)
  Update upload-artifact to v4 (NOAA-EMC#3216)
  Prevent duplicate case generation in generate_workflows.sh (NOAA-EMC#3217)
  Update g-w to cycle with C1152 ATM (NOAA-EMC#3206)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Post processing errors resulting from GFS HR4 test run
10 participants