Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove anl from post task and replace with new task #1979

Merged
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
4c8e147
Updated Rocoto driver and removed anl dependency.
henrywinterbottom-wxdev Oct 24, 2023
f4867a7
Added new Rocoto script for postanl application.
henrywinterbottom-wxdev Oct 24, 2023
78229a9
Added postanl job/task.
henrywinterbottom-wxdev Oct 24, 2023
7be0b1c
Updated GFS tasks.
henrywinterbottom-wxdev Oct 24, 2023
e24b29a
Turning off tests for debugging purposes.
henrywinterbottom-wxdev Oct 25, 2023
93ef711
Additional cubed-sphere resolution support.
henrywinterbottom-wxdev Oct 25, 2023
04ab5d9
Reverting to develop actions.
henrywinterbottom-wxdev Oct 25, 2023
a849c9e
Updated/corrected order of post task applications.
henrywinterbottom-wxdev Oct 25, 2023
096808b
Addressed reviewer comments.
henrywinterbottom-wxdev Oct 25, 2023
445ed50
Updated as per reviewer comment.
henrywinterbottom-wxdev Oct 26, 2023
1e47906
Removing script due to redundancy.
henrywinterbottom-wxdev Oct 27, 2023
4bd36e8
Created link to post.sh in order to remove redundancy.
henrywinterbottom-wxdev Oct 27, 2023
d6759a7
Merge branch 'NOAA-EMC:develop' into feature/postanl
HenryRWinterbottom Oct 27, 2023
0efaad5
Merge branch 'feature/postanl' of https://github.com/HenryWinterbotto…
henrywinterbottom-wxdev Oct 27, 2023
a3eec68
Updates requested by reviewers.
henrywinterbottom-wxdev Oct 27, 2023
0210554
Update workflow/rocoto/gfs_tasks.py
HenryRWinterbottom Oct 27, 2023
5d77c57
Reverting change.
henrywinterbottom-wxdev Oct 27, 2023
e472620
Merge branch 'NOAA-EMC:develop' into feature/postanl
HenryRWinterbottom Oct 30, 2023
91efd43
Removed hack/workaround.
henrywinterbottom-wxdev Oct 30, 2023
c48cd43
Docstring update.
henrywinterbottom-wxdev Oct 30, 2023
688a641
Removed all references to add_anl_to_post.
henrywinterbottom-wxdev Oct 30, 2023
95725ac
Removed from method as per reviewer request.
henrywinterbottom-wxdev Oct 30, 2023
bafe0b1
Merge branch 'NOAA-EMC:develop' into feature/postanl
HenryRWinterbottom Oct 31, 2023
fef6cb4
Corrected bug related to path dependency.
henrywinterbottom-wxdev Oct 31, 2023
47b8668
Merge branch 'NOAA-EMC:develop' into feature/postanl
HenryRWinterbottom Oct 31, 2023
4f79827
Merge branch 'NOAA-EMC:develop' into feature/postanl
HenryRWinterbottom Nov 1, 2023
fa9ada5
Update gfs_tasks.py
HenryRWinterbottom Nov 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 1 addition & 5 deletions jobs/rocoto/post.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,7 @@ status=$?
export job="post"
export jobid="${job}.$$"

if [ ${FHRGRP} = 'anl' ]; then
fhrlst="anl"
else
fhrlst=$(echo ${FHRLST} | sed -e 's/_/ /g; s/f/ /g; s/,/ /g')
fi
fhrlst=$(echo ${FHRLST} | sed -e 's/_/ /g; s/f/ /g; s/,/ /g')

#---------------------------------------------------------------
for fhr in ${fhrlst}; do
Expand Down
29 changes: 29 additions & 0 deletions jobs/rocoto/postanl.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#! /usr/bin/env bash
WalterKolczynski-NOAA marked this conversation as resolved.
Show resolved Hide resolved

source "${HOMEgfs}/ush/preamble.sh"

###############################################################
## NCEP post driver script
## FHRGRP : forecast hour group to post-process (e.g. 0)
## FHRLST : forecast hourlist to be post-process (e.g. anl)
###############################################################

# Source FV3GFS workflow modules
. ${HOMEgfs}/ush/load_fv3gfs_modules.sh
Fixed Show fixed Hide fixed
status=$?
[[ ${status} -ne 0 ]] && exit ${status}
Fixed Show fixed Hide fixed

export job="postanl"
export jobid="${job}.$$"

fhrlst=$(echo ${FHRLST} | sed -e 's/_/ /g; s/f/ /g; s/,/ /g')
Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed

#---------------------------------------------------------------
for fhr in ${fhrlst}; do
export post_times=${fhr}
${HOMEgfs}/jobs/JGLOBAL_ATMOS_POST
Fixed Show fixed Hide fixed
status=$?
[[ ${status} -ne 0 ]] && exit ${status}
Fixed Show fixed Hide fixed
done

exit 0
2 changes: 1 addition & 1 deletion workflow/applications/gfs_cycled.py
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ def get_task_names(self):
"""

gdas_gfs_common_tasks_before_fcst = ['prep']
gdas_gfs_common_tasks_after_fcst = ['post']
gdas_gfs_common_tasks_after_fcst = ['postanl', 'post']
# if self.do_ocean: # TODO: uncomment when ocnpost is fixed in cycled mode
# gdas_gfs_common_tasks_after_fcst += ['ocnpost']
gdas_gfs_common_tasks_after_fcst += ['vrfy']
Expand Down
2 changes: 1 addition & 1 deletion workflow/hosts/awspw.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,4 @@ LOCALARCH: 'NO'
ATARDIR: '/NCEPDEV/${HPSS_PROJECT}/1year/${USER}/${machine}/scratch/${PSLOT}' # TODO: This will not yet work from AWS.
MAKE_NSSTBUFR: 'NO'
MAKE_ACFTBUFR: 'NO'
SUPPORTED_RESOLUTIONS: ['C48'] # TODO: Test and support all cubed-sphere resolutions.
SUPPORTED_RESOLUTIONS: ['C48', 'C96'] # TODO: Test and support all cubed-sphere resolutions.
23 changes: 20 additions & 3 deletions workflow/rocoto/gfs_tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -554,16 +554,33 @@ def _fcst_cycled(self):
def post(self):
add_anl_to_post = False
if self.app_config.mode in ['cycled']:
add_anl_to_post = True
add_anl_to_post = False
WalterKolczynski-NOAA marked this conversation as resolved.
Show resolved Hide resolved

return self._post_task('post', add_anl_to_post=add_anl_to_post)

def postanl(self):
aerorahul marked this conversation as resolved.
Show resolved Hide resolved
postenvars = self.envars.copy()
for key, value in postenvar_dict.items():
postenvars.append(rocoto.create_envar(name=key, value=str(value)))

deps = []
atm_hist_path = self._template_to_rocoto_cycstring(self._base["COM_ATMOS_HISTORY_TMPL"])
data = f'{atm_hist_path}/{self.cdump}.t@Hz.atm.loganl.txt'
dep_dict = {'type': 'data', 'data': data}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)
resources = self.get_resource('post')
task = create_wf_task('postanl', resources, cdump=self.cdump, envar=postenvars, dependency=dependencies,
cycledef=self.cdump)

return task

def ocnpost(self):
if self.app_config.mode in ['forecast-only']: # TODO: fix ocnpost in cycled mode
return self._post_task('ocnpost', add_anl_to_post=False)

def _post_task(self, task_name, add_anl_to_post=False):
if task_name not in ['post', 'ocnpost']:
if task_name not in ['postanl', 'post', 'ocnpost']:
HenryRWinterbottom marked this conversation as resolved.
Show resolved Hide resolved
raise KeyError(f'Invalid post-processing task: {task_name}')

if task_name in ['ocnpost']:
Expand Down Expand Up @@ -620,7 +637,7 @@ def _get_postgroups(cdump, config, add_anl=False):
postenvars.append(rocoto.create_envar(name=key, value=str(value)))

varname1, varname2, varname3 = 'grp', 'dep', 'lst'
varval1, varval2, varval3 = _get_postgroups(self.cdump, self._configs[task_name], add_anl=add_anl_to_post)
varval1, varval2, varval3 = _get_postgroups(self.cdump, self._configs[task_name], add_anl=False)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
varval1, varval2, varval3 = _get_postgroups(self.cdump, self._configs[task_name], add_anl=False)
varval1, varval2, varval3 = _get_postgroups(self.cdump, self._configs[task_name])

vardict = {varname2: varval2, varname3: varval3}

cycledef = 'gdas_half,gdas' if self.cdump in ['gdas'] else self.cdump
Expand Down