SFS crashed at MOM6

[bluefinweiwei]

Hello!

I used the latest "develop" branch to run cold-start SFS with the UFS-replay data as IC, but the model kept crashing. The problem seems to be associated with MOM6. Your timely help is much appreciated. Thanks. - Weiwei (attn: @shansun6)

This is how I set up my experiment:
./setup_expt.py sfs forecast-only --app S2S --pslot test --idate 2018050100 --edate 2018050100 --resdetatmos 96 --resdetocean 1.0 --configdir ...

Below are relevant warning/error messages.
30: WARNING from PE 0: MEKE_init: Initializing MEKE with a local equilibrium balance.
30:
30: NOTE from PE 0: MOM_restart: Unable to find MEKE_Le queried by pointer, probably because of the suspect comparison of pointers by ASSOCIATED.
30: NOTE from PE 0: MOM_restart: Unknown restart variable MEKE_Le queried for initialization.
30:
30: WARNING from PE 0: MEKE_init: Initializing MEKE with a local equilibrium balance.

WARNING from PE 19: btstep: eta has dropped below bathyT:

42: Image PC Routine Line Source
42: gfs_model.x 0000000006484087 Unknown Unknown Unknown
42: gfs_model.x 000000000569C159 mpp_mod_mp_mpp_er 72 mpp_util_mpi.inc
42: gfs_model.x 00000000049D9080 mom_error_handler 191 MOM_error_handler.F90
42: gfs_model.x 00000000046C82AC mom_mp_extract_su 4060 MOM.F90
42: gfs_model.x 00000000046B2B23 mom_mp_step_mom_ 1053 MOM.F90
42: gfs_model.x 00000000046860CD mom_ocean_model_n 630 mom_ocean_model_nuopc.F90
42: gfs_model.x 000000000464E7AF mom_cap_mod_mp_mo 1770 mom_cap.F90

[jiandewang]

you can try -apps SFS instaed of S2S.
I have never run 1x1 ocean using S2S under g-w but Neil's group ran C96+1x1 ocean for SFS.
BTW, can you show me your ocean IC and MOM_input ?

[bluefinweiwei]

Thanks a lot @jiandewang

• I tried -apps SFS but was told: "error: argument --app: invalid choice: 'SFS' (choose from 'ATM', 'ATMA', 'ATMW', 'S2S', 'S2SA', 'S2SW', 'S2SWA')". I believe SFS is not an option for --app
• Please find my ocean IC at HPSS: /NCEPDEV/emc-marine/2year/Neil.Barton/ICS//HR4/C96mx100/2018050100.tar (I also tried other years - same error).
• Sorry I don't know what MOM_input is. Is it the same thing as ocean IC?

[aerorahul]

I think the executable is not correct for this model configuration.
I would have expected the sfs_model.x executable to be used. Instead I see gfs_model.x in the runtime log.
SFS requires the model to be built with SFS options, and the resulting executable is sfs_model.x. gfs_model.x is the result of building the UFS-weather-model with GFS options.

[bluefinweiwei]

@aerorahul Thanks for catching that. Since I've tried different executables, I'm afraid I pasted the GFS one. Sorry about that. Please find the run/error log with sfs_model.x as executable, showing the same error.

42: sfs_model.x 0000000006483F77 Unknown Unknown Unknown
42: sfs_model.x 000000000569C049 mpp_mod_mp_mpp_er 72 mpp_util_mpi.inc
42: sfs_model.x 00000000049D9080 mom_error_handler 191 MOM_error_handler.F90
42: sfs_model.x 00000000046C82AC mom_mp_extract_su 4060 MOM.F90
42: sfs_model.x 00000000046B2B23 mom_mp_step_mom_ 1053 MOM.F90
42: sfs_model.x 00000000046860CD mom_ocean_model_n 630 mom_ocean_model_nuopc.F90
42: sfs_model.x 000000000464E7AF mom_cap_mod_mp_mo 1770 mom_cap.F90

[jiandewang]

MOM_input is inside "fcst.xxx" run dir. Also we need to see more info. on your error file. Try to paste the section started from "WARNING......"

[bluefinweiwei]

Okay, then MOM_input is exactly the same as the ocean IC provided by UFS-replay as that file was linked from where I saved the UFS-replay (i.e., $ICSDIR).

I'm attaching my log file for your use. I ran the experiment on Hera.
Feel free to let me know if you need any other information. Thank you @jiandewang .
sfs_fcst_mem000_seg0.log

[jiandewang]

MOM_input is a ASCII file that you set up parameters, it shouldn't be the same as IC file which is netcdf file

[jiandewang]

check your input.nml, in the end, there is one line "input_filename"
is it = "n" or "r" ?

[bluefinweiwei]

@jiandewang Gotcha, I found the MOM_input (attached I had to change it to .txt to attach it) and checked the input_filename in input.nml, where "input_filename = r"
MOM_input.txt

[jiandewang]

did you make any changes to MOM_input after you checked out the g-w ?

[bluefinweiwei]

Nope, made no changes, as you know I just found out where the file was located :-)

…

On Mon, Apr 7, 2025, 12:24 PM jiandewang ***@***.***> wrote: did you make any changes to MOM_input after you checked out the g-w ? — Reply to this email directly, view it on GitHub <#3552 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKEOR2TYRHOIJWPTQXXVYXL2YK7GHAVCNFSM6AAAAAB2SKL6VOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENZVGQ4DCMY> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[jiandewang]

I checked your run log, it ran but died for science reason
WARNING from PE 17: Extreme surface sfc_state detected: i= 154 j= 241 lon=-146.500 lat= 58.226 x=-146.500 y= 58.226 D= 4.6003E+03 SSH= 9.0917E+01

here you have 9m of ssh.

Are you sure you didn't mess up on IC date ? All components shall use the IC for the same date.
If you think this is not the case, I can lead you to Yangxing who ran this IC with the setting as yours

[YangxingZheng-NOAA]

I have run SFS baseline experiment (hydro, mono along with 10 ensemble members) with Replay ICs for 2018050100 cycle on Hera. It is successfully completed.

[bluefinweiwei]

Thanks @YangxingZheng-NOAA. Glad to know there’s a working version. We used the default setup from the top branch, but with the IC data we used, with hydro + mono and MOM6_RESTART* =“n”. The model still crashed with the following error msg. Perhaps you could you point me and @shansun6 to where/how you ran your experiment on Hera?
23: forrtl: severe (17): syntax error in NAMELIST input, unit -5, file Internal Formatted NML Read, line -1, position 18
23: Image PC Routine Line Source
23: sfs_model.x 00000000051946A8 Unknown Unknown Unknown
23: sfs_model.x 00000000051D520E Unknown Unknown Unknown
23: sfs_model.x 00000000051D2AB0 Unknown Unknown Unknown
23: sfs_model.x 00000000026216B1 gfs_typedefs_mp_c 4251 GFS_typedefs.F90

[YangxingZheng-NOAA]

Hi, below is some dir information for my SFS run on Hera:
Global-Workflow dir = /scratch1/NCEPDEV/climate/Yangxing.Zheng/SFS/global-workflow
EXPDIR=/scratch1/NCEPDEV/climate/Yangxing.Zheng/SFS/work/EXPDIR/sfsc96mx100_replayICs
COMROOT=/scratch1/NCEPDEV/climate/Yangxing.Zheng/SFS/work/COMROOT/sfsc96mx100_replayICs

If you run experiments on Hera, there are replay ICs available on Hera:
/scratch2/NCEPDEV/stmp1/Neil.Barton/ICs/HR4/C96mx100
You can directly use the ICs.

[YangxingZheng-NOAA]

How to run SFS on Hera for C96mx100:
Download and Compile Code:
git clone --recursive https://github.com/NeilBarton-NOAA/global-workflow.git SFS
cd SFS
git checkout SFS_baseline_hydro
git checkout --recurse-submodules SFS_baseline_hydro
cd sorc
sh build_all.sh -w -y -j 8
sh link_workflow.sh

Below is a referenced procedure to follow to run SFS:
(1) User defined variables:
HOMEgfs=”location of code”
machine=”machine used (e.g., hera, wcoss2, orion, hercules)”
HPC_ACCOUNT=”cpu account for machine”
RUNTESTS=”location to run model (somewhere on work/scratch directory)”
pslot=”user defined experiment name”

(2) Set up Experiment Directories:
YAML=”${HOMEgfs}/ci/cases/sfs/C96mx100_S2S.yaml”
export TOPICDIR=
/scratch2/NCEPDEV/stmp1/Neil.Barton/ICs (on hera)
export HPC_ACCOUNT RUNTESTS pslot
source ${HOMEgfs}/workflow/gw_setup.sh
${HOMEgfs}/workflow/create_experiment.py --yaml "${YAML}"

(3) Add rocotorun command to crontab:
EXPDIR=${RUNTESTS}/EXPDIR/${pslot}
xml_file=${EXPDIR}/${pslot}.xml
db_file=${EXPDIR}/${pslot}.db
cron_file=${EXPDIR}/${pslot}.crontab
rocotorun -d ${db_file} -w ${xml_file}
crontab -l | cat - ${cron_file} | crontab -

Other Notes for SFS:
In the XML file:
Increase “cyclethrottle” (25) to run more cycle points/dates at a time
Increase “taskthrottle” (250) to run more jobs at a time
Manually put in dates under <!-- Define the cycles →
e.g., 200305010000 200305010000 24:00:00
After a set of runs, remove the .db files in the EXPDIR, add additional dates, and perform a rocotorun

[bluefinweiwei]

@YangxingZheng-NOAA (and @jiandewang) Many thanks for the comprehensive instructions. Appreciate it.

I noticed you used Neil’s repo/branch, which has always worked for us. However, what has been troubling us is the develop branch at the NOAA-EMC/global-workflow repo.
git clone --recursive https://github.com/NOAA-EMC/global-workflow.git

I guess we can switch to Neil’s repo/branch for our work. And it would be very helpful if someone could help address issues with the develop branch at the NOAA-EMC/global-workflow repo, as some community users may not know about Neil’s repo/branch.