Stopping of a LIS-DA run (related with state perturbation and analysis increments)

[DonghyuckYoon]

Hi, I am Donghyuck Yoon, a Ph.D. student at UNIST, South Korea.
I tried to assimilate SMAP L3 data during 1-year open-loop LIS run (2016).
But, the run was canceled after running a certain time, which is the same with the state perturbation frequency.
For example, the run provided as the attachment file was stopped after 1 hour, and it is the same with state perturbation frequency in the configure file.
The canceled time can be changed by adjusting the state perturbation frequency, but other frequencies (forcing perturbation and observation perturbation frequencies) didn't affect the break time.
Also, I found that my LIS run stopped after a 1-hour run if the "Data assimilation exclude analysis increments" option was set to 0, regardless of state perturbation frequency. If the option was set to 1, then it stopped after the state perturbation frequency.
So, are there any reasons for the stopping of a LIS run?
I attached my LIS configure and log.
Always welcome your comments.
Thanks,
Donghyuck
LISevaluation.zip

[dmocko]

Hi Donghyuck,

Is there any error message at all in the job's standard or error output (or to the screen)?

Also, please check the end of all lislog files, not just the .0000 file. One of the processors may show the error.

David

[dmocko]

Also, this is probably not the cause of your crash, but you should consider having the template open water timestep be the same as the LSM timestep. This is what you currently have:

Template open water timestep: "10mn"
Noah.3.3 model timestep: "15mn"

[DonghyuckYoon]

David,
Thank you for your reply.
I tried LIS run by using a single CPU, but there had the same result (suddenly stopped, and no error messages in the log file).
This is the error output on the screen.
forrtl: error (65): floating invalid
Image PC Routine Line Source
LIS 00000000014085FE Unknown Unknown Unknown
libpthread-2.17.s 00002AD0803A1370 Unknown Unknown Unknown
LIS 0000000000DF1155 module_sf_noah33l 3978 module_sf_noah33lsm.F90
LIS 0000000000DE8991 module_sf_noah33l 667 module_sf_noah33lsm.F90
LIS 0000000000FBF0E0 noah33_main_ 740 noah33_main.F90
LIS 000000000065E897 lis_lsmmod_mp_lis 552 LIS_lsmMod.F90
LIS 00000000006F82AC lis_surfacemodelm 246 LIS_surfaceModelMod.F90
LIS 00000000012ABF59 retrospective_run 196 retrospective_runMod.F90
LIS 0000000000D32634 MAIN__ 68 lisdrv.F90
LIS 000000000041175E Unknown Unknown Unknown
libc-2.17.so 00002AD0808D1B35 __libc_start_main Unknown Unknown
LIS 0000000000411669 Unknown Unknown Unknown
Thanks,
Donghyuck

[dmocko]

In the LISF public-7.3 release, I believe that line 3978 of LISF/lis/surfacemodels/noah.3.3/module_sf_noah33lsm.F90 is:
THKS = (THKQTZ ** QZ)* (THKO ** (1. - QZ))

QZ is the quartz, and you likely have a grid point where the QZ is zero, causing the floating invalid.

Can you go into LISF/lis/make, run "gmake realclean", then run the configure script again - this time choosing "-1" for the compilation option for debug mode, then run again, and copy the error message? It will give a more exact error message (although it may be the same as above).

Also, can you share your SOILPARM.TBL file? You may be trying to use a file that does not match up with the LIS-Noah-3.3 code.

Finally, please be sure that all of your LANDMASK=1 grid boxes in your lis_input file have a non-water TEXTURE index value. You may be trying to run a grid point without a defined soil TEXTURE type, so the quartz value is not defined from the SOILPARM.TBL file.

David

[DonghyuckYoon]

David,

Great thanks for your detailed reply.
I'll try your suggestions and report results soon.
First, please check the SOILPARM.TBL file.
Donghyuck
SOILPARM.zip

[DonghyuckYoon]

David,

I tried LIS run by using debug mode, but, there were almost the same messages on the screen, and log file.
Please find the below outputs.
Thanks,
Donghyuck

forrtl: error (65): floating invalid
Image PC Routine Line Source
LIS 00000000022B82FE Unknown Unknown Unknown
libpthread-2.17.s 00002B76AC7DF370 Unknown Unknown Unknown
LIS 00000000016A6120 module_sf_noah33l 3978 module_sf_noah33lsm.F90
LIS 0000000001698F83 module_sf_noah33l 667 module_sf_noah33lsm.F90
LIS 00000000019A76EF noah33_main_ 740 noah33_main.F90
LIS 000000000089F1B8 lsmrun_ 474 LIS_lsm_FTable.c
LIS 000000000089320E lis_lsmmod_mp_lis 552 LIS_lsmMod.F90
LIS 00000000009ACFE5 lis_surfacemodelm 246 LIS_surfaceModelMod.F90
LIS 000000000202F6A6 retrospective_run 196 retrospective_runMod.F90
LIS 000000000098BD02 lisrun_ 200 LIS_runmode_FTable.c
LIS 00000000015BFD13 MAIN__ 68 lisdrv.F90
LIS 00000000004114DE Unknown Unknown Unknown
libc-2.17.so 00002B76ACD0FB35 __libc_start_main Unknown Unknown
LIS 00000000004113E9 Unknown Unknown Unknown

[DonghyuckYoon]

David,
Gladly, the LIS normally worked when I turned off the open water model.
I think it related to your comment (mismatch of water and non-water grid).
Thanks for your reply.
Donghyuck

[DonghyuckYoon]

Moreover, I have the plan to use other soil moisture retrieval instead of SMAP.
I believe ASCAT data is widely used to assimilate soil moisture in the LIS framework, so, if you don't mind, would you please let me know the URL that can download ASCAT data? (especially format for LIS input)
By the way, what is the difference between ASCAT and SMOPS soil moisture data?
(According to NOAA, SMOPS combines various soil moisture retrievals from multiple satellite sensors, which include GPM, SMAP, GCOM-W1, SMOS, Metop-A, and Metop-B ㅡ no ASCAT. But, there is "SMOPS-ASCAT data assimilation test case" on the LIS webpage, so I am confusing.)
Thanks,
Donghyuck

[dmocko]

Donghyuck,

Great - glad that it works for you now. I can confirm that you are using the correct SOILPARM.TBL file for LIS-Noah-3.3.

SMOPS has a blended product, as well as individual layers for each sensor. ASCAT is one of the layers in SMOPS, so that is the "SMOPS-ASCAT" product that we use in LIS. See the right-hand side of this page: https://www.ospo.noaa.gov/Products/land/smops/index.html

The format of the SMOPS-ASCAT data that works in LIS is GRIB-2. The filename convention changes starting on 05 Oct 2017.
Before then the filename is: smops_d20171004_s180000_cness.gr2
After, the file name is: NPR_SMOPS_CMAP_D2017100500.gr2

We download the most recent GRIB-2 datasets from this site: https://satepsanone.nesdis.noaa.gov/pub/product/smops/TestData/

I'm not sure if there's a long-term archive of the GRIB-2 version of the data. Thoughts - @karsenau - on where Donghyuck (and other LIS users) can get this data archive?

David