This repository contains Python scripts for manipulating the monthly data produced by the MODIS cloud group in support of the MODIS simulator for climate models. The data are described in
Pincus et al, 2023: Updated observations of clouds by MODIS for global model assessment, available at https://crew.ldeo.columbia.edu/people/robert-pincus/

The data record available at the time of this writing extends from July 2002 to Feb 2023.

Downloading data

The download-data-with-wget.sh script attempts to download the entire 16Gb data record to the directory specified by the environment variable MODIS_DATA_CACHE_DIR. Downloading the data requires an authentication token for the website in environment variable EARTHDATA_TOKEN. The simple process for obtaining a token is described on the data distribution website.

At this writing the NASA http server is not always responsive and experiences "Internal Server Error"s a small fraction of the time, so it may take several hours for data to be downloaded and the script might need to be run more than once (previously downloaded files will be skipped by wget).

Post-processing data

As described in the documentation paper, the original form of the data contains variables which are not likely to be used by most analysts. The Python script process-local-data process data for a range of dates (by default, the entire data record) to produce time series of scalar variables and properly-normalized joint histograms, which can be written to netCDF files and/or a Zarr store (both are enabled by default). The original data are taken from MODIS_DATA_CACHE_DIR; the revised files are written to the local directory.

Python scripts rely on the modules listed in environment.yml. These may be installed with conda:

conda env create -f environment.yml
conda activate modis-cosp-data

Reproducing figures from the description paper

Figures from the data description paper can be generated from the post-processed data (or a subset containing July 2021) with the Jupyter notebook MODIS-COSP-Data-Paper-Figures.md. The notebook is saved in the compact Jupytext format, which is installed via environment.yml. Starting the Jupyter notebook and setting variables figDir and dataDir should reproduce the figures.