Skip to content

Commit 7cd882a

Browse files
author
Gregory E. Maurer
committed
Updating documentation
1 parent a668b67 commit 7cd882a

5 files changed

+225
-83
lines changed

‎README.md

+12-83
Original file line numberDiff line numberDiff line change
@@ -1,93 +1,22 @@
11
# NMEG_FluxProc
22

33
[This](https://github.com/gremau/NMEG_FluxProc) is the repository for
4-
FluxProc code used to process data from the New Mexico Elevation Gradient.
5-
It is primarily written in and called from MATLAB.
4+
FluxProc code used to process data from the New Mexico Elevation Gradient,
5+
an ecological research project based at the University of New Mexico.
6+
For project information see [the project website](http://biology.unm.edu/litvak/res_NM_elev.html).
7+
This code is used to process and quality assure eddy covariance and other
8+
environmental sensor data.
69

10+
## Documentation
711

8-
## Local setup
12+
Documentation is found in the `doc` directory -- start [here](doc/README.md).
913

10-
### Data and configuration directories
14+
## Data releases
1115

12-
The FluxProc code performs operations on data in a designated directory
13-
("FLUXROOT"). An example FLUXROOT directory with data from an imaginary
14-
site can be downloaded from the socorro ftp and used to test the FluxProc
15-
code.
16+
Many of versions of NMEG flux data have been produced by this code as it has
17+
changed over the years. Recently we have made the effort to version the data
18+
using a release system that has associated git tags. Release information is
19+
on the [Releases page](doc/Releases.md) of the docs.
1620

17-
Site specific configuration files must also be present in the FLUXROOT
18-
path, and FluxProc is currently set to look for them in
19-
"FLUXROOT/FluxProcConfig". Configuration files for NMEG sites, including
20-
the test site mentioned above can be found [here]
21-
(https://github.com/gremau/NMEG_FluxProcConfig).
2221

23-
### Paths and environment variables.
2422

25-
An environment variable must be set for FluxProc to find the FLUXROOT
26-
directory on the local file structure. In your `startup.m` file, add
27-
these lines:
28-
29-
setenv('FLUXROOT', '/.../')
30-
31-
where "/.../" is the path to the FLUXROOT directory. This will add the
32-
needed environment variable each time you start MATLAB.
33-
34-
Once this is done, start MATLAB, add the NMEG_FluxProc to your path:
35-
36-
addpath('path/to/FluxProc')
37-
38-
Enter the directory:
39-
40-
cd 'path/to/FluxProc'
41-
42-
The rest of the paths needed for FluxProc can be set using
43-
44-
fluxproc_setpaths
45-
46-
Now FluxProc code should be initialized and ready to use the data and
47-
configuration files in the FLUXROOT directory.
48-
49-
50-
## Task scripts
51-
52-
Common tasks have scripts that can be run with common configurations, and are easily modified. These scripts can be found in the [scripts](scripts/) directory. Each of these scripts can be set to run for a list of sites and years and to overwrite existing output files or not.
53-
54-
### Create new "fluxall" files
55-
56-
Fluxall files (`{site}_{year}_fluxall.txt`) should contain raw data from all sensors at a site for one year. The [script_make_fluxall.m](/scripts/script_make_fluxall.m) script will make these files, primarily by calling [card_data_processor.m](card_data_processor.m) in various configurations and reading the raw data in 'toa5' and 'ts_data' directories. Though these files should contain all sensor data, in practice there are some sites with dataloggers that have not been configured to be merged into the fluxall file (namely the Valles Caldera sites).
57-
58-
### Create new "qc", "for_gapfilling", and "for_gapfilling_filled" files
59-
60-
There are several files created from the NMEG quality control pipeline, all output to the "processed_flux" directory. These are:
61-
62-
1. "qc" files (`{site}_{years}_fluxall_qc.txt`): Contain all variables that are quality-controlled and then output by the [UNM_RemoveBadData.m](UNM_RemoveBadData.m) script.
63-
64-
2. "for_gapfilling" files (`{site}_flux_all_{year}_for_gap_filling.txt`): Also output by [UNM_RemoveBadData.m](UNM_RemoveBadData.m) script and contain a subset of quality-controlled variables in a format ready to be filled with ancillary met data.
65-
66-
3. "for_gapfilling_filled" files (`{site}_flux_all_{year}_for_gap_filling_filled.txt`): Same as the file above, but gaps in the met variables have been filled with ancillary met data by the [UNM_fill_met_gaps_from_nearby_site.m](UNM_fill_met_gaps_from_nearby_site.m) script.
67-
68-
To make these files, run the [script_make_qc_gf.m](/scripts/script_make_qc_gf.m). This script may also run the REddyProc gapfilling tool by calling on the [R code from the Max Planck institute](https://www.bgc-jena.mpg.de/bgi/index.php/Services/REddyProcWebRPackage), and the output (also in 'processed_flux') can be used to make AmeriFlux files, below, if desired.
69-
70-
### Create new AmeriFlux files
71-
72-
AmeriFlux files (`{af-site}_{year}_gapfilled.txt` and `{af-site}_{year}_with_gaps.txt`) contain quality controlled sensor data, gapfilled met data, gapfilled fluxes, and partitioned C fluxes. There are several steps currently needed to create them.
73-
74-
1. Send the "for_gapfilling_filled" file for each site/year to the [MPI EddyProc web service](http://www.bgc-jena.mpg.de/~MDIwork/eddyproc/upload.php). This service provides gapfilled and partitioned flux data, and is the way we currently have to get Lasslop partitioned fluxes used for the lower elevation NMEG sites.
75-
76-
2. Once you receive notification that the partitioner has finished (by email), copy the job number and run [download_gapfilled_partitioned_flux.m](/retrieve_card_data/download_gapfilled_partitioned_flux.m) with the job number as an argument. This will download the resulting files to the "processed_flux" directory.
77-
78-
3. Run [script_make_ameriflux.m](/scripts/script_make_ameriflux.m), which will call the `UNM_Ameriflux_File_Maker.m` with the specified configuration options and output the new AmeriFlux files to "FLUXROOT/FluxOut/".
79-
80-
### Create soil "qc" and "qc_rbd" files
81-
82-
Running the [script_make_soilmet_qc.m](/scripts/script_make_soilmet_qc.m) script for a site/year runs [soil_met_correct.m](soil_met_correct.m), which assembles all soil temperature, water content, heat flux, and TCAV sensor outputs, applies temperature corrections to VWC, and then removes bad data. The two output files, which are placed in "processed_soil", are:
83-
84-
1. "qc" files (`{site}_{year}_soilmet_qc.txt`): Contains all sensor outputs, with raw outputs converted to VWC and temperature corrected (if applicable), for a site/year.
85-
86-
2. "qc_rbd" files (`{site}_{year}_soilmet_qc.txt`): Same as above, but with outliers, level shifts, and other bad data removed on a site/year basis (see [soil_met_correct.m](soil_met_correct.m) for details.
87-
88-
89-
## Additional documentation (in doc/)
90-
91-
* [Ancillary Met Data](/doc/Anc_Met_Readme.md)
92-
* [The old README](/doc/old_README.md)
93-
* [The old README](/doc/old_README.md)

‎doc/AncillaryMetData.md

+110
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# AncillaryMetData
2+
3+
This file contains instructions for downloading ancillary data used to fill
4+
missing NMEG site meteorology data. Unless otherwise specified, this information
5+
applies to the `FLUXROOT/Ancillary_met_data` directory, and new data files
6+
should be put inside that directory.
7+
8+
9+
## Valles Caldera met stations ( MCon and PPine )
10+
11+
There is a network of stations in the Valles that have been managed by different entities over the years. NPS staff now maintain the stations, but data are available through the Desert Research Institute (DRI) Western Regional Climate Center (WRCC).
12+
13+
All sites at VCNP can be seen here [here](http://www.wrcc.dri.edu/vallescaldera/). You can click on the site or...
14+
15+
Data for Jemez, the closest to PPine, can be accessed from:
16+
17+
<http://www.wrcc.dri.edu/cgi-bin/rawMAIN3.pl?nmxjem>
18+
19+
Data for Redondo, the closest to MCon, can be accessed from:
20+
21+
<http://www.wrcc.dri.edu/cgi-bin/rawMAIN3.pl?nmvrdd>
22+
23+
Data for VC Headquarters, can be accessed from:
24+
25+
<http://www.wrcc.dri.edu/cgi-bin/rawMAIN3.pl?nmvhvc>
26+
27+
From each page select "Data Lister" and fill in the form for dates/format desired (delimited format,short headers, metric units, etc). The password needed to download data is "wrcc14" (or wrcc15 for Jemez?). The headers have changed periodically, so after downloading all data (2006-present) the files have been broken down into periods with the same header. Original files are in the "raw_DRI_VC_files" directory, and this directory contains the doctored files. The "parse_valles_met_data.m" script should do a pretty good job of stitching these back together.
28+
29+
### NOTE
30+
31+
Data from some, but not all, of these VCNP met stations are also available in different format from the Sev LTER network. There was once a parser (see git history) for these files and in some ways they are a little easier to use. They can be found here:
32+
33+
<http://tierra.unm.edu:8080/research/climate/meteorology/VCNP/index>
34+
35+
using username:sevuser and password:mes4paje . In the past these same files were
36+
at:
37+
38+
<http://sev.lternet.edu/research/climate/meteorology/VCNP/index.html>
39+
40+
41+
## SNOTEL Stations ( MCon and PPine )
42+
43+
Can be navigated to from the interactive map at the [NRCS site](http://www.wcc.nrcs.usda.gov/snow/). Navigate to the desired site page where there is a data download interface. Select hourly or daily csv data from standard SNOTEL sensors for the desired calendar year (all days), then press the View Historic button. This should download a file with a 'XXX_STAND_YEAR=YYYY.csv' filename, where XXX is the site code and YYYY is the year. Save this in the 'SNOTEL_daily' or 'SNOTEL_hourly' directory as appropriate.
44+
45+
We currently use data from Quemazon (708), Senorita Divide #2 (744), and Vacas Locas (1017).
46+
47+
48+
## Sevilleta sites (GLand, SLand, New_Gland)
49+
50+
We use the Sevilleta LTER met network to fill our sites there. These sites are managed by Doug Moore. There are several options for acquiring data.
51+
52+
1. Contact Doug directly (dmoore@sevilleta.unm.edu)
53+
2. Raw data from all sites is posted periodically to the web at:
54+
55+
http://sev.lternet.edu/research/climate/meteorology/SEV/outputXX.dat
56+
57+
where XX is the year, OR
58+
59+
http://sev.lternet.edu/data (Meteorology data link)
60+
61+
3. Get the raw wireless data from the socorro ftp server at:
62+
63+
ftp://eddyflux@socorro.unm.edu/export/db/work/wireless/data/current/met
64+
65+
66+
## Global Historical Climate Network data from NCDC (JSav)
67+
68+
GHCN data is available at <http://www.ncdc.noaa.gov/cdo-web/datasets>. Follow
69+
the link to "Daily Summaries". Then select the time period, and search
70+
for the sitename (ESTANCIA and PROGRESSO are near JSav and PJ sites). Once the
71+
site is found add it to the cart and:
72+
73+
1. Select the desired format (Custom Daily CSV), adjust date range if needed
74+
2. Select station metadata outputs (all - station name, location, flags) and set
75+
units to METRIC
76+
3. Select data outputs (all - precip, temp, + all subcategories)
77+
4. Enter email address and submit order
78+
79+
A link to the data will be sent via email. The raw datafile should be parseable
80+
by MATLAB. Note that data outputs change over the years, so it may be wise to
81+
always get data files for the entire period (2006/01/01-present)
82+
83+
84+
## PRISM data
85+
86+
Daily PRISM data is downloaded as zipped archives of 1-day .bil files (a georeferenced data format). Yearly archives for the continental US can be downloaded [here](http://prism.oregonstate.edu/recent/). Note that PRISM data is provisional for 6 months. If desired, provisional data can be downloaded [here](http://prism.oregonstate.edu/6month/). Save the archives in the 'PRISM_daily/raw_bil' directory.
87+
88+
There are 2 python scripts and a configuration file needed to process the .bil
89+
file archives into usable precip data. These can be found in [this
90+
repository](http://github.com/gremau/NMEG_utils).
91+
92+
* `site_coords.txt` is a list of site names and lat/lon coordinates
93+
* `bilParser.py` is defines a `BilFile` object with a method to extract a data value at a given coordinate location. It also defines some functions to extract data from particular types of bilFiles (monthly, provisional, etc)
94+
* `getPRISMdata.py` is the master script. It sets needed parameters and then
95+
* makes calls to the `bilParser` functions (and by extension, `BilFile` methods) to extract data for each day, year, and site, and then outputs a csv for each year.
96+
97+
I use the anaconda python distribution and run getPRISMdata.py with ipython, but other python distributions that include numPy, pandas, and matplotlib should work. In addition to python, [OsGeo GDAL](http://www.gdal.org/) and its python bindings need to be installed. Decent instructions for this can be found [here](http://pythongisandstuff.wordpress.com/2011/07/07/installing-gdal-and-ogr-for-python-on-windows/). For the Sandia lab computer, this has all been done.
98+
99+
100+
## DayMet data
101+
102+
DayMet has its own single pixel extractor program (daymet_multiple_extraction.jar) that can be downloaded [here](http://daymet.ornl.gov/singlepixel.html). This currently resides in the 'DayMet' directory.
103+
104+
* Sites to be extracted should be added to latlon.txt.
105+
* Open a terminal in this directory and run `./daymet_multiple_extraction.sh`
106+
107+
Currently Daymet only seems to be available through 2013
108+
109+
110+

‎doc/Install_and_Configure.md

+53
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# Installation and configuration
2+
3+
The source code can be downloaded from [the GitHub repository](https://github.com/gremau/NMEG_FluxProc). MATLAB should be installed, and some tasks will requre cygwin (for the bash shell), Campbell Scientific's CardConvert utility, and R tobe installed and on the path.
4+
5+
6+
## Local setup
7+
8+
### Data and configuration directories
9+
10+
NMEG_Fluxproc scripts and functions reqire access to a local data
11+
directory (`FLUXROOT`) and a configuration file directory (`FluxProcConfig`).
12+
On the local machine, MATLAB must be able to find these two directories.
13+
14+
#### FLUXROOT
15+
16+
The NMEG_FluxProc scripts and functions performs operations on data in a
17+
designated path (termed `FLUXROOT`). An example FLUXROOT directory with data
18+
from an imaginary site can be downloaded from the socorro ftp and used to
19+
test the NMEG_FluxProc code.
20+
21+
#### FluxProcConfig
22+
23+
Site specific configuration files must also be present in the `FLUXROOT`
24+
path, and NMEG_FluxProc is currently set to look for them in
25+
`FLUXROOT/FluxProcConfig`. Configuration files for NMEG sites, including
26+
the test site mentioned above can be found [here]
27+
(https://github.com/gremau/NMEG_FluxProcConfig).
28+
29+
### Paths and environment variables.
30+
31+
An environment variable must be set for FluxProc to find the FLUXROOT
32+
directory on the local file structure. In your `startup.m` file, add
33+
these lines:
34+
35+
setenv('FLUXROOT', '/.../')
36+
37+
where "/.../" is the path to the FLUXROOT directory. This will add the
38+
needed environment variable each time you start MATLAB.
39+
40+
Once this is done, start MATLAB, add the NMEG_FluxProc to your path:
41+
42+
addpath('path/to/FluxProc')
43+
44+
Enter the directory:
45+
46+
cd 'path/to/FluxProc'
47+
48+
The rest of the paths needed for FluxProc can be set using
49+
50+
fluxproc_setpaths
51+
52+
Now FluxProc code should be initialized and ready to use the data and
53+
configuration files in the FLUXROOT directory.

‎doc/README.md

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# NMEG_FluxProc Documentation
2+
3+
This is the documentation for the NMEG_FluxProc code repository, which is
4+
primarily used to process and quality assure data from the New Mexico
5+
Elevation Gradient. It is written in and called from MATLAB (mostly).
6+
7+
* [Installing and configuring on a local machine](Install_and_Configure.md)
8+
* [Common tasks and scripts](TaskScripts.md)
9+
* [Data releases (and corresponding git tags)](DataReleases.md)
10+
* [Managing and updating ancillary met data](AncillaryMetData.md)
11+
* [The old README](/doc/old_README.md) by Tim Hilton

‎doc/TaskScripts.md

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Scripts for common tasks
2+
3+
## Task scripts
4+
5+
Common tasks have scripts that can be run with common configurations, and are easily modified. These scripts can be found in the [scripts](scripts/) directory. Each of these scripts can be set to run for a list of sites and years and to overwrite existing output files or not.
6+
7+
### Create new "fluxall" files
8+
9+
Fluxall files (`{site}_{year}_fluxall.txt`) should contain raw data from all sensors at a site for one year. The [script_make_fluxall.m](/scripts/script_make_fluxall.m) script will make these files, primarily by calling [card_data_processor.m](card_data_processor.m) in various configurations and reading the raw data in 'toa5' and 'ts_data' directories. Though these files should contain all sensor data, in practice there are some sites with dataloggers that have not been configured to be merged into the fluxall file (namely the Valles Caldera sites).
10+
11+
### Create new "qc", "for_gapfilling", and "for_gapfilling_filled" files
12+
13+
There are several files created from the NMEG quality control pipeline, all output to the "processed_flux" directory. These are:
14+
15+
1. "qc" files (`{site}_{years}_fluxall_qc.txt`): Contain all variables that are quality-controlled and then output by the [UNM_RemoveBadData.m](UNM_RemoveBadData.m) script.
16+
17+
2. "for_gapfilling" files (`{site}_flux_all_{year}_for_gap_filling.txt`): Also output by [UNM_RemoveBadData.m](UNM_RemoveBadData.m) script and contain a subset of quality-controlled variables in a format ready to be filled with ancillary met data.
18+
19+
3. "for_gapfilling_filled" files (`{site}_flux_all_{year}_for_gap_filling_filled.txt`): Same as the file above, but gaps in the met variables have been filled with ancillary met data by the [UNM_fill_met_gaps_from_nearby_site.m](UNM_fill_met_gaps_from_nearby_site.m) script.
20+
21+
To make these files, run the [script_make_qc_gf.m](/scripts/script_make_qc_gf.m). This script may also run the REddyProc gapfilling tool by calling on the [R code from the Max Planck institute](https://www.bgc-jena.mpg.de/bgi/index.php/Services/REddyProcWebRPackage), and the output (also in 'processed_flux') can be used to make AmeriFlux files, below, if desired.
22+
23+
### Create new AmeriFlux files
24+
25+
AmeriFlux files (`{af-site}_{year}_gapfilled.txt` and `{af-site}_{year}_with_gaps.txt`) contain quality controlled sensor data, gapfilled met data, gapfilled fluxes, and partitioned C fluxes. There are several steps currently needed to create them.
26+
27+
1. Send the "for_gapfilling_filled" file for each site/year to the [MPI EddyProc web service](http://www.bgc-jena.mpg.de/~MDIwork/eddyproc/upload.php). This service provides gapfilled and partitioned flux data, and is the way we currently have to get Lasslop partitioned fluxes used for the lower elevation NMEG sites.
28+
29+
2. Once you receive notification that the partitioner has finished (by email), copy the job number and run [download_gapfilled_partitioned_flux.m](/retrieve_card_data/download_gapfilled_partitioned_flux.m) with the job number as an argument. This will download the resulting files to the "processed_flux" directory.
30+
31+
3. Run [script_make_ameriflux.m](/scripts/script_make_ameriflux.m), which will call the `UNM_Ameriflux_File_Maker.m` with the specified configuration options and output the new AmeriFlux files to "FLUXROOT/FluxOut/".
32+
33+
### Create soil "qc" and "qc_rbd" files
34+
35+
Running the [script_make_soilmet_qc.m](/scripts/script_make_soilmet_qc.m) script for a site/year runs [soil_met_correct.m](soil_met_correct.m), which assembles all soil temperature, water content, heat flux, and TCAV sensor outputs, applies temperature corrections to VWC, and then removes bad data. The two output files, which are placed in "processed_soil", are:
36+
37+
1. "qc" files (`{site}_{year}_soilmet_qc.txt`): Contains all sensor outputs, with raw outputs converted to VWC and temperature corrected (if applicable), for a site/year.
38+
39+
2. "qc_rbd" files (`{site}_{year}_soilmet_qc.txt`): Same as above, but with outliers, level shifts, and other bad data removed on a site/year basis (see [soil_met_correct.m](soil_met_correct.m) for details.

0 commit comments

Comments
 (0)