Skip to content

Commit ab3ed60

Browse files
author
Gregory E. Maurer
committed
Documentation updates
1 parent 7cd882a commit ab3ed60

5 files changed

+272
-7
lines changed

‎README.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,9 @@ Documentation is found in the `doc` directory -- start [here](doc/README.md).
1414
## Data releases
1515

1616
Many of versions of NMEG flux data have been produced by this code as it has
17-
changed over the years. Recently we have made the effort to version the data
17+
changed over the years. Recently we have made the effort to version NMEG data
1818
using a release system that has associated git tags. Release information is
19-
on the [Releases page](doc/Releases.md) of the docs.
19+
on the [Data Releases page](doc/DataReleases.md) of the docs.
2020

2121

2222

‎doc/DataReleases.md

+36
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# NMEG Data Releases
2+
3+
Data collection and quality assurance is ongoing, but QA'd data files must
4+
periodically be "released" for submission to AmeriFlux or other collaborators.
5+
Recently we have begun a release system in which a version of NMEG data files
6+
is created with an associated git tag in the repository. These verisoned files
7+
can then be submitted, and if needed, releases can be sub-versioned to fix
8+
issues until the next release.
9+
10+
## Current releases
11+
12+
Unless otherwise noted, associated git tags with the same name will be found on
13+
the master branch of NMEG_FluxProc
14+
15+
* **FLUXNET2015_a** removed the Litvak lab USTAR threshold filter from FC, H,
16+
and LE, so that output data were more representative of the raw flux (rather
17+
than NEE or similar). In gapfilled files, a USTAR filter was applied to FC_F,
18+
by the MPI Eddyproc tool prior to gapfilling (Reichstein 2005 method).
19+
20+
- Note that the nighttime filter problem (USTAR filtering at night) was fixed,
21+
and this git tag was moved to reflect that.
22+
23+
24+
* **FLUXNET2015** This release accumulates all QA improvements made by Greg (and
25+
some others) from 2014-2016, including calibration changes, timing shifts,
26+
relaxing filters, improved ancillary met gapfilling, etc. This release
27+
maintains the Litvak lab USTAR filtering thresholds, and no additional USTAR
28+
filters were applied during gapfilling.
29+
30+
## Upcoming release changes to be aware of
31+
32+
* The next release removes an incorrect 30 minute shift so that timestamps
33+
fall at the end of each 30 minute averaging period (as stamped by the
34+
datalogger). In prior releases timestamps signify the start of the averaging
35+
period. AmeriFlux will need to be notified of this.
36+

‎doc/Install_and_Configure.md

+5-5
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
The source code can be downloaded from [the GitHub repository](https://github.com/gremau/NMEG_FluxProc). MATLAB should be installed, and some tasks will requre cygwin (for the bash shell), Campbell Scientific's CardConvert utility, and R tobe installed and on the path.
44

55

6-
## Local setup
6+
## Local machine setup
77

88
### Data and configuration directories
99

@@ -28,9 +28,9 @@ the test site mentioned above can be found [here]
2828

2929
### Paths and environment variables.
3030

31-
An environment variable must be set for FluxProc to find the FLUXROOT
32-
directory on the local file structure. In your `startup.m` file, add
33-
these lines:
31+
An environment variable must be set for MATLAB to find the FLUXROOT
32+
(and FluxProcConfig) directory on the local machine's file structure. In your
33+
`startup.m` file, add these lines:
3434

3535
setenv('FLUXROOT', '/.../')
3636

@@ -49,5 +49,5 @@ The rest of the paths needed for FluxProc can be set using
4949

5050
fluxproc_setpaths
5151

52-
Now FluxProc code should be initialized and ready to use the data and
52+
Now NMEG_FluxProc code should be initialized and ready to use the data and
5353
configuration files in the FLUXROOT directory.

‎doc/README.md

+11
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,19 @@ This is the documentation for the NMEG_FluxProc code repository, which is
44
primarily used to process and quality assure data from the New Mexico
55
Elevation Gradient. It is written in and called from MATLAB (mostly).
66

7+
## Doc pages
8+
79
* [Installing and configuring on a local machine](Install_and_Configure.md)
810
* [Common tasks and scripts](TaskScripts.md)
911
* [Data releases (and corresponding git tags)](DataReleases.md)
12+
* Some sites have [secondary data loggers](SecondaryLoggers.md) (other than
13+
'flux')
1014
* [Managing and updating ancillary met data](AncillaryMetData.md)
1115
* [The old README](/doc/old_README.md) by Tim Hilton
16+
17+
## Filetype metadata
18+
19+
Some of the intermediate processing files (fluxall files, qc files, etc) have
20+
metadata guides that make the columns somewhat more clear'
21+
22+
* See `FLUXROOT/SiteData/metadata/`

‎doc/SecondaryLoggers.md

+218
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
# Secondary Data Loggers
2+
3+
Notes on using data from secondary dataloggers at sites that have them.
4+
Sometimes the data from these secondary loggers requires tricks to find,
5+
convert, or otherwise get into the workflow of the rest of this processing
6+
code. Unless otherwise noted, data and other information for these loggers can
7+
be found in `FLUXROOT/SiteData/{SiteName}/secondary_loggers/`.
8+
9+
Some of these might be a tad out of date (filesystem changes).
10+
11+
12+
## JSav
13+
14+
There was a secondary datalogger here, and its TOA5 files are in
15+
`secondary_loggers'. They are successfully read and appended to the fluxall
16+
files by the NMEG_FluxProc code.
17+
18+
19+
## MCon
20+
21+
### Precipitation station
22+
23+
Raw files archived in `secondary_loggers/precip/`, but there is currently no
24+
procedure to load, convert, and concatenate them into fluxall files.
25+
26+
### SAHRA datalogger
27+
28+
#### Raw data file
29+
30+
The most current data file from the SAHRA station located at our MCon site is `MCon_SAHRA_data_20061001_20130601.dat`. This should contain all data that the station collected. If needed, these data are available from:
31+
32+
<http://www.wrcc.dri.edu/cgi-bin/rawMAIN.pl?nmvcnx>
33+
(Main top-level URL: <http://www.wrcc.dri.edu/index.html> )
34+
35+
The password for data older than 30 days is wrcc14. The .dat files included in this directory were downloaded in delimited (Win/PC) format.
36+
37+
Depending on your selections when exporting files from the website, there may be some HTML at the beginning and end of the file. Be sure to strip this out first.
38+
39+
#### Converting raw HydraProbe voltages
40+
41+
The soil sensors are Steven's HydraProbes. An observation from this sensor consists of 4 voltages and there are proprietary programs that convert these voltages to SWC, Tsoil, and other values. Program for converting individual observations and time-series files (`Hydra.exe` and `Hyd_file.exe`) can be downloaded from:
42+
43+
<http://www.stevenswater.com/software/downloads.aspx#Hydra>
44+
45+
Data from HydraProbe sensors must be run through `Hyde_file.exe` in a specific file format. The functions associated with the `parse_MCon_SAHRA_data.m` function in the main MATLAB pipeline should be able to create a good input file for `Hyd_file.exe` from the raw SAHRA data file.
46+
47+
48+
## MCon_SS
49+
50+
### Precipitation station
51+
52+
Raw files archived in `secondary_loggers/precip/`, but there is currently no
53+
procedure to load, convert, and concatenate them into fluxall files.
54+
55+
### Soil station
56+
57+
Raw files archived in `secondary_loggers/soil/`, and they are successfully read and appended to the fluxall files by the NMEG_FluxProc code.
58+
59+
### Sapflow station
60+
61+
Raw files archived in `secondary_loggers/sap/`, but there is currently no
62+
procedure to load, convert, and concatenate them into fluxall files (I think
63+
they are on 15min frequency).
64+
65+
## PJ
66+
67+
### Current soil_sap cr23x data
68+
69+
This data is archived in `secondary_loggers/soil_sap/` and is read by the MATLAB
70+
code and appended into fluxall files after header changes are resolved.
71+
72+
### Morillas compilations from 2009-2013
73+
74+
These have filenames like: `cr23x_PJ_2009_01_01_0030_compilation.dat`
75+
76+
2009-2013 compilations are currently based on the compilations by Laura
77+
Morillas. The original compilations used can be found in
78+
`soil_sap/LM_CR23xCompilations_orig/` and there are backup copies on the
79+
socorro ftp server in the "Laura M" directory.
80+
81+
A couple of changes have been made to these to make them work with the header
82+
resolution system in the processing code:
83+
84+
1. In some cases header names have been edited to be more usable
85+
2. The filename has been standardized.
86+
3. Filename includes a date so header change timing is parsed by matlab code
87+
88+
### Morillas QCd compilations from 2009-2013
89+
90+
2009-2013 QCd soil data are currently based on work of Laura
91+
Morillas. The qc'd compilations used can be found in
92+
`soil_sap/LM_CR23xCompilations_qc/` and there are backup copies on the
93+
socorro ftp server in the "Tower_sites/PJ/Data" directory
94+
95+
### Recent files (2014 forward)
96+
97+
These are simple compilations of the raw datalogger files that are appended to
98+
as new data is retrieved by Jonathan. New data should be retrieved from
99+
socorro at:
100+
101+
`/export/db/work/eddyflux/Tower sites/Tower Sites/PJ/PJ/Data/PJ_CR23X_Data`
102+
103+
and can then be appended to an existing file, or a new file can be created by
104+
pasting headers into the raw datalogger file. Older datalogger files are in the
105+
`PJC_23X_YEAR/` directories.
106+
107+
### "Soil Complete" files
108+
109+
These can be made using the `mirror_PJ_soil_to_local` bash script in the
110+
`secondary_loggers` directory. The last time I tried this it didn't work
111+
particularly well, perhaps due to header changes. According to Tim's former
112+
documentation:
113+
114+
The bash script `mirror_PJ_soil_to_local` mirrors the data in
115+
`socorro.unm.edu:/export/db/work/eddyflux/Tower sites/Tower Sites/PJ/PJ/Data/PJ_CR23X_Data`
116+
to this directory.
117+
118+
It then concatenates all the data files together and removes malformatted lines, creating `PJ_YYYY_soil_complete.dat`.
119+
120+
The matlab code that creates the soil Ameriflux files looks in this directory for `PJ_YYYY_soil_complete.dat`.
121+
122+
So, to update PJ soil data:
123+
124+
1. run `mirror_PJ_soil_to_local`
125+
2. process the soil data as usual in Matlab.
126+
127+
### Other compilations (`other_compilations`)
128+
129+
Assorted random compilations of cr23x data (including "soil complete" compilations) that have been used in the past. Haven't really looked at all of these very thorougly yet.
130+
131+
132+
## PJ_girdle
133+
134+
### Current cr23x data
135+
136+
This data is archived in `secondary_loggers/soil_sap/` and is read by the MATLAB code and appended into fluxall files after header changes are resolved.
137+
138+
### Morillas compilations from 2009-2013
139+
140+
These have filenames like: `cr23x_PJ_girdle_2009_01_01_0030_compilation.dat`
141+
142+
2009-2013 compilations are currently based on the compilations by Laura
143+
Morillas. The original compilations used can be found in
144+
`cr23x_files/LM_CR23xCompilations_orig/` and there are backup copies on the
145+
socorro ftp server in the "Laura M" directory.
146+
147+
A couple of changes have been made to these to make them work with the header
148+
resolution system in the processing code:
149+
150+
1. In some cases header names have been edited to be more usable
151+
2. The filename has been standardized.
152+
3. Filename includes a date so header change timing is parsed by matlab code
153+
154+
### Morillas QCd compilations from 2009-2013
155+
156+
2009-2013 QCd soil data are currently based on work of Laura
157+
Morillas. The qc'd compilations used can be found in
158+
`cr23x_files/LM_CR23xCompilations_qc/` and there are backup copies on the
159+
socorro ftp server in the "Tower_sites/PJ_girdle/Data" directory.
160+
161+
### Recent files (2014 forward)
162+
163+
These are simple compilations of the raw datalogger files that are appended to
164+
as new data is retrieved by Jonathan. New data should be retrieved from
165+
socorro at:
166+
167+
`/export/db/work/eddyflux/Tower sites/Tower Sites/PJ/PJ_girdle/Data/PJ_girdle_ CR23X_Data`
168+
169+
and can then be appended to an existing file, or a new file can be created by
170+
pasting headers into the raw datalogger file. Older datalogger files are in the
171+
`PJG_23X_YEAR/` directories.
172+
173+
### "Soil Complete" files
174+
175+
These can be made using the `mirror_PJ_soil_to_local` bash script in the PJ
176+
Control directory. The last time I tried this it didn't work particularly well, perhaps due to header changes. According to Tim's former documentation:
177+
178+
The bash script `mirror_PJ_soil_to_local` mirrors the data in
179+
`socorro.unm.edu:/export/db/work/eddyflux/Tower sites/Tower Sites/PJ/PJ/Data/PJ_CR23X_Data`
180+
to this directory.
181+
182+
It then concatenates all the data files together and removes malformatted lines, creating `PJ_YYYY_soil_complete.dat`.
183+
184+
The matlab code that creates the soil Ameriflux files looks in this directory for `PJ_YYYY_soil_complete.dat`.
185+
186+
So, to update PJ soil data:
187+
188+
1. run `mirror_PJ_soil_to_local`
189+
2. process the soil data as usual in Matlab.
190+
191+
### Other compilations (`other_compilations`)
192+
193+
Assorted random compilations of cr23x data (including "soil complete" compilations) that have been used in the past. Haven't really looked at all of these very thorougly yet.
194+
195+
196+
## PPine
197+
198+
### DRI logger
199+
200+
in `secondary_loggers/DRI_logger/`
201+
202+
The zip archive `PPine_2008_2012_raw_soil_data.zip` contains all known PPine soil data files provided by the DRI team that (formerly?) ran the soil station at the site.
203+
204+
To the best of my knowledge, this system has four soil profiles with Steven's
205+
HydraProbe sensors at 5", 20", and 50" (a total of 12 T/VWC measurements).
206+
207+
The data span 1 Jan 2008 to early 2015. These data have been concatenated into a file called `PPine_soil_data_20080101_20150128.dat`. The concatenation was performed in Matlab using the code in `concatenate_all_PPine_soil_data.m`, a copy of which lives in this zip archive. There is also a version of this file that is run from the main MATLAB pipeline (same name).
208+
209+
See matlab documentation for `concatenate_all_PPine_soil_data` for more
210+
extensive usage instructions.
211+
212+
The `PPine_soil_data_20080101_20150128.dat` can be parsed by the `parse_PPine_soil_data.m` function.
213+
214+
215+
#### archive/
216+
217+
In the `archive` directory there are also some soil data files that may or may not be of interest.
218+

0 commit comments

Comments
 (0)