|
| 
META_PROC - DF FILE PROCESSING
------------------------------
Under CDIP's new processing structure, meta_proc is the program that
manages the processing of diskfarm files. It handles all data types -
energy, temperature, wind, etc. - and produces all of the standard CDIP data
products: spectral files, databank files, and parameter files.
A. The processing archive
All of the information that meta_proc uses to manage the processing of df
files comes from the processing archives. A station's processing archive
(e.g. .stations/095/095.proc) is basically an instruction book, explaining
exactly how that station's files should be processed. Since the processing
procedure for a station can change over time, archive files are organized by
time frames. Thus once meta_proc determines the time of a file, it
can consult the processing archive and determine the correct processing
procedure. (See .docs/processing/processing_archive.txt for more details.)
B. Program flow - pre-, mid-, and post- processing
To process a file, meta_proc first opens and loads the processing archive, and
then it assembles a list of tasks to be completed. It categorizes these
tasks into three groups: pre-, mid-, and post-processing.
I. Pre-processing: Currently there is only one type of pre-processing
task: gauge comparison. This pre-processing is used to ensure that
pressure sensors in close proximity (e.g. within an array) are in
agreement. If one of the sensors does not agree with the others, it is
flagged bad and not processed further.
II. Mid-processing: The bulk of meta_proc's work comes in mid-processing.
All of the spectral, databank, and report products are made by the
mid-processing routines. There are, in fact, two types of mid-processing:
default and combined.
a. The default processing works with single-sensor data - a
temperature time series is run through the temperature routine,
a directional buoy file is run through datawell processing, etc.
b. The combined processing uses multiple files to generate products -
a wind speed file and a wind direction file are combined to
produce a databank wind entry, four water column files are put through
array processing to produce a directional spectral file, etc.
III. Post-processing: Currently there is only one type of post-processing:
parameter file generation. During the mid-processing, all of the
parameters calculated for each product are saved. Meta_proc's final task
is to add these parameters to the appropriate parameter files.
Two special data structures are used to pass information between meta_proc
subroutines. If any pre-processing is performed, an 'energy_block' is used
to pass the edited time series, the transform coefficients, etc. into the
mid-processing. Similarly, the mid-processing routines write parameter data
into the 'pm_info_block', which is then passed into the post-processing.
INPUT FILES PROCESSING TASKS OUTPUT FILES
----------- ---------------- ------------
'meta_proc rd08300200201150845'
|
V
083.proc -> Load processing archive
|
V
ix08300200201 -> Read index, assemble df file list
|
V
df08301200201... -> PRE-PROCESSING:
df08302200201... -> gauge_comparisons
... -> \
df08306200201... -> \
|
V /
/
MID-PROCESSING: --> sp08301, sp08302..
energy --> de08301, de08305..
directional --> sp083p2, dd083p2..
df08307200201... -> barometer --> db08307200201
tidal \ --> + reports, .update
\
|
V /
/
POST-PROCESSING:
run_parameter --> pm083p1200201
|
V
Finished.
C. Subdirectories and subroutines
Descriptions of meta_proc's major subroutines are given below.
gauge_comparisons - pre-processing agreement checks for pressure sensors.
Can be used for any number of sensors; a prerequisite
for array processing.
energy - processes single-point pressure sensors and
non-directional buoys.
detox - processes directional buoy data.
temperature - processes temperature data.
wind - processes wind and wind buoy data.
directional - performs array processing on wave data.
update_reports - routines for writing to the .mondat and .update reports.
parameter - routines for writing to the parameter files.
|
