 The data were first
converted from binary to ASCII format (as raw counts) and then converted to physical
units. Optical data were calibrated using the calibration factors provided by
Satlantic, Inc. Missing, saturated, and anomalous data were replaced with NaNs.
Repeated and out-of-order data were removed. The data were then despiked as
follows:
Despiking
Method
The despiking method used was
based on a combination of statistical and subjective criteria.
The input parameters and
typical values used (in brackets) were as follows:
numav : number of data
points to average in a running average (30)
n : least number of
standard deviations from the mean that is acceptable (2)
minstd : minimum standard
deviation used (value varies)
initav : estimated mean for
the good data points in the first set of 30 data points (value varies)
The steps taken were as
follows:
A running average and
standard deviation is calculated in groups of 30 (numav) data points.
If the standard deviation
is less than a guessed lower estimate for the standard deviation (minstd), the guessed
estimate (minstd) is used instead (this prevents stds of zero).
For the first 30 points, if
any of the points deviate from a guess for the initial mean (initav) by more than n
minimum standard deviations (n*minstd), they are removed.
After the initial set of 30
points, the despiking method is as follows: Each data point is compared with the mean and
standard deviation of the previous 30 points. If it differs from the mean by more than n
standard deviations, it is removed.
After the initial despiking
has been made, a second despiking is performed on data where any remaining spikes are
obvious (such as latitude, longitude, and occasionally SST data). This second type of
despiking is simply based on visual estimation of the maximum deviation that should occur
between data points.
Optical despiking
Before applying the above method to radiance data, each set of radiance measurements was
first divided by Ed490 to remove diurnal variations. Poor data points were then determined
by recording the positions of spikes present in these ratios. The corresponding data
points were removed from the radiance data. Poor data points in the Ed490 data set were
assumed to correspond to the poor data points present in the ratio of Lu412 to Ed490. Note
that this method also removes some good data points, since some of the data points removed
from the radiance data may have been caused by poor Ed490 values, and vice versa.
Time
The decimal day was calculated from the day of year and datatime. The datatime is the
satellite GMT time minus the data age.
Webpage by Jasmine
S. Bartlett, Oregon State University. |