IOOS QARTOD software (ioos_qc)
Mathew Biddle
mathew.biddle@noaa.gov
https://orcid.org/0000-0003-4897-1669
NOAA/NOS/IOOS
Materials available at https://github.com/MathewBiddle/WIO_workshop.
This presentation will demonstrate how to run ioos_qc on a time-series dataset. ioos_qc implements the Quality Assurance / Quality Control of Real Time Oceanographic Data (QARTOD).
Key Objectives of QARTOD¶
- Establish authoritative QA/QC procedures for the U.S. IOOS core variables, as necessary, including detailed information about the sensors and procedures used to measure the variables.
- Produce written manuals for these QA/QC procedures
- From the list of individual QA/QC procedures and guidelines developed, define a baseline set of QA/QC procedures that can be used for certification of RCOOS data providers
- Facilitate QA/QC integration with Global Ocean Observing System (GOOS) and other international ocean observation efforts
- Engage the Federal Agencies and IOOS Regions that are part of, or contribute to, US IOOS who will use the established QA/QC procedure
- Work efficiently, without duplication of effort, to facilitate the implementation of common QA/QC procedures amongst US IOOS Partners.
Let's go get these data¶
We will be using the water level data from a fixed station in Kotzebue, AK.
We will get the data from the AOOS ERDDAP server.
from erddapy import ERDDAP
e = ERDDAP(
server="http://erddap.aoos.org/erddap/",
protocol="tabledap"
)
e.dataset_id = "kotzebue-alaska-water-level"
e.constraints = {
"time>=": "2018-09-05T21:00:00Z",
"time<=": "2019-07-10T19:00:00Z",
}
Return data and metadata¶
import cf_xarray
import netCDF4
data = e.to_xarray()
data
<xarray.Dataset> Size: 1MB
Dimensions: (timeseries: 1, obs: 7241)
Coordinates:
latitude (timeseries) float64 8B ...
longitude (timeseries) float64 8B ...
time (obs) datetime64[ns] 58kB ...
Dimensions without coordinates: timeseries, obs
Data variables: (12/21)
station (timeseries) object 8B ...
rowSize (timeseries) int32 4B ...
water_surface_above_mhw (obs) float64 58kB ...
water_surface_above_mhw_qc_agg (obs) float64 58kB ...
water_surface_above_mhw_qc_tests (obs) float64 58kB ...
water_surface_above_mllw (obs) float64 58kB ...
... ...
water_surface_above_mtl_qc_agg (obs) float64 58kB ...
water_surface_above_mtl_qc_tests (obs) float64 58kB ...
z (obs) float64 58kB ...
water_surface_above_mhhw (obs) float64 58kB ...
water_surface_above_mhhw_qc_agg (obs) float64 58kB ...
water_surface_above_mhhw_qc_tests (obs) float64 58kB ...
Attributes: (12/54)
cdm_data_type: TimeSeries
cdm_timeseries_variables: station,longitude,latitude
contributor_email: dggspubs@alaska.gov,information@aoos.org,,...
contributor_name: Alaska Division of Geological & Geophysica...
contributor_role: collaborator,sponsor,contributor,processor
contributor_role_vocabulary: NERC
... ...
station_id: 100053
summary: Timeseries data from 'Kotzebue, AK (KZTA2)...
time_coverage_end: 2019-07-10T19:00:00Z
time_coverage_start: 2018-09-05T21:00:00Z
title: Kotzebue, AK (KZTA2)
Westernmost_Easting: -162.566752data.cf
Discrete Sampling Geometry:
CF Roles: timeseries_id: ['station']
Coordinates:
CF Axes: X: ['longitude']
Y: ['latitude']
T: ['time']
Z: n/a
CF Coordinates: longitude: ['longitude']
latitude: ['latitude']
time: ['time']
vertical: n/a
Cell Measures: area, volume: n/a
Standard Names: latitude: ['latitude']
longitude: ['longitude']
time: ['time']
Bounds: n/a
Grid Mappings: n/a
Data Variables:
Cell Measures: area, volume: n/a
Standard Names: aggregate_quality_flag: ['water_surface_above_mhhw_qc_agg']
altitude: ['z']
quality_flag: ['water_surface_above_mhhw_qc_tests']
sea_surface_height_above_sea_level: ['water_surface_above_mhhw']
Bounds: n/a
Grid Mappings: n/a
data.cf.axes
{'X': ['longitude'], 'Y': ['latitude'], 'T': ['time']}
data.cf.coordinates
{'longitude': ['longitude'], 'latitude': ['latitude'], 'time': ['time']}
data.cf.standard_names
{'latitude': ['latitude'],
'longitude': ['longitude'],
'time': ['time'],
'altitude': ['z'],
'sea_surface_height_above_sea_level': ['water_surface_above_mhhw'],
'aggregate_quality_flag': ['water_surface_above_mhhw_qc_agg'],
'quality_flag': ['water_surface_above_mhhw_qc_tests']}
Let's plot the raw data¶
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(15, 3.75))
ax.plot(data['time'],data['water_surface_above_mhhw'])
ax.grid(True)
Build the QC configuration¶
Below we create a simple Quality Assurance/Quality Control (QA/QC) configuration that will be used as input for ioos_qc. All the interval values are in the same units as the data.
For more information on the tests and recommended values for QA/QC check the documentation of each test and its inputs: https://ioos.github.io/ioos_qc/api/ioos_qc.html#module-ioos_qc.qartod
qc_config = {
"qartod": {
"gross_range_test": {
"suspect_span": [ -2, 3],
"fail_span": [-10, 10]
},
"flat_line_test": {
"tolerance": 0.001,
"suspect_threshold": 10800,
"fail_threshold": 21600
},
"spike_test": {
"suspect_threshold": 0.8,
"fail_threshold": 3,
}
}
}
nice_print(qc_config)
{
"qartod": {
"gross_range_test": {
"suspect_span": [
-2,
3
],
"fail_span": [
-10,
10
]
},
"flat_line_test": {
"tolerance": 0.001,
"suspect_threshold": 10800,
"fail_threshold": 21600
},
"spike_test": {
"suspect_threshold": 0.8,
"fail_threshold": 3
}
}
}
Run the QC tests with the supplied configuration¶
from ioos_qc.config import QcConfig
qc = QcConfig(qc_config)
variable_name = data.cf.standard_names["sea_surface_height_above_sea_level"][0]
qc_results = qc.run(
inp=data[variable_name],
tinp=data.cf["time"],
)
nice_print(qc_results['qartod'])
{
"gross_range_test": "[1 1 1 ... 1 1 1]",
"flat_line_test": "[1 1 1 ... 1 1 1]",
"spike_test": "[2 1 1 ... 1 1 2]"
}
The results are returned in a dictionary format, similar to the input configuration, with a mask for each test. The results range from 1 to 4 meaning:
| flag | meaning |
|---|---|
| 1 | data passed the QA/QC |
| 2 | did not run on this data point |
| 3 | flag as suspect |
| 4 | flag as failed |
Let's look at the gross_range test results¶
The gross range test test should fail data outside the $\pm$ 10 range and suspect data below -2, and greater than 3. As one can easily see all the major spikes are flagged as expected.
plot_results(
data,
variable_name,
qc_results,
title,
"gross_range_test"
)
qc_config['qartod']['gross_range_test']
{'suspect_span': [-2, 3], 'fail_span': [-10, 10]}
Let's look at the spike test results¶
An actual spike test, based on a data increase threshold, flags similar spikes to the gross range test but also indetifies other suspect unusual increases in the series.
plot_results(
data,
variable_name,
qc_results,
title,
"spike_test"
)
qc_config['qartod']['spike_test']
{'suspect_threshold': 0.8, 'fail_threshold': 3}
Let's look at the flat_line test results¶
The flat line test identifies issues with the data where values are "stuck."
ioos_qc succefully identified a huge portion of the data where that happens and flagged a smaller one as suspect. (Zoom in the red point to the left to see this one.)
plot_results(
data,
variable_name,
qc_results,
title,
"flat_line_test"
)
qc_config['qartod']['flat_line_test']
{'tolerance': 0.001, 'suspect_threshold': 10800, 'fail_threshold': 21600}
What tests are currently available in ioos_qc for QARTOD?¶
import ioos_qc
for func in dir(ioos_qc.qartod):
if "test" in func:
print(func)
attenuated_signal_test climatology_test density_inversion_test flat_line_test gross_range_test location_test rate_of_change_test spike_test
Where can you find more information?¶
- IOOS CodeLab example: https://ioos.github.io/ioos_code_lab/content/code_gallery/data_analysis_and_visualization_notebooks/2020-02-14-QARTOD_ioos_qc_Water-Level-Example.html
- Example notebooks: https://github.com/ioos/ioos_qc/tree/master/docs/source/examples
- Source documentation: https://ioos.github.io/ioos_qc/
- QARTOD manuals: https://ioos.noaa.gov/project/qartod/
Thank you!
Mathew Biddle
Mathew.Biddle@noaa.gov
https://orcid.org/0000-0003-4897-1669
NOAA/NOS/IOOS
Materials available at https://github.com/MathewBiddle/WIO_workshop.