OPeNDAP QC OPeNDAP QC ImplementationImplementation
Use CaseUse Case
SECOORA/SEACOOS ActivitiesSECOORA/SEACOOS ActivitiesUNC, USC, USF, UMiami, SKIOUNC, USC, USF, UMiami, SKIO
Sara Haines, UNCSara Haines, UNCWednesday, 27 February 2008Wednesday, 27 February 2008
Implementing QC forImplementing QC for Regional Data Sharing Regional Data Sharing
SECOORA/SEACOOS ActivitiesSECOORA/SEACOOS ActivitiesUNC, USC, USF, UMiami, SKIOUNC, USC, USF, UMiami, SKIO
Sara Haines, UNCSara Haines, UNCWednesday, 27 February 2008Wednesday, 27 February 2008
What is SEACOOS?What is SEACOOS?Southeast Atlantic Coastal Ocean Observing SystemSoutheast Atlantic Coastal Ocean Observing SystemKick-off meeting October 2002Kick-off meeting October 2002Observing, modeling, outreach, and data managementObserving, modeling, outreach, and data managementPartnership of only a few Partnership of only a few http://www.seacoos.orghttp://www.seacoos.org
What is SECOORA?What is SECOORA?Southeast Coastal Ocean Observing Regional Southeast Coastal Ocean Observing Regional AssociationAssociationFall 2006Fall 2006Many of the same people and ideas are now part of Many of the same people and ideas are now part of SECOORASECOORAStakeholders and established governance, more diverseStakeholders and established governance, more diversehttp://www.secoora.orghttp://www.secoora.org
SEACOOS Data SEACOOS Data ManagementManagement
Share data from a wide range of instruments Share data from a wide range of instruments (CTD, HFRADAR, ADCP, Met-packages)(CTD, HFRADAR, ADCP, Met-packages)From many different platforms (ships, aircraft, From many different platforms (ships, aircraft, buoys, moorings, satellite, AUVs, drifters)buoys, moorings, satellite, AUVs, drifters)Initial parameters addressed were winds, Initial parameters addressed were winds, temperature, salinity, water level and currentstemperature, salinity, water level and currents
We needed standards but there were noneWe needed standards but there were none– Created a Common Data Language (CDL)Created a Common Data Language (CDL)– Established a Data Dictionary (included by MMI)Established a Data Dictionary (included by MMI)
Data Flow using OPeNDAPData Flow using OPeNDAP
Best of both worldsBest of both worlds
SEACOOS CDL v2.0SEACOOS CDL v2.0
Specifies required and recommended Specifies required and recommended metadata fields and attributes for metadata fields and attributes for Provides several canonical formats Provides several canonical formats modeled on how measurement taken modeled on how measurement taken (e.g. ship-mounted ADCP, or moored (e.g. ship-mounted ADCP, or moored string of CTDs)string of CTDs)Initially intended for netCDF, but could Initially intended for netCDF, but could be used for other formats such as HDF, be used for other formats such as HDF, XMLXML
SEACOOS CDL v2.0SEACOOS CDL v2.0ExamplesExamples
• Fixed point
• Fixed profile
• Moving profile
• Fixed map
• Moving 2D map
• Moving 3D point
SECOORA QA/QC Workshop SECOORA QA/QC Workshop March 9-10, 2006March 9-10, 2006
Data Sharing and Data Quality of Ocean Currents and Data Sharing and Data Quality of Ocean Currents and Temperature towards an Operational Data Management Temperature towards an Operational Data Management
SystemSystem
Regional Association (RA) ResponsibilitiesRegional Association (RA) Responsibilities– Aggregator of data and/or access to distributed dataAggregator of data and/or access to distributed data– QC of aggregated dataQC of aggregated data– Assuring data available (Is a cataloged sensor/data Assuring data available (Is a cataloged sensor/data
there for time desired?)there for time desired?)– Catalogue of assets Catalogue of assets – Ratify and “support” a few well chosen standardsRatify and “support” a few well chosen standards– Make it easy as possible to submit dataMake it easy as possible to submit data
SECOORA QA/QC Workshop SECOORA QA/QC Workshop March 9-10, 2006March 9-10, 2006
List of QC Tests and Who is Responsible:List of QC Tests and Who is Responsible:– Data AvailabilityData Availability DP DP (or RA)(or RA)– Range checksRange checks DP DP (or RA)(or RA)– Time ContinuityTime Continuity DP DP (or RA)(or RA)– Rate of ChangeRate of Change DP DP (or RA)(or RA)– Data FormatData Format DP DP (or RA)(or RA)– Instrument uncertaintiesInstrument uncertainties DP DP – Nearest NeighborNearest Neighbor RA RA (or DP)(or DP)– ClimatologyClimatology RA RA (or DP)(or DP)– Model ComparisonModel Comparison RA RA (or DP)(or DP)
Based on QARTOD recommended testsBased on QARTOD recommended tests
Implementation Sprint for QC Implementation Sprint for QC July 11-14, 2006July 11-14, 2006
Focus on QC or the step that supports the output of Focus on QC or the step that supports the output of high quality data high quality data Assume QA addressed by those responsible for proper Assume QA addressed by those responsible for proper operation of instrumentation and assessment of normal operation of instrumentation and assessment of normal operationoperationDraw on QARTOD findings and IOOS DMAC Draw on QARTOD findings and IOOS DMAC recommendationsrecommendationsWe needed more standardsWe needed more standards– Extended the Common Data Language (draft SEACOOS CDL Extended the Common Data Language (draft SEACOOS CDL
v3.0) which defined required information v3.0) which defined required information – Specified TestsSpecified Tests– Specified Flags (specific and aggregate)Specified Flags (specific and aggregate)
Draft SEACOOS CDL with QC Draft SEACOOS CDL with QC (v3.0)(v3.0)
Wind, SST, CurrentWind, SST, Current
For each variable of interest, and for each For each variable of interest, and for each instance of data for that variable, there will instance of data for that variable, there will be multiple QA/QC tests and multiple be multiple QA/QC tests and multiple QA/QC flags. QA/QC flags.
Test FlagsTest Flags
Aggregate FlagAggregate Flag
Example SST Example SST
Stub OutStub Out
Specifications for data providers (required Specifications for data providers (required and recommended) on OPeNDAP-servers and recommended) on OPeNDAP-servers (Draft CDL v3.0)(Draft CDL v3.0)
Specifications for regional provider (Draft Specifications for regional provider (Draft CDL v3.0)CDL v3.0)
Regional search and database schema to Regional search and database schema to accept QC flags and info (beta)accept QC flags and info (beta)
GUI Control (beta)GUI Control (beta)
Stub outStub out
Lack funds and resources for data managers Lack funds and resources for data managers to do QA/QC work of implementing data to do QA/QC work of implementing data model model
SECOORA/SEACOOS only lacked individual SECOORA/SEACOOS only lacked individual institutions to output QC Flags and info on institutions to output QC Flags and info on OPeNDAP-servers using this data modelOPeNDAP-servers using this data model
Next StepsNext Steps
Add variables for testing (currents, waves)Add variables for testing (currents, waves)Define and implement analysis of aggregated QA/QC Define and implement analysis of aggregated QA/QC effectiveness. effectiveness. Nearest neighbor, add model comparisonNearest neighbor, add model comparisonUncertainty valuesUncertainty valuesSensor specific assessment, tests and flagsSensor specific assessment, tests and flags
Revisit draft CDL for currents and temperature with Revisit draft CDL for currents and temperature with QARTOD 4 findingsQARTOD 4 findingsContinue monitoring IOOS DMAC and QARTOD as Continue monitoring IOOS DMAC and QARTOD as new QA/QC recommendations, parameters and new QA/QC recommendations, parameters and technologies emergetechnologies emerge
