PROJECT DATA MANAGEMENT PLANNING

Steve Williams

NCAR Earth Observing Laboratory (EOL)

Boulder, Colorado

3rd Drought Research Initiative Workshop

Calgary, Alberta

17-19 January 2008

DATA MANAGEMENT

HOW MANY LEGS DOES THIS ELEPHANT HAVE?

DATA MANAGEMENT CONSIDERATIONS

• Data Policy, Protocol, and Plan• Data Types and Inventory• Data Availability, Transmission, and Collection

• Data Formats and Documentation• Real-time Data Requirements• Data Processing and Quality Control

• Data and Metadata Archival• Data Access and Distribution• Coordination with other Programs

• Data Integration, Analysis, and Application• Data Stewardship

“The beauty about standards is that there are so many to choose from”

- Anonymous

XML, THREDDSWeb display (HTML, XSLT)Text filesGIS (internet mapping, desktop GIS, Open GIS Services)ISO19115

Export

XML Files (local or distributed)WWW formsScience data formats / databases

Import

NGDC Metadata Management System

National Geophysical Data Center Database

• The NGDC Metadata System presently supports the Federal Geographic Data Committee (FGDC) Content Standard for Digital Geospatial Metadata as well as a number of extensions to that standard.

• In addition, the FGDC provides a standard structure for XML documents that can be used several national data portals (i.e. NASA’s Global Change Master Directory, ESRI’s Geography Network, and the FGDC Data Clearinghouses).

ADaM DataMining EnvironmentADaM DataMining Environment(University of Alabama-Huntsville)(University of Alabama-Huntsville)

MiningResults

Mining Engine (ADaM)AnalysisModules

InputModules

OutputModules

Analysis/Vis Tools

Knowledge Base

Distributed Clients

Web-basedWorkstation

basedOther Systems

Common Client API

Data Stores

Data Mining Server

Event/Relationship SearchSystem

Coordinated Energy and water-cycle Observations Peroject

A Well Organized Data Archive System

Data Integrating/Archiving

Center at University of Tokyo and JAXA of

Japan

Model Output Data Archiving Center at the World Data Center

for Climate, Max-Planck Institute for Meteorology of

Germany

In-Situ Data Archiving Center at NCAR (National

Center for Atmospheric Research) of USA

GrADS-DODS Analysis Server

GRIB dataNetCDF data

GrADS data

etc..

datasets in any format supported by GrADS

Result cache

holds temporary data (uploaded, generated by a previous operation, or transferred directly from another server) for use in remote analysis

GrADS batch mode

interface code

DODS server libraries

Server

performs analysis

operations

manages sessions, translates dataset

names

Java servlet

supports extended request types for analysis, upload

internet

DODS data and requests

DODS client libraries

GrADS

Matlab

IDL

etc..

data appears to client as local file, in a standard format (i.e, NetCDF, etc.)

Client

Encapsulated Analysis Requests

Data Source

Data/Doc Arrive at EOL

Visual Inspection of Data and Plots

Available On-Line

EOL Updates Status Table and Detailed Notes

Reference Site Data Flow

Apply Auto/Manual Data/Doc Consistency Checks

Format Verification

Gross Limit

Checks

Exact/Inexact Dup Records

Data/Flag/Doc Checks

Generate Flagging and Site Statistics

Merge Data

Embedded Accumulative PrecipitationEmbedded Accumulative Precipitation

Composite Data Sets at NCAR/EOL

A composite dataset is a collection (over some time period and region) of similar data (e.g. surface meteorological) from a variety of sources, put into a common format, and passed through a uniform quality control.

Why does NCAR/EOL develop composites?- Provides data in a uniform format with QC.- Allows determination of network/site problems.- Useful for model applications.- Prevents duplication of effort.

Hourly Surface Meteorological Data Composite (2991 stations)

1-min sites (* 385)AWOS (+ 335)RAWS (* 220)MesoWest (+ 94)HPCN (o 138)RWIS (+ 279)GPSMET (o 153)CO CoAgMet (* 17)FL FAWN (+ 5)IA IEM (+ 88)IL ICN (o 19)IN PAAWS (* 7)KS GWMD5 (* 10)MI MAWN (o 33)MO CAWS (* 21)OH OARDC (o 11)OK ARS Micro (o 42)OK Mesonet (+ 119)TX LCRA (o 102)TX TNRCC (+ 47)West TX Meso (o 39)Texas ET (o 23)15 Other Networks (o 804)

GIS Mapserver

Climate Prediction Program for the Americas (CPPA)

Objectives:• Quantify the sources and limits of

predictability of climate variations on intra-seasonal to interannual time scale

• Improve predictive understanding and model simulations of ocean, atmosphere and land-surface processes, including the ability to quantify uncertainty

• Advance NOAA’s operational climate forecasts, monitoring, and analysis systems by transferring research to operation

• Develop climate-based hydrologic forecasting capabilities for decision support and water resource applications

PACS

Climate Predictability

Atmosphere-OceanInteractions

Land-AtmosphereInteractions

Operational ClimatePrediction, Monitoring,

and Analysis

Climate-Based Hydrologic Forecasting and Water Resources

Application

Research Components

Mission: Improve operational intra-seasonal to interannual hydroclimatic predictions for the Americas with quantified uncertainties sufficient for making informed decisions

CPPA Drought Research ProjectsObjective: to identify the contributions of anomalous boundary forcing and climate

anomalies to the initiation, intensification, and demise of persistent droughts over North America

– Warm Season Predictability of Great Plains Hydroclimate (Nigam)• It is found that external moisture fluxes are of primary importance for precipitation variability in

contrast to the local recycling of precipitation over central US

– Tropical Influences on Recent and Historical Droughts over North America (Huang and Seager)

• A drying trend is found to be driven by both the change in atmospheric moisture content and a

change in atmospheric circulation pattern under global warming.

– Roles of Land Surface Processes and Large-Scale Atmospheric Circulation in Summer Drought in the Southeast United States (R. Fu)

• provided observational evidence to show that the interannual variations of summer rainfall anomalies in SE US has been doubled in strength, and identified potential causes of such a change.

– Seasonal Cycle of Drought (D. Gutzler)

– About 7 more drought focused projects will be funded in FY08

CPPA Drought Monitor and Seasonal Prediction

NLDAS-based Drought Monitor is based on realtime and retrospective National Land Data Assimilation System (NLDAS) with multiple land models.

Compared to the existing operational US Drought Monitor, NLDAS-based Monitor System is 1) objective2) quantifiable3) reproducible4) can manifest short & long time scales

NLDAS-Based Drought MonitorOperational Drought Monitor

New Drought Prediction

MultipleLand/Hydrologic

Models

Climate Forecasts (GCMs and Official)

Initial land conditions

Downscaling & bias correction