Date post: | 27-Mar-2015 |
Category: |
Documents |
Upload: | miguel-hensley |
View: | 214 times |
Download: | 0 times |
1
Large-Scale Data Management Challenges
Climate, Water, and Weather Data
Kenneth GalluppiDirector, Disaster and Environmental Programs
Renaissance Computing InstituteUniversity of North Carolina at Chapel Hill
NOAA - National Climatic Data CenterEd Kearns
2
Goal of Collaborations
• Enable cutting edge, Grand Challenge multidisciplinary science through the federation of data-grids of climate, hydrological, and weather data, with other geospatially and socially relevant datasets.
– Understanding of regional impacts of climate change on water availability and society trends
– Understanding and prediction of catastrophic weather-driven events under climatatic change
– Communicate risk/crisis knowledge non-specialists
3
Challenges of Data• Integration of Large, Multidisciplinary Datasets
– NCDC and NOAA Centers, SDSC, and others– Discover, access, integration, utility [not store/retrieve]
• Linkage of Datasets to Computational Models– Input/outputs for real-time model forecasting– Model-to-observation comparison– Climatic models for reanalysis and prediction
• Access to Large Reference Data– Climate Reanalysis Datasets, 1 PetaByte– NWS DataCube for aviation and emergencies
4
Collaboration and Datagrids
National ClimaticData Center
Emergency ManagementResearch Program
Federal Agencies Academic Research
NOAA Mission:
Weather & Water Serve Society’s Needs for Weather and Water Information
Ecosystems Protect, Restore, and Manage the Use of Coastal and Ocean Resources through an Ecosystem Approach to Management
Climate Understand Climate Variability and Change to Enhance Society’s Ability to Plan and Respond
To understand and predict changes in Earth’s environment and conserve and manage coastal and marine resources to meet our nation’s economic, social, and environmental needs
NCDC’s Place in NOAA’s MissionNCDC’s Place in NOAA’s Mission
Commerce & Transportation Support the Nation’s Commerce with Information for Safe,Efficient, and Environmentally Sound Transportation
Mission Support Provide Critical Support for NOAA’s Mission
NOAA Goals:
Data Supports NOAA/NCDC Mission
Data supports NOAA/NCDC Mission
• NCDC will need to function in a wider information landscape with a NOAA Federated Archive– Support distributed data management and services
• Interoperable with DataNet, Earth System Grid, GEO-IDE, EOSDIS, etc.– netCDF, LDM, CF conventions, ISO 19115-2
• Move out of the Box and into the Cloud (networked)– Utilize highly distributed storage and computing (RENCI, Oak Ridge
National Lab
• Implement supporting technologies to enable interoperability with Designated Communities (OGC, WMS/WFS)
• Institute rules-based data management to enable true federation of NOAA Centers of Data – iRODS
Comprehensive Large Array-data Stewardship System (CLASS) Storage
(reanalysis)
The National Environmental Data ArchiveThe National Environmental Data Archive
NOAA’s Data Centers Will Function in a NOAA’s Data Centers Will Function in a Wider Information LandscapeWider Information Landscape
ORNL,ESG
NSF DataNet
DAPs Data Mgmt
IPCC International Sources
International Sources
NEAAT
Climate Services using Federated DB’sNOAA’s Data Centers will need to provide access to petabytes of data that are distributed across multiple NOAA facilities
Be able to integrate these data with data from other disciplines (environmental, biological, social, etc..) that are distributed on other databases both in the public and private sector domain
Export data to common data formats - Shapefile, Well-Known Text, Arc/Info ASCII GRID, Gridded and Raw NetCDF, GeoTIFF and KMZ (Google Earth)
Coordinated, efficient,integrated, interoperable
Data Systems
Space Observations
Ocean Observations
Land Surface Observation
Atmospheric Observations
Discipline-Specific View Whole-System View
Current systems are program-specific, focused, individually efficient.But incompatible, not integrated, isolated from one another and from wider environmental community
1
Support :Disaster reductionHuman HealthClimateWater ResourcesWeatherOcean ResourcesAgriculture & Land-UseEcosystems
NOAA/NCDC Climate Services
11
NCDC-RENCI Potential Use Cases
• Catastrophic Event Modeling and Observations• Climate Reanalysis Datasets
– Climate records everywhere, for 30 years– 1-PetaByte– Regional and local sub-setting– Ten’s of thousands of users
• Multi-sensed Gridded Precipitation Climatology• Extreme Event Climatology• Green Energy, physical-social science Integration
12
High Level View of HIS Service Oriented ArchitectureAs of October 2009, 1,867,108 sites and4,336,790,286 data values where available through the HIS from federal, state, and academic data providers.
There have been 543,144 “GetValues” data requests from Feb 2008 to Oct 2009 .
http://his.cuahsi.org
Discovery
Hydroseek
Analysis
MATLAB , Excel , GIS, R, …
Modeling
Web services and WaterMLto transmit hydrologic
data in a standard way
GetSitesGetSiteInfoGetVariableInfoGetValues
Access
HydroDesktop
HIS Centralcatalog hydrologic data and metadatastore and
share hydrologic data
HIS Server
ODM
Observatories
publication and archival
of field data
3rd- Party Servers
include data from
others
HydroModeler
Hydrology Community
13
HIS Service Oriented Architecture
Maximize Data Access and UtilityMaximize Data Access and Utility
15
Data and Model Integration Neededto Support Hydrologic Science
ObservationsHydrologic Models
Weather and Climate Models
Physical Data
Socioeconomic Data
CUAHSI HIS
DFC
RHESSys
TOPS
ADAS
Meteorology, Hydrology, Ecological Models
WRF RHESSYSHEC-RAS
ADCIRCADCIRC
Scientific Research
Historical Re-Analysis
Disaster Planning
Disaster Response
Agricultural Forecasts
Ag Decision Support
Public Dissemination
Economic Planning
etc …
Sensor Data Bus
TOPS
StateClimateOffice
Sensor Cloud• National Weather Service• Department of Transportation / FAA• USGS NWIS, USFS• Buoys, Stream Gauges, Soil Moisture• People with mobile devices • etc …
CHPS
Enablement
17
Use Case: National Water Model
Terrain in the Neuse River Basin, NC constructed from 390 million LiDAR measurements
Flooding in the Mississippi River Basin, August 1993 observed from satellite imageryHydrologic scientist have expressed a “grand research
challenge” of building a National Water Model for flood and drought applications.
Achieving this goal will require a system like DFC to handle the massive data requirements.
Source: nasa.gov
Source: terrain.cs.duke.edu
18
CUAHSI Case Study• Hydrology Grand Challenge Problem: National Water
Model– How much water is available in the Nation’s water resources?– Currently, hydrologic models are implemented at the watershed-scale
(county)– Hydrologists plan to scale physically-based models to national level
• Provide CI, Policies & Sustainability for Water Model Data– Gathering, analysis, dissemination and preservation– Policies for quality control, metadata harvesting, versioning and usage– Enables the data required for real-time analysis for flood and drought
modeling– Enables integrating data from “new sources”– Enables new science, outreach, decision making and disaster recovery– Integration of Predictive Models, Real-time Data and Historic Data
• Technical Solutions– Too many systems/solutions, home grown to programs (CUAHSI)– Standards (ODM, OGC, Virtual USA, etc.– Federal enterprises
– NOAA, CLASS general, heavy system– Oracle front end to large tape system
• Unique• Handling large sets with limited skills• Multidisciplinary, formats are not enough, but knowledge• Federal
– Has to work, has to preserve– Observation systems are getting more complex– Users are more sophisticated and demanding more