Unidata:Providing data, developing tools, and

building community for the geosciences

Joshua YoungCommunity Services Manager

March 2015

Unidata: Community Driven

• Unidata means the entire community • Unidata Program Center (UPC) means the staff in

Boulder• UPC is governed by two committees composed of academic

community members ( with ex officio participation from government partners, etc).• The Users Committee advises on our services for the

community• The Strategic Advisory Committee advises on the

management of the UPC

• There are many opportunities for community involvement!

Unidata: Services

• Facilitate access to (real-time) data by the University research and education community

• Support the community in their use of the data• Help advocate on behalf of the community• Develop open source tools and infrastructure for

data access, analysis, visualization, and data management

• Advance metadata standards for the earth science community

• Support for users of our services at support@unidata.ucar.edu

Internet Data Distribution (IDD): Real-Time Data Distribution

The “Push” Model

Over 200 sites. Approx 15 GB/hourUnidata’s LDM• Protocol and client/server software• Event-driven data distribution• Supports subscription to subsets of data feeds

Unidata: The Products

The IDD and LDM Data:

• Forecast Model Outputs from NCEP, FNMOC, CMC

• Satellite data from GOES East & West; and COSMIC

• RADAR Level II, III, & National Mosaics

• Lightening data• Aircraft-borne Observation Data • SuomiNet• Textual Weather Bulletin Data

Thematic Real-time Environmental Data Distribution Services (THREDDS)

•THREDDS is middleware to bridge the gap between data providers and data users.

•The goal is to simplify the discovery and use of scientific data and to allow scientific publications and educational materials to reference scientific data.

•The THREDDS effort consists of two main areas: the THREDDS Data Server (TDS) and the Common Data Model (CDM) / netCDF-Java library.

Unidata’s TDS’

Visualizing and Analyzing Data

The Integrated Data Viewer (IDV)Developed and supported at the UPC

AWIPS II

How Does AWIPS II Work?

Visualizing and Analyzing Data

GEMPAK McIDAS-X

Visualization tools supported by the UPC but developed elsewhere

Network Common Data Form (netCDF)

• A set of software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.

• Data in netCDF format is: • Self-Describing �• Portable �• Scalable �• Appendable �• Sharable �• Archivable�

RosettaRosetta is a web-based service that provides an easy, wizard-

based interface for data collectors to transform their datalogger generated ASCII output into:

• Climate and Forecast (CF) compliant netCDF files,

• Complete with metadata describing what data are contained in the file

• Instruments used to collect the data

The use of CF compliant netCDF files allows the use of standard services; however, with knowing that the observational community appreciates the ease of use of ASCII files, methods for transforming the netCDF back into a user defined CSV or spreadsheet format are also built-in.

RAMADDAThe Repository for Archiving, Managing and Accessing Diverse Data (RAMADDA), was originally developed at the Unidata Program Center and is now an Open Source project managed by Geode Systems, LLC.

RAMADDA is a full-featured scientific data management system, allowing you to:

• store virtually any type of scientific data

• manage access to and sharing of the data

• generate quick visualizations of many data types

Unidata continues to be actively involved with the RAMADDA community in numerous ways:

• RAMADDA is tightly integrated with the IDV

• Unidata's Teaching Resources project relies on RAMADDA

• Unidata's Motherlode RAMADDA server makes a wide selection of observational data and forecast model output available

Teaching Resources Network

The Teaching Resources Network allows educators to create, access, and share collections of resource material related to atmospheric, oceanic, and other earth system phenomena.

It is designed for sharing scientific data and products, teaching tools such as lesson plans and guided exercises, and tools for displaying data.

To access the Teaching Resources Network visit:• http://motherlode.ucar.edu/repository/alias/

TeachingResourceNetwork

Unidata: Community Opportunities

• Annual Training from July 20th to August 5th covering Unidata tech and Python, TDS, LDM, AWIPS II, and IDV with RAMADDA

• Unidata Users Workshop June 22nd to 25th

• Nominations for the DeSouza Award and committee memberships

• Summer internships

• Equipment awards

Unidata: The Future The Challenges/Opportunties:• Exponential growth in data• Decreasing community capacity for

hardware and system administrationSome of our Responses:• Move to the Cloud• Leverage the community support for

Python

Unidata: The Contacts

The UPC:• http://www.unidata.ucar.edu/• support@unidata.ucar.edu • @unidata• www.facebook.com/Unidata Support

Me:• jwyoung@ucar.edu• @unidata_josh• www.slideshare.net/JoshYoung8• 303-497-8646

Unidata is one of the University Corporation for Atmospheric

Research (UCAR)'s Community Programs (UCP), and is funded

primarily by the National Science Foundation (Grant NSF-1344155).

Back-up Slides

NCEP Model Data Description

FNMOC Model Data Description

CMC Model Data Description

COSMIC Satellite Date Description

GOES Satellite Data Description

Level II RADAR Data Description

Level III RADAR Data Description

National Mosaics RADAR Data Description

Lightening Data

Aircraft-Borne Observation Data (ACARS)

GPS Meteorological Data (SuomiNet)

Textual Weather Bulletin Data