Establishing a User-Driven, World-Establishing a User-Driven, World-Class Oceanographic Data Center by Class Oceanographic Data Center by the Right People, in the Right Place , the Right People, in the Right Place , and at the Right Timeand at the Right Time
L. Charles Sun
National Center for Ocean Research
20-24 June, 2005, Taipei, Taiwan
2
OutlineOutline
1. Time, Place, and People2. Steps in Establishing an NODC3. Mission and Role of an NODC4. QC and QA5. Products and Services6. Information Technology7. Organizational Considerations and Chart8. “Collaboratory” 9. IDARS, Argo & GTSPP: Three examples of “Collaboratories”10. Data Portal: “Gateway” to Ocean Data11. Climate Data Portal: The Proven Prototype12. Other Technologies for the Collaboratory13. The Future
3
Time, Place, and PeopleTime, Place, and People
Time: Since 1975 ~Place: The Center of the worldPeople: We are the right people
4
Steps in Establishing an NODC - ISteps in Establishing an NODC - I
1 Recruit a team of interested parties to propose a mission and organizational model for the center.
2 Construct a draft mission.3 Conduct negotiations with the potential
partners.
5
Steps in Establishing an NODC - IISteps in Establishing an NODC - II
4 Prepare a draft administrative organization.
5 Prepare a final version of the mission and information on partnerships for final approval.
6
Organization ChartOrganization Chart
Office of the DirectorDirector
Deputy DirectorAssociate Director
Ocean DynamicsChief
Data Base ManagementChief
Information Technology Chief
Staff
Data ProcessingResearch Data and
Product Development
Data ArchivalDatabase Development and
Maintenance
NetworkingOperating System Maintenance
Hardware/software purchase and Maintenance
LibraryChief
Service
7
Mission of an NODCMission of an NODC
To safeguard versions of oceanographic data and information.
To provide high quality data to a wide variety of users in a timely and useful manner.
8
Roles of an NODCRoles of an NODC
Conventional role – as a minimumContemporary role – in response to
advances in data collection and information technology
9
Conventional Role - IConventional Role - I
Receive data, perform quality control, archive and disseminate it on request.
Keep copies of all or part of its data holdings in the format in which the data were received.
Developing and protecting national archives of oceanographic data
10
Conventional Role - IIConventional Role - II
Produce and provide inventories of its holdings on request.
Referral of the users to sources of additional data and information not stored in the NODC.
Participate in international oceanographic data and information exchange.
11
Contemporary Role - IContemporary Role - I
Receive data via electronic networks on a daily basis, process the data immediately, and provide outputs to the user or to the data collectors for data in question.
Report the results of quality control directly to data collectors as part of the quality assurance module for the system.
12
Contemporary Role - IIContemporary Role - II
Process and publish data on the Internet and on CD/DVD-ROMs.
Publish statistical studies and atlases of oceanographic variables.
Performing a level of quality control on its data holdings
13
Quality Control and AssuranceQuality Control and Assurance
Data can be detected easily by a data center Obvious errors such as an impossible date and
time and location
Data cannot usually be detected by a data center Subtle errors such as an instrument may be off
calibration
14
Information Technologies - IInformation Technologies - I
Data Storage/ArchiveData ProcessingLocal Area NetworkingWide Area Networking – the Internet
(and the GTS)
15
Information Technologies - IIInformation Technologies - II
Publishing DVD/CD–ROMsGraphics Capability (Graphical
Information System)Software Development &
ImplementationHardware procurement &
Maintenance
16
Products Development - IProducts Development - I
Work with the client to determine what the real need. Examples of data products include atlases, datasets of ocean observations filtered by area, time and variables observed
17
Products Development - IIProducts Development - II
Review the world wide web sites of existing NODCs for ideas and examples of data and Information products.
18
ServicesServices
Providing directory and inventory information
Acting as a referral centerReceiving data for specific
processing followed by delivery of the processed data
19
Organizational ConsiderationsOrganizational Considerations
A centralized data center A distributed data center
Centers of Data : “Data Portals” or “Virtual Collaboratories”
Data CenterData Center
Center of Data ACenter of Data A Center of Data BCenter of Data B Center of Data CCenter of Data C
20
What is a Collaboratory?What is a Collaboratory?
The fusion of computers and electronic communications has the potential to dramatically enhance the output and productivity of researchers. A major step toward realizing that potential can come from combining the interests of the scientific community at large with those of the computer science and engineering community to create integrated, tool-oriented computing and communication systems to support scientific collaboration. Such systems can be called "collaboratories."
From "National Collaboratories - Applying Information Technology for Scientific Research," Committee on a National Collaboratory, National Research Council. National Academy Press, Washington, D. C., 1993.
21
AcknowledgementAcknowledgement
Soreide, N. N. and L. C. Sun, 1999:
Virtual Collaboratory: How Climate Research can be done Collaboratively using the Internet. U.S. – China Symposium and Workshop on Climate variability, September 21-24, 1999, Beijing, China
Presented by Len Pietrafesa, North Carolina State University.
22
Collaboratory Infrastructure Collaboratory Infrastructure Data Portal
– Computer and networking hardware and software – Increased network bandwidth/speed– Next Generation Internet (NGI) connection
Visualization– Interactive Java graphics– 3D, Virtual Reality, collaborative virtual environments– immersion technology CAVE, ImmersaDesk...
Relationships:– Observing System Project Offices– Research community, Academia...– Other Collaboratory nodes– Steering Committee
23
International Steering Committee
CollaboratoryPartner
CollaboratoryPartner
Collaboratory
Partner
Collaboratory Partners & CustomersProviders of Data & Information
Users of Data & Information
Observations&
Satellite Groups
Modeling&
ForecastingGroups
ResearchGroups
New Users Educational Administrators General Public
Structure of the Collaboratory for Ocean Research
24
IDARSIDARS** as an example... as an example...
• Real-Time Coastal Water Temperature Data• Real-Time Argo Profile Data• Real-Time Global Temperature and Salinity
Profile Data• Time Series Data• NOAA CoastWatch AVHRR SST Images
http://www.nodc.noaa.gov/idars/
*Interactive Data Access and Retrieval System
25
Argo as an example...Argo as an example...
26
GTSPPGTSPP** as an example... as an example...
** Global Temperature-Salinity Profile Program
27
Argo and GTSPP Argo and GTSPP
Argo and GTSPP set a standard in the international ocean data management community
Data dissemination in near-real time– Researcher involvement has assured data quality
Benefits of data dissemination– Wide use of Argo and GTSPP data – Traditional research, modeling, forecasting groups– Related disciplines, educational, administrative, public
With recent advances in technology, we can do much more...
28
Distributed Object TechnologyDistributed Object Technology
Data servers and datasets are objects – software packages of procedures and data that contain their own context
Solid, commercial underpinning for distributed object technology in the ocean sciences
30
The Data Portal: a “gateway” to ocean dataThe Data Portal: a “gateway” to ocean data
Why do we need a Data Portal?– Each center of data provides a highly customized Web
sites for their data• but different datasets have different navigation and interface
characteristics• so the user faces a bewildering spectrum of data access
interfaces and locations
Data Portal is single, uniform, consistent “gateway” to ocean data in a common format
• User goes to a single location and sees a consistent interface• Complements the customized data access
31
Data Portal/Visualization/CollaborationData Portal/Visualization/Collaboration
Traditional users:ModelersForecastersResearchers
New users:EducatorsStudentsGeneral Public
Data & Data & Information UsersInformation Users
Distributed data Observed data Satellite data Data and information products Model outputsVisualizationUniform network accessUniform network access
32
WebBrowser
JavaApplication
User
Network
CORBA*
Client Support
Java Servlet
Graphics
One or more Web Servers
TAO data support
CORBA*
Data
Observing System Server
Data
Common Object Request Broker Architecture (CORBA) is an industry standard Middleware. CORBA is used in the NOAAServer software from which this effort will leverage. Based on performance indicators, Java Remote Method Invocation (RMI), an alternative middleware, could easily be substituted for CORBA.
CORBA*
Network
Data ServerData Portal
33
WebBrowser
JavaApplication
User
Network
CORBA*
Client Support
Java Servlet
Graphics
One or more Web Servers
Drifter Data support
CORBA*
Data
TAO data support
CORBA*
Data
Observing System Servers
Data
Common Object Request Broker Architecture (CORBA) is an industry standard Middleware. CORBA is used in the NOAAServer software from which this effort will leverage. Based on performance indicators, Java Remote Method Invocation (RMI), an alternative middleware, could easily be substituted for CORBA.
CORBA*
Network
Data ServersData Portal
34
WebBrowser
JavaApplication
User
Network
CORBA*
Client Support
Java Servlet
Graphics
One or more Web Servers
Drifter Data support
CORBA*
Data
TAO data support
CORBA*
Data
Observing System Servers
In-Situ/Satellite data support
CORBA*
Data
In-Situ/Satellite Data Servers
Model data support
CORBA*
Data
Model Output Servers
Data
Gridded data support
CORBA*
Data
Gridded Data ServersCommon Object Request Broker Architecture (CORBA) is an industry standard Middleware. CORBA is used in the NOAAServer software from which this effort will leverage. Based on performance indicators, Java Remote Method Invocation (RMI), an alternative middleware, could easily be substituted for CORBA.
CORBA*
Network
Data ServersData Portal
35
How do we build a Data Portal?How do we build a Data Portal?
Build on a proven prototype– connects 5 geographically distributed data
servers in Silver Spring, Boulder, Seattle– CORBA for network connections– unified interactive Java graphics – data from distributed servers are co-plotted
together on the same axis on the users desktop
http://www.pmel.noaa.gov/~nns/noaaserver/nodc-coads-tao.htmlhttp://www.pmel.noaa.gov/~nns/noaaserver/coads-tao-raster.html
36
BoulderCO
Prototype Data Portal: CDPPrototype Data Portal: CDP**
SeattleWA
Silver Spring
MD
HonoluluHI
*Climate Data Portal
37
Climate Data Portal Sample PlotsClimate Data Portal Sample Plots
38
Data Selection : Web InterfaceData Selection : Web Interface
Utilizes CORBA for network connections.
Utilizes EPIC Web Technology:– Java Applets– JavaScript– Java Servlets
Searches data by keywords, location and time ranges.
39
Web Interface sWeb Interface screen Shotscreen Shots
40
Other Technologies for the Other Technologies for the Collaboratory:Collaboratory:
Networks (100 Megabits/sec today, 10 Gigabits/sec in future)– Next Generation Internet (NGI) and Internet 2
Visualization– Interactive Java graphics– 3D, Virtual reality– Immersion technology
Collaboration tools– high-speed telecommunications systems for advanced
collaboration applications– tele-immersion systems allow individuals at different locations to
share a single virtual environment– Use networks not airplanes for collaboration
41
Virtual RealityVirtual Reality
Virtual Reality lets the scientist touch the data, move into it, and see it from different viewpoints– The realism of virtual reality enables the scientist and the lay
person to understand complex ideas more easily – Scientists using virtual reality affirm this new technology
discloses features of their data and model outputs which were undiscovered with standard visualization techniques
Virtual reality can be approachable and affordable Widens audience for scientific data and information
– Government administrators and decision makers– Educators and students– General public
Some examples follow…
Courtesy of Nancy N. Soreides, PMEL
42
Why use Virtual Reality?Why use Virtual Reality?
Virtual reality modeling language (VRML) rendering of temperatures and sea surface topography along the equator in the tropical Pacific, viewed from South America, showing the dynamics of El Nino and La Nina.
Using an inexpensive PC and a web browser with a free plug-in, the images can be rotated, animated, and zoomed. Changes in the equatorial Pacific during El Nino and La Nina are clearly understood by scientist and layman. http://www.pmel.noaa.gov/toga-tao/vis/vrml/ or http://www.pmel.noaa.gov/vrml
El Nino La Nina
Courtesy of Nancy N. Soreides, PMEL
43
Stereographic Virtual RealityStereographic Virtual Reality3D, interactive virtual reality visualizations are not difficult for a scientist to create or to view, from the web or from the desktop, and the effect can be enhanced dramatically by including the capability of stereographic viewing.
With a PC and a 99-cent pair of red/green sci-fi glasses, the spheres and vectors will pop out of the page in stereo, revealing the true 3D location of the fish, the steep slopes of the bathymetry, and the vertical motions near the submarine canyon.
The images can be rotated, animated and zoomed. http://www.pmel.noaa.gov/~hermann/vrml/stereo.html
Fish larvae and velocity vectors in a submarine canyon, from a circulation model of Pribolof Canyon in the Bering Sea. Use red/green glasses to see images on the right in stereo.
Stereo
Stereo
Courtesy of Nancy N. Soreides, PMEL
44
Immersive devices provide the graphical illusion of being in a three-dimensional space by displaying visual output in 3D and stereo, and by allowing navigation through the space.
Navigating through our virtual environments and viewing the data from different vantage points greatly increases our ability to perform analysis of scientific data.
The impact of such visualizations in person is stunning, and must be experienced by the scientist to be fully comprehended .
Users of these advanced immersion technologies affirm that no other techniques provide a similar sense of presence and insight into their datasets.
Immersive Virtual RealityImmersive Virtual Reality
Courtesy of Nancy N. Soreides, PMEL
45
The CAVEThe CAVE
The CAVE is a multi-person, high resolution, 3D graphics video and audio virtual environment. The size of a small room (10x10x10 foot), it consists of rear-projected screen walls and a front-projected floor.
Using special "stereoscopic" glasses inside a CAVE, scientists are fully immersed in their data. Images appear to float in space, with the user free to "walk" around them, yet maintain a proper perspective.
The CAVE was the first virtual reality technology to allow multiple users to immerse themselves fully in the same virtual environment at the same time.
View of the CAVE
Scientist inside the CAVE
CAVES have been deployed in academia, government, and industry, including NASA, NCAR, NCSA, Argon National Laboratory, Caterpillar Corp., General Motors, among others.
http://www.pyramidsystems.com/CAVE.htmlCourtesy of Nancy N. Soreides, PMEL
46
The ImmersaDeskThe ImmersaDesk
Courtesy of Nancy N. Soreides, PMEL
47
The FutureThe Future
“The development of scientific data manipulation and visualization capabilities requires an integrated systems approach … [including] the end-to-end flow of data from generation to storage to interactive visualization, and must support data retrieval, data mining, and sophisticated interactive presentation and navigation capabilities.”
“Data Exploration of petabyte databases will required both technology development and altered work patterns for research scientists and engineers.”*
* Data and Visualization Corridors, Report on the 1998 DVC Workshop Series, Edited by Paul H. Smith and John van Rosendale, Sponsored by the Department of Energy and the National Science Foundation, 1998.
Courtesy of Nancy N. Soreides, PMEL