The IPY Data and Information Service—How do we get there?

IPY Data WorkshopCambridge, England3 March 2006

World Data Center for Glaciology, BoulderFacilitating the international exchange of snow and ice data

Mark A. Parsons IPY Data Policy and Management Sub-committeeIPY Data and Information ServiceElectronic Geophysical Year

IPY1

IPY2

IGY (IPY3) IPY4

IPYDIS; Mark A. Parsons, 3 March 20063 QuickTime™ and aGraphics decompressor

are needed to see this picture.

What will IPY4 bring? The Challenge!• Will researchers be able to find all the data

relevant to their research and see relationships between data sets. Access

• Will they be able to merge and integrate different data sets across experiments and disciplines? Interoperability

• Will they be able to subset, visualize, and transform the data? Usability

• Will they be able to retrieve and understand IPY4 data in 2050? Preservation

IPY4

IPYDIS; Mark A. Parsons, 3 March 20064 QuickTime™ and aGraphics decompressor

are needed to see this picture.

Organization of IPY Data Management

IPY Joint Committee

Data Policy & Management Subcommittee• scientists• data managers• funding agencies

ProgrammeOffice Data

& InformationService

eGY

Projects Data Centers, Virtual Observatories, etc.

Users

IPYDIS; Mark A. Parsons, 3 March 20065 QuickTime™ and aGraphics decompressor

are needed to see this picture.

IPYDIS; Mark A. Parsons, 3 March 20066 QuickTime™ and aGraphics decompressor

are needed to see this picture.

Alternate Views of the DIS

DIS?

DIS?

IPYDIS; Mark A. Parsons, 3 March 20067 QuickTime™ and aGraphics decompressor

are needed to see this picture.

Systems and Innovation

31%40%

28%23%

15% 18%

53% 33%46%

49%

51%54%

16%27% 26% 28%

34%28%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1994 1996 1998 2000 2002 2004

The Standish Group’s “CHAOS report”. An assessment of over 40,000 IT application projects

Succeeded

“Challenged”

Failed

“We're entering a new world in which data may be more important than software.”

- Tim O'Reilly

IPYDIS; Mark A. Parsons, 3 March 20068 QuickTime™ and aGraphics decompressor

are needed to see this picture.

The People Part

Service counts.

“A striking proportion of project difficulties stem from people in both customer and supplier organisations failing to implement known best practice.”

— Oxford University/Computer Weekly survey of public and private sector IT projects (emphasis added)

However, people are much more able to adapt to change, uncertainty, and messy systems

IPYDIS; Mark A. Parsons, 3 March 20069 QuickTime™ and aGraphics decompressor

are needed to see this picture.

The People Part: Science and Data Management

• Many have stated the need to involve scientists in data management, but…

• It is also important to involve data managers in conducting science.• Field Experiments:

• 20% increase in data quality (Parsons, et al. 2004)• 70% of experiment cost is data assembly

(Bernhardsen 1992, Longley, et al. 2001)

• Observing systems

IPYDIS; Mark A. Parsons, 3 March 200610 QuickTime™ and aGraphics decompressor

are needed to see this picture.

Preservation and Access—Two Peas in a Pod

Scientific Data Stewardship:• “preservation and responsive supply of

reliable and comprehensive data, products, and information for use in building new knowledge to…”

—US Global Climate Research Program, 1998

• “the long-term preservation of the scientific integrity, monitoring and improving the quality, and the extraction of further knowledge from the data”

— H. Diamond et al., NOAA/NESDIS, 2003

IPYDIS; Mark A. Parsons, 3 March 200611 QuickTime™ and aGraphics decompressor

are needed to see this picture.

Access. What is it?

• Preservation requirements are well defined in the Open Archive Information System (OAIS) Reference Model, but

• No similar model for access requirements • Not even a common definition of “access” and

what restricts it• Unique access requirements for social science

data and non-digital collections (physical samples, photographs, audio, etc.)

“Facts are terrible things if left sprawling and unattended…”

- Norman Cousins

IPYDIS; Mark A. Parsons, 3 March 200612 QuickTime™ and aGraphics decompressor

are needed to see this picture.

Standards—Essential but Cumbersome

• Some Possibilities:• ISO19115 metadata standard• OAIS Reference Model• OGC data transfer standards• Other OGC Standards• “Web Services” (WSDL, SOAP)• Other XML-based standards (GML, OAI-PMH,

RSS,…)• Etc, etc,

• No New Standards!

“We must not … start from any and every accepted opinion, but only from those we have defined — those accepted by our judges or by those whose authority they recognize.”

—Aristotle c. 350 BC

IPYDIS; Mark A. Parsons, 3 March 200613 QuickTime™ and aGraphics decompressor

are needed to see this picture.

011000101001001111010111000111101100101010001110011100101010011101010100111000110101101000010000100101001001010110010010001010100100100101010101001010100101001010100000111110010110101010110100010111101011010110101010011000101001001111010111000111101100101010001110011100101010011101010100111000110101101000010000100101001001010110010010001010100100100101010101001010100101001010100000111110010110101010110100010111101011

Issues with the Data Itself

Formats:• Archives and users may have different

needs• Consider four themes (Raymond, 2004)

• Transparency• Interoperability• Extensibility• Storage or transaction economy

“We often get blinded by the forms in which content is produced, rather than the job that the content does.” - Tim O’Reilly

IPYDIS; Mark A. Parsons, 3 March 200614 QuickTime™ and aGraphics decompressor

are needed to see this picture.

Other Questions and Issues• How interoperable can we be? What does “portal”

mean to you?• How do maximize use of existing data systems and

structures? CODATA? WDCs?• How does IPY data fit into current operational

systems?• What about GEOSS—can IPY be a prototype?• Which technological trends can help us?

(ontologies, virtual observatories, portals, etc.)• How do we incorporate historical data?• Need a solid business model esp. for the long-term

IPYDIS; Mark A. Parsons, 3 March 200615 QuickTime™ and aGraphics decompressor

are needed to see this picture.

Breakout Groups

Methods for Data discovery—portalsPaul Berkman, room 370

Ensuring data submission and publicatio--carrots and sticks

Jim Moore, room 303bSemantics, ontologies, and language

Heather Lane, main room

IPYDIS; Mark A. Parsons, 3 March 200616 QuickTime™ and aGraphics decompressor

are needed to see this picture.

Charge to Breakout Groups

1. Determine rapporteur2. Explicitly define problem(s)3. Identify options to solve problem4. Recommend steps to solve problem5. Present to whole group for feedback6. Revise7. Write up results to be part of a larger

workshop report. Include outstanding issues, next steps, etc.

Workshop report will be presented to broader IPY research community for feedback and buy in.

IPYDIS; Mark A. Parsons, 3 March 200617 QuickTime™ and aGraphics decompressor

are needed to see this picture.

Data Systems Today

© N. Carr 2006

IPYDIS; Mark A. Parsons, 3 March 200618 QuickTime™ and aGraphics decompressor

are needed to see this picture.

What they need to become

© N. Carr 2006