Cyberinfrastructure Status

July, 2011

2

• NSF reverse site visit• Refactoring and cleanup after review preparations• Coordinating Node technology changes (reducing

latency, generalizing)• Authentication• Identity management• Access control• TeraGrid Compute Node design

2011 – First half

3

• Public release of cyberinfrastructure• Synchronization, replication, authentication,

authorization, resolution, discovery, retrieval

• Coordinating Nodes• UNM, UCSB, ORC

• Member Nodes• KNB, Dryad, ORNL-DAAC, AKN, NBII, CUAHSI, replication

targets, Merritt, other Metacat, other Mercury

• Investigator Toolkit• Libraries, Web interface, Zotero, R, Morpho, DataONE

Drive

Goals for 2011

4

The Major Components

Investigator Toolkit

Web Interface Analysis, Visualization Data Management

Client LibrariesJava Python Command Line

Member Nodes

Service Interfaces

Bridge to non-DataONE Services

Data Repository

Coordinating NodesService Interfaces

Object Store Index

Resolution

RegistrationReplication

Discovery

Identifiers

MonitorPreservation

Catalog

Identity Authen

5

Investigator Toolkit SoftwareSearchPortal R Client Morpho

Client LibrariesJava Python Command Line

Member Node Software

Metacat

Coordinating Node SoftwareService Interfaces

Object Store Index

Software Delivered at Public Release

Zotero DataONE FS (Excel)

Dryad

GMN (CUASHI)

(Merritt) Preservation MonitorCatalogIdentifiers

Replication DiscoveryResolution Registration

Mendeley…

DataONE Service Programming Interface (SPI)

6

Tier 1: Public read, synchronization, search, resolveTier 2: Read with access controlTier 3: Write using client toolsTier 4: Able to operate as a replication target

• Some Member Nodes with Tier 4 support and corresponding Coordinating Node functionality required for 2011

Member Node Tiers

7

Node Tier Status Priority ReleaseKNB 4 2 1 2011ORNL DAAC 2 1 1 2011Dryad 2 ? 2 1 2011CUAHSI 2 1 1 2011AKN 4 1 1 2011Merritt 2 0 2 2011 ?Fedora ? 1 3 2012Repl. Targets (GMN) 4 2 1 2011TeraGrid 4 1 2 2012Mercury based 2 1 2 2011 ?Metacat based 4 1 3 2011?

Member Nodes

Status: 0 = Not implemented, 1=partial, 2=functional, 3=completePriority: 1 = High, 2= medium, 3= low

8

Coordinating Node ServicesService Tier Status PriorityNode registry 1 2 1Object storage 1 3 2Identifier resolution 1 3 2Synchronization 1 1 1Discovery 1 1 1Log retrieval 1 0 2Identity mapping 2 2 2Session state 2 2 2Identifier reservation 3 3 3Replication 4 0 2

Status: 0 = Not implemented, 1=partial, 2=functional, 3=completePriority: 1 = High, 2= medium, 3= low

9

Coordinating Node Components

cn_service

cn_rest cn_rest_proxy

Metacatidentity_manager

nodes_registry

synchro-nization

replic-ationindexer

SOLR Postgres

JMS

LDAP

MercurySearch

10

Component Tier Status Priority ReleaseLibclient_python 4 2 1 2011Libclient_java 4 2 1 2011R plugin 2 (3) 1 1 2011D1Drive - FUSE 2 1 1 2011D1Drive - Dokan 2 1 1 2011D1Drive - WebDAV 2 0 3 2012Mercury UI 2 2 1 2011Morpho 3 1 2 2012Excel 3 1 3 2012Zotero, Mendeley 2 1 2 2011

Investigator Toolkit

Status: 0 = Not implemented, 1=partial, 2=functional, 3=completePriority: 1 = High, 2= medium, 3= low

11

Collect

Assure

Describe

Deposit

Preserve

Discover

Integrate

Analyze

Data Lifecycle

11

Morpho

12

Other PiecesComponent Status Priority ReleaseDocumentation 2 1 2011

Member Node test service 1 1 2011Integration testing 2 1 2011Monitoring 1 2 2011Hardware 2 1 2011

Continuous build 2Issue tracking 3Repository 3Conferencing 2

Status: 0 = Not implemented, 1=partial, 2=functional, 3=completePriority: 1 = High, 2= medium, 3= low

13

14

Infrastructure will change over time. • Message between services• Behavior of services, components• Message semantics

How to accommodate change while maintaining operations?

Versions

15

Infrastructure Version: Tag identifying the overall functional capabilities, types, messages supported by the DataONE infrastructure

Component Version:Tag of a specific component indicating the current revision of the component

Example: Version XXX of Metacat supports version YYY of the DataONE infrastructure

Infrastructure + Component Versions

16

1. Changes to Types Schema2. Update test data3. Build libcommon Pyxb / JIBX 4. Propagate changes through libclient5. Propagate changes through remainder of components6. Deploy CN test7. Update test Member Nodes8. Deploy staging9. Update / synchronize content to match production10. Switch staging and production

Where to update all Member Node stacks?Where to update all Investigator Tools

DataONE Infrastructure Release

17

Release Coordinating Nodes• Touches everything deployed – MNs, ITK, monitor, ... • Likely to require content migration, synchronization• Expensive

Update Member Nodes• Vendor driven updates (e.g. core changes to Metacat)• DataONE driven updates (e.g. access policy fix)• Known contact points for updates

Update client tools• Automated version checks?• How to notify users?• How to notify third party developers?

Component Updates

18

For an infrastructure upgrade:• What is the acceptable latency between change

announcement and infrastructure back in sync?• Should DataONE maintain old versions of service

interfaces? If so how many revisions?• Some changes may not be backward compatible• Option to make old version CNs readonly while content

migrated to new systems?

Version Latency and Acceptable Delta

19

New package available with clear indication of affected components, signed off by component developersAffect CNs:

• Install on staging, process metadata updates, lock production, switch.

Affect MNs:• Notify instance administrators of update• Install updates and verify operation• Registry picks up updated services

Affect ITK:• Post update information• Notify active clients of update• Notify third party developers of changes

Administrative Process for Updates

20

21

July • New hardware deployed at UNM, UCSB, ORC• CN development with focus on Auth and synchronization• Integration test services

August • CN development focus on indexing, replication• Deploy CN staging to new hardware• ITK libraries updated• Full Tier 1 functionality deployed

September • Identity mapping, authentication, authorization• Evaluation of Tier 1 functionality• Support Tier 2 operations

October • Support Tier 3 and 4 operations on specific Member Nodes• CNs support replication, load testing of CNs• Evaluation of Tier 2 and initial tier 3 support

November • Final evaluation of functionality• Incorporate or schedule suggestions• Final stress and usability testing

December • Final documentation, testing• Announcement of initial release

General Timeline

22