Ocean Observatories Initiative OOI Cyberinfrastructure Overview Matthew Arrott VMware Presentation March 5, 2010 OOI CI LCO Review, Feb Goals for this Presentation OOI CI LCO Review, Feb Overview The OOI Integrated Observatory OOI Network Architecture OOI Logical Architecture OOI CI LCO Review, Feb Distributed System of Systems Building Blocks Messaging Fabric Exchange Spaces Capability Container Execution Environments Federation and Governance OOI CI LCO Review, Feb OOI Program Overview OOI CI LCO Review, Feb OOI Integrated Observatory OOI CI LCO Review, Feb Integrated Observatory OOI CI LCO Review, Feb Science & Education Operations Interactive Ocean Observing Interactive Ocean Modeling & Data Assimilation Automated Data Product Generation Discipline-Driven Semantic Organization of Data Interactive Instrument Network Integrated Observatory Management User-Driven Integration of Resource Observatory Activity Model 9 Goal after Release 5 OOI CI LCO Review, Feb Network Architecture OOI CI LCO Review, Feb Network Architecture OOI CI LCO Review, Feb International Research Networks OOI CI LCO Review, Feb National Research Networks OOI National Network OOI CI LCO Review, Feb OOI CI LCO Review, Feb Network Deployment OOI CI LCO Review, Feb OOI Logical Architecture Architecture documentation available at:ecture+and+Design OOI CI LCO Review, Feb Release-1 OOI CI Functions Instrument and platform interfaces Instrument and platform control from the observatory Direct access to instruments and platforms Data acquisition from sensors and external observatories Management of science data and other observatory information Data inventory and metadata annotation Data ingestion, i.e. canonicalization and metadata annotation Data discovery, cataloging and presentation Data distribution, replication, streaming Data persistence, assurance, retrieval Process and service execution Virtualization of computing Elastic scaling to demand High availability services Service-orientation Reliable message based asynchronous communication Security: identity and policy management Multi facility system integration Sensing & Acquisition Data Management Common Operating Infrastructure Common Execution Infrastructure Interface Adapters Application Infrastructure OOI CI LCO Review, Feb Resource the Base Object CI governed Resource Standard and user metadata attributes (in OOI convention) References to other resources Categories Information resource Physical (stateful) resource Taskable resource OOI CI LCO Review, Feb The Capability Container Ingredients: Capability: function, service, component Plumbing: Messaging, security Container: drop it all in as needed Packaging: label it, tape it shut done! OOI CI LCO Review, Feb Secure Reliable Messaging Capability Container Capability Container Capability Container Capability Container Capability Container Capability (Service) Capability (Service) Capability (Service) Capability (Service) Capability (Service) OOI CI LCO Review, Feb The Exchange Applications communicate through Exchange Spaces Exchange Spaces are namespaces of communicators Applications need to enroll in Exchange Spaces Governance applies within Exchange Spaces OOI CI LCO Review, Feb Exchange Space and Points Exchange Points are the message routing and queuing resources of Exchange Spaces Communicators play the role of Producer, Consumer, and Distributer (Broker) OOI CI LCO Review, Feb Exchange Space Exchange Space is comprised of Distributed Application Facility (DAF) Distributed IPC Facility (DIF) DIF Member Process OOI CI LCO Review, Feb OOI CI LCO Review, Feb Messaging Abstraction Capability Container Capability Container Capability Container Send Message From: name2 To: name4 In: Exchg-Space1 Action: invoke-service OOI CI LCO Review, Feb Common Message Format Capability Container Provides message handling through interceptors After a service sends a message, before it receives a message Message signing and validation Policy enforcement Governance tracking Provides a common message format for all CI messages Based in ACL FIPA message structure Content, encoding, ontology OOI CI LCO Review, Feb Services and Resources as Agents Resource Agent Resource Agent Proxy Agent Capability Container Capability Container Capability Container Capability Container OOI CI LCO Review, Feb Scenario An instrument, a physical resource, is represented by an agent to the system and its users Users request control of the instrument Capabilities are projected into another domain of authority by a proxy agent OOI CI LCO Review, Feb COI Infrastructure Services The Common Operating Infrastructure (COI) is the integration & communication environment for all the other subsystem services OOI CI LCO Review, Feb Integration Strategy Capability Container Secure reliable messaging Exchange Service-oriented architecture Non-central governance COI Infrastructure Services OOI CI LCO Review, Feb CI Interfaces and Subsystems OOI CI LCO Review, Feb Preparing Services for Deployment Instrument Point Acquisition Point Ingest Point Application Point Storage Point Access Portal OOI CI LCO Review, Feb Deployment Scenario OOI CI LCO Review, Feb Deployment Plan and Strategy Deployment of capability containers in multiple execution environments by leveraging virtualization The CEI will make it work CyberPoP (Cyberinfrastructure Point of Presence): physical deployment of one or multiple capability containers Different types OOI CI LCO Review, Feb Available, Elastic Execution OOI CI LCO Review, Feb CEI Service Interactions Planner Controller (Executive) Provisioner Fault Monitoring and Compensation Registry Resource (Taskable) OV2 CEI OOI CI LCO Review, Feb Scenario: Planner and Controller The pattern Planner- Controller-Resource can repeat itself at multiple (nested levels) OV2 CEI OOI CI LCO Review, Feb CEI Architecture Overview OOI CI LCO Review, Feb EPU EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) HA Service (OOI Application) VM (Deployable Unit) VM (Deployable Unit) VM (Deployable Unit) Application Software (Deployable Type) OOI CI LCO Review, Feb CEI Architecture Overview OOI CI LCO Review, Feb OOI CI LCO Review, Feb CEI Service Interaction OOI CI LCO Review, Feb Planner ControllerProvisioner Monitoring Information Deployable Type Registry Context Broker IaaS AdapterIaaS Exchange VM info Proc Info Other Info OOI CI LCO Review, Feb Elastic Processing Unit Elastic Processing Unit (EPU): High- Availability Services with policy based scaling to demand OOI CI LCO Review, Feb EPU Controller Register HA service EPU Worker N1 EPU Worker N1 Msg to N EPU Worker N2 EPU Worker N2 EPU Worker NX EPU Worker NX OOI CI LCO Review, Feb EPU: Prototype 1 EPU design and prototype: Policy-configurable planner/scheduler Scaling policy: size of work queue All components integrated with Magnet/AMQP OOI CI LCO Review, Feb OOI CI LCO Review, Feb Federation & Governance Other areas of computing facilitated greatly by a movement to a communication oriented architecture. OOI CI LCO Review, Feb Federation OOI CI LCO Review, Feb Governance Architecting with RM-ODP J. Putman B OOI CI LCO Review, Feb Distribute Application Framework OOI CI LCO Review, Feb Thank you OOI CI LCO Review, Feb Motivating Governance Administering collaborations Based on framing normative relationships among peers Abstracting away from low-level details OOI, broadly: many stakeholders; many resources; longevity of decades Exchange spaces, narrowly: abstractions for communicating; assembly of multiple topologies for messaging; analogous to traditional enterprise integration patterns OOI CI LCO Review, Feb Elements of a Service Engagement Enactment: doing the domain work what the end user cares most about Administration: captured via contracts Partnerships Rules of encounter Identity Enforcement OOI CI LCO Review, Feb What is Governance? Broadly, administering service engagements IT Governance: How IT resources are administered SOA Governance: How services are created, deployed, removed, Currently, governance is manual Low productivity Poor scalability for fine-grained, real time governance decisions Hidden, implicit considerations yield low confidence and poor maintainability OOI CI LCO Review, Feb Why Governance? Stakeholders using resources to best serve individual and collective needs Share resources in a controlled manner Configure and reconfigure dynamically Enable unanticipated uses for resources Respect human organizational needs In a nutshell, stakeholders administer themselves OOI CI LCO Review, Feb Separation of Concerns Protocol: specifying the interactions among autonomous parties Policy: specifying the decision making of each autonomous party as it participates in various protocols Behavior: specifying the implementation that realizes the interactions OOI CI LCO Review, Feb Exchange Space Use Case OOI CI LCO Review, Feb OOI CI LCO Review, Feb Messaging View of Enrollment OOI CI LCO Review, Feb OOI CI LCO Review, Feb Community Affiliation Use Case OOI CI LCO Review, Feb OOI CI LCO Review, Feb Combined Scenario, Schematically OOI CI LCO Review, Feb OOI CI LCO Review, Feb IdM Architecture Overview OOI CI LCO Review, Feb Conclusions OOI CI LCO Review, Feb Summary OOI CI LCO Review, Feb Next Steps OOI CI LCO Review, Feb Thanks ! OOI CI LCO Review, Feb Scope Release 1: Data Distribution Network (2011) Ingest data (products) from external data sources; both streaming and cataloged Characterize data sources with their metadata attributes (format, structure, meaning) Transform external data into internal OOI data formats, for stream-based distribution and persistence Distribute data via streaming and DAP servers to data analysts and numerical modelers Provide initial instrument integration, control and sensor data acquisition Provide instrument and platform direct access Provide service integration and execution platform OOI CI LCO Review, Feb Scope of Future Releases Release 2: Managed Instrument Network (2012) Fully managed instrument activation and control; ready for deployment on OOI moorings and cable infrastructure Qualified data products based on automated data processing Release 3: On-Demand Measurement Processing (2013) Observatory management and resource scheduling On demand and event driven measurements Data processing workflows Release 4: Integrated Modeling Network (2014) Interactive analysis and synthesis Numerical model integration Release 5: Interactive Ocean Observatory (2015) Full control of data, processes, models, instruments Closed loop, adaptive sensing and platform control OOI CI LCO Review, Feb Release Schedule OOI CI LCO Review, Feb Back-End Infrastructure OOI CI LCO Review, Feb Exchange Points and the DIF OOI CI LCO Review, Feb Message Brokers over DIF